Null coalescing operators
StackOverflow has many questions <http://stackoverflow.com/search?q=%5Bpython%5D+null+coalesce> on the topic of null coalescing operators in Python, but I can't find any discussions of them on this list or in any of the PEPs. Has the addition of null coalescing operators into Python ever been discussed publicly? Python has an "or" operator that can be used to coalesce false-y values, but it does not have an operator to coalesce "None" exclusively. C# has nice operators for handling null: "??" (null coalesce), "?." (null-aware member access), and "?[]" (null-aware index access). They are concise and easy to reason about. I think these would be a great addition to Python. As a motivating example: when writing web services, I often want to change the representation of a non-None value but also need to handle None gracefully. I write code like this frequently: response = json.dumps({ 'created': created.isoformat() if created is not None else None, 'updated': updated.isoformat() if updated is not None else None, ... }) With a null-aware member access operator, I could write this instead: response = json.dumps({ 'created': created?.isoformat(), 'updated': updated?.isoformat(), ... }) I can implement this behavior myself in pure Python, but it would be (a) nice to have it the in the standard library, and (b) even nicer to have an operator in the language, since terseness is the goal. I assume that this has never been brought up in the past because it's so heinously un-Pythonic that you'd have to be a fool to risk the public mockery and shunning associated with asking this question. Well, I guess I'm that fool: flame away... Thanks, Mark
On Fri, Sep 18, 2015 at 10:42:59AM -0700, Mark Haase wrote:
Hmmm, I use this NullObject class when I want to do stuff similar to what you've described: class NullObject(object): """ This is a helper class that does its best to pretend to be forgivingly null-like. >>> n = NullObject() >>> n None >>> n.foo None >>> n.foo.bar.moo None >>> n.foo().bar.moo(True).cat().hello(False, abc=123) None >>> n.hornet(afterburner=True).shotdown(by=n().tomcat) None >>> n or 1 1 >>> str(n) '' >>> int(n) 0 >>> len(n) 0 """ def __getattr__(self, name): return self def __getitem__(self, item): return self def __call__(self, *args, **kwds): return self def __nonzero__(self): return False def __repr__(self): return repr(None) def __str__(self): return '' def __int__(self): return 0 def __len__(self): return 0 Source: https://github.com/tpn/tpn/blob/master/lib/tpn/util.py#L1031 Sample use: https://github.com/enversion/enversion/blob/master/lib/evn/change.py#L1300 class ChangeSet(AbstractChangeSet): @property def top(self): """ Iff one child change is present, return it. Otherwise, return an instance of a NullObject. """ if self.child_count != 1: return NullObject() else: top = None for child in self: top = child break return top @property def is_tag_create(self): return self.top.is_tag_create @property def is_tag_remove(self): return self.top.is_tag_remove @property def is_branch_create(self): return self.top.is_branch_create @property def is_branch_remove(self): return self.top.is_branch_remove Having self.top potentially return a NullObject simplifies the code for the four following properties.
If you can alter the part that creates `created` or `updated` to return a NullObject() instead of None when applicable, you could call `created.isoformat()` with out the addition clause.
Thanks, Mark
Trent.
On Sep 18, 2015, at 11:21, Trent Nelson <trent@snakebite.org> wrote:
I believe it was raised as a side issue during other discussions (conditional expressions, exception-handling expressions, one of the pattern-matching discussions), but I personally can't remember anyone ever writing a serious proposal. I think Armin from PyPy also has a blog post mentioning the idea somewhere, as a spinoff of his arguments against PEP 484 (which turned into a more general "what's wrong with Python's type system and what could be done to fix it). One last place to look, although it'll be harder to search for, is every time people discuss whether things like dict.get are a wart on the language (because there should be a fully general way to do the equivalent) or a feature (because it's actually only useful in a handful of cases, and it's better to mark them explicitly than to try to generalize). But my guess is that the discussion hasn't actually been had in sufficient depth to avoid having it here. (Although even if I'm right, that doesn't mean more searching isn't worth doing--to find arguments and counter arguments you may have missed, draw parallels to successes and failures in other languages, etc.) And, even if Guido hates the idea out of hand, or someone comes up with a slam-dunk argument against it, this could turn into one of those cases where it's worth someone gathering all the info and shepherding the discussion just to write a PEP for Guido to reject explicitly. Personally, for whatever my opinion is worth (not that much), I don't have a good opinion on how it would work in Python without seeing lots of serious examples or trying it out. But I think this would be relatively easy to hack in at the tokenizer level with a quick&dirty import hook. I'll attempt it some time this weekend, in hopes that people can play with the feature. Also, it might be possible to do it less hackily with MacroPy (or it might already be part of MacroPy--often Haoyi's time machine is as good as Guido's).
This is a very Smalltalk-y solution, which isn't a bad thing. I think having a singleton instance of NullObject (like None is a singleton instance of NoneType) so you can use is-tests, etc. might make it better, but that's arguable. The biggest problem is that you have to write (or wrap) every API to return NullObjects instead of None, and likewise to take NullObjects. (And, if you use a PEP 484 checker, it won't understand that an optional int can hold a NullObject.) Also, there's no way for NullObject to ensure that spam(NullObject) returns NullObject for any function spam (or, more realistically, for any function except special cases, where it's hard to define what counts as a special case but easy to understand intuitively). And finally, there's no obvious way to make NullObject raise when you want it to raise. With syntax for nil coalescing, this is easy: ?. returns None for None, while . raises AttributeError. With separate types instead, you're putting the distinction at the point (possibly far away) where the value is produced, rather than the point where it's used. As a side note, my experience in both Smalltalk and C# is that at some point in a large program, I'm going to end up hackily using a distinction between [nil] and nil somewhere because I needed to distinguish between an optional optional spam that "failed" at the top level vs. one that did so at the bottom level. I like the fact that in Haskell or Swift I can actually distinguish "just nil" from "nil" when I need to but usually don't have to (and the code is briefer when I don't have to), but I don't know whether that's actually essential (the [nil]) hack almost always works, and isn't that hard to read if it's used sparsely, which it almost always is).
FWIW, I generally hate odd punctuation like this (@ notwithstanding) but I'm not against the idea itself -- maybe a different syntax can be invented, or maybe I could be persuaded that it's okay. On Fri, Sep 18, 2015 at 12:28 PM, Andrew Barnert via Python-ideas < python-ideas@python.org> wrote:
-- --Guido van Rossum (python.org/~guido)
Andrew, thanks for putting together that hack. I will check it out. Guido, you lost me at "hate odd punctuation ... maybe a different syntax can be invented". Do you mean introducing a new keyword or implementing this as a function? Or do you mean that some other punctuation might be less odd? I'm willing to write a PEP, even if it's only purpose is to get shot down. On Fri, Sep 18, 2015 at 3:45 PM, Guido van Rossum <guido@python.org> wrote:
-- Mark E. Haase 202-815-0201
On Sep 18, 2015, at 12:28, Andrew Barnert via Python-ideas <python-ideas@python.org> wrote:
Personally, for whatever my opinion is worth (not that much), I don't have a good opinion on how it would work in Python without seeing lots of serious examples or trying it out. But I think this would be relatively easy to hack in at the tokenizer level with a quick&dirty import hook. I'll attempt it some time this weekend, in hopes that people can play with the feature. Also, it might be possible to do it less hackily with MacroPy (or it might already be part of MacroPy--often Haoyi's time machine is as good as Guido's).
You can download a quick&dirty hack at https://github.com/abarnert/nonehack This only handles the simple case of identifier?.attribute; using an arbitrary target on the left side of the . doesn't work, and there are no other none-coalescing forms like ?(...) or ?[...]. (The latter would be easy to add; the former, I don't think so.) But that's enough to handle the examples in the initial email. So, feel free to experiment with it, and show off code that proves the usefulness of the feature. Also, if you can think of a better syntax that will make Guido less sad, but don't know how to implement it as a hack, let me know and I'll try to do it for you.
On Sat, Sep 19, 2015 at 3:42 AM, Mark Haase <mehaase@gmail.com> wrote:
Python generally doesn't special-case None, so having a bit of magic that works only on that one object seems a little odd. For comparison purposes, Pike has something very similar to what you're describing, but Pike *does* treat the integer 0 as special, so it makes good sense there. Pike code that wants to return "a thing or NULL" will return an object or the integer 0, where Python code will usually return an object or None. I can't think of any situation in Python where the language itself gives special support to None, other than it being a keyword. You're breaking new ground. But in my opinion, the practicality is worth it. The use of None to represent the SQL NULL value [1], the absence of useful return value, or other "non-values", is pretty standard. I would define the operator pretty much the way you did above, with one exception. You say: created?.isoformat() # is equivalent to created.isoformat() if created is not None else None but this means there needs to be some magic, because it should be equally possible to write: created?.year # equivalent to created.year if created is not None else None which means that sometimes it has to return None, and sometimes (lambda *a,**ka: None). Three possible solutions: 1) Make None callable. None.__call__(*a, **ka) always returns None. 2) Special-case the immediate call in the syntax, so the equivalencies are a bit different. 3) Add another case: func?(args) evaluates func, and if it's None, evaluates to None without calling anything. Option 1 would potentially mask bugs in a lot of unrelated code. I don't think it's a good idea, but maybe others disagree. Option 2 adds a grammatical distinction that currently doesn't exist. When you see a nullable attribute lookup, you have to check to see if it's a method call, and if it is, do things differently. That means there's a difference between these: func = obj?.attr; func() obj?.attr() Option 3 requires a bit more protection, but is completely explicit. It would also have use in other situations. Personally, I support that option; it maintains all the identities, is explicit that calling None will yield None, and doesn't need any magic special cases. It does add another marker, though: created?.isoformat?() # is equivalent to created.isoformat() if created is not None and created.isoformat is not None else None As to the syntax... IMO this needs to be compact, so ?. has my support. With subscripting, should it be "obj?[idx]" or "obj[?idx]" ? FWIW Pike uses the latter, but if C# uses the former, there's no one obvious choice. ChrisA [1] Or non-value, depending on context
On 18/09/15 23:37, Chris Angelico wrote:
Python generally doesn't special-case None, so having a bit of magic that works only on that one object seems a little odd.
So the answer here is to introduce a "magic" hook that None can make use of (but also other classes). I can't think of an appropriate word, so I'll use "foo" to keep it suitably abstract. If the foo operator uses the magic method "__foo__" to mean "return an object to be used in place of the operand should it be considered ... false? [or some other definition - I'm not sure]" then any class can implement that method to return an appropriate proxy object. If that was a postfix operator which has a high precedence, then: bar = foo? bar.isoformat() and the original syntax suggestion: bar = foo?.isoformat() ... are equivalent. "?." is not a new operator. "?" is. This is essentially a slight refinement of Chris's case 3 -
E.
Apologies for the self-reply. I just wanted to clarify a couple of things. On 18/09/15 23:56, Erik wrote:
Not "false", I think. The "foo" operator is meant to mean "I will go on to use the resulting object in any way imaginable and it must cope with that and return a value from any attempts to use it that will generally mean 'no'" (*).
Which is clearly wrong - the first part should be: baz = foo? bar = baz.isoformat() E. (*) Should we call the operator "shrug"?
On 19.09.2015 01:47, Ryan Gonzalez wrote:
What about "apply"? It's the closest thing to "fmap" I can think of that won't coblnfuse people...
Are you sure? I think "maybe" better reflects the purpose of "?". Nevertheless, I would love to see support for the maybe monad in Python. Best, Sven
On Sep 18, 2015, at 16:58, Sven R. Kunze <srkunze@mail.de> wrote:
I think this, and the whole discussion of maybe and fmap, is off the mark here. It's trivial to create a maybe type in Python. What's missing is the two things that make it useful: (1) pattern matching, and (2) a calling syntax and a general focus on HOFs that make fmap natural. Without at least one of those, maybe isn't useful. And adding either of those to Python is a huge proposal, much larger than null coalescing, and a lot less likely to gain support. Also, the monadic style of failure propagation directly competes with the exception-raising style, and they're both contagious. A well-designed language and library can have both side by side if it, e.g., rigorously restricts exceptions to only truly exceptional cases, but the boat for that sailed decades ago in Python. So just having them side by side would lead to the exact same problems as C++ code that mixes exception-based and status-code-based APIs, or JavaScript code that mixes exceptions and errbacks or promise.fail handlers. Personally, whenever I think to myself "I could really use maybe here" in some Python code, that's a sign that I'm not thinking Pythonically, and either need to switch gears in my brain or switch languages. Just like when I start thinking about how I could get rid of that with statement with an RAII class, and maybe an implicit conversion operator....
Ryan Gonzalez <rymg19@gmail.com> writes:
http://stackoverflow.com/questions/8507200/maybe-kind-of-monad-in-python
On Sep 18, 2015, at 15:56, Erik <python@lucidity.plus.com> wrote:
I like this (modulo the corrections later in the thread). It's simpler and more flexible than the other options, and also comes closer to resolving the "spam?.eggs" vs. "spam?.cheese()" issue, by requiring "spam?.cheese?()". Obviously "spam?" returns something with a __getattr__ method that just passes through to spam.__getattr__, except that on NoneType it returns something with a __getattr__ that always returns None. That solves the eggs case. Next, "spam?.cheese?" returns something with a __call__ method that just passed through to spam?.cheese.__call__, except that on NoneType it returns something with a __call__ that always returns None. That solves the cheese case. If you make None? return something whose other dunder methods also return None (except for special cases like __repr__), this also gives you "spam ?+ 3". (I'm not sure if that's a good thing or a bad thing...) Of course there's no way to do "spam ?= 3" (but I'm pretty sure that's a good thing). So, do we need a dunder method for the "?" operator? What else would you use it for besides None?
On Sat, Sep 19, 2015 at 10:49 AM, Andrew Barnert <abarnert@yahoo.com> wrote:
Obviously "spam?" returns something with a __getattr__ method that just passes through to spam.__getattr__, except that on NoneType it returns something with a __getattr__ that always returns None. That solves the eggs case.
Next, "spam?.cheese?" returns something with a __call__ method that just passed through to spam?.cheese.__call__, except that on NoneType it returns something with a __call__ that always returns None. That solves the cheese case.
Hang on, how do you do this? How does the operator know the difference between "spam?", which for None has to have __getattr__ return None, and "spam?.cheese?" that returns (lambda: None)? ChrisA
On Sep 18, 2015, at 18:00, Chris Angelico <rosuav@gmail.com> wrote:
All you need to make this work is: * "spam?" returns NoneQuestion if spam is None else spam * NoneQuestion.__getattr__(self, *args, **kw) returns None. * NoneQuestion.__call__(self, *args, **kw) returns None. Optionally, you can add more None-returning methods to NoneQuestion. Also, whether NoneQuestion is a singleton, has an accessible name, etc. are all bikesheddable. I think it's obvious what happens is "spam" is not None and "spam.cheese" is, or of both are None, but if not, I can work them through as well.
On 2015-09-19 02:10, Andrew Barnert via Python-ideas wrote:
I see it as "spam? doing "Maybe(spam)" and then attribute access checking returning None if the wrapped object is None and getting the attribute from it if not. I think that the optimiser could probably avoid the use of Maybe in cases like "spam?.cheese".
On 2015-09-19 02:39, MRAB wrote:
I've thought of another issue: If you write "spam?(sing_lumberjack_song())", won't it still call sing_lumberjack_song even if spam is None? After all, Python evaluates the arguments before looking up the call, so it won't know that "spam" is None until it tries to call "spam?". That isn't a problem with "spam.sing_lumberjack_song() if spam is not None else None" or if it's optimised to that, but "m = spam?; m(sing_lumberjack_song())" is a different matter. perhaps a "Maybe" object should also support "?" so you could write "m = spam?; m?(sing_lumberjack_song())". "Maybe" could be idempotent, so "Maybe(Maybe(x))" returns the same result as "Maybe(x)".
On Sep 18, 2015, at 18:52, MRAB <python@mrabarnett.plus.com> wrote:
You're right; I didn't think about that. But I don't think that's a problem. I believe C#, Swift, etc. all evaluate the arguments in their equivalent. And languages like ObjC that do automatic nil coalescing for all method calls definitely evaluate them. If you really want to switch on spam and not call sing_lumberjack_song, you can always do that manually, right?
That actually makes sense just for its own reasons. Actually, now that I think about it, the way I defined it above already gives you think: if spam? is spam if it's anything but None, then spam?? is always spam?, right?
Andrew, I really like that idea. Turning back to the null coalescing operator (spelled ?? in other languages), how do you think that fits in? Consider this syntax:
None? or 1 1
This works if NoneQuestion overrides __nonzero__ to return False.
0? or 1 0
This doesn't work, because 0? returns 0, and "0 or 1" is 1. We could try this instead, if NoneQuestion overrides __or__:
This looks a little ugly, and it would be nice (as MRAB pointed out) if null coalescing short circuited.
None? or None?
This also doesn't work quite right. If both operands are None, we want the expression to evaluate to None, not NoneQuestion. *Should null coalescing be a separate operator? And if so, are "?" and "??" too similar?* Can anybody think of realistic use cases for overriding a magic method for the "?" operator? I would like to include such use cases in a PEP. One possible use case: being able to coalesce empty strings.
On Fri, Sep 18, 2015 at 9:10 PM, Andrew Barnert via Python-ideas < python-ideas@python.org> wrote:
-- Mark E. Haase 202-815-0201
On Sat, Sep 19, 2015 at 12:06 PM, Mark E. Haase <mehaase@gmail.com> wrote:
Frankly, I think this is a bad idea. You're potentially coalescing multiple things with the same expression, and we already have a way of spelling that: the "or" operator. If you don't want a generic "if it's false, use this", and don't want a super-specific "if it's None, use this", then how are you going to define what it is? And more importantly, how do you reason about the expression "s1? or s2" without knowing exactly what types coalesce to what? Let's keep the rules simple. Make this a special feature of the None singleton, and all other objects simply return themselves - for the same reason that a class isn't allowed to override the "is" operator. ChrisA
On Sep 18, 2015, at 19:06, Mark E. Haase <mehaase@gmail.com> wrote:
I don't think there's any easy way to make "spam? or 1" work any better than "spam or 1" already does, partly for the reasons you give below, but also because it doesn't seem to fit the design in any obvious way. I guess that means postix ? doesn't quite magically solve everything...
This also doesn't work quite right. If both operands are None, we want the expression to evaluate to None, not NoneQuestion. Should null coalescing be a separate operator? And if so, are "?" and "??" too similar?
As MRAB pointed out, there seem to be good reasons to let spam?? mean the same thing as spam? (and that follows automatically from the simplest possible definition, the one I gave above). So I think "spam ?? eggs" is ambiguous between the postfix operator and the infix operator without lookahead, at least to a human, and possibly to the compiler as well. I suppose ?: as in ColdFusion might work, but (a) ewwww, (b) it regularly confuses novices to CF, and (c) it's impossible to search for, because ?: no matter how you quote it gets you the C ternary operator....
This seems like a bad idea. Empty strings are already falsey. If you want this behavior, why not just use "s1 or s2", which already works, and for obvious reasons?
This is likely going to get shot down quickly... I know CoffeeScript is not regarded too well in this community (well, at least based on Guido's remarks on parsing it), but what if x? was shorthand for x is None? In CS, it's called the existential operator. On September 18, 2015 10:30:48 PM CDT, Andrew Barnert via Python-ideas <python-ideas@python.org> wrote:
-- Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity.
On Fri, Sep 18, 2015 at 05:49:36PM -0700, Andrew Barnert via Python-ideas wrote:
Ah, and now my enthusiasm for the whole idea is gone... In my previous response, I imagined spam?.attr to be syntactic sugar for `None if spam is None else spam.attr`. But having ? be an ordinary operator that returns a special Null object feels too "Design Pattern-y" to me. I think the Null design pattern is actually harmful, and I would not like to see this proposal implemented this way. (In another email, Andrew called the special object something like NoneMaybe or NoneQuestion, I forget which. I'm going to call the object Null, since that's less typing.) The Null object pattern sounds like a great idea at first, but I find it to be a code smell at best and outright harmful at worst. If you are passing around an object which is conceptually None, but unlike None reliably does nothing without raising an exception no matter what you do with it, that suggests to me that something about your code is not right. If your functions already accept None, then you should just use None. If they don't accept None, then why are you trying to smuggle None into them using a quasi-None that unconditionally hides errors? Here are some problems with the Null pattern as I see it: (1) Suppose that spam? returns a special Null object, and Null.attr itself returns Null. (As do Null[item] and Null(arg), of course.) This matches the classic Null object design pattern, and gives us chaining for free: value = obj?.spam.eggs.cheese But now `value` is Null, which may not be what we expect and may in fact be a problem if we're expecting it to be "an actual value, or None" rather than our quasi-None Null object. Because `value` is now a Null, every time we pass it to a function, we risk getting new Nulls in places that shouldn't get them. If a function isn't expecting None, we should get an exception, but Null is designed to not raise exceptions no matter what you do with it. So we risk contaminating our data with Nulls in unexpected places. Eventually, of course, there comes a time where we need to deal with the actual value. With the Null pattern in place, we have to deal with two special cases, not one: # I assume Null is a singleton, otherwise use isinstance if filename is not None and filename is not Null: os.unlink(filename) A small nuisance, to be sure, but part of the reason why I really don't think much of the Null object pattern. It sounds good on paper, but I think it's actually more dangerous and inconvenient than the problem it tries to solve. (2) We can avoid the worst of the Null design (anti-)pattern by having Null.attr return None instead of Null. Unfortunately, that means we've lost automatic chaining. If you have an object that might be None, we have to explicitly use the ? operator after each lookup except the last: value = obj?.spam?.eggs?.cheese which is (a) messy, (b) potentially inefficient, and (c) potentially hides subtle bugs. Here is a scenario where it hides bugs. Suppose obj may be None, but if it is not, then obj.spam *must* be a object with an eggs attribute. If obj.spam is None, that's a bug that needs fixing. Suppose we start off by writing the obvious thing: obj?.spam.eggs but that fails because obj=None raises an exception: obj? returns Null Null.spam returns None None.eggs raises So to protect against that, we might write: obj?.spam?.eggs but that protects against too much, and hides the fact that obj.spam exists but is None. As far as I am concerned, any use of a Null object has serious downsides. If people want to explicitly use it in their own code, well, good luck with that. I don't think Python should be making it a built-in. I think the first case, the classic Null design pattern, is actually *better* because the downsides are anything but subtle, and people will soon learn not to touch it with a 10ft pole *wink*, while the second case, the "Null.attr gives None" case, is actually worse because it isn't *obviously* wrong and can subtly hide bugs. How does my earlier idea of ? as syntactic sugar compare with those? In that case, there is no special Null object, there's only None. So we avoid the risk of Null infection, and avoid needing to check specially for Null. It also avoids the bug-hiding scenario: obj?.spam.eggs.cheese is equivalent to: None if obj is None else obj.spam.eggs If obj is None, we get None, as we expect. If it is not None, we get obj.spam.eggs as we expect. If obj.spam is wrongly None, then we get an exception, as we should. -- Steve
Andrew Barnert via Python-ideas writes:
So, do we need a dunder method for the "?" operator? What else would you use it for besides None?
NaNs in a pure-Python implementation of float or Decimal. (This is not a practical suggestion.) A true SQL NULL type. It's always bothered me that most ORMs map NULL to None but there are plenty of other ways to inject None into a Python computation. (This probably isn't a practical suggestion either unless Random832's suggestion of ?() establishing a lexical context were adopted.) The point is that Maybe behavior is at least theoretically useful in subcategories, with special objects other than None. Sven's suggestion of calling this the "monad" operator triggers a worry in me, however. In Haskell, the Monad type doesn't enforce the monad laws, only the property of being an endofunctor. That apparently turns out to be enough in practice to make the Monad type very useful. However, in Python we have no way to enforce that property. I don't have the imagination to come up with a truly attractive nuisance here, and this operator doesn't enable general functorial behavior, so maybe it's not a problem.
On 19.09.2015 07:14, Stephen J. Turnbull wrote:
I definitely agree here. Internally, we have a guideline telling us to avoid None or NULL whenever possible. Andrew's remark about 'code smell' is definitely appropriate. There was a great discussion some years ago on one of the RDF semantics mailing list about the semantics of NULL (in RDF). It turned out to have 6 or 7 semantics WITHOUT any domain-specific focus (don't know, don't exists, is missing, etc. -- can't remember all of them). I feel that is one reason why Python programs should avoid None: we don't guess.
Sleeping one night over it, I now tend to change my mind regarding this. Maybe, it's *better to DEAL with None as in remove* *them* from the code, from the database, from the YAML files and so forth *instead**of *making it easier to work with them. Restricting oneself, would eventually lead to more predictable designs. Does this makes sense somehow? Issue is, None is so convenient to work with. You only find out the code smell when you discover a "NoneType object does not have attribute X" exception some months later and start looking where the heck the None could come from. What can we do here?
Sven R. Kunze writes:
That's exactly what should happen (analogous to a "signalling NaN"). The problem is if you are using None as a proxy for a NULL in another subsystem that has "NULL contagion" (I prefer that to "coalescing"). At this point the thread ends for me because I'm not going try to tell the many libraries that have chosen to translate NULL to None and vice versa that they are wrong.
"Uptalk" is an interesting speech pattern where every sentence sounds like a question. Google it, there's some interesting research. The "null pattern" is terrible. Uptalk should not be considered a unary operator that returns a magical value. It's a modifier on other operators (somewhat similar to the way "+=" and friends are formed). In case someone missed it, uptalk should test for None, not for a falsey value. I forgot to think about the scope of the uptalk operator (i.e. what is skipped when it finds a None). There are some clear cases (the actual implementation should avoid double evaluation of the tested expression, of course): a.b?.c.d[x, y](p, q) === None if a.b is None else a.b.c.d[x, y](p, q) a.b?[x, y].c.d(p, q) === None if a.b is None else a.b[x, y].c.d(p, q) a.b?(p, q).c.d[x, y] === None if a.b is None else a.b(p, q).c.d[x, y] But what about its effect on other operators in the same expression? I think this is reasonable: a?.b + c.d === None if a is None else a.b + c.d OTOH I don't think it should affect shortcut boolean operators (and, or): a?.b or x === (None if a is None else a.b) or x It also shouldn't escape out of comma-separated lists, argument lists, etc.: (a?.b, x) === ((None if a is None else a.b), x) f(a?.b) === f((None if a is None else a.b)) Should it escape from plain parentheses? Which of these is better? (a?.b) + c === (None if a is None else a.b) + c # Fails unless c overloads None+c (a?.b) + c === None if a is None else (a.b) + c # Could be surprising if ? is deeply nested Here are some more edge cases / hypergeneralizations: {k1?: v1, k2: v2} === {k2: v2} if k1 is None else {k1: v1, k2: v2} # ?: skips if key is None # But what to do to skip None values? Could we give ?= a meaning in assignment, e.g. x ?= y could mean: if y is not None: x = y More fun: x ?+= y could mean: if x is None: x = y elif y is not None: y += y You see where this is going. Downhill fast. :-) -- --Guido van Rossum (python.org/~guido)
On 2015-09-19 17:21, Guido van Rossum wrote:
It shouldn't escape beyond anything having a lower precedence.
Shouldn't that be: if x is not None: x = y ? It's the value before the '?' that's tested.
Or: if x is None: pass else: x += y
You see where this is going. Downhill fast. :-)
Could it be used postfix: a +? b === None if b is None else a + b -?a === None if a is None else -a or both prefix and postfix: a ?+? b === None if a is None or b is None else a + b ?
On Sat, Sep 19, 2015 at 9:21 AM, Guido van Rossum <guido@python.org> wrote:
This is a bit weird to me. Essentially ?. takes precedence over a following +. But would you also expect it to take precedence over a preceding one as well? That's inconsistent. c.d + a?.b === None if a is None else c.d + a.b or c.d + a?.b === c.d + None if a is None else c.d + a.b I think that ?. ?[] and ?() should affect other operators at the same precedence level only, i.e., each other and . [] and (). This seems the most logical to me. And I just looked up the C# documentation on MSDN and it does the same thing: https://msdn.microsoft.com/en-us/library/dn986595.aspx
Agree. It also should not escape grouping parenthesis even though that might not be useful. It would be very weird if a parenthesized expression did something other than evaluate the expression inside it, period. (a?.b).c === None.c if a is None else (a.b).c === temp = a?.b; temp.c (x or a?.b).c === (x or (None if a is none else a.b)).c Yes, None.c is going to raise an exception. That's better than just getting None IMHO. --- Bruce Check out my new puzzle book: http://J.mp/ingToConclusions Get it free here: http://J.mp/ingToConclusionsFree (available on iOS)
On 19.09.2015 14:48, Stephen J. Turnbull wrote:
Not my point, Stephen. My point is, you better avoid None (despite its convenience) because you are going to have a hard time finding its origin later in the control flow. Question still stands: is None really necessary to justify the introduction of convenience operators like "?." etc.?
The problem is if you are using None as a proxy for a NULL in another subsystem that has "NULL contagion" (I prefer that to "coalescing").
How would you solve instead? Best, Sven
I think the core issue is that, whether or not it should be used, APIs already return None values, so a convenience operator might as well be added. On September 19, 2015 2:09:48 PM CDT, "Sven R. Kunze" <srkunze@mail.de> wrote:
-- Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity.
2015-09-19 21:24 GMT+02:00 Ryan Gonzalez <rymg19@gmail.com>:
I'm curious on which API returning None, a major bonus on using python is that I pretty never stumbled upon the equivalent of NullPointerException. Moreover, I wonder if that this convenience operator will do something more than hide bugs.
On Sep 19, 2015, at 14:47, Guido van Rossum <guido@python.org> wrote:
The TypeErrors usually come from novices. There are many of StackOverflow questions asking why they can't add spam.get_text() + "\n" where they don't show you the implementation of get_text, or the exception they got, but you just know they forgot a return statement at the end and the exception was a TypeError about adding NoneType and str.
On Sat, Sep 19, 2015 at 12:02:42AM +0100, MRAB wrote:
No. If I understand the idea, obj?.attr returns None if obj is None, otherwise returns obj.attr. The question mark (shrug operator?) applies to `obj` *before* the attribute lookup, so it should appear *before* the dot (since we read from left-to-right). The heuristic for remembering the order is that the "shrug" (question mark) operator applies to obj, so it is attached to obj, before any subsequent operation. For the sake of brevity, using @ as a placeholder for one of attribute access, item/key lookup, or function call, then we have: obj?@ as syntactic sugar for: None if obj is None else obj@ Furthermore, we should be able to chain a sequence of such @s: paperboy.receive(customer?.trousers.backpocket.wallet.extract(2.99)) being equivalent to: paperboy.receive(None if customer is None else customer.trousers.backpocket.wallet.extract(2.99) ) Let's just assume we have a good reason for chaining lookups that isn't an egregious violation of the Law of Demeter, and not get into a debate over OOP best practices, okay? :-) Suppose that wallet itself may also be None. Then we can easily deal with that situation too: paperboy.receive(customer?.trousers.backpocket.wallet?.extract(2.99)) which I think is a big win over either of these two alternatives: # 1 paperboy.receive(None if customer is None else None if customer.trousers.backpocket.wallet is None else customer.trousers.backpocket.wallet.extract(2.99) ) # 2 if customer is not None: wallet = customer.trousers.backpocket.wallet if wallet is not None: paperboy.receive(wallet.extract(2.99)) It's a funny thing, I'm usually not a huge fan of symbols outside of maths operators, and I strongly dislike the C ? ternary operator, but this one feels really natural to me. I didn't have even the most momentary "if you want Perl, you know where to find it" thought. -- Steve
On Fri, Sep 18, 2015 at 8:41 PM, Steven D'Aprano <steve@pearwood.info> wrote:
I do, but at least the '?' is part of an operator, not part of the name (as it is in Ruby?). I really, really, really don't like how it looks, but here's one thing: the discussion can be cut short and focus almost entirely on whether this is worth making Python uglier (and whether it's even ugly :-). The semantics are crystal clear and it's obvious that the way it should work is by making "?.", ?(" and "?[" new operators or operator pairs -- the "?" should not be a unary postfix operator but a symbol that combines with certain other symbols. Let me propose a (hyper?)generalization: it could be combined with any binary operation, e.g. "a?+b" would mean "None if a is None else a+b". Sadly (as hypergeneralizations tend to do?) this also leads to a negative observation: what if I wanted to write "None if b is None else a+b"? (And don't be funny and say I should swap a and b -- they could be strings.) Similar for what if you wanted to do this with a unary operator, e.g. None if x is None else -x. Maybe we could write "a+?b" and "-?x"? But I don't think the use cases warrant these much. Finally, let's give it a proper name -- let's call it the uptalk operator. -- --Guido van Rossum (python.org/~guido)
Guido van Rossum <guido@python.org> writes:
Let me propose a (hyper?)generalization: it could be combined with any binary operation, e.g. "a?+b" would mean "None if a is None else a+b".
I'd have read it as "None if a is None or b is None else a+b". If you want to only do it for one of the operands you should be explicit. I'm not sure if I have a coherent argument for why this shouldn't apply to ?[, though.
On Fri, Sep 18, 2015 at 11:21 PM, Guido van Rossum <guido@python.org> wrote:
I really liked this whole thread, and I largely still do -- I?think -- but I'm not sure I like how `?` suddenly prevents whole blocks of code from being evaluated. Anything within the (...) or [...] is now skipped (IIUC) just because a `?` was added, which seems like it could have side effects on the surrounding state, especially since I expect people will use it for squashing/silencing or as a convenient trick after the fact, possibly in code they did not originally write. If the original example included a `?` like so: response = json.dumps?({ 'created': created?.isoformat(), 'updated': updated?.isoformat(), ... }) should "dumps" be None, the additional `?` (although though you can barely see it) prevents *everything else* from executing. This may cause confusion about what is being executed, and when, especially once nesting (to any degree really) and/or chaining comes into play! Usually when I want to use this pattern, I find I just need to write things out more. The concept itself vaguely reminds me of PHP's use of `@` for squashing errors. In my opinion, it has some utility but has too much potential impact on program flow without being very noticeable. If I saw more than 1 per line, or a couple within a few lines, I think my ability to quickly identify -> analyze -> comprehend possible routes in program control flow decreases. I feel like I'll fault more, double back, and/or make sure I forevermore look harder for sneaky `?`s. I probably need to research more examples of how such a thing is used in real code, today. This will help me get a feel for how people might want to integrate the new `?` capability into their libraries and apis, maybe that will ease my readability reservations. Thanks, -- C Anthony
On Sat, Sep 19, 2015 at 03:17:07AM -0500, C Anthony Risinger wrote:
I don't think this is any different from other short-circuiting operators, particularly `and` and the ternary `if` operator: result = obj and obj.method(expression) result = obj.method(expression) if obj else default In both cases, `expression` is not evaluated if obj is falsey. That's the whole point.
We're still discussing the syntax and semantics of this, so I could be wrong, but my understanding of this is that the *first* question mark prevents the expressions in the parens from being executed: json.dumps?( ... ) evaluates as None if json.dumps is None, otherwise it evaluates the arguments and calls the dumps object. In other words, rather like this: _temp = json.dumps # temporary value if _temp is None: response = None else: response = _temp({ 'created': None if created is None else created.isoformat(), 'updated': None if updated is None else updated.isoformat(), ... }) del _temp except the _temp name isn't actually used. The whole point is to avoid evaluating an expression (attribute looking, index/key lookup, function call) which will fail if the object is None, and if you're not going to call the function, why evaluate the arguments to the function?
Well, yes, people can abuse most any syntax.
I had to look up PHP's @ and I must say I'm rather horrified. According to the docs, all it does is suppress the error reporting, it does nothing to prevent or recover from errors. There's not really an equivalent in Python, but I suppose this is the closest: # similar to PHP's $result = @(expression); try: result = expression except: result = None This is nothing like this proposal. It doesn't suppress arbitrary errors. It's more like a conditional: # result = obj?(expression) if obj is None: result = None else: result = obj(expression) If `expression` raises an exception, it will still be raised, but only if it is actually evaluated, just like anything else protected by an if...else or short-circuit operator. -- Steve
On Sat, Sep 19, 2015 at 7:06 AM, Steven D'Aprano <steve@pearwood.info> wrote:
Sure, but those all have white space and I can read what's happening. The `?` could appear anywhere without break. I don't like that, but, opinion.
Yes that is how I understand it as well. I'm suggesting it's hard to see. I understand the concept as "None cancellation", because if the left is None, the right is cancelled. This lead me here: * This is great, want to use all the time! * First-level language support, shouldn't I use? Does feels useful/natural * How can I make my APIs cancellation-friendly? * I can write None-centric APIs, that often collapse to None * Now maybe user code does stuff like `patient.visits?.september?.records` to get all records in September (if any, else None) * Since both `?` points would *prefer* None, if the result is None, I now have to jump around looking for who done it * If I don't have debugger ATM, I'm breaking it up a lot for good 'ol print(...), only way * I don't think I like this any more :( I especially don't like the idea of seeing it multiple times quickly, and the potential impact to debugging. The truth is I want to like this but I feel like it opens a can of worms (as seen by all the wild operators this proposal "naturally" suggests).
I did say vaguely :) but it is extremely hideous I agree. The part that made me think of this is the would be desire for things to become None (so, or example, wanting to avoid throwing typed/informative exceptions if possible) so they'd then be more useful with `?`. -- C Anthony
On Sun, Sep 20, 2015 at 09:10:32AM +0300, Serhiy Storchaka wrote:
Absolutely nothing. I'm not convinced that we should generalise this beyond the three original examples of attribute access, item lookup and function call. I think that applying ? to arbitrary operators is a case of "YAGNI". Or perhaps, "You Shouldn't Need It". Mark's original motivating use-case strikes me as both common and unexceptional. We might write: # spam may be None, or some object result = spam or spam.attr # better, as it doesn't misbehave when spam is falsey result = None if spam is None else spam.attr and it seems reasonable to me to want a short-cut for that use-case. But the generalisations to arbitrary operators suggested by Guido strike me as too much, too far. As he says, going downhill, and quickly. Consider these two hypotheticals: spam ?+ eggs # None if spam is None or eggs is None else spam + eggs needle ?in haystack # None if needle is None or haystack is None else needle in haystack Briefer (more concise) is not necessarily better. At the point you have *two* objects in the one term that both need to be checked for None, that is in my opinion a code smell and we shouldn't provide a short-cut disguising that. Technically, x.y x[y] and x(y) aren't operators, but for the sake of convenience I'll call them such. Even though these are binary operators, the ? only shortcuts according to the x, not the y. So we can call these ?. ?[] ?() operators "pseudo-unary" operators rather than binary operators. Are there any actual unary operators we might want to apply this uptalk/shrug operator to? There are (if I remember correctly) only three unary operators: + - and ~. I don't think there are any reasonable use-cases for writing (say): value = ?-x that justifies making this short-cut available. So as far as I am concerned, the following conditions should apply: - the uptalk/shrug ? "operator" should not apply to actual binary operators where both operands need to be checked for None-ness (e.g. arithmetic operators, comparison operators) - it should not apply to arithmetic unary operators + - and ~ - it might apply to pseudo-operators where only the lefthand argument is checked for None-ness, that is, x.y x[y] and x(y), written as x?.y x?[y] and x?(y). If I had to choose between generalising this to all operators, or not having it at all, I'd rather not have it at all. A little bit of uptalk goes a long way, once we have ? appearing all over the place in all sorts of expressions, I think it's too much. -- Steve
On Sep 20, 2015, at 00:31, Steven D'Aprano <steve@pearwood.info> wrote:
I agree. Seeing how far you can generalize something and whether you can come up with a simple rule that makes all of your use cases follow naturally can be fun, but it isn't necessarily the best design. Also, by not trying to generalize uptalk-combined operators (or uptalk as a postfix unary operator of its own, which I was earlier arguing for...), the question of how we deal with ?? or ?= (if we want them) can be "the same way every other language does", rather than seeing what follows from the general rule and then convincing ourselves that's what we wanted. Also, I think trying to generalize to all operators is a false generalization, since the things we're generalizing from aren't actually operators (and not just syntactically--e.g., stylistically, they're never surrounded by spaces--which makes a pretty big difference in the readability impact of a character as heavy as "?") in the first place. Personally, I think ?? is the second most obviously useful after ?. (there's a reason it's the one with the oldest and widest pedigree); we need ?() because Python, unlike C# and friends, unifies member and method access; ?[] doesn't seem as necessary but it's such an obvious parallel to ?() that I think people will expect it; ?= is potentially as confusing as it is helpful. So, my suggestion would be just the first four. And keeping them simple, and consistent with other languages, no trying to extend the protection to other operators/accesses, no extra short-circuiting, nothing. So: spam ?? eggs === spam if spam is not None else eggs spam?.eggs === spam.eggs if spam is not None else None spam?(eggs) === spam(eggs) if spam is not None else None spam?[eggs] === spam[eggs] if spam is not None else None That's easy to define, easy to learn and remember, and pretty consistent with other languages. The one big difference is that what you write as "spam?.eggs(cheese)" in C# has to be "spam?.eggs?(cheese)" in Python, but I don't think that's a big problem. After all, in Python, spam.eggs is a first-class object, and one that's commonly passed or stored, so the obvious way to look at "spam.eggs(cheese)" is as explicitly chaining two separate things together (a __getattr__ with a descriptor __get__, and a __call__), so why shouldn't uptalking both operations be explicit?
On Sun, Sep 20, 2015 at 5:31 PM, Steven D'Aprano <steve@pearwood.info> wrote:
That's how all Python's short-circuiting works - based on the value of what's on the left, decide whether or not to evaluate what's on the right. (Well, nearly all - if/else evaluates the middle first, but same difference.) This is another form of short-circuiting; "x[y]" evaluates x, then if that's None, doesn't bother evaluating y because it can't affect the result. ChrisA
On Sun, Sep 20, 2015 at 07:38:18PM +1000, Chris Angelico wrote:
I think you are mistaken about x[y]: py> None[print("side effect")] side effect Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'NoneType' object is not subscriptable That's why x?[y] is a proposal. -- Steve
On Mon, Sep 21, 2015 at 1:50 PM, Steven D'Aprano <steve@pearwood.info> wrote:
Oops, that was a typo in my statement. I meant "x?[y]" should behave that way - once it's discovered that x is None, the evaluation of y can't affect the result, and so it doesn't get evaluated (as per the normal short-circuiting rules). Yes, x[y] has to evaluate both x and y (after all, the value of y is passed to __getitem__). Sorry for the confusion. ChrisA
On 20 September 2015 at 08:31, Steven D'Aprano <steve@pearwood.info> wrote:
Agreed. Does this need to be an operator? How about the following: class Maybe: def __getattr__(self, attr): return None def __getitem__(self, idx): return None def __call__(self, *args, **kw): return None def maybe(obj): return Maybe() if obj is None else obj attr = maybe(obj).spam elt = maybe(obj)[n] result = maybe(callback)(args) The Maybe class could be hidden, and the Maybe() object a singleton (making my poor naming a non-issue :-)) and if it's felt sufficiently useful, the maybe() function could be a builtin. Usage of the result of maybe() outside of the above 3 contexts should simply be "not supported" - don't worry about trying to stop people doing weird things, just make it clear that the intent is only to support the 3 given idiomatic usages. Paul.
On Sun, Sep 20, 2015 at 12:05:52PM +0100, Paul Moore <p.f.moore@gmail.com> wrote:
PyMaybe - a Python implementation of the Maybe pattern. Seems to be quite elaborated. https://github.com/ekampf/pymaybe
Paul.
Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
Paul Moore's idea is WAAYY better than the ugly ? pseudo-operator. `maybe()` reads just like a regular function (because it is), and we don't need to go looking for Perl (nor Haskell) in some weird extra syntax that will confuse beginners. On Sun, Sep 20, 2015 at 4:05 AM, Paul Moore <p.f.moore@gmail.com> wrote:
-- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On the day I started this thread, I wrote a Python module that does what maybe() does. I hadn't seen PyMaybe yet, and I couldn't think of any good names for my module's functions, so my module was disappointingly ugly. PyMaybe is exactly what I *wish* I had written that day. For comparison, here's the code from my first post in this thread and it's maybe-ized version. response = json.dumps({ 'created': created?.isoformat(), 'updated': updated?.isoformat(), ... }) response = json.dumps({ 'created': maybe(created).isoformat(), 'updated': maybe(updated).isoformat(), ... }) Pros: 1. No modification to Python grammar. 2. More readable: it's easy to overlook ? when skimming quickly, but "maybe()" is easy to spot. 3. More intuitive: the name "maybe" gives a hint at what it might do, whereas if you've never seen "?." you would need to google it. (Googling punctuation is obnoxious.) Cons: 1. Doesn't short circuit: "maybe(family_name).upper().strip()" will fail if family_name is None.[1] You might try "maybe(maybe(family_name).upper()).strip()", but that is tricky to read and still isn't quite right: if family_name is not None, then it *should* be an error if "upper" is not an attribute of it. The 2-maybes form covers up that error. I'm sure there will be differing opinions on whether this type of operation should short circuit. Some will say that we shouldn't be writing code that way: if you need to chain calls, then use some other syntax. But I think the example of upper case & strip is a good example of a perfectly reasonable thing to do. These kinds of operations are pretty common when you're interfacing with some external system or external data that has a concept of null (databases, JSON, YAML, argparse, any thin wrapper around C library, etc.). This conversation has really focused on the null aware attribute access, but the easier and more defensible use case is the null coalesce operator, spelled "??" in C# and Dart. It's easy to find popular packages that use something like "retries = default if default is not None else cls.DEFAULT" to supply default instances.[2] Other packages do something like "retries = default or cls.DEFAULT"[3], which is worse because it easy to overlook the implicit coalescing of the left operand. In fact, the top hit for "python null coalesce" is StackOverflow, and the top-voted answer says to use "or".[4] (The answer goes on to explain the nuance of using "or" to coalesce, but how many developers read that far?) *In the interest of finding some common ground, I'd like to get some feedback on the coalesce operator.* Maybe that conversation will yield some insight into the other "None aware" operators. A) Is coalesce a useful feature? (And what are the use cases?) B) If it is useful, is it important that it short circuits? (Put another way, could a function suffice?) C) If it should be an operator, is "??" an ugly spelling? >>> retries = default ?? cls.DEFAULT D) If it should be an operator, are any keywords more aesthetically pleasing? (I expect zero support for adding a new keyword.) >>> retries = default else cls.DEFAULT >>> retries = try default or cls.DEFAULT >>> retries = try default else cls.DEFAULT >>> retries = try default, cls.DEFAULT >>> retries = from default or cls.DEFAULT >>> retries = from default else cls.DEFAULT >>> retries = from default, cls.DEFAULT My answers: A) It's useful: supplying default instances for optional values is an obvious and common use case. B) It should short circuit, because the patterns it replaces (using ternary operator or "or") also do. C) It's too restrictive to cobble a new operator out of existing keywords; "??" isn't hard to read when it is separated by whitespace, as Pythonistas typically do between a binary operator and its operands. D) I don't find any of these easier to read or write than "??". [1] I say "should", but actually PyMaybe does something underhanded so that this expression does not fail: "maybe(foo).upper()" returns a "Nothing" instance, not "None". But Nothing has "def __repr__(self): return repr(None)". So if you try to print it out, you'll think you have a None instance, but it won't behave like one. If you try to JSON serialize it, you get a hideously confusing error: "TypeError: None is not JSON serializable". For those not familiar: the JSON encoder can definitely serialize None: it becomes a JSON "null". A standard implementation of maybe() should _not_ work this way. [2] https://github.com/shazow/urllib3/blob/master/urllib3/util/retry.py#L148 [3] https://github.com/kennethreitz/requests/blob/46ff1a9a543cc4d33541aa64c94f50... [4] http://stackoverflow.com/a/4978745/122763 On Sun, Sep 20, 2015 at 6:50 PM, Guido van Rossum <guido@python.org> wrote:
-- Mark E. Haase 202-815-0201
On 21.09.2015 04:35, Mark E. Haase wrote:
A) Is coalesce a useful feature? (And what are the use cases?)
I limit myself to materializing default arguments as in: def a(b=None): b = b or {} ... Because its a well known theme (and issue) of the mutability of default arguments of Python.
The only difference between "or" and "??" is that "??" is None only, right? At least to me, the given use case above does not justify the introduction of "??".
Yes, "or" suffices in that case.
B) It should short circuit, because the patterns it replaces (using ternary operator or "or") also do.
They look ugly and unpleasant because they remind you to reduce the usage of None; not to make dealing with it more pleasant.
"or" is easier to type (no special characters), I don't need to explain it to new staff, and it's more pleasant to the eye. I remember my missis telling me, after I showed her some C# code, that programmers tend to like weird special characters. Well, that might certainly be true. Special characters increase the visual noise and the mental strain when reading. They make the lines they are in special. I don't see anything special with "or" and with the single use case I have for it. :) Best, Sven
On 09/20/2015 11:57 PM, Chris Angelico wrote:
From reading these, I think the lowest-level/purest change would be to accommodate testing for "not None". Something I've always thought Python should be able to do in a nicer more direct way. We could add a "not None" specific boolean operators just by appending ! to them. while! x: <--> while x != None: if! x: <--> if x != None: a or! b <--> b if a != None else a a and! b <--> a if a != None else b not! x <--> x if x != None else None Those expressions on the right are very common and are needed because of None, False, and 0, are all False values. It would make for much simpler expressions and statements where they are used and be more efficient as these are likely to be in loops going over *many* objects. So it may also result in a fairly nice speed improvement for many routines. While the consistency argument says "if!" should be equivalent to "if not", I feel the practicality argument leans towards it being specific to "if obj != None". I believe testing for "not None" is a lot more common than testing for "None". Usually the only difference is how the code is arranged. I like how it simplifies/clarifies the common cases above. It would be especially nice in comprehensions. Cheers, Ron
Just to cut this thread short, I'm going to reject PEP 505, because ? is just too ugly to add to Python IMO. Sorry. I commend Mark for his clean write-up, without being distracted, giving some good use cases. I also like that he focused on a minimal addition to the language and didn't get distracted by hyper-generalizations. I also like that he left out f?(...) -- the use case is much weaker; usually it's the object whose method you're calling that might be None, as in title?.upper(). Some nits for the PEP: - I don't think it ever gives the priority for the ?? operator. What would "a ?? b or c" mean? - You don't explain why it's x ?? y but x ?= y. I would have expected either x ? y or x ??= y. - You don't explain or show how far ?. reaches; I assume x?y.z is equivalent to None if x is None else x.y.z, so you don't have to write x?.y?.z just to handle x.y.z if x is None. - The specification section is empty. -- --Guido van Rossum (python.org/~guido)
PEP-505 isn't anywhere close to being finished. I only submitted the draft because somebody off list asked me to send a draft so I could get a PEP number assigned. So I literally sent him what I had open in my text editor, which was just a few minutes of brain dumping and had several mistakes (grammatical and technical). If there's absolutely no point in continuing to work on it, I'll drop it. But from the outset, I thought the plan was to present this in its best light (and similar to the ternary operator PEP, offer several alternatives) if for no other reason than to have a good record of the reasoning for rejecting it. I'm sorry if I misunderstood the PEP process; I would have kept it to myself longer if I knew the first submission was going to be reviewed critically. I thought this e-mail chain was more of an open discussion on the general idea, not specifically a referendum on the PEP itself. On Mon, Sep 21, 2015 at 11:40 AM, Guido van Rossum <guido@python.org> wrote:
-- Mark E. Haase 202-815-0201
On Mon, Sep 21, 2015 at 8:58 AM, Mark E. Haase <mehaase@gmail.com> wrote:
I apologize for having misunderstood the status of your PEP. I think it would be great if you finished the PEP. As you know the ? operator has its share of fans as well as detractors, and I will happily wait until more of a consensus appears. I hope you can also add a discussion to the PEP of ideas (like some of the hyper-generalizations) that were considered and rejected -- summarizing a discussion is often a very important goal of a PEP. I think you have made a great start already! --Guido
-- --Guido van Rossum (python.org/~guido)
On 9/21/2015 1:07 PM, Guido van Rossum wrote:
Add me to the detractors of what I have read so far ;-). In arithmetic, 1/0 and 0/0 both stop the calculation. My hand calculator literally freezes until I hit 'on' or 'all clear'. Early computers also stopped, maybe with an instruction address and core dump. Three orthogonal solutions are: test y before x/y, so one can do something else; introduce catchable exceptions, so one can do something else; introduce contagious special objects ('inf' and 'nan'), which at some point can be tested for, so one can do something else. Python introduced 'inf' and 'nan' but did not use them to replace ZeroDivisionError. Some languages lacking exceptions introduce a contagious null object. Call it Bottom. Any operation on Bottom yields Bottom. Python is not such a language. None is anti-contagious; most operations raise an exception. I agree with Paul Moore that propagating None is generally a bad idea. It merely avoids the inevitable exception. Or is it inevitable? Trying to avoid exceptions naturally leads to the hypergeneralization of allowing '?' everywhere. Instead of trying to turn None into Bottom, I think a better solution would be a new, contagious, singleton Bottom object with every possible special method, all returning Bottom. Anyone could write such for their one use. Someone could put it on pypi to see if there how useful it would be. I agree with Ron Adam that the narrow issue is that bool(x) is False is sometimes too broad and people dislike of spelling out 'x is not None'. So abbreviate that with a unary operator; 'is not None', is a property of objects, not operators. I think 'x!' or 'x?', either meaning 'x is not None', might be better than a new binary operator. The former, x!, re-uses ! in something close to its normal meaning: x really exists. -- Terry Jan Reedy
On Mon, Sep 21, 2015 at 2:23 PM, Terry Reedy <tjreedy@udel.edu> wrote:
I think this is the PyMaybe solution. What I don't like about it is that it is dynamic -- when used incorrectly (or even correctly?) Bottom could end up being passed into code that doesn't expect it. That's bad -- "if x is None" returns False when x is Bottom, so code that isn't prepared for Bottom may well misbehave. In contrast, PEP 505 only affects code that is lexically near the ? operator. (You may see a trend here. PEP 498 is also carefully designed to be locally-scoped.)
I don't think the big issue is bool(x) being too broad. That's what the binary ?? operator is trying to fix, but to me the more useful operators are x?.y and x?[y], both of which would still require repetition of the part on the left when spelled using ??. This is important when x is a more complex expression that is either expensive or has a side-effect. E.g. d.get(key)?.upper() would currently have to be spelled as (some variant of) "None if d.get(key) is None else d.get(key).upper()" and the ?? operator doesn't really help for the repetition -- it would still be "d.get(key) ?? d.get(key).upper()". In general to avoid this repetition you have to introduce a local variable, but that's often awkward and interrupts the programmer's "flow". The ? solves that nicely. The key issue with this proposal to me is how it affects readability of code that uses it, given that there isn't much uniformity across languages in what ? means -- it could be part of a method name indicating a Boolean return value (Ruby) or a conditional operator (C and most of its descendents) or some kind of shortcut. So this is the issue I have to deal with (and thought I had dealt with by prematurely rejecting the PEP, but I've had a change of heart and am now waiting for the PEP to be finished). -- --Guido van Rossum (python.org/~guido)
On 9/21/2015 5:48 PM, Guido van Rossum wrote:
I agree with Paul Moore that propagating None is generally a bad idea. It merely avoids the inevitable exception.
To me, this is the key idea in opposition to proposals that make propagating None easier.
try: x = d.get(key).upper() except AttributeError: x = None is also a no-repeat equivalent when d.values are all strings. I agree than "x = d.get(key)?.upper()" is a plausible abbreviation. But I am much more likely to want "x = ''" or another exception as the alternative. I guess some other pythonistas like keeping None around more than I do ;-). -- Terry Jan Reedy
On Mon, Sep 21, 2015 at 3:45 PM, Terry Reedy <tjreedy@udel.edu> wrote:
(I didn't write that, you [Terry] did. It looks like our mailers don't understand each other's quoting conventions. :-( )
Eew. That try/except is not only very distracting and interrupts the flow of both the writer and the reader, it may also catch errors, e.g. what if the method being called raises an exception (not a problem with upper(), but definitely with user-defined methods). -- --Guido van Rossum (python.org/~guido)
On 09/21/2015 04:45 PM, Terry Reedy wrote:
I think it's one of those things that depends on what you're doing. From a web-development perspective, you rarely keep _anything_ around for very long, so there's rarely an issue of `None` sneaking in somewhere unexpectedly and then causing a surprise exception way down the line. Typical use cases are things like: "If this database query returns a User, I want to get their name and return that in the JSON dict from my API, otherwise I want None, which will be serialized to a JSON null, clearly indicating that there is no user here." My jaw dropped a bit when I saw it asserted in this thread that functions returning "useful value or None" is an anti-pattern. I write functions like that all the time, and I consider it a useful and necessary Python idiom. I would hate to rewrite all that code to either deal with exceptions or add default-value-argument boilerplate to all of them; when "no result" is an expected and normal possibility from a function, letting the calling code deal with None however it chooses is much nicer than either of those options. I don't love the ? syntax, but I would certainly use the feature discussed here happily and frequently. Carl
On Mon, Sep 21, 2015 at 3:56 PM, Carl Meyer <carl@oddbird.net> wrote:
+1 Some language features are "prescriptive," designed to encourage particular ways of writing things. Others are "respective," recognizing the variety of ways people write things and respecting that variety. Python has None and generally respects use of it. To say that using None is an anti-pattern is something I would strongly disagree with. Yes, NPE errors are a problem, but eliminating null/None does not eliminate those errors. It merely replaces one common error with an assortment of other errors. I like the feature. I have been asking for features like this for years and the number of times I have written the longer forms is too many to count. I like the ?. ?[] ?() ?? syntax. I think: (1) it's strongly related to the . [] () syntax; (2) any syntax that uses a keyword is either not syntactically related to . [] () or mixes a keyword and punctuation, both of which I dislike; (3) it's the same syntax as used in other languages (yes, Python is not C# or Dart but there's a good reason Python uses ^ for xor, ** for power, += for add to, etc.) --- Bruce
On 2015-09-21 15:56, Carl Meyer wrote:
I agree that it's a fine thing. The thing is, it's an API choice. If your API is "return such-and-such or None", then anyone who calls your function knows they have to check for None and do the right thing. I think this is fine if None really does indicate something like "no result". (The re module uses None return values this way.) It seems to me that a lot of the "problem" that these null-coalescing proposals are trying to solve is dealing with APIs that return None when they really ought to be raising an exception or returning some kind of context-appropriate empty value. If you're doing result = someFunction() and then result.attr.upper() and it's failing because result.attr is None, to me that's often a sign that the API is fragile, and the result object that someFunction returns should have its attr set to an empty string, not None. In other words, if you really want "a null result that I can call all kinds of string methods on and treat it like a string", you should be returning an empty string. If you want "a null result I can subscript and get an integer", you should be returning some kind of defaultdict-like object that has a default zero value. Or whatever. There isn't really such a thing as "an object to which I want to be able to do absolutely anything and have it work", because there's no type-general notion of what "work" means. From a duck-typing perspective, if you expect users to try to do anything with a value you return, what they might reasonably want to do should be a clue as to what kind of value you should return. That still leaves the use-case where you're trying to interoperate with some external system that may have missing values, but I don't see that as super compelling. Getting an exception when you do some['big']['json']['object']['value'] and one of the intermediate ones isn't there is a feature; the bug is the JavaScripty mentality of just silently passing around "undefined". To my mind, Python APIs that wrap such external data sources should ideally take the opportunity to improve on them and make them more Pythonic, by providing sensible, context-relevant defaults instead of propagating a generic "null" value willy-nilly. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown
On Mon, Sep 21, 2015 at 4:47 PM, Random832 <random832@fastmail.com> wrote:
Sorry, my bad. Indeed, x ?? y tries to fix the issue that "x or y" uses y if x is falsey. Still this seems a lesser problem to me than the problem solved by x?.a and x?[y]. Most of the time in my code it is actually fine to use the default if the LHS is an empty string. -- --Guido van Rossum (python.org/~guido)
On Mon, Sep 21, 2015 at 2:48 PM, Guido van Rossum <guido@python.org> wrote:
As we are in the process of writing a PEP and uses of ?/?? in other languages, why not speak about thecurrent usage of `?` / `??` in the Python community ? (I'll try to state only facts, excuse any passage that might seem like a personal opinion) Can the PEP include the fact that `? and `??` have been in use in the Scientific Python community for 10 to 14 years now, and that any Scientific Python user who have touched IPython will tell you that ? and ?? are for getting help. (?? try to pull the source, while ? does not, but let's not get into details). This include the fact that any IDE (like spyder) which use IPython under the hood have this feature. The usage of `?` is even visible on Python.org main page [3] (imgur screenshot), which invite the user to launch an interactive console saying:
object? -> Details about 'object', use 'object??' for extra details. In [1]:
leading the casual user thinking that this is a Python feature. This fact is even including in Books and introduction to python. Sometime without mentioning that the feature is IPython Specific, and does not work in Python repl/scripts. Examples in Cyrile's rossant "Learning IPython for Interactive Computing and Data Visualization"[1] introduce Python with the second code/repl example beeing about `?`. Book extract :
A google search also give for eaxample: Python for beginners online tutorial[2] which does rapidly the same: Tuto snippet:
Doing even worse as they replace the IPython prompt `In[x]:` with `>>>` literally showing that `>>> len?` works. Which imply that it should work on a plain Python REPL. As someone that have to regularly teach Python, and interact with new Python users, it will be hard to explain that `?` and `??` have different meaning depending on the context, and that most book on Scientific Python are wrong/inaccurate.
It will be even harder if we have to remove the usage of `?`/`??`[4]. I also want to note that the use of `?`/`??` is not present to just being or end of identifiers as it can also be used use to search for names:
But this usage is not as widespread as extracting help about objects, and seem less relevant, though I'm not sure:
Please take these fact into consideration when making a decision/writing the Pep. Thanks, -- M [1]: That's one of the only book for which I have (legally) the sources, and that I bother to grepped through. [2]: http://www.pythonforbeginners.com/basics/ipython-a-short-introduction [3]: http://imgur.com/d0Vs7Xr [4]: I'll have to hire a body guard to prevent people to pursue me to the end of the earth with a chainsaw. I'm sure you know that feeling.
On Tue, Sep 22, 2015 at 10:34 AM, Matthias Bussonnier <bussonniermatthias@gmail.com> wrote:
Are there any uses like this that would involve a question mark followed by some other punctuation? The main proposal would be for x?.y and similar; if "x ?? y" can't be used because of a conflict with ipython, I'm sure it could be changed ("x ?! y" would be cute). ChrisA
Hi Chris, On Mon, Sep 21, 2015 at 5:44 PM, Chris Angelico <rosuav@gmail.com> wrote:
As far as I can tell, no `?`/`??` behave in IPython like a unary operator[1] and don't conflict. We could distinguish from any use case so far mentioned in this discussion (as far as I can tell). I just hope that the PEP will not slide in the direction of `foo?` alone being valid and equivalent to `maybe(x)`, or returning an object. I'm concern about teaching/semantics as for me `x?.y` read like "The attribute y of the help of x" so roughly `help(x).y`. The `x ?? y` is less ambiguous (imho) as it is clearly an operator with the space on each side.
I'm sure it could be changed ("x ?! y" would be cute).
Is that the WTF operator ? [2] Joke aside, ?! should not conflict either, but `!` and `!!` also have their own meaning in IPython Like the following is valid (which is a useless piece of code, but to show the principles) if you are interested in the Python syntax extension we have. my_files = ! ls ~/*.txt for i,file in enumerate(my_files): raw = !cat $file !cat $raw > {'%s-%s'%(i,file.upper())} Thanks, -- M [1]: but is not really an operator, I'm not sure what a=print? ; a =?print or ?a=print would do. [2]: http://stackoverflow.com/questions/7825055/what-does-the-c-operator-do
Matthias Bussonnier writes:
But the syntax is extremely restrictive. Both "None? 1 + 2" and "None ?" are SyntaxErrors, as are "a?.attr" and even "*int*? ". Prefixing the help operator also gives help, and in that case whitespace may separate the operator from the word (apparently defined as non-whitespace, and any trailing detritus is ignored). Perhaps the prefix form (a little less natural for people coming directly from natural languages, I guess) should be emphasized -- there are no proposals for unary prefix use of "?" or "??". So, this is the kind of DWIM that I doubt will confuse many users, at least not for very long. Do you envision a problem keeping IPython facilities separate from Python language syntax? Technically, I think the current rules that divide valid IPython requests for help from Python syntax (valid or invalid) should continue to work. Whether it would confuse users, I doubt, but there are possible surprises for some (many?) users, I suppose. Definitely, it should be mentioned in the PEP, but Python syntax is something that Python defines; shells and language variants have to be prepared to deal with Python syntax changes.
leading the casual user thinking that this is a Python feature.
Casual users actually expect software to DWIM in my experience. The rule "leading help or stuck-right-on-the-end help works, elsewhere it means something else" (including no meaning == SyntaxError) is intuitively understood by them already, I'm sure. Also, I rather doubt that "casual users" will encounter "?." or "??" until they're not so casual anymore.
I've never had a question about the context-sensitivity of "%" in IPython. Have you?
and that most book on Scientific Python are wrong/inaccurate.
I'm afraid that's Scientific Python's cross to bear, not Python's.
It will be even harder if we have to remove the usage of `?`/`??`[4].
Not to worry about that. IPython can define its own syntax for parsing out help requests vs. Python syntax. I doubt you'll have to modify the current rules in any way.
But again "*int* ?" is a SyntaxError. This is the kind of thing most casual users can easily work with. (At least speakers of American English. In email text, my Indian students love to separate trailing punctuation from the preceding word for some reason, but they would certainly learn quickly that you can't do that in IPython.) Again, I agree it would be useful to mention this in the PEP, but as far as I can see there really isn't a conflict. The main thing I'd want to know to convince me there's a risk would be if a lot of users are confused by "%quickref" (an IPython command) vs. "3 % 2" (a Python expression).
Hi Stephen, Thanks for the response and the time you took to investigate,
Yes I’m not worrying for the time being, the current syntax proposal does not conflict, and I just wanted to describe a few usage and reference to consideration in the PEP. I prefer to give the PEP authors all the cards so that they can work out a proposal that fits majority of the people. I don’t want to let people write a pep, go through iteration and once they are happy with it complain that it does not fit my needs.
So, this is the kind of DWIM that I doubt will confuse many users, at least not for very long.
I do not think we are in contact with the same users. Yes, I see users confused by % syntax, just last Thursday, amy neighbor reported my that IPython was printing only for element of tuple, but was working for list: In [1]: print((1)) 1 In [2]: print([1]) [1] Yes people do get confused for %magics, vs modulo vs %-format, less because module is number, % for strings. But it gets betterwith time.
For no with current proposal, no no problem to keep them separate.
I can see the Python `??`/`?` vs IPython `?`/`??` being one explicit point in our docs/teaching/tutorial. I cannot say how much confusion this will be into our user head, I think the greater confusion will be the double meaning plus the fact that’s a Python 3.6+ only feature. So I doubt it will be taught before a few years. Though it is still another small difficulty. I guess we will start to get this king of experience with 3.5 now that @ is there both for decorator and __matmul__ (thanks for 3.5 in general BTW)
Definitely, it should be mentioned in the PEP,
Thanks,
Yes, we are prepared, but our user don’t always understand :-) I’m wondering if there wouldn’t be a way for interpreter to a actually help Python beta-test some syntax changes, at least at the REPL level. Like website do user testing.
In my domain people get confronted to advance syntax really rapidly, one feedback that I have for such weird syntax, especially when you are new to python, is that you don’t even now how to Google for this kind of thing (especially for non english speaker). Trying to put a name on *arg and **kwarg is hard, google is starting to get better, but still ignore chars like ?,+,- The google search for `Python` and `Python ??` seem to be identical.
Cf above yes, but more form the side where people don’t get what modulo mean, but fair enough it was non native english, and they were confused by indent/implement/increment being roughly the same word with 3 completely different meaning. Though the number of people I see using modulo is low, then they use numpy.mod on arrays once we get to numerics. And in string formatting we push for .format(*args, **kwargs). But I’ll try to gather some statistics around.
I hope that won’t change in the final PEP, and I’m not too worry, worse case we use more reg-ex, and shift the problem elsewhere :-)
French also separate punctuation, I’m still torn on that. But if we have to slightly change the rules, so be it.
Again, I agree it would be useful to mention this in the PEP, but as far as I can see there really isn't a conflict.
Happy you agree on point 1, and that you confirm point 2.
Will try to get more qualitative info. Thanks, -- M
IIRC, back when the ternary conditional was suggested, and the C-style ?: was proposed, Guido declared that no way was his language ever going to use ? an operator. So if IPython took that as a promise that they can use ? without fear of ambiguity, you can't blame them too much.... I'm not saying they have a right to expect/demand that Guido never change his mind about anything anywhere ever, just that maybe they get a little extra consideration on backward compatibility with their use of ? than with their use of ! or % (which have been in use as operators or parts of operators for decades).
On 21 September 2015 at 23:56, Carl Meyer <carl@oddbird.net> wrote:
Maybe my use of the phrase "anti-pattern" was too strong (i thought it implied a relatively mild "this causes problems"). Having the caller deal with problems isn't bad, but in my experience, too often the caller *doesn't* deal with the possibility None return. It feels rather like C's practice of returning error codes which never get checked. But as I said, YMMV, and my experience is clearly different from yours.
I don't love the ? syntax, but I would certainly use the feature discussed here happily and frequently.
If we're back to discussing indexing and attribute access rather than ??, maybe -> would work? obj->attr meaning None if obj is None else obj.attr obj->[n] meaning None if obj is None else obj[attr] obj->(args) meaning None if obj is None else obj(args) I think Matthias Bussonnier's point that ? and ?? is heavily used in IPython is a good one. Python traditionally doesn't introduce new punctuation (@ for decorators was AFAIK the last one). I thought that was precisely to leave the space of unused characters available for 3rd party tools. Paul
On 2015-09-22 03:59:07 +0000, Andrew Barnert via Python-ideas said:
I just wanted to quickly comment on what my original stance was regarding IPython's extensions to the base Python language. This was where I stood as I made decisions when the project was basically just me, and over time we've mostly adopted this as project policy. We fully acknowledge that IPython has to be a strict superset of the Python language, and we are most emphatically *not* a fork of the lanugage intended to be incompatible. We've added some extensions by hijacking a few characters that are invalid in the base language for thigns we deemed to be useful while working interactively, but we always accept that, if the language moves in our direction, it's our job to pack up and move again to a new location. In fact, that already happened once: before Python 2.4, our prefix for "magic functions" was the @ character, and when that was introduced as the decorator prefix, we had to scramble. We carefully decided to pick %, knowing that an existing binary operator would be unlikely to be added also as a new unary prefix. Now, accepting that as our reality doesn't mean that at least we don't want to *inform* you folks of what our uses are, so that at least you can consider them in your decision-making process. Since in some cases, that means there's an established ~ 15 years of a community with a habit of using a particular syntax for something, that may be confused if things change. So at least, we want to let you know. Due precisely to these recent conversations (I had a very similar thread a few days ago with Nick about the ! operator, which we also use in all kinds of nasty ways), we have started documenting more precisely all these differences, so the question "where exactly does IPython go beyond Python" can be answered in one place. You can see the progress here: https://github.com/ipython/ipython/pull/8821 We hope this will be merged soon into our docs, and it should help you folks have a quick reference for these questions. Finally, I want to emphasize that these things aren't really changing much anymore, this is all fairly stable. All these choices have by now stabilized, we only introduced the @ -> % transition when python 2.4 forced us, and more recently we introduced the notion of having a double-%% marker for "cell magics", but that was ~ 4 years ago, and it didn't require a new character, only allowing it to be doubled. Best, f
On 2015-09-23 02:21, Fernando Perez wrote:
From the examples I've seen, the "?" and "??" occur at the end of the line. The proposed 'operators' "?.", "?[", "?(" and "??" wouldn't occur at the end of the line (or, if they did, they'd be inside parentheses, brackets, or braces). So is there really a conflict, in practice?
beginning of line can happened too. ?print is equivalent to print?
As stated in previous mails, with current state of proposal no it does not conflict, we should be able to distinguish the two cases. We are just informing the pep authors and contributors of the syntax hijack that we did and currently have in IPython. -- M
On Mon, Sep 21, 2015 at 05:23:42PM -0400, Terry Reedy wrote:
I agree with Paul Moore that propagating None is generally a bad idea.
As I understand it, you and Paul are describing a basic, simple idiom which is ubiquitous across Python code: using None to stand in for "no such value" when the data type normally used doesn't otherwise have something suitable. Consequently I really don't understand what you and Paul have against it.
It merely avoids the inevitable exception.
I think you meant to say it merely *postpones* the inevitable exception. But that's wrong, there's nothing inevitable about an exception here. It's not *hard* to deal with "value-or-None". It's just tedious, which is why a bit of syntactic sugar may appeal. [...]
In one of my earlier posts, I discussed this Null object design pattern. I think it is an anti-pattern. If people want to add one to their own code, it's their foot, but I certainly don't want to see it as a built-in. Thank goodness Guido has already ruled that out :-)
I agree with Ron Adam that the narrow issue is that bool(x) is False is sometimes too broad and people dislike of spelling out 'x is not None'.
I don't think that is the motivation of the original proposal, nor is it one I particularly care about. I think that there is a level of inconvenience below which it's not worth adding yet more syntax just to save a few characters. That inconvenience is not necessarily just to do with the typing, it may be conceptual, e.g. we have "x != y" rather than "not x == y". I think that x is not None fails to reach that minimum level of inconvenience to justify syntactic sugar, but obj.method() if x is not None else None does exceed the level. So I am mildly interested in null-coalescing versions of attribute and item/key lookup, but not at all interested in making the "x is not None" part *alone* shorter.
Bring it back to the original post's motivating use-case. Slightly paraphrased, it was something like: value = obj.method() if obj is not None else None Having x! as a short-cut for "x is not None" makes this a bit shorter to write: value = obj.method() if obj! else None but it is still very boilerplatey and verbose compared to the suggested: value = obj?.method() -- Steve
On 22.09.2015 05:15, Steven D'Aprano wrote:
There is not a single "no such value". As I mentioned before, when discussing NULL values on the RDF mailing list, we discovered 6 or 7 domain-agnostic meanings.
I can tell from what I've seen that people use None for: all kinds of various interesting semantics depending on the variable, on the supposed type and on the function such as: - +infinity for datetimes but only if it signifies the end of a timespan - current datetime - mixing both - default item in a list like [1, 2, None, 4, 9] (putting in 5 would have done the trick) - ... Really? Just imagine a world where Python and other systems would have never invented None, NULLs or anything like that.
It's a sign of bad design. So, syntactic sugar does not help when doing toilet paper programming (hope that translation works for English). Best, Sven
On Wed, Sep 23, 2015 at 4:22 AM, Sven R. Kunze <srkunze@mail.de> wrote:
What this means is that your boundaries can be a datetime or None, where None means "no boundary at this end".
- current datetime - mixing both
I don't know of a situation where None means "now"; can you give an example?
- default item in a list like [1, 2, None, 4, 9] (putting in 5 would have done the trick)
What does this mean? Is this where you're taking an average or somesuch, and pretending that the None doesn't exist? That seems fairly consistent with SQL. Mostly, this does still represent "no such value". ChrisA
On 23.09.2015 00:53, Chris Angelico wrote:
Yes.
range_start = <proper datetime> range_end = <proper datetime or one of the above> So, if you need something that ranges from 2015-01-01 to now, you basically say the range is expanding. Depending on the function/method, it either means until forever, or now.
Imagine you render (as in HTML and the like) 1, 2, 4, 9 and instead of the None you render a 3. Now, the rendering engine needs to special-check None to put in a pre-defined value. Furthermore, all places where you need that list [1, 2, None, 9], you basically need to special-check None and act appropriately. (Of course it was not that simple but you get the idea. The numbers stand for fairly complex objects drawn from the database.)
Mostly, this does still represent "no such value".
Point was "no such value" sucks. It can be a blend of every other value and semantics depending on the function, type and so forth. It's too convenient that people would not use it. As the example with the list shows us, the 3 could have easily be put into the database as it behaves exactly the same as the other objects. The same goes for the special datetime objects. The lack of thinking and appropriate default objects, lead to the usage of None. People tend to use None for everything that is special and you end up with something really nasty to debug. Not why people don't find it problematic when I said "we found 6/7 domain-agnostic semantics for NULL". The "no such value" can be any of them OR a blend of them. That, I don't want to see in the code; that's all. Btw. having the third issue of above, I could add another domain-agnostic meaning for None: "too lazy to create a pre-defined object but instead using None".
What about re-using try? Crystal does this ( http://play.crystal-lang.org/#/r/gf5): v = "ABC" puts nil == v.try &.downcase # prints true v = nil puts nil == v.try &.downcase # prints false Python could use something like: v = 'ABC' print(v try.downcase is None) # prints False v = None print(v try.downcase is None) # prints True (Of course, the syntax would be a little less...weird!) On Mon, Sep 21, 2015 at 10:40 AM, Guido van Rossum <guido@python.org> wrote:
-- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something’s wrong. http://kirbyfan64.github.io/
On 09/21/2015 03:44 PM, Terry Reedy wrote:
Yes
My observation is that because None is the default return value from functions (and methods), it is already a special case. While that isn't directly related to None values in general, I think it does lend weight to treating it specially. Having None specific bool-type operators is both cleaner and more efficient and avoids issues with false values. The byte code might look like this...
It would not be sensitive to False values. Applying the op to the object wouldn't quite work as expected. What would x! return? True, or the object if not None? And if it returns the object, what does this do when the value is 0 or False, or an empty container. result = x! or y # not the same as result = x or! y The maybe(x) function would work the same as x! in this case. I also think a trailing unary operator is kind of weird. But I get the feeling from Guido response about over generalizations that it may be too big of a change, and I agree it is a new concept I haven't seen anywhere else. Maybe one to think about over time, and not to rush into. Cheers, Ron
Mark E. Haase writes:
To me, it's less defensible. Eg, currently TOOWTDI for "??" is the idiom quoted above. I sorta like the attribute access, attribute fetch, and function call versions, though I probably won't use them. Also some functions need to accept None as an actual argument, and the module defines a module-specific sentinel. The inability to handle such sentinels is a lack of generality that the "x if x is not sentinel else y" idiom doesn't suffer from, so "??" itself can't become TOOWTDI. I don't write "def foo(default):" (ever that I can recall), so using "default" in retries = default if default is not None else cls.DEFAULT confuses me. Realistically, I would be writing retries = retries if retries is not None else cls.RETRIES (or perhaps the RHS would be "self.retries"). That doesn't look that bad to me (perhaps from frequent repetition). It's verbose, but I don't see a need to chain it, unlike "?.". For "?.", some Pythonistas would say "just don't", but I agree that often it's natural to chain.
Worse? It's true that it's more risky because it's all falsies, not just the None sentinel, but I think "consenting adults" applies here. I don't know about the packages you looked at, but I often use "x = s or y" where I really want to trap the falsey value of the expected type, perhaps as well as None, and I use the "x if s is not sentinel else y" idiom to substitute default values. I also use "or" in scripty applications and unit test setup functions where I want compact expression and I don't expect long-lived objects to be passed so I can easily figure out where the non-None falsey came from anyway.
A) Is coalesce a useful feature? (And what are the use cases?)
Yes, for the whole group of operators. Several use cases for the other operators have already been proposed, but I wouldn't use them myself in current or past projects, and don't really foresee that changing. -0 for the group on the IAGNI principle. But for "??" specifically, it's just more compact AFAICS. I don't see where I would use x ?? y ?? z, so the compactness doesn't seem like that great a benefit. In practice, I think the use cases for "??" would be a strict subset of the use cases for the ternary operator, so you have to argue that "this special case *is* special enough" to have its own way to do it. I don't think it is. -1
C) If it should be an operator, is "??" an ugly spelling?
>>> retries = default ?? cls.DEFAULT
Looks like metasyntax from pseudo-code that didn't get fixed to me. That would probably change if other ?x operators were added though. I have no comment on short-circuiting (no relevant experience), or keyword vs. punctuation spellings. On second thought:
I kinda like this if-less else syntax for the symmetry with else-less if. But on second thought I think it would persistently confuse me when reading, because it would be extremely natural to expect it to be another way of spelling "default or cls.DEFAULT". "try ... else ..." also has its attraction, but I suppose that would fail for the same reasons that the ternary operator is spelled "x if y else z" rather than "if y then x else z".
On 21.09.2015 11:05, Andrew Barnert via Python-ideas wrote:
That's cool. It reads nice (at least for a non-native speaker). Also chaining else reads nice: final_value = users_value else apps_value else systems_value
Use parentheses if you mix up if-else and else. ;) Btw. the same applies for: a + b * c + d If you don't know from you education that b*c would have been evaluated first, then it's not obvious either. Best, Sven
Andrew Barnert writes:
I don't know what an LL(1) parser could do offhand. As a human, I would parse that greedily as (a if b else c) else d. But the point's actually moot, as I'm -1 on the "??" operator in any form in favor of the explicit "a if a is not None else b" existing syntax. And to be honest, the fact that a truly symmetric "if-less else" would have "or" semantics, not "??" semantics, bothers me more than the technical issue of whether anybody could actually parse it.
On 21 September 2015 at 03:35, Mark E. Haase <mehaase@gmail.com> wrote:
A) Is coalesce a useful feature? (And what are the use cases?)
There seem to be a few main use cases: 1. Dealing with functions that return a useful value or None to signal "no value". I suspect the right answer here is actually to rewrite the function to not do that in the first place. "Useful value or None" seems like a reasonable example of an anti-pattern in Python. 2. People forgetting a return at the end of the function. In that case, the error, while obscure, is reasonable, and should be fixed by fixing the function, not by working around it in the caller. 3. Using a library (or other function outside your control) that uses the "useful value or None" idiom. You have to make the best of a bad job here, but writing an adapter function that hides the complexity doesn't seem completely unreasonable. Nor does just putting the test inline and accepting that you're dealing with a less than ideal API. 4. Any others? I can't think of anything. Overall, I don't think coalesce is *that* useful, given that it seems like it'd mainly be used in situations where I'd recommend a more strategic fix to the code.
B) If it is useful, is it important that it short circuits? (Put another way, could a function suffice?)
Short circuiting is important, but to me that simply implies that the "useful value or None" approach is flawed *because* it needs short-circuiting to manage. In lazy languages like Haskell, the Maybe type is reasonable because short-circuiting is a natural consequence of laziness, and so not a problem. In languages like C#, the use of null as a sentinel probably goes back to C usage of NULL (i.e., it may not be a good approach there either, but history and common practice make it common enough that a fix is needed).
C) If it should be an operator, is "??" an ugly spelling?
>>> retries = default ?? cls.DEFAULT
Arbitrary punctuation as operators is not natural in Python, something like this should be a keyword IMO.
Reusing existing keywords (specifically, all of the above) looks clumsy and forced to me. I agree that proposals to add a new keyword will probably never get off the ground, but none of the above suggestions look reasonable to me, and I can't think of anything else that does (particularly if you add "must be parseable" as a restriction!) Overall, I'm -0.5 on a "coalesce" operator. I can't see it having sufficient value, and I can't think of a syntax I'd consider justifying it. But if someone were to come up with a Guido-like blindingly obvious way to spell the operation, I would be fine with that (and may even start using it more often than I think). Paul
On Mon, Sep 21, 2015 at 8:55 PM, Paul Moore <p.f.moore@gmail.com> wrote:
The alternative being to raise an exception? It's generally easier, when you can know in advance what kind of object you're expecting, to have a None return when there isn't one. For example, SQLAlchemy has .get(id) to return the object for a given primary key value, and it returns None if there's no such row in the database table - having to wrap that with try/except would be a pain. This isn't an error condition, and it's not like the special case of iteration (since an iterator could yield any value, it's critical to have a non-value way of signalling "end of iteration"). I don't want to see everything forced to "return or raise" just because someone calls this an anti-pattern. ChrisA
On 21 September 2015 at 14:27, Chris Angelico <rosuav@gmail.com> wrote:
Agreed, that's not what should happen. It's hard to give examples without going into specific cases, but as an example, look at dict.get. The user can supply a "what to return if the key doesn't exist" argument. OK, many people leave it returning the default None, but they don't *have* to - dict.get itself doesn't do "useful value or None", it does "useful value or user-supplied default". All I'm saying is that people should look at *why* their functions return None instead of a useful result, and see if they can do better. My contention is that (given free rein) many times they can. Of course not all code has free rein, not all developers have the time to look for perfect APIs, etc. But in that case, returning a placeholder None (and accepting a little ugliness at the call site) isn't an impossible price to pay. Nothing more than "I don't think the benefit justifies adding a new operator to Python". Paul
On 21.09.2015 15:27, Chris Angelico wrote:
I don't think both approaches are mutual exclusive. They can both exist and provide whenever I need the right thing. Depending on the use-case, one needs to decide: If I know, the value definitely needs to be a dictionary, I use dict[...]. If I know, the value is definitely optional and I can't do anything about it, I use dict.get('key'[, default]). If I definitely don't know, I use dict[...] to get my hands on a real example with out that key if that every happens and don't waste time for special-handling a possible None return value. Best, Sven
On Mon, Sep 21, 2015 at 11:55:55AM +0100, Paul Moore wrote:
I think that's a bit strong. Or perhaps much too strong. There are times where you can avoid the "None or value" pattern, since there is a perfectly decent empty value you can use instead of None. E.g. if somebody doesn't have a name, you can use "" instead of None, and avoid special treatment. But that doesn't always work. Suppose you want an optional (say) Dog object. There isn't such a thing as an empty Dog, so you have to use some other value to represent the lack of Dog. One could, I suppose, subclass Dog and build a (singleton? borg?) NoDog object, but that's overkill and besides it doesn't scale well if you have multiple types that need the same treatment. So I don't think it is correct, or helpful, to say that we should avoid the "None versus value" pattern. Sometimes we can naturally avoid it, but it also has perfectly reasonable uses.
Go back to the original use-case given, which, paraphrasing, looks something like this: result = None if value is None else value['id'].method() I don't think we can reject code like the above out of hand as un-Pythonic or an anti-pattern. It's also very common, and a little verbose. It's not bad when the value is a name, but sometimes it's an expression, in which case it's both verbose and inefficient: result = None if spam.eggs(cheese) is None else spam.eggs(cheese)['id'].method() Contrast: result = spam.eggs(cheese)?['id'].method() which only calculates the expression to the left of the ?[ once. An actual real-life example where we work around this by using a temporary name that otherwise isn't actually used for anything: mo = re.match(needle, haystack) if mo: substr = mo.group() else: substr = None I think it is perfectly reasonable to ask for syntactic sugar to avoid having to write code like the above: substr = re.match(needle, haystack)?.group() That's not to say we necessarily should add sugar for this, since there is no doubt there are disadvantages as well (mostly that many people dislike the ? syntax), but in principle at least it would certainly be nice to have and useful.
Nothing needs short-circuiting, at least in a language with imperative assignment statements. You can always avoid the need for short-circuits with temporary variables, and sometimes that's the right answer: not everything needs to be a one-liner, or an expression. But sometimes it is better if it could be.
C) If it should be an operator, is "??" an ugly spelling?
>>> retries = default ?? cls.DEFAULT
I assume the ?? operator is meant as sugar for: retries = cls.DEFAULT if default is None else default I prefer to skip the "default" variable and use the standard idiom: if retries is None: retries = cls.DEFAULT I also worry about confusion caused by the asymmetry between ?? and the other three ? cases: # if the left side is None, return None, else evaluate the right side spam?.attr spam?['id'] spam?(arg) # if the left side is None, return the right side, else return the left spam ?? eggs but perhaps I'm worried over nothing. -- Steve
On 21 September 2015 at 15:41, Steven D'Aprano <steve@pearwood.info> wrote:
Well, (1) Mark had focused on the "coalesce" operator ??, not the ?. variant, and that is less obviously useful here, and (2) I find the former version more readable. YMMV on readability of course - which is why I added the proviso that if someone comes up with an "obviously right" syntax, I may well change my mind. But the options suggested so far are all far less readable than a simple multi-line if (maybe with a temporary variable) to me, at least. By the way, in your example you're passing on the "none or useful" property by making substr be either the matched value or None. In real life, I'd probably do something more like mo = re.match(needle, haystack) if mo: process(mo.group()) else: no_needle() possibly with inline code if process or no_needle were simple. But it is of course easy to pick apart examples - real code isn't always that tractable. For example we get (what seems to me like) a *lot* of bug reports about "None has not attribute foo" style errors in pip. My comments here are based on my inclinations about how I would fix them in pip - I'd always go back to *why* we got a None, and try to avoid getting the None in the first place. But that's not always easy to do. Again, of course, YMMV. Paul
On 9/21/2015 10:56 AM, Paul Moore wrote:
By the way, in your example you're passing on the "none or useful" property by making substr be either the matched value or None.
I agree that dealing with None immediately is better. In real
try: process(re.match(needle, haystack).group()) except AttributeError: # no match no_needle() is equivalent unless process can also raise AttributeError. -- Terry Jan Reedy
On Mon, Sep 21, 2015 at 04:23:03PM -0400, Terry Reedy wrote:
It is difficult to guarantee what exceptions a function will, or won't, raise. Even if process() is documented as "only raising X", I wouldn't be confident that the above might not disguise a bug in process as "no needle". This is why the standard idiom for using re.match is to capture the result first, then test for truthiness (a MatchObject), or None-ness, before processing it. -- Steve
On Sun, Sep 20, 2015 at 12:05:52PM +0100, Paul Moore wrote:
Sadly, I think it does. Guido has (I think) ruled out the Null object design pattern, which makes me glad because I think it is horrid. But your Maybe class below is a watered down, weak version that (in my opinion) isn't worth bothering with. See below. class Maybe: def __getattr__(self, attr): return None def __getitem__(self, idx): return None def __call__(self, *args, **kw): return None def maybe(obj): return Maybe() if obj is None else obj And in action: py> maybe("spam").upper() # Works fine. 'SPAM' py> maybe(None).upper() Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'NoneType' object is not callable It also fails for chained lookups: maybe(obj).spam['id'].ham will fail for the same reason. You could write this: maybe(maybe(obj).upper)() maybe(maybe(maybe(obj).spam)['id']).ham but that's simply awful. Avoiding that problem is why the Null object returns itself, but we've rightly ruled that out. This is why I think that if this is worth doing, it has to be some sort of short-circuiting operator or pseudo-operator: expression ? .spam.eggs.cheese can short-circuit the entire chain .spam.eggs.cheese, not just the first component. Otherwise, I don't think it's worth doing. -- Steve
On Fri, Sep 18, 2015, at 18:37, Chris Angelico wrote:
More or less - it'd only look up the attribute once.
?[ has the benefit of being consistent with ?. - and ?(, for that matter. It actually suggests a whole range of null-coalescing operators. ?* for multiply? A lot of these things are done already by the normal operators for statically-typed nullable operands in C#. That could get hairy fast - I just thought of a radical alternative that I'm not even sure if I support: ?(expr) as a lexical context that changes the meaning of all operators.
I've now read PEP 505, and I would like to comment. Executive summary: - I have very little interest in the ?? and ?= operators, but don't object to them: vote +0 - I have a little more interest in ?. and ?[ provided that the precedence allows coalescing multiple look-ups from a single question mark: vote +1 - if it uses the (apparent?) Dart semantics, I am opposed: vote -1 - if the syntax chosen uses || or !| as per Nick's suggestion, I feel the cryptic and ugly syntax is worse than the benefit: vote -1 In more detail: I'm sympathetic to the idea of reducing typing, but I think it is critical to recognise that reducing typing is not always a good thing. If it were, we would always name our variables "a", "b" etc, the type of [] would be "ls", and we would say "frm col impt ODt". And if you have no idea what that last one means, that's exactly my point. Reducing typing is a good thing *up to a point*, at which time it becomes excessively terse and cryptic. One of the things I like about Python is that it is not Perl: it doesn't have an excess of punctuation and short-cuts. Too much syntactic sugar is a bad thing. The PEP suggests a handful of new operators: (1) Null Coalescing Operator spam ?? eggs equivalent to a short-circuiting: spam if spam is not None else eggs I'm ambivalent about this. I don't object to it, but nor does it excite me in the least. I don't think the abbreviated syntax gains us enough in expressiveness to make up for the increase in terseness. In its favour, it can reduce code duplication, and also act as a more correct alternative to `spam or eggs`. (See the PEP for details.) So I'm a very luke-warm +0 on this part of the PEP. (2) None coalescing assignment spam ?= eggs being equivalent to: if spam is None: spam = eggs For the same reasons as above, I'm luke-warm on this: +0. (3) Null-Aware Member Access Operator spam?.attr being equivalent to spam.attr if spam is not None else None To me, this passes the test "does it add more than it costs in cryptic punctuation?", so I'm a little more positive about this. If my reading is correct, the PEP underspecifies the behaviour of this when there is a chain of attribute accesses. Consider: spam?.eggs.cheese This can be interpreted two ways: (a) (spam.eggs.cheese) if spam is not None else None (b) (spam.eggs if spam is not None).cheese but the PEP doesn't make it clear which behaviour they have in mind. Dart appears to interpret it as (b), as the reference given in the PEP shows this example: [quote] You can chain ?. calls, for example: obj?.child?.child?.getter [quote] http://blog.sethladd.com/2015/07/null-aware-operators-in-dart.html That would seem to imply that obj?.child.child.getter would end up trying to evaluate null.child if the first ?. operator returned null. I don't think the Dart semantics is useful, indeed it is actively harmful in that it can hide bugs: Suppose we have an object which may be None, but if not, it must have an attribute spam which in turn must have an attribute eggs. This implies that spam must not be None. We want: obj.spam.eggs if obj is not None else None Using the Dart semantics, we chain ?. operators and get this: obj?.spam?.eggs If obj is None, the expression correctly returns None. If obj is not None, and obj.spam is not None, the expression correctly returns eggs. But it is over-eager, and hides a bug: if obj.spam is None, you want to get an AttributeError, but instead the error is silenced and you get None. So I'm -1 with the Dart semantics, and +1 otherwise. (3) Null-Aware Index Access Operator spam?[item] being similar to spam.attr. Same reasoning applies to this as for attribute access. Nick has suggested using || instead of ??, and similar for the other operators. I don't think this is attractive at all, but the deciding factor which makes Nick's syntax a -1 for me is that it is inconsistent and confusing. He has to introduce a !| variation, so the user has to remember when to use two |s and when to use a ! instead, whether the ! goes before or after the | and that !! is never used. -- Steve
Steven, thanks for the reply. Just to clarify: the current PEP draft was not meant to be read -- it was just a placeholder to get a PEP # assigned. I didn't realize that new PEPs are published in an RSS feed! I do appreciate your detailed feedback, though. Your interpretation of Dart's semantics is correct, and I agree that's absolutely the wrong way to do it. C# does have the short-circuit semantics that you're looking for. To you, and to everybody else in this thread: I am reading every single message, and I'm working on a draft worthy of your time and attention that incorporates all of these viewpoints and offers several, competing alternatives. I will announce the draft on this list when I'm further along. Proposing a new operator is tremendously difficult in Python, because this community doesn't like complex or ugly punctuation. (And adding a new keyword won't happen any time soon.) A similar debate surrounded the ternary operator PEP[1]. That PEP's author eventually held a vote on the competing alternatives, including an option for "don't do anything". I'm hoping to hold a similar referendum on this PEP once I've had time to work on it a bit more. [1] https://www.python.org/dev/peps/pep-0308/ On Wed, Sep 23, 2015 at 12:47 PM, Steven D'Aprano <steve@pearwood.info> wrote:
-- Mark E. Haase 202-815-0201
On Wed, Sep 23, 2015 at 11:47 AM, Steven D'Aprano <steve@pearwood.info> wrote:
from collections import OrderedDict
I have to kind of agree here. In reality, I don't see any issues like this with approach (a).
-- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something’s wrong. http://kirbyfan64.github.io/
On 23 September 2015 at 17:47, Steven D'Aprano <steve@pearwood.info> wrote:
I've now read PEP 505, and I would like to comment.
Having read the various messages in this thread, and then your summary (which was interesting, because it put a lot of the various options next to each other) I have to say: 1. The "expanded" versions using if..else are definitely pretty unreadable and ugly (for all the variations). But in practice, I'd be very unlikely to use if expressions in this case - I'd be more likely to expand the whole construct, probably involving an if *statement*. Comparing a multi-line statement to an operator is much harder to do in a general manner. So I guess I can see the benefits, but I suspect the operators won't be used in practice as much as people are implying (in much the same way that the use if the if expression is pretty rare in real Python code, as opposed to examples). 2. All of the punctuation-based suggestions remain ugly to my eyes. ? is too visually striking, and has too many other associations for me ("help" in IPython, and as a suffix for variable names from Lisp). Nick's || version looked plausible, but the inconsistent !| variations bother me. 3. People keep referring to "obj ?? default" in comparison to "obj or default". The comparison is fine - as is the notion that we are talking about a version that simply replaces a truth test with an "is None" test. But to me it also says that we should be looking for a keyword, not a punctuation operator - the "or" version reads nicely, and the punctuation version looks very cryptic in comparison. I can't think of a good keyword, or a viable way to use a keyword for the ?. ?[ and ?( variations, but I wish I could. Summary - I don't mind the addition of the functionality, although I don't think it's crucial. But I really dislike the punctuation. The benefits don't justify the cost for me. Paul
On Fri, Sep 18, 2015 at 10:42:59AM -0700, Mark Haase wrote:
Hmmm, I use this NullObject class when I want to do stuff similar to what you've described: class NullObject(object): """ This is a helper class that does its best to pretend to be forgivingly null-like. >>> n = NullObject() >>> n None >>> n.foo None >>> n.foo.bar.moo None >>> n.foo().bar.moo(True).cat().hello(False, abc=123) None >>> n.hornet(afterburner=True).shotdown(by=n().tomcat) None >>> n or 1 1 >>> str(n) '' >>> int(n) 0 >>> len(n) 0 """ def __getattr__(self, name): return self def __getitem__(self, item): return self def __call__(self, *args, **kwds): return self def __nonzero__(self): return False def __repr__(self): return repr(None) def __str__(self): return '' def __int__(self): return 0 def __len__(self): return 0 Source: https://github.com/tpn/tpn/blob/master/lib/tpn/util.py#L1031 Sample use: https://github.com/enversion/enversion/blob/master/lib/evn/change.py#L1300 class ChangeSet(AbstractChangeSet): @property def top(self): """ Iff one child change is present, return it. Otherwise, return an instance of a NullObject. """ if self.child_count != 1: return NullObject() else: top = None for child in self: top = child break return top @property def is_tag_create(self): return self.top.is_tag_create @property def is_tag_remove(self): return self.top.is_tag_remove @property def is_branch_create(self): return self.top.is_branch_create @property def is_branch_remove(self): return self.top.is_branch_remove Having self.top potentially return a NullObject simplifies the code for the four following properties.
If you can alter the part that creates `created` or `updated` to return a NullObject() instead of None when applicable, you could call `created.isoformat()` with out the addition clause.
Thanks, Mark
Trent.
On Sep 18, 2015, at 11:21, Trent Nelson <trent@snakebite.org> wrote:
I believe it was raised as a side issue during other discussions (conditional expressions, exception-handling expressions, one of the pattern-matching discussions), but I personally can't remember anyone ever writing a serious proposal. I think Armin from PyPy also has a blog post mentioning the idea somewhere, as a spinoff of his arguments against PEP 484 (which turned into a more general "what's wrong with Python's type system and what could be done to fix it). One last place to look, although it'll be harder to search for, is every time people discuss whether things like dict.get are a wart on the language (because there should be a fully general way to do the equivalent) or a feature (because it's actually only useful in a handful of cases, and it's better to mark them explicitly than to try to generalize). But my guess is that the discussion hasn't actually been had in sufficient depth to avoid having it here. (Although even if I'm right, that doesn't mean more searching isn't worth doing--to find arguments and counter arguments you may have missed, draw parallels to successes and failures in other languages, etc.) And, even if Guido hates the idea out of hand, or someone comes up with a slam-dunk argument against it, this could turn into one of those cases where it's worth someone gathering all the info and shepherding the discussion just to write a PEP for Guido to reject explicitly. Personally, for whatever my opinion is worth (not that much), I don't have a good opinion on how it would work in Python without seeing lots of serious examples or trying it out. But I think this would be relatively easy to hack in at the tokenizer level with a quick&dirty import hook. I'll attempt it some time this weekend, in hopes that people can play with the feature. Also, it might be possible to do it less hackily with MacroPy (or it might already be part of MacroPy--often Haoyi's time machine is as good as Guido's).
This is a very Smalltalk-y solution, which isn't a bad thing. I think having a singleton instance of NullObject (like None is a singleton instance of NoneType) so you can use is-tests, etc. might make it better, but that's arguable. The biggest problem is that you have to write (or wrap) every API to return NullObjects instead of None, and likewise to take NullObjects. (And, if you use a PEP 484 checker, it won't understand that an optional int can hold a NullObject.) Also, there's no way for NullObject to ensure that spam(NullObject) returns NullObject for any function spam (or, more realistically, for any function except special cases, where it's hard to define what counts as a special case but easy to understand intuitively). And finally, there's no obvious way to make NullObject raise when you want it to raise. With syntax for nil coalescing, this is easy: ?. returns None for None, while . raises AttributeError. With separate types instead, you're putting the distinction at the point (possibly far away) where the value is produced, rather than the point where it's used. As a side note, my experience in both Smalltalk and C# is that at some point in a large program, I'm going to end up hackily using a distinction between [nil] and nil somewhere because I needed to distinguish between an optional optional spam that "failed" at the top level vs. one that did so at the bottom level. I like the fact that in Haskell or Swift I can actually distinguish "just nil" from "nil" when I need to but usually don't have to (and the code is briefer when I don't have to), but I don't know whether that's actually essential (the [nil]) hack almost always works, and isn't that hard to read if it's used sparsely, which it almost always is).
FWIW, I generally hate odd punctuation like this (@ notwithstanding) but I'm not against the idea itself -- maybe a different syntax can be invented, or maybe I could be persuaded that it's okay. On Fri, Sep 18, 2015 at 12:28 PM, Andrew Barnert via Python-ideas < python-ideas@python.org> wrote:
-- --Guido van Rossum (python.org/~guido)
Andrew, thanks for putting together that hack. I will check it out. Guido, you lost me at "hate odd punctuation ... maybe a different syntax can be invented". Do you mean introducing a new keyword or implementing this as a function? Or do you mean that some other punctuation might be less odd? I'm willing to write a PEP, even if it's only purpose is to get shot down. On Fri, Sep 18, 2015 at 3:45 PM, Guido van Rossum <guido@python.org> wrote:
-- Mark E. Haase 202-815-0201
On Sep 18, 2015, at 12:28, Andrew Barnert via Python-ideas <python-ideas@python.org> wrote:
Personally, for whatever my opinion is worth (not that much), I don't have a good opinion on how it would work in Python without seeing lots of serious examples or trying it out. But I think this would be relatively easy to hack in at the tokenizer level with a quick&dirty import hook. I'll attempt it some time this weekend, in hopes that people can play with the feature. Also, it might be possible to do it less hackily with MacroPy (or it might already be part of MacroPy--often Haoyi's time machine is as good as Guido's).
You can download a quick&dirty hack at https://github.com/abarnert/nonehack This only handles the simple case of identifier?.attribute; using an arbitrary target on the left side of the . doesn't work, and there are no other none-coalescing forms like ?(...) or ?[...]. (The latter would be easy to add; the former, I don't think so.) But that's enough to handle the examples in the initial email. So, feel free to experiment with it, and show off code that proves the usefulness of the feature. Also, if you can think of a better syntax that will make Guido less sad, but don't know how to implement it as a hack, let me know and I'll try to do it for you.
On Sat, Sep 19, 2015 at 3:42 AM, Mark Haase <mehaase@gmail.com> wrote:
Python generally doesn't special-case None, so having a bit of magic that works only on that one object seems a little odd. For comparison purposes, Pike has something very similar to what you're describing, but Pike *does* treat the integer 0 as special, so it makes good sense there. Pike code that wants to return "a thing or NULL" will return an object or the integer 0, where Python code will usually return an object or None. I can't think of any situation in Python where the language itself gives special support to None, other than it being a keyword. You're breaking new ground. But in my opinion, the practicality is worth it. The use of None to represent the SQL NULL value [1], the absence of useful return value, or other "non-values", is pretty standard. I would define the operator pretty much the way you did above, with one exception. You say: created?.isoformat() # is equivalent to created.isoformat() if created is not None else None but this means there needs to be some magic, because it should be equally possible to write: created?.year # equivalent to created.year if created is not None else None which means that sometimes it has to return None, and sometimes (lambda *a,**ka: None). Three possible solutions: 1) Make None callable. None.__call__(*a, **ka) always returns None. 2) Special-case the immediate call in the syntax, so the equivalencies are a bit different. 3) Add another case: func?(args) evaluates func, and if it's None, evaluates to None without calling anything. Option 1 would potentially mask bugs in a lot of unrelated code. I don't think it's a good idea, but maybe others disagree. Option 2 adds a grammatical distinction that currently doesn't exist. When you see a nullable attribute lookup, you have to check to see if it's a method call, and if it is, do things differently. That means there's a difference between these: func = obj?.attr; func() obj?.attr() Option 3 requires a bit more protection, but is completely explicit. It would also have use in other situations. Personally, I support that option; it maintains all the identities, is explicit that calling None will yield None, and doesn't need any magic special cases. It does add another marker, though: created?.isoformat?() # is equivalent to created.isoformat() if created is not None and created.isoformat is not None else None As to the syntax... IMO this needs to be compact, so ?. has my support. With subscripting, should it be "obj?[idx]" or "obj[?idx]" ? FWIW Pike uses the latter, but if C# uses the former, there's no one obvious choice. ChrisA [1] Or non-value, depending on context
On 18/09/15 23:37, Chris Angelico wrote:
Python generally doesn't special-case None, so having a bit of magic that works only on that one object seems a little odd.
So the answer here is to introduce a "magic" hook that None can make use of (but also other classes). I can't think of an appropriate word, so I'll use "foo" to keep it suitably abstract. If the foo operator uses the magic method "__foo__" to mean "return an object to be used in place of the operand should it be considered ... false? [or some other definition - I'm not sure]" then any class can implement that method to return an appropriate proxy object. If that was a postfix operator which has a high precedence, then: bar = foo? bar.isoformat() and the original syntax suggestion: bar = foo?.isoformat() ... are equivalent. "?." is not a new operator. "?" is. This is essentially a slight refinement of Chris's case 3 -
E.
Apologies for the self-reply. I just wanted to clarify a couple of things. On 18/09/15 23:56, Erik wrote:
Not "false", I think. The "foo" operator is meant to mean "I will go on to use the resulting object in any way imaginable and it must cope with that and return a value from any attempts to use it that will generally mean 'no'" (*).
Which is clearly wrong - the first part should be: baz = foo? bar = baz.isoformat() E. (*) Should we call the operator "shrug"?
On 19.09.2015 01:47, Ryan Gonzalez wrote:
What about "apply"? It's the closest thing to "fmap" I can think of that won't coblnfuse people...
Are you sure? I think "maybe" better reflects the purpose of "?". Nevertheless, I would love to see support for the maybe monad in Python. Best, Sven
On Sep 18, 2015, at 16:58, Sven R. Kunze <srkunze@mail.de> wrote:
I think this, and the whole discussion of maybe and fmap, is off the mark here. It's trivial to create a maybe type in Python. What's missing is the two things that make it useful: (1) pattern matching, and (2) a calling syntax and a general focus on HOFs that make fmap natural. Without at least one of those, maybe isn't useful. And adding either of those to Python is a huge proposal, much larger than null coalescing, and a lot less likely to gain support. Also, the monadic style of failure propagation directly competes with the exception-raising style, and they're both contagious. A well-designed language and library can have both side by side if it, e.g., rigorously restricts exceptions to only truly exceptional cases, but the boat for that sailed decades ago in Python. So just having them side by side would lead to the exact same problems as C++ code that mixes exception-based and status-code-based APIs, or JavaScript code that mixes exceptions and errbacks or promise.fail handlers. Personally, whenever I think to myself "I could really use maybe here" in some Python code, that's a sign that I'm not thinking Pythonically, and either need to switch gears in my brain or switch languages. Just like when I start thinking about how I could get rid of that with statement with an RAII class, and maybe an implicit conversion operator....
Ryan Gonzalez <rymg19@gmail.com> writes:
http://stackoverflow.com/questions/8507200/maybe-kind-of-monad-in-python
On Sep 18, 2015, at 15:56, Erik <python@lucidity.plus.com> wrote:
I like this (modulo the corrections later in the thread). It's simpler and more flexible than the other options, and also comes closer to resolving the "spam?.eggs" vs. "spam?.cheese()" issue, by requiring "spam?.cheese?()". Obviously "spam?" returns something with a __getattr__ method that just passes through to spam.__getattr__, except that on NoneType it returns something with a __getattr__ that always returns None. That solves the eggs case. Next, "spam?.cheese?" returns something with a __call__ method that just passed through to spam?.cheese.__call__, except that on NoneType it returns something with a __call__ that always returns None. That solves the cheese case. If you make None? return something whose other dunder methods also return None (except for special cases like __repr__), this also gives you "spam ?+ 3". (I'm not sure if that's a good thing or a bad thing...) Of course there's no way to do "spam ?= 3" (but I'm pretty sure that's a good thing). So, do we need a dunder method for the "?" operator? What else would you use it for besides None?
On Sat, Sep 19, 2015 at 10:49 AM, Andrew Barnert <abarnert@yahoo.com> wrote:
Obviously "spam?" returns something with a __getattr__ method that just passes through to spam.__getattr__, except that on NoneType it returns something with a __getattr__ that always returns None. That solves the eggs case.
Next, "spam?.cheese?" returns something with a __call__ method that just passed through to spam?.cheese.__call__, except that on NoneType it returns something with a __call__ that always returns None. That solves the cheese case.
Hang on, how do you do this? How does the operator know the difference between "spam?", which for None has to have __getattr__ return None, and "spam?.cheese?" that returns (lambda: None)? ChrisA
On Sep 18, 2015, at 18:00, Chris Angelico <rosuav@gmail.com> wrote:
All you need to make this work is: * "spam?" returns NoneQuestion if spam is None else spam * NoneQuestion.__getattr__(self, *args, **kw) returns None. * NoneQuestion.__call__(self, *args, **kw) returns None. Optionally, you can add more None-returning methods to NoneQuestion. Also, whether NoneQuestion is a singleton, has an accessible name, etc. are all bikesheddable. I think it's obvious what happens is "spam" is not None and "spam.cheese" is, or of both are None, but if not, I can work them through as well.
On 2015-09-19 02:10, Andrew Barnert via Python-ideas wrote:
I see it as "spam? doing "Maybe(spam)" and then attribute access checking returning None if the wrapped object is None and getting the attribute from it if not. I think that the optimiser could probably avoid the use of Maybe in cases like "spam?.cheese".
On 2015-09-19 02:39, MRAB wrote:
I've thought of another issue: If you write "spam?(sing_lumberjack_song())", won't it still call sing_lumberjack_song even if spam is None? After all, Python evaluates the arguments before looking up the call, so it won't know that "spam" is None until it tries to call "spam?". That isn't a problem with "spam.sing_lumberjack_song() if spam is not None else None" or if it's optimised to that, but "m = spam?; m(sing_lumberjack_song())" is a different matter. perhaps a "Maybe" object should also support "?" so you could write "m = spam?; m?(sing_lumberjack_song())". "Maybe" could be idempotent, so "Maybe(Maybe(x))" returns the same result as "Maybe(x)".
On Sep 18, 2015, at 18:52, MRAB <python@mrabarnett.plus.com> wrote:
You're right; I didn't think about that. But I don't think that's a problem. I believe C#, Swift, etc. all evaluate the arguments in their equivalent. And languages like ObjC that do automatic nil coalescing for all method calls definitely evaluate them. If you really want to switch on spam and not call sing_lumberjack_song, you can always do that manually, right?
That actually makes sense just for its own reasons. Actually, now that I think about it, the way I defined it above already gives you think: if spam? is spam if it's anything but None, then spam?? is always spam?, right?
Andrew, I really like that idea. Turning back to the null coalescing operator (spelled ?? in other languages), how do you think that fits in? Consider this syntax:
None? or 1 1
This works if NoneQuestion overrides __nonzero__ to return False.
0? or 1 0
This doesn't work, because 0? returns 0, and "0 or 1" is 1. We could try this instead, if NoneQuestion overrides __or__:
This looks a little ugly, and it would be nice (as MRAB pointed out) if null coalescing short circuited.
None? or None?
This also doesn't work quite right. If both operands are None, we want the expression to evaluate to None, not NoneQuestion. *Should null coalescing be a separate operator? And if so, are "?" and "??" too similar?* Can anybody think of realistic use cases for overriding a magic method for the "?" operator? I would like to include such use cases in a PEP. One possible use case: being able to coalesce empty strings.
On Fri, Sep 18, 2015 at 9:10 PM, Andrew Barnert via Python-ideas < python-ideas@python.org> wrote:
-- Mark E. Haase 202-815-0201
On Sat, Sep 19, 2015 at 12:06 PM, Mark E. Haase <mehaase@gmail.com> wrote:
Frankly, I think this is a bad idea. You're potentially coalescing multiple things with the same expression, and we already have a way of spelling that: the "or" operator. If you don't want a generic "if it's false, use this", and don't want a super-specific "if it's None, use this", then how are you going to define what it is? And more importantly, how do you reason about the expression "s1? or s2" without knowing exactly what types coalesce to what? Let's keep the rules simple. Make this a special feature of the None singleton, and all other objects simply return themselves - for the same reason that a class isn't allowed to override the "is" operator. ChrisA
On Sep 18, 2015, at 19:06, Mark E. Haase <mehaase@gmail.com> wrote:
I don't think there's any easy way to make "spam? or 1" work any better than "spam or 1" already does, partly for the reasons you give below, but also because it doesn't seem to fit the design in any obvious way. I guess that means postix ? doesn't quite magically solve everything...
This also doesn't work quite right. If both operands are None, we want the expression to evaluate to None, not NoneQuestion. Should null coalescing be a separate operator? And if so, are "?" and "??" too similar?
As MRAB pointed out, there seem to be good reasons to let spam?? mean the same thing as spam? (and that follows automatically from the simplest possible definition, the one I gave above). So I think "spam ?? eggs" is ambiguous between the postfix operator and the infix operator without lookahead, at least to a human, and possibly to the compiler as well. I suppose ?: as in ColdFusion might work, but (a) ewwww, (b) it regularly confuses novices to CF, and (c) it's impossible to search for, because ?: no matter how you quote it gets you the C ternary operator....
This seems like a bad idea. Empty strings are already falsey. If you want this behavior, why not just use "s1 or s2", which already works, and for obvious reasons?
This is likely going to get shot down quickly... I know CoffeeScript is not regarded too well in this community (well, at least based on Guido's remarks on parsing it), but what if x? was shorthand for x is None? In CS, it's called the existential operator. On September 18, 2015 10:30:48 PM CDT, Andrew Barnert via Python-ideas <python-ideas@python.org> wrote:
-- Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity.
On Fri, Sep 18, 2015 at 05:49:36PM -0700, Andrew Barnert via Python-ideas wrote:
Ah, and now my enthusiasm for the whole idea is gone... In my previous response, I imagined spam?.attr to be syntactic sugar for `None if spam is None else spam.attr`. But having ? be an ordinary operator that returns a special Null object feels too "Design Pattern-y" to me. I think the Null design pattern is actually harmful, and I would not like to see this proposal implemented this way. (In another email, Andrew called the special object something like NoneMaybe or NoneQuestion, I forget which. I'm going to call the object Null, since that's less typing.) The Null object pattern sounds like a great idea at first, but I find it to be a code smell at best and outright harmful at worst. If you are passing around an object which is conceptually None, but unlike None reliably does nothing without raising an exception no matter what you do with it, that suggests to me that something about your code is not right. If your functions already accept None, then you should just use None. If they don't accept None, then why are you trying to smuggle None into them using a quasi-None that unconditionally hides errors? Here are some problems with the Null pattern as I see it: (1) Suppose that spam? returns a special Null object, and Null.attr itself returns Null. (As do Null[item] and Null(arg), of course.) This matches the classic Null object design pattern, and gives us chaining for free: value = obj?.spam.eggs.cheese But now `value` is Null, which may not be what we expect and may in fact be a problem if we're expecting it to be "an actual value, or None" rather than our quasi-None Null object. Because `value` is now a Null, every time we pass it to a function, we risk getting new Nulls in places that shouldn't get them. If a function isn't expecting None, we should get an exception, but Null is designed to not raise exceptions no matter what you do with it. So we risk contaminating our data with Nulls in unexpected places. Eventually, of course, there comes a time where we need to deal with the actual value. With the Null pattern in place, we have to deal with two special cases, not one: # I assume Null is a singleton, otherwise use isinstance if filename is not None and filename is not Null: os.unlink(filename) A small nuisance, to be sure, but part of the reason why I really don't think much of the Null object pattern. It sounds good on paper, but I think it's actually more dangerous and inconvenient than the problem it tries to solve. (2) We can avoid the worst of the Null design (anti-)pattern by having Null.attr return None instead of Null. Unfortunately, that means we've lost automatic chaining. If you have an object that might be None, we have to explicitly use the ? operator after each lookup except the last: value = obj?.spam?.eggs?.cheese which is (a) messy, (b) potentially inefficient, and (c) potentially hides subtle bugs. Here is a scenario where it hides bugs. Suppose obj may be None, but if it is not, then obj.spam *must* be a object with an eggs attribute. If obj.spam is None, that's a bug that needs fixing. Suppose we start off by writing the obvious thing: obj?.spam.eggs but that fails because obj=None raises an exception: obj? returns Null Null.spam returns None None.eggs raises So to protect against that, we might write: obj?.spam?.eggs but that protects against too much, and hides the fact that obj.spam exists but is None. As far as I am concerned, any use of a Null object has serious downsides. If people want to explicitly use it in their own code, well, good luck with that. I don't think Python should be making it a built-in. I think the first case, the classic Null design pattern, is actually *better* because the downsides are anything but subtle, and people will soon learn not to touch it with a 10ft pole *wink*, while the second case, the "Null.attr gives None" case, is actually worse because it isn't *obviously* wrong and can subtly hide bugs. How does my earlier idea of ? as syntactic sugar compare with those? In that case, there is no special Null object, there's only None. So we avoid the risk of Null infection, and avoid needing to check specially for Null. It also avoids the bug-hiding scenario: obj?.spam.eggs.cheese is equivalent to: None if obj is None else obj.spam.eggs If obj is None, we get None, as we expect. If it is not None, we get obj.spam.eggs as we expect. If obj.spam is wrongly None, then we get an exception, as we should. -- Steve
Andrew Barnert via Python-ideas writes:
So, do we need a dunder method for the "?" operator? What else would you use it for besides None?
NaNs in a pure-Python implementation of float or Decimal. (This is not a practical suggestion.) A true SQL NULL type. It's always bothered me that most ORMs map NULL to None but there are plenty of other ways to inject None into a Python computation. (This probably isn't a practical suggestion either unless Random832's suggestion of ?() establishing a lexical context were adopted.) The point is that Maybe behavior is at least theoretically useful in subcategories, with special objects other than None. Sven's suggestion of calling this the "monad" operator triggers a worry in me, however. In Haskell, the Monad type doesn't enforce the monad laws, only the property of being an endofunctor. That apparently turns out to be enough in practice to make the Monad type very useful. However, in Python we have no way to enforce that property. I don't have the imagination to come up with a truly attractive nuisance here, and this operator doesn't enable general functorial behavior, so maybe it's not a problem.
On 19.09.2015 07:14, Stephen J. Turnbull wrote:
I definitely agree here. Internally, we have a guideline telling us to avoid None or NULL whenever possible. Andrew's remark about 'code smell' is definitely appropriate. There was a great discussion some years ago on one of the RDF semantics mailing list about the semantics of NULL (in RDF). It turned out to have 6 or 7 semantics WITHOUT any domain-specific focus (don't know, don't exists, is missing, etc. -- can't remember all of them). I feel that is one reason why Python programs should avoid None: we don't guess.
Sleeping one night over it, I now tend to change my mind regarding this. Maybe, it's *better to DEAL with None as in remove* *them* from the code, from the database, from the YAML files and so forth *instead**of *making it easier to work with them. Restricting oneself, would eventually lead to more predictable designs. Does this makes sense somehow? Issue is, None is so convenient to work with. You only find out the code smell when you discover a "NoneType object does not have attribute X" exception some months later and start looking where the heck the None could come from. What can we do here?
Sven R. Kunze writes:
That's exactly what should happen (analogous to a "signalling NaN"). The problem is if you are using None as a proxy for a NULL in another subsystem that has "NULL contagion" (I prefer that to "coalescing"). At this point the thread ends for me because I'm not going try to tell the many libraries that have chosen to translate NULL to None and vice versa that they are wrong.
"Uptalk" is an interesting speech pattern where every sentence sounds like a question. Google it, there's some interesting research. The "null pattern" is terrible. Uptalk should not be considered a unary operator that returns a magical value. It's a modifier on other operators (somewhat similar to the way "+=" and friends are formed). In case someone missed it, uptalk should test for None, not for a falsey value. I forgot to think about the scope of the uptalk operator (i.e. what is skipped when it finds a None). There are some clear cases (the actual implementation should avoid double evaluation of the tested expression, of course): a.b?.c.d[x, y](p, q) === None if a.b is None else a.b.c.d[x, y](p, q) a.b?[x, y].c.d(p, q) === None if a.b is None else a.b[x, y].c.d(p, q) a.b?(p, q).c.d[x, y] === None if a.b is None else a.b(p, q).c.d[x, y] But what about its effect on other operators in the same expression? I think this is reasonable: a?.b + c.d === None if a is None else a.b + c.d OTOH I don't think it should affect shortcut boolean operators (and, or): a?.b or x === (None if a is None else a.b) or x It also shouldn't escape out of comma-separated lists, argument lists, etc.: (a?.b, x) === ((None if a is None else a.b), x) f(a?.b) === f((None if a is None else a.b)) Should it escape from plain parentheses? Which of these is better? (a?.b) + c === (None if a is None else a.b) + c # Fails unless c overloads None+c (a?.b) + c === None if a is None else (a.b) + c # Could be surprising if ? is deeply nested Here are some more edge cases / hypergeneralizations: {k1?: v1, k2: v2} === {k2: v2} if k1 is None else {k1: v1, k2: v2} # ?: skips if key is None # But what to do to skip None values? Could we give ?= a meaning in assignment, e.g. x ?= y could mean: if y is not None: x = y More fun: x ?+= y could mean: if x is None: x = y elif y is not None: y += y You see where this is going. Downhill fast. :-) -- --Guido van Rossum (python.org/~guido)
On 2015-09-19 17:21, Guido van Rossum wrote:
It shouldn't escape beyond anything having a lower precedence.
Shouldn't that be: if x is not None: x = y ? It's the value before the '?' that's tested.
Or: if x is None: pass else: x += y
You see where this is going. Downhill fast. :-)
Could it be used postfix: a +? b === None if b is None else a + b -?a === None if a is None else -a or both prefix and postfix: a ?+? b === None if a is None or b is None else a + b ?
On Sat, Sep 19, 2015 at 9:21 AM, Guido van Rossum <guido@python.org> wrote:
This is a bit weird to me. Essentially ?. takes precedence over a following +. But would you also expect it to take precedence over a preceding one as well? That's inconsistent. c.d + a?.b === None if a is None else c.d + a.b or c.d + a?.b === c.d + None if a is None else c.d + a.b I think that ?. ?[] and ?() should affect other operators at the same precedence level only, i.e., each other and . [] and (). This seems the most logical to me. And I just looked up the C# documentation on MSDN and it does the same thing: https://msdn.microsoft.com/en-us/library/dn986595.aspx
Agree. It also should not escape grouping parenthesis even though that might not be useful. It would be very weird if a parenthesized expression did something other than evaluate the expression inside it, period. (a?.b).c === None.c if a is None else (a.b).c === temp = a?.b; temp.c (x or a?.b).c === (x or (None if a is none else a.b)).c Yes, None.c is going to raise an exception. That's better than just getting None IMHO. --- Bruce Check out my new puzzle book: http://J.mp/ingToConclusions Get it free here: http://J.mp/ingToConclusionsFree (available on iOS)
On 19.09.2015 14:48, Stephen J. Turnbull wrote:
Not my point, Stephen. My point is, you better avoid None (despite its convenience) because you are going to have a hard time finding its origin later in the control flow. Question still stands: is None really necessary to justify the introduction of convenience operators like "?." etc.?
The problem is if you are using None as a proxy for a NULL in another subsystem that has "NULL contagion" (I prefer that to "coalescing").
How would you solve instead? Best, Sven
I think the core issue is that, whether or not it should be used, APIs already return None values, so a convenience operator might as well be added. On September 19, 2015 2:09:48 PM CDT, "Sven R. Kunze" <srkunze@mail.de> wrote:
-- Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity.
2015-09-19 21:24 GMT+02:00 Ryan Gonzalez <rymg19@gmail.com>:
I'm curious on which API returning None, a major bonus on using python is that I pretty never stumbled upon the equivalent of NullPointerException. Moreover, I wonder if that this convenience operator will do something more than hide bugs.
On Sep 19, 2015, at 14:47, Guido van Rossum <guido@python.org> wrote:
The TypeErrors usually come from novices. There are many of StackOverflow questions asking why they can't add spam.get_text() + "\n" where they don't show you the implementation of get_text, or the exception they got, but you just know they forgot a return statement at the end and the exception was a TypeError about adding NoneType and str.
On Sat, Sep 19, 2015 at 12:02:42AM +0100, MRAB wrote:
No. If I understand the idea, obj?.attr returns None if obj is None, otherwise returns obj.attr. The question mark (shrug operator?) applies to `obj` *before* the attribute lookup, so it should appear *before* the dot (since we read from left-to-right). The heuristic for remembering the order is that the "shrug" (question mark) operator applies to obj, so it is attached to obj, before any subsequent operation. For the sake of brevity, using @ as a placeholder for one of attribute access, item/key lookup, or function call, then we have: obj?@ as syntactic sugar for: None if obj is None else obj@ Furthermore, we should be able to chain a sequence of such @s: paperboy.receive(customer?.trousers.backpocket.wallet.extract(2.99)) being equivalent to: paperboy.receive(None if customer is None else customer.trousers.backpocket.wallet.extract(2.99) ) Let's just assume we have a good reason for chaining lookups that isn't an egregious violation of the Law of Demeter, and not get into a debate over OOP best practices, okay? :-) Suppose that wallet itself may also be None. Then we can easily deal with that situation too: paperboy.receive(customer?.trousers.backpocket.wallet?.extract(2.99)) which I think is a big win over either of these two alternatives: # 1 paperboy.receive(None if customer is None else None if customer.trousers.backpocket.wallet is None else customer.trousers.backpocket.wallet.extract(2.99) ) # 2 if customer is not None: wallet = customer.trousers.backpocket.wallet if wallet is not None: paperboy.receive(wallet.extract(2.99)) It's a funny thing, I'm usually not a huge fan of symbols outside of maths operators, and I strongly dislike the C ? ternary operator, but this one feels really natural to me. I didn't have even the most momentary "if you want Perl, you know where to find it" thought. -- Steve
On Fri, Sep 18, 2015 at 8:41 PM, Steven D'Aprano <steve@pearwood.info> wrote:
I do, but at least the '?' is part of an operator, not part of the name (as it is in Ruby?). I really, really, really don't like how it looks, but here's one thing: the discussion can be cut short and focus almost entirely on whether this is worth making Python uglier (and whether it's even ugly :-). The semantics are crystal clear and it's obvious that the way it should work is by making "?.", ?(" and "?[" new operators or operator pairs -- the "?" should not be a unary postfix operator but a symbol that combines with certain other symbols. Let me propose a (hyper?)generalization: it could be combined with any binary operation, e.g. "a?+b" would mean "None if a is None else a+b". Sadly (as hypergeneralizations tend to do?) this also leads to a negative observation: what if I wanted to write "None if b is None else a+b"? (And don't be funny and say I should swap a and b -- they could be strings.) Similar for what if you wanted to do this with a unary operator, e.g. None if x is None else -x. Maybe we could write "a+?b" and "-?x"? But I don't think the use cases warrant these much. Finally, let's give it a proper name -- let's call it the uptalk operator. -- --Guido van Rossum (python.org/~guido)
Guido van Rossum <guido@python.org> writes:
Let me propose a (hyper?)generalization: it could be combined with any binary operation, e.g. "a?+b" would mean "None if a is None else a+b".
I'd have read it as "None if a is None or b is None else a+b". If you want to only do it for one of the operands you should be explicit. I'm not sure if I have a coherent argument for why this shouldn't apply to ?[, though.
On Fri, Sep 18, 2015 at 11:21 PM, Guido van Rossum <guido@python.org> wrote:
I really liked this whole thread, and I largely still do -- I?think -- but I'm not sure I like how `?` suddenly prevents whole blocks of code from being evaluated. Anything within the (...) or [...] is now skipped (IIUC) just because a `?` was added, which seems like it could have side effects on the surrounding state, especially since I expect people will use it for squashing/silencing or as a convenient trick after the fact, possibly in code they did not originally write. If the original example included a `?` like so: response = json.dumps?({ 'created': created?.isoformat(), 'updated': updated?.isoformat(), ... }) should "dumps" be None, the additional `?` (although though you can barely see it) prevents *everything else* from executing. This may cause confusion about what is being executed, and when, especially once nesting (to any degree really) and/or chaining comes into play! Usually when I want to use this pattern, I find I just need to write things out more. The concept itself vaguely reminds me of PHP's use of `@` for squashing errors. In my opinion, it has some utility but has too much potential impact on program flow without being very noticeable. If I saw more than 1 per line, or a couple within a few lines, I think my ability to quickly identify -> analyze -> comprehend possible routes in program control flow decreases. I feel like I'll fault more, double back, and/or make sure I forevermore look harder for sneaky `?`s. I probably need to research more examples of how such a thing is used in real code, today. This will help me get a feel for how people might want to integrate the new `?` capability into their libraries and apis, maybe that will ease my readability reservations. Thanks, -- C Anthony
On Sat, Sep 19, 2015 at 03:17:07AM -0500, C Anthony Risinger wrote:
I don't think this is any different from other short-circuiting operators, particularly `and` and the ternary `if` operator: result = obj and obj.method(expression) result = obj.method(expression) if obj else default In both cases, `expression` is not evaluated if obj is falsey. That's the whole point.
We're still discussing the syntax and semantics of this, so I could be wrong, but my understanding of this is that the *first* question mark prevents the expressions in the parens from being executed: json.dumps?( ... ) evaluates as None if json.dumps is None, otherwise it evaluates the arguments and calls the dumps object. In other words, rather like this: _temp = json.dumps # temporary value if _temp is None: response = None else: response = _temp({ 'created': None if created is None else created.isoformat(), 'updated': None if updated is None else updated.isoformat(), ... }) del _temp except the _temp name isn't actually used. The whole point is to avoid evaluating an expression (attribute looking, index/key lookup, function call) which will fail if the object is None, and if you're not going to call the function, why evaluate the arguments to the function?
Well, yes, people can abuse most any syntax.
I had to look up PHP's @ and I must say I'm rather horrified. According to the docs, all it does is suppress the error reporting, it does nothing to prevent or recover from errors. There's not really an equivalent in Python, but I suppose this is the closest: # similar to PHP's $result = @(expression); try: result = expression except: result = None This is nothing like this proposal. It doesn't suppress arbitrary errors. It's more like a conditional: # result = obj?(expression) if obj is None: result = None else: result = obj(expression) If `expression` raises an exception, it will still be raised, but only if it is actually evaluated, just like anything else protected by an if...else or short-circuit operator. -- Steve
On Sat, Sep 19, 2015 at 7:06 AM, Steven D'Aprano <steve@pearwood.info> wrote:
Sure, but those all have white space and I can read what's happening. The `?` could appear anywhere without break. I don't like that, but, opinion.
Yes that is how I understand it as well. I'm suggesting it's hard to see. I understand the concept as "None cancellation", because if the left is None, the right is cancelled. This lead me here: * This is great, want to use all the time! * First-level language support, shouldn't I use? Does feels useful/natural * How can I make my APIs cancellation-friendly? * I can write None-centric APIs, that often collapse to None * Now maybe user code does stuff like `patient.visits?.september?.records` to get all records in September (if any, else None) * Since both `?` points would *prefer* None, if the result is None, I now have to jump around looking for who done it * If I don't have debugger ATM, I'm breaking it up a lot for good 'ol print(...), only way * I don't think I like this any more :( I especially don't like the idea of seeing it multiple times quickly, and the potential impact to debugging. The truth is I want to like this but I feel like it opens a can of worms (as seen by all the wild operators this proposal "naturally" suggests).
I did say vaguely :) but it is extremely hideous I agree. The part that made me think of this is the would be desire for things to become None (so, or example, wanting to avoid throwing typed/informative exceptions if possible) so they'd then be more useful with `?`. -- C Anthony
On Sun, Sep 20, 2015 at 09:10:32AM +0300, Serhiy Storchaka wrote:
Absolutely nothing. I'm not convinced that we should generalise this beyond the three original examples of attribute access, item lookup and function call. I think that applying ? to arbitrary operators is a case of "YAGNI". Or perhaps, "You Shouldn't Need It". Mark's original motivating use-case strikes me as both common and unexceptional. We might write: # spam may be None, or some object result = spam or spam.attr # better, as it doesn't misbehave when spam is falsey result = None if spam is None else spam.attr and it seems reasonable to me to want a short-cut for that use-case. But the generalisations to arbitrary operators suggested by Guido strike me as too much, too far. As he says, going downhill, and quickly. Consider these two hypotheticals: spam ?+ eggs # None if spam is None or eggs is None else spam + eggs needle ?in haystack # None if needle is None or haystack is None else needle in haystack Briefer (more concise) is not necessarily better. At the point you have *two* objects in the one term that both need to be checked for None, that is in my opinion a code smell and we shouldn't provide a short-cut disguising that. Technically, x.y x[y] and x(y) aren't operators, but for the sake of convenience I'll call them such. Even though these are binary operators, the ? only shortcuts according to the x, not the y. So we can call these ?. ?[] ?() operators "pseudo-unary" operators rather than binary operators. Are there any actual unary operators we might want to apply this uptalk/shrug operator to? There are (if I remember correctly) only three unary operators: + - and ~. I don't think there are any reasonable use-cases for writing (say): value = ?-x that justifies making this short-cut available. So as far as I am concerned, the following conditions should apply: - the uptalk/shrug ? "operator" should not apply to actual binary operators where both operands need to be checked for None-ness (e.g. arithmetic operators, comparison operators) - it should not apply to arithmetic unary operators + - and ~ - it might apply to pseudo-operators where only the lefthand argument is checked for None-ness, that is, x.y x[y] and x(y), written as x?.y x?[y] and x?(y). If I had to choose between generalising this to all operators, or not having it at all, I'd rather not have it at all. A little bit of uptalk goes a long way, once we have ? appearing all over the place in all sorts of expressions, I think it's too much. -- Steve
On Sep 20, 2015, at 00:31, Steven D'Aprano <steve@pearwood.info> wrote:
I agree. Seeing how far you can generalize something and whether you can come up with a simple rule that makes all of your use cases follow naturally can be fun, but it isn't necessarily the best design. Also, by not trying to generalize uptalk-combined operators (or uptalk as a postfix unary operator of its own, which I was earlier arguing for...), the question of how we deal with ?? or ?= (if we want them) can be "the same way every other language does", rather than seeing what follows from the general rule and then convincing ourselves that's what we wanted. Also, I think trying to generalize to all operators is a false generalization, since the things we're generalizing from aren't actually operators (and not just syntactically--e.g., stylistically, they're never surrounded by spaces--which makes a pretty big difference in the readability impact of a character as heavy as "?") in the first place. Personally, I think ?? is the second most obviously useful after ?. (there's a reason it's the one with the oldest and widest pedigree); we need ?() because Python, unlike C# and friends, unifies member and method access; ?[] doesn't seem as necessary but it's such an obvious parallel to ?() that I think people will expect it; ?= is potentially as confusing as it is helpful. So, my suggestion would be just the first four. And keeping them simple, and consistent with other languages, no trying to extend the protection to other operators/accesses, no extra short-circuiting, nothing. So: spam ?? eggs === spam if spam is not None else eggs spam?.eggs === spam.eggs if spam is not None else None spam?(eggs) === spam(eggs) if spam is not None else None spam?[eggs] === spam[eggs] if spam is not None else None That's easy to define, easy to learn and remember, and pretty consistent with other languages. The one big difference is that what you write as "spam?.eggs(cheese)" in C# has to be "spam?.eggs?(cheese)" in Python, but I don't think that's a big problem. After all, in Python, spam.eggs is a first-class object, and one that's commonly passed or stored, so the obvious way to look at "spam.eggs(cheese)" is as explicitly chaining two separate things together (a __getattr__ with a descriptor __get__, and a __call__), so why shouldn't uptalking both operations be explicit?
On Sun, Sep 20, 2015 at 5:31 PM, Steven D'Aprano <steve@pearwood.info> wrote:
That's how all Python's short-circuiting works - based on the value of what's on the left, decide whether or not to evaluate what's on the right. (Well, nearly all - if/else evaluates the middle first, but same difference.) This is another form of short-circuiting; "x[y]" evaluates x, then if that's None, doesn't bother evaluating y because it can't affect the result. ChrisA
On Sun, Sep 20, 2015 at 07:38:18PM +1000, Chris Angelico wrote:
I think you are mistaken about x[y]: py> None[print("side effect")] side effect Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'NoneType' object is not subscriptable That's why x?[y] is a proposal. -- Steve
On Mon, Sep 21, 2015 at 1:50 PM, Steven D'Aprano <steve@pearwood.info> wrote:
Oops, that was a typo in my statement. I meant "x?[y]" should behave that way - once it's discovered that x is None, the evaluation of y can't affect the result, and so it doesn't get evaluated (as per the normal short-circuiting rules). Yes, x[y] has to evaluate both x and y (after all, the value of y is passed to __getitem__). Sorry for the confusion. ChrisA
On 20 September 2015 at 08:31, Steven D'Aprano <steve@pearwood.info> wrote:
Agreed. Does this need to be an operator? How about the following: class Maybe: def __getattr__(self, attr): return None def __getitem__(self, idx): return None def __call__(self, *args, **kw): return None def maybe(obj): return Maybe() if obj is None else obj attr = maybe(obj).spam elt = maybe(obj)[n] result = maybe(callback)(args) The Maybe class could be hidden, and the Maybe() object a singleton (making my poor naming a non-issue :-)) and if it's felt sufficiently useful, the maybe() function could be a builtin. Usage of the result of maybe() outside of the above 3 contexts should simply be "not supported" - don't worry about trying to stop people doing weird things, just make it clear that the intent is only to support the 3 given idiomatic usages. Paul.
On Sun, Sep 20, 2015 at 12:05:52PM +0100, Paul Moore <p.f.moore@gmail.com> wrote:
PyMaybe - a Python implementation of the Maybe pattern. Seems to be quite elaborated. https://github.com/ekampf/pymaybe
Paul.
Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
Paul Moore's idea is WAAYY better than the ugly ? pseudo-operator. `maybe()` reads just like a regular function (because it is), and we don't need to go looking for Perl (nor Haskell) in some weird extra syntax that will confuse beginners. On Sun, Sep 20, 2015 at 4:05 AM, Paul Moore <p.f.moore@gmail.com> wrote:
-- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On the day I started this thread, I wrote a Python module that does what maybe() does. I hadn't seen PyMaybe yet, and I couldn't think of any good names for my module's functions, so my module was disappointingly ugly. PyMaybe is exactly what I *wish* I had written that day. For comparison, here's the code from my first post in this thread and it's maybe-ized version. response = json.dumps({ 'created': created?.isoformat(), 'updated': updated?.isoformat(), ... }) response = json.dumps({ 'created': maybe(created).isoformat(), 'updated': maybe(updated).isoformat(), ... }) Pros: 1. No modification to Python grammar. 2. More readable: it's easy to overlook ? when skimming quickly, but "maybe()" is easy to spot. 3. More intuitive: the name "maybe" gives a hint at what it might do, whereas if you've never seen "?." you would need to google it. (Googling punctuation is obnoxious.) Cons: 1. Doesn't short circuit: "maybe(family_name).upper().strip()" will fail if family_name is None.[1] You might try "maybe(maybe(family_name).upper()).strip()", but that is tricky to read and still isn't quite right: if family_name is not None, then it *should* be an error if "upper" is not an attribute of it. The 2-maybes form covers up that error. I'm sure there will be differing opinions on whether this type of operation should short circuit. Some will say that we shouldn't be writing code that way: if you need to chain calls, then use some other syntax. But I think the example of upper case & strip is a good example of a perfectly reasonable thing to do. These kinds of operations are pretty common when you're interfacing with some external system or external data that has a concept of null (databases, JSON, YAML, argparse, any thin wrapper around C library, etc.). This conversation has really focused on the null aware attribute access, but the easier and more defensible use case is the null coalesce operator, spelled "??" in C# and Dart. It's easy to find popular packages that use something like "retries = default if default is not None else cls.DEFAULT" to supply default instances.[2] Other packages do something like "retries = default or cls.DEFAULT"[3], which is worse because it easy to overlook the implicit coalescing of the left operand. In fact, the top hit for "python null coalesce" is StackOverflow, and the top-voted answer says to use "or".[4] (The answer goes on to explain the nuance of using "or" to coalesce, but how many developers read that far?) *In the interest of finding some common ground, I'd like to get some feedback on the coalesce operator.* Maybe that conversation will yield some insight into the other "None aware" operators. A) Is coalesce a useful feature? (And what are the use cases?) B) If it is useful, is it important that it short circuits? (Put another way, could a function suffice?) C) If it should be an operator, is "??" an ugly spelling? >>> retries = default ?? cls.DEFAULT D) If it should be an operator, are any keywords more aesthetically pleasing? (I expect zero support for adding a new keyword.) >>> retries = default else cls.DEFAULT >>> retries = try default or cls.DEFAULT >>> retries = try default else cls.DEFAULT >>> retries = try default, cls.DEFAULT >>> retries = from default or cls.DEFAULT >>> retries = from default else cls.DEFAULT >>> retries = from default, cls.DEFAULT My answers: A) It's useful: supplying default instances for optional values is an obvious and common use case. B) It should short circuit, because the patterns it replaces (using ternary operator or "or") also do. C) It's too restrictive to cobble a new operator out of existing keywords; "??" isn't hard to read when it is separated by whitespace, as Pythonistas typically do between a binary operator and its operands. D) I don't find any of these easier to read or write than "??". [1] I say "should", but actually PyMaybe does something underhanded so that this expression does not fail: "maybe(foo).upper()" returns a "Nothing" instance, not "None". But Nothing has "def __repr__(self): return repr(None)". So if you try to print it out, you'll think you have a None instance, but it won't behave like one. If you try to JSON serialize it, you get a hideously confusing error: "TypeError: None is not JSON serializable". For those not familiar: the JSON encoder can definitely serialize None: it becomes a JSON "null". A standard implementation of maybe() should _not_ work this way. [2] https://github.com/shazow/urllib3/blob/master/urllib3/util/retry.py#L148 [3] https://github.com/kennethreitz/requests/blob/46ff1a9a543cc4d33541aa64c94f50... [4] http://stackoverflow.com/a/4978745/122763 On Sun, Sep 20, 2015 at 6:50 PM, Guido van Rossum <guido@python.org> wrote:
-- Mark E. Haase 202-815-0201
On 21.09.2015 04:35, Mark E. Haase wrote:
A) Is coalesce a useful feature? (And what are the use cases?)
I limit myself to materializing default arguments as in: def a(b=None): b = b or {} ... Because its a well known theme (and issue) of the mutability of default arguments of Python.
The only difference between "or" and "??" is that "??" is None only, right? At least to me, the given use case above does not justify the introduction of "??".
Yes, "or" suffices in that case.
B) It should short circuit, because the patterns it replaces (using ternary operator or "or") also do.
They look ugly and unpleasant because they remind you to reduce the usage of None; not to make dealing with it more pleasant.
"or" is easier to type (no special characters), I don't need to explain it to new staff, and it's more pleasant to the eye. I remember my missis telling me, after I showed her some C# code, that programmers tend to like weird special characters. Well, that might certainly be true. Special characters increase the visual noise and the mental strain when reading. They make the lines they are in special. I don't see anything special with "or" and with the single use case I have for it. :) Best, Sven
On 09/20/2015 11:57 PM, Chris Angelico wrote:
From reading these, I think the lowest-level/purest change would be to accommodate testing for "not None". Something I've always thought Python should be able to do in a nicer more direct way. We could add a "not None" specific boolean operators just by appending ! to them. while! x: <--> while x != None: if! x: <--> if x != None: a or! b <--> b if a != None else a a and! b <--> a if a != None else b not! x <--> x if x != None else None Those expressions on the right are very common and are needed because of None, False, and 0, are all False values. It would make for much simpler expressions and statements where they are used and be more efficient as these are likely to be in loops going over *many* objects. So it may also result in a fairly nice speed improvement for many routines. While the consistency argument says "if!" should be equivalent to "if not", I feel the practicality argument leans towards it being specific to "if obj != None". I believe testing for "not None" is a lot more common than testing for "None". Usually the only difference is how the code is arranged. I like how it simplifies/clarifies the common cases above. It would be especially nice in comprehensions. Cheers, Ron
Just to cut this thread short, I'm going to reject PEP 505, because ? is just too ugly to add to Python IMO. Sorry. I commend Mark for his clean write-up, without being distracted, giving some good use cases. I also like that he focused on a minimal addition to the language and didn't get distracted by hyper-generalizations. I also like that he left out f?(...) -- the use case is much weaker; usually it's the object whose method you're calling that might be None, as in title?.upper(). Some nits for the PEP: - I don't think it ever gives the priority for the ?? operator. What would "a ?? b or c" mean? - You don't explain why it's x ?? y but x ?= y. I would have expected either x ? y or x ??= y. - You don't explain or show how far ?. reaches; I assume x?y.z is equivalent to None if x is None else x.y.z, so you don't have to write x?.y?.z just to handle x.y.z if x is None. - The specification section is empty. -- --Guido van Rossum (python.org/~guido)
PEP-505 isn't anywhere close to being finished. I only submitted the draft because somebody off list asked me to send a draft so I could get a PEP number assigned. So I literally sent him what I had open in my text editor, which was just a few minutes of brain dumping and had several mistakes (grammatical and technical). If there's absolutely no point in continuing to work on it, I'll drop it. But from the outset, I thought the plan was to present this in its best light (and similar to the ternary operator PEP, offer several alternatives) if for no other reason than to have a good record of the reasoning for rejecting it. I'm sorry if I misunderstood the PEP process; I would have kept it to myself longer if I knew the first submission was going to be reviewed critically. I thought this e-mail chain was more of an open discussion on the general idea, not specifically a referendum on the PEP itself. On Mon, Sep 21, 2015 at 11:40 AM, Guido van Rossum <guido@python.org> wrote:
-- Mark E. Haase 202-815-0201
On Mon, Sep 21, 2015 at 8:58 AM, Mark E. Haase <mehaase@gmail.com> wrote:
I apologize for having misunderstood the status of your PEP. I think it would be great if you finished the PEP. As you know the ? operator has its share of fans as well as detractors, and I will happily wait until more of a consensus appears. I hope you can also add a discussion to the PEP of ideas (like some of the hyper-generalizations) that were considered and rejected -- summarizing a discussion is often a very important goal of a PEP. I think you have made a great start already! --Guido
-- --Guido van Rossum (python.org/~guido)
On 9/21/2015 1:07 PM, Guido van Rossum wrote:
Add me to the detractors of what I have read so far ;-). In arithmetic, 1/0 and 0/0 both stop the calculation. My hand calculator literally freezes until I hit 'on' or 'all clear'. Early computers also stopped, maybe with an instruction address and core dump. Three orthogonal solutions are: test y before x/y, so one can do something else; introduce catchable exceptions, so one can do something else; introduce contagious special objects ('inf' and 'nan'), which at some point can be tested for, so one can do something else. Python introduced 'inf' and 'nan' but did not use them to replace ZeroDivisionError. Some languages lacking exceptions introduce a contagious null object. Call it Bottom. Any operation on Bottom yields Bottom. Python is not such a language. None is anti-contagious; most operations raise an exception. I agree with Paul Moore that propagating None is generally a bad idea. It merely avoids the inevitable exception. Or is it inevitable? Trying to avoid exceptions naturally leads to the hypergeneralization of allowing '?' everywhere. Instead of trying to turn None into Bottom, I think a better solution would be a new, contagious, singleton Bottom object with every possible special method, all returning Bottom. Anyone could write such for their one use. Someone could put it on pypi to see if there how useful it would be. I agree with Ron Adam that the narrow issue is that bool(x) is False is sometimes too broad and people dislike of spelling out 'x is not None'. So abbreviate that with a unary operator; 'is not None', is a property of objects, not operators. I think 'x!' or 'x?', either meaning 'x is not None', might be better than a new binary operator. The former, x!, re-uses ! in something close to its normal meaning: x really exists. -- Terry Jan Reedy
On Mon, Sep 21, 2015 at 2:23 PM, Terry Reedy <tjreedy@udel.edu> wrote:
I think this is the PyMaybe solution. What I don't like about it is that it is dynamic -- when used incorrectly (or even correctly?) Bottom could end up being passed into code that doesn't expect it. That's bad -- "if x is None" returns False when x is Bottom, so code that isn't prepared for Bottom may well misbehave. In contrast, PEP 505 only affects code that is lexically near the ? operator. (You may see a trend here. PEP 498 is also carefully designed to be locally-scoped.)
I don't think the big issue is bool(x) being too broad. That's what the binary ?? operator is trying to fix, but to me the more useful operators are x?.y and x?[y], both of which would still require repetition of the part on the left when spelled using ??. This is important when x is a more complex expression that is either expensive or has a side-effect. E.g. d.get(key)?.upper() would currently have to be spelled as (some variant of) "None if d.get(key) is None else d.get(key).upper()" and the ?? operator doesn't really help for the repetition -- it would still be "d.get(key) ?? d.get(key).upper()". In general to avoid this repetition you have to introduce a local variable, but that's often awkward and interrupts the programmer's "flow". The ? solves that nicely. The key issue with this proposal to me is how it affects readability of code that uses it, given that there isn't much uniformity across languages in what ? means -- it could be part of a method name indicating a Boolean return value (Ruby) or a conditional operator (C and most of its descendents) or some kind of shortcut. So this is the issue I have to deal with (and thought I had dealt with by prematurely rejecting the PEP, but I've had a change of heart and am now waiting for the PEP to be finished). -- --Guido van Rossum (python.org/~guido)
On 9/21/2015 5:48 PM, Guido van Rossum wrote:
I agree with Paul Moore that propagating None is generally a bad idea. It merely avoids the inevitable exception.
To me, this is the key idea in opposition to proposals that make propagating None easier.
try: x = d.get(key).upper() except AttributeError: x = None is also a no-repeat equivalent when d.values are all strings. I agree than "x = d.get(key)?.upper()" is a plausible abbreviation. But I am much more likely to want "x = ''" or another exception as the alternative. I guess some other pythonistas like keeping None around more than I do ;-). -- Terry Jan Reedy
On Mon, Sep 21, 2015 at 3:45 PM, Terry Reedy <tjreedy@udel.edu> wrote:
(I didn't write that, you [Terry] did. It looks like our mailers don't understand each other's quoting conventions. :-( )
Eew. That try/except is not only very distracting and interrupts the flow of both the writer and the reader, it may also catch errors, e.g. what if the method being called raises an exception (not a problem with upper(), but definitely with user-defined methods). -- --Guido van Rossum (python.org/~guido)
On 09/21/2015 04:45 PM, Terry Reedy wrote:
I think it's one of those things that depends on what you're doing. From a web-development perspective, you rarely keep _anything_ around for very long, so there's rarely an issue of `None` sneaking in somewhere unexpectedly and then causing a surprise exception way down the line. Typical use cases are things like: "If this database query returns a User, I want to get their name and return that in the JSON dict from my API, otherwise I want None, which will be serialized to a JSON null, clearly indicating that there is no user here." My jaw dropped a bit when I saw it asserted in this thread that functions returning "useful value or None" is an anti-pattern. I write functions like that all the time, and I consider it a useful and necessary Python idiom. I would hate to rewrite all that code to either deal with exceptions or add default-value-argument boilerplate to all of them; when "no result" is an expected and normal possibility from a function, letting the calling code deal with None however it chooses is much nicer than either of those options. I don't love the ? syntax, but I would certainly use the feature discussed here happily and frequently. Carl
On Mon, Sep 21, 2015 at 3:56 PM, Carl Meyer <carl@oddbird.net> wrote:
+1 Some language features are "prescriptive," designed to encourage particular ways of writing things. Others are "respective," recognizing the variety of ways people write things and respecting that variety. Python has None and generally respects use of it. To say that using None is an anti-pattern is something I would strongly disagree with. Yes, NPE errors are a problem, but eliminating null/None does not eliminate those errors. It merely replaces one common error with an assortment of other errors. I like the feature. I have been asking for features like this for years and the number of times I have written the longer forms is too many to count. I like the ?. ?[] ?() ?? syntax. I think: (1) it's strongly related to the . [] () syntax; (2) any syntax that uses a keyword is either not syntactically related to . [] () or mixes a keyword and punctuation, both of which I dislike; (3) it's the same syntax as used in other languages (yes, Python is not C# or Dart but there's a good reason Python uses ^ for xor, ** for power, += for add to, etc.) --- Bruce
On 2015-09-21 15:56, Carl Meyer wrote:
I agree that it's a fine thing. The thing is, it's an API choice. If your API is "return such-and-such or None", then anyone who calls your function knows they have to check for None and do the right thing. I think this is fine if None really does indicate something like "no result". (The re module uses None return values this way.) It seems to me that a lot of the "problem" that these null-coalescing proposals are trying to solve is dealing with APIs that return None when they really ought to be raising an exception or returning some kind of context-appropriate empty value. If you're doing result = someFunction() and then result.attr.upper() and it's failing because result.attr is None, to me that's often a sign that the API is fragile, and the result object that someFunction returns should have its attr set to an empty string, not None. In other words, if you really want "a null result that I can call all kinds of string methods on and treat it like a string", you should be returning an empty string. If you want "a null result I can subscript and get an integer", you should be returning some kind of defaultdict-like object that has a default zero value. Or whatever. There isn't really such a thing as "an object to which I want to be able to do absolutely anything and have it work", because there's no type-general notion of what "work" means. From a duck-typing perspective, if you expect users to try to do anything with a value you return, what they might reasonably want to do should be a clue as to what kind of value you should return. That still leaves the use-case where you're trying to interoperate with some external system that may have missing values, but I don't see that as super compelling. Getting an exception when you do some['big']['json']['object']['value'] and one of the intermediate ones isn't there is a feature; the bug is the JavaScripty mentality of just silently passing around "undefined". To my mind, Python APIs that wrap such external data sources should ideally take the opportunity to improve on them and make them more Pythonic, by providing sensible, context-relevant defaults instead of propagating a generic "null" value willy-nilly. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown
On Mon, Sep 21, 2015 at 4:47 PM, Random832 <random832@fastmail.com> wrote:
Sorry, my bad. Indeed, x ?? y tries to fix the issue that "x or y" uses y if x is falsey. Still this seems a lesser problem to me than the problem solved by x?.a and x?[y]. Most of the time in my code it is actually fine to use the default if the LHS is an empty string. -- --Guido van Rossum (python.org/~guido)
On Mon, Sep 21, 2015 at 2:48 PM, Guido van Rossum <guido@python.org> wrote:
As we are in the process of writing a PEP and uses of ?/?? in other languages, why not speak about thecurrent usage of `?` / `??` in the Python community ? (I'll try to state only facts, excuse any passage that might seem like a personal opinion) Can the PEP include the fact that `? and `??` have been in use in the Scientific Python community for 10 to 14 years now, and that any Scientific Python user who have touched IPython will tell you that ? and ?? are for getting help. (?? try to pull the source, while ? does not, but let's not get into details). This include the fact that any IDE (like spyder) which use IPython under the hood have this feature. The usage of `?` is even visible on Python.org main page [3] (imgur screenshot), which invite the user to launch an interactive console saying:
object? -> Details about 'object', use 'object??' for extra details. In [1]:
leading the casual user thinking that this is a Python feature. This fact is even including in Books and introduction to python. Sometime without mentioning that the feature is IPython Specific, and does not work in Python repl/scripts. Examples in Cyrile's rossant "Learning IPython for Interactive Computing and Data Visualization"[1] introduce Python with the second code/repl example beeing about `?`. Book extract :
A google search also give for eaxample: Python for beginners online tutorial[2] which does rapidly the same: Tuto snippet:
Doing even worse as they replace the IPython prompt `In[x]:` with `>>>` literally showing that `>>> len?` works. Which imply that it should work on a plain Python REPL. As someone that have to regularly teach Python, and interact with new Python users, it will be hard to explain that `?` and `??` have different meaning depending on the context, and that most book on Scientific Python are wrong/inaccurate.
It will be even harder if we have to remove the usage of `?`/`??`[4]. I also want to note that the use of `?`/`??` is not present to just being or end of identifiers as it can also be used use to search for names:
But this usage is not as widespread as extracting help about objects, and seem less relevant, though I'm not sure:
Please take these fact into consideration when making a decision/writing the Pep. Thanks, -- M [1]: That's one of the only book for which I have (legally) the sources, and that I bother to grepped through. [2]: http://www.pythonforbeginners.com/basics/ipython-a-short-introduction [3]: http://imgur.com/d0Vs7Xr [4]: I'll have to hire a body guard to prevent people to pursue me to the end of the earth with a chainsaw. I'm sure you know that feeling.
On Tue, Sep 22, 2015 at 10:34 AM, Matthias Bussonnier <bussonniermatthias@gmail.com> wrote:
Are there any uses like this that would involve a question mark followed by some other punctuation? The main proposal would be for x?.y and similar; if "x ?? y" can't be used because of a conflict with ipython, I'm sure it could be changed ("x ?! y" would be cute). ChrisA
Hi Chris, On Mon, Sep 21, 2015 at 5:44 PM, Chris Angelico <rosuav@gmail.com> wrote:
As far as I can tell, no `?`/`??` behave in IPython like a unary operator[1] and don't conflict. We could distinguish from any use case so far mentioned in this discussion (as far as I can tell). I just hope that the PEP will not slide in the direction of `foo?` alone being valid and equivalent to `maybe(x)`, or returning an object. I'm concern about teaching/semantics as for me `x?.y` read like "The attribute y of the help of x" so roughly `help(x).y`. The `x ?? y` is less ambiguous (imho) as it is clearly an operator with the space on each side.
I'm sure it could be changed ("x ?! y" would be cute).
Is that the WTF operator ? [2] Joke aside, ?! should not conflict either, but `!` and `!!` also have their own meaning in IPython Like the following is valid (which is a useless piece of code, but to show the principles) if you are interested in the Python syntax extension we have. my_files = ! ls ~/*.txt for i,file in enumerate(my_files): raw = !cat $file !cat $raw > {'%s-%s'%(i,file.upper())} Thanks, -- M [1]: but is not really an operator, I'm not sure what a=print? ; a =?print or ?a=print would do. [2]: http://stackoverflow.com/questions/7825055/what-does-the-c-operator-do
Matthias Bussonnier writes:
But the syntax is extremely restrictive. Both "None? 1 + 2" and "None ?" are SyntaxErrors, as are "a?.attr" and even "*int*? ". Prefixing the help operator also gives help, and in that case whitespace may separate the operator from the word (apparently defined as non-whitespace, and any trailing detritus is ignored). Perhaps the prefix form (a little less natural for people coming directly from natural languages, I guess) should be emphasized -- there are no proposals for unary prefix use of "?" or "??". So, this is the kind of DWIM that I doubt will confuse many users, at least not for very long. Do you envision a problem keeping IPython facilities separate from Python language syntax? Technically, I think the current rules that divide valid IPython requests for help from Python syntax (valid or invalid) should continue to work. Whether it would confuse users, I doubt, but there are possible surprises for some (many?) users, I suppose. Definitely, it should be mentioned in the PEP, but Python syntax is something that Python defines; shells and language variants have to be prepared to deal with Python syntax changes.
leading the casual user thinking that this is a Python feature.
Casual users actually expect software to DWIM in my experience. The rule "leading help or stuck-right-on-the-end help works, elsewhere it means something else" (including no meaning == SyntaxError) is intuitively understood by them already, I'm sure. Also, I rather doubt that "casual users" will encounter "?." or "??" until they're not so casual anymore.
I've never had a question about the context-sensitivity of "%" in IPython. Have you?
and that most book on Scientific Python are wrong/inaccurate.
I'm afraid that's Scientific Python's cross to bear, not Python's.
It will be even harder if we have to remove the usage of `?`/`??`[4].
Not to worry about that. IPython can define its own syntax for parsing out help requests vs. Python syntax. I doubt you'll have to modify the current rules in any way.
But again "*int* ?" is a SyntaxError. This is the kind of thing most casual users can easily work with. (At least speakers of American English. In email text, my Indian students love to separate trailing punctuation from the preceding word for some reason, but they would certainly learn quickly that you can't do that in IPython.) Again, I agree it would be useful to mention this in the PEP, but as far as I can see there really isn't a conflict. The main thing I'd want to know to convince me there's a risk would be if a lot of users are confused by "%quickref" (an IPython command) vs. "3 % 2" (a Python expression).
Hi Stephen, Thanks for the response and the time you took to investigate,
Yes I’m not worrying for the time being, the current syntax proposal does not conflict, and I just wanted to describe a few usage and reference to consideration in the PEP. I prefer to give the PEP authors all the cards so that they can work out a proposal that fits majority of the people. I don’t want to let people write a pep, go through iteration and once they are happy with it complain that it does not fit my needs.
So, this is the kind of DWIM that I doubt will confuse many users, at least not for very long.
I do not think we are in contact with the same users. Yes, I see users confused by % syntax, just last Thursday, amy neighbor reported my that IPython was printing only for element of tuple, but was working for list: In [1]: print((1)) 1 In [2]: print([1]) [1] Yes people do get confused for %magics, vs modulo vs %-format, less because module is number, % for strings. But it gets betterwith time.
For no with current proposal, no no problem to keep them separate.
I can see the Python `??`/`?` vs IPython `?`/`??` being one explicit point in our docs/teaching/tutorial. I cannot say how much confusion this will be into our user head, I think the greater confusion will be the double meaning plus the fact that’s a Python 3.6+ only feature. So I doubt it will be taught before a few years. Though it is still another small difficulty. I guess we will start to get this king of experience with 3.5 now that @ is there both for decorator and __matmul__ (thanks for 3.5 in general BTW)
Definitely, it should be mentioned in the PEP,
Thanks,
Yes, we are prepared, but our user don’t always understand :-) I’m wondering if there wouldn’t be a way for interpreter to a actually help Python beta-test some syntax changes, at least at the REPL level. Like website do user testing.
In my domain people get confronted to advance syntax really rapidly, one feedback that I have for such weird syntax, especially when you are new to python, is that you don’t even now how to Google for this kind of thing (especially for non english speaker). Trying to put a name on *arg and **kwarg is hard, google is starting to get better, but still ignore chars like ?,+,- The google search for `Python` and `Python ??` seem to be identical.
Cf above yes, but more form the side where people don’t get what modulo mean, but fair enough it was non native english, and they were confused by indent/implement/increment being roughly the same word with 3 completely different meaning. Though the number of people I see using modulo is low, then they use numpy.mod on arrays once we get to numerics. And in string formatting we push for .format(*args, **kwargs). But I’ll try to gather some statistics around.
I hope that won’t change in the final PEP, and I’m not too worry, worse case we use more reg-ex, and shift the problem elsewhere :-)
French also separate punctuation, I’m still torn on that. But if we have to slightly change the rules, so be it.
Again, I agree it would be useful to mention this in the PEP, but as far as I can see there really isn't a conflict.
Happy you agree on point 1, and that you confirm point 2.
Will try to get more qualitative info. Thanks, -- M
IIRC, back when the ternary conditional was suggested, and the C-style ?: was proposed, Guido declared that no way was his language ever going to use ? an operator. So if IPython took that as a promise that they can use ? without fear of ambiguity, you can't blame them too much.... I'm not saying they have a right to expect/demand that Guido never change his mind about anything anywhere ever, just that maybe they get a little extra consideration on backward compatibility with their use of ? than with their use of ! or % (which have been in use as operators or parts of operators for decades).
On 21 September 2015 at 23:56, Carl Meyer <carl@oddbird.net> wrote:
Maybe my use of the phrase "anti-pattern" was too strong (i thought it implied a relatively mild "this causes problems"). Having the caller deal with problems isn't bad, but in my experience, too often the caller *doesn't* deal with the possibility None return. It feels rather like C's practice of returning error codes which never get checked. But as I said, YMMV, and my experience is clearly different from yours.
I don't love the ? syntax, but I would certainly use the feature discussed here happily and frequently.
If we're back to discussing indexing and attribute access rather than ??, maybe -> would work? obj->attr meaning None if obj is None else obj.attr obj->[n] meaning None if obj is None else obj[attr] obj->(args) meaning None if obj is None else obj(args) I think Matthias Bussonnier's point that ? and ?? is heavily used in IPython is a good one. Python traditionally doesn't introduce new punctuation (@ for decorators was AFAIK the last one). I thought that was precisely to leave the space of unused characters available for 3rd party tools. Paul
On 2015-09-22 03:59:07 +0000, Andrew Barnert via Python-ideas said:
I just wanted to quickly comment on what my original stance was regarding IPython's extensions to the base Python language. This was where I stood as I made decisions when the project was basically just me, and over time we've mostly adopted this as project policy. We fully acknowledge that IPython has to be a strict superset of the Python language, and we are most emphatically *not* a fork of the lanugage intended to be incompatible. We've added some extensions by hijacking a few characters that are invalid in the base language for thigns we deemed to be useful while working interactively, but we always accept that, if the language moves in our direction, it's our job to pack up and move again to a new location. In fact, that already happened once: before Python 2.4, our prefix for "magic functions" was the @ character, and when that was introduced as the decorator prefix, we had to scramble. We carefully decided to pick %, knowing that an existing binary operator would be unlikely to be added also as a new unary prefix. Now, accepting that as our reality doesn't mean that at least we don't want to *inform* you folks of what our uses are, so that at least you can consider them in your decision-making process. Since in some cases, that means there's an established ~ 15 years of a community with a habit of using a particular syntax for something, that may be confused if things change. So at least, we want to let you know. Due precisely to these recent conversations (I had a very similar thread a few days ago with Nick about the ! operator, which we also use in all kinds of nasty ways), we have started documenting more precisely all these differences, so the question "where exactly does IPython go beyond Python" can be answered in one place. You can see the progress here: https://github.com/ipython/ipython/pull/8821 We hope this will be merged soon into our docs, and it should help you folks have a quick reference for these questions. Finally, I want to emphasize that these things aren't really changing much anymore, this is all fairly stable. All these choices have by now stabilized, we only introduced the @ -> % transition when python 2.4 forced us, and more recently we introduced the notion of having a double-%% marker for "cell magics", but that was ~ 4 years ago, and it didn't require a new character, only allowing it to be doubled. Best, f
On 2015-09-23 02:21, Fernando Perez wrote:
From the examples I've seen, the "?" and "??" occur at the end of the line. The proposed 'operators' "?.", "?[", "?(" and "??" wouldn't occur at the end of the line (or, if they did, they'd be inside parentheses, brackets, or braces). So is there really a conflict, in practice?
beginning of line can happened too. ?print is equivalent to print?
As stated in previous mails, with current state of proposal no it does not conflict, we should be able to distinguish the two cases. We are just informing the pep authors and contributors of the syntax hijack that we did and currently have in IPython. -- M
On Mon, Sep 21, 2015 at 05:23:42PM -0400, Terry Reedy wrote:
I agree with Paul Moore that propagating None is generally a bad idea.
As I understand it, you and Paul are describing a basic, simple idiom which is ubiquitous across Python code: using None to stand in for "no such value" when the data type normally used doesn't otherwise have something suitable. Consequently I really don't understand what you and Paul have against it.
It merely avoids the inevitable exception.
I think you meant to say it merely *postpones* the inevitable exception. But that's wrong, there's nothing inevitable about an exception here. It's not *hard* to deal with "value-or-None". It's just tedious, which is why a bit of syntactic sugar may appeal. [...]
In one of my earlier posts, I discussed this Null object design pattern. I think it is an anti-pattern. If people want to add one to their own code, it's their foot, but I certainly don't want to see it as a built-in. Thank goodness Guido has already ruled that out :-)
I agree with Ron Adam that the narrow issue is that bool(x) is False is sometimes too broad and people dislike of spelling out 'x is not None'.
I don't think that is the motivation of the original proposal, nor is it one I particularly care about. I think that there is a level of inconvenience below which it's not worth adding yet more syntax just to save a few characters. That inconvenience is not necessarily just to do with the typing, it may be conceptual, e.g. we have "x != y" rather than "not x == y". I think that x is not None fails to reach that minimum level of inconvenience to justify syntactic sugar, but obj.method() if x is not None else None does exceed the level. So I am mildly interested in null-coalescing versions of attribute and item/key lookup, but not at all interested in making the "x is not None" part *alone* shorter.
Bring it back to the original post's motivating use-case. Slightly paraphrased, it was something like: value = obj.method() if obj is not None else None Having x! as a short-cut for "x is not None" makes this a bit shorter to write: value = obj.method() if obj! else None but it is still very boilerplatey and verbose compared to the suggested: value = obj?.method() -- Steve
On 22.09.2015 05:15, Steven D'Aprano wrote:
There is not a single "no such value". As I mentioned before, when discussing NULL values on the RDF mailing list, we discovered 6 or 7 domain-agnostic meanings.
I can tell from what I've seen that people use None for: all kinds of various interesting semantics depending on the variable, on the supposed type and on the function such as: - +infinity for datetimes but only if it signifies the end of a timespan - current datetime - mixing both - default item in a list like [1, 2, None, 4, 9] (putting in 5 would have done the trick) - ... Really? Just imagine a world where Python and other systems would have never invented None, NULLs or anything like that.
It's a sign of bad design. So, syntactic sugar does not help when doing toilet paper programming (hope that translation works for English). Best, Sven
On Wed, Sep 23, 2015 at 4:22 AM, Sven R. Kunze <srkunze@mail.de> wrote:
What this means is that your boundaries can be a datetime or None, where None means "no boundary at this end".
- current datetime - mixing both
I don't know of a situation where None means "now"; can you give an example?
- default item in a list like [1, 2, None, 4, 9] (putting in 5 would have done the trick)
What does this mean? Is this where you're taking an average or somesuch, and pretending that the None doesn't exist? That seems fairly consistent with SQL. Mostly, this does still represent "no such value". ChrisA
On 23.09.2015 00:53, Chris Angelico wrote:
Yes.
range_start = <proper datetime> range_end = <proper datetime or one of the above> So, if you need something that ranges from 2015-01-01 to now, you basically say the range is expanding. Depending on the function/method, it either means until forever, or now.
Imagine you render (as in HTML and the like) 1, 2, 4, 9 and instead of the None you render a 3. Now, the rendering engine needs to special-check None to put in a pre-defined value. Furthermore, all places where you need that list [1, 2, None, 9], you basically need to special-check None and act appropriately. (Of course it was not that simple but you get the idea. The numbers stand for fairly complex objects drawn from the database.)
Mostly, this does still represent "no such value".
Point was "no such value" sucks. It can be a blend of every other value and semantics depending on the function, type and so forth. It's too convenient that people would not use it. As the example with the list shows us, the 3 could have easily be put into the database as it behaves exactly the same as the other objects. The same goes for the special datetime objects. The lack of thinking and appropriate default objects, lead to the usage of None. People tend to use None for everything that is special and you end up with something really nasty to debug. Not why people don't find it problematic when I said "we found 6/7 domain-agnostic semantics for NULL". The "no such value" can be any of them OR a blend of them. That, I don't want to see in the code; that's all. Btw. having the third issue of above, I could add another domain-agnostic meaning for None: "too lazy to create a pre-defined object but instead using None".
What about re-using try? Crystal does this ( http://play.crystal-lang.org/#/r/gf5): v = "ABC" puts nil == v.try &.downcase # prints true v = nil puts nil == v.try &.downcase # prints false Python could use something like: v = 'ABC' print(v try.downcase is None) # prints False v = None print(v try.downcase is None) # prints True (Of course, the syntax would be a little less...weird!) On Mon, Sep 21, 2015 at 10:40 AM, Guido van Rossum <guido@python.org> wrote:
-- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something’s wrong. http://kirbyfan64.github.io/
On 09/21/2015 03:44 PM, Terry Reedy wrote:
Yes
My observation is that because None is the default return value from functions (and methods), it is already a special case. While that isn't directly related to None values in general, I think it does lend weight to treating it specially. Having None specific bool-type operators is both cleaner and more efficient and avoids issues with false values. The byte code might look like this...
It would not be sensitive to False values. Applying the op to the object wouldn't quite work as expected. What would x! return? True, or the object if not None? And if it returns the object, what does this do when the value is 0 or False, or an empty container. result = x! or y # not the same as result = x or! y The maybe(x) function would work the same as x! in this case. I also think a trailing unary operator is kind of weird. But I get the feeling from Guido response about over generalizations that it may be too big of a change, and I agree it is a new concept I haven't seen anywhere else. Maybe one to think about over time, and not to rush into. Cheers, Ron
Mark E. Haase writes:
To me, it's less defensible. Eg, currently TOOWTDI for "??" is the idiom quoted above. I sorta like the attribute access, attribute fetch, and function call versions, though I probably won't use them. Also some functions need to accept None as an actual argument, and the module defines a module-specific sentinel. The inability to handle such sentinels is a lack of generality that the "x if x is not sentinel else y" idiom doesn't suffer from, so "??" itself can't become TOOWTDI. I don't write "def foo(default):" (ever that I can recall), so using "default" in retries = default if default is not None else cls.DEFAULT confuses me. Realistically, I would be writing retries = retries if retries is not None else cls.RETRIES (or perhaps the RHS would be "self.retries"). That doesn't look that bad to me (perhaps from frequent repetition). It's verbose, but I don't see a need to chain it, unlike "?.". For "?.", some Pythonistas would say "just don't", but I agree that often it's natural to chain.
Worse? It's true that it's more risky because it's all falsies, not just the None sentinel, but I think "consenting adults" applies here. I don't know about the packages you looked at, but I often use "x = s or y" where I really want to trap the falsey value of the expected type, perhaps as well as None, and I use the "x if s is not sentinel else y" idiom to substitute default values. I also use "or" in scripty applications and unit test setup functions where I want compact expression and I don't expect long-lived objects to be passed so I can easily figure out where the non-None falsey came from anyway.
A) Is coalesce a useful feature? (And what are the use cases?)
Yes, for the whole group of operators. Several use cases for the other operators have already been proposed, but I wouldn't use them myself in current or past projects, and don't really foresee that changing. -0 for the group on the IAGNI principle. But for "??" specifically, it's just more compact AFAICS. I don't see where I would use x ?? y ?? z, so the compactness doesn't seem like that great a benefit. In practice, I think the use cases for "??" would be a strict subset of the use cases for the ternary operator, so you have to argue that "this special case *is* special enough" to have its own way to do it. I don't think it is. -1
C) If it should be an operator, is "??" an ugly spelling?
>>> retries = default ?? cls.DEFAULT
Looks like metasyntax from pseudo-code that didn't get fixed to me. That would probably change if other ?x operators were added though. I have no comment on short-circuiting (no relevant experience), or keyword vs. punctuation spellings. On second thought:
I kinda like this if-less else syntax for the symmetry with else-less if. But on second thought I think it would persistently confuse me when reading, because it would be extremely natural to expect it to be another way of spelling "default or cls.DEFAULT". "try ... else ..." also has its attraction, but I suppose that would fail for the same reasons that the ternary operator is spelled "x if y else z" rather than "if y then x else z".
On 21.09.2015 11:05, Andrew Barnert via Python-ideas wrote:
That's cool. It reads nice (at least for a non-native speaker). Also chaining else reads nice: final_value = users_value else apps_value else systems_value
Use parentheses if you mix up if-else and else. ;) Btw. the same applies for: a + b * c + d If you don't know from you education that b*c would have been evaluated first, then it's not obvious either. Best, Sven
Andrew Barnert writes:
I don't know what an LL(1) parser could do offhand. As a human, I would parse that greedily as (a if b else c) else d. But the point's actually moot, as I'm -1 on the "??" operator in any form in favor of the explicit "a if a is not None else b" existing syntax. And to be honest, the fact that a truly symmetric "if-less else" would have "or" semantics, not "??" semantics, bothers me more than the technical issue of whether anybody could actually parse it.
On 21 September 2015 at 03:35, Mark E. Haase <mehaase@gmail.com> wrote:
A) Is coalesce a useful feature? (And what are the use cases?)
There seem to be a few main use cases: 1. Dealing with functions that return a useful value or None to signal "no value". I suspect the right answer here is actually to rewrite the function to not do that in the first place. "Useful value or None" seems like a reasonable example of an anti-pattern in Python. 2. People forgetting a return at the end of the function. In that case, the error, while obscure, is reasonable, and should be fixed by fixing the function, not by working around it in the caller. 3. Using a library (or other function outside your control) that uses the "useful value or None" idiom. You have to make the best of a bad job here, but writing an adapter function that hides the complexity doesn't seem completely unreasonable. Nor does just putting the test inline and accepting that you're dealing with a less than ideal API. 4. Any others? I can't think of anything. Overall, I don't think coalesce is *that* useful, given that it seems like it'd mainly be used in situations where I'd recommend a more strategic fix to the code.
B) If it is useful, is it important that it short circuits? (Put another way, could a function suffice?)
Short circuiting is important, but to me that simply implies that the "useful value or None" approach is flawed *because* it needs short-circuiting to manage. In lazy languages like Haskell, the Maybe type is reasonable because short-circuiting is a natural consequence of laziness, and so not a problem. In languages like C#, the use of null as a sentinel probably goes back to C usage of NULL (i.e., it may not be a good approach there either, but history and common practice make it common enough that a fix is needed).
C) If it should be an operator, is "??" an ugly spelling?
>>> retries = default ?? cls.DEFAULT
Arbitrary punctuation as operators is not natural in Python, something like this should be a keyword IMO.
Reusing existing keywords (specifically, all of the above) looks clumsy and forced to me. I agree that proposals to add a new keyword will probably never get off the ground, but none of the above suggestions look reasonable to me, and I can't think of anything else that does (particularly if you add "must be parseable" as a restriction!) Overall, I'm -0.5 on a "coalesce" operator. I can't see it having sufficient value, and I can't think of a syntax I'd consider justifying it. But if someone were to come up with a Guido-like blindingly obvious way to spell the operation, I would be fine with that (and may even start using it more often than I think). Paul
On Mon, Sep 21, 2015 at 8:55 PM, Paul Moore <p.f.moore@gmail.com> wrote:
The alternative being to raise an exception? It's generally easier, when you can know in advance what kind of object you're expecting, to have a None return when there isn't one. For example, SQLAlchemy has .get(id) to return the object for a given primary key value, and it returns None if there's no such row in the database table - having to wrap that with try/except would be a pain. This isn't an error condition, and it's not like the special case of iteration (since an iterator could yield any value, it's critical to have a non-value way of signalling "end of iteration"). I don't want to see everything forced to "return or raise" just because someone calls this an anti-pattern. ChrisA
On 21 September 2015 at 14:27, Chris Angelico <rosuav@gmail.com> wrote:
Agreed, that's not what should happen. It's hard to give examples without going into specific cases, but as an example, look at dict.get. The user can supply a "what to return if the key doesn't exist" argument. OK, many people leave it returning the default None, but they don't *have* to - dict.get itself doesn't do "useful value or None", it does "useful value or user-supplied default". All I'm saying is that people should look at *why* their functions return None instead of a useful result, and see if they can do better. My contention is that (given free rein) many times they can. Of course not all code has free rein, not all developers have the time to look for perfect APIs, etc. But in that case, returning a placeholder None (and accepting a little ugliness at the call site) isn't an impossible price to pay. Nothing more than "I don't think the benefit justifies adding a new operator to Python". Paul
On 21.09.2015 15:27, Chris Angelico wrote:
I don't think both approaches are mutual exclusive. They can both exist and provide whenever I need the right thing. Depending on the use-case, one needs to decide: If I know, the value definitely needs to be a dictionary, I use dict[...]. If I know, the value is definitely optional and I can't do anything about it, I use dict.get('key'[, default]). If I definitely don't know, I use dict[...] to get my hands on a real example with out that key if that every happens and don't waste time for special-handling a possible None return value. Best, Sven
On Mon, Sep 21, 2015 at 11:55:55AM +0100, Paul Moore wrote:
I think that's a bit strong. Or perhaps much too strong. There are times where you can avoid the "None or value" pattern, since there is a perfectly decent empty value you can use instead of None. E.g. if somebody doesn't have a name, you can use "" instead of None, and avoid special treatment. But that doesn't always work. Suppose you want an optional (say) Dog object. There isn't such a thing as an empty Dog, so you have to use some other value to represent the lack of Dog. One could, I suppose, subclass Dog and build a (singleton? borg?) NoDog object, but that's overkill and besides it doesn't scale well if you have multiple types that need the same treatment. So I don't think it is correct, or helpful, to say that we should avoid the "None versus value" pattern. Sometimes we can naturally avoid it, but it also has perfectly reasonable uses.
Go back to the original use-case given, which, paraphrasing, looks something like this: result = None if value is None else value['id'].method() I don't think we can reject code like the above out of hand as un-Pythonic or an anti-pattern. It's also very common, and a little verbose. It's not bad when the value is a name, but sometimes it's an expression, in which case it's both verbose and inefficient: result = None if spam.eggs(cheese) is None else spam.eggs(cheese)['id'].method() Contrast: result = spam.eggs(cheese)?['id'].method() which only calculates the expression to the left of the ?[ once. An actual real-life example where we work around this by using a temporary name that otherwise isn't actually used for anything: mo = re.match(needle, haystack) if mo: substr = mo.group() else: substr = None I think it is perfectly reasonable to ask for syntactic sugar to avoid having to write code like the above: substr = re.match(needle, haystack)?.group() That's not to say we necessarily should add sugar for this, since there is no doubt there are disadvantages as well (mostly that many people dislike the ? syntax), but in principle at least it would certainly be nice to have and useful.
Nothing needs short-circuiting, at least in a language with imperative assignment statements. You can always avoid the need for short-circuits with temporary variables, and sometimes that's the right answer: not everything needs to be a one-liner, or an expression. But sometimes it is better if it could be.
C) If it should be an operator, is "??" an ugly spelling?
>>> retries = default ?? cls.DEFAULT
I assume the ?? operator is meant as sugar for: retries = cls.DEFAULT if default is None else default I prefer to skip the "default" variable and use the standard idiom: if retries is None: retries = cls.DEFAULT I also worry about confusion caused by the asymmetry between ?? and the other three ? cases: # if the left side is None, return None, else evaluate the right side spam?.attr spam?['id'] spam?(arg) # if the left side is None, return the right side, else return the left spam ?? eggs but perhaps I'm worried over nothing. -- Steve
On 21 September 2015 at 15:41, Steven D'Aprano <steve@pearwood.info> wrote:
Well, (1) Mark had focused on the "coalesce" operator ??, not the ?. variant, and that is less obviously useful here, and (2) I find the former version more readable. YMMV on readability of course - which is why I added the proviso that if someone comes up with an "obviously right" syntax, I may well change my mind. But the options suggested so far are all far less readable than a simple multi-line if (maybe with a temporary variable) to me, at least. By the way, in your example you're passing on the "none or useful" property by making substr be either the matched value or None. In real life, I'd probably do something more like mo = re.match(needle, haystack) if mo: process(mo.group()) else: no_needle() possibly with inline code if process or no_needle were simple. But it is of course easy to pick apart examples - real code isn't always that tractable. For example we get (what seems to me like) a *lot* of bug reports about "None has not attribute foo" style errors in pip. My comments here are based on my inclinations about how I would fix them in pip - I'd always go back to *why* we got a None, and try to avoid getting the None in the first place. But that's not always easy to do. Again, of course, YMMV. Paul
On 9/21/2015 10:56 AM, Paul Moore wrote:
By the way, in your example you're passing on the "none or useful" property by making substr be either the matched value or None.
I agree that dealing with None immediately is better. In real
try: process(re.match(needle, haystack).group()) except AttributeError: # no match no_needle() is equivalent unless process can also raise AttributeError. -- Terry Jan Reedy
On Mon, Sep 21, 2015 at 04:23:03PM -0400, Terry Reedy wrote:
It is difficult to guarantee what exceptions a function will, or won't, raise. Even if process() is documented as "only raising X", I wouldn't be confident that the above might not disguise a bug in process as "no needle". This is why the standard idiom for using re.match is to capture the result first, then test for truthiness (a MatchObject), or None-ness, before processing it. -- Steve
On Sun, Sep 20, 2015 at 12:05:52PM +0100, Paul Moore wrote:
Sadly, I think it does. Guido has (I think) ruled out the Null object design pattern, which makes me glad because I think it is horrid. But your Maybe class below is a watered down, weak version that (in my opinion) isn't worth bothering with. See below. class Maybe: def __getattr__(self, attr): return None def __getitem__(self, idx): return None def __call__(self, *args, **kw): return None def maybe(obj): return Maybe() if obj is None else obj And in action: py> maybe("spam").upper() # Works fine. 'SPAM' py> maybe(None).upper() Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'NoneType' object is not callable It also fails for chained lookups: maybe(obj).spam['id'].ham will fail for the same reason. You could write this: maybe(maybe(obj).upper)() maybe(maybe(maybe(obj).spam)['id']).ham but that's simply awful. Avoiding that problem is why the Null object returns itself, but we've rightly ruled that out. This is why I think that if this is worth doing, it has to be some sort of short-circuiting operator or pseudo-operator: expression ? .spam.eggs.cheese can short-circuit the entire chain .spam.eggs.cheese, not just the first component. Otherwise, I don't think it's worth doing. -- Steve
On Fri, Sep 18, 2015, at 18:37, Chris Angelico wrote:
More or less - it'd only look up the attribute once.
?[ has the benefit of being consistent with ?. - and ?(, for that matter. It actually suggests a whole range of null-coalescing operators. ?* for multiply? A lot of these things are done already by the normal operators for statically-typed nullable operands in C#. That could get hairy fast - I just thought of a radical alternative that I'm not even sure if I support: ?(expr) as a lexical context that changes the meaning of all operators.
I've now read PEP 505, and I would like to comment. Executive summary: - I have very little interest in the ?? and ?= operators, but don't object to them: vote +0 - I have a little more interest in ?. and ?[ provided that the precedence allows coalescing multiple look-ups from a single question mark: vote +1 - if it uses the (apparent?) Dart semantics, I am opposed: vote -1 - if the syntax chosen uses || or !| as per Nick's suggestion, I feel the cryptic and ugly syntax is worse than the benefit: vote -1 In more detail: I'm sympathetic to the idea of reducing typing, but I think it is critical to recognise that reducing typing is not always a good thing. If it were, we would always name our variables "a", "b" etc, the type of [] would be "ls", and we would say "frm col impt ODt". And if you have no idea what that last one means, that's exactly my point. Reducing typing is a good thing *up to a point*, at which time it becomes excessively terse and cryptic. One of the things I like about Python is that it is not Perl: it doesn't have an excess of punctuation and short-cuts. Too much syntactic sugar is a bad thing. The PEP suggests a handful of new operators: (1) Null Coalescing Operator spam ?? eggs equivalent to a short-circuiting: spam if spam is not None else eggs I'm ambivalent about this. I don't object to it, but nor does it excite me in the least. I don't think the abbreviated syntax gains us enough in expressiveness to make up for the increase in terseness. In its favour, it can reduce code duplication, and also act as a more correct alternative to `spam or eggs`. (See the PEP for details.) So I'm a very luke-warm +0 on this part of the PEP. (2) None coalescing assignment spam ?= eggs being equivalent to: if spam is None: spam = eggs For the same reasons as above, I'm luke-warm on this: +0. (3) Null-Aware Member Access Operator spam?.attr being equivalent to spam.attr if spam is not None else None To me, this passes the test "does it add more than it costs in cryptic punctuation?", so I'm a little more positive about this. If my reading is correct, the PEP underspecifies the behaviour of this when there is a chain of attribute accesses. Consider: spam?.eggs.cheese This can be interpreted two ways: (a) (spam.eggs.cheese) if spam is not None else None (b) (spam.eggs if spam is not None).cheese but the PEP doesn't make it clear which behaviour they have in mind. Dart appears to interpret it as (b), as the reference given in the PEP shows this example: [quote] You can chain ?. calls, for example: obj?.child?.child?.getter [quote] http://blog.sethladd.com/2015/07/null-aware-operators-in-dart.html That would seem to imply that obj?.child.child.getter would end up trying to evaluate null.child if the first ?. operator returned null. I don't think the Dart semantics is useful, indeed it is actively harmful in that it can hide bugs: Suppose we have an object which may be None, but if not, it must have an attribute spam which in turn must have an attribute eggs. This implies that spam must not be None. We want: obj.spam.eggs if obj is not None else None Using the Dart semantics, we chain ?. operators and get this: obj?.spam?.eggs If obj is None, the expression correctly returns None. If obj is not None, and obj.spam is not None, the expression correctly returns eggs. But it is over-eager, and hides a bug: if obj.spam is None, you want to get an AttributeError, but instead the error is silenced and you get None. So I'm -1 with the Dart semantics, and +1 otherwise. (3) Null-Aware Index Access Operator spam?[item] being similar to spam.attr. Same reasoning applies to this as for attribute access. Nick has suggested using || instead of ??, and similar for the other operators. I don't think this is attractive at all, but the deciding factor which makes Nick's syntax a -1 for me is that it is inconsistent and confusing. He has to introduce a !| variation, so the user has to remember when to use two |s and when to use a ! instead, whether the ! goes before or after the | and that !! is never used. -- Steve
Steven, thanks for the reply. Just to clarify: the current PEP draft was not meant to be read -- it was just a placeholder to get a PEP # assigned. I didn't realize that new PEPs are published in an RSS feed! I do appreciate your detailed feedback, though. Your interpretation of Dart's semantics is correct, and I agree that's absolutely the wrong way to do it. C# does have the short-circuit semantics that you're looking for. To you, and to everybody else in this thread: I am reading every single message, and I'm working on a draft worthy of your time and attention that incorporates all of these viewpoints and offers several, competing alternatives. I will announce the draft on this list when I'm further along. Proposing a new operator is tremendously difficult in Python, because this community doesn't like complex or ugly punctuation. (And adding a new keyword won't happen any time soon.) A similar debate surrounded the ternary operator PEP[1]. That PEP's author eventually held a vote on the competing alternatives, including an option for "don't do anything". I'm hoping to hold a similar referendum on this PEP once I've had time to work on it a bit more. [1] https://www.python.org/dev/peps/pep-0308/ On Wed, Sep 23, 2015 at 12:47 PM, Steven D'Aprano <steve@pearwood.info> wrote:
-- Mark E. Haase 202-815-0201
On Wed, Sep 23, 2015 at 11:47 AM, Steven D'Aprano <steve@pearwood.info> wrote:
from collections import OrderedDict
I have to kind of agree here. In reality, I don't see any issues like this with approach (a).
-- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something’s wrong. http://kirbyfan64.github.io/
On 23 September 2015 at 17:47, Steven D'Aprano <steve@pearwood.info> wrote:
I've now read PEP 505, and I would like to comment.
Having read the various messages in this thread, and then your summary (which was interesting, because it put a lot of the various options next to each other) I have to say: 1. The "expanded" versions using if..else are definitely pretty unreadable and ugly (for all the variations). But in practice, I'd be very unlikely to use if expressions in this case - I'd be more likely to expand the whole construct, probably involving an if *statement*. Comparing a multi-line statement to an operator is much harder to do in a general manner. So I guess I can see the benefits, but I suspect the operators won't be used in practice as much as people are implying (in much the same way that the use if the if expression is pretty rare in real Python code, as opposed to examples). 2. All of the punctuation-based suggestions remain ugly to my eyes. ? is too visually striking, and has too many other associations for me ("help" in IPython, and as a suffix for variable names from Lisp). Nick's || version looked plausible, but the inconsistent !| variations bother me. 3. People keep referring to "obj ?? default" in comparison to "obj or default". The comparison is fine - as is the notion that we are talking about a version that simply replaces a truth test with an "is None" test. But to me it also says that we should be looking for a keyword, not a punctuation operator - the "or" version reads nicely, and the punctuation version looks very cryptic in comparison. I can't think of a good keyword, or a viable way to use a keyword for the ?. ?[ and ?( variations, but I wish I could. Summary - I don't mind the addition of the functionality, although I don't think it's crucial. But I really dislike the punctuation. The benefits don't justify the cost for me. Paul
participants (29)
-
Akira Li -
Andrew Barnert -
Brendan Barnwell -
Bruce Leban -
C Anthony Risinger -
Carl Meyer -
Chris Angelico -
David Mertz -
Erik -
Fernando Perez -
Greg Ewing -
Guido van Rossum -
Joseph Jevnik -
Mark E. Haase -
Mark Haase
-
Matthias Bussonnier -
MRAB -
Oleg Broytman -
Paul Moore -
Random832 -
Ron Adam -
Ryan Gonzalez -
Serhiy Storchaka -
Stephen J. Turnbull -
Steven D'Aprano -
Sven R. Kunze -
Terry Reedy -
Trent Nelson -
Xavier Combelle