PEP 505 (None-aware operators) for Python 3.11

Hello, I've been following PEP 505 (https://www.python.org/dev/peps/pep-0505/) since it was first proposed for Python 3.8. It's been deferred for some time and I'm interested in seeing it in Python 3.11, but I know there were also a number of objections which resulted in it being deferred (including by one of the original authors, Mr. Dower). I did email both Mr. Dower and Mr. Haase and they graciously gave me their permission to bring it up on this list for discussion and hopefully final pronouncement one way or the other. I personally believe that the PEP will result in a significant reduction in boilerplate code, and it is substantially similar to the same operators now found in a number of other languages, especially C# and JavaScript (https://wikipedia.org/wiki/Null_coalescing_operator and https://wikipedia.org/wiki/Safe_navigation_operator). I believe strong and valid arguments can be made about the use of None being a fundamental flaw in some types of coding (and that adding additional support for it to the language will increase the use of None in this way), but I also believe there are many use cases in programming where it is by far the simplest way to express various semantics, and the fact exists that None is already used extensively in large quantities of code, and further that there is already a great deal of code written to constantly test against None and break out of a statement without throwing an error. I also understand the argument that especially the maybe-dot (?.) and maybe-subscript (?[) operators can decrease readability of code and also believe these are valid arguments against it. While I believe the existence and use of these operators in other languages definitely helps the case that these can be used and understood successfully, I think it is entirely valid to either consider other syntax (though I prefer the chosen syntax of PEP 505), or even to reduce PEP 505 to having only the coalesce operator (??) and the maybe-assign operator (??=). Separately, I have implemented a pure-Python solution for PEP505 (which is definitely rather beta) which might help test the waters for a final implementation in CPython (though the CPython implementation would of course be much more efficient). It can be found at https://pypi.org/project/pep505/ Thanks, Doug

Thanks -- this is the kind of work that helps a PEP get accepted. I am personally in favor of accepting PEP 505, and I hope that your work and the discussion that will undoubtedly follow here will help convince the Steering Council to accept it. --Guido On Thu, Oct 14, 2021 at 10:38 AM Doug Swarin <dswarin@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>

I tried to implement this in CPython by modifying a downloaded source code, but I can't seem to fix the problem of the "maybe" operators segfaulting when being used with literal immutables. The maybe-assign/coalesce operators were implemented successfully though.

Hello Doug, On Thu, Oct 14, 2021 at 03:45:07PM -0000, Doug Swarin wrote:
I believe strong and valid arguments can be made about the use of None being a fundamental flaw in some types of coding
Can you elaborate on that? Obviously it is not always appropriate to use None, but I've never seen it called a *fundamental* flaw. I know that the null pointer has been called a billion-dollar mistake, but Python's None is not a null pointer. -- Steve

Steven D'Aprano wrote:
I apologize that I may have spoken too strongly here. When I emailed Mr. Dower, he mentioned that he now believes the implementation of these operators would lead people to a style of coding which would lead to the proliferation of None as an exception-less error result and also throughout data structures. My understanding is that his current preference is to focus on functional composition and styles of programming that disallow the use of None. I certainly don't mean to speak for him and I hope he will weigh in with a more detailed explanation of his thoughts and objections, but I personally disagree as a matter of pure practicality. It's just plain useful to be able to easily take non-values and safely deal with them without having to constantly check for None or to catch and inspect exceptions to see if it's a case that can be ignored. I did indeed think about connecting None to the 'billion dollar mistake' but decided against it since as you say None is not a null pointer, and I should have chosen my words a little more carefully when revising my initial post (likely by removing the word 'fundamental'). Doug

Thanks for allowing me to speak for myself, as I said I would when contacted off list :) But as it happens, you've summarised my position very accurately:
he now believes the implementation of these operators would lead people to a style of coding which would lead to the proliferation of None as an exception-less error result and also throughout data structures. My understanding is that his current preference is to focus on functional composition and styles of programming that disallow the use of None.
As additional context, C# and the .NET Framework have just gone through the exercise of making (and propagating) non-nullable reference types (and their "null" is far more like our None than C's NULL). Our type checkers are also pushing in this direction by requiring Optional[]. The argument in all cases is that it allows for simpler code that can safely ignore non-values because they cannot be passed. Adding ??, ?. and ?[, particularly in their short-circuiting evaluation form, encourages more complex code by adding value-based branching within an expression. And it encourages it on the "outside" of the API, not within it. Which means API designers can more easily justify returning None because their caller can use None-aware operators to basically ignore it. (I would argue that the API designer should work harder to create an API that doesn't ever have to return None.) To be clear, I'm thinking very much in terms of "impact on what people will consider to be Pythonic API design over the next 10 years", that is, what people think they *should* do with this, rather than simply what they *could*. So it's all very hypothetical and I have no data for it, much like when it was decided that Optional[] would be mandatory on types where None is permitted. And with that, I'm bowing out of the fresh discussion (unless the SC wants to discuss it directly with me). I'll continue advocating for tools/features that help people create better APIs, but I'm not going to spend a lot of time arguing against those that I believe will not. Cheers, Steve On 10/15/2021 3:08 AM, Doug Swarin wrote:

NoneType is just another type, and in type checking scenarios should be expressed with `Optional[type]` or more preferably in the future `type | None`; `None` is not a non-value. Assuming what I just wrote is true, I don't get what the basis of this thread is; what am I missing? On Mon, 2021-10-18 at 14:13 +0100, Steve Dower wrote:

On Mon, Oct 18, 2021 at 9:35 AM Paul Bryan <pbryan@anode.ca> wrote:
To me the thread isn't about type checking. It is about APIs that are built into the language that special-case None, in particular dict.get(). In certain cases, encountered commonly in certain styles of coding (data science? web programming?), users encounter data that is structured as dicts nested multiple levels. This is often out of the user's control, as the dict is returned by reading a JSON value whose structure is controlled by some other framework (often not specific to Python). For example, if we have a config structure like this: config = { "timeout": 0.1, "handler: { "timeout-override": 0.4, "method-name": "plot", "parameters": { "x": 10, "y": "auto", } } } where the convention is that keys at any level may be omitted altogether and config itself may be NOne, then to safely access the value of config["handler"]["parameters"]["y"] we would have to write y = None # Default if config is not None: handler = config.get("handler") if handler is not None: parameters = handler.get("parameters") if parameters is not None: y = parameters.get("y") This kind of pattern (and all the various other ways of writing it, e.g. using the walrus or passing {} as the second argument to dict.get()) can be *very* common if that's the kind of data you're given and that's the kind of app you have to write, and you can't control the format of the data. Using ?. this can be written as y = config?.get("handler")?.get("parameters")?.get("y") More examples are in PEP 505 itself, see https://www.python.org/dev/peps/pep-0505/#examples -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>

On Mon, 18 Oct 2021 at 19:29, Guido van Rossum <guido@python.org> wrote:
For this particular usage, I'd much rather have a functional API, like y = get_config(config, "handler", "parameters", "y") I understand that writing many helpers like that nested_get is a chore, and having language support for the operation avoids that chore - but I'm with Steve in not wanting to see the ?.get() pattern become "idiomatic Python" as it encourages people *not* to design APIs like nested_get that allow the user to not even be aware of all that behind-the-scenes complexity. Sure, you could argue that the ?. operator makes it easier to write something like get_config, but it's not *that* hard: def get_config(config, *keys): value = config for key in keys: if value is None: break value = value.get(key) return value And the problem with the ?.get style is that it doesn't hide anything - what if you want/need to change your config data structure (because the JSON you're reading changes its layout, say)? Without encapsulation, you can't. And if it's *that* common, adding a stdlib function or a new dict method is also an option, which doesn't need a language feature and demonstrates good (IMO) API design for people to copy. Anyway, much like Steve, I don't expect to spend a lot of time fighting this proposal. But I will be sad if it gets accepted, and even more so if ?. becomes idiomatic in user code. Paul

On Mon, Oct 18, 2021 at 6:49 PM Paul Moore <p.f.moore@gmail.com> wrote:
I agree with Paul here... and am pretty sure I did the last time this went around. Whenever the issue comes up, it's about JSON. Or *maybe* about something very similar that goes by another name or format details. And there already exists a pretty good, pretty standard, approach called JSONPath that deals with exactly this kind of thing. This, in turn, is largely the same as XPath which serves the same role for XML documents. I don't think it necessarily needs to be in the standard library, but the mini-language for extracting data from trees with frequently missing branches can very well simply be a mini-language. It'll wind up a lot like XPath/JSONPath, but something a little bit different could be good too. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.

I should have added that I also don't feel I want to go at bat to fight for this PEP. I do observe that it looks like the folks used to building large systems (in Python or other languages) don't seem to like it, while it seems to appeal to folks writing simpler code (the abundant majority of Python users, but not of Python core devs). I worry that the experienced folks may perhaps be a little too eager to protect newbies from shooting themselves in the foot. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>

On Tue, 19 Oct 2021 at 00:55, Guido van Rossum <guido@python.org> wrote:
I should have added that I also don't feel I want to go at bat to fight for this PEP. I do observe that it looks like the folks used to building large systems (in Python or other languages) don't seem to like it, while it seems to appeal to folks writing simpler code (the abundant majority of Python users, but not of Python core devs). I worry that the experienced folks may perhaps be a little too eager to protect newbies from shooting themselves in the foot.
Possibly. But in *my* case, I'm not arguing from the position of a large system builder, but from that of a script writer (albeit an experienced one) and code maintainer (usually of my own code). I find y = config?.get("handler")?.get("parameters")?.get("y") unreadable and confusing, and I'd probably advise strongly against it if someone ever showed me code containing it. I see y = get_config(config, "handler", "parameters", "y") as *far* more readable and expressing the intent more directly. Yes, I find Steve's arguments persuasive, but they are not the ones I'd be concerned with when advising a newcomer. Rather I'd be saying "do you see how using a named function expresses your intent better?" and "do you see how writing a small function hides the messiness of checking for None so that your main code is cleaner?" *Shrug* I guess I just don't understand how people can look at a string of ?.get() and see it as readable and obvious :-( Paul

Le 18/10/2021 à 20:26, Guido van Rossum a écrit :
Sure, but the EAFP version is not that bad: try: y = config["handler"]["parameters"]["y"] except KeyError: y = None which could be further simplified with an exception-catching expression (caveat: keyword usage is pure improvisation, it sounds good, but is probably broken :-) : y = config["handler"]["parameters"]["y"] with KeyError as None The PEP authors would probably reject this as "hiding errors in code", which is true, but none-aware operators also hide errors… Cheers, Baptiste

On Tue, Oct 19, 2021 at 9:39 AM Chris Angelico <rosuav@gmail.com> wrote:
In case it saves anyone a couple clicks: https://www.python.org/dev/peps/pep-0463/ I also prefer more syntactic help with exceptions, rather than more syntax emphasizing None's uniqueness. None and its ilk often conflate too many qualities. For example, is it missing because it doesn't exist, it never existed, or because we never received a value, despite knowing it must exist? The languages SAS and R support at least 27 varieties of NA, allowing un-tagged, and tagged with the letters A-Z to help someone create distinctions between different kinds of nothingness. IEEE-754 allows about 16 million possible NaNs, which I believe was intended to allow floating point instructions to pass error messages along. If the motivation for this operator is chained lookups, how about adding a feature to the operator module, first? It seems natural to add a keyword-only argument to `attrgetter`, and it's a lighter touch than implementing a new operator. If use becomes widespread, that gives more weight to PEP 505. def attrgetter(*attrs, none_aware=False) https://docs.python.org/3/library/operator.html#operator.attrgetter Apologies if attrgetter has already been discussed. I didn't see mention of it in PEP 505.

Hi, On Wed, Oct 20, 2021 at 2:33 AM Michael Selik <mike@quantami.com> wrote:
Me too, but could you provide me an example where try-except approach is more readable when trying to chain attribute lookups (like in the database book-publisher-owner example I have provided before).
I think that a code where those three qualities happen all at once is in fact a bad code design. But the truth is that None has been and will be used to denote each of those qualities when writing a code because of its ease of use in a normal workflow, where time constraints are tight.
I remember using inhouse solution like this at a certain workplace - a method accepting list of string arguments and an object, returning the value being the result of chained attribute access. And it worked all right. The problem I have with such approaches is that the name of the attrs are passed as strings. This makes it less practical in day-to-day situations when writing code. Automated class refactors, available in most of the IDEs also seem to not be able to support such field renames, whereas attribute lookup via `.` is properly detected. The same goes with context help available in most IDEs - you'll get no useful information when writing a string (IDE does not know if you are trying to write a name of the class' field), whereas when using the dot this works. And it'll probably be implemented for maybe-dot operator in no time looking at other languages' support.

On Wed, Oct 20, 2021 at 1:16 AM Piotr Waszkiewicz <waszka23@gmail.com> wrote:
I'd echo the others' examples, taking inspiration from PEP 463.
I understand the preference for attributes over strings, but many of the none-aware examples use keys and indices. If JSON is the main culprit for deeply nested structures, then you're already using strings and not attributes. Adding features to `operator` wouldn't preclude accepting PEP 505, so why not get started with a less controversial change that provides much of the value? If PEP 505 is accepted, it would need support in the `operator` module. Might as well design that aspect of the implementation now.

Hi, On Wed, Oct 20, 2021 at 5:44 PM Michael Selik <mike@quantami.com> wrote:
Do you think about something along those lines? ``` phone = book.publisher.owner.phone except AttributeError: None ``` I don't mind this syntax but it would have to be supported by static type checkers and IDEs. And currently something like this is not: ``` try: phone = book.publisher.owner.phone except AttributeError: phone = None ``` mypy complains: ``` error: Item "None" of "Optional[Publisher]" has no attribute "owner" ```
I have nothing against introducing such a new feature to the `operator` apart from this one problem mentioned before (using strings which are not properly detected by IDE), and I agree that could be a good start. I've seen chained-attributes-lookups solutions in quite a few places already and I think that there would actually be people benefiting from such addition. Although I must admit that personally I don't see many benefits of using strings for attribute lookups due to typing and IDE issues mentioned before. Even for JSON data, in my own projects I tend to write dataclasses wrapping parsed dict in order to benefit from IDE tooltips.
If PEP 505 is accepted, it would need support in the `operator` module. Might as well design that aspect of the implementation now.
I'm sorry but I don't know if I understand that sentence correctly. You mean we would have to add an "explicit" function that behaves like a maybe-dot operator? Is it actually a requirement when adding new operators?

On Wed, Oct 20, 2021 at 9:18 AM Piotr Waszkiewicz <waszka23@gmail.com> wrote:
Yes, that seems reasonable.
That sounds like a feature request for mypy. Would creating a new operator make it easier to implement analysis of that situation would mypy? My guess is not. Checking the AST to see if there's a try/except AttributeError sounds comparable to checking for the use of a none-aware operator. I'm completely ignorant of how mypy does its analysis, so that's just a wild guess. If PEP 505 is accepted, it would need support in the `operator` module.
The documentation of the `operator` module says, "The operator module exports a set of efficient functions corresponding to the intrinsic operators of Python." It feels like there's an implicit "all" in there. The table of correspondences looks exhaustive. I haven't noticed any exceptions. https://docs.python.org/3/library/operator.html#mapping-operators-to-functio...

On Wed, Oct 20, 2021 at 9:39 PM Michael Selik <mike@quantami.com> wrote:
Nice, I think I would also be able to get used to that notation. It's good to know that there are others supporting the PEP 463, maybe it'd be possible to propose it once again.
I'm not sure, just wanting to point out that the `AttributeError` syntax is not completely equivalent to the maybe-dot operator here. This example would actually hide a real AttributeError problem, e.g. if the `publisher` didn't have an owner field. So I guess there may be a need to introduce a new exception type, e.g. `NoneAttributeAccess` before mypy can safely allow any attribute access in such situation.
Thank you very much, I wasn't aware of that module before. Will look into that. I don't want to prolong this conversation too much, as I feel like I get your point and agree with it to some (rather great) extent. That doesn't change my attitude towards this PEP 505 proposal though, as I feel that if the general consensus would be towards accepting this change it will bring some quality of life improvements in a usual day-to-day work, when dealing with not-so-ideal code. I'd be also interested in seeing PEP 463 being resurrected as it looks like there are some folks here interested in restarting the discussion about it. Thank you very much for the fruitful discussion and broadening my knowledge.

On Wed, Oct 20, 2021 at 06:17:59PM +0200, Piotr Waszkiewicz wrote:
This is not equivalent to PEP 505's None-aware operators. The semantics are very different, and it is much less safe. If you misspell an attribute: book.publisher.onwer = Owner(...) then the `except AttributeError` code will silently hide that error and return None. PEP 505 does not do that. If you use the wrong type: book.publisher = "O'Reilly Books" then the `except` version will silently hide the error and return None. PEP 505 does not. Versions of this that rely on catching AttributeError are simply wrong and are an anti-pattern. They catch too much and silently turn errors into silent wrong behaviour. PEP 505 does not fall into that trap. -- Steve

Hello, Le 21/10/2021 à 07:59, Steven D'Aprano a écrit :
This is not true as a general rule: the PEP 505 examples with `dict.get()` do catch too much. Also, if `None` is not special, you are free to make up your own `NoValue` sentinel object, which can raise specific exceptions on attribute access, item access, etc: class NoValueError(Exception): pass class NoValueAttributeError(AttributeError, NoValueError): pass and so on… So this particular point seems solvable. Cheers, Baptiste

On Thu, Oct 21, 2021 at 10:49:35AM +0200, Baptiste Carvello wrote:
The problem there is not the None-aware operators, but the use of dict.get. That's a good reason to re-think None-aware subscripting. dict?['key'] will still raise if you mistype the key, while dict.get does not. Even if we limit ourselves to dict.get: return obj?.get('spam')?.get('eggs') doesn't protect against mispellings of the keys, *due to dict.get*, but it does protect against mispelling "get" (unlikely). More importantly it also protects against type errors: obj['spam'] = {'eggs', value} # oops, a set, not a dict or equivalent. (I don't mean to limit this to just typos.) Now if we write: obj?.get('spam')?.get('eggs') the first attribute lookup will return a set and the second will fail because sets don't have a get method. Where as if we use exceptions: try: obj['spam]['eggs'] except (TypeError, KeyError): return None the error is silently suppressed and we get None when we should get a TypeError. -- Steve

On Tue, Oct 19, 2021 at 05:09:42PM -0700, Michael Selik wrote:
Does it matter if different functions have different semantic interpretations for None?
Yes, and after something like 30-40 years of IEEE-754 supporting NAN payloads, the number of systems that actually use them can probably be counted on the fingers of one hand :-( Ironically, one of those systems is R, which -- so I have been lead to believe -- uses distict NANs to represent those 27 tagged NA values. Back in the 1980s, one of the earliest systems which supported IEEE-754 maths was the Apple Numeric Toolkit. Apple's maths routines generated NANs with documented payloads for certain errors, e.g: * NAN(1) invalid sqrt * NAN(2) invalid addition such as INF + -INF * NAN(34) invalid argument to inverse trig functions In a complex computation, it was sometimes useful to see why a NAN was generated. Alas, when Apple moved their maths routines into hardware, the MC68881 coprocessor always generated NANs with payload 255, and that useful debugging information was lost. 30+ years later, and we cannot easily, reliably or portably use NAN payloads. Most people don't care. If we offerred them a dozen or a thousand distinct sentinels for all the various kinds of missing data, how many people would use them and how many would just stick to plain old None?
I agree that this is a nice way forward, and a useful function in its own right. The only thing is that I would argue for a different colour of the bikeshed: def getattr_chain(obj, *attrs, default): # like obj.a.b.c.d # if any attribute is missing, # raises if default is not given # otherwise returns default getattr is far more commonly used than attrgetter. -- Steve

On Thu, Oct 21, 2021 at 2:52 AM Steven D'Aprano <steve@pearwood.info> wrote:
In data science, I have been frustrated by the sparsity of ways of spelling "missing value." Besides the distinction Michael points out, and that Steven did in relation to NaNs with payloads, I encounter missingness of various other sorts as well. Crucially, an important kind of missing data is data where the value I received seems unreliable and I have decided to *impute* missingness rather than accept a value I believe is unreliable. But there is also something akin to what Michael points out (maybe it's just an example). For example, "middle name" is something that some people simply do not have, other people choose not to provide on a survey, and others still we just don't know anything beyond "it's not there." Of course, when I impute missingness, I can do so at various stages of data cleaning, and for various different reasons or confidences. None (or NaN) are sort of OK, but carrying metadata as to the nature of missingness would be nice. So my strawman suggestion is tagging None's. I suppose spellings like `None[reason]` or `None(reason)` are appealing. An obvious problem that I recognize is that it's not obvious this can "play nice" with the common idiom `if mydata is not None: ...`. None really is a singleton, and a "tagged singleton" or "annotated singleton" probably doesn't work well with Python's object model. My goal, of course, would be to have TaggedNone be a kind of subclass of None, in the same way that bool is a subclass of int, and hence True is a kind of 1. However, I'd want a large number of custom None's, with some sort of accessible string or numeric code or something to inspect which one it was. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.

On Fri, Oct 22, 2021 at 3:23 AM David Mertz, Ph.D. <david.mertz@gmail.com> wrote:
Might be worth redirecting this to -ideas.
Besides the distinction Michael points out, and that Steven did in relation to NaNs with payloads, I encounter missingness of various other sorts as well. Crucially, an important kind of missing data is data where the value I received seems unreliable and I have decided to *impute* missingness rather than accept a value I believe is unreliable.
But there is also something akin to what Michael points out (maybe it's just an example). For example, "middle name" is something that some people simply do not have, other people choose not to provide on a survey, and others still we just don't know anything beyond "it's not there."
And some people have more than one (I have a brother with two of them). Not the best example to use, since names have WAY more complexities than different types of absence, but there are other cases where that sort of thing comes up. For instance, if someone says on a survey that s/he is in Australia, and then you ask for a postcode, then leaving it blank should be recorded as "chose not to provide"; but if the country is listed as Timor-Leste / East Timor, then "not applicable" would be appropriate, since the country doesn't use postal codes.
Of course, when I impute missingness, I can do so at various stages of data cleaning, and for various different reasons or confidences. None (or NaN) are sort of OK, but carrying metadata as to the nature of missingness would be nice.
Right. Using postcodes as an example again, for someone in Australia, a postcode of "E3B 0H8" doesn't make sense, as that isn't the format we use. So you could wipe that out and replace it with "No postal code, malformed data entered".
But this is where I start to disagree. None should remain a singleton, but "no data available" could be its own thing, tied in with the way that you do your data storage and stats. As such, you wouldn't be checking it with 'is', so you wouldn't have that problem (the Python 'is' operator will only ever test for actual object identity). Keep None simple and dependable, and then "Missing Data" can be an entire class of values if you so desire. ChrisA

El jue, 21 oct 2021 a las 10:25, David Mertz, Ph.D. (<david.mertz@gmail.com>) escribió:
class MissingReason(enum.Enum): no_middle_name = 1 not_provided = 2 unknown = 3 middle_name: str | MissingReason This gives you more structured and precise data than None-plus-string-tag and avoids breaking the ecosystem by changing the meaning and behavior of None.

On Thu, Oct 21, 2021 at 01:46:27PM +1100, Steven D'Aprano wrote:
On Tue, Oct 19, 2021 at 05:09:42PM -0700, Michael Selik wrote:
On further thought, I no longer agree. Or at least, I think we need to think a lot harder about the API before adding any sort of chained attribute getter into the stdlib. If we get it wrong, we'll be stuck with it until Python 5000. The problem is that any sort of code equivalent to: try: return obj.chain.of.attribute.lookups except AttributeError: return None risks silently hiding genuine coding errors. This seems to be an anti-pattern, or at least a foot-gun. And it is certainly not equivalent to, or a replacement for, PEP 505. Same applies to variants similar to attrgetter. -- Steve

Baptiste Carvello wrote:
y = config["handler"]["parameters"]["y"] with KeyError as None
I love the look of this! While it doesn't address everything that PEP505 does, that's IMO a good thing, because - as other people mentioned already - None does too many things already. On another note, the whole discussion reminds me of wg21.link/p0798, which is about adding something quite similar to C++ (was accepted into C++23), and contains a decent overview of how other languages solve the same thing. Even though the approach there is probably not applicable (as all python types are implicitly `Optional` in the sense of "can be None", i.e. that API would have to be added all the way down to `object`), it's still ironic that C++'s `and_then` looks more pythonic than what's proposed here.

This is very reminiscent of the (rejected) PEP 463, Exception-catching expressions (which I still hope will be resurrected some day). It would allow you to write y = (config["handler"]["parameters"]["y"] except KeyError: None) (possibly the parentheses might not be required) which IMO is even better; `except` makes the intent clearer than `with`. Best wishes Rob Cliffe On 20/10/2021 03:05, h.vetinari@gmx.com wrote:

Data point: I find all the examples in PEP 505 less readable using the proposed new operators. Trying to explain why: The syntax feels *too* compact (Perl-like?) - when reading it, every time you see a None-aware operator (*if* you notice it), you have to jerk to a halt and say, "Whoa! What's going on here?". I have some sympathy with this use case of exploring a nested-dict (JSON derived?) structure, but this can be written as try: config.get("handler").get("parameters").get("y") except Attribute Error: # Handle missing value. or something similar (I haven't tested it). Even using the new operators, in a realistic case you would probably have to test if the result is None and take different action accordingly. Best wishes Rob Cliffe On 18/10/2021 19:26, Guido van Rossum wrote:

Big +1 from me. I've been looking forward to having None-aware operators in Python as I find them a very neat language addition. For me personally the main advantage of having maybe-dot (?.) operator is the ability to express certain logic in a much clearer way, for example:
Imagine wanting to get the phone number of the person that published a certain book from the database. In this situation, with maybe-dot operator I can write:
phone_number = book.publisher?.owner?.phone
Getting None value could mean that the book has not been published yet (no publisher relation), owner does not exist or does not have a phone number associated with it. Either way it doesn't matter because the only thing that we wanted to retrieve is the phone number and not the whole context of why it has a certain value and not the other. Personally I find this syntax much more readable than other, "more explicit" expressions like:
phone_number = book.publisher.owner.phone if (book.publisher is not None and book.publisher.owner is not None) else None
Best regards, On Thu, Oct 14, 2021 at 7:37 PM Doug Swarin <dswarin@gmail.com> wrote:

Okay, I'll let myself get sucked into responding ONE TIME, but only because you gave me such a nice API to work with :) On 10/18/2021 9:11 PM, Piotr Waszkiewicz wrote:
Consider today, you wrote this as "book.publisher.owner.phone". You would potentially get AttributeError, from any one of the elements - no way to tell which, and no way to react. Generally, AttributeError indicates that you've provided a value to an API which doesn't fit its pattern. In other words, it's an error about the *type* rather than the value. But in this case, the (semantic, not implementation) *type* is known and correct - it's a publisher! It just happens that the API designed it such that when the *value* is unknown, the *type* no longer matches. This is PRECISELY the kind of (IMHO, bad) API design that None-aware operators will encourage. Consider an alternative: class ForeignKey: ... def __bool__(self): return not self.is_dbnull def value(self): if self.is_dbnull: return self.Type.empty() # that is, DBModel.empty() return self._value class DBModel: @classmethod def empty(cls): return cls(__secret_is_empty_flag=True) def __bool__(self): return not self._is_empty def __getattr__(self, key): if not self: t = self._get_model_type(key) return t.empty() if isinstance(t, DBModel) else None ... class User(DBModel): phone: str | None class Publisher(DBModel): owner: ForeignKey[User] class Book(DBModel) publisher: ForeignKey[Publisher] Okay, so as the API implementer, I've had to do a tonne more work. That's fine - *that's my job*. The user hasn't had to stick "| None" everywhere (and when we eventually get around to allowing named arguments in indexing then they could use "ForeignKey[User, non_nullable=True]", but I guess for now that would be some subclass of ForeignKey). But now here's the example again:
book.publisher.owner.phone
If there is no publisher, it'll return None. If there is no owner, it'll return None. If the owner has no phone number, it'll return None. BUT, if you misspell "owner", it will raise AttributeError, because you referenced something that is not part of the *type*. And that error will be raised EVERY time, not just in the cases where 'publisher' is non-null. It takes away the random value-based errors we've come to love from poorly coded web sites and makes them reliably based on the value's type (and doesn't even require a type checker ;) ). Additionally, if you want to explicitly check whether a FK is null, you can do everything with regular checks: if book.publisher.owner: # we know the owner! else: # we don't know # Get all owner names - including where the name is None - but only if # Mrs. None actually published a book (and not just because we don't # know a book's publisher or a publisher's owner) owners = {book.id: book.publisher.owner.name for book in all_books if book.publisher.owner} # Update a null FK with a lazy lookup book.publisher = book.publisher or publishers.get(...) You can't do anything useful with a native None here besides test for it, and there are better ways to do that test. So None is not a useful value compared to a rich DBModel subclass that *knows* it is empty. --- So to summarise my core concern - allowing an API designer to "just use None" is a cop out, and it lets people write lazy/bad APIs rather than coming up with good ones. Cheers, Steve

Thank you very much for this exhaustive explanation and example. I really like it and agree with you that the implementation provided by your example is much more well designed. The problem I have with it is that I feel like it assumes that I have a way to introduce such changes when writing the API, whereas I was talking more about using existing solutions. The provided example was based on common situations encountered when writing the code in Django. It may be that I'm using this tool not to its full potential, but it seems like many of your points were about bad API design, not the actual maybe-dot operator. Such operator would make it much easier to work with those frameworks that do not allow for more readable/well thought-out solutions. I'm thinking about it as a syntactic sugar that can speed things up (and even make it more readable) in a situation where there is no time and/or budget to come up with a better architecture (like e.g. the one you have provided). The reality is that usually from the business perspective nobody wants a well designed system, only the functioning one. It may be that I misunderstood your response and didn't provide a valid answer - please let me know in that case. Best regards On Tue, Oct 19, 2021 at 1:29 AM Steve Dower <steve.dower@python.org> wrote:

I will also say that I don't believe the safe-navigation operators necessarily compromise type safety. PEP 505 explicitly rejects having them catch `AttributeError` or `KeyError` (and I agree with this rejection). It's not the default behavior of objects to return None when an unknown attribute is read, so attempting to access `book?.publisher?.onwer?.name` will still fail with `AttributeError`. Type checkers would also continue being able to check such navigation. Doug

Steve Dower wrote:
Okay, I'll let myself get sucked into responding ONE TIME, but only because you gave me such a nice API to work with :)
This actually pushed me hard towards adding the null-aware operators. I agree that the named-function approach Paul suggests is better. I admit that there are times when a comprehensive model is good, and that if the DB schema is automatically generated, it should look something like this. But this is long enough that I would be unhappy if it were what I had to read to understand the database in the first place. (Yes, I know Java beans are pretty widely tolerated, but ... that doesn't make longer code desirable.)

Most of the discussion so far has been focused on (?.). Tbh though, I'm more interested in (??) and (??=). Just reading through code, I constantly notice boilerplate like this which could easily be substituted. variable = some_function(...) if variable is None: variable = [] # some default value # a bit better with an assignment expression if (variable := some_function(...)) is None: variable = [] # or worse with an if expression variable = some_function(...) if some_function(...) else [] # also possible with :=, but not much better variable = x if (x := some_function(...)) else [] # using the coalesce operator would be much more readable IMO variable = some_function(...) ?? [] If (?.) and (?[) are rejected / deferred, maybe there is interest in seeing at least (??) and (??=) through?

On Sat, Oct 23, 2021 at 6:20 AM Marc Mueller <cdce8p@gmail.com> wrote:
Bear in mind that these last ones are exactly equivalent to the "or" operator, as they'll use the default if you have any falsy value. variable = some_function(...) or []
I'm actually more interested in a better idiom for non-constant function default arguments, since that's the place where this kind of thing often comes up. A nice ??= operator might help if your default is None, but if you then change the default to be object(), you can't use ??= any more. As a bonus, the docs for such an argument could actually say what the default really is: def bisect_right(a, x, lo=0, hi=len(a), *, key=None): ... except that it'd need some adornment to say that it's late-bound. ChrisA

Isn't that in itself a good argument in favor of (??) ? By missing to add 'is None', I would have already added a subtle bug that could be quite difficult to find. (??) could prevent that while also being more readable.
True, but from my experience 'None' is just by far the most common default. Why not improve how we handle it?

Marc Mueller writes:
True, but from my experience 'None' is just by far the most common default. Why not improve how we handle it?
The question is whether this is an improvement in the long run. When some falsies are expected, in-range values, "if arg is None: ..." or "x = default if arg is None else arg" is sufficiently precise and concise. I don't see "arg ?? default" as an improvement worthy of syntax. One the minus side, as David Mertz testifies, there is a maze of missing value use cases, all alike (or is that a missing maze of values, all alike?) These operators would further encourage conflating them all into None. At least in theory, that's not an improvement in handling None, that's a loss of precision in handling an important subset of Nones. I just don't see much value in ??. I don't have an opinion on the ?. and ?[] versions. I can see that they could make chained operations much more concise than nested if ... else expressions, and at least somewhat more precise (at least if you want to avoid deep nesting of try ... except KeyError statements, which I suppose most everybody would like to avoid). Steve

On Sun, Oct 24, 2021 at 12:35 AM Marc Mueller <cdce8p@gmail.com> wrote:
Perhaps, but your alternate versions were identically buggy, which means that - in all probability - the bug wouldn't have been relevant. For instance, if you're asking for a file-like object, and will use stdout if no argument is passed, it's absolutely fine to use None as a default and "or" to replace it with sys.stdout, because only a pathological example will actually cause problems. And that's fine. There are myriad use-cases for a None-coalescing operator that would work just fine without one; if Python had such an operator, sure, it could be used, but they're not strong arguments for adding one.
There are three situations to consider here: 1) Situations where there's no falsy value that would make sense 2) Situations where None is used and there are other falsy values that would make sense 3) Situations where any object makes sense The only ones where None-coalescing is truly significant are the second group. In the first group, "or" and "??" would behave identically, so why bother introducing a new operator? And in the third group, the default can't be None, so it has to be an artificially-invented object. All three would be improved by a late-binding feature, but only a strict subset of them would benefit from directly replacing None with something else. Like several others here, I am much more interested in the operators that do "if None, give back None, otherwise fetch an attribute/item". If we end up getting None-coalescing for free along with that, so be it, but I'm not really pushing for that part. Proper argument late-binding is an entirely orthogonal proposal, and would have many benefits that aren't part of PEP 505 or related proposals, but would do most of what we want from the ?? operator. PEP 505 has some oddities that are hard to explain, though - and I mean that not in the sense that the PEP itself is hard to read, but that I would have to go to some effort to explain what my code does (picture a mentor/student relationship here). If I offer a line of code saying <<x = spam()?.ham.eggs or 5>>, what happens if spam returns None? Unpacking it step by step reveals a bit more complexity than it first seems to have. Python tries to minimize such situations, doing them only when there is a very clear benefit (for instance, "3 < x < 8" is very different from "(3 < x) < 8", but that's because it is immensely valuable), and I do have misgivings about that part of it. Otherwise, though, I am in favour of this and would make good use of it. (And would do so regardless of the ultimate decision on the chaining.) ChrisA

Hi, I wanted to propose replacing ? with -> in the none aware syntax. This makes the expression look like a chain, and I think that most people intuitively can understand how the chain might get broken (and what would happen in that case). For example: zipcode = person->.address->['zipcode'] I've proposed this alternative syntax on Doug's github project, and I think he didn't like it so much (or at least, his co-contributor didn't like it), but I still wanted to propose it here as well, because I do think it's a good solution to the complaints that the ? operator looks cryptic. Best regards, Maarten

Hi Maarten, I like the suggestion but I'm not sure if the real problem with the PEP505 is the symbol/operator used. Reading through the whole discussion I'm under the impression that the idea of the None value being treated as an indicator of the missing attribute is what prevents this PEP from happening. Apart from that I'm all for it, (probably) regardless of the syntax (as long as it remains short). Best regards, Piotr On Thu, Sep 15, 2022 at 5:07 PM Maarten Nieber <mnieber@gmail.com> wrote:

Hi Piotr, doesn't Doug's reply of 8:03 address this point? As he says, the none-aware operator never gives you None when you ask for a missing attribute (these cases always raise an exception). If we look at these two alternatives phone1 = book.publisher?.owner.phone phone2 = book.publisher.owner.phone if book.publisher else None then they behave exactly the same. If we would misspell "owner" then in both versions we'd get the same AttributeError under the same conditions. Best, Maarten

Hi Maarten, I'm sorry for the confusion - it was bad wording on my part. What I really meant was that the problem with the None-aware operator, and the reason why PEP505 has not been accepted for such a long time, is that there's no consensus regarding the need for it (and not necessarily the problem with the operator used to represent it - although this topic has also been raised). Take a look at the response by Steve Dower: https://mail.python.org/archives/list/python-dev@python.org/message/BRTRKGY6... Best regards, Piotr On Thu, Sep 15, 2022 at 6:54 PM Maarten Nieber <mnieber@gmail.com> wrote:

So to summarise my core concern - allowing an API designer to "just use None" is a cop out, and it lets people write lazy/bad APIs rather
Hi all, I've only here found out that there is a discussion going on about those none-aware operators and my first thought was "great, finally!". FWIW, I'd be happy with the syntax suggestion in the PEP, since '?' looks rather intuitive to me to mean something like "maybe". However, I then read the mentioned post of Steve Dower, with the final summary: than coming up with good ones. This is a very good point. In fact, I've never really thought about it that way and of course he's totally right that "SomeType | None" (or Optional[SomeType], which also somehow made me feel that this usage is fairly intended) is not optimal, at least for user defined types/classes. The problem is, that I never actually thought about his suggested way. And I wouldn't be surprised if this holds for many other people as well. Maybe it would be great to boldly mention these thoughts in the documentation at an appropriate place. In my opinion, there are at least the following good places where this would fit nicely: - The documentation of the dataclasses (https://docs.python.org/3/library/dataclasses.html), since this is probably the most common use case for the "| None" pattern. Going further, the dataclasses functionality might even be extended to make it simpler to generate such null-types (or however they are called), so that it is no longer "a tonne more work". - Linters like pylint could emit a note when seeing the "| None" pattern, linking to the explanation about why it is possibly not the best way to do it. - The documentation of the discussed None-aware operators. Since these new operators are closely coupled to the arguably suboptimal "| None" pattern, it is probably good to tell folks right there why they should consider better alternatives. As mentioned, I absolutely see Steve's point. However, there are many Python scripts/programs working without a complex API, where this "| None" pattern may still have its legitimate uses and the none-aware operators can make code easier to read (and write). Best regards, Philipp

Personally I think returning None is a fine API design, and IMO the concerns about this pattern are overblown. Note that X|None is no different than the "Maybe X" pattern that functional programmers are so fond of. On Mon, Sep 19, 2022 at 8:02 AM Philipp Burch <phip@hb9etc.ch> wrote:
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>

On 19. 09. 22 17:58, Guido van Rossum wrote:
I must disagree here. With `X|None` there is no way to do `Maybe[Maybe[X]]`, and FP is all about proper composition. One practical consequence is that "Maybe X" doesn't have the problem that PEP 661 (Sentinel Values) tries to solve.

Hi Philipp, That's a really good idea, and I'd really want to see it being implemented. Having said that, I wonder how many people would actually use this concept of "none-representing" objects. In my 10+ years of programming experience I've never seen anybody eager to use more complex structures or object representations of such concepts outside of an academic environment (and even there usually when dealing with personal or proof-of-concept projects). Nowadays I write mostly more business-oriented code, and I haven't been successful in finding a workplace where "business" wants to pay for preparing "good looking" code - they want a functioning one. That's where this whole concept of using none values is used the most (at least from my experience). So to reiterate: I think that the idea presented by Steve is very appealing, and I'd really like to work with such projects, but the reality is that it's really hard to stumble upon such. And I would really much *prefer to have none-aware operators that would make my life easier working with "real world" projects that rely too heavily on none values*, even if it's academically not correct. On Mon, Sep 19, 2022, 17:06 Philipp Burch <phip@hb9etc.ch> wrote:

On Sun, Sep 18, 2022 at 09:43:26PM +0200, Philipp Burch wrote:
I don't think that `SomeType|None` is necessarily a lazy or bad API. Whether it is "optimal" or not will depend on the specific SomeType involved, not to mention other tradeoffs and constraints such as time and money available. (Oh, to have the luxury of spending two months to "Do It Right" for a project needed in a week on a budget...) But what's not "optimal" is expecting everyone who writes a class and needs to represent the lack of an instance to duplicate those None-like semantics within the class. The distinction you make between user-defined and non-user-defined classes doesn't hold water. If you allow that (say) `int|None` **might** be acceptable, then why would `Integer|None` **necessarily** be lazy and bad just because int is written in C and built into the intepreter while Integer is written in Python by a user? The laziness/badness of `SomeType|None` must depend on the semantics of SomeType, not the implementation language or who wrote it. "Our API was lazy and bad because it uses None, but then it got accepted into Python as a builtin, so now it is no longer lazy or bad." /s Of course people can write None-like instances of their SomeType classes, if it makes sense for their application. But the idea that any and every use (or even a majority) of Optional types is a "lazy/bad API" is, I think, unrealistic puritism, not to mention unfairly disparaging towards the programmers who write those APIs.
The problem is, that I never actually thought about his suggested way.
Some people have, and found that it doesn't always simplify the API that much, or at all. If you end up replacing `obj is None` with `obj == NULL` or `obj.isnull()` then you haven't really gained much. -- Steven

Hi Steven! On 21.09.22 13:17, Steven D'Aprano wrote:
I considered a while if "user-defined types/classes" is the right wording, but didn't find a better alternative. The point I'm making is that `X|None` could especially then hide bugs if X defines attributes that will often be accessed in a chain. For int or float, there are only a few methods, which are arguably rarely used and chaining is probably not used at all. Plus, providing nulltypes for primitives would feel very awkward. Consider ``` x: int | None = 5 # ... if x is not None: y = 1 << x.bit_cont() ``` The typo with bit_cont() is hidden whenever x is None, so it might be missed by unit tests with limited coverage. However, using a nullable variant of it would probably make it look like this: ``` class Int(int): # ... class NoneInt(Int): # ... x = Int(5) # With the option for NoneInt() # ... x_bit_count = x.bit_cont() if x_bit_count == NoneInt: y = 1 << x_bit_count ``` This is certainly not pretty for something basic as an int, plus there will most likely be a performance hit when replacing every int by Int(), not to mention compatibility issues with APIs/libraries expecting plain int. On the other hand, more complex classes like in the mentioned publisher API could indeed profit from such null-types, since a chain like ``` book?.publisher?.owner?.namme ``` will hide the typo with namme whenever any of book, publisher or owner is None, so there is much more potential here to miss it in a unit test. This is actually the distinction that I wanted to make, not in the technical sense if a class is defined in the standard library or not.
Certainly. But without None-aware operators, the gain in readability can be large, if you can just write ``` name = book.publisher.owner.name if name is not None: # No NoneStr here, see above # ... ``` versus ``` if (book is not None) and (book.publisher is not None) and ...: # ... ``` Just to be clear: I've been thankful for the comment and explanation of Steve Dower, because I've never thought about it that way and now see constructs in my code, where such a concept really would have simplified things and improved robustness. For others to profit as well, I would be glad to see this mentioned in the docs, so that it can be found even when not actually looking for it. But I certainly don't want to say that this concept must be followed everywhere or that "X|None" is necessarily a bad thing. After all, I'd still love to see Python supporting such None-aware operators. Best regards, Philipp

Thanks -- this is the kind of work that helps a PEP get accepted. I am personally in favor of accepting PEP 505, and I hope that your work and the discussion that will undoubtedly follow here will help convince the Steering Council to accept it. --Guido On Thu, Oct 14, 2021 at 10:38 AM Doug Swarin <dswarin@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>

I tried to implement this in CPython by modifying a downloaded source code, but I can't seem to fix the problem of the "maybe" operators segfaulting when being used with literal immutables. The maybe-assign/coalesce operators were implemented successfully though.

Hello Doug, On Thu, Oct 14, 2021 at 03:45:07PM -0000, Doug Swarin wrote:
I believe strong and valid arguments can be made about the use of None being a fundamental flaw in some types of coding
Can you elaborate on that? Obviously it is not always appropriate to use None, but I've never seen it called a *fundamental* flaw. I know that the null pointer has been called a billion-dollar mistake, but Python's None is not a null pointer. -- Steve

Steven D'Aprano wrote:
I apologize that I may have spoken too strongly here. When I emailed Mr. Dower, he mentioned that he now believes the implementation of these operators would lead people to a style of coding which would lead to the proliferation of None as an exception-less error result and also throughout data structures. My understanding is that his current preference is to focus on functional composition and styles of programming that disallow the use of None. I certainly don't mean to speak for him and I hope he will weigh in with a more detailed explanation of his thoughts and objections, but I personally disagree as a matter of pure practicality. It's just plain useful to be able to easily take non-values and safely deal with them without having to constantly check for None or to catch and inspect exceptions to see if it's a case that can be ignored. I did indeed think about connecting None to the 'billion dollar mistake' but decided against it since as you say None is not a null pointer, and I should have chosen my words a little more carefully when revising my initial post (likely by removing the word 'fundamental'). Doug

Thanks for allowing me to speak for myself, as I said I would when contacted off list :) But as it happens, you've summarised my position very accurately:
he now believes the implementation of these operators would lead people to a style of coding which would lead to the proliferation of None as an exception-less error result and also throughout data structures. My understanding is that his current preference is to focus on functional composition and styles of programming that disallow the use of None.
As additional context, C# and the .NET Framework have just gone through the exercise of making (and propagating) non-nullable reference types (and their "null" is far more like our None than C's NULL). Our type checkers are also pushing in this direction by requiring Optional[]. The argument in all cases is that it allows for simpler code that can safely ignore non-values because they cannot be passed. Adding ??, ?. and ?[, particularly in their short-circuiting evaluation form, encourages more complex code by adding value-based branching within an expression. And it encourages it on the "outside" of the API, not within it. Which means API designers can more easily justify returning None because their caller can use None-aware operators to basically ignore it. (I would argue that the API designer should work harder to create an API that doesn't ever have to return None.) To be clear, I'm thinking very much in terms of "impact on what people will consider to be Pythonic API design over the next 10 years", that is, what people think they *should* do with this, rather than simply what they *could*. So it's all very hypothetical and I have no data for it, much like when it was decided that Optional[] would be mandatory on types where None is permitted. And with that, I'm bowing out of the fresh discussion (unless the SC wants to discuss it directly with me). I'll continue advocating for tools/features that help people create better APIs, but I'm not going to spend a lot of time arguing against those that I believe will not. Cheers, Steve On 10/15/2021 3:08 AM, Doug Swarin wrote:

NoneType is just another type, and in type checking scenarios should be expressed with `Optional[type]` or more preferably in the future `type | None`; `None` is not a non-value. Assuming what I just wrote is true, I don't get what the basis of this thread is; what am I missing? On Mon, 2021-10-18 at 14:13 +0100, Steve Dower wrote:

On Mon, Oct 18, 2021 at 9:35 AM Paul Bryan <pbryan@anode.ca> wrote:
To me the thread isn't about type checking. It is about APIs that are built into the language that special-case None, in particular dict.get(). In certain cases, encountered commonly in certain styles of coding (data science? web programming?), users encounter data that is structured as dicts nested multiple levels. This is often out of the user's control, as the dict is returned by reading a JSON value whose structure is controlled by some other framework (often not specific to Python). For example, if we have a config structure like this: config = { "timeout": 0.1, "handler: { "timeout-override": 0.4, "method-name": "plot", "parameters": { "x": 10, "y": "auto", } } } where the convention is that keys at any level may be omitted altogether and config itself may be NOne, then to safely access the value of config["handler"]["parameters"]["y"] we would have to write y = None # Default if config is not None: handler = config.get("handler") if handler is not None: parameters = handler.get("parameters") if parameters is not None: y = parameters.get("y") This kind of pattern (and all the various other ways of writing it, e.g. using the walrus or passing {} as the second argument to dict.get()) can be *very* common if that's the kind of data you're given and that's the kind of app you have to write, and you can't control the format of the data. Using ?. this can be written as y = config?.get("handler")?.get("parameters")?.get("y") More examples are in PEP 505 itself, see https://www.python.org/dev/peps/pep-0505/#examples -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>

On Mon, 18 Oct 2021 at 19:29, Guido van Rossum <guido@python.org> wrote:
For this particular usage, I'd much rather have a functional API, like y = get_config(config, "handler", "parameters", "y") I understand that writing many helpers like that nested_get is a chore, and having language support for the operation avoids that chore - but I'm with Steve in not wanting to see the ?.get() pattern become "idiomatic Python" as it encourages people *not* to design APIs like nested_get that allow the user to not even be aware of all that behind-the-scenes complexity. Sure, you could argue that the ?. operator makes it easier to write something like get_config, but it's not *that* hard: def get_config(config, *keys): value = config for key in keys: if value is None: break value = value.get(key) return value And the problem with the ?.get style is that it doesn't hide anything - what if you want/need to change your config data structure (because the JSON you're reading changes its layout, say)? Without encapsulation, you can't. And if it's *that* common, adding a stdlib function or a new dict method is also an option, which doesn't need a language feature and demonstrates good (IMO) API design for people to copy. Anyway, much like Steve, I don't expect to spend a lot of time fighting this proposal. But I will be sad if it gets accepted, and even more so if ?. becomes idiomatic in user code. Paul

On Mon, Oct 18, 2021 at 6:49 PM Paul Moore <p.f.moore@gmail.com> wrote:
I agree with Paul here... and am pretty sure I did the last time this went around. Whenever the issue comes up, it's about JSON. Or *maybe* about something very similar that goes by another name or format details. And there already exists a pretty good, pretty standard, approach called JSONPath that deals with exactly this kind of thing. This, in turn, is largely the same as XPath which serves the same role for XML documents. I don't think it necessarily needs to be in the standard library, but the mini-language for extracting data from trees with frequently missing branches can very well simply be a mini-language. It'll wind up a lot like XPath/JSONPath, but something a little bit different could be good too. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.

I should have added that I also don't feel I want to go at bat to fight for this PEP. I do observe that it looks like the folks used to building large systems (in Python or other languages) don't seem to like it, while it seems to appeal to folks writing simpler code (the abundant majority of Python users, but not of Python core devs). I worry that the experienced folks may perhaps be a little too eager to protect newbies from shooting themselves in the foot. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>

On Tue, 19 Oct 2021 at 00:55, Guido van Rossum <guido@python.org> wrote:
I should have added that I also don't feel I want to go at bat to fight for this PEP. I do observe that it looks like the folks used to building large systems (in Python or other languages) don't seem to like it, while it seems to appeal to folks writing simpler code (the abundant majority of Python users, but not of Python core devs). I worry that the experienced folks may perhaps be a little too eager to protect newbies from shooting themselves in the foot.
Possibly. But in *my* case, I'm not arguing from the position of a large system builder, but from that of a script writer (albeit an experienced one) and code maintainer (usually of my own code). I find y = config?.get("handler")?.get("parameters")?.get("y") unreadable and confusing, and I'd probably advise strongly against it if someone ever showed me code containing it. I see y = get_config(config, "handler", "parameters", "y") as *far* more readable and expressing the intent more directly. Yes, I find Steve's arguments persuasive, but they are not the ones I'd be concerned with when advising a newcomer. Rather I'd be saying "do you see how using a named function expresses your intent better?" and "do you see how writing a small function hides the messiness of checking for None so that your main code is cleaner?" *Shrug* I guess I just don't understand how people can look at a string of ?.get() and see it as readable and obvious :-( Paul

Le 18/10/2021 à 20:26, Guido van Rossum a écrit :
Sure, but the EAFP version is not that bad: try: y = config["handler"]["parameters"]["y"] except KeyError: y = None which could be further simplified with an exception-catching expression (caveat: keyword usage is pure improvisation, it sounds good, but is probably broken :-) : y = config["handler"]["parameters"]["y"] with KeyError as None The PEP authors would probably reject this as "hiding errors in code", which is true, but none-aware operators also hide errors… Cheers, Baptiste

On Tue, Oct 19, 2021 at 9:39 AM Chris Angelico <rosuav@gmail.com> wrote:
In case it saves anyone a couple clicks: https://www.python.org/dev/peps/pep-0463/ I also prefer more syntactic help with exceptions, rather than more syntax emphasizing None's uniqueness. None and its ilk often conflate too many qualities. For example, is it missing because it doesn't exist, it never existed, or because we never received a value, despite knowing it must exist? The languages SAS and R support at least 27 varieties of NA, allowing un-tagged, and tagged with the letters A-Z to help someone create distinctions between different kinds of nothingness. IEEE-754 allows about 16 million possible NaNs, which I believe was intended to allow floating point instructions to pass error messages along. If the motivation for this operator is chained lookups, how about adding a feature to the operator module, first? It seems natural to add a keyword-only argument to `attrgetter`, and it's a lighter touch than implementing a new operator. If use becomes widespread, that gives more weight to PEP 505. def attrgetter(*attrs, none_aware=False) https://docs.python.org/3/library/operator.html#operator.attrgetter Apologies if attrgetter has already been discussed. I didn't see mention of it in PEP 505.

Hi, On Wed, Oct 20, 2021 at 2:33 AM Michael Selik <mike@quantami.com> wrote:
Me too, but could you provide me an example where try-except approach is more readable when trying to chain attribute lookups (like in the database book-publisher-owner example I have provided before).
I think that a code where those three qualities happen all at once is in fact a bad code design. But the truth is that None has been and will be used to denote each of those qualities when writing a code because of its ease of use in a normal workflow, where time constraints are tight.
I remember using inhouse solution like this at a certain workplace - a method accepting list of string arguments and an object, returning the value being the result of chained attribute access. And it worked all right. The problem I have with such approaches is that the name of the attrs are passed as strings. This makes it less practical in day-to-day situations when writing code. Automated class refactors, available in most of the IDEs also seem to not be able to support such field renames, whereas attribute lookup via `.` is properly detected. The same goes with context help available in most IDEs - you'll get no useful information when writing a string (IDE does not know if you are trying to write a name of the class' field), whereas when using the dot this works. And it'll probably be implemented for maybe-dot operator in no time looking at other languages' support.

On Wed, Oct 20, 2021 at 1:16 AM Piotr Waszkiewicz <waszka23@gmail.com> wrote:
I'd echo the others' examples, taking inspiration from PEP 463.
I understand the preference for attributes over strings, but many of the none-aware examples use keys and indices. If JSON is the main culprit for deeply nested structures, then you're already using strings and not attributes. Adding features to `operator` wouldn't preclude accepting PEP 505, so why not get started with a less controversial change that provides much of the value? If PEP 505 is accepted, it would need support in the `operator` module. Might as well design that aspect of the implementation now.

Hi, On Wed, Oct 20, 2021 at 5:44 PM Michael Selik <mike@quantami.com> wrote:
Do you think about something along those lines? ``` phone = book.publisher.owner.phone except AttributeError: None ``` I don't mind this syntax but it would have to be supported by static type checkers and IDEs. And currently something like this is not: ``` try: phone = book.publisher.owner.phone except AttributeError: phone = None ``` mypy complains: ``` error: Item "None" of "Optional[Publisher]" has no attribute "owner" ```
I have nothing against introducing such a new feature to the `operator` apart from this one problem mentioned before (using strings which are not properly detected by IDE), and I agree that could be a good start. I've seen chained-attributes-lookups solutions in quite a few places already and I think that there would actually be people benefiting from such addition. Although I must admit that personally I don't see many benefits of using strings for attribute lookups due to typing and IDE issues mentioned before. Even for JSON data, in my own projects I tend to write dataclasses wrapping parsed dict in order to benefit from IDE tooltips.
If PEP 505 is accepted, it would need support in the `operator` module. Might as well design that aspect of the implementation now.
I'm sorry but I don't know if I understand that sentence correctly. You mean we would have to add an "explicit" function that behaves like a maybe-dot operator? Is it actually a requirement when adding new operators?

On Wed, Oct 20, 2021 at 9:18 AM Piotr Waszkiewicz <waszka23@gmail.com> wrote:
Yes, that seems reasonable.
That sounds like a feature request for mypy. Would creating a new operator make it easier to implement analysis of that situation would mypy? My guess is not. Checking the AST to see if there's a try/except AttributeError sounds comparable to checking for the use of a none-aware operator. I'm completely ignorant of how mypy does its analysis, so that's just a wild guess. If PEP 505 is accepted, it would need support in the `operator` module.
The documentation of the `operator` module says, "The operator module exports a set of efficient functions corresponding to the intrinsic operators of Python." It feels like there's an implicit "all" in there. The table of correspondences looks exhaustive. I haven't noticed any exceptions. https://docs.python.org/3/library/operator.html#mapping-operators-to-functio...

On Wed, Oct 20, 2021 at 9:39 PM Michael Selik <mike@quantami.com> wrote:
Nice, I think I would also be able to get used to that notation. It's good to know that there are others supporting the PEP 463, maybe it'd be possible to propose it once again.
I'm not sure, just wanting to point out that the `AttributeError` syntax is not completely equivalent to the maybe-dot operator here. This example would actually hide a real AttributeError problem, e.g. if the `publisher` didn't have an owner field. So I guess there may be a need to introduce a new exception type, e.g. `NoneAttributeAccess` before mypy can safely allow any attribute access in such situation.
Thank you very much, I wasn't aware of that module before. Will look into that. I don't want to prolong this conversation too much, as I feel like I get your point and agree with it to some (rather great) extent. That doesn't change my attitude towards this PEP 505 proposal though, as I feel that if the general consensus would be towards accepting this change it will bring some quality of life improvements in a usual day-to-day work, when dealing with not-so-ideal code. I'd be also interested in seeing PEP 463 being resurrected as it looks like there are some folks here interested in restarting the discussion about it. Thank you very much for the fruitful discussion and broadening my knowledge.

On Wed, Oct 20, 2021 at 06:17:59PM +0200, Piotr Waszkiewicz wrote:
This is not equivalent to PEP 505's None-aware operators. The semantics are very different, and it is much less safe. If you misspell an attribute: book.publisher.onwer = Owner(...) then the `except AttributeError` code will silently hide that error and return None. PEP 505 does not do that. If you use the wrong type: book.publisher = "O'Reilly Books" then the `except` version will silently hide the error and return None. PEP 505 does not. Versions of this that rely on catching AttributeError are simply wrong and are an anti-pattern. They catch too much and silently turn errors into silent wrong behaviour. PEP 505 does not fall into that trap. -- Steve

Hello, Le 21/10/2021 à 07:59, Steven D'Aprano a écrit :
This is not true as a general rule: the PEP 505 examples with `dict.get()` do catch too much. Also, if `None` is not special, you are free to make up your own `NoValue` sentinel object, which can raise specific exceptions on attribute access, item access, etc: class NoValueError(Exception): pass class NoValueAttributeError(AttributeError, NoValueError): pass and so on… So this particular point seems solvable. Cheers, Baptiste

On Thu, Oct 21, 2021 at 10:49:35AM +0200, Baptiste Carvello wrote:
The problem there is not the None-aware operators, but the use of dict.get. That's a good reason to re-think None-aware subscripting. dict?['key'] will still raise if you mistype the key, while dict.get does not. Even if we limit ourselves to dict.get: return obj?.get('spam')?.get('eggs') doesn't protect against mispellings of the keys, *due to dict.get*, but it does protect against mispelling "get" (unlikely). More importantly it also protects against type errors: obj['spam'] = {'eggs', value} # oops, a set, not a dict or equivalent. (I don't mean to limit this to just typos.) Now if we write: obj?.get('spam')?.get('eggs') the first attribute lookup will return a set and the second will fail because sets don't have a get method. Where as if we use exceptions: try: obj['spam]['eggs'] except (TypeError, KeyError): return None the error is silently suppressed and we get None when we should get a TypeError. -- Steve

On Tue, Oct 19, 2021 at 05:09:42PM -0700, Michael Selik wrote:
Does it matter if different functions have different semantic interpretations for None?
Yes, and after something like 30-40 years of IEEE-754 supporting NAN payloads, the number of systems that actually use them can probably be counted on the fingers of one hand :-( Ironically, one of those systems is R, which -- so I have been lead to believe -- uses distict NANs to represent those 27 tagged NA values. Back in the 1980s, one of the earliest systems which supported IEEE-754 maths was the Apple Numeric Toolkit. Apple's maths routines generated NANs with documented payloads for certain errors, e.g: * NAN(1) invalid sqrt * NAN(2) invalid addition such as INF + -INF * NAN(34) invalid argument to inverse trig functions In a complex computation, it was sometimes useful to see why a NAN was generated. Alas, when Apple moved their maths routines into hardware, the MC68881 coprocessor always generated NANs with payload 255, and that useful debugging information was lost. 30+ years later, and we cannot easily, reliably or portably use NAN payloads. Most people don't care. If we offerred them a dozen or a thousand distinct sentinels for all the various kinds of missing data, how many people would use them and how many would just stick to plain old None?
I agree that this is a nice way forward, and a useful function in its own right. The only thing is that I would argue for a different colour of the bikeshed: def getattr_chain(obj, *attrs, default): # like obj.a.b.c.d # if any attribute is missing, # raises if default is not given # otherwise returns default getattr is far more commonly used than attrgetter. -- Steve

On Thu, Oct 21, 2021 at 2:52 AM Steven D'Aprano <steve@pearwood.info> wrote:
In data science, I have been frustrated by the sparsity of ways of spelling "missing value." Besides the distinction Michael points out, and that Steven did in relation to NaNs with payloads, I encounter missingness of various other sorts as well. Crucially, an important kind of missing data is data where the value I received seems unreliable and I have decided to *impute* missingness rather than accept a value I believe is unreliable. But there is also something akin to what Michael points out (maybe it's just an example). For example, "middle name" is something that some people simply do not have, other people choose not to provide on a survey, and others still we just don't know anything beyond "it's not there." Of course, when I impute missingness, I can do so at various stages of data cleaning, and for various different reasons or confidences. None (or NaN) are sort of OK, but carrying metadata as to the nature of missingness would be nice. So my strawman suggestion is tagging None's. I suppose spellings like `None[reason]` or `None(reason)` are appealing. An obvious problem that I recognize is that it's not obvious this can "play nice" with the common idiom `if mydata is not None: ...`. None really is a singleton, and a "tagged singleton" or "annotated singleton" probably doesn't work well with Python's object model. My goal, of course, would be to have TaggedNone be a kind of subclass of None, in the same way that bool is a subclass of int, and hence True is a kind of 1. However, I'd want a large number of custom None's, with some sort of accessible string or numeric code or something to inspect which one it was. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.

On Fri, Oct 22, 2021 at 3:23 AM David Mertz, Ph.D. <david.mertz@gmail.com> wrote:
Might be worth redirecting this to -ideas.
Besides the distinction Michael points out, and that Steven did in relation to NaNs with payloads, I encounter missingness of various other sorts as well. Crucially, an important kind of missing data is data where the value I received seems unreliable and I have decided to *impute* missingness rather than accept a value I believe is unreliable.
But there is also something akin to what Michael points out (maybe it's just an example). For example, "middle name" is something that some people simply do not have, other people choose not to provide on a survey, and others still we just don't know anything beyond "it's not there."
And some people have more than one (I have a brother with two of them). Not the best example to use, since names have WAY more complexities than different types of absence, but there are other cases where that sort of thing comes up. For instance, if someone says on a survey that s/he is in Australia, and then you ask for a postcode, then leaving it blank should be recorded as "chose not to provide"; but if the country is listed as Timor-Leste / East Timor, then "not applicable" would be appropriate, since the country doesn't use postal codes.
Of course, when I impute missingness, I can do so at various stages of data cleaning, and for various different reasons or confidences. None (or NaN) are sort of OK, but carrying metadata as to the nature of missingness would be nice.
Right. Using postcodes as an example again, for someone in Australia, a postcode of "E3B 0H8" doesn't make sense, as that isn't the format we use. So you could wipe that out and replace it with "No postal code, malformed data entered".
But this is where I start to disagree. None should remain a singleton, but "no data available" could be its own thing, tied in with the way that you do your data storage and stats. As such, you wouldn't be checking it with 'is', so you wouldn't have that problem (the Python 'is' operator will only ever test for actual object identity). Keep None simple and dependable, and then "Missing Data" can be an entire class of values if you so desire. ChrisA

El jue, 21 oct 2021 a las 10:25, David Mertz, Ph.D. (<david.mertz@gmail.com>) escribió:
class MissingReason(enum.Enum): no_middle_name = 1 not_provided = 2 unknown = 3 middle_name: str | MissingReason This gives you more structured and precise data than None-plus-string-tag and avoids breaking the ecosystem by changing the meaning and behavior of None.

On Thu, Oct 21, 2021 at 01:46:27PM +1100, Steven D'Aprano wrote:
On Tue, Oct 19, 2021 at 05:09:42PM -0700, Michael Selik wrote:
On further thought, I no longer agree. Or at least, I think we need to think a lot harder about the API before adding any sort of chained attribute getter into the stdlib. If we get it wrong, we'll be stuck with it until Python 5000. The problem is that any sort of code equivalent to: try: return obj.chain.of.attribute.lookups except AttributeError: return None risks silently hiding genuine coding errors. This seems to be an anti-pattern, or at least a foot-gun. And it is certainly not equivalent to, or a replacement for, PEP 505. Same applies to variants similar to attrgetter. -- Steve

Baptiste Carvello wrote:
y = config["handler"]["parameters"]["y"] with KeyError as None
I love the look of this! While it doesn't address everything that PEP505 does, that's IMO a good thing, because - as other people mentioned already - None does too many things already. On another note, the whole discussion reminds me of wg21.link/p0798, which is about adding something quite similar to C++ (was accepted into C++23), and contains a decent overview of how other languages solve the same thing. Even though the approach there is probably not applicable (as all python types are implicitly `Optional` in the sense of "can be None", i.e. that API would have to be added all the way down to `object`), it's still ironic that C++'s `and_then` looks more pythonic than what's proposed here.

This is very reminiscent of the (rejected) PEP 463, Exception-catching expressions (which I still hope will be resurrected some day). It would allow you to write y = (config["handler"]["parameters"]["y"] except KeyError: None) (possibly the parentheses might not be required) which IMO is even better; `except` makes the intent clearer than `with`. Best wishes Rob Cliffe On 20/10/2021 03:05, h.vetinari@gmx.com wrote:

Data point: I find all the examples in PEP 505 less readable using the proposed new operators. Trying to explain why: The syntax feels *too* compact (Perl-like?) - when reading it, every time you see a None-aware operator (*if* you notice it), you have to jerk to a halt and say, "Whoa! What's going on here?". I have some sympathy with this use case of exploring a nested-dict (JSON derived?) structure, but this can be written as try: config.get("handler").get("parameters").get("y") except Attribute Error: # Handle missing value. or something similar (I haven't tested it). Even using the new operators, in a realistic case you would probably have to test if the result is None and take different action accordingly. Best wishes Rob Cliffe On 18/10/2021 19:26, Guido van Rossum wrote:

Big +1 from me. I've been looking forward to having None-aware operators in Python as I find them a very neat language addition. For me personally the main advantage of having maybe-dot (?.) operator is the ability to express certain logic in a much clearer way, for example:
Imagine wanting to get the phone number of the person that published a certain book from the database. In this situation, with maybe-dot operator I can write:
phone_number = book.publisher?.owner?.phone
Getting None value could mean that the book has not been published yet (no publisher relation), owner does not exist or does not have a phone number associated with it. Either way it doesn't matter because the only thing that we wanted to retrieve is the phone number and not the whole context of why it has a certain value and not the other. Personally I find this syntax much more readable than other, "more explicit" expressions like:
phone_number = book.publisher.owner.phone if (book.publisher is not None and book.publisher.owner is not None) else None
Best regards, On Thu, Oct 14, 2021 at 7:37 PM Doug Swarin <dswarin@gmail.com> wrote:

Okay, I'll let myself get sucked into responding ONE TIME, but only because you gave me such a nice API to work with :) On 10/18/2021 9:11 PM, Piotr Waszkiewicz wrote:
Consider today, you wrote this as "book.publisher.owner.phone". You would potentially get AttributeError, from any one of the elements - no way to tell which, and no way to react. Generally, AttributeError indicates that you've provided a value to an API which doesn't fit its pattern. In other words, it's an error about the *type* rather than the value. But in this case, the (semantic, not implementation) *type* is known and correct - it's a publisher! It just happens that the API designed it such that when the *value* is unknown, the *type* no longer matches. This is PRECISELY the kind of (IMHO, bad) API design that None-aware operators will encourage. Consider an alternative: class ForeignKey: ... def __bool__(self): return not self.is_dbnull def value(self): if self.is_dbnull: return self.Type.empty() # that is, DBModel.empty() return self._value class DBModel: @classmethod def empty(cls): return cls(__secret_is_empty_flag=True) def __bool__(self): return not self._is_empty def __getattr__(self, key): if not self: t = self._get_model_type(key) return t.empty() if isinstance(t, DBModel) else None ... class User(DBModel): phone: str | None class Publisher(DBModel): owner: ForeignKey[User] class Book(DBModel) publisher: ForeignKey[Publisher] Okay, so as the API implementer, I've had to do a tonne more work. That's fine - *that's my job*. The user hasn't had to stick "| None" everywhere (and when we eventually get around to allowing named arguments in indexing then they could use "ForeignKey[User, non_nullable=True]", but I guess for now that would be some subclass of ForeignKey). But now here's the example again:
book.publisher.owner.phone
If there is no publisher, it'll return None. If there is no owner, it'll return None. If the owner has no phone number, it'll return None. BUT, if you misspell "owner", it will raise AttributeError, because you referenced something that is not part of the *type*. And that error will be raised EVERY time, not just in the cases where 'publisher' is non-null. It takes away the random value-based errors we've come to love from poorly coded web sites and makes them reliably based on the value's type (and doesn't even require a type checker ;) ). Additionally, if you want to explicitly check whether a FK is null, you can do everything with regular checks: if book.publisher.owner: # we know the owner! else: # we don't know # Get all owner names - including where the name is None - but only if # Mrs. None actually published a book (and not just because we don't # know a book's publisher or a publisher's owner) owners = {book.id: book.publisher.owner.name for book in all_books if book.publisher.owner} # Update a null FK with a lazy lookup book.publisher = book.publisher or publishers.get(...) You can't do anything useful with a native None here besides test for it, and there are better ways to do that test. So None is not a useful value compared to a rich DBModel subclass that *knows* it is empty. --- So to summarise my core concern - allowing an API designer to "just use None" is a cop out, and it lets people write lazy/bad APIs rather than coming up with good ones. Cheers, Steve

Thank you very much for this exhaustive explanation and example. I really like it and agree with you that the implementation provided by your example is much more well designed. The problem I have with it is that I feel like it assumes that I have a way to introduce such changes when writing the API, whereas I was talking more about using existing solutions. The provided example was based on common situations encountered when writing the code in Django. It may be that I'm using this tool not to its full potential, but it seems like many of your points were about bad API design, not the actual maybe-dot operator. Such operator would make it much easier to work with those frameworks that do not allow for more readable/well thought-out solutions. I'm thinking about it as a syntactic sugar that can speed things up (and even make it more readable) in a situation where there is no time and/or budget to come up with a better architecture (like e.g. the one you have provided). The reality is that usually from the business perspective nobody wants a well designed system, only the functioning one. It may be that I misunderstood your response and didn't provide a valid answer - please let me know in that case. Best regards On Tue, Oct 19, 2021 at 1:29 AM Steve Dower <steve.dower@python.org> wrote:

I will also say that I don't believe the safe-navigation operators necessarily compromise type safety. PEP 505 explicitly rejects having them catch `AttributeError` or `KeyError` (and I agree with this rejection). It's not the default behavior of objects to return None when an unknown attribute is read, so attempting to access `book?.publisher?.onwer?.name` will still fail with `AttributeError`. Type checkers would also continue being able to check such navigation. Doug

Steve Dower wrote:
Okay, I'll let myself get sucked into responding ONE TIME, but only because you gave me such a nice API to work with :)
This actually pushed me hard towards adding the null-aware operators. I agree that the named-function approach Paul suggests is better. I admit that there are times when a comprehensive model is good, and that if the DB schema is automatically generated, it should look something like this. But this is long enough that I would be unhappy if it were what I had to read to understand the database in the first place. (Yes, I know Java beans are pretty widely tolerated, but ... that doesn't make longer code desirable.)

Most of the discussion so far has been focused on (?.). Tbh though, I'm more interested in (??) and (??=). Just reading through code, I constantly notice boilerplate like this which could easily be substituted. variable = some_function(...) if variable is None: variable = [] # some default value # a bit better with an assignment expression if (variable := some_function(...)) is None: variable = [] # or worse with an if expression variable = some_function(...) if some_function(...) else [] # also possible with :=, but not much better variable = x if (x := some_function(...)) else [] # using the coalesce operator would be much more readable IMO variable = some_function(...) ?? [] If (?.) and (?[) are rejected / deferred, maybe there is interest in seeing at least (??) and (??=) through?

On Sat, Oct 23, 2021 at 6:20 AM Marc Mueller <cdce8p@gmail.com> wrote:
Bear in mind that these last ones are exactly equivalent to the "or" operator, as they'll use the default if you have any falsy value. variable = some_function(...) or []
I'm actually more interested in a better idiom for non-constant function default arguments, since that's the place where this kind of thing often comes up. A nice ??= operator might help if your default is None, but if you then change the default to be object(), you can't use ??= any more. As a bonus, the docs for such an argument could actually say what the default really is: def bisect_right(a, x, lo=0, hi=len(a), *, key=None): ... except that it'd need some adornment to say that it's late-bound. ChrisA

Isn't that in itself a good argument in favor of (??) ? By missing to add 'is None', I would have already added a subtle bug that could be quite difficult to find. (??) could prevent that while also being more readable.
True, but from my experience 'None' is just by far the most common default. Why not improve how we handle it?

Marc Mueller writes:
True, but from my experience 'None' is just by far the most common default. Why not improve how we handle it?
The question is whether this is an improvement in the long run. When some falsies are expected, in-range values, "if arg is None: ..." or "x = default if arg is None else arg" is sufficiently precise and concise. I don't see "arg ?? default" as an improvement worthy of syntax. One the minus side, as David Mertz testifies, there is a maze of missing value use cases, all alike (or is that a missing maze of values, all alike?) These operators would further encourage conflating them all into None. At least in theory, that's not an improvement in handling None, that's a loss of precision in handling an important subset of Nones. I just don't see much value in ??. I don't have an opinion on the ?. and ?[] versions. I can see that they could make chained operations much more concise than nested if ... else expressions, and at least somewhat more precise (at least if you want to avoid deep nesting of try ... except KeyError statements, which I suppose most everybody would like to avoid). Steve

On Sun, Oct 24, 2021 at 12:35 AM Marc Mueller <cdce8p@gmail.com> wrote:
Perhaps, but your alternate versions were identically buggy, which means that - in all probability - the bug wouldn't have been relevant. For instance, if you're asking for a file-like object, and will use stdout if no argument is passed, it's absolutely fine to use None as a default and "or" to replace it with sys.stdout, because only a pathological example will actually cause problems. And that's fine. There are myriad use-cases for a None-coalescing operator that would work just fine without one; if Python had such an operator, sure, it could be used, but they're not strong arguments for adding one.
There are three situations to consider here: 1) Situations where there's no falsy value that would make sense 2) Situations where None is used and there are other falsy values that would make sense 3) Situations where any object makes sense The only ones where None-coalescing is truly significant are the second group. In the first group, "or" and "??" would behave identically, so why bother introducing a new operator? And in the third group, the default can't be None, so it has to be an artificially-invented object. All three would be improved by a late-binding feature, but only a strict subset of them would benefit from directly replacing None with something else. Like several others here, I am much more interested in the operators that do "if None, give back None, otherwise fetch an attribute/item". If we end up getting None-coalescing for free along with that, so be it, but I'm not really pushing for that part. Proper argument late-binding is an entirely orthogonal proposal, and would have many benefits that aren't part of PEP 505 or related proposals, but would do most of what we want from the ?? operator. PEP 505 has some oddities that are hard to explain, though - and I mean that not in the sense that the PEP itself is hard to read, but that I would have to go to some effort to explain what my code does (picture a mentor/student relationship here). If I offer a line of code saying <<x = spam()?.ham.eggs or 5>>, what happens if spam returns None? Unpacking it step by step reveals a bit more complexity than it first seems to have. Python tries to minimize such situations, doing them only when there is a very clear benefit (for instance, "3 < x < 8" is very different from "(3 < x) < 8", but that's because it is immensely valuable), and I do have misgivings about that part of it. Otherwise, though, I am in favour of this and would make good use of it. (And would do so regardless of the ultimate decision on the chaining.) ChrisA

Hi, I wanted to propose replacing ? with -> in the none aware syntax. This makes the expression look like a chain, and I think that most people intuitively can understand how the chain might get broken (and what would happen in that case). For example: zipcode = person->.address->['zipcode'] I've proposed this alternative syntax on Doug's github project, and I think he didn't like it so much (or at least, his co-contributor didn't like it), but I still wanted to propose it here as well, because I do think it's a good solution to the complaints that the ? operator looks cryptic. Best regards, Maarten

Hi Maarten, I like the suggestion but I'm not sure if the real problem with the PEP505 is the symbol/operator used. Reading through the whole discussion I'm under the impression that the idea of the None value being treated as an indicator of the missing attribute is what prevents this PEP from happening. Apart from that I'm all for it, (probably) regardless of the syntax (as long as it remains short). Best regards, Piotr On Thu, Sep 15, 2022 at 5:07 PM Maarten Nieber <mnieber@gmail.com> wrote:

Hi Piotr, doesn't Doug's reply of 8:03 address this point? As he says, the none-aware operator never gives you None when you ask for a missing attribute (these cases always raise an exception). If we look at these two alternatives phone1 = book.publisher?.owner.phone phone2 = book.publisher.owner.phone if book.publisher else None then they behave exactly the same. If we would misspell "owner" then in both versions we'd get the same AttributeError under the same conditions. Best, Maarten

Hi Maarten, I'm sorry for the confusion - it was bad wording on my part. What I really meant was that the problem with the None-aware operator, and the reason why PEP505 has not been accepted for such a long time, is that there's no consensus regarding the need for it (and not necessarily the problem with the operator used to represent it - although this topic has also been raised). Take a look at the response by Steve Dower: https://mail.python.org/archives/list/python-dev@python.org/message/BRTRKGY6... Best regards, Piotr On Thu, Sep 15, 2022 at 6:54 PM Maarten Nieber <mnieber@gmail.com> wrote:

So to summarise my core concern - allowing an API designer to "just use None" is a cop out, and it lets people write lazy/bad APIs rather
Hi all, I've only here found out that there is a discussion going on about those none-aware operators and my first thought was "great, finally!". FWIW, I'd be happy with the syntax suggestion in the PEP, since '?' looks rather intuitive to me to mean something like "maybe". However, I then read the mentioned post of Steve Dower, with the final summary: than coming up with good ones. This is a very good point. In fact, I've never really thought about it that way and of course he's totally right that "SomeType | None" (or Optional[SomeType], which also somehow made me feel that this usage is fairly intended) is not optimal, at least for user defined types/classes. The problem is, that I never actually thought about his suggested way. And I wouldn't be surprised if this holds for many other people as well. Maybe it would be great to boldly mention these thoughts in the documentation at an appropriate place. In my opinion, there are at least the following good places where this would fit nicely: - The documentation of the dataclasses (https://docs.python.org/3/library/dataclasses.html), since this is probably the most common use case for the "| None" pattern. Going further, the dataclasses functionality might even be extended to make it simpler to generate such null-types (or however they are called), so that it is no longer "a tonne more work". - Linters like pylint could emit a note when seeing the "| None" pattern, linking to the explanation about why it is possibly not the best way to do it. - The documentation of the discussed None-aware operators. Since these new operators are closely coupled to the arguably suboptimal "| None" pattern, it is probably good to tell folks right there why they should consider better alternatives. As mentioned, I absolutely see Steve's point. However, there are many Python scripts/programs working without a complex API, where this "| None" pattern may still have its legitimate uses and the none-aware operators can make code easier to read (and write). Best regards, Philipp

Personally I think returning None is a fine API design, and IMO the concerns about this pattern are overblown. Note that X|None is no different than the "Maybe X" pattern that functional programmers are so fond of. On Mon, Sep 19, 2022 at 8:02 AM Philipp Burch <phip@hb9etc.ch> wrote:
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>

On 19. 09. 22 17:58, Guido van Rossum wrote:
I must disagree here. With `X|None` there is no way to do `Maybe[Maybe[X]]`, and FP is all about proper composition. One practical consequence is that "Maybe X" doesn't have the problem that PEP 661 (Sentinel Values) tries to solve.

Hi Philipp, That's a really good idea, and I'd really want to see it being implemented. Having said that, I wonder how many people would actually use this concept of "none-representing" objects. In my 10+ years of programming experience I've never seen anybody eager to use more complex structures or object representations of such concepts outside of an academic environment (and even there usually when dealing with personal or proof-of-concept projects). Nowadays I write mostly more business-oriented code, and I haven't been successful in finding a workplace where "business" wants to pay for preparing "good looking" code - they want a functioning one. That's where this whole concept of using none values is used the most (at least from my experience). So to reiterate: I think that the idea presented by Steve is very appealing, and I'd really like to work with such projects, but the reality is that it's really hard to stumble upon such. And I would really much *prefer to have none-aware operators that would make my life easier working with "real world" projects that rely too heavily on none values*, even if it's academically not correct. On Mon, Sep 19, 2022, 17:06 Philipp Burch <phip@hb9etc.ch> wrote:

On Sun, Sep 18, 2022 at 09:43:26PM +0200, Philipp Burch wrote:
I don't think that `SomeType|None` is necessarily a lazy or bad API. Whether it is "optimal" or not will depend on the specific SomeType involved, not to mention other tradeoffs and constraints such as time and money available. (Oh, to have the luxury of spending two months to "Do It Right" for a project needed in a week on a budget...) But what's not "optimal" is expecting everyone who writes a class and needs to represent the lack of an instance to duplicate those None-like semantics within the class. The distinction you make between user-defined and non-user-defined classes doesn't hold water. If you allow that (say) `int|None` **might** be acceptable, then why would `Integer|None` **necessarily** be lazy and bad just because int is written in C and built into the intepreter while Integer is written in Python by a user? The laziness/badness of `SomeType|None` must depend on the semantics of SomeType, not the implementation language or who wrote it. "Our API was lazy and bad because it uses None, but then it got accepted into Python as a builtin, so now it is no longer lazy or bad." /s Of course people can write None-like instances of their SomeType classes, if it makes sense for their application. But the idea that any and every use (or even a majority) of Optional types is a "lazy/bad API" is, I think, unrealistic puritism, not to mention unfairly disparaging towards the programmers who write those APIs.
The problem is, that I never actually thought about his suggested way.
Some people have, and found that it doesn't always simplify the API that much, or at all. If you end up replacing `obj is None` with `obj == NULL` or `obj.isnull()` then you haven't really gained much. -- Steven

Hi Steven! On 21.09.22 13:17, Steven D'Aprano wrote:
I considered a while if "user-defined types/classes" is the right wording, but didn't find a better alternative. The point I'm making is that `X|None` could especially then hide bugs if X defines attributes that will often be accessed in a chain. For int or float, there are only a few methods, which are arguably rarely used and chaining is probably not used at all. Plus, providing nulltypes for primitives would feel very awkward. Consider ``` x: int | None = 5 # ... if x is not None: y = 1 << x.bit_cont() ``` The typo with bit_cont() is hidden whenever x is None, so it might be missed by unit tests with limited coverage. However, using a nullable variant of it would probably make it look like this: ``` class Int(int): # ... class NoneInt(Int): # ... x = Int(5) # With the option for NoneInt() # ... x_bit_count = x.bit_cont() if x_bit_count == NoneInt: y = 1 << x_bit_count ``` This is certainly not pretty for something basic as an int, plus there will most likely be a performance hit when replacing every int by Int(), not to mention compatibility issues with APIs/libraries expecting plain int. On the other hand, more complex classes like in the mentioned publisher API could indeed profit from such null-types, since a chain like ``` book?.publisher?.owner?.namme ``` will hide the typo with namme whenever any of book, publisher or owner is None, so there is much more potential here to miss it in a unit test. This is actually the distinction that I wanted to make, not in the technical sense if a class is defined in the standard library or not.
Certainly. But without None-aware operators, the gain in readability can be large, if you can just write ``` name = book.publisher.owner.name if name is not None: # No NoneStr here, see above # ... ``` versus ``` if (book is not None) and (book.publisher is not None) and ...: # ... ``` Just to be clear: I've been thankful for the comment and explanation of Steve Dower, because I've never thought about it that way and now see constructs in my code, where such a concept really would have simplified things and improved robustness. For others to profit as well, I would be glad to see this mentioned in the docs, so that it can be found even when not actually looking for it. But I certainly don't want to say that this concept must be followed everywhere or that "X|None" is necessarily a bad thing. After all, I'd still love to see Python supporting such None-aware operators. Best regards, Philipp
participants (22)
-
Antoine Pitrou
-
Baptiste Carvello
-
Chris Angelico
-
David Mertz, Ph.D.
-
Doug Swarin
-
Guido van Rossum
-
h.vetinari@gmx.com
-
Jelle Zijlstra
-
Jeremiah Vivian
-
Jim J. Jewett
-
Maarten Nieber
-
Marc Mueller
-
Michael Selik
-
Paul Bryan
-
Paul Moore
-
Petr Viktorin
-
Philipp Burch
-
Piotr Waszkiewicz
-
Rob Cliffe
-
Stephen J. Turnbull
-
Steve Dower
-
Steven D'Aprano