Thanks! Bit of discussion below about precedence, but thanks for spotting the typos.

On 18Jul2018 1318, MRAB wrote:

On 2018-07-18 18:43, Steve Dower wrote:

...

Grammar changes

The following rules of the Python grammar are updated to read::

augassign: ('+=' | '-=' | '*=' | '@=' | '/=' | '%=' | '&=' | '|=' | '^=' |                   '<<=' | '>>=' | '**=' | '//=' | '??=')

power: coalesce ['**' factor]       coalesce: atom_expr ['??' factor]       atom_expr: ['await'] atom trailer*       trailer: ('(' [arglist] ')' |                 '[' subscriptlist ']' |                 '?[' subscriptlist ']' |                 '.' NAME |                 '?.' NAME)

The precedence is higher than I expected. I think of it more like 'or'. What is its precedence in the other languages?

Yes, I expected this to be the contentious part. I may have to add a bit of discussion.

Mostly, I applied intuition rather than copying other languages on precedence (and if you could go through my non-git history, you'd see I tried four other places ;) ). The most "obvious" cases were these::

a ?? 1 + b()

b ** a() ?? 2

In the first case, both "(a ?? 1) + b()" and "a ?? (1 + b())" make sense, so it's really just my own personal preference that I think it looks like the first. If you flip the operands to get "b() + a ?? 1" then you end up with either "b() + (a ?? 1)" or "(b() + a) ?? 1", then it's more obvious that the latter doesn't make any sense (why would __add__ return None?), and so binding more tightly than "+" helps write sensible expressions with fewer parentheses.

Similarly, I feel like "b ** (a() ?? 2)" makes more sense than "(b ** a()) ?? 2", where for the latter we would have to assume a __pow__ implementation that returns None, or one that handles being passed None without raising a TypeError.

Contrasting this with "or", it is totally legitimate for arithmetic operators to return falsey values.

As I open the text file to correct the typos, I see this is what I tried to capture with:

Inserting the ``coalesce`` rule in this location ensures that expressions resulting in ``None`` are naturally coalesced before they are used in operations that would typically raise ``TypeError``.

Take (2 ** a.b) ?? 0. The result of __pow__ is rarely going to be None, unless we train all the builtin types to do so (which, incidentally, I am not proposing and have no intention of proposing), whereas something like "2 ** coord?.exponent" attempting to call "2.__pow__(None)" seems comparatively likely. (Unfortunately, nobody writes code like this yet :) So there aren't any real-life examples. Originally I didn't include "??" in the proposal, but it became obvious in the examples that the presence of None-propagating operators ?. and ?[] just cause more pain without having the None-terminating operator ?? as well.)

...

Inserting the ``coalesce`` rule in this location ensures that expressions resulting in ``None`` are natuarlly coalesced before they are used in

Typo "natuarlly".

Thanks.

...

assert a == 'value'       assert b == ''       assert c == '0' and any(os.scandir('/'))

Wouldn't the last assertion fail, because c == 0?

Correct, another typo.

Cheers, Steve

On Thu, Jul 19, 2018 at 9:55 AM, Giampaolo Rodola' g.rodola@gmail.com wrote: [ please trim quotes, you just quoted the entire PEP in your post ]

With all due respect (and I am sorry for being “vocal” about a PEP once again) I find this simply ugly. To me this basically doesn’t look like python anymore, so a strong -1 from me.

I'd love to hear an explanation of WHY this doesn't look like Python any more. For instance, is the + operator somehow wrong for Python, and it should have been the word "add"? People complain about proposals with words like these, but the best explanation I've ever had is "well, Python uses words and this proposal uses punctuation".

Personally, I'm +0 on this. It'd be a few small wins here and there, nothing huge, and I could easily live without it; but it's something that I know some people will love.

ChrisA

Chris Angelico wrote:

I'd love to hear an explanation of WHY this doesn't look like Python any more. For instance, is the + operator somehow wrong for Python, and it should have been the word "add"?

There's a very long tradition of using the symbol "+" to represent addition, so it's something most people are familiar with. There's no such tradition for the new operators being proposed.

On Thu, 19 Jul 2018 19:11:33 +1000 Chris Angelico rosuav@gmail.com wrote:

On Thu, Jul 19, 2018 at 4:06 PM, Greg Ewing greg.ewing@canterbury.ac.nz wrote:

...

Chris Angelico wrote:

...

I'd love to hear an explanation of WHY this doesn't look like Python any more. For instance, is the + operator somehow wrong for Python, and it should have been the word "add"?

There's a very long tradition of using the symbol "+" to represent addition, so it's something most people are familiar with. There's no such tradition for the new operators being proposed.

Okay. What about bitwise operators, then? They don't have centuries of mathematical backing to support them, yet it isn't considered "unpythonic" to have &|^~ peppering our code.

They have decades of widespread presence in other programming languages, though.

Coalescing None to a value is _at least_ as common as performing bit manipulations in integers.

Certainly, but spelling that as a "?*" operator is a syntactical novelty.

Consider that for the ternary operator, Python chose "B if A else C" over "A ? B : C", even though the latter had precedent in several languages.

Regards

Antoine.

On Thu, Jul 19, 2018, 5:12 AM Chris Angelico rosuav@gmail.com wrote:

On Thu, Jul 19, 2018 at 4:06 PM, Greg Ewing greg.ewing@canterbury.ac.nz wrote:

...

There's a very long tradition of using the symbol "+" to represent addition, so it's something most people are familiar with. There's no such tradition for the new operators being proposed.

Okay. What about bitwise operators, then? They don't have centuries of mathematical backing to support them, yet it isn't considered "unpythonic" to have &|^~ peppering our code.

I have quite literally NEVER seem Python code with much use of the bitwise operators. I guess the closest I've come is in some NumPy and Pandas code where filters cannot use plain 'and' and ' or' but every student would find it more intuitive if they could. E.g.

myDf[(myDf.field1 > 4) & (myDf.field2 < 2)]

Everyone still gets tripped up by the need for those parentheses because of operator precedence. But this already yells or "special domain" rather than "plain Python".

Indeed, it not uncommon or unreasonable to recommend using np.bitwise_and() to avoid those confusing operators.

In the case where bitwise masking is used in its C manner, the code screams special domain even more loudly. It definitely feels strongly unPythonic, but it's often a reasonable compromise for dealing with just a little bit of binary data without having to write a C extension.

I would ALWAYS want the code that used bitwise operators wrapped in a separate function that most users and developers didn't need to look at, but rather they'd call a more Pythonic API for the overall operation.

A huge difference is that bitwise operators do something you simply cannot do other ways in Python at all. None-aware operators *at best* allow you to write something with an existing straightforward approach using a couple fewer lines. I think the result is ALWAYS less clear to read. Code golf is an anti-goal in Python.

On Thu, Jul 19, 2018 at 11:37 AM Brendan Barnwell brenbarn@brenbarn.net wrote:

    As far as I can see, these null-coalescing operators would break

that model. The PEP doesn't seem to provide for a "real" magic method allowing users to override the actual behavior of the method. (You can only override __has_value__ to hook into it, but not define by fiat what A ?? B does, as you can with other operators.) And I think the reason for this is that the operator itself is too specific, much more specific in semantics than other operators. (I had similar doubts about adding the matrix-multiplication operator @.)

It is just as specific as the `is` operator, and for the same reason.

Elazar

Let me just address this point:

2018-07-19 20:36 GMT+02:00 Brendan Barnwell brenbarn@brenbarn.net:

    As far as I can see, these null-coalescing operators would break

that model. The PEP doesn't seem to provide for a "real" magic method allowing users to override the actual behavior of the method. (You can only override __has_value__ to hook into it, but not define by fiat what A ?? B does, as you can with other operators.) And I think the reason for this is that the operator itself is too specific, much more specific in semantics than other operators. (I had similar doubts about adding the matrix-multiplication operator @.)

I think the actual reason is that it is a short-cutting operator, and none of the shortcutting operators (and, or, if/else) have an associated method. They cannot have, since they induce a non-standard evaluation order, hence their effect cannot be emulated with a method invocation.

Stephan

Thanks everyone for the feedback and discussion so far. I want to address some of the themes, so apologies for not quoting individuals and for doing this in one post instead of twenty.

------

* "It looks like line noise"

Thanks for the feedback. There's nothing constructive for me to take from this.

* "I've never needed this"

Also not very actionable, but as background I'll say that this was exactly my argument against adding them to C#. But my coding style has adapted to suit (for example, I'm more likely to use "null" as a default value and have a single function implementation than two mostly-duplicated overloads).

* "It makes it more complex" * "It's harder to follow the flow"

Depends on your measure of complexity. For me, I prioritise "area under the indentation" as my preferred complexity metric (more lines*indents == more complex), as well as left-to-right reading of each line (more random access == more complex). By these measures, ?. significantly reduces the complexity over any of the current or future alternatives::

def f(a=None): name = 'default' if a is not None: user = a.get_user() if user is not None: name = user.name print(name)

def f(a=None): if a is not None: user = a.get_user() name = user.name if user is not None else 'default' print(name) else print('default')

def f(a=None): user = a.get_user() if a is not None else None name = user.name if user is not None else 'default' print(name)

def f(a=None): print(user.name if (user := a.get_user() if a is not None else None) is not None else 'default')

def f(a=None): print(a?.get_user()?.name ?? 'none')

* "We have 'or', we don't need '??'"

Nearly-agreed, but I think the tighter binding on ?? makes it more valuable and tighter test make it valuable in place of 'or'.

For example, compare:

a ** b() or 2 # actual: (a ** b()) or 2 a ** b() ?? 2 # proposed: a ** (b() ?? 2)

In the first, the presence of 'or' implies that either b() or __pow__(a, b()) could return a non-True value. This is correct (it could return 0 if a == 0). And the current precedence results in the result of __pow__ being used for the check.

In the second one, the presence of the '??' implies that either b() or __pow__(a, b()) could return None. The latter should never happen, and so the choices are to make the built-in types propagate Nones when passed None (uhh... no) or to make '??' bind to the closer part of the expression.

(If you don't think it's likely enough that a function could return [float, None], then assume 'a ** b?.c ?? 2' instead.)

* "We could have '||', we don't need '??'"

Perhaps, though this is basically just choosing the bikeshed colour. In the absence of a stronger argument, matching existing languages equivalent operators instead of operators that do different things in those languages should win.

* "We could have 'else', we don't need '??'"

This is the "a else 'default'" rather than "a ?? 'default'" proposal, which I do like the look of, but I think it will simultaneously mess with operator precedence and also force me to search for the 'if' that we actually need to be comparing "(a else 'default')" vs. "a ?? 'default'"::

x = a if b else c else d x = a if (b else c) else d x = a if b else (c else d)

* "It's not clear whether it's 'is not None' or 'hasattr' checks"

I'm totally sympathetic to this. Ultimately, like everything else, this is a concept that has to be taught/learned rather than known intrinsically.

The main reasons for not having 'a?.b' be directly equivalent to getattr(a, 'b', ???) is that you lose the easy ability to find typos, and we also already have the getattr() approach.

(Aside: in this context, why should the result be 'None' if an attribute is missing? For None, the None value propagates (getattr(a, 'b', a)), while for falsies you could argue the same thing applies. But for a silently handled AttributeError? You still have to make the case that None is special here, just special as a return value vs. special as a test.)

* "The semantics of this example changed from getattr() with ?."

Yes, this was a poor example. On re-reading, all of the checks are indeed looking for optional attributes, rather than looking them up on optional targets. I'll find a better one (I've certainly seen and/or written code like this that was intended to avoid crashing on None, but I stopped my search of the stdlib too soon after finding this example).

* "Bitwise operators"

Uh... yeah. Have fun over there :)

* "Assumes the only falsie ever returned [in some context] is None"

I argue that it assumes the only falsie you want to replace with a different value is None. In many cases, I'd expect the None to be replaced with a falsie of the intended type:

x = maybe_get_int() ?? 0 y = maybe_get_list() ?? []

Particularly for the second case, if you are about to mutate the list, then it could be very important that you don't replace the provided reference with your own, just because it happens to be empty right now (this is the logging example in the PEP). Supporting a default value in the get() calls is obviously a better way to do this though, provided you have the ability to modify the API.

But for me, the ?? operator makes its strongest case when you are getting attributes from potentially None values:

x = maybe_get()?.get_int() ?? 0 y = maybe_get()?.get_list() ?? []

In this case, using 'or' may replace an intentionally falsie value with your own, while using a default parameter is still going to leave you with None if the first maybe_get() returned nothing.

"?." without "??" feels like a trap, while "??" without "?." feels largely unnecessary. But both together lets you turn many lines of code into a much shorter snippet that reads left-to-right and lets you assume success until you reach the "??". That, for me, is peak readability.

Steve Dower writes:

But for me, the ?? operator makes its strongest case when you are getting attributes from potentially None values:

 x = maybe_get()?.get_int() ?? 0
 y = maybe_get()?.get_list() ?? []

In this case, using 'or' may replace an intentionally falsie value with your own,

I wish the PEP's examples had such cases marked and explained. I didn't understand the context for most examples (ie, *why* the left operand could be None, and why that needed to be distinguished from "intentionally falsie values").

In the "find_content_type" example, None is just a way of saying "oops, you handle it" that is less "cute" than "". "or" is far more resilient in that context than "??" would be. I am a mail geek, and *much* more likely to encounter that code than the others. I would be distressed if the email module or Mailman grew a lot of "??". Email is a domain where resilience really matters because you can receive arbitrary data over SMTP, there's no provision for rejecting garbage, and all too often you do get garbage. :-/

while using a default parameter is still going to leave you with None if the first maybe_get() returned nothing.

"?." without "??" feels like a trap, while "??" without "?." feels largely unnecessary. But both together lets you turn many lines of code into a much shorter snippet that reads left-to-right and lets you assume success until you reach the "??".

I find that pretty persuasive. Perhaps that should be added to PEP 8.

I would advocate write those

x = maybe_get() ?. get_int() ?? 0 y = maybe_get() ?. get_list() ?? []

to emphasize that this is an operator that "does something" to the operand, rather than "just" extracting an attribute. (No, I'm not going to try to make that more precise, so feel free to ignore.) I'm not sure about the ?[ operator:

x = a?[b] ?? c x = a ?[b] ?? c x = a ?[ b] ?? c x = a ?[ b ] ?? c

I guess I like the second line best (and

x = a?[b]??c

is right out, of course).

I do think Terry Reedy's criticism that in many cases having maybe_get() return None rather than an appropriate default object is bad API design may be important. I suspect that the "a ?. b ?? c" pattern will discourage refactoring of that code in favor of writing short workarounds.

my vote would go to `A otherwise B` since it's unambiguous, the case you care about the state of comes first, and it doesn't trip your brain up looking for 'if'. :)

And I’d hope “otherwise” is a rare variable name :-)

- CHB

---

Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On 2018-07-19 02:11, Chris Angelico wrote:

-snip- As far as I can see, these null-coalescing operators would break that model. The PEP doesn't seem to provide for a "real" magic method allowing users to override the actual behavior of the method. (You can only override __has_value__ to hook into it, but not define by fiat what A ?? B does, as you can with other operators.) And I think the reason for this is that the operator itself is too specific, much more specific in semantics than other operators. (I had similar doubts about adding the matrix-multiplication operator @.)

    People keep saying that this null-coalescing behavior is so common 

and useful, etc., but that hasn't been my experience at all. In my experience, the desire to shortcut this kind of logic is more often a sign of corner-cutting and insufficiently specified data formats, and is likely to cause bugs later on. Eventually it has to actually matter whether something is None or not, and these operators just kick that can down the road. In terms of their abstract meaning, they are not remotely close to as common or useful as operators like & and |.

Fully agree with you on this, so -1 on those new operators. I like operators (I was behind one of the early proposals for matmul operator (even proposed a bunch of others, for orthogonality purpose: element-wise/object-as-a-whole is a dichotomy that can happen for other things that multiplication), but they need to benefit from operators specifics: precendence order, tradition meaning direct translation of formula/recipies already written with a similar notation, and be generic enough to allow productive use of overloading (instead of confusing use).

Matmul have this (with possible exception for overloading, but they are quite a few other mathematical entities where you want to have more than 1 multiplication, beside matrices).

Those new operators, not so much, the only benefit is more compact notation of something that is already quite compact (A if A is not None else B) and very explicit (it reads like pseudocode, which is a huge strengh of python). ?? and associates is much more cryptic, force you to think much harder about precedence, just to be slightly more compact. Granted, compact notation can be valuable, but usually if you want to couple multiple such things (like you often do with matmul), and precedence rules allows you to do it elegantly combine multiple operator with infix notation (again matmul is present in formulas that have scalar mul, + and -) . Do we do this with None-coalescence? I don't think so, not often anyway... Do we want to do this more with None-coalescence? Not me, certainly not: When I None coalesce, I want it to be as clear and atomic as possible because it's a corner case.

so strong -1...

On Friday, July 20, 2018 at 11:12:26 AM UTC+2, Chris Angelico wrote:

On Fri, Jul 20, 2018 at 5:10 PM, Grégory Lielens <gregory...@gmail.com javascript:> wrote:

Excuse me, that wasn't my words you quoted. Watch your citations please :)

ChrisA

Yes, sorry, it was Brendan Barnwell. And I can not edit my previous post....:-/

On 19/07/18 07:06, Greg Ewing wrote:

Chris Angelico wrote:

...

I'd love to hear an explanation of WHY this doesn't look like Python any more. For instance, is the + operator somehow wrong for Python, and it should have been the word "add"?

There's a very long tradition of using the symbol "+" to represent addition, so it's something most people are familiar with. There's no such tradition for the new operators being proposed.

There is, actually, it's just not a long one. C# has had null-aware operators for a while, for example. I'll admit the syntax sits better in a C-like language, though.

On 20/07/18 01:30, Greg Ewing wrote:

Rhodri James wrote:

...

On 19/07/18 07:06, Greg Ewing wrote:

...

There's no such tradition for the new operators being proposed.

There is, actually, it's just not a long one.  C# has had null-aware operators for a while, for example.

THere's a precedent, yes, but I wouldn't call it a tradition. A substantial period of time is part of the definition of the word.

Wikipedia:

"A tradition is a belief or behavior passed down within a group or society with symbolic meaning or special significance with origins in the past."

Merriam-Webster:

"the handing down of information, beliefs, or customs from one generation to another."

I don't think C# has been around long enought to span multiple generations of programmers.

I go with SF fandom's traditional :-) definition: "somebody did it once." If it's been done more than once, it's an honoured tradition.

On 20 July 2018 at 13:16, Chris Angelico rosuav@gmail.com wrote:

On Fri, Jul 20, 2018 at 10:14 PM, Rhodri James rhodri@kynesim.co.uk wrote:

...

I go with SF fandom's traditional :-) definition: "somebody did it once." If it's been done more than once, it's an honoured tradition.

But if Shakespeare did it, it's just the way the language is.

I think Fortran is the programming world's Shakespeare.

Or maybe COBOL:

"And lo from yonder file a record doth appear"...

Paul

One of the big problems with most of the proposed spellings is that question marks to a learner will feel like truthiness tests and not None tests. I would propose adding a new default statement for the most useful part of this PEP, the ?= assignment

currently: if x is None: x = blah

with `?=`: x ?= blah

with defalult: default x: blah

This a) makes it obvious even to an amateur python-reader that x will only be changed if it does not yet exist, and b) has no similarity to the truthiness checks involved with the standard meaning of `?` as the ternary operator in many other languages.

On Fri, Jul 20, 2018 at 8:30 AM Chris Angelico rosuav@gmail.com wrote:

On Fri, Jul 20, 2018 at 10:24 PM, Paul Moore p.f.moore@gmail.com wrote:

...

On 20 July 2018 at 13:16, Chris Angelico rosuav@gmail.com wrote:

...

On Fri, Jul 20, 2018 at 10:14 PM, Rhodri James rhodri@kynesim.co.uk

wrote:

...

...

...

I go with SF fandom's traditional :-) definition: "somebody did it

once."

...

...

...

If it's been done more than once, it's an honoured tradition.

But if Shakespeare did it, it's just the way the language is.

I think Fortran is the programming world's Shakespeare.

Or maybe COBOL:

"And lo from yonder file a record doth appear"...

Hah! But I was thinking of all those uber-obvious features like "a + b" meaning addition. Those symbols, and the infix style, aren't "traditions" - they're the baseline that we measure everything else against. Also, the use of decimal digits to represent literals; if you *don't* use decimal, you're unusual. (Which makes the Shakespeare Programming Language [1] an ironic example, since it doesn't use decimal digits for numeric literals.)

ChrisA [1] http://shakespearelang.sourceforge.net/report/shakespeare/ _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

The examples in the PEP strengthen my opinion. I think every one of them read much more clearly in the existing syntax. I can only understand any by mentally inserting 'if Foo is None' everywhere... Basically mentally replacing the "code golf" syntax with the actual intent that is spelled like it is in current Python.

On Wed, Jul 18, 2018, 8:05 PM David Mertz mertz@gnosis.cx wrote:

As before, I'm a strong -1 on this.

I recognize the semantic utility (albeit, I've needed it much less than some folks testify). But the obscure characters are accumulating far too fast with this.

Even with PEP 572, I would have preferred 'as' and restrictions on where it's allowed, but ':=' is more familiar from numerous languages and pseudo-code notations. '??' and '?.' and ?[]' are really just marching into APL and Perl territory. Yes, I know such operators exist in other languages, but it feels very unpythonic.

On Wed, Jul 18, 2018, 1:46 PM Steve Dower steve.dower@python.org wrote:

...

Possibly this is exactly the wrong time to propose the next big syntax change, since we currently have nobody to declare on it, but since we're likely to argue for a while anyway it probably can't hurt (and maybe this will become the test PEP for whoever takes the reins?).

FWIW, Guido had previously indicated that he was generally favourable towards most of this proposal, provided we could figure out coherent semantics. Last time we tried, that didn't happen, so this time I've made the semantics much more precise, have implemented and verified them, and made much stronger statements about why we are proposing these.

Additional thanks to Mark Haase for writing most of the PEP. All the fair and balanced parts are his - all the overly strong opinions are mine.

Also thanks to Nick Coghlan for writing PEPs 531 and 532 last time we went through this - if you're unhappy with "None" being treated as a special kind of value, I recommend reading those before you start repeating them.

There is a formatted version of this PEP at https://www.python.org/dev/peps/pep-0505/

My current implementation is at https://github.com/zooba/cpython/tree/pep-505 (though I'm considering removing some of the new opcodes I added and just generating more complex code - in any case, let's get hung up on the proposal rather than the implementation :) )

Let the discussions begin!

---

PEP: 505 Title: None-aware operators Version: $Revision$ Last-Modified: $Date$ Author: Mark E. Haase mehaase@gmail.com, Steve Dower steve.dower@python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 18-Sep-2015 Python-Version: 3.8

Abstract

Several modern programming languages have so-called "``null``-coalescing" or "``null``- aware" operators, including C# [1]_, Dart [2]_, Perl, Swift, and PHP (starting in version 7). These operators provide syntactic sugar for common patterns involving null references.

• The "``null``-coalescing" operator is a binary operator that returns

its left operand if it is not ``null``. Otherwise it returns its right operand.

• The "``null``-aware member access" operator accesses an instance

member only if that instance is non-``null``. Otherwise it returns ``null``. (This is also called a "safe navigation" operator.)

• The "``null``-aware index access" operator accesses an element of a

collection only if that collection is non-``null``. Otherwise it returns ``null``. (This is another type of "safe navigation" operator.)

This PEP proposes three ``None``-aware operators for Python, based on the definitions and other language's implementations of those above. Specifically:

• The "``None`` coalescing`` binary operator ``??`` returns the left

hand side if it evaluates to a value that is not ``None``, or else it evaluates and returns the right hand side. A coalescing ``??=`` augmented assignment operator is included.

• The "``None``-aware attribute access" operator ``?.`` evaluates the

complete expression if the left hand side evaluates to a value that is not ``None``

• The "``None``-aware indexing" operator ``?[]`` evaluates the complete expression if the left hand site evaluates to a value that is not

``None``

Syntax and Semantics

Specialness of ``None``

The ``None`` object denotes the lack of a value. For the purposes of these operators, the lack of a value indicates that the remainder of the expression also lacks a value and should not be evaluated.

A rejected proposal was to treat any value that evaluates to false in a Boolean context as not having a value. However, the purpose of these operators is to propagate the "lack of value" state, rather that the "false" state.

Some argue that this makes ``None`` special. We contend that ``None`` is already special, and that using it as both the test and the result of these operators does not change the existing semantics in any way.

See the `Rejected Ideas`_ section for discussion on the rejected approaches.

Grammar changes

The following rules of the Python grammar are updated to read::

 augassign: ('+=' | '-=' | '*=' | '@=' | '/=' | '%=' | '&=' | '|=' |

'^=' | '<<=' | '>>=' | '**=' | '//=' | '??=')

 power: coalesce ['**' factor]
 coalesce: atom_expr ['??' factor]
 atom_expr: ['await'] atom trailer*
 trailer: ('(' [arglist] ')' |
           '[' subscriptlist ']' |
           '?[' subscriptlist ']' |
           '.' NAME |
           '?.' NAME)

Inserting the ``coalesce`` rule in this location ensures that expressions resulting in ``None`` are natuarlly coalesced before they are used in operations that would typically raise ``TypeError``. Like ``and`` and ``or`` the right-hand expression is not evaluated until the left-hand side is determined to be ``None``. For example::

 a, b = None, None
 def c(): return None
 def ex(): raise Exception()

 (a ?? 2 ** b ?? 3) == a ?? (2 ** (b ?? 3))
 (a * b ?? c // d) == a * (b ?? c) // d
 (a ?? True and b ?? False) == (a ?? True) and (b ?? False)
 (c() ?? c() ?? True) == True
 (True ?? ex()) == True
 (c ?? ex)() == c()

Augmented coalescing assignment only rebinds the name if its current value is ``None``. If the target name already has a value, the right-hand side is not evaluated. For example::

 a = None
 b = ''
 c = 0

 a ??= 'value'
 b ??= undefined_name
 c ??= shutil.rmtree('/')    # don't try this at home, kids

 assert a == 'value'
 assert b == ''
 assert c == '0' and any(os.scandir('/'))

Adding new trailers for the other ``None``-aware operators ensures that they may be used in all valid locations for the existing equivalent operators, including as part of an assignment target (more details below). As the existing evaluation rules are not directly embedded in the grammar, we specify the required changes here.

Assume that the ``atom`` is always successfully evaluated. Each ``trailer`` is then evaluated from left to right, applying its own parameter (either its arguments, subscripts or attribute name) to produce the value for the next ``trailer``. Finally, if present, ``await`` is applied.

For example, ``await a.b(c).d[e]`` is currently parsed as ``['await', 'a', '.b', '(c)', '.d', '[e]']`` and evaluated::

 _v = a
 _v = _v.b
 _v = _v(c)
 _v = _v.d
 _v = _v[e]
 await _v

When a ``None``-aware operator is present, the left-to-right evaluation may be short-circuited. For example, ``await a?.b(c).d?[e]`` is evaluated::

 _v = a
 if _v is not None:
     _v = _v.b
     _v = _v(c)
     _v = _v.d
     if _v is not None:
         _v = _v[e]
 await _v

.. note:: ``await`` will almost certainly fail in this context, as it would in the case where code attempts ``await None``. We are not proposing to add a ``None``-aware ``await`` keyword here, and merely include it in this example for completeness of the specification, since the ``atom_expr`` grammar rule includes the keyword. If it were in its own rule, we would have never mentioned it.

Parenthesised expressions are handled by the ``atom`` rule (not shown above), which will implicitly terminate the short-circuiting behaviour of the above transformation. For example, ``(a?.b ?? c).d?.e`` is evaluated as::

 # a?.b
 _v = a
 if _v is not None:
     _v = _v.b

 # ... ?? c
 if _v is None:
     _v = c

 # (...).d?.e
 _v = _v.d
 if _v is not None:
     _v = _v.e

When used as an assignment target, the ``None``-aware operations may only be used in a "load" context. That is, ``a?.b = 1`` and ``a?[b] = 1`` will raise ``SyntaxError``. Use earlier in the expression (``a?.b.c = 1``) is permitted, though unlikely to be useful unless combined with a coalescing operation::

 (a?.b ?? d).c = 1

Examples

This section presents some examples of common ``None`` patterns and shows what conversion to use ``None``-aware operators may look like.

Standard Library

Using the ``find-pep505.py`` script[3]_ an analysis of the Python 3.7 standard library discovered up to 678 code snippets that could be replaced with use of one of the ``None``-aware operators::

 $ find /usr/lib/python3.7 -name '*.py' | xargs python3.7

find-pep505.py <snip> Total None-coalescing `if` blocks: 449 Total [possible] None-coalescing `or`: 120 Total None-coalescing ternaries: 27 Total Safe navigation `and`: 13 Total Safe navigation `if` blocks: 61 Total Safe navigation ternaries: 8

Some of these are shown below as examples before and after converting to use the new operators.

From ``bisect.py``::

 def insort_right(a, x, lo=0, hi=None):
     # ...
     if hi is None:
         hi = len(a)
     # ...

After updating to use the ``??=`` augmented assignment statement::

 def insort_right(a, x, lo=0, hi=None):
     # ...
     hi ??= len(a)
     # ...

From ``calendar.py``::

 encoding = options.encoding
 if encoding is None:
     encoding = sys.getdefaultencoding()
 optdict = dict(encoding=encoding, css=options.css)

After updating to use the ``??`` operator::

 optdict = dict(encoding=encoding ?? sys.getdefaultencoding(),
                css=options.css)

From ``dis.py``::

 def _get_const_info(const_index, const_list):
     argval = const_index
     if const_list is not None:
         argval = const_list[const_index]
     return argval, repr(argval)

After updating to use the ``?[]`` and ``??`` operators::

 def _get_const_info(const_index, const_list):
     argval = const_list?[const_index] ?? const_index
     return argval, repr(argval)

From ``inspect.py``::

 for base in object.__bases__:
     for name in getattr(base, "__abstractmethods__", ()):
         value = getattr(object, name, None)
         if getattr(value, "__isabstractmethod__", False):
             return True

After updating to use the ``?.`` operator (and deliberately not converting to use ``any()``)::

 for base in object.__bases__:
     for name in base?.__abstractmethods__ ?? ():
         if object?.name?.__isabstractmethod__:
             return True

From ``os.py``::

 if entry.is_dir():
     dirs.append(name)
     if entries is not None:
         entries.append(entry)
 else:
     nondirs.append(name)

After updating to use the ``?.`` operator::

 if entry.is_dir():
     dirs.append(name)
     entries?.append(entry)
 else:
     nondirs.append(name)

jsonify

This example is from a Python web crawler that uses the Flask framework as its front-end. This function retrieves information about a web site from a SQL database and formats it as JSON to send to an HTTP client::

 class SiteView(FlaskView):
     @route('/site/<id_>', methods=['GET'])
     def get_site(self, id_):
         site = db.query('site_table').find(id_)

         return jsonify(
             first_seen=site.first_seen.isoformat() if

site.first_seen is not None else None, id=site.id, is_active=site.is_active, last_seen=site.last_seen.isoformat() if site.last_seen is not None else None, url=site.url.rstrip('/') )

Both ``first_seen`` and ``last_seen`` are allowed to be ``null`` in the database, and they are also allowed to be ``null`` in the JSON response. JSON does not have a native way to represent a ``datetime``, so the server's contract states that any non-``null`` date is represented as an ISO-8601 string.

Without knowing the exact semantics of the ``first_seen`` and ``last_seen`` attributes, it is impossible to know whether the attribute can be safely or performantly accessed multiple times.

One way to fix this code is to replace each conditional expression with an explicit value assignment and a full ``if``/``else`` block::

 class SiteView(FlaskView):
     @route('/site/<id_>', methods=['GET'])
     def get_site(self, id_):
         site = db.query('site_table').find(id_)

         first_seen_dt = site.first_seen
         if first_seen_dt is None:
             first_seen = None
         else:
             first_seen = first_seen_dt.isoformat()

         last_seen_dt = site.last_seen
         if last_seen_dt is None:
             last_seen = None
         else:
             last_seen = last_seen_dt.isoformat()

         return jsonify(
             first_seen=first_seen,
             id=site.id,
             is_active=site.is_active,
             last_seen=last_seen,
             url=site.url.rstrip('/')
         )

This adds ten lines of code and four new code paths to the function, dramatically increasing the apparent complexity. Rewriting using the ``None``-aware attribute operator results in shorter code with more clear intent::

 class SiteView(FlaskView):
     @route('/site/<id_>', methods=['GET'])
     def get_site(self, id_):
         site = db.query('site_table').find(id_)

         return jsonify(
             first_seen=site.first_seen?.isoformat(),
             id=site.id,
             is_active=site.is_active,
             last_seen=site.last_seen?.isoformat(),
             url=site.url.rstrip('/')
         )

Grab

The next example is from a Python scraping library called `Grab < https://github.com/lorien/grab/blob/4c95b18dcb0fa88eeca81f5643c0ebfb114bf728... ab/upload.py https://github.com/lorien/grab/blob/4c95b18dcb0fa88eeca81f5643c0ebfb114bf728/grab/upload.py

...

`_::

 class BaseUploadObject(object):
     def find_content_type(self, filename):
         ctype, encoding = mimetypes.guess_type(filename)
         if ctype is None:
             return 'application/octet-stream'
         else:
             return ctype

 class UploadContent(BaseUploadObject):
     def __init__(self, content, filename=None, content_type=None):
         self.content = content
         if filename is None:
             self.filename = self.get_random_filename()
         else:
             self.filename = filename
         if content_type is None:
             self.content_type = self.find_content_type(self.filename)
         else:
             self.content_type = content_type

 class UploadFile(BaseUploadObject):
     def __init__(self, path, filename=None, content_type=None):
         self.path = path
         if filename is None:
             self.filename = os.path.split(path)[1]
         else:
             self.filename = filename
         if content_type is None:
             self.content_type = self.find_content_type(self.filename)
         else:
             self.content_type = content_type

This example contains several good examples of needing to provide default values. Rewriting to use conditional expressions reduces the overall lines of code, but does not necessarily improve readability::

 class BaseUploadObject(object):
     def find_content_type(self, filename):
         ctype, encoding = mimetypes.guess_type(filename)
         return 'application/octet-stream' if ctype is None else ctype

 class UploadContent(BaseUploadObject):
     def __init__(self, content, filename=None, content_type=None):
         self.content = content
         self.filename = (self.get_random_filename() if filename
             is None else filename)
         self.content_type = (self.find_content_type(self.filename)
             if content_type is None else content_type)

 class UploadFile(BaseUploadObject):
     def __init__(self, path, filename=None, content_type=None):
         self.path = path
         self.filename = (os.path.split(path)[1] if filename is
             None else filename)
         self.content_type = (self.find_content_type(self.filename)
             if content_type is None else content_type)

The first ternary expression is tidy, but it reverses the intuitive order of the operands: it should return ``ctype`` if it has a value and use the string literal as fallback. The other ternary expressions are unintuitive and so long that they must be wrapped. The overall readability is worsened, not improved.

Rewriting using the ``None`` coalescing operator::

 class BaseUploadObject(object):
     def find_content_type(self, filename):
         ctype, encoding = mimetypes.guess_type(filename)
         return ctype ?? 'application/octet-stream'

 class UploadContent(BaseUploadObject):
     def __init__(self, content, filename=None, content_type=None):
         self.content = content
         self.filename = filename ?? self.get_random_filename()
         self.content_type = content_type ??

self.find_content_type(self.filename)

 class UploadFile(BaseUploadObject):
     def __init__(self, path, filename=None, content_type=None):
         self.path = path
         self.filename = filename ?? os.path.split(path)[1]
         self.content_type = content_type ??

self.find_content_type(self.filename)

This syntax has an intuitive ordering of the operands. In ``find_content_type``, for example, the preferred value ``ctype`` appears before the fallback value. The terseness of the syntax also makes for fewer lines of code and less code to visually parse, and reading from left-to-right and top-to-bottom more accurately follows the execution flow.

Rejected Ideas

The first three ideas in this section are oft-proposed alternatives to treating ``None`` as special. For further background on why these are rejected, see their treatment in `PEP 531 https://www.python.org/dev/peps/pep-0531/`_ and `PEP 532 https://www.python.org/dev/peps/pep-0532/`_ and the associated discussions.

No-Value Protocol

The operators could be generalised to user-defined types by defining a protocol to indicate when a value represents "no value". Such a protocol may be a dunder method ``__has_value__(self)` that returns ``True`` if the value should be treated as having a value, and ``False`` if the value should be treated as no value.

With this generalization, ``object`` would implement a dunder method equivalent to this::

 def __has_value__(self):
     return True

``NoneType`` would implement a dunder method equivalent to this::

 def __has_value__(self):
     return False

In the specification section, all uses of ``x is None`` would be replaced with ``not x.__has_value__()``.

This generalization would allow for domain-specific "no-value" objects to be coalesced just like ``None``. For example the ``pyasn1`` package has a type called ``Null`` that represents an ASN.1 ``null``::

 >>> from pyasn1.type import univ
 >>> univ.Null() ?? univ.Integer(123)
 Integer(123)

Similarly, values such as ``math.nan`` and ``NotImplemented`` could be treated as representing no value.

However, the "no-value" nature of these values is domain-specific, which means they *should* be treated as a value by the language. For example, ``math.nan.imag`` is well defined (it's ``0.0``), and so short-circuiting ``math.nan?.imag`` to return ``math.nan`` would be incorrect.

As ``None`` is already defined by the language as being the value that represents "no value", and the current specification would not preclude switching to a protocol in the future (though changes to built-in objects would not be compatible), this idea is rejected for now.

Boolean-aware operators

This suggestion is fundamentally the same as adding a no-value protocol, and so the discussion above also applies.

Similar behavior to the ``??`` operator can be achieved with an ``or`` expression, however ``or`` checks whether its left operand is false-y and not specifically ``None``. This approach is attractive, as it requires fewer changes to the language, but ultimately does not solve the underlying problem correctly.

Assuming the check is for truthiness rather than ``None``, there is no longer a need for the ``??`` operator. However, applying this check to the ``?.`` and ``?[]`` operators prevents perfectly valid operations applying

Consider the following example, where ``get_log_list()`` may return either a list containing current log messages (potentially empty), or ``None`` if logging is not enabled::

 lst = get_log_list()
 lst?.append('A log message')

If ``?.`` is checking for true values rather than specifically ``None`` and the log has not been initialized with any items, no item will ever be appended. This violates the obvious intent of the code, which is to append an item. The ``append`` method is available on an empty list, as are all other list methods, and there is no reason to assume that these members should not be used because the list is presently empty.

Further, there is no sensible result to use in place of the expression. A normal ``lst.append`` returns ``None``, but under this idea ``lst?.append`` may result in either ``[]`` or ``None``, depending on the value of ``lst``. As with the examples in the previous section, this makes no sense.

As checking for truthiness rather than ``None`` results in apparently valid expressions no longer executing as intended, this idea is rejected.

Exception-aware operators

Arguably, the reason to short-circuit an expression when ``None`` is encountered is to avoid the ``AttributeError`` or ``TypeError`` that would be raised under normal circumstances. As an alternative to testing for ``None``, the ``?.`` and ``?[]`` operators could instead handle ``AttributeError`` and ``TypeError`` raised by the operation and skip the remainder of the expression.

This produces a transformation for ``a?.b.c?.d.e`` similar to this::

 _v = a
 try:
     _v = _v.b
 except AttributeError:
     pass
 else:
     _v = _v.c
     try:
         _v = _v.d
     except AttributeError:
         pass
     else:
         _v = _v.e

One open question is which value should be returned as the expression when an exception is handled. The above example simply leaves the partial result, but this is not helpful for replacing with a default value. An alternative would be to force the result to ``None``, which then raises the question as to why ``None`` is special enough to be the result but not special enough to be the test.

Secondly, this approach masks errors within code executed implicitly as part of the expression. For ``?.``, any ``AttributeError`` within a property or ``__getattr__`` implementation would be hidden, and similarly for ``?[]`` and ``__getitem__`` implementations.

Similarly, simple typing errors such as ``{}?.ietms()`` could go unnoticed.

Existing conventions for handling these kinds of errors in the form of the ``getattr`` builtin and the ``.get(key, default)`` method pattern established by ``dict`` show that it is already possible to explicitly use this behaviour.

As this approach would hide errors in code, it is rejected.

``None``-aware Function Call

The ``None``-aware syntax applies to attribute and index access, so it seems natural to ask if it should also apply to function invocation syntax. It might be written as ``foo?()``, where ``foo`` is only called if it is not None.

This has been deferred on the basis of the proposed operators being intended to aid traversal of partially populated hierarchical data structures, *not* for traversal of arbitrary class hierarchies. This is reflected in the fact that none of the other mainstream languages that already offer this syntax have found it worthwhile to support a similar syntax for optional function invocations.

A workaround similar to that used by C# would be to write ``maybe_none?.__call__(arguments)``. If the callable is ``None``, the expression will not be evaluated. (The C# equivalent uses ``?.Invoke()`` on its callable type.)

``?`` Unary Postfix Operator

To generalize the ``None``-aware behavior and limit the number of new operators introduced, a unary, postfix operator spelled ``?`` was suggested. The idea is that ``?`` might return a special object that could would override dunder methods that return ``self``. For example, ``foo?`` would evaluate to ``foo`` if it is not ``None``, otherwise it would evaluate to an instance of ``NoneQuestion``::

 class NoneQuestion():
     def __call__(self, *args, **kwargs):
         return self

     def __getattr__(self, name):
         return self

     def __getitem__(self, key):
         return self

With this new operator and new type, an expression like ``foo?.bar[baz]`` evaluates to ``NoneQuestion`` if ``foo`` is None. This is a nifty generalization, but it's difficult to use in practice since most existing code won't know what ``NoneQuestion`` is.

Going back to one of the motivating examples above, consider the following::

 >>> import json
 >>> created = None
 >>> json.dumps({'created': created?.isoformat()})``

The JSON serializer does not know how to serialize ``NoneQuestion``, nor will any other API. This proposal actually requires *lots of specialized logic* throughout the standard library and any third party library.

At the same time, the ``?`` operator may also be **too general**, in the sense that it can be combined with any other operator. What should the following expressions mean?::

 >>> x? + 1
 >>> x? -= 1
 >>> x? == 1
 >>> ~x?

This degree of generalization is not useful. The operators actually proposed herein are intentionally limited to a few operators that are expected to make it easier to write common code patterns.

Built-in ``maybe``

Haskell has a concept called `Maybe https://wiki.haskell.org/Maybe`_ that encapsulates the idea of an optional value without relying on any special keyword (e.g. ``null``) or any special instance (e.g. ``None``). In Haskell, the purpose of ``Maybe`` is to avoid separate handling of "something" and nothing".

A Python package called `pymaybe https://pypi.org/p/pymaybe/`_ provides a rough approximation. The documentation shows the following example::

 >>> maybe('VALUE').lower()
 'value'

 >>> maybe(None).invalid().method().or_else('unknown')
 'unknown'

The function ``maybe()`` returns either a ``Something`` instance or a ``Nothing`` instance. Similar to the unary postfix operator described in the previous section, ``Nothing`` overrides dunder methods in order to allow chaining on a missing value.

Note that ``or_else()`` is eventually required to retrieve the underlying value from ``pymaybe``'s wrappers. Furthermore, ``pymaybe`` does not short circuit any evaluation. Although ``pymaybe`` has some strengths and may be useful in its own right, it also demonstrates why a pure Python implementation of coalescing is not nearly as powerful as support built into the language.

The idea of adding a builtin ``maybe`` type to enable this scenario is rejected.

Just use a conditional expression

Another common way to initialize default values is to use the ternary operator. Here is an excerpt from the popular `Requests package < https://github.com/kennethreitz/requests/blob/14a555ac716866678bf17e43e23230... a8149f5/requests/models.py#L212 https://github.com/kennethreitz/requests/blob/14a555ac716866678bf17e43e23230d81a8149f5/requests/models.py#L212

...

`_::

 data = [] if data is None else data
 files = [] if files is None else files
 headers = {} if headers is None else headers
 params = {} if params is None else params
 hooks = {} if hooks is None else hooks

This particular formulation has the undesirable effect of putting the operands in an unintuitive order: the brain thinks, "use ``data`` if possible and use ``[]`` as a fallback," but the code puts the fallback *before* the preferred value.

The author of this package could have written it like this instead::

 data = data if data is not None else []
 files = files if files is not None else []
 headers = headers if headers is not None else {}
 params = params if params is not None else {}
 hooks = hooks if hooks is not None else {}

This ordering of the operands is more intuitive, but it requires 4 extra characters (for "not "). It also highlights the repetition of identifiers: ``data if data``, ``files if files``, etc.

When written using the ``None`` coalescing operator, the sample reads::

 data = data ?? []
 files = files ?? []
 headers = headers ?? {}
 params = params ?? {}
 hooks = hooks ?? {}

References

.. [1] C# Reference: Operators (https://msdn.microsoft.com/en-us/library/6a71f45d.aspx)

.. [2] A Tour of the Dart Language: Operators ( https://www.dartlang.org/docs/dart-up-and-running/ch02.html#operators)

.. [3] Associated scripts (https://github.com/python/peps/tree/master/pep-0505/)

Copyright

This document has been placed in the public domain. _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

"APL and Perl territory" means "use lots of punctuation characters in somewhat cryptic ways, often combining several for a distinct semantics." I did not mean "APL and Perl use those specific characters with the proposed meaning."

On Thu, Jul 19, 2018, 9:39 AM Steven D'Aprano steve@pearwood.info wrote:

On Wed, Jul 18, 2018 at 08:05:56PM -0400, David Mertz wrote:

...

'??' and '?.' and ?[]' are really just marching into APL and Perl territory. Yes, I know such operators exist in other languages, but it feels very unpythonic.

If these features exist in "other languages", and *don't* exist in APL or Perl, how are they marching into APL and Perl territory?

Perl territory, like this perhaps?

print "hello world\n"; @days = ("Monday", "Tuesday", "Wednesday"); print $days[0]

Yes, I can see why we use "Perl syntax" as an insult *wink*

Okay, okay, my examples are a bit unfair. I deliberately chose examples where the syntax is almost identical to Python's. Aside from the array and scalar sigils @ and $ the above could be Python.

Tens of thousands of non-English speakers have had to learn the meaning of what might as well be meaningless, random sets of symbols (to them) like "class", "import", "while" and "True". If they can do so, perhaps we English-speakers should stop complaining about how hard it is to memorise the meaning of a couple of symbols like ??. Surely its no more difficult than learning the various meanings of ** and [] which we've already done.

*Its just spelling*. If it is a useful and well-defined feature, we'll get used to the spelling soon enough.

That's not to say that spelling is not important *at all*, or that we should never prefer words to symbols. But if the only objection we have is "this is useful but I don't like the spelling so -1" then that's usually a pretty weak argument against the feature.

-- Steve _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On 2018-07-19 06:38, Steven D'Aprano wrote:

*Its just spelling*. If it is a useful and well-defined feature, we'll get used to the spelling soon enough.

That's not to say that spelling is not important *at all*, or that we should never prefer words to symbols. But if the only objection we have is "this is useful but I don't like the spelling so -1" then that's usually a pretty weak argument against the feature.

But we already have a spelling for the most common case. It is:

x = a if a is not None else b

That is the only use case of any of these operators that is actually common enough for me to care about --- but it's still not common enough to warrant the creation of a new operator, let alone multiple new operators.

On Thu, Jul 19, 2018, 9:39 AM Steven D'Aprano steve@pearwood.info wrote:

*Its just spelling*. If it is a useful and well-defined feature, we'll get used to the spelling soon enough.

That's not to say that spelling is not important *at all*, or that we should never prefer words to symbols. But if the only objection we have is "this is useful but I don't like the spelling so -1" then that's usually a pretty weak argument against the feature.

I could not disagree more. Spelling is extremely important to readability, and that only goes away partially with familiarity. As a teaching language, moreover, there will always be a late share of Python users who haven't become familiar, and for whom "executable pseudo-code" is a big advantage.

I don't believe Python should be ONLY for those who do not already know it, but neither should it be even half so arcane as this PEP would make it.

Here's are well defined features of the J programming language, for comparison... very compact too:

For example, in the following contrived expression the exclamation point https://en.m.wikipedia.org/wiki/Exclamation_point !refers to three distinct functions:

2!!7!4

The function below can be used to list all of the prime numbers between 1 and R with:

2_&{&/x!/:2_!x}'!R

On Sun, Jul 22, 2018 at 01:56:35AM +0200, Giampaolo Rodola' wrote:

On Thu, Jul 19, 2018 at 3:39 PM Steven D'Aprano steve@pearwood.info wrote:

...

Tens of thousands of non-English speakers have had to learn the meaning of what might as well be meaningless, random sets of symbols (to them) like "class", "import", "while" and "True". If they can do so, perhaps we English-speakers should stop complaining about how hard it is to memorise the meaning of a couple of symbols like ??.

"class", "import", "while" and "True" are keywords, not symbols.

They are only key WORDS if you are an English speaker. If your language doesn't use the Latin script, they don't even look like words. They look like gibberish: ∌≇⊅∇∫

A symbol is more cryptic than a keyword

I was using "symbol" in its most general sense, "a visible sign or representation of an idea". Words, letters, icons, emblems etc are all symbols. Sorry for being unclear.

hence it comes at a higher cost in terms of readability.

It is not clear to me that this is always true even when it comes to non-word symbols. I don't think that "+" is harder to read than "standard_mathematics_operators_numeric_addition" for example.

It is possible to be too verbose as well as too terse. This is why we aren't copying COBOL.

which is the reason why conditionals use keywords instead symbols:

• we say "and" instead of "&&"

[...]

Indeed we do. But we also say:

- we say "+" instead of "add" - we say "//" instead of "floor division" - we say "**" instead of "exponentiation" - we say "&" instead of "bitwise AND" - we say "f( ... )" instead of "call f with arguments ..."

etc. Python has no shortage of non-word symbols:

== != ~ - + * ** / // @ % ^ & | << >> < <= > >= [] ()

[...]

...

*Its just spelling*. If it is a useful and well-defined feature, we'll get used to the spelling soon enough.

Such an argument may apply to other languages but not Python.

I disagree. I think it applies equally to Python. Python functions and keywords are usually English words, but not always:

def elif

have to be memorised even by English speakers. If we gained a function or even a keyword from Italian, let's say "ripetere", would that really change the nature of Python? I don't think so. English speakers are adaptable, we don't so much borrow words from other languages as chase them down the alley and mug them for new vocabulary.

The same applies to non-word symbols. Look at how quickly and easily people adapted to @ (the commercial "at" sign, a variation of multiplication) as a separator in emails, not to mention #hashtags.

I'll admit that the number and variety of new operators gives me some reason to pause, but for the simplest and most obvious case, the proposed ?? operator, I think that the fears about readability are grossly exaggerated.

The philosophy of the language is all about readability and beauty since day 1, and the main reason why Python got so successful despite being slow.

Let's talk about Python's readability: the first time I saw Python code, I had *no idea* how to read it. This was Python 1.5, long before comprehensions, @ decorator syntax, etc. I could read Pascal, Forth, Hypertalk, BASIC and Fortran, and I looked at Python and couldn't make heads or tails of it. It was full of cryptic idioms like:

for i in range(len(sequence))

which is meaningless until you learn what range does and learn to read it as a for loop:

for i = 0 to len(sequence) - 1

And as for slice notation:

suffix = word[n:]

that might as well have been advanced quantum mechanics transliterated into Korean by a native Navaho speaker for all I could read it. I knew round brackets were used for function calls f() but the only experience I had with square and curly brackets was in mathematics, where they are used as second and third levels of brackets:

x = {[a (b+1)][b (a - 1)] - 1}/2

Now *that* I understood. What on earth was this mysterious {'a': 1} syntax that I keep seeing in Python code?

My initial experience with this bizarre, cryptic language Python was so off-putting that I put it aside for literally three or four years before coming back to it.

Of course I believe that Python is, generally speaking, a beautiful and elegant language, extremely readable. But readability doesn't exist in a vacuum. We still need to learn the symbols of the language, not just operators like ** % and // but words as well. To appreciate the elegance of the language, we need to know the common idioms, as well as the semantics of keywords and functions.

Unfortunately, ?? does have one disadvantage: even though it is used as an operator in a number of languages, Google doesn't seem to have indexed it. Goggling for "??" is unhelpful. But then googling for "@ python" is similarly unhelpful. Clearly decorators was a mistake. /s

That's why we have mandatory indentation, to say one. We don't *have to* get used to idioms which can decrease readability. When we do there must be a good reason and the spelling should matter (I remember the debates about decorators' syntax).

Exactly. Years later, do we still think that @decorator syntax is unreadable and unnecessary?

In my opinion, writing

expression if expression is None else default

is the *opposite* of Pythonic, it is verbose and the DRY violation is inelegant (as well as inefficient). I'd much rather use:

expression ?? default

although with PEP 572 approved, there is an alternative:

temp := expression if temp is None else default

which avoids the DRY violation but is more verbose than even the first version.

On 22 July 2018 at 02:54, Steven D'Aprano steve@pearwood.info wrote:

I'll admit that the number and variety of new operators gives me some reason to pause, but for the simplest and most obvious case, the proposed ?? operator, I think that the fears about readability are grossly exaggerated.

Certainly *my* concerns about readability are around the other proposed operators (?[ and ?. in particular).

In my opinion, writing

expression if expression is None else default

is the *opposite* of Pythonic, it is verbose and the DRY violation is inelegant (as well as inefficient). I'd much rather use:

expression ?? default

Agreed. But the PEP proposes three other operators, and it's not at all clear to me that those are such clear wins.

Paul

On Sun, Jul 22, 2018 at 3:55 AM Steven D'Aprano steve@pearwood.info wrote:

Indeed we do. But we also say:

• we say "+" instead of "add"
• we say "//" instead of "floor division"
• we say "**" instead of "exponentiation"
• we say "&" instead of "bitwise AND"
• we say "f( ... )" instead of "call f with arguments ..."

[...]

I don't think that "+" is harder to read than

"standard_mathematics_operators_numeric_addition"

Please let's drop the argument that + - * / = and ? are the same. They clearly are not. Anybody learned those symbols at elementary schools, all programming languages have them and using math in programming is common enough to justify a symbol over a keyword. "a + b" is literally just an addition and nothing else. The "?" variants have multiple meanings, spellings and implications:

- "a ?? b" means "b is chosen over a if a is None"

- "a ??= b" means "a is set to b if a is None"

- "a?.b" means "a.b is executed but only if a is not None"

- "a?[2] ?? 3" means "index 2 of list a is picked up if a is not None, else use 3"

"a?.b"and "a?[2]" in particular go way beyond the mere "it's not pretty" argument which, I concur, can be subjective, as you don't know where evaluation stops. Even "a ??= b" goes beyond that as it introduces yet another assignment operator (the third, as we now have = and :=). So again, I don't think it's fair to dismiss the whole thing as "it's just another symbol" or "it's like a + b".

As for bitwise operators: they are kinda obscure and low-levelish and when I bump into them I still have to pause to reason what's going on. The difference with ? though is that you basically have no other way to do the same thing. Also they are much more rare and also are present in many other languages since... forever.

On Sun, Jul 22, 2018 at 12:26 PM Paul Moore p.f.moore@gmail.com wrote:

On 22 July 2018 at 11:13, Giampaolo Rodola' g.rodola@gmail.com wrote:

...

• "a?[2] ?? 3" means "index 2 of list a is picked up if a is not None,

else

...

use 3"

Actually, doesn't it mean

if a is not None, pick up index 2 of the list. If a is None, OR IF a[2] IS NONE, then use 3. If a is None but a[2] is not None, use a[2].

?

Which is subtly different, and probably at least as prone to "accidental" errors as some of the constructs the None-aware operators are intended to replace.

Paul

Yes, I think you're right.

On Sun, Jul 22, 2018 at 12:13:04PM +0200, Giampaolo Rodola' wrote:

On Sun, Jul 22, 2018 at 3:55 AM Steven D'Aprano steve@pearwood.info wrote:

[...]

...

I don't think that "+" is harder to read than "standard_mathematics_operators_numeric_addition"

Please let's drop the argument that + - * / = and ? are the same.

In context, the argument was that non-word symbols are not always worse than words, not that all symbols are "the same".

Obviously symbols that we use regularly will be more familiar and easier to recognise than symbols we use rarely. I still have to make a conscious action to recall which of ∩ and ∪ is set union and intersection, and don't ask me what set symmetric difference is. And even after 20 years of Python I still occasionally write ^ for exponentiation instead of **.

But if we insist that every symbol we use is instantly recognisable and intuitively obvious to every programmer, we're putting the bar for acceptance impossibly high.

They clearly are not. Anybody learned those symbols at elementary schools, all programming languages have them and using math in programming is common enough to justify a symbol over a keyword. "a + b" is literally just an addition and nothing else.

That's not quite correct: '+' in Python is used for both addition and sequence concatenation. And with operator overloading, it can be anything at all. But it is *usually* addition, or concatenation.

I don't know what it is like in your country, but here in Australia, I don't know any school that teaches * for multiplication except in programming classes, which is not a core subject. The usual symbol we have for multiplication is × and sometimes ⋅ (dot operator).

This is a good point: after learning * for multiplication, it becomes so familiar that most of us forget that we haven't been using it forever. It becomes second-nature. In the same way that @ for decorators has become second nature, or slice notation. Both of which are terribly mysterious to people just starting out.

We shouldn't judge proposals on how mysterious they are the first time we see them, because everything is mysterious the first time. We should try to look forward to when we've seen them ten or twenty times. How will the "usefulness versus surprise" trade-off appear when, let's say, 50% of the surprise has been worn away with experience?

As a community, we're risk-adverse. I understand why we should be conservative in what we add to the language (once added, it cannot easily be removed if it turns out to be a mistake) but on Python-Ideas we regularly demand levels of obviousness and "readability" that existing syntax does not reach.

(For example, the dot operator for attribute access fails the "syntax should not look like grit on Tim's monitor" test.)

I believe that it is fine to prefer that new syntax is no harder to learn or use than (for example) // or ** (neither of which is taught in maths class), or slice notation. But I don't think it is fair, or desirable, to demand levels of readability greater than what we already have in the less common corners of the language.

All the obvious operators are already in use. Anything we add now is going to be a little bit niche, a little bit unusual. It's not like we're going to suddenly realise we forgot to include a "subtract" operator. So it is perfectly natural that any new operators won't be as familiar as + or - operators. But it might become as familiar as ** or << operators, or some of the less common regex patterns, and that's okay. Not everything needs to be as recognisable as the plus sign.

The "?" variants have multiple meanings, spellings and implications:

[...]

Indeed. And I think we ought to think carefully about the benefits and costs of all of those variants separately.

To me, the ?? operator seems like a clear and obvious win. The other variants are more complex and the benefit is not as obvious to me, so I haven't decided where I stand on them.

On Sun, Jul 22, 2018, 8:11 AM Steven D'Aprano steve@pearwood.info wrote:

On Sun, Jul 22, 2018 at 12:13:04PM +0200, Giampaolo Rodola' wrote:

...

Please let's drop the argument that + - * / = and ? are the same.

We shouldn't judge proposals on how mysterious they are the first time we see them, because everything is mysterious the first time. All the obvious operators are already in use. Anything we add now is going to be a little bit niche, a little bit unusual. It's not like we're going to suddenly realise we forgot to include a "subtract" operator. So it is perfectly natural that any new operators won't be as familiar as + or - operators. But it might become as familiar as ** or << operators, or some of the less common regex patterns,

I'm glad you've moved to better acknowledging the relative familiarity of symbols. This simple can't be brushed away as a non-concern, as you did in prior posts.

We're not promoting or reaching Python to Martians with entirely different systems of writing. Nor, for that matter, even to humans of the 14th entirety CE, before some basic math symbology became universal.

+, -, /, <, >, <=, >= are taught to small children throughout the world, over the last 5 or more generations (the compound inequalities drawn in combined glyphs, but iconically very close). Yes they are characters with no inherent meaning, but they are as well known as the sounds of Latin letters, or probably more. '*' is a bit more computer specific. 'x' or some nicer typographic variation is taught to children. But '*' is close to universal in computer languages. '**' is funny. It's used in many computer languages, but ^ is also a commonplace and probably more iconic for "superscript", the actual middle school notation on exponentiation.

Actually, I have a ten Pythin line command-line calculator that accepts 'x' and '^' in their "obvious" meanings.

The various question mark operators are only reasonable to compare to e.g. @ for either __matmul__ or introducing decorators. Or to bitwise operators like |, ~, ^, <<. But in both of those areas, the programs are ones dealing with specialized domains where "experts" are necessary to understand the semantics, not only the iconography.

Here the question mark operators are very different. They are proposed as general constructs with no particularly special area of use. Programmers or hobbyists who come from different or no programming language would encounter them. Even as a scientific Python export, my linear algebra is weak... if I see code with a bunch of '@' marks I know I'm not really going to understand it, but before I look I'm warned about the domain difficulties by the comments, function names, and so on.

The question marks are basic—though obscure and hard to reason about–flow control.

To me, the ?? operator seems like a clear and obvious win. The other

variants are more complex and the benefit is not as obvious to me, so I haven't decided where I stand on them.

Yes, that is the only one I'm merely -0 on. The others I'm -1000. Even experienced developers in this thread keep tripping over the actual semantics of those others, and the PEP proposer is unsure about what's best for some edge behaviors that are actually really important were they to be adopted.

The most any of these can possibly do is save a few characters in a ternary line, or perhaps a few lines of if/elif that make the intention far more obvious.

On Sun, Jul 22, 2018 at 11:22 PM, Grégory Lielens gregory.lielens@gmail.com wrote:

The ?? operator is probably the less scary one regarding legibility, and in guessing (or remembering) what it exactly does... Well, at least I think I understand what it does exactly, but if I'm not wrong there, what it does is also quite simple and minimal.

A function returning it's first non-None argument (or None, if all args are None) will provide the same functionality, with not much typing. You have parenthesis for the call, but you will probably need them anyway to group things, for correcting precedence, or helping the reader to parse your expression even if precedence was right. You have an extra call, so ?? may be more efficient...maybe.

Is that a reason enough, together with a few letters saved typing, to introduce ?? ? Not for me...

You forget that the operator will *short-circuit*. It will not evaluate the second argument if the first argument is None. You cannot do this with a function, other than with a hack like a lambda function.

THAT is reason enough for an operator.

ChrisA

On Sun, Jul 22, 2018 at 11:26:15PM +1000, Chris Angelico wrote:

You forget that the operator will *short-circuit*. It will not evaluate the second argument if the first argument is None. You cannot do this with a function, other than with a hack like a lambda function.

We keep running up to this issue. What if there was a language supported, non-hackish way to officially delay evaluation of expressions until explicitly requested?

That would allow us to write a function:

func(arg, default = delayed-expression)

and avoid new punctuation. Then the only bike-shedding will be where the function should live and what it is called and how many arguments it ought to take...

On Mon, Jul 23, 2018 at 10:48:23AM +0100, Steve Dower wrote:

On 23Jul2018 0151, Steven D'Aprano wrote:

...

What if there was a language supported, non-hackish way to officially delay evaluation of expressions until explicitly requested?

The current spelling for this is "lambda: delayed-expression" and the way to request the value is "()". :)

(I'm not even being that facetious here. People ask for delayed expressions all the time, and it's only 7 characters, provided the callee knows they're getting it, and the semantics are already well defined and likely match what you want.)

I know you not being facetious, and delaying computation through a function call is not an awful solution. But its not a great solution either. Contrast the elegance of syntax with delayed evaluation:

1/x if x != 0 else func(y)

versus the lambda solution:

if_else(lambda: 1/x, x != 0, lambda: func(y))()

For clarity, or perhaps the opposite *wink* I've kept the same order of arguments. It's not just the extra seven characters (plus spaces) per delayed expression, or the extra parentheses at the end to force evaluation, but the ease at which we can forget and write this:

if_else(1/x, x != 0, func(y))

Anyway, it was just an idle thought, not in a state to compete with this PEP. And even if it were, I'd still prefer to see at least ?? as a dedicated operator rather than a function.

On Sun, Jul 22, 2018 at 2:10 PM Steven D'Aprano steve@pearwood.info wrote:

On Sun, Jul 22, 2018 at 12:13:04PM +0200, Giampaolo Rodola' wrote:

...

On Sun, Jul 22, 2018 at 3:55 AM Steven D'Aprano steve@pearwood.info wrote:

[...]

...

...

I don't think that "+" is harder to read than "standard_mathematics_operators_numeric_addition"

Please let's drop the argument that + - * / = and ? are the same.

[...] But if we insist that every symbol we use is instantly recognisable and intuitively obvious to every programmer, we're putting the bar for acceptance impossibly high.

I personally don't find "a ?? b" too bad (let's say I'm -0 about it) but idioms such as "a?.b", "a ??= b" and "a?[3] ?? 4" look too Perl-ish to me, non pythonic and overall not explicit, no matter what the chosen symbol is gonna be. It looks like they want to do too much for the sole reason of allowing people to write more compact code and save a few lines. Compact code is not necessarily a good thing, especially when it comes at the expense of readability and explicitness, as I think is this case.

All the obvious operators are already in use. Anything we add now is going to be a little bit niche, a little bit unusual.

That's basically my point. And I know I'll sound very conservative here but to me that is a valid enough reason to not take action or be extremely careful at the very least. Not to state the obvious but it's not that we *have to* use the remaining unused symbols just because they're there.

On Sun, Jul 22, 2018 at 3:38 PM Chris Angelico rosuav@gmail.com wrote:

On Sun, Jul 22, 2018 at 11:35 PM, Giampaolo Rodola' g.rodola@gmail.com wrote:

...

On Sun, Jul 22, 2018 at 2:10 PM Steven D'Aprano steve@pearwood.info wrote:

...

On Sun, Jul 22, 2018 at 12:13:04PM +0200, Giampaolo Rodola' wrote:

...

On Sun, Jul 22, 2018 at 3:55 AM Steven D'Aprano steve@pearwood.info wrote:

[...]

...

...

I don't think that "+" is harder to read than "standard_mathematics_operators_numeric_addition"

Please let's drop the argument that + - * / = and ? are the same.

[...] But if we insist that every symbol we use is instantly recognisable and intuitively obvious to every programmer, we're putting the bar for acceptance impossibly high.

I personally don't find "a ?? b" too bad (let's say I'm -0 about it) but idioms such as "a?.b", "a ??= b" and "a?[3] ?? 4" look too Perl-ish to me, non pythonic and overall not explicit, no matter what the chosen symbol is gonna be.

Please explain what is not explicit about it. "a?.b" is very simple and perfectly explicit: it means "None if a is None else a.b". What does "not explicit" mean, other than "I don't like this code"?

I find it less explicit mainly because it does 3 things at once: check if attribute is None, use it if it's not None and continue the evaluation from left to right. I find that logic to be more explicit when living on different lines or is clearly delimited by keywords and spaces. ? has no spaces, it's literally "variable names interrupted by question marks" and evaluation can stop at any time while scanning the line from left to right. Multiple "?" can live on the same line so that's incentive to write one-liners, really, and to me one-liners are always less explicit than the same logic split on multiple lines.

On Sun, Jul 22, 2018 at 10:01 PM Chris Angelico rosuav@gmail.com wrote:

On Mon, Jul 23, 2018 at 1:09 AM, Giampaolo Rodola' g.rodola@gmail.com wrote:

...

On Sun, Jul 22, 2018 at 3:38 PM Chris Angelico rosuav@gmail.com wrote: I find it less explicit mainly because it does 3 things at once: check if attribute is None, use it if it's not None and continue the evaluation from left to right. I find that logic to be more explicit when living on different lines or is clearly delimited by keywords and spaces. ? has no spaces, it's literally "variable names interrupted by question marks" and evaluation can stop at any time while scanning the line from left to right. Multiple "?" can live on the same line so that's incentive to write one-liners, really, and to me one-liners are always less explicit than the same logic split on multiple lines.

Ah, I see what you mean. Well, think about what actually happens when you write "lst.sort()". In terms of "hidden behaviour", there is far FAR more of it in existing syntax than in the new proposals.

I am not sure I'm following you (what does lst.sort() have to do with "?"?).

Which is back to what Steven said: people demand such a high bar for new syntax that few existing pieces of syntax would pass it.

Probably. That's what happens when a language is mature. Personally I don't think that's a bad thing.

On Mon, Jul 23, 2018 at 6:43 AM, Giampaolo Rodola' g.rodola@gmail.com wrote:

On Sun, Jul 22, 2018 at 10:01 PM Chris Angelico rosuav@gmail.com wrote:

...

On Mon, Jul 23, 2018 at 1:09 AM, Giampaolo Rodola' g.rodola@gmail.com wrote:

...

On Sun, Jul 22, 2018 at 3:38 PM Chris Angelico rosuav@gmail.com wrote: I find it less explicit mainly because it does 3 things at once: check if attribute is None, use it if it's not None and continue the evaluation from left to right. I find that logic to be more explicit when living on different lines or is clearly delimited by keywords and spaces. ? has no spaces, it's literally "variable names interrupted by question marks" and evaluation can stop at any time while scanning the line from left to right. Multiple "?" can live on the same line so that's incentive to write one-liners, really, and to me one-liners are always less explicit than the same logic split on multiple lines.

Ah, I see what you mean. Well, think about what actually happens when you write "lst.sort()". In terms of "hidden behaviour", there is far FAR more of it in existing syntax than in the new proposals.

I am not sure I'm following you (what does lst.sort() have to do with "?"?).

The "." in "lst.sort" is an operator. How much hidden behaviour is there in that? Do you actually even know every possible thing that can happen? Don't feel bad if you don't - it's not an indictment of your quality as a programmer, but an acknowledgement that Python's attribute access is incredibly complicated.

...

Which is back to what Steven said: people demand such a high bar for new syntax that few existing pieces of syntax would pass it.

Probably. That's what happens when a language is mature. Personally I don't think that's a bad thing.

I do. It means people place crazily high demands on new proposals. Imagine if we were talking about people, rather than features in a language; imagine if, to join the Warriors Guild, you had to first slay a red dragon with nothing but a rusty dagger, despite none of the existing members having done so. Is that reasonable to ask? Can you say "well, the guild is mature now, so yeah, it's a good thing"?

ChrisA

On Sun, Jul 22, 2018, 4:56 PM Chris Angelico rosuav@gmail.com wrote:

It means people place crazily high demands on new proposals.

I think the bar has been much too low for introducing new features over the last 5 years or so. Internal changes like the new dictionary implementation are fine, but user-facing changes should be exceedingly rare in the base language. This proposal doesn't come remotely close to such a good standard.

I was consistently +0 on the 572 idea, as long as its worst excesses were trimmed, as in the final PEP. But after reading this discussion, I almost reconsider that opinion since its social effect seems to be a move towards accepting wild and unnecessary changes that "might be useful" for a few unusual programming patterns.

Honestly, if you want Perl, and as many different ways to approach each problem as there are programmers (each with their own syntax niche), that language continues to be fully working. I'm not even writing that to be dismissive... There are actually some pretty and interesting ideas over there. But I very much want Python not to be like that, and to do most of my work in a readable language with as few special characters/signils as feasible.

On Sun, Jul 22, 2018 at 11:51 PM Giampaolo Rodola' g.rodola@gmail.com wrote:

On Sun, Jul 22, 2018 at 10:55 PM Chris Angelico rosuav@gmail.com wrote:

...

On Mon, Jul 23, 2018 at 6:43 AM, Giampaolo Rodola' g.rodola@gmail.com wrote:

...

On Sun, Jul 22, 2018 at 10:01 PM Chris Angelico rosuav@gmail.com wrote:

...

On Mon, Jul 23, 2018 at 1:09 AM, Giampaolo Rodola' g.rodola@gmail.com wrote:

...

On Sun, Jul 22, 2018 at 3:38 PM Chris Angelico rosuav@gmail.com wrote: I find it less explicit mainly because it does 3 things at once: check if attribute is None, use it if it's not None and continue the evaluation from left to right. I find that logic to be more explicit when living on different lines or is clearly delimited by keywords and spaces. ? has no spaces, it's literally "variable names interrupted by question marks" and evaluation can stop at any time while scanning the line from left to right. Multiple "?" can live on the same line so that's incentive to write one-liners, really, and to me one-liners are always less explicit than the same logic split on multiple lines.

Ah, I see what you mean. Well, think about what actually happens when you write "lst.sort()". In terms of "hidden behaviour", there is far FAR more of it in existing syntax than in the new proposals.

I am not sure I'm following you (what does lst.sort() have to do with "?"?).

The "." in "lst.sort" is an operator. How much hidden behaviour is there in that? Do you actually even know every possible thing that can happen? Don't feel bad if you don't - it's not an indictment of your quality as a programmer, but an acknowledgement that Python's attribute access is incredibly complicated.

I'm going to engage into a discussion [...]

s/I'm going/I'm not going

On Mon, Jul 23, 2018 at 12:08 AM Chris Angelico rosuav@gmail.com wrote:

On Mon, Jul 23, 2018 at 7:51 AM, Giampaolo Rodola' g.rodola@gmail.com wrote:

...

On Sun, Jul 22, 2018 at 10:55 PM Chris Angelico rosuav@gmail.com wrote:

...

On Mon, Jul 23, 2018 at 6:43 AM, Giampaolo Rodola' g.rodola@gmail.com wrote:

...

On Sun, Jul 22, 2018 at 10:01 PM Chris Angelico rosuav@gmail.com wrote:

...

On Mon, Jul 23, 2018 at 1:09 AM, Giampaolo Rodola' g.rodola@gmail.com wrote:

...

On Sun, Jul 22, 2018 at 3:38 PM Chris Angelico rosuav@gmail.com wrote: I find it less explicit mainly because it does 3 things at once: check if attribute is None, use it if it's not None and continue the evaluation from left to right. I find that logic to be more explicit when living on different lines or is clearly delimited by keywords and spaces. ? has no spaces, it's literally "variable names interrupted by question marks" and evaluation can stop at any time while scanning the line from left to right. Multiple "?" can live on the same line so that's incentive to write one-liners, really, and to me one-liners are always less explicit than the same logic split on multiple lines.

Ah, I see what you mean. Well, think about what actually happens when you write "lst.sort()". In terms of "hidden behaviour", there is far FAR more of it in existing syntax than in the new proposals.

I am not sure I'm following you (what does lst.sort() have to do with "?"?).

The "." in "lst.sort" is an operator. How much hidden behaviour is there in that? Do you actually even know every possible thing that can happen? Don't feel bad if you don't - it's not an indictment of your quality as a programmer, but an acknowledgement that Python's attribute access is incredibly complicated.

I'm [not] going to engage into a discussion about the analogy between "?" and "." because simply there is none. It doesn't prove anything except that you're not really interested in having a serious discussion about the pros and cons of this PEP: you just want it to happen no matter what.

That's because the dot already exists in the language, and you have become so accustomed to it that you don't see it any more. You've just proven my point.

You're back at "since we have X that justifies the addition of Y" [1] and AFAICT that's the only argument you have provided so far in a 100+ messages discussion.

[1] https://mail.python.org/pipermail/python-ideas/2018-July/052068.html

On Mon, Jul 23, 2018 at 2:38 AM Steven D'Aprano steve@pearwood.info wrote:

On Mon, Jul 23, 2018 at 12:59:20AM +0200, Giampaolo Rodola' wrote:

...

You're back at "since we have X that justifies the addition of Y" [1] and AFAICT that's the only argument you have provided so far in a 100+ messages discussion.

The PEP itself justifies the addition of Y.

This thread seems to be unnecessarily heated.

Other languages have these operators, and so they aren't a wild idea. That said, from what I've seen, Swift optionals are very different things, and Python really has nothing like them.

In Python, is None really special enough to need an operator like this?

One issue for me is that the trivial case is already a one-liner:

if a is None: a = 10

And it works for other things too:

if a is -1: a = 10

if not a: a = 10

Again, is None special enough in Python to need this operator? I don't know.

And that leads to a simple question: how many times does this actually occur in real-world by python code? -- i.e. how many times do I want to check the value of an existing label, and, finding it is None (and None specifically), then assign it a value?

What does a scan through the existing core library say?

I'm +0 on this proposal.

Best wishes,

N

On Mon, Jul 23, 2018 at 8:08 AM Grégory Lielens gregory.lielens@gmail.com wrote:

On Monday, July 23, 2018 at 8:24:45 AM UTC+2, Nicholas Cole wrote:

...

And that leads to a simple question: how many times does this actually occur in real-world by python code? -- i.e. how many times do I want to check the value of an existing label, and, finding it is None (and None specifically), then assign it a value?

The PEP present a few examples.

I think the compactness and clarity is really gained when doing a descent into an "attribute tree", where any (or selected) members can be None, like getting back to the example by Steven (spell-corrected, and expanded):

meal = obj?.spam?.eggs.tomato?.cheese ?? "Mozzarella"

Where, to put the proposal in even better light, I have assumed that eggs instances always have a tomato attributes (i.e. are never None).

This is indeed much more compact that the current version, and arguably more readable (it's easier to parse once you know the syntax...but you have to know the special syntax that will be used for this kind of stuff only). The more you deepen the attribute access, the more you gain, but of course deep attribute access is not so common. The new operators may make deep attribute hierarchies (or deeply nested dicts/lists) slightly more common, and there also I do not think it's a good thing.

That above example looks terrible to read (to me). Yes, that's a subjective statement. This example from the PEP is even worse:

Example:

if entry.is_dir(): dirs.append(name) if entries is not None: entries.append(entry) else: nondirs.append(name)

After updating to use the ?. operator:

if entry.is_dir(): dirs.append(name) entries?.append(entry) else: nondirs.append(name)

In the first, it's totally clear to me when entries would be appended to and when it wouldn't. The second I had to read several times before I could see what was going on. The fact that it is already embedded in an if...else statement perhaps made it harder to understand.

I guess people will say that we will just get used to the new syntax, but I don't see the benefit of making that particular "if" statement more compact, and leaving all the others in place. Or to put it another way, I'm just not convinced that "None" is sufficiently special.

N.

On Mon, 23 Jul 2018 10:51:31 +0100 Steve Dower steve.dower@python.org wrote:

Which is the most important operator?

Personally, I think '?.' is the most valuable.

For me, it's the most contentious. The fact that a single '?' added to a regular line of Python code can short-circuit execution silently is a net detriment to readability, IMHO.

In a code review, this means I must be careful about '?' sigils lest I miss important bug magnets.

Regards

Antoine.

Le 23/07/2018 à 12:25, Steve Dower a écrit :

On 23Jul2018 1111, Antoine Pitrou wrote:

...

On Mon, 23 Jul 2018 10:51:31 +0100 Steve Dower steve.dower@python.org wrote:

...

Which is the most important operator?

Personally, I think '?.' is the most valuable.

For me, it's the most contentious. The fact that a single '?' added to a regular line of Python code can short-circuit execution silently is a net detriment to readability, IMHO.

The only time it would short-circuit is when it would otherwise raise AttributeError for trying to access an attribute from None, which is also going to short-circuit.

But AttributeError is going to bubble up as soon as it's raised, unless it's explicitly handled by an except block. Simply returning None may have silent undesired effects (perhaps even security flaws).

This whole thing reminds of PHP's malicious "@" operator.

Regards

Antoine.

Le 23/07/2018 à 12:38, Steve Dower a écrit :

General comment to everyone (not just Antoine): these arguments have zero value to me. Feel free to keep making them, but I am uninterested.

So you're uninterested in learning from past mistakes?

You sound like a child who thinks their demands should be satisfied because they are the center of the world.

Regards

Antoine.

Maybe it would help if you mention in which context you will benefit the most? If the python sub-community related to this context agree "?? and friends" is a good idea, then it will add weight to the proposal. Else, probably better to forget it.

It seems related to JSON, but as I have never used it, it's a wild guess.

Anyway, it's a special pattern with deep attribute hierarchies, whose traversal is shortcutted when one attribute happen to be None. This is common, with 2 exception: -it's rare it's really deep, or then it is arbitrarily deep and you need a procedural descent, not a fixed expression. -None break the descend, but then so does missing attributes. You address the first with special syntax. not the second, so I suspect in your case the second does not happen. Or rarely happen.

Hope this will help finding why some finds the ? operators add little, while some others think they add enough to overcome python traditional operator-averse nature.

On Monday, July 23, 2018 at 1:10:03 PM UTC+2, Steve Dower wrote:

On 23Jul2018 1145, Antoine Pitrou wrote:

...

Le 23/07/2018 à 12:38, Steve Dower a écrit :

...

General comment to everyone (not just Antoine): these arguments have zero value to me. Feel free to keep making them, but I am uninterested.

So you're uninterested in learning from past mistakes?

You sound like a child who thinks their demands should be satisfied because they are the center of the world.

Sorry if it came across like that, it wasn't the intention. A bit of context on why you think it's a mistake would have helped, but if it's a purely subjective "I don't like the look of it" (as most similar arguments have turned out) then it doesn't add anything to enhancing the PEP. As a result, I do not see any reason to engage with this class of argument.

I hope you'll also notice that I've been making very few demands in this thread, and have indicated a number of times that I'm very open to adjusting the proposal in the face of honest and useful feedback.

Cheers, Steve _______________________________________________ Python-ideas mailing list Python...@python.org javascript: https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Paul Moore wrote:

This is my impression, as well. It seems like something that's helpful in dealing with unstructured object hierarchies with lots of optional attributes - which is where JSON tends to be used.

But given that, I'm really much more interested in seeing the new operators compared against a well-written "JSON object hierarchy traversal" library than against raw Python code.

Good point, I did not think about that when suggesting to give some context, but indeed if it's linked to a library in particular, there is always the possibility to choose another object than None as the "nothing here" marker.

One that will absorb getitems and getattr accesses so that ?. and ?[] behavior is reproduced by plain . and []. Guard/NoValues should be chosen so that typical lib use is easier.

Anyway, addressing partially-populated nodes would need lib support: None-coalescence will not help when you traverse a dict/attribute hierarchy where some nodes implement some attributes/keys but not others. It help only when a node is either regular, or a guard object without any attributes...So an irregular tree, but not too irregular ;-)...

I personally do almost only classical programming, and I am somewhat opposed to OOP in general. So here my somewhat outlandish view, (and I am biased becase I will probably never need this feature).

First thoughts after reading the PEP: what is so super-special and fundamental about None value? Is it more special and required to treat more often than "true" or "false" values in similar manner? Is there statistics for occurence of code checking 'None' versus 'not None'?

So:

if x is None: x = 10

converts to:

x ??= 10

But what if one need to do something else than just return a value? E.g. I want to insert `print(x)` in the first example? Or check against other value than 'None'? Supposed to write it so then rewrite it so, then if I need it other way - again rewrite it so.

So from the PEP I understand only two things: 1) de-facto None became special in some contexts. 2) one may need to short-circuit an expression if None pops up.

So it is syntax good only for special kind of applications I suppose. It seems motivating examples come mainly from situations where one has a Python script serving some database framework or something alike. So it smells like specialized usage and design-specific pattern. It looks (from a passer-by POV) like the proposed syntax tries to treat some sticking out parts of something that may not be necessarily present in an application at all. Of course that maybe argued easily because many concepts that does not directly belong to programming are sometimes appearing in form of dedicated syntax. Though such cases not so common in Python.

I don't know whether the feature adressed by '?.' is often in those contexts, but first thing that comes to mind - it regards only experts and NOT intended for reading the code that I am not familiar with.

E.g. this "improved" code in examples:

def find_module(self, fullname, path): return getattr(self, 'find_spec', None)?.__call__(fullname, path)?.loader

I could not decipher this and I have tried for a while.

I could probably understand the motivation better if it were some more or less fundamental or general pattern, but is it? IDK

-----

More important that the syntax part is really worrying. Does everybody realize that there are only 2 or 3 ASCII characters left free? IIRC those are question mark "?", exclamation mark "!" and dollar sign "$".

Imagine that in some time someone comes up with a good general syntax feature that would require new symbol - and there is nothing left. That's not funny actually. This makes decisions about symbols into agony and heats up discussions to extreme because some PEP authors maybe hoping for a better-looking symbol for their 'child' while other proposals and possible future proposals may be more important.

At this time point I think one should first consider exploring the possibility to add non-ASCII characters to syntax. I think it is resonable because Unicode became standard. It should at least help to relax the atmosphere around syntax proposals.

'Niche' features IMO should be better added as functions, or, if non-ASCII symbols could be added, as some '2nd class' symbols (i.e. not so nice looking). And nicely looking symbols should be reserved for future proposals for general syntax features which are potentially useful for wider user groups.

On Mon, Jul 23, 2018 at 12:09:00PM +0100, Steve Dower wrote:

On 23Jul2018 1145, Antoine Pitrou wrote:

...

Le 23/07/2018 à 12:38, Steve Dower a écrit :

...

General comment to everyone (not just Antoine): these arguments have zero value to me. Feel free to keep making them, but I am uninterested.

So you're uninterested in learning from past mistakes?

You sound like a child who thinks their demands should be satisfied because they are the center of the world.

Sorry if it came across like that, it wasn't the intention.

Steve, I don't think you should apologise to somebody who has just quoted you out of context and insulted you for no good reason.

In context, your comment about "these arguments" was a perfectly reasonable thing to say. You didn't say you were uninterested in *all* arguments, only certain kinds of low-value arguments that keep getting made over and over again. Antoine snipped the context, unfairly accused you on zero evidence of being "uninterested in learning from past mistakes", and described you as a self-centred child.

I think Antoine should be apologising to you, not the other way around.

On Mon, Jul 23, 2018 at 5:52 AM Steve Dower steve.dower@python.org wrote:

The PEP author is unsure about how it works

I wish this statement had come with some context, because the only thing I'm unsure about is what I'm supposed to be unsure about.

In general—as I haven't been shy of saying—I find the entire idea awful. I recognize you have done sincere and high quality work in arguing for it; it just feels like a very wrong direction for Python.

But bracketing that, I'm probably the person you have in mind in that comment. And it's funny that you write you are unsure what you are supposed to be unsure about, but the very next section is exactly what I had in mind. Of course I don't mean that if implemented the semantics would be ambiguous... rather, the proper "swallowing" of different kinds of exceptions is not intuitively obvious, not even to you, Steve. And if some decision was reached and documented, it would remain unclear to new (or even experienced) users of the feature.

I'm borderline on ?[] right now. Honestly, I think it works best if it

also silently handles LookupError (e.g. for traversing a loaded JSON dict), but then it's inconsistent with ?. which I think works best if it handles None but allows AttributeError.

Moreover, at a couple points in your clarification, you say "ignoring descriptors." But ultimately, the language cannot do that. When a programmer writes `obj?.descriptor?.other_descriptor` SOMETHING has to happen (no matter what actually happens within the code of the property).

This can certainly be specified in some unambiguous way, but I believe that any decision made will be strongly counter-intuitive for certain concrete code.

On Mon, Jul 23, 2018 at 11:52 AM Steve Dower steve.dower@python.org wrote:

I'm borderline on ?[] right now. Honestly, I think it works best if it also silently handles LookupError (e.g. for traversing a loaded JSON dict), but then it's inconsistent with ?. which I think works best if it handles None but allows AttributeError.

That would easily make typos pass unnoticed:

request.context.user.usernme # raises AttributeError request?.context?.user?.usernme # return None

Same goes for LookupError: if a key or index is missing on 'a?[b]' I do want an exception. If I don't, which should be the exception rather than the rule, I will simply take the risk myself and do:

default = '<no name>' try: name = d['user']['details']['name'] or default except KeyError: name = default

But certainly there should be no native syntax encouraging me to do any of that. Talking about arbitrarily swallowing exceptions is the worst direction this proposal can take as it breaks yet another fundamental Python Zen: "errors should never pass silently". IMO this shows how fundamentally detached from the base philosophy of the language this whole idea is.

-- Giampaolo - http://grodola.blogspot.com

On Mon, Jul 23, 2018 at 2:23 AM Nicholas Cole nicholas.cole@gmail.com wrote:

One issue for me is that the trivial case is already a one-liner:

if a is None: a = 10

Yes, if you have no indentation and a 1-character name, then it fits on a single line. If you have a longer expression and/or side effects, then it's not a one-liner anymore.

And that leads to a simple question: how many times does this actually occur in real-world by python code? -- i.e. how many times do I want to check the value of an existing label, and, finding it is None (and None specifically), then assign it a value?

What does a scan through the existing core library say?

Please read the PEP before you shoot it down. It answers this _exact_ question.

On Mon, Jul 23, 2018 at 3:12 AM Steven D'Aprano steve@pearwood.info wrote:

...

? has no spaces, it's literally "variable names interrupted by question marks" and evaluation can stop at any time while scanning the line from left to right.

Just like ordinary attribute access.

This is the point I was making earlier: you accept existing punctuation doing these things:

try:
    obj.spam.egsg.tomato.cheese  # oops a typo
except AttributeError:
    # evaluation can stop at any time
    ...

while demanding a higher standard for new punctuation.

All of your criticisms of ? punctuation applies to . as well.

I don't think they do. For once, "a.b" does one and one thing only, "a?.b" does two and that's a fundamental difference (explicitness). It does so by introducing a brand new operator ("?") which can be spelled in two forms ("a?.b" and "a?[b]") by using two adjacent symbols not interrupted by any space, which is an absolute first in the Python syntax and that's the second and fundamental difference. I cannot move the same criticism to the "a.b" form: it's simpler, it does one thing and it uses one symbol.

Your argument is basically a revisitation of "it's just another symbol" and "it's like a + b" which you have being pulling different times in this thread already. You want to imply that since symbols are already used in the grammar (and "." in particular) then it's perfectly fine to also have "?" and its spell variants (which are 4 in total). I don't think that's how changes of such importance should be discussed.

...

Multiple "?" can live on the same line so that's incentive to write one-liners, really, and to me one-liners are always less explicit than the same logic split on multiple lines.

Explicit is not always better.

import this

is much better than:

for location in sys.path:
    try:
        for file in os.listdir(location):
            if os.splitext(file) in ('.pyc', '.py', '.so'):
                ...

I honestly don't see how this example is related with anything discussed so far.

If punctuation is unreadable and Perlish, so is "." and ":" punctuation. If ?? is bad because it is "implicit", then so is import or sorted.

I'm not even sure how to reply to this.

On Mon, Jul 23, 2018 at 6:53 PM Steven D'Aprano steve@pearwood.info wrote:

On Mon, Jul 23, 2018 at 02:04:17PM +0200, Giampaolo Rodola' wrote:

...

"a?.b" does two and that's a fundamental difference (explicitness).

How is "two things" less explicit than "one thing"? Comments like the above is why I think that "explicit" and "implicit" are used to mean "I like it" and "I don't like it" rather than being objective arguments, or indeed having anything to do with explicitness or implicitness.

This:

v = a?.b

...*implicitly* checks if value is not None [and continues execution]. This:

v = a if a.b is not None: v = a.b

...*explicitly* checks if value is not None and continues execution. If for some reason '?'[ is also going to swallow LookupError then *that* would further decrease explicitness, because LookupError would be nowhere in sight, the same way "if", "is", "not", "None", ":", "new line" are nowhere in sight in the 'a?.b' example. Some argued "A ?? B" is less explicit than "A if A is not None else B" for the same reason. One may argue that silently returning None instead of raising AttributeError is also less explicit. This - and this only - is my argument about explicitness. It doesn't have to do with how many things are hidden behind an import statement or what happens on sorted() (that's comparing apples and oranges). I hope it's clear now.

...

It does so by introducing a brand new operator ("?") which can be spelled in two forms ("a?.b" and "a?[b]") by using two adjacent symbols not interrupted by any space, which is an absolute first in the Python syntax

It isn't a first. Many existing operators use two adjacent symbols not interrupted by a space:

e.g. == <= >= != ** // << >> += -= *= etc.

You say 'a == b'. You can't say 'a ?. b' (not that it matters, it would be less intuitive anyway). You can't because '.?' is the only couple of contiguous symbols requiring "something" before and after with no spaces in between, and that's a first in the language. The argument about this is that it's ugly and less readable. My additional argument at the beginning of this thread was that if you add PEP-572 to the mix you dangerously enter into Perl territory:

foo(x=(x := a?.b?[c] ?? d))

...

and that's the second and fundamental difference. I cannot move the same criticism to the "a.b" form: it's simpler, it does one thing and it uses one symbol.

You criticised ?. because it can interupt left-to-right execution:

a?.b?.c?.d

True. But so can a single dot:

a.b.c.d

is no more guaranteed to execute all the way to the right.

The difference is that 'a.b.c.d' will result in AttributeError as soon as something is None while 'a?.b?.c?.d' will return None instead.

Likewise the logical operators "or" and "and" are designed to short-circuit. If ?? and friends are a mistake because they short-circuit, why aren't "or" and "and" mistakes? I'm not asking this as a rhetorical question. If you think there is a reason why it is okay for or/and to short-circuit, but it is bad for ?? and friends to short-circuit, then please explain why they are different. I will be very happy to listen to your arguments.

The argument about this is that '?.' short-circuits execution *silently*. Instead of AttributeError you get None. You may chain ?. in order to lazily traverse a long tree, inadvertently assign None to a variable, continue code execution and fail later rather than sooner:

email = request?.context?.user?.email # None ... sendmail(subject, body, email)

Some (Antoine) rightly argued this may even have security implications (replace 'email' with 'password').

On 2018-07-23 23:05, Giampaolo Rodola' wrote:

On Mon, Jul 23, 2018 at 6:53 PM Steven D'Aprano steve@pearwood.info wrote:

...

On Mon, Jul 23, 2018 at 02:04:17PM +0200, Giampaolo Rodola' wrote:

...

"a?.b" does two and that's a fundamental difference (explicitness).

How is "two things" less explicit than "one thing"? Comments like the above is why I think that "explicit" and "implicit" are used to mean "I like it" and "I don't like it" rather than being objective arguments, or indeed having anything to do with explicitness or implicitness.

This:

 v = a?.b

...*implicitly* checks if value is not None [and continues execution].

It's no more implicit than 'or' checking for falseness.

This:

 v = a
 if a.b is not None:
     v = a.b

...*explicitly* checks if value is not None and continues execution. If for some reason '?'[ is also going to swallow LookupError then *that* would further decrease explicitness, because LookupError would be nowhere in sight, the same way "if", "is", "not", "None", ":", "new line" are nowhere in sight in the 'a?.b' example. Some argued "A ?? B" is less explicit than "A if A is not None else B" for the same reason. One may argue that silently returning None instead of raising AttributeError is also less explicit. This - and this only - is my argument about explicitness. It doesn't have to do with how many things are hidden behind an import statement or what happens on sorted() (that's comparing apples and oranges). I hope it's clear now.

...

...

It does so by introducing a brand new operator ("?") which can be spelled in two forms ("a?.b" and "a?[b]") by using two adjacent symbols not interrupted by any space, which is an absolute first in the Python syntax

It isn't a first. Many existing operators use two adjacent symbols not interrupted by a space:

e.g. == <= >= != ** // << >> += -= *= etc.

You say 'a == b'. You can't say 'a ?. b' (not that it matters, it would be less intuitive anyway). You can't because '.?' is the only couple of contiguous symbols requiring "something" before and after with no spaces in between, and that's a first in the language.

You _can_ say 'a ?. b', just as you _can_ say 'a . b'.

Also, it's not a couple of contiguous symbols, it's a single symbol, just as '<=' is a single symbol (and you can't put a space in the middle of that either). [snip]

On Tue, Jul 24, 2018 at 12:05:14AM +0200, Giampaolo Rodola' wrote:

This:

v = a?.b

...*implicitly* checks if value is not None [and continues execution].

Do you agree that:

obj.attribute x + 1 func(arg)

explicitly looks up an attribute on obj, explicitly adds 1 to x, and explicitly calls func with a single argument? I don't think that we have to write COBOL-esque code to be explicit:

GET ATTRIBUTE "attribute" FROM obj ADD 1 TO x CALL FUNCTION func WITH ARGUMENT arg

I don't accept that the use of punctuation makes something implicit. But if you want to argue that anything with punctuation is "implicit", then okay, Python has lots of implicit punctuation.

By definition, ?. checks for None before doing the attribute lookup. That is completely explicit, regardless of how it is spelled:

obj?.attribute

NULL-AWARE GET ATTRIBUTE "attribute" FROM obj

null_aware_getattr(obj, "attribute")

getattr_if_not_none(obj, "attribute")

But what certainly *is* implicity is David Mertz' suggestion for a magical None-aware proxy:

x.attribute

The only way to tell whether that was an ordinary attribute lookup or a none-aware lookup would be to carefully inspect x and find out whether it was an instance of the None-aware proxy class or not.

This

v = a
if a.b is not None:
    v = a.b

...*explicitly* checks if value is not None and continues execution.

If you are trying to match the behaviour of a?.b above, it is also completely buggy and doesn't do what is intended.

# Equivalent of a?.b v = a if v is not None: v = v.b

If for some reason '?'[ is also going to swallow LookupError

What makes you think that ?[...] will swallow LookupError?

Please don't argue against misfeatures that the PEP doesn't propose. Nothing in PEP 505 swallows any exceptions. Swallowing exceptions is explicitly rejected, and swallowing LookupError isn't part of the proposal.

[...]

One may argue that silently returning None instead of raising AttributeError is also less explicit.

And again, you are arguing against a misfeature which PEP 505 does not propose. The ?. operator will not suppress AttributeErrors.

# Wrong! No! This is not what the PEP proposes! obj = 1.234 assert obj?.hexx is None

[...]

...

It isn't a first. Many existing operators use two adjacent symbols not interrupted by a space:

e.g. == <= >= != ** // << >> += -= *= etc.

You say 'a == b'. You can't say 'a ?. b' (not that it matters, it would be less intuitive anyway). You can't because '.?' is the only couple of contiguous symbols requiring "something" before and after with no spaces in between, and that's a first in the language.

Why do you think spaces aren't allowed? The PEP explicitly says that the new operators can be used wherever the regular operators can be used:

"The maybe-dot and maybe-subscript operators are added as trailers for atoms, so that they may be used in all the same locations as the regular operators"

and explicitly shows the grammar changes required:

trailer: ('(' [arglist] ')' | '[' subscriptlist ']' | '?[' subscriptlist ']' | '.' NAME | '?.' NAME)

That tells me that ?. will be legal anywhere . is legal, so if x . y is legal (and it is) so will x ?. y be legal.

[...]

The difference is that 'a.b.c.d' will result in AttributeError as soon as something is None while 'a?.b?.c?.d' will return None instead.

Correct. Because sometimes you want an AttributeError, and sometimes you want None.

You are criticising the operator for doing what it is designed and intended to do. You might as well criticise getattr(obj, 'spam', None) for returning None.

If you want an AttributeError, then don't use ?. and use ordinary . instead.

...

Likewise the logical operators "or" and "and" are designed to short-circuit. If ?? and friends are a mistake because they short-circuit, why aren't "or" and "and" mistakes? I'm not asking this as a rhetorical question. If you think there is a reason why it is okay for or/and to short-circuit, but it is bad for ?? and friends to short-circuit, then please explain why they are different. I will be very happy to listen to your arguments.

The argument about this is that '?.' short-circuits execution *silently*.

Other short-circuit operators also short-circuit execution silently. That's what they are designed to do.

# This isn't what actually happens. py> x = 0 py> result = x and 1/x __main__:1: UserWarning: Short-cut operation occurred, the right hand operand was not evaluated!!! Do not panic, this is the expected behaviour!!! py> print(result) 0

Instead of AttributeError you get None. You may chain ?. in order to lazily traverse a long tree,

Correct, that is what it is designed to do.

inadvertently assign None to a variable, continue code execution and fail later rather than sooner:

 email = request?.context?.user?.email  # None
 ...
 sendmail(subject, body, email)

Some (Antoine) rightly argued this may even have security implications (replace 'email' with 'password').

I'll have to remember that. Whenever there's any proposal for a feature I don't like, just claim it "may even have security implications". Who needs evidence when we have Fear, Uncertainty, Doubt?

On Tue, Jul 24, 2018, 5:50 AM Steven D'Aprano steve@pearwood.info wrote:

But what certainly *is* implicity is David Mertz' suggestion for a magical None-aware proxy:

x.attribute

The only way to tell whether that was an ordinary attribute lookup or a none-aware lookup would be to carefully inspect x and find out whether it was an instance of the None-aware proxy class or not.

Every use I've suggested for the magic proxy is similar to:

NullCoalesce(cfg).user.profile.food

Yes, the class is magic. That much more so in the library I published last night that utilizes wrapt.ObjectProxy. But it's also pretty explicit in that an actual *word* announces that funny stuff is going to happen on the same line.

Of course the this could be abused with:

cfg = NoneCoalesce(cfg) ... 1000 lines ... do_something(cfg)

But then, I could also write a property that actually started a computation of the millionth digit of pi while launching a DDoS attack on python.org when a user accessed 'x.attribute'.

NoneCoalesce or GreedyAccess are magic, but in their intended use, they are as little magical as possible to deal with messy nested data.

On Tuesday, July 24, 2018 at 1:38:42 PM UTC+2, Rhodri James wrote:

-snip- I'm still of the opinion that both approaches are trying to solve a problem that's too niche to merit them, BTW.

That's also my impression. Hence the second approach: it does not require any change to python, it's just a tool for that niche, so imho it's the right approach. Such a wrapper can even be provided by the lib that produced such nested attributes in the first place. The operator are a tool which seems designed to the same niche issue, but is exposed as a core language feature. It may be interesting if it provides a lot of side benefits, so the tool is so braodly useful it outgrowned it's niche origin. At this point, I do not think it does.

Both approaches should not be exposed as core language, but as facilitating tools to be used if, in your application, you have to traverse deep semi-regular attributes hierarchies. The operator approach, by definition, is part of the core, so -1 The wrapper does not need to be included in python, so the discussion is meaningless: people define the tools they want and use them as they please. Personally, I think it would be nice to have that in the standard distrib, especially wrapt or something as powerful. But I don't really care in fact, what good is that now I know about it so I can use it...

On 24/07/18 13:07, Richard Damon wrote:

The fact that you changed NullCoalesce into Foo to show lack of explicitness seems a straw-man. Words are FULL of meaning, while symbols are less so. The biggest issue I see with the use of ? here is that ? does have some meaning, it says we are going to be (or have) asked a question, it doesn’t tell us what the question is. Most of the other symbols used have a long history of meaning (yes = has the problem that historically it has had two possible meanings). To me, ? gives no indication that it is going to ask about Nullness.

Oh, I don't disagree with you about ? not giving any much indication that it's about Nullness except for the (relatively short) history of using it to mean exactly that in C# etc. However I don't think that a class of whatever name doing something magic is any better.

On Tue, Jul 24, 2018, 7:38 AM Rhodri James rhodri@kynesim.co.uk wrote:

On 24/07/18 12:02, David Mertz wrote:

...

Every use I've suggested for the magic proxy is similar to:

NullCoalesce(cfg).user.profile.food

Yes, the class is magic. That much more so in the library I published

last

...

night that utilizes wrapt.ObjectProxy. But it's also pretty explicit in that an actual*word* announces that funny stuff is going to happen on

the

...

same line.

  Foo(cfg).user.profile.food

Is that explicit that funny stuff is going to happen on the same line? I wouldn't generally assume so, I'd just assume the coder created a throwaway object to get at an attribute.

Foo isn't a very indicative name. NoneCoalesce (or NullCoalesce, or GreedyAccess) is. But if I had any doubt, I could read the docstring for Foo to find out how magical it was, and in what way. I definitely know *something* is happening by creating that new instance in the line.

I'm still of the opinion that both approaches are trying to solve a

problem that's too niche to merit them, BTW.

That doesn't make sense to me. You think my little library shouldn't be allowed on PyPI? I don't force the couple classes on anyone, but if they happen to help someone (or PyMaybe, or some other library, does) they don't change anything about Python itself. Syntax is a very different matter.

I *definitely* don't think a little tool I wrote in a couple hours last night belongs in the standard library (with most of the heavy lifting actually done by wrapt—which is really well designed, and is also not in the standard library). I also don't think PyMaybe belongs there, even though it's a couple years old.

Now, perhaps, if 'coalescing' is widely used for a year or two. And bugs are fixed. And the API is tweaked based on experience with it's use. And so on... At that point, *maybe* something derived from it might be appropriate.

On Tue, Jul 24, 2018, 9:08 AM Rhodri James rhodri@kynesim.co.uk wrote:

On 24/07/18 14:02, David Mertz wrote:

...

On Tue, Jul 24, 2018, 7:38 AM Rhodri James rhodri@kynesim.co.uk wrote:

...

I'm still of the opinion that both approaches are trying to solve a problem that's too niche to merit them, BTW.

That doesn't make sense to me. You think my little library shouldn't be allowed on PyPI? I don't force the couple classes on anyone, but if they happen to help someone (or PyMaybe, or some other library, does) they

don't

...

change anything about Python itself. Syntax is a very different matter.

I have no objection to anyone putting anything on PyPI. Putting it (or an equivalent) in the standard library is much more problematical, and you were talking about that. I think your little library is a much richer source of bugs than you think it is, and dealing with messy data access isn't a good enough reason to put that much temptation in front of naive users.

-- Rhodri James *-* Kynesim Ltd

On Tue, Jul 24, 2018, 9:09 AM Chris Angelico rosuav@gmail.com wrote:

...

...

...

x = Foo(cfg).user.profile x.food

Remember, these lines could be a very long way apart. You could pass 'x' to a function unrelated to the line of code that created it. And 'x.food' still has to have the magic. You can't mandate that the call to Coalesce be on the same line as the attribute access - Python doesn't work that way.

So your perfectly ordinary dot operator now does magic in addition to normal attribute access. See why it's a dangerous thing?

Yes, of course. That's why I would recommend best practice is to unbox or otherwise use the conditional value as close to the magic code as feasible.

Likewise, as I noted a little while ago, 'x.food' could equally well be a property that executed arbitrarily slow, magical, obscure, or even malicious operations. Equally, 'x + y' could do absolutely anything if we define .__add__() or .__radd__() methods.

Everything in Python is magical in that sense, but we should deliberately keep the magic constrained to the amount needed.

On Wed, Jul 25, 2018 at 8:00 PM Steven D'Aprano steve@pearwood.info wrote:

...

Every use I've suggested for the magic proxy is similar to: NullCoalesce(cfg).user.profile.food

Thanks Steven! Several things you suggest urge some minor improvements to my slightly-more-than-toy library `coalescing` ( https://pypi.org/project/coalescing/).

I trust that you don't expect everyone to religiously follow the exact letter of your examples. Being able to extract out subexpressions into variables is an important programming technique, and we ought to be able to do things like this:

config = NullCoalesce(cfg)
process(config)

# later...
user = config.user
user_records[user.id] = user
food = get_preferred_food(user)

Yes, absolutely! In many cases this will JUST WORK because of the magic in wrapt.ObjectProxy. But it also makes me realize that I should provide a plain old function `unbox()` as well as the method. In particular, `unbox(obj)` can and will figure out whether the object is really a proxy at all, and if not simply return `obj` itself. Then you do no harm by scattering unbox() calls in places where they may or may not be needed.

I also think that even though it was a typo at first, the name NullCoalesce is better than NoneCoalesce. I'll provide an alias.

spam?.attribute
eggs.attribute

One of those is ordinary attribute access that can raise AttributeError if the object is None, and the other is None-aware "maybe dot" attribute access that returns None if the object is None.

That is disingenuous, I think. Can this raise an AttributeError?

spam?.eggs?.bacon

Of course it can! And this is exactly the pattern used in many examples in the PEP and the discussion. So the PEP would create a situation where code will raise AttributeError in a slightly—and subtly—different set of circumstances than plain attribute access will.

The fact that half the readers of this thread won't immediately see just when that AttributeError might occur (based on lots of prior misunderstandings by smart posters) makes the operator that much more magical.

On Wed, Jul 25, 2018 at 9:20 PM Chris Angelico rosuav@gmail.com wrote:

On Thu, Jul 26, 2018 at 11:02 AM, David Mertz mertz@gnosis.cx wrote:

...

That is disingenuous, I think. Can this raise an AttributeError?

spam?.eggs?.bacon

Of course it can! And this is exactly the pattern used in many examples

in

...

the PEP and the discussion. So the PEP would create a situation where

code

...

will raise AttributeError in a slightly—and subtly—different set of circumstances than plain attribute access will.

I don't understand. If it were to raise AttributeError, it would be because spam (or spam.eggs) isn't None, but doesn't have an attribute eggs (or bacon). Exactly the same as regular attribute access. How is it slightly different? Have I missed something?

That was my reaction, too.

food = spam?.eggs?.bacon

Can be rewritten as:

food = spam if spam is not None and spam.eggs is not None: food = spam.eggs.bacon

They both behave identically, no? Maybe I missed the point David was trying to make.

So now at least TWO proponents of 505 cannot successfully translate a very simple example taken almost directly from the PEP!

Is that REALLY a good argument for it being helpful, and not being a bug magnet?!

On Wed, Jul 25, 2018 at 9:57 PM Chris Angelico rosuav@gmail.com wrote:

On Thu, Jul 26, 2018 at 11:45 AM, Nicholas Chammas nicholas.chammas@gmail.com wrote:

...

On Wed, Jul 25, 2018 at 9:20 PM Chris Angelico rosuav@gmail.com wrote:

...

On Thu, Jul 26, 2018 at 11:02 AM, David Mertz mertz@gnosis.cx wrote:

...

That is disingenuous, I think. Can this raise an AttributeError?

spam?.eggs?.bacon

Of course it can! And this is exactly the pattern used in many

examples

...

...

...

in the PEP and the discussion. So the PEP would create a situation where code will raise AttributeError in a slightly—and subtly—different set of circumstances than plain attribute access will.

I don't understand. If it were to raise AttributeError, it would be because spam (or spam.eggs) isn't None, but doesn't have an attribute eggs (or bacon). Exactly the same as regular attribute access. How is it slightly different? Have I missed something?

That was my reaction, too.

food = spam?.eggs?.bacon

Can be rewritten as:

food = spam
if spam is not None and spam.eggs is not None:
    food = spam.eggs.bacon

They both behave identically, no? Maybe I missed the point David was

trying

...

to make.

Aside from questions of repeated evaluation/assignment, yes. The broad semantics should be the same.

(If you want to get technical, "spam" gets evaluated exactly once, "spam.eggs" a maximum of once, and "food" gets assigned exactly once. Your equivalent may evaluate and assign multiple times.)

ChrisA _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On Wed, Jul 25, 2018 at 10:29 PM Chris Angelico rosuav@gmail.com wrote:

food = spam?.eggs?.bacon

can be rewritten as

_tmp = spam if _tmp is not None: _tmp = _tmp.eggs if _tmp is not None: _tmp = _tmp.bacon food = _tmp

Yes, that looks right. Well, you need a `del _tmp` at the end; but it's almost right. My point was that both you and Nicholas Chammas failed to recognize that the other translation was wrong... I recognize it does something "kinda similar." But the semantics of the operators are just plain hard to grok, even by their strongest advocates.

I can write lots of things that are "mostly correct" already in Python. Most easily, I can write:

try: food = spam.eggs.bacon except: food = None

That does what is actually needed about 95% of the time. It's also clear and easy to understand.

It is *actually impossible* to

perfectly represent short-circuiting semantics in Python!

It's INCREDIBLY EASY to represent short-circuiting semantics in Python! What on earth are you talking about? That's what the if/elif/else blocks do.

And before you go "well that proves my point, this suggestion is bad", let's

apply the same test to a few other pieces of syntax. Rewrite the following statements without using the syntactic feature named in the comment:

This is childishly simple:

# 1) Decorators @deco def func(): ...

def func(): ... func = deco(func)

OK, this one is harder. The "mostly correct" version is easy. But the actual full version is nuanced (see https://www.python.org/dev/peps/pep-0380/ for details).

# 2) "yield from"

def chain(*iters): for iter in iters: yield from iter

# The simple approximation: for iter in iters:

for _ in iter:

yield iter

# 3) and the big one: generator expressions # yes, I'm deliberately using x multiple ways here def f(x): return x*x x = range(10) x = (f(x) for x in x if x % 2)

I'm not going to bother with that. I'd fire anyone who wrote it, after code review. Minus the abuse of names, it's just:

def gen(xs):

for x in xs:

if x % 2:

yield f(x)

x = gen(xs)

Btw. Here's a way of spelling the proposed syntax that gets the semantics right:

...

...

# pip install coalescing NullCoalesce(spam).eggs.bacon

On Wed, Jul 25, 2018 at 10:14 PM David Mertz mertz@gnosis.cx wrote:

So now at least TWO proponents of 505 cannot successfully translate a very simple example taken almost directly from the PEP!

Is that REALLY a good argument for it being helpful, and not being a bug magnet?!

On Wed, Jul 25, 2018 at 9:57 PM Chris Angelico rosuav@gmail.com wrote:

...

On Thu, Jul 26, 2018 at 11:45 AM, Nicholas Chammas nicholas.chammas@gmail.com wrote:

...

On Wed, Jul 25, 2018 at 9:20 PM Chris Angelico rosuav@gmail.com

wrote:

...

...

On Thu, Jul 26, 2018 at 11:02 AM, David Mertz mertz@gnosis.cx wrote:

...

That is disingenuous, I think. Can this raise an AttributeError?

spam?.eggs?.bacon

Of course it can! And this is exactly the pattern used in many

examples

...

...

...

in the PEP and the discussion. So the PEP would create a situation where code will raise AttributeError in a slightly—and subtly—different set of circumstances than plain attribute access will.

I don't understand. If it were to raise AttributeError, it would be because spam (or spam.eggs) isn't None, but doesn't have an attribute eggs (or bacon). Exactly the same as regular attribute access. How is it slightly different? Have I missed something?

That was my reaction, too.

food = spam?.eggs?.bacon

Can be rewritten as:

food = spam
if spam is not None and spam.eggs is not None:
    food = spam.eggs.bacon

They both behave identically, no? Maybe I missed the point David was

trying

...

to make.

Aside from questions of repeated evaluation/assignment, yes. The broad semantics should be the same.

(If you want to get technical, "spam" gets evaluated exactly once, "spam.eggs" a maximum of once, and "food" gets assigned exactly once. Your equivalent may evaluate and assign multiple times.)

ChrisA _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

-- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.