Hello,
As was mentioned many times on the list, PEP634-PEP636 are thoroughly prepared and good materials, many thanks to their authors. PEP635 "Motivation and Rationale" (https://www.python.org/dev/peps/pep-0635/) stands out among the 3 however: while reading it, chances that you'll get a feeling of "residue", accumulating a section over section. By the end of reading, you may get a well-formed feeling that you've read a half-technical, half-marketing material, which is intended to "sell" a particular idea among many other very viable ideas, by shoehorning some concepts, downplaying other ideas, and at the same time, light-heartedly presenting drawbacks of its subject one.
Just to give one example, literally at the very beginning, at the "Pattern Matching and OO" section (3rd heading) it says:
Pattern matching is complimentary to the object-oriented paradigm.
It's not until the very end of document, in the "History and Context" it tells the whole truth:
With its emphasis on abstraction and encapsulation, object-oriented programming posed a serious challenge to pattern matching.
You may wonder how "complimentary" and "posed a serious challenge" relate to each other. While they're definitely not contradictory, starting the document with light-hearted "complimentary" can be seen as trying to set the stage where readers don't pay enough attention to the problem. And it kinda worked: only now [1] wider community discovers the implications of "Class Patterns" choices. (As a note, PEP635 does well on explaining them, and I'm personally sold on that, but it's *tough* choice, not the *obvious* choice).
There're many more examples like that in the PEP635, would take too much space to list them all. However, PEP635 refers to the paper:
Kohn et al., Dynamic Pattern Matching with Python https://doi.org/10.1145/3426422.3426983 (Accepted by DLS 2020. The link will go live after Nov. 17; a preview PDF can be obtained from the first author.)
As that citation suggests, the paper is not directly linked from the PEP635. But the preprint is now accessible from the conference page, https://conf.researchr.org/home/dls-2020?#event-overview (direct link as of now: https://gvanrossum.github.io//docs/PyPatternMatching.pdf).
That paper is written at much higher academic standard, and a pleasure to read. I recommend it to everyone who read PEP635 (note that it was written with PEP622 in mind, but conceptual differences are minor). With it, I noticed just 2 obvious issues:
Section 4.3. Named Constants
It would clearly be desirable to allow named constants in patterns as a replacement and extension of literals. However, Python has no concept of a constant, i.e. all variables are mutable (even where the values themselves are immutable).
So, unlike PEP635, the paper pinpoints right the root of PEP634's problems: lack of constants in Python (on the language level). This is just the point which was raised on the mailing list either (https://mail.python.org/archives/list/python-dev@python.org/message/WV2UA4AK...).
Under strict academic peer review, the paper would have been returned for elaboration, with a note like: "Given that nowadays many dynamic languages (PHP, JavaScript, Lua, etc.) have support for constants, and lack of constants poses a serious usability challenge to your proposal, please explain why you chose to proceed anyway (and apply workarounds), instead of first introducing the concept of constants to the language. (Given that amount of work to implement pattern matching is certainly an order of magnitude larger than to introduce constants)."
But the paper wasn't returned for elaboration, so we'll keep wondering why the authors chose such a backward process.
Section 6.1. Scope
The granularity of the scope of local variables is at the level of functions and frames. [...] The only way to restrict the scope of a variable to part of a function’s body (such as a case clause) would be to actively delete the variable when leaving the block. This would, however, not restore any previous value of a local variable in the function’s scope.
This is a misconception ("the only way") which is repeated almost one to one on PEP635 either. If anything, the above describes how pseudo-scoping is currently implemented for exception vars in "except Exception as e:" clause (more info: https://mail.python.org/pipermail/python-dev/2019-January/155991.html), which is hardly a "best practice", and definitely not the only way.
How to support multiple variable scopes in one stack frame is not a rocket science at all. One just need to remember how C did that since ~forever. And that's: (for unoptimized programs) variables in different scopes live in disjoint subsections of a frame. (For optimized programs, variables with disjoint liveness can share the same locations in a frame).
The only reasons for not implementing the same solution in Python would be intertia of thought and "but it's not done anywhere else in Python". Yeah, but nobody del'eted local variables behind users' backs either, before somebody started to do that for exception clause variables. And the whole of pattern matching is such that either one thing, or another, but will be done for the first time in Python. For example, nobody before could imagine that one can write "Point(x, y)", and get x and y assigned, and now we're facing just that [1]. (Again, I personally love it, though think that "Point(>x, >y)" is an interesting option to address the tensions).
In that regard, the current PEP634 and friends miss too many interesting and useful opportunities (constants in language core and true scoping for match'es, to name a few). Well, that happens. But they try to shoehorn too much of "we didn't do" into "it's not possible" or "it doesn't make sense", or "let's workaround it in adhoc ways" and that raises eyebrows, leading to concerns of whether the proposals are actually "raw" as of yet.
[1] People expressing surprise at "Class Patterns" syntax: https://mail.python.org/archives/list/python-dev@python.org/message/F66J72JU... https://mail.python.org/archives/list/python-dev@python.org/message/Q2ARJULL...
On Sun, Nov 15, 2020 at 12:48:50PM +0300, Paul Sokolovsky wrote:
Just to give one example, literally at the very beginning, at the "Pattern Matching and OO" section (3rd heading) it says:
If it's the third heading, it's not *literally* at the very beginning.
Pattern matching is complimentary to the object-oriented paradigm.
It's not until the very end of document, in the "History and Context" it tells the whole truth:
With its emphasis on abstraction and encapsulation, object-oriented programming posed a serious challenge to pattern matching.
You may wonder how "complimentary" and "posed a serious challenge" relate to each other.
There's no need to wonder. Just keep reading:
"In short: in object-oriented programming, we can no longer view objects as tagged tuples. The arguments passed into the constructor do not necessarily specify the attributes or fields of the objects. Moreover, there is no longer a strict ordering of an object's fields and some of the fields might be private and thus inaccessible. And on top of this, the given object might actually be an instance of a subclass with slightly different structure.
To address this challenge, patterns became increasingly independent of the original tuple constructors."
The challenge was not, as you imply, that OOP was a challenger to pattern matching by offering an alternative idiom that made pattern matching obsolete. The challenge was to update pattern matching implementations to work with objects that don't map well to tagged tuples.
[...]
As that citation suggests, the paper is not directly linked from the PEP635. But the preprint is now accessible from the conference page, https://conf.researchr.org/home/dls-2020?#event-overview (direct link as of now: https://gvanrossum.github.io//docs/PyPatternMatching.pdf).
It would be nice to have the PEP link to the paper. I found it very useful for my understanding to compare how other languages solved or avoiding these issues.
I disagree strongly with your further points about Named Constants and Scopes. I think you make unjustified, and mostly likely wrong, statements like:
Under strict academic peer review, the paper would have been returned for elaboration,
amount of work to implement pattern matching is certainly an order of magnitude larger than to introduce constants
The only reasons for not implementing the same solution [block scopes] in Python would be intertia of thought and "but it's not done anywhere else in Python".
but in an effort to try to keep matters focused, I won't elaborate.
Having said that, I would like to see some discussion on what it would take to bring in constants to the language in a fully backwards compatible fashion, without hurting performance. Perhaps it is as easy as you say, perhaps not. If it is easy, that might work for pattern matching. It is isn't easy, it would be nice to know why.
Here's a toy proposal, not intended to be taken too seriously: for local constants, inside a function, the compiler could detect and reject (with a SyntaxError) more than one attempt to bind a value to a name declared as constant, with no runtime cost. Outside of functions, we could add one extra scope between locals and globals, a "const" scope, which implements write-once semantics. This would make every global and builtin name lookup more costly.
Looking at the paper, I like the look of Elixer's pin operator:
case ^mylib.STATUS_OK, result:
would match mylib.STATUS_OK by value, without binding, and capture result. I think that's better than the PEP's current DWIM (Do What I Mean) approach that a dotted identifier must be a constant and an undotted identifier is a capture pattern.
On the other hand, I can see myself forgetting the pin and accidentally binding to (pseudo-)constants:
case mylib.STATUS_OK, result:
would accidentally replace the STATUS_OK code. Ouch.
Maybe we should flip the meaning of pin and have it mark capture patterns?
case mylib.STATUS_OK, ^result:
Here if I forgot the pin, I'd likely get a NameError.
Hello,
On Sun, 15 Nov 2020 22:05:46 +1100 Steven D'Aprano steve@pearwood.info wrote:
On Sun, Nov 15, 2020 at 12:48:50PM +0300, Paul Sokolovsky wrote:
Just to give one example, literally at the very beginning, at the "Pattern Matching and OO" section (3rd heading) it says:
If it's the third heading, it's not *literally* at the very beginning.
Thanks for stopping by that. 3rd of 26, but yeah, "beginning-ness is in the eye of beholder".
[]
You may wonder how "complimentary" and "posed a serious challenge" relate to each other.
There's no need to wonder. Just keep reading:
I read it in full, and explained my concerns with the wording (of PEP625 in general): is seems to be carefully chosen to lead the reader to believe that the view presented in PEP634 is the only viable choice, and to downplay other alternatives (too much, and unfairly).
[]
It would be nice to have the PEP link to the paper. I found it very useful for my understanding to compare how other languages solved or avoiding these issues.
I disagree strongly with your further points about Named Constants and Scopes. I think you make unjustified, and mostly likely wrong, statements like:
Under strict academic peer review, the paper would have been returned for elaboration,
Well, this is not a statement, but a speculation, based on reading many complaints from (usually junior) researches of how it's hard to get a paper published/conf-accepted when a subject concern improvement to an existing technique - there're oftentimes multiple rounds of rejections with suggestions to prove the point that it's indeed "improvement" and how it compares to actual and possible alternatives.
amount of work to implement pattern matching is certainly an order of magnitude larger than to introduce constants
The only reasons for not implementing the same solution [block scopes] in Python would be intertia of thought and "but it's not done anywhere else in Python".
By these two I stand, yeah. (And ready to be proven wrong, be ashamed, and learn from it.)
but in an effort to try to keep matters focused, I won't elaborate.
Having said that, I would like to see some discussion on what it would take to bring in constants to the language in a fully backwards compatible fashion, without hurting performance. Perhaps it is as easy as you say, perhaps not. If it is easy, that might work for pattern matching. It is isn't easy, it would be nice to know why.
Thanks, that's exactly what I'd like to see too. And while it was pointed previously that this is orthogonal matter to pattern matching, the common sense keeps kicking in, suggesting that it's a *prerequisite* to having non-adhoc pattern matching design.
Here's a toy proposal,
[]
The baseline of my version is much simpler:
# This makes "const" a kind of hard keyword for this module from __future__ import const
FOO: const = 1 # obviously, this is constant Foo_Bar: const = 2 # this is too foo: const = 3 # and like it or not, but this is constant either
"const" annotation is recognized by compiler (scope resolving component, aka symtable builder to be exact) as a property orthogonal to localness/globalness. Literally, "symtable" module (https://docs.python.org/3/library/symtable.html#symtable.Symbol) will grow Symbol.is_const() method.
The rest should be clear - if there's:
case sym:
Where Symbol.is_const() is true for "sym", then value of that symbol is used for matching (instead of treating it as a binding target).
This definitely would be very beginning of const's journey in Python, but exactly the amount needed to address concerns with "force enum usage" adhockery in PEP634.
[]
Looking at the paper, I like the look of Elixer's pin operator:
case ^mylib.STATUS_OK, result:
And I don't. I think that "==", as used in PEP642v2, much better fits with Python syntax and overall feel (that's also what you and I suggested, before PEP642v2 picked it up, right?).
But again, the whole idea to introduce const's is to drop the need for any "pin operator" in 90% of cases, when constants are used. "Pin operator" will be needed only to match against a real variable:
FOO: const = 1 BAR: const = 2
i_dont_know_yet = FOO if sth() else BAR
match a: case == i_dont_know_yet: print("We're better than Rust")
[]
Maybe we should flip the meaning of pin and have it mark capture patterns?
case mylib.STATUS_OK, ^result:
But that's the humble idea, the epiphany of which I humbly spread around for a month (as an alternative to both const-introduction (we won't get around that!) or "=="-marking). Except that you used the wrong arrow - it shouldn't point away from capturing variable, but right into the heart of it:
case mylib.STATUS_OK, >result: case mylib.STATUS_OK, >>result: case mylib.STATUS_OK, ->result:
- all equally intuitive (aka: even if you don't get it yourself, once told, you're unlikely to forget what it means).
case mylib.STATUS_OK, ^result: case mylib.STATUS_OK, ?result: case mylib.STATUS_OK, @result:
- are much less so IMHO.
[]
Steve
On Sun, Nov 15, 2020 at 03:18:29PM +0300, Paul Sokolovsky wrote:
amount of work to implement pattern matching is certainly an order of magnitude larger than to introduce constants
[...]
[Steve]
Here's a toy proposal,
[]
The baseline of my version is much simpler:
# This makes "const" a kind of hard keyword for this module from __future__ import const
FOO: const = 1 # obviously, this is constant Foo_Bar: const = 2 # this is too foo: const = 3 # and like it or not, but this is constant either
"const" annotation is recognized by compiler (scope resolving component, aka symtable builder to be exact) as a property orthogonal to localness/globalness. Literally, "symtable" module (https://docs.python.org/3/library/symtable.html#symtable.Symbol) will grow Symbol.is_const() method.
Oh, well, if all it takes is to add a new keyword, then constants are easy! No need to worry about how constantness affects name resolution, or how the syntax interacts with type-hints:
spam: const: Widget = build_widget(*args) # Maybe this is better? # let spam: Widget = build_widget(*args)
or how "constant" interacts with mutability:
spam: const = [] spam.append(99) # Allowed? spam.extend(items) spam.sort()
or for that matter, whether or not constants are actually constant.
spam: const = 1 spam = 2
If constants aren't actually constant, but just a tag on symbols, then you would be right, it probably would be trivially easy to add "constants" to Python.
But I think most people agree making them behave as constants is a pretty important feature. *The* most critical feature of all, really.
Given a declared constant:
# Module a.py x: const = None
how do you prevent not just code in module `a` from rebinding the value:
x = 1 globals()['x'] = 2 exec('x = 3', globals())
but other modules as well:
# Module b.py import a a.x = 'spam' vars(a)['x'] = 'spam' exec('x = 3', vars(a))
Hello,
On Mon, 16 Nov 2020 08:39:30 +1100 Steven D'Aprano steve@pearwood.info wrote:
[]
The baseline of my version is much simpler:
# This makes "const" a kind of hard keyword for this module from __future__ import const
FOO: const = 1 # obviously, this is constant
Oh, well,
To start with, in the original thread I wanted to concentrate on issues of the PEP634/PEP635, and whether these issues are prominent enough to hope for them being addressed. So, I changed the subject (and we'd soon be excused to python-ideas, indeed, I cc: there).
if all it takes is to add a new keyword, then constants are easy!
A new annotation. And the new subject is "constants in Python: Starting simple and gradually adding more", which hopefully should set the theme. Please remember how we arrived here: it's from the fact that PEP634 doesn't allow for the following trivial code:
MY_CONST = 1 match foo: case MY_CONST:
We're considering (another alternative) how to address that. Please don't make a leap towards supporting (at once) const-ness equivalent to statically typed languages.
No need to worry about how constantness affects name resolution, or how the syntax interacts with type-hints:
"const" is *the* type hint.
spam: const: Widget = build_widget(*args)
That syntax is clearly invalid. And composability of type annotations (aka type hints) is a known, looming issue. We now have https://www.python.org/dev/peps/pep-0593/ , but in all fairness, it seems like a stopgap measure rather than an elegant solution. (In full fairness, entire "typing" module's annotations aren't very elegant, but as we know, it's not a part of language core, but part of CPython's stdlib. That means it's only *one* of possible annotation schemes for *Python*).
So, under PEP593, the above would be written
spam: Annotated[Widget, const] = build_widget(*args)
If you want a glimpse of what alternatives might look, then: given that "|" is going to be used for union types, why not try "&" for composition?
spam: Widget & const = build_widget(*args)
But again, for "initial support of constants in Python, prompted by the introduction of pattern matching facilities", we don't need to worry about all that right away.
# Maybe this is better? # let spam: Widget = build_widget(*args)
As you remember, at the beginning of my proposal, I wrote "The baseline of my version ...". Baseline == level 0. For "level 2", I considered
const spam: Widget = ...
I don't think that "let" should be used. We're putting emphasis on *constantness* (not lexical scoping of variables [immutable by functional tradition, though that's "minor"]). JavaScript (now) has both "const" and "let", and I think that's pretty reasonable approach. So, let's save "let" for possible later uses.
So, what's "level 1"?
As I mentioned, under "level 0", "from __future__ import const" has a magic meaning (akin to other "from __future__"'s). Under "level 1", "const" is just a normal annotation symbol imported from a module. So, following would be possible:
========= from __future__ import const
FOO: const = 1
def fun(): const = 1 # Just a variable in function scope
def sub(): # Gimme powerz back from __future__ import const BAR: const = 2
# Back to annotation in global scope BAZ: const = 3 =========
"level 0" should be implementable in CPython in ~1hr. "level 1" is realistically what we should shoot for. "level 2" (the dedicated keyword), I'm still not sure about. "const" is very baseline annotation, and deserves a dedicated keyword. But the underlying problem is composability of (arbitrary) annotations. I'd rather keep searching for graal in that regard, before giving up and introduce a dedicated thing just for "const".
or how "constant" interacts with mutability:
spam: const = [] spam.append(99) # Allowed? spam.extend(items) spam.sort()
"const" is an annotation just like any other. And it affects *runtime* in the same as any other annotation in CPython affects it: in no way.
"const" is however introduced as a hint for *compile-time*, so compiler could make some simple inferences and maybe even optimizations based on it.
or for that matter, whether or not constants are actually constant.
spam: const = 1 spam = 2
Compiler can, and thus would, catch that as an error (warning for a beta version?).
If constants aren't actually constant, but just a tag on symbols, then you would be right, it probably would be trivially easy to add "constants" to Python.
Right.
But I think most people agree making them behave as constants is a pretty important feature.
As mentioned, "const" is just an annotation like any other, except compiler has some insight into it. Dealing with runtime is distant goal for CPython. (Which is for now offloaded to static type checkers and libraries/alternative Python implementations.)
*The* most critical feature of all, really.
Given a declared constant:
# Module a.py x: const = Nonehow do you prevent not just code in module `a` from rebinding the value:
Outside of the current scope of the discussion.
However, if you're interested in that topic, then I've implemented it in my Python dialect, Pycopy (https://github.com/pfalcon/pycopy). It's on my TODO to post an RFC to python-ideas for beating.
[]
case mylib.STATUS_OK, >result: case mylib.STATUS_OK, >>result: case mylib.STATUS_OK, ->result:
The second problem with those is that ">" has a very strong tie to "greater than". I think -> or even >> *might* be enough to overcome that, but I'm not comfortable.
(The first problem, of course, is that the PEP authors are still reluctant to require any sigil at all, for what they assume will quickly become the default case.)
-jJ
Hi Paul,
Thank you for your comments on the DLS'20 paper. I am glad to hear that it helps paint a clear(er) picture of pattern matching in Python. However, please let me set the record straight in a few regards.
First, there is no 'shoehorning' or 'deception' in our pattern matching PEPs. This notion of our team standing against the Python community or trying to outsmart it is quite absurd and unnecessary, really. At the end of the day, you might find that the PEPs are a boiled-down version of months of intense work and discussions (as Brandt has already pointed out elsewhere) and thus necessarily terse in some parts. Pattern matching is a complex subject and the PEPs have been written with a sincere effort to convey the big picture and give a tour of this feature in an accessible manner. Our entire discussion is all openly available to anyone interested.
Second, our work on pattern matching is based on a very simple premise/idea: *bring pattern matching to Python*. In particular, it is *not* to 'bring Python to pattern matching'. We want to introduce pattern matching as a new feature to Python as Python stands right now, without the need to first modify Python and make it more static, C-like, functional or whatever. As I have mentioned before, pattern matching is an opt-in feature, something with absolutely no influence on the behaviour of any Python program you have ever written before. Not introducing constants or a completely new kind of scopes to Python first is certainly not a lack of academic rigour, oversight on our part because we are not smart enough, or trick to coerce people into accepting pattern matching. It just means that we propose to introduce pattern matching to Python as it is right now.
Third, the academic paper has a different scope than the PEPs. In case you missed it: after the first version of PEP 622, we listened to the reactions and suggestions from the community and went back to overhaul the entire design and build as much as possible on the raised concerns and cummulative experience of the community. One major outcome of this was to reduce the overall proposal to the absolute core features of what we need for pattern matching (another was to put more effort into writing an accessible account of what we propose and why we think the various design features are a good choice, leading to three PEPs with different aims). The academic paper outlines a much larger scope of possibilities than the PEPs, whereas the PEPs are more grounded in the pragmatic and taking one step at a time. In short, the academic paper lays out an entire journey, a vision, whereas the PEPs propose a first step forward.
Finally, the reason why the academic paper has not been referred to before is simply that it will officially be published coming (!) week. Since the process of peer review is now complete, we can already provide a preprint. It was clear from the outset that the PEPs will contain a link to the paper as soon as it becomes publicly available.
Of course, we encourage anyone interested in it to read the academic paper. Since its focus is somewhat complementary to the PEPs, it might highlight some ideas behind our design that might not be made clear enough in the PEPs. But while doing so, please keep in mind that the paper is a vision of a bigger picture and as such distinct from the PEPs!
Kind regards, Tobias
Quoting Paul Sokolovsky pmiscml@gmail.com:
Hello,
As was mentioned many times on the list, PEP634-PEP636 are thoroughly prepared and good materials, many thanks to their authors. PEP635 "Motivation and Rationale" (https://www.python.org/dev/peps/pep-0635/) stands out among the 3 however: while reading it, chances that you'll get a feeling of "residue", accumulating a section over section. By the end of reading, you may get a well-formed feeling that you've read a half-technical, half-marketing material, which is intended to "sell" a particular idea among many other very viable ideas, by shoehorning some concepts, downplaying other ideas, and at the same time, light-heartedly presenting drawbacks of its subject one.
Just to give one example, literally at the very beginning, at the "Pattern Matching and OO" section (3rd heading) it says:
Pattern matching is complimentary to the object-oriented paradigm.
It's not until the very end of document, in the "History and Context" it tells the whole truth:
With its emphasis on abstraction and encapsulation, object-oriented programming posed a serious challenge to pattern matching.
You may wonder how "complimentary" and "posed a serious challenge" relate to each other. While they're definitely not contradictory, starting the document with light-hearted "complimentary" can be seen as trying to set the stage where readers don't pay enough attention to the problem. And it kinda worked: only now [1] wider community discovers the implications of "Class Patterns" choices. (As a note, PEP635 does well on explaining them, and I'm personally sold on that, but it's *tough* choice, not the *obvious* choice).
There're many more examples like that in the PEP635, would take too much space to list them all. However, PEP635 refers to the paper:
Kohn et al., Dynamic Pattern Matching with Python https://doi.org/10.1145/3426422.3426983 (Accepted by DLS 2020. The link will go live after Nov. 17; a preview PDF can be obtained from the first author.)
As that citation suggests, the paper is not directly linked from the PEP635. But the preprint is now accessible from the conference page, https://conf.researchr.org/home/dls-2020?#event-overview (direct link as of now: https://gvanrossum.github.io//docs/PyPatternMatching.pdf).
That paper is written at much higher academic standard, and a pleasure to read. I recommend it to everyone who read PEP635 (note that it was written with PEP622 in mind, but conceptual differences are minor). With it, I noticed just 2 obvious issues:
Section 4.3. Named Constants
It would clearly be desirable to allow named constants in patterns as a replacement and extension of literals. However, Python has no concept of a constant, i.e. all variables are mutable (even where the values themselves are immutable).
So, unlike PEP635, the paper pinpoints right the root of PEP634's problems: lack of constants in Python (on the language level). This is just the point which was raised on the mailing list either (https://mail.python.org/archives/list/python-dev@python.org/message/WV2UA4AK...).
Under strict academic peer review, the paper would have been returned for elaboration, with a note like: "Given that nowadays many dynamic languages (PHP, JavaScript, Lua, etc.) have support for constants, and lack of constants poses a serious usability challenge to your proposal, please explain why you chose to proceed anyway (and apply workarounds), instead of first introducing the concept of constants to the language. (Given that amount of work to implement pattern matching is certainly an order of magnitude larger than to introduce constants)."
But the paper wasn't returned for elaboration, so we'll keep wondering why the authors chose such a backward process.
Section 6.1. Scope
The granularity of the scope of local variables is at the level of functions and frames. [...] The only way to restrict the scope of a variable to part of a function’s body (such as a case clause) would be to actively delete the variable when leaving the block. This would, however, not restore any previous value of a local variable in the function’s scope.
This is a misconception ("the only way") which is repeated almost one to one on PEP635 either. If anything, the above describes how pseudo-scoping is currently implemented for exception vars in "except Exception as e:" clause (more info: https://mail.python.org/pipermail/python-dev/2019-January/155991.html), which is hardly a "best practice", and definitely not the only way.
How to support multiple variable scopes in one stack frame is not a rocket science at all. One just need to remember how C did that since ~forever. And that's: (for unoptimized programs) variables in different scopes live in disjoint subsections of a frame. (For optimized programs, variables with disjoint liveness can share the same locations in a frame).
The only reasons for not implementing the same solution in Python would be intertia of thought and "but it's not done anywhere else in Python". Yeah, but nobody del'eted local variables behind users' backs either, before somebody started to do that for exception clause variables. And the whole of pattern matching is such that either one thing, or another, but will be done for the first time in Python. For example, nobody before could imagine that one can write "Point(x, y)", and get x and y assigned, and now we're facing just that [1]. (Again, I personally love it, though think that "Point(>x, >y)" is an interesting option to address the tensions).
In that regard, the current PEP634 and friends miss too many interesting and useful opportunities (constants in language core and true scoping for match'es, to name a few). Well, that happens. But they try to shoehorn too much of "we didn't do" into "it's not possible" or "it doesn't make sense", or "let's workaround it in adhoc ways" and that raises eyebrows, leading to concerns of whether the proposals are actually "raw" as of yet.
[1] People expressing surprise at "Class Patterns" syntax: https://mail.python.org/archives/list/python-dev@python.org/message/F66J72JU... https://mail.python.org/archives/list/python-dev@python.org/message/Q2ARJULL...
-- Best regards, Paul mailto:pmiscml@gmail.com _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/EQX6GK7C... of Conduct: http://python.org/psf/codeofconduct/
Hello Tobias,
On Sun, 15 Nov 2020 13:22:07 +0100 Tobias Kohn kohnt@tobiaskohn.ch wrote:
Hi Paul,
Thank you for your comments on the DLS'20 paper. I am glad to hear that it helps paint a clear(er) picture of pattern matching in Python. However, please let me set the record straight in a few regards.
That's definitely true, and thanks for this reply too, as it indeed sets many point straight regarding the PEP635. I'd say that I'm on my side largely satisfied both with reading materials and responses of Brandt and yours. But we still have PEPs on the plate which raise concern from different people (not just myself), so please let me continue.
First, there is no 'shoehorning' or 'deception'
Hey, I didn't use that word ;-)
in our pattern matching PEPs. This notion of our team standing against the Python community or trying to outsmart it is quite absurd and unnecessary, really. At the end of the day, you might find that the PEPs are a boiled-down version of months of intense work and discussions (as Brandt has already pointed out elsewhere) and thus necessarily terse in some parts.
This is very good point, and I easily believe it. In an alternative universe, if there would be any bias, conscious or subconscious, it's still would be good-intentioned, for sure. For example, suppose that you, the PEP authors, thought out all choices, and found insurmountable issues with all of them, but one. Knowing that the SC would need to decide on them, it would be only naturally to present material in a way to not take their precious time with dead-ends, and present only the winning proposal. There's absolutely nothing wrong with that, I myself could do the same.
The whole culprit under this point of view are exactly some people from community who continue to search for gold in a mine you know empty. The role of community is at all not clear. It's only relatively recently aspects of Python decision-making caught wide attention, with PEP572. And as we know, that was a point where decision process changed considerably either. And... we still don't know how it works in details (only learning it). Maybe it's largely a matter between PEP authors and SC, then PEPs are definitely "ok" and nothing can be improved.
What concerns community has, can be seen from some posts, e.g. https://mail.python.org/archives/list/python-dev@python.org/message/IG4VNKZF... (the title: "PEP 622 railroaded through?"). I'm sorry if I join this perceived "conspiracy theory". Just as that's not the intention of PEPs, neither it's the intention of feedback like that. Finding omissions (yes, after so much work on PEPs!) and hope for improvements is the intention.
Pattern matching is a complex subject and the PEPs have been written with a sincere effort to convey the big picture and give a tour of this feature in an accessible manner. Our entire discussion is all openly available to anyone interested.
Second, our work on pattern matching is based on a very simple premise/idea: *bring pattern matching to Python*. In particular, it is *not* to 'bring Python to pattern matching'. We want to introduce pattern matching as a new feature to Python as Python stands right now, without the need to first modify Python and make it more static, C-like, functional or whatever.
Makes sense, and your choice. But not entirely consistent with what you write in the paper:
In contrast to pattern matching in other dynamic languages [3, 11], we implement it as a compiled feature of the language in the tradition of static languages [6, 22] rather than introducing patterns as first-class objects.
Again, it's your right to stop where you see reasonable. It's the right of reviewers to point out that you stopped half-way, and even criticize such a choice.
As I have mentioned before, pattern matching is an opt-in feature, something with absolutely no influence on the behaviour of any Python program you have ever written before.
Another good argument I fully agree with my. I still have a counter-argument: pattern matching seems like an important and conceptually deep enough change, which could be compared to switching Python to proper lexical scoping, or switching multiple inheritance to C3 algorithm away from the adhoc one which was used before. Not done right, pattern matching risks integrity and soundness of language.
Not introducing constants or a completely new kind of scopes to Python first is certainly not a lack of academic rigour, oversight on our part because we are not smart enough, or trick to coerce people into accepting pattern matching. It just means that we propose to introduce pattern matching to Python as it is right now.
Makes sense again. And you're well aware of counter-arguments, as polar as "we don't need pattern matching in Python". You definitely do, as was a subsection in PEP622 with literal heading "Don't do this, pattern matching is hard to learn". But if you accept that such a point of view exist, you should also accept that "we don't need *such* pattern matching" viewpoint also exists.
Straight off, that's not me. But after digging into this stuff, I now understand better people who say that PEP634 isn't good enough. I no longer think they're progress-impeders or just ungrateful dudes. Rather, I listen to an ideas to not haste with it, and see what can be improved.
Third, the academic paper has a different scope than the PEPs. In case you missed it: after the first version of PEP 622, we listened to the reactions and suggestions from the community and went back to overhaul the entire design and build as much as possible on the raised concerns and cummulative experience of the community.
I didn't miss it, and that's much appreciated. (I for one was in "homework" mode on that stuff all this time, and my interest to dig deeper is based on "why some other people aren't satisfied with it".)
One major outcome of this was to reduce the overall proposal to the absolute core features of what we need for pattern matching (another was to put more effort into writing an accessible account of what we propose and why we think the various design features are a good choice, leading to three PEPs with different aims).
I again appreciate that, and it's very visible, and definitely an improvement. But for example, in academic research, as far as I know, you rarely get suggestions to just narrow your scope. Usually, it's narrowing scope *and* going deeper on some aspects. As you mention, academic research and PEPs are different matters, but I wonder if there're some points in common still.
The academic paper outlines a much larger scope of possibilities than the PEPs, whereas the PEPs are more grounded in the pragmatic and taking one step at a time. In short, the academic paper lays out an entire journey, a vision, whereas the PEPs propose a first step forward.
Finally, the reason why the academic paper has not been referred to before is simply that it will officially be published coming (!) week. Since the process of peer review is now complete, we can already provide a preprint. It was clear from the outset that the PEPs will contain a link to the paper as soon as it becomes publicly available.
Thanks again for making the paper available and good luck at the conference next week (I assume there will be virtual presentation or something)!
PEP-wise, I don't know what to take from all this. How I'd formulate it: for a slightest chance that at the SC review there would be feedback like "we don't think that the PEPs considered all options/implications thoroughly enough, and ask for additional consideration of alternatives", then to not treat it like "you must be kidding us", but find champions among the PEP team for different alternative approaches (as Brandt mentioned, there were such), and try to look into those alternatives again.
Thanks!
Of course, we encourage anyone interested in it to read the academic paper. Since its focus is somewhat complementary to the PEPs, it might highlight some ideas behind our design that might not be made clear enough in the PEPs. But while doing so, please keep in mind that the paper is a vision of a bigger picture and as such distinct from the PEPs!
Kind regards, Tobias
Quoting Paul Sokolovsky pmiscml@gmail.com:
Hello,
As was mentioned many times on the list, PEP634-PEP636 are thoroughly prepared and good materials, many thanks to their authors. PEP635 "Motivation and Rationale" (https://www.python.org/dev/peps/pep-0635/) stands out among the 3 however: while reading it, chances that you'll get a feeling of "residue", accumulating a section over section. By the end of reading, you may get a well-formed feeling that you've read a half-technical, half-marketing material, which is intended to "sell" a particular idea among many other very viable ideas, by shoehorning some concepts, downplaying other ideas, and at the same time, light-heartedly presenting drawbacks of its subject one.
Just to give one example, literally at the very beginning, at the "Pattern Matching and OO" section (3rd heading) it says:
Pattern matching is complimentary to the object-oriented paradigm.
It's not until the very end of document, in the "History and Context" it tells the whole truth:
With its emphasis on abstraction and encapsulation, object-oriented programming posed a serious challenge to pattern matching.
You may wonder how "complimentary" and "posed a serious challenge" relate to each other. While they're definitely not contradictory, starting the document with light-hearted "complimentary" can be seen as trying to set the stage where readers don't pay enough attention to the problem. And it kinda worked: only now [1] wider community discovers the implications of "Class Patterns" choices. (As a note, PEP635 does well on explaining them, and I'm personally sold on that, but it's *tough* choice, not the *obvious* choice).
There're many more examples like that in the PEP635, would take too much space to list them all. However, PEP635 refers to the paper:
Kohn et al., Dynamic Pattern Matching with Python https://doi.org/10.1145/3426422.3426983 (Accepted by DLS 2020. The link will go live after Nov. 17; a preview PDF can be obtained from the first author.)
As that citation suggests, the paper is not directly linked from the PEP635. But the preprint is now accessible from the conference page, https://conf.researchr.org/home/dls-2020?#event-overview (direct link as of now: https://gvanrossum.github.io//docs/PyPatternMatching.pdf).
That paper is written at much higher academic standard, and a pleasure to read. I recommend it to everyone who read PEP635 (note that it was written with PEP622 in mind, but conceptual differences are minor). With it, I noticed just 2 obvious issues:
Section 4.3. Named Constants
It would clearly be desirable to allow named constants in patterns as a replacement and extension of literals. However, Python has no concept of a constant, i.e. all variables are mutable (even where the values themselves are immutable).
So, unlike PEP635, the paper pinpoints right the root of PEP634's problems: lack of constants in Python (on the language level). This is just the point which was raised on the mailing list either (https://mail.python.org/archives/list/python-dev@python.org/message/WV2UA4AK...).
Under strict academic peer review, the paper would have been returned for elaboration, with a note like: "Given that nowadays many dynamic languages (PHP, JavaScript, Lua, etc.) have support for constants, and lack of constants poses a serious usability challenge to your proposal, please explain why you chose to proceed anyway (and apply workarounds), instead of first introducing the concept of constants to the language. (Given that amount of work to implement pattern matching is certainly an order of magnitude larger than to introduce constants)."
But the paper wasn't returned for elaboration, so we'll keep wondering why the authors chose such a backward process.
Section 6.1. Scope
The granularity of the scope of local variables is at the level of functions and frames. [...] The only way to restrict the scope of a variable to part of a function’s body (such as a case clause) would be to actively delete the variable when leaving the block. This would, however, not restore any previous value of a local variable in the function’s scope.
This is a misconception ("the only way") which is repeated almost one to one on PEP635 either. If anything, the above describes how pseudo-scoping is currently implemented for exception vars in "except Exception as e:" clause (more info: https://mail.python.org/pipermail/python-dev/2019-January/155991.html), which is hardly a "best practice", and definitely not the only way.
How to support multiple variable scopes in one stack frame is not a rocket science at all. One just need to remember how C did that since ~forever. And that's: (for unoptimized programs) variables in different scopes live in disjoint subsections of a frame. (For optimized programs, variables with disjoint liveness can share the same locations in a frame).
The only reasons for not implementing the same solution in Python would be intertia of thought and "but it's not done anywhere else in Python". Yeah, but nobody del'eted local variables behind users' backs either, before somebody started to do that for exception clause variables. And the whole of pattern matching is such that either one thing, or another, but will be done for the first time in Python. For example, nobody before could imagine that one can write "Point(x, y)", and get x and y assigned, and now we're facing just that [1]. (Again, I personally love it, though think that "Point(>x, >y)" is an interesting option to address the tensions).
In that regard, the current PEP634 and friends miss too many interesting and useful opportunities (constants in language core and true scoping for match'es, to name a few). Well, that happens. But they try to shoehorn too much of "we didn't do" into "it's not possible" or "it doesn't make sense", or "let's workaround it in adhoc ways" and that raises eyebrows, leading to concerns of whether the proposals are actually "raw" as of yet.
[1] People expressing surprise at "Class Patterns" syntax: https://mail.python.org/archives/list/python-dev@python.org/message/F66J72JU... https://mail.python.org/archives/list/python-dev@python.org/message/Q2ARJULL...
-- Best regards, Paul mailto:pmiscml@gmail.com _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/EQX6GK7C... of Conduct: http://python.org/psf/codeofconduct/
Complimentary != Complementary
On Sun, Nov 15, 2020, 4:51 AM Paul Sokolovsky pmiscml@gmail.com wrote:
Hello,
As was mentioned many times on the list, PEP634-PEP636 are thoroughly prepared and good materials, many thanks to their authors. PEP635 "Motivation and Rationale" (https://www.python.org/dev/peps/pep-0635/) stands out among the 3 however: while reading it, chances that you'll get a feeling of "residue", accumulating a section over section. By the end of reading, you may get a well-formed feeling that you've read a half-technical, half-marketing material, which is intended to "sell" a particular idea among many other very viable ideas, by shoehorning some concepts, downplaying other ideas, and at the same time, light-heartedly presenting drawbacks of its subject one.
Just to give one example, literally at the very beginning, at the "Pattern Matching and OO" section (3rd heading) it says:
Pattern matching is complimentary to the object-oriented paradigm.
It's not until the very end of document, in the "History and Context" it tells the whole truth:
With its emphasis on abstraction and encapsulation, object-oriented programming posed a serious challenge to pattern matching.
You may wonder how "complimentary" and "posed a serious challenge" relate to each other. While they're definitely not contradictory, starting the document with light-hearted "complimentary" can be seen as trying to set the stage where readers don't pay enough attention to the problem. And it kinda worked: only now [1] wider community discovers the implications of "Class Patterns" choices. (As a note, PEP635 does well on explaining them, and I'm personally sold on that, but it's *tough* choice, not the *obvious* choice).
There're many more examples like that in the PEP635, would take too much space to list them all. However, PEP635 refers to the paper:
Kohn et al., Dynamic Pattern Matching with Python https://doi.org/10.1145/3426422.3426983 (Accepted by DLS 2020. The link will go live after Nov. 17; a preview PDF can be obtained from the first author.)
As that citation suggests, the paper is not directly linked from the PEP635. But the preprint is now accessible from the conference page, https://conf.researchr.org/home/dls-2020?#event-overview (direct link as of now: https://gvanrossum.github.io//docs/PyPatternMatching.pdf).
That paper is written at much higher academic standard, and a pleasure to read. I recommend it to everyone who read PEP635 (note that it was written with PEP622 in mind, but conceptual differences are minor). With it, I noticed just 2 obvious issues:
Section 4.3. Named Constants
It would clearly be desirable to allow named constants in patterns as a replacement and extension of literals. However, Python has no concept of a constant, i.e. all variables are mutable (even where the values themselves are immutable).
So, unlike PEP635, the paper pinpoints right the root of PEP634's problems: lack of constants in Python (on the language level). This is just the point which was raised on the mailing list either ( https://mail.python.org/archives/list/python-dev@python.org/message/WV2UA4AK... ).
Under strict academic peer review, the paper would have been returned for elaboration, with a note like: "Given that nowadays many dynamic languages (PHP, JavaScript, Lua, etc.) have support for constants, and lack of constants poses a serious usability challenge to your proposal, please explain why you chose to proceed anyway (and apply workarounds), instead of first introducing the concept of constants to the language. (Given that amount of work to implement pattern matching is certainly an order of magnitude larger than to introduce constants)."
But the paper wasn't returned for elaboration, so we'll keep wondering why the authors chose such a backward process.
Section 6.1. Scope
The granularity of the scope of local variables is at the level of functions and frames. [...] The only way to restrict the scope of a variable to part of a function’s body (such as a case clause) would be to actively delete the variable when leaving the block. This would, however, not restore any previous value of a local variable in the function’s scope.
This is a misconception ("the only way") which is repeated almost one to one on PEP635 either. If anything, the above describes how pseudo-scoping is currently implemented for exception vars in "except Exception as e:" clause (more info: https://mail.python.org/pipermail/python-dev/2019-January/155991.html), which is hardly a "best practice", and definitely not the only way.
How to support multiple variable scopes in one stack frame is not a rocket science at all. One just need to remember how C did that since ~forever. And that's: (for unoptimized programs) variables in different scopes live in disjoint subsections of a frame. (For optimized programs, variables with disjoint liveness can share the same locations in a frame).
The only reasons for not implementing the same solution in Python would be intertia of thought and "but it's not done anywhere else in Python". Yeah, but nobody del'eted local variables behind users' backs either, before somebody started to do that for exception clause variables. And the whole of pattern matching is such that either one thing, or another, but will be done for the first time in Python. For example, nobody before could imagine that one can write "Point(x, y)", and get x and y assigned, and now we're facing just that [1]. (Again, I personally love it, though think that "Point(>x, >y)" is an interesting option to address the tensions).
In that regard, the current PEP634 and friends miss too many interesting and useful opportunities (constants in language core and true scoping for match'es, to name a few). Well, that happens. But they try to shoehorn too much of "we didn't do" into "it's not possible" or "it doesn't make sense", or "let's workaround it in adhoc ways" and that raises eyebrows, leading to concerns of whether the proposals are actually "raw" as of yet.
[1] People expressing surprise at "Class Patterns" syntax:
https://mail.python.org/archives/list/python-dev@python.org/message/F66J72JU...
https://mail.python.org/archives/list/python-dev@python.org/message/Q2ARJULL...
-- Best regards, Paul mailto:pmiscml@gmail.com _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/EQX6GK7C... Code of Conduct: http://python.org/psf/codeofconduct/
On 2020-11-15 09:48, Paul Sokolovsky wrote:
Hello,
As was mentioned many times on the list, PEP634-PEP636 are thoroughly prepared and good materials, many thanks to their authors. PEP635 "Motivation and Rationale" (https://www.python.org/dev/peps/pep-0635/) stands out among the 3 however: while reading it, chances that you'll get a feeling of "residue", accumulating a section over section. By the end of reading, you may get a well-formed feeling that you've read a half-technical, half-marketing material, which is intended to "sell" a particular idea among many other very viable ideas, by shoehorning some concepts, downplaying other ideas, and at the same time, light-heartedly presenting drawbacks of its subject one.
Just to give one example, literally at the very beginning, at the "Pattern Matching and OO" section (3rd heading) it says:
Pattern matching is complimentary to the object-oriented paradigm.
That looks like a mistake to me; it should be "complementary".
[snip]
Hello,
On Sun, 15 Nov 2020 16:58:09 +0000 MRAB python@mrabarnett.plus.com wrote:
[]
Pattern matching is complimentary to the object-oriented paradigm.
That looks like a mistake to me; it should be "complementary".
This way or that, my point is that even the word "orthogonal" wouldn't give it fairness in that context. It should have gone straight to "conflicting", to raise the alertness level to the somewhat-hard on the perception matters which (eventually, a dozen or so headings below) follow. Then maybe there wouldn't be surprises several months later like "Oh really, 'case Point(x=a, y=b):' assigns to a and b??77"
(Only maybe, perception is hard thing to play with. Though I truly believe that people would pick it up subconsciously quickly enough, after making a few mistakes. Then again, that's why I propose for consideration an option to write it `case Point(x = >a, y = >b):`, to cut mistakes to ~0).
[]
On 15/11/20 10:48 pm, Paul Sokolovsky wrote:
[from PEP 635]
Pattern matching is complimentary to the object-oriented paradigm.
BTW, there seems to be a typo here -- I think it's meant to be "complementary".
please explain why you chose to proceed anyway (and apply workarounds), instead of first introducing the concept of constants to the language. (Given that amount of work to implement pattern matching is certainly an order of magnitude larger than to introduce constants)."
That's not certain at all, and moreover it's not just a matter of work, it's a language design issue that would require its own extensive investigation and debate. Members of the intended audience (people very familiar with Python and its technicalities) can be expected to understand this, so the PEP doesn't need to spell it out.
How to support multiple variable scopes in one stack frame is not a rocket science at all. One just need to remember how C did that
We can't just do it "like C does" because C requires variables to be declared and Python doesn't.
The only reasons for not implementing the same solution in Python would be intertia of thought and "but it's not done anywhere else in Python".
No, other reasons include that it would require making unrelated language changes that open various other wormcans.
nobody del'eted local variables behind users' backs either, before somebody started to do that for exception clause variables.
There was a good reason for that having to do with reference cycles. Nobody liked it much, but we didn't really have a choice. There is no such pressing need for special scope rules in match statements.
Hello,
On Mon, 16 Nov 2020 12:21:35 +1300 Greg Ewing greg.ewing@canterbury.ac.nz wrote:
[]
please explain why you chose to proceed anyway (and apply workarounds), instead of first introducing the concept of constants to the language. (Given that amount of work to implement pattern matching is certainly an order of magnitude larger than to introduce constants)."
That's not certain at all, and moreover it's not just a matter of work, it's a language design issue that would require its own extensive investigation and debate.
Everything would need discussion, maybe even debate. So, currently we debate various conceptual holes in the pattern matching proposal (like inability to match against a plain simple constant). We could discuss/debate things which would allow to resolve that and other issues in more proper way, that's the whole point.
And we don't speak about some obscure "innovative" idea. Const'ness aka immutability is well-known and widely used feature in programming languages. So, we at least can be sure we're on the right path. Whereas now we're standing on one lag and wonder why our walking that way is awkward.
Members of the intended audience (people very familiar with Python and its technicalities) can be expected to understand this, so the PEP doesn't need to spell it out.
So, the passage you quoted (and the whole original mail) was mostly related to the DLS'20 paper (https://gvanrossum.github.io//docs/PyPatternMatching.pdf), not PEP. I won't argue what PEP should do (under one interpretation, given that PEPs are accepted by SC, it might be even empty if SC is ready to accept it like that).
But for a scientific paper which already considers various choices, omitting more detailed discussion of a pretty obvious choice caught some attention.
How to support multiple variable scopes in one stack frame is not a rocket science at all. One just need to remember how C did that
We can't just do it "like C does" because C requires variables to be declared and Python doesn't.
Again, my original mail criticized such simplistic write-offs in the DLS'20 paper, and in a reply to it, we again get the same story.
So, let's go over it again. You probably say that C uses following syntax to define a variable in a particular scope:
<type> <var>;
e.g.:
int var;
Right, Python doesn't have that syntax. But it has other syntaxes to designate that a variable is defined in a particular scope:
# e is defined in the scope of the following block except Exception as e:
# x is defined in the scope of a comprehension [x for x in seq]
Then for pattern matching,
case a, b:
*Could* define a, b in the scope of following block (and not beyond). That would have pros and cons, an obvious pro is that failing-to-match patterns would not litter in surrounding namespace, and con that "pattern matching as expression" usage would be more harder. If anything, use of block scoping could be considered to alleviate the littering problem, because the above should be equivalent to:
case _1, _2: a, b = _1, _2
Where _1 and _2 *are* block-scoped *variables* (even if a and b aren't).
Since then, "The semantics of pattern matching for Python" topic presented very similar ideas, except that it again tried to preserve the tabu on block-level scoping and argued for stack-machine workarounds. Instead of mentioning the shining-thru idea that it's perfect usecase for generic block-level scoping.
Finally (and that's why this thread is brought up again), in a parallel thread on python-ideas (https://mail.python.org/archives/list/python-ideas@python.org/thread/3ZKYV6W...), we see that block scoping would fit the bill to alleviate some issues with the scoping of "for" loop variables.
The only reasons for not implementing the same solution in Python would be intertia of thought and "but it's not done anywhere else in Python".
No, other reasons include that it would require making unrelated language changes that open various other wormcans.
Let's strive for solutions which follow the best practices in the programming language design (const'ness, block-level scoping are 2 good examples), and that should increase benefits/effort ratio of the solutions.
nobody del'eted local variables behind users' backs either, before somebody started to do that for exception clause variables.
There was a good reason for that having to do with reference cycles. Nobody liked it much, but we didn't really have a choice. There is no such pressing need for special scope rules in match statements.
"Undefined behavior" regarding how failed matches may affect surrounding namespace is unlikely something to be proud of.
Whether it's "pressing" or not, I won't debate. The point I want to show is that we have some history of workarounds for scoping-related matters. We're about to make workarounds (or just close eyes and say "there's no problem") in regard to pattern matching and/or "for" scoping improvement.
Whereas all these cases seem to fit the bill of the generic block-level scoping pretty well, and discussing that approach might have bigger long-term benefits than keeping to apply adhoc workarounds.
-- Greg
On Sun, Nov 29, 2020 at 12:10:39AM +0300, Paul Sokolovsky wrote:
And we don't speak about some obscure "innovative" idea. Const'ness aka immutability is well-known and widely used feature in programming languages.
Constantness and immutability are not synonyms.
Immutability refers to *objects*. "abc" is an immutable object; `[]` is not, it is mutable.
Constantness refers to *name bindings*: once the name is bound to an object, it cannot be re-bound to a different object.
So you can have:
- variables bound to a mutable object; - variables bound to an immutable object; - constants bound to a mutable object; - constants bound to an immutable object.
and you could have the same object bound to multiple names, some of which could be variables and some constants.
A mutable constant could still be mutated:
const spam = [] spam.append(None) # Allowed.
but not rebound:
spam = [1, 2] # Forbidden.
So, let's go over it again. You probably say that C uses following syntax to define a variable in a particular scope:
<type> <var>;
e.g.:
int var;
Right, Python doesn't have that syntax. But it has other syntaxes to designate that a variable is defined in a particular scope:
# e is defined in the scope of the following block except Exception as e:
That is incorrect. e is defined in the *current* scope. There is no "scope of the following block".
Exception blocks are a special case, because there is dedicated code to unbind (delete) the "e" name when the block is left. But the name is local to the current scope, the block does not create a new scope.
That same applies to all other blocks.
# x is defined in the scope of a comprehension [x for x in seq]
Then for pattern matching,
case a, b:
*Could* define a, b in the scope of following block (and not beyond). That would have pros and cons, an obvious pro is that failing-to-match patterns would not litter in surrounding namespace, and con that "pattern matching as expression" usage would be more harder. If anything, use of block scoping could be considered to alleviate the littering problem
The "littering problem" will be a non-problem in practice.
Let's strive for solutions which follow the best practices in the programming language design (const'ness, block-level scoping are 2 good examples),
There isn't even a single definition for block-scoping. C block scoping and Java block scoping are not the same. So which one is best practice?
At *best* it is a matter of personal taste whether you like or dislike block scoping, but I'm going to stick my head out and say that it is an unnecessary complication that will cause more annoyance and confusion in Python than benefit.
"Undefined behavior" regarding how failed matches may affect surrounding namespace is unlikely something to be proud of.
We should not use the term *undefined behaviour* because that carries too much baggage from C, where undefined behaviour is a truly frightening thing.
In Python, it just means implementation defined. We have lots of implementation defined behaviour:
- caching of small ints and strings;
- whether changes to locals() effects local variables;
- performance of string concatenation;
- many floating point operations, e.g. trig functions, powers;
- presence or absense of many platform-dependent features;
- (in the past) Unicode narrow or wide builds;
- details of precisely when objects will be garbage collected;
- iteration order of sets;
and I expect there are others.
Let us be clear: failed matches do not affect *surrounding* namespaces unless you declare capture variables as global or nonglobal. They *might* affect the current namespace, but not surrounding namespaces. Failed matches will not leak out to surrounding namespaces.
Hello,
On Sun, 29 Nov 2020 11:36:45 +1100 Steven D'Aprano steve@pearwood.info wrote:
On Sun, Nov 29, 2020 at 12:10:39AM +0300, Paul Sokolovsky wrote:
And we don't speak about some obscure "innovative" idea. Const'ness aka immutability is well-known and widely used feature in programming languages.
Constantness and immutability are not synonyms.
I agree. I'm just trying A/B testing to reach wider audience. I prefer the term (and keyword) "const", and the full term "constant variable". But if spelled like that, we immediately get nitpicks like "That doesn't sound well in colloquial English! Constant... variable? That's oxymoron!". These nitpicks miss that it's a term from a programming language domain, not a colloquial phrase. But we'll cover those nitpicks in detail elsewhere (in the python-ideas thread).
Immutability refers to *objects*. "abc" is an immutable object; `[]` is not, it is mutable.
Btw, nowadays "immutable variable" is also a pretty established term, thanks to a particular proglingo from some media company (forgot it's name (lingo's, not company's), something like fungus).
[]
Right, Python doesn't have that syntax. But it has other syntaxes to designate that a variable is defined in a particular scope:
# e is defined in the scope of the following block except Exception as e:
That is incorrect. e is defined in the *current* scope. There is no "scope of the following block".
Exception blocks are a special case, because there is dedicated code to unbind (delete) the "e" name when the block is left. But the name is local to the current scope, the block does not create a new scope.
It seems that I didn't convey my message well enough. What you render is "old way of thinking". The new way of thinking which I promote is: "'except as e' was an attempt to introduce block-level scoping, with truly subpar implementation details (like introducing block scope without real block scope)".
[]
*Could* define a, b in the scope of following block (and not beyond). That would have pros and cons, an obvious pro is that failing-to-match patterns would not litter in surrounding namespace, and con that "pattern matching as expression" usage would be more harder. If anything, use of block scoping could be considered to alleviate the littering problem
The "littering problem" will be a non-problem in practice.
"except as e" already confuses people:
----- e = 2.718281828
try: if random() < 0.1: 1/0 except Exception as e: print("Ha-ha! We're are not afraid of: %s" % e)
# Bugreport: phase-of-moon dependent, my e is gone, gone!!11 print("In e we trust: %f" % e) -----
People already get confused by the "for" behavior, as the parallel thread on python-idea.
With the bright future foretold for the pattern matching, you want to take a bet how many people will get confused by the poltergeist effect failed matches may have on the surrounding namespace? Well, I'd call that "cowboy attitude in programming language design" ;-).
Let's strive for solutions which follow the best practices in the programming language design (const'ness, block-level scoping are 2 good examples),
There isn't even a single definition for block-scoping. C block scoping and Java block scoping are not the same. So which one is best practice?
In theory, the best one is that of lambda calculus ;-). In practice, as you suggest, detail may vary by practical considerations. We'd certainly make it blend well with the rest of Python.
At *best* it is a matter of personal taste whether you like or dislike block scoping, but I'm going to stick my head out and say that it is an unnecessary complication that will cause more annoyance and confusion in Python than benefit.
That's useful opinion to hear, thanks. (But not the only one).
The problem is that we already apply adhoc workarounds which already lead to annoyance and confusion. So you would need to compare amounts of existing annoyance with (predicted!) future annoyance for explicit block-scopes. My bet is that a well-proven technique, applied in a general fashion (but integrated well with the rest of language) would only alleviate levels of annoyance and confusion.
"Undefined behavior" regarding how failed matches may affect surrounding namespace is unlikely something to be proud of.
We should not use the term *undefined behaviour* because that carries too much baggage from C, where undefined behaviour is a truly frightening thing.
Me not useth, PEP634 doth: https://www.python.org/dev/peps/pep-0634/#side-effects-and-undefined-behavio...
In Python, it just means implementation defined.
In C it's the same. And you know the tales where it led it. (I'm personally still not afraid of it, but why purposefully establish it where it's not due?)
[]
Let us be clear: failed matches do not affect *surrounding* namespaces unless you declare capture variables as global or nonglobal. They *might* affect the current namespace, but not surrounding namespaces. Failed matches will not leak out to surrounding namespaces.
The problem is that intuitively (just like with "for"), "case a, b if a != b:" opens a new namespace for "a" and "b". That's why I talk about "surrounding namespace". Then if a particular case matching succeeds, weak of us (myself including) expect "a" and "b" to magically appear outside the "case" too. But if the case didn't match, nope, I don't expect "a" and "b" to appear there, it's not intuitive at all ;-).
-- Steve
[]
Paul Sokolovsky writes:
Well, I'd call that "cowboy attitude in programming language design" ;-).
That was uncalled for, especially since you're selling an idea without an implementation yourself.
We'd certainly make it blend well with the rest of Python.
But how long will that take? People have wanted f-strings since forever (1.5 is as far back as I go, PEP 215 was July 2000). We got them in 3.6 (December 2016), the delay basically due to "blending" issues. And that was for a quite self-contained feature.
This one strikes me as likely to be messy. How does we get consistency if we don't change for? How do block locals interact with global, nonlocal, and locals()? Do we need a block_locals()?
Currently all suites in a multiarmed statement (if, for, while, try) are in the same scope. I suspect that when I'm reading other people's code I'd almost certainly read a let/const var = init_val as suite- local in such contexts, while in my own code I'd probably want to use it as statement-local a lot. Of course that latter is easy to handle by creating a new suite with something like "if 1" (as you suggest elsewhere), but creating unconditional 1-statement suites just to declare block-locals seems excessively inelegant. :-/ But requiring a "block" statement to create a slope in common cases like a one-armed if or for is equally inelegant (and surely more common).
The problem is that intuitively (just like with "for"), "case a, b if a != b:" opens a new namespace for "a" and "b".
I don't find a new namespace in either of those contexts intuitive at all. Nor do I find it unintuitive at all. A language will define rules for scoping, I'll learn them.
Hello,
On Mon, 30 Nov 2020 01:54:53 +0900 "Stephen J. Turnbull" turnbull.stephen.fw@u.tsukuba.ac.jp wrote:
Paul Sokolovsky writes:
Well, I'd call that "cowboy attitude in programming language design" ;-).
That was uncalled for, especially since you're selling an idea without an implementation yourself.
Ok, I posted my "cowboy" implementation (i.e. a prototype) of block-level scoping for the "for" now: https://mail.python.org/archives/list/python-ideas@python.org/thread/MMEIYOK... , to make that even.
Also to clarify, that referred to difference in approaches in response to particular issue(s) raised. One thing is to say "it's hard to implement it better with the limited VM infrastructure and resources we have" (that of course leads to further questions/discussion), it's different thing to say to the same matter "issue exists, but it's not really an issue".
We'd certainly make it blend well with the rest of Python.
But how long will that take? People have wanted f-strings since forever (1.5 is as far back as I go, PEP 215 was July 2000). We got them in 3.6 (December 2016), the delay basically due to "blending" issues. And that was for a quite self-contained feature.
If anything, "Unlikely Adventures of Python in 3.x Era", is what motivates me to continue this discussion. Alternatively, consider what a conceptual jump happened in 2.1, when function-level nested scoping was introduced. Maybe one of these years would be good time, after so many usability changes, to also tighten up theoretical side with block-level scoping.
This one strikes me as likely to be messy. How does we get consistency if we don't change for? How do block locals interact with global, nonlocal, and locals()? Do we need a block_locals()?
The simplest answer is that they don't (interact with existing overdynamic features). They are new enough, opt-in concepts, and so can start from blank page. Alternatively, if debuggers, etc. are tied to locals() (vs more low-level code object introspection), could add implementation-defined support for them in locals() for simple implementations like CPython (block-local names will be mangled).
Currently all suites in a multiarmed statement (if, for, while, try) are in the same scope.
Well, I definitely advocate a solution where a Python suite == block scope, which for example means that "true" vs "false" suites of the "if" are different scopes.
I suspect that when I'm reading other people's code I'd almost certainly read a let/const var = init_val as suite- local in such contexts, while in my own code I'd probably want to use it as statement-local a lot.
It's unclear how block-level scoping can be useful for single-statement scopes (beyond special cases like comprehensions), because well, that single statement will be assignment to such a variable (but nothing will read it). I even suspect we might imagine different things when talk about this, perhaps examples would help.
Of course that latter is easy to handle by creating a new suite with something like "if 1" (as you suggest elsewhere), but creating unconditional 1-statement suites just to declare block-locals seems excessively inelegant. :-/
This is somewhat backwards to the motivation I try to promote. My motivation is:
1. We have adhoc block-scope-alikes already. 2. We have on the plate cases where block scope may help. 3. So, convert to "real" block scoping there. 4. Don't stop there, and *let* people use explicit block-level vars if they need.
It's "let", don't "make" people use it. If you *really* need block-level local, perhaps just being able to use it (without rehashing half of the language) is good enough, for starters.
That said, if you want my bet for the best block-level introduction way, not requiring new keywords, it's:
with: # yeah, empty with! let x = 1
But requiring a "block" statement to create a slope in common cases like a one-armed if or for is equally inelegant (and surely more common).
Arm on an "if" is already a block scope, per above.
The problem is that intuitively (just like with "for"), "case a, b if a != b:" opens a new namespace for "a" and "b".
I don't find a new namespace in either of those contexts intuitive at all. Nor do I find it unintuitive at all. A language will define rules for scoping, I'll learn them.
In "case a, b if ...:", if the case *doesn't* match, do you expect a and b be assigned? If you don't, then you intuitively expect special scoping rules for at least the "case" line itself, and the block scope is the best well-proven formalism in the proglanguage theory/practice which matches that requirement.
Paul Sokolovsky writes:
Also to clarify, [cowboy attitude] referred to difference in approaches in response to particular issue(s) raised. One thing is to say "it's hard to implement it better with the limited VM infrastructure and resources we have" (that of course leads to further questions/discussion), it's different thing to say to the same matter "issue exists, but it's not really an issue".
Of course those are *different*. Where I differ with you is on whether "I understand the case are describing and that you want to do something about it, but we don't consider that a problem" is "cowboy attitude".
My experience is that the development community will care when it perceives that a change serves the greater community (eg, f-strings and data classes, which are universally loved), or when a sufficiently large coalition of committers can be convinced. That's what you need to do, and accusing the core developers of cowboy attitude is not a good start IMO.
Maybe one of these years would be good time, after so many usability changes, to also tighten up theoretical side with block-level scoping.
Maybe it is. I certainly hope not, because I like Python as it is, where I rarely have to deal with declarations separate from instantiation, and code is highly dynamic without a compiler complaining, perhaps even going on strike, any time I want to do something whose semantics I know to be sound even though the compiler can't prove that from the syntax.
Yes, I know you claim that your proposed syntax won't be in my face, and quite possibly you're right with respect to the obstreperous compiler. But I also know that you make claims that are provably false, such as not making me use the syntax. True, short of XKCD 538 you can't make me write it, but if it exists somebody will make me read it.
The simplest answer is that they don't (interact with existing overdynamic features).
You say "overdynamic", I say "works for me".
They are new enough, opt-in concepts, and so can start from blank page.
You're BS-ing here -- you back off in the next sentence (quoted below). And from the community's point of view, no syntax is truly opt-in because we often work with Other People's Code. We need to read it, and often in open source we are modifying code whose style is "owned" by someone else -- if I were to work on code you maintain, I'm pretty sure you wouldn't hesitate to query "shouldn't this variable be block local" or even refuse to accept code that didn't have the block locals you think it should have.
A lot of effort and persuasion (and more or less silent acceptance of the slings and arrows from outraged dynamicists) went into the introduction of type hints. I think this will require more of the same (although maybe not: type hints are intentionally pervasive for those who use them, these declarations will presumably be occasional).
Alternatively, if debuggers, etc. are tied to locals() (vs more low-level code object introspection), could add implementation-defined support for them in locals() for simple implementations like CPython (block-local names will be mangled).
Sure, but we don't have an implementation of that support yet, do we? And adding the syntax will impose burdens on other implementations that claim to support the full language, as well as quite possibly inducing implementation differences here, unless you specify the debugger support etc. My point is that these are the kinds of complications that frequently delay acceptance of features for a release or two.
I suspect that when I'm reading other people's code I'd almost certainly read a let/const var = init_val as suite- local in such contexts, while in my own code I'd probably want to use it as statement-local a lot.
It's unclear how block-level scoping can be useful for single-statement scopes (beyond special cases like comprehensions), because well, that single statement will be assignment to such a variable (but nothing will read it). I even suspect we might imagine different things when talk about this, perhaps examples would help.
By "if statement" I mean
if foo(): pass elif bar(): pass else: pass
That is a single conditional *statement* with three suites that each might constitute an individual block or together constitute a single block (encompassing the if and elif condition expressions as well). I believe that's how the Python Language Reference uses "statement".
- We have adhoc block-scope-alikes already.
Yes, there's the problematic deletion of exception-capture variables, but I don't think the plural is justified. Comprehensions are *expressions*, not suites. Python is a *high level* language: the fact that
[x for x in y]
effectively expands to
list((x for x in y)) ,
implicitly defining a generator function and iterating that, doesn't bother me at all. It's an expression, and there's no reason for its internals to be available to the scope it occurs in, any more than v or x in
def foo(y): v = [] for x in y: v.append(x) return v
should be available in the scope where foo is called. AIUI, the decision to expose x in the comprehension was so that the explanation "the comprehension above is equivalent to the suite in foo (except that no name is bound to the object denoted by 'v')" would be exact, including the leakage of x (but not v) into the containing scope. That's a pretty arbitrary reason, but ease of explanation is valued in Python, as is simplicity of implementation. I think you have to admit that given that generator expressions exist, "list((x for x in y))" is about as simple a semantics and implementation as you could ask for.
- We have on the plate cases where block scope may help.
I'll take your word for that. I don't know that I've ever run into any, and I do know that I've never thought "Damn, I wish I could use block locals here" or "I wish the author of this code had used block locals here" in Python (though I occasionally use block scope in C). Of course *I* don't matter, but the relative balance between folks with your experience and folks with mine does.
- So, convert to "real" block scoping there.
Whoa! First you have to convince the core devs that block scope really does help, and that it's the best way to improve the language's ability to express solutions to these issues. I'll trust you that it *does* address the issues. But whether it's best is entirely unclear to me. Perhaps there are are semantic tweaks like changing the intuitive explanation of comprehension from "a list comprehension is equivalent to an obvious inlined for statement" to "a list comprehension is list applied to the obvious genexp".
You don't like that particular semantic tweak because its implementation involves creating an "unnecessary" function, I know, but I do, because it unifies two similar expressions (actually, at least four), making one a trivial extension of the other.
- Don't stop there, and *let* people use explicit block-level vars if
they need.
It's "let", don't "make" people use it.
Please, that's just factually wrong. If you write it, I have to read it if I want to understand your program. If you don't intend to write code for me to read, why do we need to have the same language? Fork, and use your improved Python-like language for your private code. You'll still be able to read my code, much of which is intended for public use and visibility.
If you *really* need block-level local, perhaps just being able to use it (without rehashing half of the language) is good enough, for starters.
You continue to overstate your case. Nobody *really* *needs* block locals. Like type hints, they may be a great convenience for detecting certain kinds of logic errors, but they don't allow you to do any computation in the language you couldn't already do in the same number of lines, even the same number of tokens.
But requiring a "block" statement to create a slope in common cases like a one-armed if or for is equally inelegant (and surely more common).
But not necessarily equally costly if misunderstood, and the "always need a block statement to create a block" approach is easier to understand and explain correctly. And I disagree with you that mere verbosity is "inelegant". Concise syntax is a virtue, but so is consistent syntax. IMO, elegance lies in achieving both in the same construct. I'm not sure if elegance is possible here, consistent with existing Python syntax.
In "case a, b if ...:", if the case *doesn't* match, do you expect a and b be assigned?
Mu. I can both not expect them to be assigned, and not expect them to not be assigned, at the same time. I can learn either.
[When] you intuitively expect special scoping rules for at least the "case" line itself, [...] the block scope is the best well- proven formalism in the proglanguage theory/practice which matches that requirement.
Sure, but Python is not a Lisp. Adding arbitrary user-defined block scopes is not just a matter of one more pair of parentheses. So "best" has to be constrained by "Pythonicity", and I don't think your computer-scientific proofs can handle Pythonicity yet.
Hello,
On Mon, 30 Nov 2020 14:45:26 +0900 "Stephen J. Turnbull" turnbull.stephen.fw@u.tsukuba.ac.jp wrote:
Paul Sokolovsky writes:
Also to clarify, [cowboy attitude] referred to difference in approaches in response to particular issue(s) raised. One thing is to say "it's hard to implement it better with the limited VM infrastructure and resources we have" (that of course leads to further questions/discussion), it's different thing to say to the same matter "issue exists, but it's not really an issue".
Of course those are *different*. Where I differ with you is on whether "I understand the case are describing and that you want to do something about it, but we don't consider that a problem" is "cowboy attitude".
Right, all people are different. That's great!
My experience is that the development community will care when it perceives that a change serves the greater community (eg, f-strings and data classes, which are universally loved), or when a sufficiently large coalition of committers can be convinced. That's what you need to do, and accusing the core developers of cowboy attitude is not a good start IMO.
Sorry, but there may be a suggestion of tactics of sneaking somebody's "pet feature" past the attention of core developers by employing sycophant techniques. That's certainly not me. Let the criticism begin. Both of "doing it" and "not doing it".
Maybe one of these years would be good time, after so many usability changes, to also tighten up theoretical side with block-level scoping.
Maybe it is. I certainly hope not, because I like Python as it is,
But then you've already lost, because today's CPython (3.9) is not like that of a year ago, and that one is not like that of 2 years ago, and it all has been changing at alarming pace. So, calling for "stop the world" right now seems to be a bit ungrounded, if anything, newer changes proposed take inspiration in the changes already done.
where I rarely have to deal with declarations separate from instantiation, and code is highly dynamic without a compiler complaining, perhaps even going on strike, any time I want to do something whose semantics I know to be sound even though the compiler can't prove that from the syntax.
Good news is that it all stays. But if you want all that, perhaps you appreciate that some other people may want different things too. Like, more speed (JIT), better scalability to complex codebases, tighter formal side (syntax/semantics), etc.
Yes, I know you claim that your proposed syntax won't be in my face, and quite possibly you're right with respect to the obstreperous compiler. But I also know that you make claims that are provably false, such as not making me use the syntax. True, short of XKCD 538 you can't make me write it, but if it exists somebody will make me read it.
Ok, it will be it will be a fun cute read, calling for existing memories (or for youngest of us actually teaching them useful things, which apply outside Python too).
The simplest answer is that they don't (interact with existing overdynamic features).
You say "overdynamic", I say "works for me".
... And some say "slow".
They are new enough, opt-in concepts, and so can start from blank page.
You're BS-ing here -- you back off in the next sentence (quoted below). And from the community's point of view, no syntax is truly opt-in because we often work with Other People's Code. We need to read it, and often in open source we are modifying code whose style is "owned" by someone else -- if I were to work on code you maintain, I'm pretty sure you wouldn't hesitate to query "shouldn't this variable be block local" or even refuse to accept code that didn't have the block locals you think it should have.
But if you anticipate that there may be people who will use block-local vars in Python "when they should have been", don't you think that "saving" them from that by simply not letting them hold of block-scoped vars is a bit ... umm, backward way to tackle the issue?
Beyond that, "it's going to be a fun cute read". If anything, we can judge by experience of others. If you see a "let" in JavaScript code, you can google up "javascript let" and in a minute you will be back to your pleasurable code-reading. And "let" ("still", "so far") doesn't appear that often in JavaScript code. Unlike it's counterpart "const" which has the same block scope, but very obvious semantics, so many people won't even google it on first sight (only later).
Do you think it would be worse than that with us? I don't.
A lot of effort and persuasion (and more or less silent acceptance of the slings and arrows from outraged dynamicists) went into the introduction of type hints. I think this will require more of the same (although maybe not: type hints are intentionally pervasive for those who use them, these declarations will presumably be occasional).
And block scope is initially introduce as internal compilation pipeline feature at all, but with an immediate notice: it's not intended to be hidden forever, many other languages offer it as "public API", so let's be getting used to "measure up" it for Python too.
And strangely, this latter notice is getting more (surface) attention than the initial idea to resolve corner semantic cases for existing or to-be-added constructs.
Alternatively, if debuggers, etc. are tied to locals() (vs more low-level code object introspection), could add implementation-defined support for them in locals() for simple implementations like CPython (block-local names will be mangled).
Sure, but we don't have an implementation of that support yet, do we?
Only idea and a quick prototype, which still allows to get some "feel" of what it may be like.
And adding the syntax will impose burdens on other implementations that claim to support the full language,
Oh, so again, it's a bit too late to get conscientious in that regard, after stuffing f-strings and walrus operator into the language, shaking up dictionaries from ol' good "mapping" computer-science data structure into some adhoc, CPython-specific little monster, etc.
as well as quite possibly inducing implementation differences here, unless you specify the debugger support etc. My point is that these are the kinds of complications that frequently delay acceptance of features for a release or two.
The claim was that block scoping is not a rocket science, but baseline compiler science. The underlying machinery is not complex (it's variable name rewriting, that's all). Surely implementation always has its own side, but overall, this feature should be a smaller (but "deeper") change than many others.
Aspects of debugger support is not part of the language (syntax/semantics), but an implementation detail of a particular implementation. Certainly, if debugger support is important for such one, it will be addressed.
I suspect that when I'm reading other people's code I'd almost certainly read a let/const var = init_val as suite- local in such contexts, while in my own code I'd probably want to use it as statement-local a lot.
It's unclear how block-level scoping can be useful for single-statement scopes (beyond special cases like comprehensions), because well, that single statement will be assignment to such a variable (but nothing will read it). I even suspect we might imagine different things when talk about this, perhaps examples would help.
By "if statement" I mean
if foo(): pass elif bar(): pass else: passThat is a single conditional *statement* with three suites that each might constitute an individual block or together constitute a single block (encompassing the if and elif condition expressions as well).
Right. And more specifically, this "if" statement may be a block with other statements (at the same level), with which it shares block scope.
I believe that's how the Python Language Reference uses "statement".
- We have adhoc block-scope-alikes already.
Yes, there's the problematic deletion of exception-capture variables, but I don't think the plural is justified.
But "adhoc" is not a swear-word. It means that each case was addressed with its own approach, each somewhat (or quite) different. What's proposed instead is a common way to address them, a couple of more cases "on the plate", and hopefully, some of the future challenges too.
Comprehensions are *expressions*, not suites.
Yes. But expression can be used as a statement. And block/suite is defined as "one or more statements". That's how we can apply machinery for "block scope" to expressions.
Python is a *high level* language: the fact that
[x for x in y]effectively expands to
list((x for x in y)) ,implicitly defining a generator function and iterating that, doesn't bother me at all.
That's good, because how it's actually implemented in the code is implementation detail, which should bother people working on the compiler. And block scoping is initially introduced exactly to address hurdles the compiler has with implementing some language features. (Where "hurdles" include "being less optimal than could be".)
It's an expression, and there's no reason for its internals to be available to the scope it occurs in, any more than v or x in
def foo(y): v = [] for x in y: v.append(x) return v
From the point of view of lambda calculus, function abstraction, application, and alpha/beta conversion all go in the same pack, so what you write is definitely right ;-). Implementation-wise however, some constructs are less efficient than other, e.g. functions are almost often heavier-weight than expressions/statements (even in machine code). That's why a baseline compiler optimization is inlining. Here we vice-versa, "outline" a comprehension expression to a function. That can be improved upon.
[]
that given that generator expressions exist, "list((x for x in y))" is about as simple a semantics and implementation as you could ask for.
Absolutely right. Simple implementation. But not the most efficient one. And with block-scoping infrastructure available, avoiding function overhead would be almost as easy, but tad more efficient in the end.
But most importantly, a common infrastructure will be reused to address different cases, and that's the main motivation here.
- We have on the plate cases where block scope may help.
I'll take your word for that.
One case is "undefined behavior" in case of not-matching with pattern matching, another case is peculiarities with "for" loop variable closure-capturing, on python-idea. Again, how block scoping helps there is that the same proven, generic (thus easy(er) to explain) machinery would be used. It's not a "great enabler" or something. Alternative solutions exist. Quite adhoc and/or more complex/less efficient IMHO.
I don't know that I've ever run into any, and I do know that I've never thought "Damn, I wish I could use block locals here" or "I wish the author of this code had used block locals here" in Python
But that's good, isn't it? If anything, that would be a reason to "keep calm and not worrying", isn't it? There's no conspiracy going on to put everyone in an uncomfortable position of realizing that they "did it all wrong all this time", and all their variables should have been "new variables on the block". That's totally not the case.
(though I occasionally use block scope in C).
Oh, so we're on that track already, right? It's just that we got used to think that "Python is special". Because we've been told that so many times. But after peering into it for enough time, it seems that Python is much less of that special snowflake it's drawn as sometimes. Well, that's how I arrived here. So, please hold on to that path.
Of course *I* don't matter, but the relative balance between folks with your experience and folks with mine does.
Yep, that's why we run discussions, trying to show insights we've got to other people, to check if they see something similar in there.
- So, convert to "real" block scoping there.
Whoa! First you have to convince the core devs that block scope really does help, and that it's the best way to improve the language's
Well, but you might be suggesting that I'm a kind of titan on whose shoulders lies the task of conveying some sacred knowledge to "the core devs". If it's like that, "all is lost" already (for a few years, though some things in Python history were actually lost for decades).
First, "my idea" is not mine. Block scoping is again a well-known feature used in many other languages. I just once saw and was amazed by seemingly right-on-the-surface idea to apply it to Python needs, and try to spread gospel of this revelation.
In all fairness, the best outcome could be now is that some of the core devs says something like: "Yeah, I wanted to try that idea several years ago. But then I went on Hawaii vacation for a month (much better spend of time than this stuff), and on return, started renovation in the house, that's how I forgot about it."
Otherwise I fully understand it will be a long thorny path. But somebody should start it.
ability to express solutions to these issues. I'll trust you that it *does* address the issues. But whether it's best is entirely unclear to me. Perhaps there are are semantic tweaks like changing the intuitive explanation of comprehension from "a list comprehension is equivalent to an obvious inlined for statement" to "a list comprehension is list applied to the obvious genexp".
You don't like that particular semantic tweak because its
But I absolutely like it! I just don't sit with blissful smile on my face contemplating the object of my liking, but use it as motivation to rise up and say: "That's nice, but we can do better!" (Some other things I actually don't like, e.g. "except as e" hack).
[]
If you *really* need block-level local, perhaps just being able to use it (without rehashing half of the language) is good enough, for starters.
You continue to overstate your case. Nobody *really* *needs* block
But you overstate your worry about the block scopes, and than apparently I overstate my comforting those worries.
[]
But requiring a "block" statement to create a slope in common cases like a one-armed if or for is equally inelegant (and surely more common).
But not necessarily equally costly if misunderstood, and the "always need a block statement to create a block" approach is easier to understand and explain correctly. And I disagree with you that mere verbosity is "inelegant". Concise syntax is a virtue, but so is consistent syntax. IMO, elegance lies in achieving both in the same construct. I'm not sure if elegance is possible here, consistent with existing Python syntax.
Here you respond to your own quote, and I'm losing track of what arguments apply to. So, no response from me.
[]
[When] you intuitively expect special scoping rules for at least the "case" line itself, [...] the block scope is the best well- proven formalism in the proglanguage theory/practice which matches that requirement.
Sure, but Python is not a Lisp. Adding arbitrary user-defined block scopes is not just a matter of one more pair of parentheses.
So, we want be adding Python's analog of parentheses, which is indented statement suite. Because well, that's not a cheap as a pair of parent in LISP, so *unconditionally requiring* new indented block just for block scope vars is too much. We'll use existing intended blocks. This follows ideas how it's done in most statement-based languages (except that many use braces instead of indentation).
So "best" has to be constrained by "Pythonicity", and I don't think your computer-scientific proofs can handle Pythonicity yet.
You know, "Pythonicity" term lost much of its charm over the very last few years. You started your previous mail with mentioning a case of f-strings, saying that "people waited for string interpolation since Python 1.5". I was there at that time, and remember *quite different* story. I remember regular sabotage spies from the Perl-land trying to seed confusion in the happy Python-island. But those were detected quickly and their harmful influence were humbly but with a firm hand rejected. Because explicit is better than an implicit. Because there's one way to do it. And if you have one way to say explicitly that you format a string with '"%s" % foo', why would there would be a need to hide formatting inside strings themselves and confuse the innocent?
The suddenly, .format()! And f-strings were a fatal blow to the Pythonicity As We Knew It.
With that introduction in mind, the solution I advocate is exactly Pythonic, trust me. It's based on the idea that there's no need to hamper readability and learnability by introducing heavy-weigth indented blocks without good need. Instead, it reuses existing practice of "variable annotations keywords" like "global" and "nonlocal".
In this way, it *differs* from the tradition of LISP and other functional languages, which, due to their expression (vs statement) orientation, indeed usually have a dedicated form for lexical scoping:
(using ML-like syntax)
let var1 = expr1, var2 = expr2 ... in expr
Which is itself an expression, so you can write:
(let x = 10 in x) + 1
We don't need all that (well, we don't let most of that), and thus we don't need to burden ourselves with "let" being a block statement.
Paul Sokolovsky writes:
Sorry, but there may be a suggestion of tactics of sneaking somebody's "pet feature" past the attention of core developers by employing sycophant techniques.
I've had enough of your random aspersions. They make it impossible for me to continue reading your posts. I'll join David with a "moderate" -100 on your proposals, and bid you "good day".
I hope more of the regulars here jump on this bandwagon. It will be a great day when Paul posts one of his offensive posts and there is just deafening silence.
Paul was in my (very short) kill file for years but I decided to give him another chance. And he blew it.
There is a reason why he was banned from micropython, and you all are seeing it here — he just can’t help himself.
Probably he’s hoping to be banned for a COC violation. Let’s not give him that satisfaction.
—Guido (speaking for myself only)
On Tue, Dec 1, 2020 at 15:11 Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
Paul Sokolovsky writes:
Sorry, but there may be a suggestion of tactics of sneaking somebody's "pet feature" past the attention of core developers by employing sycophant techniques.
I've had enough of your random aspersions. They make it impossible for me to continue reading your posts. I'll join David with a "moderate" -100 on your proposals, and bid you "good day". _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/3BAYVC53... Code of Conduct: http://python.org/psf/codeofconduct/
On Sat, Dec 5, 2020 at 1:53 AM Guido van Rossum guido@python.org wrote:
I hope more of the regulars here jump on this bandwagon. It will be a great day when Paul posts one of his offensive posts and there is just deafening silence.
Paul was in my (very short) kill file for years but I decided to give him another chance. And he blew it.
There is a reason why he was banned from micropython, and you all are seeing it here — he just can’t help himself.
Probably he’s hoping to be banned for a COC violation. Let’s not give him that satisfaction.
Agree. I press "Mute" GMail button when seeing offensive words from Paul in conversation. I did it several times for the last months.
—Guido (speaking for myself only)
On Tue, Dec 1, 2020 at 15:11 Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
Paul Sokolovsky writes:
Sorry, but there may be a suggestion of tactics of sneaking somebody's "pet feature" past the attention of core developers by employing sycophant techniques.
I've had enough of your random aspersions. They make it impossible for me to continue reading your posts. I'll join David with a "moderate" -100 on your proposals, and bid you "good day". _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/3BAYVC53... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido (mobile) _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/RADAUJGF... Code of Conduct: http://python.org/psf/codeofconduct/
Hello,
On Fri, 4 Dec 2020 15:52:01 -0800 Guido van Rossum guido@python.org wrote:
I hope more of the regulars here jump on this bandwagon. It will be a great day when Paul posts one of his offensive posts and there is just deafening silence.
And I maintain my hopes where they always were - that there will be discussions of the things Python misses and steps to address them.
Paul was in my (very short) kill file for years
I wear that badge proudly, thanks.
but I decided to give him another chance.
And I appreciate that.
And he blew it.
There is a reason why he was banned from micropython, and you all are seeing it here — he just can’t help himself.
For having a different opinion, and maintaining it.
Probably he’s hoping to be banned for a COC violation. Let’s not give him that satisfaction.
I won't be the first, I won't be the last. Long time ago I thought: what worst could happen? I will leave Python, like thousands before me? And I decided to stay, and even make some noises sometimes. I definitely can't reach levels of "Python3 is not Turing-complete" drama (https://www.google.com/search?q=python+3+is+not+turing+complete), but glad that my humble efforts are being noticed.
The more interesting question is what will you do on the actual technical topics raised? Nothing? Something completely different? The whole story of Python3 migration (the case above) showed that:
1. All the compatibility concerns were never addressed, to punish abusive critics of Python3 (and anyone who was persistent in their critics was abusive).
2. As a particular step, when "u" string prefix for unicode strings was reintroduced, it was made as "z", again, to punish all the critics. Sorry, abusive critics.
So, it's a real intrigue how future technical challenges will be addressed, if at all.
—Guido (speaking for myself only)
Paul,
You seem to be intent on alienating and driving away as many people as possible. If I may play the amateur psychologist for a moment, I fear that you are unconsciously sabotaging your own proposals.
Better to be the genius whose proposals are dragged down by jealous and uncomprehending lessers (such as the PyPy and CPython dev teams) than to risk having to actually deliver on your proposals, and maybe failing, by winning over the decision makers who can approve or disapprove them.
Why else would you repeat such a transparently false untruth as the old canard that the Python core devs "never addressed" concerns about the Python 2 to 3 transition?
You are talking to an audience who, for the most part, took part of that transition here and on other mailing lists, and know that your claim is unadulterated bullshit.
I use that term in the philosophical sense:
https://philosophynow.org/issues/53/On_Bullshit_by_Harry_Frankfurt
For a decade or more the CPython core devs addressed those concerns over and over again, back-porting features to Python 2.6 and 2.7, building a 2to3 tool, writing porting guides, extending the life of 2.7, and re-adding features to Python 3 to make porting easier.
What do you think you had to gain by telling them to their face (in a virtual sense) that they didn't do any of those things?
You even disprove your own claim by mentioning one of those features, the re-introduction of u"" string syntax which has *literally no use* in Python 3 except to increase compatibility with code that has to straddle the 2 to 3 boundary.
But let's put that aside and focus on your technical knowledge:
- As a particular step, when "u" string prefix for unicode strings was
reintroduced, it was made as "z", again, to punish all the critics.
You claim that the u'' prefix was turned into a z'' prefix in Python 3. Have you tried it?
>>> print(u'string') # This doesn't exist, according to Paul string >>> print(z'string') # but this does File "<stdin>", line 1 print(z'string') # but this does ^ SyntaxError: invalid syntax
-
Andrew Svetlov -
David Mertz -
Greg Ewing -
Guido van Rossum -
Jim J. Jewett -
MRAB -
Paul Sokolovsky -
Stephen J. Turnbull -
Steven D'Aprano -
Tobias Kohn