Re: PEP 622: Structural Pattern Matching (version 2)
Hi, As I wrote in an earlier email:
For a PEP to succeed it needs to show two things.
1. Exactly what problem is being solved, or need is to be fulfilled, and that is a sufficiently large problem, or need, to merit the proposed change.
2. That the proposed change is the best known solution for the problem being addressed.
IMO, PEP 622 fails on both counts.
This email addresses point 1 (for version 2). Abstract -------- Why the use of "shape" in "scare quotes"? It worries me that the abstract can't explain what PEP 622 does directly. Could you use a real example in the abstract? Using a contrived example like this seems like a straw man. It feels like it is constructed to favour the PEP, whilst being unlike any real code. Rationale and Goals -------------------
Python programs frequently need to handle data which varies in type, presence of attributes/keys, or number of elements. Typical examples are operating on nodes of a mixed structure like an AST, handling UI events of different types, processing structured input (like structured files or network messages), or “parsing” arguments for a function that can accept different combinations of types and numbers of parameters.
AST, and UI objects usually (pretty much always) form a class hierarchy, making is easy to add utility matching methods to the base class. Where matching *might* have some value is when unrelated types are involved, but you need to show that enhanced destructuring would be insufficient.
In fact, the classic 'visitor' pattern is an example of this, done in an OOP style -- but matching makes it much less tedious to write.
This might be true of the "classic" visitor pattern, but that's a straw man. The Python visitor pattern is a joy to use. Each case gets its own self contained method, clearly named, which is much better than a giant `match` statement.
Much of the code to do so tends to consist of complex chains of nested if/elif statements, including multiple calls to len(), isinstance() and index/key/attribute access. Inside those branches users sometimes need to destructure the data further to extract the required component values, which may be nested several objects deep.
There seem to be three things you want to enhance here: Unpacking; to avoid calls to `len` Type checking; to avoid calls to `isinstance`. To avoiding nesting by using complex lvalues. You fail to justify why the first two cannot be handled separately with much simpler extensions to the language and why complex lvalues are better than nesting. The examples ------------ There are only three examples, none of which are compelling. In fact, two of them serve as a warning against the additional complexity this PEP entails. Django example: ''''''''''''''' Saves two lines of code, but introduces two bugs! (Assuming that the original behavior should be preserved) If the authors of the PEP cannot use this feature correctly in the first example they give, what chance do the rest of us have? is_tuple example: ''''''''''''''''' (Repeating my earlier email) Python's support for OOP provides an alternative to ADTs. For example, by adding a simple "matches" method to Node and Leaf, `is_tuple` can be rewritten as something like: def is_tuple(node): if not isinstance(node, Node): return False return node.matches("(", ")") or node.matches("(", ..., ")") The "switch on http response codes" example. '''''''''''''''''''''''''''''''''''''''''''' This clearly demonstrates the value of symbolic constants for readability and the serious flaw in the PEP that I cannot use them. I really don't see you how you can claim that `case 406:` is more readable than `elif response == HTTP_UPGRADE_REQUIRED:` ? Preventing the use of symbolic constants or other complex rvalues is a impairment to usability. Silently failing when symbolic constants are used is terrible. I do see the value of a `switch` statement for repeated tests against the same value; its intent is clearer than repeated `elif`s. But there is no need for it to be so hard to use. Cheers, Mark.
Hi Mark, Thank you for your message. I might be able to answer some of the questions and also address some issues with the underlying assumptions in your email---after all, we would most certainly want to avoid discussing and reasoning about straw men, as you yourself have repeatedly pointed out.
Why the use of "shape" in "scare quotes"?
Because the `shape' of an object is not something that is a well-defined term, but rather addresses the intuitive understanding to explain what we are talking about. It is usually the intention of the abstract to give a rough overview, before delving into the details and more formal descriptions.
Using a contrived example like this seems like a straw man. It feels like it is constructed to favour the PEP, whilst being unlike any real code.
It is a trait of many (good) tutorials to present a new structure with an example that highlights its feature in an approachable way rather than coming up with `real code'---particularly given that real code is always surrounded by a larger context, which rather clouds any understanding. And it would only be a straw man had we constructed a contrived example of unrealistic code that we then showed to be simplified by pattern matching. That's not the case: it really is just a simple educational example to present the idea. BTW: as a matter of course, it is constructed in favour of the PEP!
There seem to be three things you want to enhance here: Unpacking; to avoid calls to `len` Type checking; to avoid calls to `isinstance`. To avoiding nesting by using complex lvalues.
Actually, no. Pattern matching is not about _avoiding_ anything, it is about a more concise syntax. The suggestion that we wanted to avoid calls to `len` or `isinstance` is similar to the idea of using a lambda to avoid a function. The objective of pattern matching is to introduce a more succinct and readable way to express structure. It's all still there but we want to focus on the structure rather than `isinstance` calls. Pattern matching already exists in some limited way in Python. Even today you can write: `a, b = value` instead of: `a = value[0] b = value[1]` The idea of this is, of course, not so much about avoiding item access, but about having a more concise syntax for it. Moreover, saying this only saves a single line and is therefore rather useless is quite beside the point. It is about a better representation of what the code is supposed to do and not about saving lines.
Saves two lines of code, but introduces two bugs! (Assuming that the original behavior should be preserved)
Thank you for pointing out the bug in our example. This highlights, however, rather the difficulty of refactoring (did I already mention the issue of the 'context' that real code is embedded in?), than any shortcoming of pattern matching as a tool. And by the way: constructive critisism would perhaps include explicitly naming the two bugs.
For example, by adding a simple "matches" method to Node and Leaf, `is_tuple` can be rewritten as something like:
This assumes that you have full control over the entire code. But if you are using some third-party library, you cannot "simply add a `matches` method" (without some substantial trickery). Moreover, there is quite some work needed to define such a `matches` function as you propose it (including support for the ellipsis as wildcard). In effect, you would have to implement large parts of the pattern matching for just this simple example. Whereas we believe that it has merit enough to have the compiler do it for a wide range of possible use cases.
I really don't see you how you can claim that `case 406:` is more readable than `elif response == HTTP_UPGRADE_REQUIRED:` ? Preventing the use of symbolic constants or other complex rvalues is a impairment to usability.
First, let me quote what the PEP has to say on that exact example:
Although this will work, it's not necessarily what the proposal is focused on.
Moreover, it is usually a good idea to put constants into a separate namespace that further describes their meaning and intended use. And that is fully supported by the syntax as proposed in the PEP. ``` match response.status: case HTTP_RESPONSE.UPGRADE_REQUIRED: ... ```
For a PEP to succeed it needs to show two things.
1. Exactly what problem is being solved, or need is to be fulfilled, and that is a sufficiently large problem, or need, to merit the proposed change.
2. That the proposed change is the best known solution for the problem being addressed.
IMO, PEP 622 fails on both counts.
This seems fair enough as far as the two issues are concerned. It might seem indeed as if pattern matching does not add anything that could not be done already in Python. Part of the problem is that pattern matching starts to really shine when the objects and data get more complex---which requires a lot of built-up and context just to get to the point. This is similar to OOP: as long as you stick with the easy introductory examples, OOP does not really provide anything that you could not solve using procedural programming. For instance, I often see critisism of Java based on some simple hello-world examples, which completely misses the point of OOP in the first place, of course. As far as referring to the best solution: may I humbly point out that pattern matching has been around for some 40 years now and has been adopted by an increasing number of languages lately. We put a lot of effort into designing a version of it that fits well with Python and covered a huge design space, while also taking all the lessons of those who came before us to heart. This is not just a new feature we propose on a whim. In the end, pattern matching is an increasingly popular feature that favours higher-level declarative expressiveness over spelling out all the details of how to check the structure of data and extract information from it. Think of it perhaps this way: pattern matching is like introducing grammars and parser generators instead of writing a new parser each time. It answers the question of: could we have something like regular expressions for graph-like objects instead of only text...? Kind regards, Tobias
Hi Tobias, In future, could you avoid editing emails when replying to them? A lot of context can get lost. On 14/07/2020 5:25 pm, Tobias Kohn wrote:
Hi Mark,
Thank you for your message. I might be able to answer some of the questions and also address some issues with the underlying assumptions in your email---after all, we would most certainly want to avoid discussing and reasoning about straw men, as you yourself have repeatedly pointed out.
Why the use of "shape" in "scare quotes"?
Because the `shape' of an object is not something that is a well-defined term, but rather addresses the intuitive understanding to explain what we are talking about. It is usually the intention of the abstract to give a rough overview, before delving into the details and more formal descriptions.
You ignore my point about explaining PEP 622 directly. Sure, the abstract must give a high level overview, but it should be able to do that without so much hand waving.
Using a contrived example like this seems like a straw man. It feels like it is constructed to favour the PEP, whilst being unlike any real code.
It is a trait of many (good) tutorials to present a new structure with an example that highlights its feature in an approachable way rather than coming up with `real code'---particularly given that real code is always surrounded by a larger context, which rather clouds any understanding. And it would only be a straw man had we constructed a contrived example of unrealistic code that we then showed to be simplified by pattern matching. That's not the case: it really is just a simple educational example to present the idea.
The example given in the PEP does seems like a "contrived example of unrealistic code", as you put it. Is there real code like this?
BTW: as a matter of course, it is constructed in favour of the PEP!
There seem to be three things you want to enhance here: Unpacking; to avoid calls to `len` Type checking; to avoid calls to `isinstance`. To avoiding nesting by using complex lvalues.
Actually, no. Pattern matching is not about _avoiding_ anything, it is about a more concise syntax. The suggestion that we wanted to avoid calls to `len` or `isinstance` is similar to the idea of using a lambda to avoid a function. The objective of pattern matching is to introduce a more succinct and readable way to express structure. It's all still there but we want to focus on the structure rather than `isinstance` calls.
I should have been clearer. Avoiding *explicit* calls to `len` and `isinstance`. Yes, the pattern matching syntax is more concise in some circumstances, but there still needs to be justification why a combination of simpler language changes is insufficient.
Pattern matching already exists in some limited way in Python. Even today you can write: `a, b = value`
I think most of us are aware of unpacking in Python ;)
instead of: `a = value[0] b = value[1]`
The idea of this is, of course, not so much about avoiding item access, but about having a more concise syntax for it. Moreover, saying this only saves a single line and is therefore rather useless is quite beside the point. It is about a better representation of what the code is supposed to do and not about saving lines.
How do you quantify "better representation" then?
Saves two lines of code, but introduces two bugs! (Assuming that the original behavior should be preserved)
Thank you for pointing out the bug in our example. This highlights, however, rather the difficulty of refactoring (did I already mention the issue of the 'context' that real code is embedded in?), than any shortcoming of pattern matching as a tool. And by the way: constructive critisism would perhaps include explicitly naming the two bugs.
I'm not sure it would be more constructive, this is not a bug report or code review after all. The point I am making is this: It is far too easy to make mistakes using the constructs proposed in PEP 622.
For example, by adding a simple "matches" method to Node and Leaf, `is_tuple` can be rewritten as something like:
This assumes that you have full control over the entire code. But if you are using some third-party library, you cannot "simply add a `matches` method" (without some substantial trickery). Moreover, there is quite some work needed to define such a `matches` function as you propose it (including support for the ellipsis as wildcard). In effect, you would have to implement large parts of the pattern matching for just this simple example. Whereas we believe that it has merit enough to have the compiler do it for a wide range of possible use cases.
You wouldn't have to implement large parts of the pattern matching, because this would be specific to one class. There are only two or three cases to deal with. If you don't have access to the original source, then it can be made a function, not a method.
I really don't see you how you can claim that `case 406:` is more readable than `elif response == HTTP_UPGRADE_REQUIRED:` ? Preventing the use of symbolic constants or other complex rvalues is a impairment to usability.
First, let me quote what the PEP has to say on that exact example:
Although this will work, it's not necessarily what the proposal is focused on.
So why does the PEP include it as one of just three examples?
Moreover, it is usually a good idea to put constants into a separate namespace that further describes their meaning and intended use. And that is fully supported by the syntax as proposed in the PEP. ``` match response.status: case HTTP_RESPONSE.UPGRADE_REQUIRED: ... ```
Are you suggesting that all constants live in a separate module? That no module can contain both constants and code?
For a PEP to succeed it needs to show two things.
1. Exactly what problem is being solved, or need is to be fulfilled, and that is a sufficiently large problem, or need, to merit the proposed change.
2. That the proposed change is the best known solution for the problem being addressed.
IMO, PEP 622 fails on both counts.
This seems fair enough as far as the two issues are concerned. It might seem indeed as if pattern matching does not add anything that could not be done already in Python. Part of the problem is that pattern matching starts to really shine when the objects and data get more complex---which requires a lot of built-up and context just to get to the point. This is similar to OOP: as long as you stick with the easy introductory examples, OOP does not really provide anything that you could not solve using procedural programming. For instance, I often see critisism of Java based on some simple hello-world examples, which completely misses the point of OOP in the first place, of course.
There are legitimate reasons to criticize OOP, especially Java. Like when it becomes an ideal, not a tool, and we drown in AbstractObjectFactoryManagers. I worry that the PEP is treating pattern matching as an ideal which we should be striving towards. That is a bad thing, IMO. The PEP needs to convince us that it is a useful tool. Providing compelling real world examples is necessary.
As far as referring to the best solution: may I humbly point out that pattern matching has been around for some 40 years now and has been adopted by an increasing number of languages lately. We put a lot of effort into designing a version of it that fits well with Python and covered a huge design space, while also taking all the lessons of those who came before us to heart. This is not just a new feature we propose on a whim.
I think we can all name program language features that have been around for 40 years and that we definitely don't want in Python :) Age is not always an indicator of quality. Pattern matching is well suited to statically typed functional languages. Python is, at its heart, a dynamically typed procedural language. (By procedural, I mean step-by-step as opposed to functional, nothing to do with whether it is OO or not).
In the end, pattern matching is an increasingly popular feature that favours higher-level declarative expressiveness over spelling out all the details of how to check the structure of data and extract information from it.
Higher-level more declarative programming is great, but there are many ways to do it. List comprehensions, dataclasses, and f-strings are examples. Each of those adds a clear benefit without being overly complex. It was also clear what code patterns they streamlined. PEP 622 fails to make clear which code patterns it streamlines. It definitely feels overly complex.
Think of it perhaps this way: pattern matching is like introducing grammars and parser generators instead of writing a new parser each time. It answers the question of: could we have something like regular expressions for graph-like objects instead of only text...?
Objects are about encapsulation. They may be graphs internally but that should be hidden. Objects should be able to provide an interface without exposing their internals, which means you can't efficiently pattern match on them. Pattern matching is for Algebraic Data Types, not Objects. Cheers, Mark.
Kind regards, Tobias
On 07/14/2020 11:05 AM, Mark Shannon wrote:
On 14/07/2020 5:25 pm, Tobias Kohn wrote:
On 07/14/2020, Mark Shannon wrote:
In future, could you avoid editing emails when replying to them? A lot of context can get lost.
I appreciate posters who take the time to trim the parts of an email that they are not replying to.
Why the use of "shape" in "scare quotes"?
Because the `shape' of an object is not something that is a well-defined term, but rather addresses the intuitive understanding to explain what we are talking about. It is usually the intention of the abstract to give a rough overview, before delving into the details and more formal descriptions.
You ignore my point about explaining PEP 622 directly. Sure, the abstract must give a high level overview, but it should be able to do that without so much hand waving.
I find the PEP abstract concrete enough. Perhaps you could write a replacement that you think does a better job?
Actually, no. Pattern matching is not about _avoiding_ anything, it is about a more concise syntax. The suggestion that we wanted to avoid calls to `len` or `isinstance` is similar to the idea of using a lambda to avoid a function. The objective of pattern matching is to introduce a more succinct and readable way to express structure. It's all still there but we want to focus on the structure rather than `isinstance` calls.
I should have been clearer. Avoiding *explicit* calls to `len` and `isinstance`. Yes, the pattern matching syntax is more concise in some circumstances, but there still needs to be justification why a combination of simpler language changes is insufficient.
My apologies if I missed it, but do you have some ideas of these simpler language changes that would do the job?
Pattern matching already exists in some limited way in Python. Even today you can write: `a, b = value`
I think most of us are aware of unpacking in Python ;)
Yes, but not everyone recognizes that unpacking is a form of pattern matching.
instead of: `a = value[0] b = value[1]`
The idea of this is, of course, not so much about avoiding item access, but about having a more concise syntax for it. Moreover, saying this only saves a single line and is therefore rather useless is quite beside the point. It is about a better representation of what the code is supposed to do and not about saving lines.
How do you quantify "better representation" then?
You mean, without large study groups and lots of money and/or time? Not easily. I can tell you that a, b = value is much more informative, and easier to read and process, than the alternative of using item assignment.
Saves two lines of code, but introduces two bugs! (Assuming that the original behavior should be preserved)
Thank you for pointing out the bug in our example. This highlights, however, rather the difficulty of refactoring (did I already mention the issue of the 'context' that real code is embedded in?), than any shortcoming of pattern matching as a tool. And by the way: constructive critisism would perhaps include explicitly naming the two bugs.
I'm not sure it would be more constructive, this is not a bug report or code review after all.
It should be. If the PEP is to be decided on it should be as correct as possible.
The point I am making is this: It is far too easy to make mistakes using the constructs proposed in PEP 622.
You're point is false. They weren't crafting new code, they were refactoring.
For example, by adding a simple "matches" method to Node and Leaf, `is_tuple` can be rewritten as something like:
This assumes that you have full control over the entire code. But if you are using some third-party library, you cannot "simply add a `matches` method" (without some substantial trickery). Moreover, there is quite some work needed to define such a `matches` function as you propose it (including support for the ellipsis as wildcard). In effect, you would have to implement large parts of the pattern matching for just this simple example. Whereas we believe that it has merit enough to have the compiler do it for a wide range of possible use cases.
You wouldn't have to implement large parts of the pattern matching, because this would be specific to one class. There are only two or three cases to deal with. If you don't have access to the original source, then it can be made a function, not a method.
So, a few lines for that example, another few for a different example, some more for that example over there -- how many times do we have to write similar code before we let Python do the grunt work for us?
Preventing the use of symbolic constants or other complex rvalues is a impairment to usability.
First, let me quote what the PEP has to say on that exact example:
Although this will work, it's not necessarily what the proposal is focused on.
So why does the PEP include it as one of just three examples?
To show that it is possible?
The PEP needs to convince us that it is a useful tool.
Or we can look in our own code and see where it would be useful. I was able to find a location within only moments of searching, and I even managed to get the refactoring correct.
Providing compelling real world examples is necessary.
I found the examples provided very compelling.
Pattern matching is well suited to statically typed functional languages. Python is, at its heart, a dynamically typed procedural language.
And this PEP will finally give us pattern matching for our dynamic language.
Higher-level more declarative programming is great, but there are many ways to do it. List comprehensions, dataclasses, and f-strings are examples. Each of those adds a clear benefit without being overly complex. It was also clear what code patterns they streamlined.
PEP 622 fails to make clear which code patterns it streamlines.
I found it very clear: - lots of conditions to see if you have the object you think you have? Use Pattern Matching.
Think of it perhaps this way: pattern matching is like introducing grammars and parser generators instead of writing a new parser each time. It answers the question of: could we have something like regular expressions for graph-like objects instead of only text...?
Objects are about encapsulation. They may be graphs internally but that should be hidden. Objects should be able to provide an interface without exposing their internals, which means you can't efficiently pattern match on them. Pattern matching is for Algebraic Data Types, not Objects.
Pattern matching is not enabling us to check for anything we weren't already checking for, it's just making it easier to see what is being checked for. And since everything in Python is an object, and a working implementation of pattern matching for Python now exists, I would say pattern matching is definitely for objects. -- ~Ethan~
Mark Shannon wrote:
In future, could you avoid editing emails when replying to them? A lot of context can get lost.
I'll add another voice to Ethan's saying that I appreciate having as much as possible trimmed. As long as people are arguing in good faith (and I assume that they are here), the loss of context is usually pretty small, and outweighed by being able to find the newly important part more quickly. (I say this as someone reading and replying through the archives, in a way that seems to mangle quoted portions -- but they are usually still good enough to be useful.)
... Yes, the pattern matching syntax is more concise in some circumstances, but there still needs to be justification why a combination of simpler language changes is insufficient.
Code that fits on a single small (24x80) screen (without getting too dense) is much easier to understand, because I can see it all at once, instead of needing to remember things while I flip back and forth. A combination of simpler changes might be fine, but shorter really is very valuable, all by itself.
Saves two lines of code, but introduces two bugs! (Assuming that the original behavior should be preserved)
That is a pretty big assumption. I'm guessing that at least one of the "bugs" is that sequences other than the built-in list and tuple are now also accepted. Maybe there are good reasons to exclude other sequences, but ... in my experience, the reason is usually that someone didn't think of it, and no one felt strongly enough to fix it yet. That would mean pattern matching led to a bug *fix*. Closed source code tends to be even more fragile, though it also tends to not see as many surprising input types in the first place.
If you don't have access to the original source, then it can be made a function, not a method.
Even when I have read access, I may not have write access. Creating and naming a separate match function every place I need to match isn't quite boilerplate, but it adds enough code to feel that way. Trying to use a single match function with parameters gets ugly in a different way. I'm not quite convinced that this PEP has found the magic solution, but the goal is clearly worthy.
On 14/07/2020 5:25 pm, Tobias Kohn wrote:
match response.status: case HTTP_RESPONSE.UPGRADE_REQUIRED:
Are you suggesting that all constants live in a separate module?
The limit to dotted names is a possibly temporary wart. But even with that restriction, there is no reason you can't gather constants on an object first, and use its attributes.
For a PEP to succeed it needs to show ... That the proposed change is the best known solution for the problem being addressed.
I think the bigger barrier is "although never is often better than right now," and hope that a better solution will be found later, and fear that backwards compatibility with this would block that better solution. (To be very explicit, I personally abstain on this, because I am not sure whether this is "good enough", nor am I confident a better solution can ever be found.)
I worry that the PEP is treating pattern matching as an ideal which we should be striving towards. That is a bad thing, IMO.
Fair. Like annotations as a typing system, you can personally ignore it if it isn't helpful, but there is still some ecosystem cost. I would like to see more old/new comparisons to judge how intrusive this will be, but eventually there will be (or not be) a leap of faith. In the past, most of those have worked out.
Pattern matching is well suited to statically typed functional languages.
and statically typed data domains and communications protocols -jJ
participants (4)
-
Ethan Furman -
Jim J. Jewett -
Mark Shannon -
Tobias Kohn