PEP 653: Precise Semantics for Pattern Matching
Hi everyone, I'd like to announce a new PEP. https://www.python.org/dev/peps/pep-0653/ It builds on PEP 634 (Structural Pattern Matching: Specification), adding: More precise semantics. A bit more customization for complex classes. Its also a bit more robust and should be faster (eventually). The syntax is unchanged, and in most cases the semantics are the same. As always, comments and suggestions are welcome. Cheers, Mark.
Thanks for taking the time to work on this, Mark. Overall, I'm skeptical of this proposal. It seems like it takes a lot of "simple" things and makes them quite complex, and takes many "static" things and makes them quite dynamic. I feel that it also misrepresents certain aspects of PEP 634. Here's a more detailed commentary:
Pattern matching will be more usable for complex classes, by allowing classes more control over which patterns they match.
I fear this is at the expense of most simple classes, which currently "just work" with PEP 634. I'm not convinced that making it easier for very complicated classes (like those in `sympy`) to participate at the expense of everyday classes is a win. For comparison, here is what (as I understand it) each PEP requires for a class to be used in a pattern such as `C(x, y, z)`: ```py class C: """A PEP 634 matchable class.""" __match_args__ = ... class C: """A PEP 653 matchable class.""" __match_kind__ = MATCH_CLASS __attributes__ = ... def __deconstruct__(self): ... ``` For a pattern with no positional subpatterns, such as like `C()` or `C(x=x, y=y, z=z)`: ```py class C: """A PEP 634 matchable class (all classes work).""" class C: """A PEP 653 matchable class.""" __match_kind__ = MATCH_CLASS __attributes__ = ... def __deconstruct__(self): ... ``` It also appears that we lose a lot of expressive "idioms" by requiring `__attributes__` to be complete. For example, the following attribute extractions work with PEP 634, but not PEP 653. ```py match subject: case object(a=_): # Match any object with an "a" attribute. ... case object(x=x, y=y, z=z): # Match any object with "x", "y", and "z" attributes, extracting them. ... ``` This also means that matching classes like `types.SimpleNamespace` are much less powerful under PEP 653, since the class must know which attributes are "allowed" to be looked up. Further, this whole specification shifts most of the matching logic from the class in the pattern to the subject itself: ```py match ChildA(), ChildB(): case Parent(a, b), Parent(x, y): ... ``` The above pattern could have completely different rules for extracting `a` and `b` vs `x` and `y` if either child overrides `__deconstruct__`. I think that is a mistake.
PEP 634 also privileges some builtin classes with a special form of matching, the "self" match. For example the pattern `list(x)` matches a list and assigns the list to `x`. By allowing classes to choose which kinds of pattern they match, other classes can use this form as well.
This is already fairly trivial to implement: ```py class C: __match_args__ = ("_self",) _self = property(lambda s: s) ``` You can even avoid adding a `_self` attribute if you do some magic with descriptors... ;). We could consider provide a decorator for this somewhere in the stdlib if there's demonstrated need (but I'm not sure there will be).
Oof. I was scratching my head for way too long at this before I noticed the previous sentence said "false". Maybe reword this section to indicate the conditions that hold *true*? :)
Security Implications Preventing the possible registering or unregistering of classes as sequences or a mappings, that PEP 634 allows, should improve security. However, the advantage is slight and is not a motivation for this PEP.
I would say the advantage is nonexistent. How is this any different than tweaking the flags in a class's `__match_kind__`?
Efficient Implementation / Implementation
I won't get too deep into this section, but on the surface most flavors of implementation/optimization present here are also possible with PEP 634. I understand that you feel there are benefits to having "rules" for what optimizations are legal, but we don't need to completely change the mechanics of the match statement in order to do so.
Because the object model is a core part of Python, implementations already handle special attribute lookup efficiently. Looking up a special attribute is much faster than performing a subclass test on an abstract base class.
But calling a `__deconstruct__` method whenever positional arguments are present will slow down normal class matches, right? I see it as mostly a wash.
In general, it seems to me that this proposal prioritizes: - complicated library code - ease of implementation in the interpreter over: - simple client code - general usability I can't say I support that, generally. When working on PEP 634, we tossed lot of more complicated, powerful protocols (like `__match__` and several others before it... don't mention the phrase "match proxy" around any of the authors!) in favor of simplicity. This sort of feels like a rehash of that. Brandt
Brandt Bucher wrote:
For a pattern with no positional subpatterns, such as like `C()` or `C(x=x, y=y, z=z)`: ...
It also appears that we lose a lot of expressive "idioms" by requiring `__attributes__` to be complete.
This also means that matching classes like `types.SimpleNamespace` are much less powerful under PEP 653, since the class must know which attributes are "allowed" to be looked up.
Never mind these three points... I *think* setting `__match_kind__ = MATCH_DEFAULT` allows for arbitrary attribute extraction like this. Perhaps make it bit clearer?
On Thu, 18 Feb 2021 at 17:35, Brandt Bucher <brandtbucher@gmail.com> wrote:
Thanks for taking the time to work on this, Mark.
Yes, thanks Mark. I'm not sure I've fully understood the PEP yet but I can see some parts that I definitely like.
I fear this is at the expense of most simple classes, which currently "just work" with PEP 634. I'm not convinced that making it easier for very complicated classes (like those in `sympy`) to participate at the expense of everyday classes is a win.
While those sympy classes are quite complicated, the relevant aspect of them is not. The issue is just that they have positional only constructor arguments. PEP 634 does not allow for positional only arguments in a class pattern in a way that can work in general. This is because it requires attributes to exist for classes that would otherwise have no need for them.
You are presuming here that the strings in __match_args__ already correspond to attributes. Otherwise the PEP 634 version above requires a property to be added corresponding to each positional argument in the constructor: class C: """A PEP 634 matchable class""" __match_args__ = ['a', 'b', 'c'] a = property(lambda self: self._rep[0]) b = property(lambda self: self._rep[1]) c = property(lambda self: self._rep[2]) At runtime case/match looks into __match_args__ and calls getattr evaluating each property one by one (even if they all derive from the same internal data structure).
PEP 653 says (although it wasn't immediately obvious to me from reading it): """ Classes which define __match_kind__ & MATCH_CLASS to be non-zero must implement one additional special attribute, and one special method: """ Then much later: """ object.__match_kind__ will be MATCH_DEFAULT. """ I think that means that a simple `class C: pass` will match keyword patterns with attributes so C(x=1) will match against any C with c.x == 1. To match positional arguments both __match_kind__, __attributes__ and __deconstruct__ would need to be defined but the end result works better for positional arguments than PEP 634 does: - It is possible to match the number of positional arguments precisely so a pattern C(x) would not match an object C(x, y) - Any invertible mapping between positional arguments and internal class state can be accommodated. I don't think it's possible to achieve those with PEP 634. A concrete example would be matching against a range class: class range: __match_args__ = ['start', 'stop', 'step'] def __init__(self, start_or_stop, stop=None, step=1): if stop is None: start = 0 stop = start_or_stop else: start = start_or_stop self.start = start self.stop = stop self.step = step With PEP 634 a pattern like range(10) will also match objects like range(10, 20) and range(10, 20, 2) etc. I think that is not what someone familiar with range would expect. With PEP 634 there is no way to define __match_args__ so that positional argument patterns with range match in a way that corresponds to its constructor syntax (or that have useful matching semantics in this case). I'm not entirely sure but I think that with PEP 653 you can implement this like: def __deconstruct__(obj): if obj.step != 1: return obj.start, obj.stop, obj.step elif obj.start != 0: return obj.start, obj.stop else: return obj.stop I think that would then mean you could use a pattern range(10) to unambiguously match range(10) without matching range(10, 20) etc. Above I suggested that any *invertible* mapping of args to state could be supported and here there is an example of a non-invertible mapping since range(0, 10) is the same as range(10). I might have misunderstood but I don't think that PEP 653 makes it possible to have a pattern range(10) and a pattern range(0, 10) both match against the same object range(10). This means that the pattern would always need to use the "canonical" form of the args (e.g, range(10) in this case). -- Oscar
Hi Oscar, Quoting Oscar Benjamin <oscar.j.benjamin@gmail.com>:
Yes, thanks Mark. I'm not sure I've fully understood the PEP yet but I can see some parts that I definitely like. [...]
As I have just noted in my response to Mark, the aspect with the "deconstructor" (or `__match__` protocol as we called it) is definitely something that I like, too. Moreover, I think that packages like sympy might make a strong argument for it. That being said, perhaps we would have to start thinking about concrete use cases first and then consider how to best provide for those with an extended match protocol. Even though I still like the idea of a flexible match method, there might be aspects that I/we overlook without knowing exactly where we want to go with this.
This is certainly more of an anti-pattern (no pun intended) than an argument for the match protocol. Shuffling positional arguments around like this seems like a rather bad idea, particularly in the context of pattern matching. I'd rather write `case range(_, stop, _)` or `case range(0, stop, 1)` and be explicit here. Kind regards, Tobias
On Fri, 19 Feb 2021 at 16:27, Tobias Kohn <kohnt@tobiaskohn.ch> wrote:
Good point! I was trying to reason out the mechanism but the alternative approach as you say would be to always use all 3 arguments. PEP 653 here actually makes it possible for the range class to affect matching so that it can only succeed against a pattern that uses all 3 arguments. The significance of range as an example is basically that it has a varying number of positional only arguments to its constructor. The most important feature of the example is not the reordering of arguments: it is the fact that with PEP 634 a pattern range(x) would have to match range(x, y) which is very unlikely to be useful in the case of range but also not generally applicable for other types either. There is no way for range (or other types) to prevent that while supporting positional argument matching through __match_args__. Under PEP 634 in general, for any class C, a pattern C(x) matches an object C(x, y) and there's no way for C to override that. To me that is sufficiently unintuitive in the abstract that no example is really needed to see where there is room for improvement. -- Oscar
Oscar Benjamin wrote:
Under PEP 634 in general, for any class C, a pattern C(x) matches an object C(x, y) and there's no way for C to override that. To me that is sufficiently unintuitive in the abstract that no example is really needed to see where there is room for improvement.
We originally kicked around (and at one point even implemented) a `__match_args_required__` attribute, which is an integer specifying a minimum required number of positional sub-patterns. For reasons I can't recall, though, it was eventually dropped. It would take care of this, and I imagine it could be quite painless to add it back if there was enough support. Here's a query for related discussions: https://github.com/gvanrossum/patma/issues?q=__match_args_required__
Hi Brandt, On 18/02/2021 5:32 pm, Brandt Bucher wrote:
In my experience, nothing "just works". It means "mostly works most of the time, and occasionally goes horribly wrong"
True, but most pure-Python classes don't have a natural order of attributes, so the pattern is likely to be `C(x=x,y=y,z=z)`. In which case no extra work is required for either PEP to pattern match.
This is not true. MATCH_DEFAULT does what you want.
You need to define what you mean by "work" :) This behaves the same for both PEPs: class C: __init__(self, x, y, z): ... match C(1,2,3): case object(x=x, y=y, z=z): ... This also works the same for both PEPs, albeit with different mechanisms: match namedtuple("n", "x, y, z")(1,2,3): case object(x=x, y=y, z=z): ... But, with PEP 634:
Which seem like "not working" to me.
This also means that matching classes like `types.SimpleNamespace` are much less powerful under PEP 653, since the class must know which attributes are "allowed" to be looked up.
Why do you say this? This should work the same for PEP 634 and 653:
Inheritance allows sub-classes to change the behavior of their super-classes. Yes, it can produce weird effects if they ignore Liskov's substitution principle. That why the modern advice is "Favour composition over inheritance". You can't prevent people writing bad code.
That's pretty obscure, IMO. You can't claim that it is trivial compared to: class C: __match_kind__ = MATCH_SELF
Good point. I'll do that.
Then please publish a precise semantics for PEP 634 pattern matching. You say that "most flavors of implementation/optimization present here are also possible with PEP 634", but without semantics you can't show that optimizations preserve the semantics.
Because the object model is a core part of Python, implementations already handle special attribute lookup efficiently. Looking up a special attribute is much faster than performing a subclass test on an abstract base class.
But calling a `__deconstruct__` method whenever positional arguments are present will slow down normal class matches, right? I see it as mostly a wash.
Why do you think it will slow it down? `__deconstruct__` is called only once. PEP 634, as written, has to perform a sequence of attribute lookups for *each* class pattern. If a class defines `__deconstruct__`, it is likely to be for a reason. For example, namedtuple's `__deconstruct__` would be: def __deconstruct__(self): return self Which is both simple and efficient.
It what way does this PEP make client code more complex, or impair usability? Evidence please. "Ease of implementation in the interpreter" improves reliability and performance. Those are important things in a language runtime :)
I can't say I support that, generally. When working on PEP 634, we tossed lot of more complicated, powerful protocols (like `__match__` and several others before it... don't mention the phrase "match proxy" around any of the authors!) in favor of simplicity. This sort of feels like a rehash of that.
If you had published these "more complicated, powerful protocols", you might be able to claim that this is a "rehash". But you didn't. Being vague is not "simplicity", it is merely hiding the complexity. Cheers, Mark.
Hi Mark, Quoting Mark Shannon <mark@hotpy.org>:
I would say that these ideas have been quite prominently published: https://www.python.org/dev/peps/pep-0622/#custom-matching-protocol https://dl.acm.org/doi/10.1145/3426422.3426983 Moreover, much of the discussion is publicly available, for instance here: https://github.com/gvanrossum/patma/issues/8 Cheers, Tobias
Hi Tobias, On 19/02/2021 5:57 pm, Tobias Kohn wrote:
But they are not referenced in PEP 634. I shouldn't have to trawl the internet to find the rejected ideas section.
That paper describes a `__match__` method, which is absent from PEP 634. Why? Cheers, Mark.
Hi Mark, Quoting Mark Shannon <mark@hotpy.org>:
You are right: you should not have to trawl the internet to get these information. However, given the history of the Pattern Matching PEPs and that PEP 622 is linked directly from PEP 634, I would think it is not that far a journey—even though there have been so many discussions that it has definitely become unwieldy to retain an overview... Anyway, the answer to both your questions lies in that the Pattern Matching feature itself is rather complex and the PEPs ended up being huge and hard to read and understand as it is. We therefore refrained from long lists of rejected ideas in PEP 634, since that has already been done in PEP 622. Moreover, the `__match__` method and protocol were removed from the PEPs to focus on a core infrastructure and keep it as simple as possible. I think such a customisable protocol will eventually add great value to pattern matching. However, one of the main arguments was to wait until we have more experience with pattern matching in Python and can give specific use cases for this extended protocol, which allow us to guide the design of the protocol. At the end of the day, I guess we quite agree that we would like to have `__match__`, `__deconstruct__` or whatever you want to name it. But the slight variations in design demonstrate that it might also be a good idea to carefully lay it out first, before adding it to the pattern matching engine. Hence, separating this from the core implementation seems like a good idea that I still fully support (despite also looking forward to having it some day in the future ^_^). Cheers, Tobias
Hi Mark, Thank you for your proposal to try and have more precise semantics for pattern matching. Of course, the proposal primarily introduces a new and extended protocol for pattern matching, upon which the 'semantics' is then based. I think it is important to recognise and discuss your work on this basis that it proposes a different protocol of how pattern matching works, before we consier whether your semantics really is 'more precise'. As I understand it, your PEP proposes two major differences: control over the matching process moves from patterns to the objects and a custom protocol is introduced (via the `__deconstruct__` method). There is also the matter of the `__match_kind__` with the intention of improving performance, which I find of minor importance here. If you go back and look at the history and development of PEP 634, you will find that we started off with a fully customisable method that we called `__match__` instead of `__deconstruct__`. The idea of more flexible and customisable pattern matching is thus certainly in the spirit of our proposal as well. However, we finally removed this part from our proposal due to strong concerns from the community that the match protocol is too complex, particularly for an initial version of pattern matching. Should the need arise, it would still be possible to later add such a customisation protocol in an upcoming version of Python. Given these concerns with respect to complexity and our concession to remove the `__match__` method, I am wondering: do you have strong arguments that would speak for inclusion of this feature in the first version of pattern matching after all? I would love to have this feature and am truly interested in any arguments in its favour. When pattern matching meets OOP, there is indeed some discussion that can be had on where the matching itself is to happen. Simply put: should an object recognise that it is an instance of a given class, or should the class recognise that an object is an instance of it? In the spirit of Python's `instancecheck` method, we opted for classes/patterns recognising whether an object matches, where the object itself is 'passive'. In other words, `case C()` would match any object that is an instance of a subclass of `C`, in line with usual OOP principles (where any instance of a subclass of `C` is still a valid instance of `C` itself). Your design turns this around and has the object take on the responsibility of checking whether it wants to match a specific pattern. This has the advantage that the object itself can morph into anything it wants to—as long as it is aware of the pattern. And therein lies the problem that IMHO fully negates one of the major advantages of pattern matching: that the patterns and the objects that are to be matched are de-coupled and fully independent. In other words: in your design it is not she who writes the patterns that controls what objects match, but he who designs the class hierarchy of the objects used. One of the great traits of pattern matching is that it works out-of-the-box even with objects from ancient or alien code bases. Pattern matching according to PEP 634 is isolated, it happens only where pattern matching is explicitly used. When writing classes, you do not have to worry about pattern matching at all, unless you want your classes to be used as patterns. And should we decide some time in the future that we want to introduce new kinds of patterns, we still do not have to change a single object because the matching happens in the patterns and not in the objects, anyway. That his would not be the case with your design is already obvious with the list of `__match_kind__` for common builtin classes, for instance. Although there are some other programming languages that chose to follow this route, I think it goes against the spirit of what is already there in Python, it violates the principle of separated concerns and is thus a rather ill-advised modification of the original protocol. Kind regards, Tobias
On Fri, 19 Feb 2021 at 15:41, Tobias Kohn <kohnt@tobiaskohn.ch> wrote:
Hi Mark,
Thank you for your proposal to try and have more precise semantics for pattern matching. Of course, the proposal primarily introduces a new and extended protocol for pattern matching, upon which the 'semantics' is then based. I think it is important to recognise and discuss your work on this basis that it proposes a different protocol of how pattern matching works, before we consier whether your semantics really is 'more precise'.
...
If you go back and look at the history and development of PEP 634, you will find that we started off with a fully customisable method that we called `__match__` instead of `__deconstruct__`. The idea of more flexible and customisable pattern matching is thus certainly in the spirit of our proposal as well. However, we finally removed this part from our proposal due to strong concerns from the community that the match protocol is too complex, particularly for an initial version of pattern matching. Should the need arise, it would still be possible to later add such a customisation protocol in an upcoming version of Python.
It's not easy now to look back over the history of all of this. My recollection of the original version of PEP 622 (if that's the right one...) was that it had an overcomplicated protocol for __match__. It needed to be simplified but in the end it was dropped. The question now is if it will be straight-forward to retrofit a protocol to PEP 634 after it is released and when backward compatibility constraints kick in. PEP 653 (as discussed here) is precisely an attempt to retrofit a protocol to PEP 634. I think the difficulties involved in achieving that will become harder rather than easier in future. -- Oscar
Hi Oscar Quoting Oscar Benjamin <oscar.j.benjamin@gmail.com>:
Without further context, I would agree with you: it is difficult to look back over the entire history of the pattern matching PEPs. However, on the one hand PEP 622 is directly linked in the current PEP 634. On the other hand, for those who actively particiated in the discussion of both PEP 622 as well as the DLS paper, I find it a bit too easy and 'convenient' to call those resources now 'hard to find'. That being said, I think it is part of your homework to first research about the history and what others have done before proposing your innovation. Granted, this is hard and very laborious work that often takes a long time and can be frustrating. But if we want to make progress and move forward, we have to stop running in circles.—To be absolutely clear here: I do not think that Mark's proposal is running in circles and I think it is fair enough to bring up these ideas. But I equally think it is well possible to acknowledge that one part of it is a discussion of existing ideas, and to have a look at why these ideas have not made it into PEP 634.
There are actually two protocols in PEP 653. One is about changing the fundamental design of pattern matching as outlined in PEP 634. This is a part that I reject for reasons presented in one of my last posts. The second one is the customisation part using `__deconstruct__` or `__match__`. This is something that I like a lot and would certainly like to see in pattern matching—alas, just being in favour of something is not a strong enough argument for it. Similar to my votum above: if we are going to discuss this, I think we need new arguments or evidence, something to show why the previous decision to drop it (for now) was wrong. For that it might be worth reading the respective section in deferred ideas of PEP 622. Some part of our decision was based on the notion that adding such a custom protocol later one will be relatively painless, but if we were wrong, please speak up and show us what we have overlooked. If you can present, for instance, some real-world use cases where this feature would enormously benefit sympy (or any other popular package), that would be helpful and a good starting point. Kind regards, Tobias
In addition to the changes proposed here that go beyond PEP-634 (which other people discuss), I find that many of the definitions fail to capture some of the basic features of PEP-634, especially when nesting patterns. Take for example: "case [int(), str()]". According to https://www.python.org/dev/peps/pep-0653/#sequence-patterns that desugars to: if $kind & MATCH_SEQUENCE: if $list is None: $list = list($value) if len($list) == len([int(), str()]): int(), str() = $list *# This is invalid!* if guard: DONE In general, the way these semantics are defined non-recursively over patterns won't be able to work with nested patterns. I value the idea of having a more accurate definition of what the semantics are, but this approach doesn't seem to work. Best, D. On Thu, 18 Feb 2021 at 16:45, Mark Shannon <mark@hotpy.org> wrote:
Thanks for taking the time to work on this, Mark. Overall, I'm skeptical of this proposal. It seems like it takes a lot of "simple" things and makes them quite complex, and takes many "static" things and makes them quite dynamic. I feel that it also misrepresents certain aspects of PEP 634. Here's a more detailed commentary:
Pattern matching will be more usable for complex classes, by allowing classes more control over which patterns they match.
I fear this is at the expense of most simple classes, which currently "just work" with PEP 634. I'm not convinced that making it easier for very complicated classes (like those in `sympy`) to participate at the expense of everyday classes is a win. For comparison, here is what (as I understand it) each PEP requires for a class to be used in a pattern such as `C(x, y, z)`: ```py class C: """A PEP 634 matchable class.""" __match_args__ = ... class C: """A PEP 653 matchable class.""" __match_kind__ = MATCH_CLASS __attributes__ = ... def __deconstruct__(self): ... ``` For a pattern with no positional subpatterns, such as like `C()` or `C(x=x, y=y, z=z)`: ```py class C: """A PEP 634 matchable class (all classes work).""" class C: """A PEP 653 matchable class.""" __match_kind__ = MATCH_CLASS __attributes__ = ... def __deconstruct__(self): ... ``` It also appears that we lose a lot of expressive "idioms" by requiring `__attributes__` to be complete. For example, the following attribute extractions work with PEP 634, but not PEP 653. ```py match subject: case object(a=_): # Match any object with an "a" attribute. ... case object(x=x, y=y, z=z): # Match any object with "x", "y", and "z" attributes, extracting them. ... ``` This also means that matching classes like `types.SimpleNamespace` are much less powerful under PEP 653, since the class must know which attributes are "allowed" to be looked up. Further, this whole specification shifts most of the matching logic from the class in the pattern to the subject itself: ```py match ChildA(), ChildB(): case Parent(a, b), Parent(x, y): ... ``` The above pattern could have completely different rules for extracting `a` and `b` vs `x` and `y` if either child overrides `__deconstruct__`. I think that is a mistake.
PEP 634 also privileges some builtin classes with a special form of matching, the "self" match. For example the pattern `list(x)` matches a list and assigns the list to `x`. By allowing classes to choose which kinds of pattern they match, other classes can use this form as well.
This is already fairly trivial to implement: ```py class C: __match_args__ = ("_self",) _self = property(lambda s: s) ``` You can even avoid adding a `_self` attribute if you do some magic with descriptors... ;). We could consider provide a decorator for this somewhere in the stdlib if there's demonstrated need (but I'm not sure there will be).
Oof. I was scratching my head for way too long at this before I noticed the previous sentence said "false". Maybe reword this section to indicate the conditions that hold *true*? :)
Security Implications Preventing the possible registering or unregistering of classes as sequences or a mappings, that PEP 634 allows, should improve security. However, the advantage is slight and is not a motivation for this PEP.
I would say the advantage is nonexistent. How is this any different than tweaking the flags in a class's `__match_kind__`?
Efficient Implementation / Implementation
I won't get too deep into this section, but on the surface most flavors of implementation/optimization present here are also possible with PEP 634. I understand that you feel there are benefits to having "rules" for what optimizations are legal, but we don't need to completely change the mechanics of the match statement in order to do so.
Because the object model is a core part of Python, implementations already handle special attribute lookup efficiently. Looking up a special attribute is much faster than performing a subclass test on an abstract base class.
But calling a `__deconstruct__` method whenever positional arguments are present will slow down normal class matches, right? I see it as mostly a wash.
In general, it seems to me that this proposal prioritizes: - complicated library code - ease of implementation in the interpreter over: - simple client code - general usability I can't say I support that, generally. When working on PEP 634, we tossed lot of more complicated, powerful protocols (like `__match__` and several others before it... don't mention the phrase "match proxy" around any of the authors!) in favor of simplicity. This sort of feels like a rehash of that. Brandt
Brandt Bucher wrote:
For a pattern with no positional subpatterns, such as like `C()` or `C(x=x, y=y, z=z)`: ...
It also appears that we lose a lot of expressive "idioms" by requiring `__attributes__` to be complete.
This also means that matching classes like `types.SimpleNamespace` are much less powerful under PEP 653, since the class must know which attributes are "allowed" to be looked up.
Never mind these three points... I *think* setting `__match_kind__ = MATCH_DEFAULT` allows for arbitrary attribute extraction like this. Perhaps make it bit clearer?
On Thu, 18 Feb 2021 at 17:35, Brandt Bucher <brandtbucher@gmail.com> wrote:
Thanks for taking the time to work on this, Mark.
Yes, thanks Mark. I'm not sure I've fully understood the PEP yet but I can see some parts that I definitely like.
I fear this is at the expense of most simple classes, which currently "just work" with PEP 634. I'm not convinced that making it easier for very complicated classes (like those in `sympy`) to participate at the expense of everyday classes is a win.
While those sympy classes are quite complicated, the relevant aspect of them is not. The issue is just that they have positional only constructor arguments. PEP 634 does not allow for positional only arguments in a class pattern in a way that can work in general. This is because it requires attributes to exist for classes that would otherwise have no need for them.
You are presuming here that the strings in __match_args__ already correspond to attributes. Otherwise the PEP 634 version above requires a property to be added corresponding to each positional argument in the constructor: class C: """A PEP 634 matchable class""" __match_args__ = ['a', 'b', 'c'] a = property(lambda self: self._rep[0]) b = property(lambda self: self._rep[1]) c = property(lambda self: self._rep[2]) At runtime case/match looks into __match_args__ and calls getattr evaluating each property one by one (even if they all derive from the same internal data structure).
PEP 653 says (although it wasn't immediately obvious to me from reading it): """ Classes which define __match_kind__ & MATCH_CLASS to be non-zero must implement one additional special attribute, and one special method: """ Then much later: """ object.__match_kind__ will be MATCH_DEFAULT. """ I think that means that a simple `class C: pass` will match keyword patterns with attributes so C(x=1) will match against any C with c.x == 1. To match positional arguments both __match_kind__, __attributes__ and __deconstruct__ would need to be defined but the end result works better for positional arguments than PEP 634 does: - It is possible to match the number of positional arguments precisely so a pattern C(x) would not match an object C(x, y) - Any invertible mapping between positional arguments and internal class state can be accommodated. I don't think it's possible to achieve those with PEP 634. A concrete example would be matching against a range class: class range: __match_args__ = ['start', 'stop', 'step'] def __init__(self, start_or_stop, stop=None, step=1): if stop is None: start = 0 stop = start_or_stop else: start = start_or_stop self.start = start self.stop = stop self.step = step With PEP 634 a pattern like range(10) will also match objects like range(10, 20) and range(10, 20, 2) etc. I think that is not what someone familiar with range would expect. With PEP 634 there is no way to define __match_args__ so that positional argument patterns with range match in a way that corresponds to its constructor syntax (or that have useful matching semantics in this case). I'm not entirely sure but I think that with PEP 653 you can implement this like: def __deconstruct__(obj): if obj.step != 1: return obj.start, obj.stop, obj.step elif obj.start != 0: return obj.start, obj.stop else: return obj.stop I think that would then mean you could use a pattern range(10) to unambiguously match range(10) without matching range(10, 20) etc. Above I suggested that any *invertible* mapping of args to state could be supported and here there is an example of a non-invertible mapping since range(0, 10) is the same as range(10). I might have misunderstood but I don't think that PEP 653 makes it possible to have a pattern range(10) and a pattern range(0, 10) both match against the same object range(10). This means that the pattern would always need to use the "canonical" form of the args (e.g, range(10) in this case). -- Oscar
Hi Oscar, Quoting Oscar Benjamin <oscar.j.benjamin@gmail.com>:
Yes, thanks Mark. I'm not sure I've fully understood the PEP yet but I can see some parts that I definitely like. [...]
As I have just noted in my response to Mark, the aspect with the "deconstructor" (or `__match__` protocol as we called it) is definitely something that I like, too. Moreover, I think that packages like sympy might make a strong argument for it. That being said, perhaps we would have to start thinking about concrete use cases first and then consider how to best provide for those with an extended match protocol. Even though I still like the idea of a flexible match method, there might be aspects that I/we overlook without knowing exactly where we want to go with this.
This is certainly more of an anti-pattern (no pun intended) than an argument for the match protocol. Shuffling positional arguments around like this seems like a rather bad idea, particularly in the context of pattern matching. I'd rather write `case range(_, stop, _)` or `case range(0, stop, 1)` and be explicit here. Kind regards, Tobias
On Fri, 19 Feb 2021 at 16:27, Tobias Kohn <kohnt@tobiaskohn.ch> wrote:
Good point! I was trying to reason out the mechanism but the alternative approach as you say would be to always use all 3 arguments. PEP 653 here actually makes it possible for the range class to affect matching so that it can only succeed against a pattern that uses all 3 arguments. The significance of range as an example is basically that it has a varying number of positional only arguments to its constructor. The most important feature of the example is not the reordering of arguments: it is the fact that with PEP 634 a pattern range(x) would have to match range(x, y) which is very unlikely to be useful in the case of range but also not generally applicable for other types either. There is no way for range (or other types) to prevent that while supporting positional argument matching through __match_args__. Under PEP 634 in general, for any class C, a pattern C(x) matches an object C(x, y) and there's no way for C to override that. To me that is sufficiently unintuitive in the abstract that no example is really needed to see where there is room for improvement. -- Oscar
Oscar Benjamin wrote:
Under PEP 634 in general, for any class C, a pattern C(x) matches an object C(x, y) and there's no way for C to override that. To me that is sufficiently unintuitive in the abstract that no example is really needed to see where there is room for improvement.
We originally kicked around (and at one point even implemented) a `__match_args_required__` attribute, which is an integer specifying a minimum required number of positional sub-patterns. For reasons I can't recall, though, it was eventually dropped. It would take care of this, and I imagine it could be quite painless to add it back if there was enough support. Here's a query for related discussions: https://github.com/gvanrossum/patma/issues?q=__match_args_required__
Hi Brandt, On 18/02/2021 5:32 pm, Brandt Bucher wrote:
In my experience, nothing "just works". It means "mostly works most of the time, and occasionally goes horribly wrong"
True, but most pure-Python classes don't have a natural order of attributes, so the pattern is likely to be `C(x=x,y=y,z=z)`. In which case no extra work is required for either PEP to pattern match.
This is not true. MATCH_DEFAULT does what you want.
You need to define what you mean by "work" :) This behaves the same for both PEPs: class C: __init__(self, x, y, z): ... match C(1,2,3): case object(x=x, y=y, z=z): ... This also works the same for both PEPs, albeit with different mechanisms: match namedtuple("n", "x, y, z")(1,2,3): case object(x=x, y=y, z=z): ... But, with PEP 634:
Which seem like "not working" to me.
This also means that matching classes like `types.SimpleNamespace` are much less powerful under PEP 653, since the class must know which attributes are "allowed" to be looked up.
Why do you say this? This should work the same for PEP 634 and 653:
Inheritance allows sub-classes to change the behavior of their super-classes. Yes, it can produce weird effects if they ignore Liskov's substitution principle. That why the modern advice is "Favour composition over inheritance". You can't prevent people writing bad code.
That's pretty obscure, IMO. You can't claim that it is trivial compared to: class C: __match_kind__ = MATCH_SELF
Good point. I'll do that.
Then please publish a precise semantics for PEP 634 pattern matching. You say that "most flavors of implementation/optimization present here are also possible with PEP 634", but without semantics you can't show that optimizations preserve the semantics.
Because the object model is a core part of Python, implementations already handle special attribute lookup efficiently. Looking up a special attribute is much faster than performing a subclass test on an abstract base class.
But calling a `__deconstruct__` method whenever positional arguments are present will slow down normal class matches, right? I see it as mostly a wash.
Why do you think it will slow it down? `__deconstruct__` is called only once. PEP 634, as written, has to perform a sequence of attribute lookups for *each* class pattern. If a class defines `__deconstruct__`, it is likely to be for a reason. For example, namedtuple's `__deconstruct__` would be: def __deconstruct__(self): return self Which is both simple and efficient.
It what way does this PEP make client code more complex, or impair usability? Evidence please. "Ease of implementation in the interpreter" improves reliability and performance. Those are important things in a language runtime :)
I can't say I support that, generally. When working on PEP 634, we tossed lot of more complicated, powerful protocols (like `__match__` and several others before it... don't mention the phrase "match proxy" around any of the authors!) in favor of simplicity. This sort of feels like a rehash of that.
If you had published these "more complicated, powerful protocols", you might be able to claim that this is a "rehash". But you didn't. Being vague is not "simplicity", it is merely hiding the complexity. Cheers, Mark.
Hi Mark, Quoting Mark Shannon <mark@hotpy.org>:
I would say that these ideas have been quite prominently published: https://www.python.org/dev/peps/pep-0622/#custom-matching-protocol https://dl.acm.org/doi/10.1145/3426422.3426983 Moreover, much of the discussion is publicly available, for instance here: https://github.com/gvanrossum/patma/issues/8 Cheers, Tobias
Hi Tobias, On 19/02/2021 5:57 pm, Tobias Kohn wrote:
But they are not referenced in PEP 634. I shouldn't have to trawl the internet to find the rejected ideas section.
That paper describes a `__match__` method, which is absent from PEP 634. Why? Cheers, Mark.
Hi Mark, Quoting Mark Shannon <mark@hotpy.org>:
You are right: you should not have to trawl the internet to get these information. However, given the history of the Pattern Matching PEPs and that PEP 622 is linked directly from PEP 634, I would think it is not that far a journey—even though there have been so many discussions that it has definitely become unwieldy to retain an overview... Anyway, the answer to both your questions lies in that the Pattern Matching feature itself is rather complex and the PEPs ended up being huge and hard to read and understand as it is. We therefore refrained from long lists of rejected ideas in PEP 634, since that has already been done in PEP 622. Moreover, the `__match__` method and protocol were removed from the PEPs to focus on a core infrastructure and keep it as simple as possible. I think such a customisable protocol will eventually add great value to pattern matching. However, one of the main arguments was to wait until we have more experience with pattern matching in Python and can give specific use cases for this extended protocol, which allow us to guide the design of the protocol. At the end of the day, I guess we quite agree that we would like to have `__match__`, `__deconstruct__` or whatever you want to name it. But the slight variations in design demonstrate that it might also be a good idea to carefully lay it out first, before adding it to the pattern matching engine. Hence, separating this from the core implementation seems like a good idea that I still fully support (despite also looking forward to having it some day in the future ^_^). Cheers, Tobias
Hi Mark, Thank you for your proposal to try and have more precise semantics for pattern matching. Of course, the proposal primarily introduces a new and extended protocol for pattern matching, upon which the 'semantics' is then based. I think it is important to recognise and discuss your work on this basis that it proposes a different protocol of how pattern matching works, before we consier whether your semantics really is 'more precise'. As I understand it, your PEP proposes two major differences: control over the matching process moves from patterns to the objects and a custom protocol is introduced (via the `__deconstruct__` method). There is also the matter of the `__match_kind__` with the intention of improving performance, which I find of minor importance here. If you go back and look at the history and development of PEP 634, you will find that we started off with a fully customisable method that we called `__match__` instead of `__deconstruct__`. The idea of more flexible and customisable pattern matching is thus certainly in the spirit of our proposal as well. However, we finally removed this part from our proposal due to strong concerns from the community that the match protocol is too complex, particularly for an initial version of pattern matching. Should the need arise, it would still be possible to later add such a customisation protocol in an upcoming version of Python. Given these concerns with respect to complexity and our concession to remove the `__match__` method, I am wondering: do you have strong arguments that would speak for inclusion of this feature in the first version of pattern matching after all? I would love to have this feature and am truly interested in any arguments in its favour. When pattern matching meets OOP, there is indeed some discussion that can be had on where the matching itself is to happen. Simply put: should an object recognise that it is an instance of a given class, or should the class recognise that an object is an instance of it? In the spirit of Python's `instancecheck` method, we opted for classes/patterns recognising whether an object matches, where the object itself is 'passive'. In other words, `case C()` would match any object that is an instance of a subclass of `C`, in line with usual OOP principles (where any instance of a subclass of `C` is still a valid instance of `C` itself). Your design turns this around and has the object take on the responsibility of checking whether it wants to match a specific pattern. This has the advantage that the object itself can morph into anything it wants to—as long as it is aware of the pattern. And therein lies the problem that IMHO fully negates one of the major advantages of pattern matching: that the patterns and the objects that are to be matched are de-coupled and fully independent. In other words: in your design it is not she who writes the patterns that controls what objects match, but he who designs the class hierarchy of the objects used. One of the great traits of pattern matching is that it works out-of-the-box even with objects from ancient or alien code bases. Pattern matching according to PEP 634 is isolated, it happens only where pattern matching is explicitly used. When writing classes, you do not have to worry about pattern matching at all, unless you want your classes to be used as patterns. And should we decide some time in the future that we want to introduce new kinds of patterns, we still do not have to change a single object because the matching happens in the patterns and not in the objects, anyway. That his would not be the case with your design is already obvious with the list of `__match_kind__` for common builtin classes, for instance. Although there are some other programming languages that chose to follow this route, I think it goes against the spirit of what is already there in Python, it violates the principle of separated concerns and is thus a rather ill-advised modification of the original protocol. Kind regards, Tobias
On Fri, 19 Feb 2021 at 15:41, Tobias Kohn <kohnt@tobiaskohn.ch> wrote:
Hi Mark,
Thank you for your proposal to try and have more precise semantics for pattern matching. Of course, the proposal primarily introduces a new and extended protocol for pattern matching, upon which the 'semantics' is then based. I think it is important to recognise and discuss your work on this basis that it proposes a different protocol of how pattern matching works, before we consier whether your semantics really is 'more precise'.
...
If you go back and look at the history and development of PEP 634, you will find that we started off with a fully customisable method that we called `__match__` instead of `__deconstruct__`. The idea of more flexible and customisable pattern matching is thus certainly in the spirit of our proposal as well. However, we finally removed this part from our proposal due to strong concerns from the community that the match protocol is too complex, particularly for an initial version of pattern matching. Should the need arise, it would still be possible to later add such a customisation protocol in an upcoming version of Python.
It's not easy now to look back over the history of all of this. My recollection of the original version of PEP 622 (if that's the right one...) was that it had an overcomplicated protocol for __match__. It needed to be simplified but in the end it was dropped. The question now is if it will be straight-forward to retrofit a protocol to PEP 634 after it is released and when backward compatibility constraints kick in. PEP 653 (as discussed here) is precisely an attempt to retrofit a protocol to PEP 634. I think the difficulties involved in achieving that will become harder rather than easier in future. -- Oscar
Hi Oscar Quoting Oscar Benjamin <oscar.j.benjamin@gmail.com>:
Without further context, I would agree with you: it is difficult to look back over the entire history of the pattern matching PEPs. However, on the one hand PEP 622 is directly linked in the current PEP 634. On the other hand, for those who actively particiated in the discussion of both PEP 622 as well as the DLS paper, I find it a bit too easy and 'convenient' to call those resources now 'hard to find'. That being said, I think it is part of your homework to first research about the history and what others have done before proposing your innovation. Granted, this is hard and very laborious work that often takes a long time and can be frustrating. But if we want to make progress and move forward, we have to stop running in circles.—To be absolutely clear here: I do not think that Mark's proposal is running in circles and I think it is fair enough to bring up these ideas. But I equally think it is well possible to acknowledge that one part of it is a discussion of existing ideas, and to have a look at why these ideas have not made it into PEP 634.
There are actually two protocols in PEP 653. One is about changing the fundamental design of pattern matching as outlined in PEP 634. This is a part that I reject for reasons presented in one of my last posts. The second one is the customisation part using `__deconstruct__` or `__match__`. This is something that I like a lot and would certainly like to see in pattern matching—alas, just being in favour of something is not a strong enough argument for it. Similar to my votum above: if we are going to discuss this, I think we need new arguments or evidence, something to show why the previous decision to drop it (for now) was wrong. For that it might be worth reading the respective section in deferred ideas of PEP 622. Some part of our decision was based on the notion that adding such a custom protocol later one will be relatively painless, but if we were wrong, please speak up and show us what we have overlooked. If you can present, for instance, some real-world use cases where this feature would enormously benefit sympy (or any other popular package), that would be helpful and a good starting point. Kind regards, Tobias
In addition to the changes proposed here that go beyond PEP-634 (which other people discuss), I find that many of the definitions fail to capture some of the basic features of PEP-634, especially when nesting patterns. Take for example: "case [int(), str()]". According to https://www.python.org/dev/peps/pep-0653/#sequence-patterns that desugars to: if $kind & MATCH_SEQUENCE: if $list is None: $list = list($value) if len($list) == len([int(), str()]): int(), str() = $list *# This is invalid!* if guard: DONE In general, the way these semantics are defined non-recursively over patterns won't be able to work with nested patterns. I value the idea of having a more accurate definition of what the semantics are, but this approach doesn't seem to work. Best, D. On Thu, 18 Feb 2021 at 16:45, Mark Shannon <mark@hotpy.org> wrote:
participants (5)
-
Brandt Bucher -
Daniel Moisset -
Mark Shannon -
Oscar Benjamin -
Tobias Kohn