Brandt Bucher wrote:

For a pattern with no positional subpatterns, such as like `C()` or `C(x=x, y=y, z=z)`: ...

It also appears that we lose a lot of expressive "idioms" by requiring `__attributes__` to be complete.

This also means that matching classes like `types.SimpleNamespace` are much less powerful under PEP 653, since the class must know which attributes are "allowed" to be looked up.

Never mind these three points... I *think* setting `__match_kind__ = MATCH_DEFAULT` allows for arbitrary attribute extraction like this. Perhaps make it bit clearer?

On Thu, 18 Feb 2021 at 17:35, Brandt Bucher <brandtbucher@gmail.com> wrote:

Thanks for taking the time to work on this, Mark.

Yes, thanks Mark. I'm not sure I've fully understood the PEP yet but I can see some parts that I definitely like.

I fear this is at the expense of most simple classes, which currently "just work" with PEP 634. I'm not convinced that making it easier for very complicated classes (like those in `sympy`) to participate at the expense of everyday classes is a win.

While those sympy classes are quite complicated, the relevant aspect of them is not. The issue is just that they have positional only constructor arguments. PEP 634 does not allow for positional only arguments in a class pattern in a way that can work in general. This is because it requires attributes to exist for classes that would otherwise have no need for them.

For comparison, here is what (as I understand it) each PEP requires for a class to be used in a pattern such as `C(x, y, z)`:

```py class C: """A PEP 634 matchable class.""" __match_args__ = ...

You are presuming here that the strings in __match_args__ already correspond to attributes. Otherwise the PEP 634 version above requires a property to be added corresponding to each positional argument in the constructor: class C: """A PEP 634 matchable class""" __match_args__ = ['a', 'b', 'c'] a = property(lambda self: self._rep[0]) b = property(lambda self: self._rep[1]) c = property(lambda self: self._rep[2]) At runtime case/match looks into __match_args__ and calls getattr evaluating each property one by one (even if they all derive from the same internal data structure).

class C: """A PEP 653 matchable class.""" __match_kind__ = MATCH_CLASS __attributes__ = ... def __deconstruct__(self): ... ```

PEP 653 says (although it wasn't immediately obvious to me from reading it): """ Classes which define __match_kind__ & MATCH_CLASS to be non-zero must implement one additional special attribute, and one special method: """ Then much later: """ object.__match_kind__ will be MATCH_DEFAULT. """ I think that means that a simple `class C: pass` will match keyword patterns with attributes so C(x=1) will match against any C with c.x == 1. To match positional arguments both __match_kind__, __attributes__ and __deconstruct__ would need to be defined but the end result works better for positional arguments than PEP 634 does: - It is possible to match the number of positional arguments precisely so a pattern C(x) would not match an object C(x, y) - Any invertible mapping between positional arguments and internal class state can be accommodated. I don't think it's possible to achieve those with PEP 634. A concrete example would be matching against a range class: class range: __match_args__ = ['start', 'stop', 'step'] def __init__(self, start_or_stop, stop=None, step=1): if stop is None: start = 0 stop = start_or_stop else: start = start_or_stop self.start = start self.stop = stop self.step = step With PEP 634 a pattern like range(10) will also match objects like range(10, 20) and range(10, 20, 2) etc. I think that is not what someone familiar with range would expect. With PEP 634 there is no way to define __match_args__ so that positional argument patterns with range match in a way that corresponds to its constructor syntax (or that have useful matching semantics in this case). I'm not entirely sure but I think that with PEP 653 you can implement this like: def __deconstruct__(obj): if obj.step != 1: return obj.start, obj.stop, obj.step elif obj.start != 0: return obj.start, obj.stop else: return obj.stop I think that would then mean you could use a pattern range(10) to unambiguously match range(10) without matching range(10, 20) etc. Above I suggested that any *invertible* mapping of args to state could be supported and here there is an example of a non-invertible mapping since range(0, 10) is the same as range(10). I might have misunderstood but I don't think that PEP 653 makes it possible to have a pattern range(10) and a pattern range(0, 10) both match against the same object range(10). This means that the pattern would always need to use the "canonical" form of the args (e.g, range(10) in this case). -- Oscar

On Fri, 19 Feb 2021 at 16:27, Tobias Kohn <kohnt@tobiaskohn.ch> wrote:

Quoting Oscar Benjamin <oscar.j.benjamin@gmail.com>:

...

I'm not entirely sure but I think that with PEP 653 you can implement this like:

def __deconstruct__(obj): if obj.step != 1: return obj.start, obj.stop, obj.step elif obj.start != 0: return obj.start, obj.stop else: return obj.stop

I think that would then mean you could use a pattern range(10) to unambiguously match range(10) without matching range(10, 20) etc.

This is certainly more of an anti-pattern (no pun intended) than an argument for the match protocol. Shuffling positional arguments around like this seems like a rather bad idea, particularly in the context of pattern matching. I'd rather write `case range(_, stop, _)` or `case range(0, stop, 1)` and be explicit here.

Good point! I was trying to reason out the mechanism but the alternative approach as you say would be to always use all 3 arguments. PEP 653 here actually makes it possible for the range class to affect matching so that it can only succeed against a pattern that uses all 3 arguments. The significance of range as an example is basically that it has a varying number of positional only arguments to its constructor. The most important feature of the example is not the reordering of arguments: it is the fact that with PEP 634 a pattern range(x) would have to match range(x, y) which is very unlikely to be useful in the case of range but also not generally applicable for other types either. There is no way for range (or other types) to prevent that while supporting positional argument matching through __match_args__. Under PEP 634 in general, for any class C, a pattern C(x) matches an object C(x, y) and there's no way for C to override that. To me that is sufficiently unintuitive in the abstract that no example is really needed to see where there is room for improvement. -- Oscar

Hi Brandt, On 18/02/2021 5:32 pm, Brandt Bucher wrote:

Thanks for taking the time to work on this, Mark.

Overall, I'm skeptical of this proposal. It seems like it takes a lot of "simple" things and makes them quite complex, and takes many "static" things and makes them quite dynamic. I feel that it also misrepresents certain aspects of PEP 634.

Here's a more detailed commentary:

...

Pattern matching will be more usable for complex classes, by allowing classes more control over which patterns they match.

I fear this is at the expense of most simple classes, which currently "just work" with PEP 634. I'm not convinced that making it easier for very complicated classes (like those in `sympy`) to participate at the expense of everyday classes is a win.

In my experience, nothing "just works". It means "mostly works most of the time, and occasionally goes horribly wrong"

For comparison, here is what (as I understand it) each PEP requires for a class to be used in a pattern such as `C(x, y, z)`:

```py class C: """A PEP 634 matchable class.""" __match_args__ = ...

class C: """A PEP 653 matchable class.""" __match_kind__ = MATCH_CLASS __attributes__ = ... def __deconstruct__(self): ... ```

True, but most pure-Python classes don't have a natural order of attributes, so the pattern is likely to be `C(x=x,y=y,z=z)`. In which case no extra work is required for either PEP to pattern match.

For a pattern with no positional subpatterns, such as like `C()` or `C(x=x, y=y, z=z)`:

```py class C: """A PEP 634 matchable class (all classes work)."""

class C: """A PEP 653 matchable class.""" __match_kind__ = MATCH_CLASS __attributes__ = ... def __deconstruct__(self): ... ```

This is not true. MATCH_DEFAULT does what you want.

It also appears that we lose a lot of expressive "idioms" by requiring `__attributes__` to be complete. For example, the following attribute extractions work with PEP 634, but not PEP 653.

```py match subject: case object(a=_): # Match any object with an "a" attribute. ... case object(x=x, y=y, z=z): # Match any object with "x", "y", and "z" attributes, extracting them. ... ```

You need to define what you mean by "work" :) This behaves the same for both PEPs: class C: __init__(self, x, y, z): ... match C(1,2,3): case object(x=x, y=y, z=z): ... This also works the same for both PEPs, albeit with different mechanisms: match namedtuple("n", "x, y, z")(1,2,3): case object(x=x, y=y, z=z): ... But, with PEP 634:

...

...

match(sympy.cos(x)): case object(n): ... n <bound method EvalfMixin.evalf of cos(x)>

Which seem like "not working" to me.

This also means that matching classes like `types.SimpleNamespace` are much less powerful under PEP 653, since the class must know which attributes are "allowed" to be looked up.

Why do you say this? This should work the same for PEP 634 and 653:

...

...

match types.SimpleNamespace(a = 1): case object(a): print(a) 1

Further, this whole specification shifts most of the matching logic from the class in the pattern to the subject itself:

```py match ChildA(), ChildB(): case Parent(a, b), Parent(x, y): ... ```

The above pattern could have completely different rules for extracting `a` and `b` vs `x` and `y` if either child overrides `__deconstruct__`. I think that is a mistake.

Inheritance allows sub-classes to change the behavior of their super-classes. Yes, it can produce weird effects if they ignore Liskov's substitution principle. That why the modern advice is "Favour composition over inheritance". You can't prevent people writing bad code.

...

PEP 634 also privileges some builtin classes with a special form of matching, the "self" match. For example the pattern `list(x)` matches a list and assigns the list to `x`. By allowing classes to choose which kinds of pattern they match, other classes can use this form as well.

This is already fairly trivial to implement:

```py class C: __match_args__ = ("_self",) _self = property(lambda s: s) ```

That's pretty obscure, IMO. You can't claim that it is trivial compared to: class C: __match_kind__ = MATCH_SELF

You can even avoid adding a `_self` attribute if you do some magic with descriptors... ;). We could consider provide a decorator for this somewhere in the stdlib if there's demonstrated need (but I'm not sure there will be).

...

All classes should ensure that the the value of `__match_kind__` follows the specification. Therefore, implementations can assume, without checking, that all the following are false: `(__match_kind__ & (MATCH_SEQUENCE | MATCH_MAPPING)) == (MATCH_SEQUENCE | MATCH_MAPPING)` `(__match_kind__ & (MATCH_SELF | MATCH_CLASS)) == (MATCH_SELF | MATCH_CLASS)` `(__match_kind__ & (MATCH_SELF | MATCH_DEFAULT)) == (MATCH_SELF | MATCH_DEFAULT)` `(__match_kind__ & (MATCH_DEFAULT | MATCH_CLASS)) == (MATCH_DEFAULT | MATCH_CLASS)`

Oof. I was scratching my head for way too long at this before I noticed the previous sentence said "false". Maybe reword this section to indicate the conditions that hold *true*? :)

Good point. I'll do that.

...

Security Implications Preventing the possible registering or unregistering of classes as sequences or a mappings, that PEP 634 allows, should improve security. However, the advantage is slight and is not a motivation for this PEP.

I would say the advantage is nonexistent. How is this any different than tweaking the flags in a class's `__match_kind__`? >

...

Efficient Implementation / Implementation

I won't get too deep into this section, but on the surface most flavors of implementation/optimization present here are also possible with PEP 634. I understand that you feel there are benefits to having "rules" for what optimizations are legal, but we don't need to completely change the mechanics of the match statement in order to do so.

Then please publish a precise semantics for PEP 634 pattern matching. You say that "most flavors of implementation/optimization present here are also possible with PEP 634", but without semantics you can't show that optimizations preserve the semantics.

...

Because the object model is a core part of Python, implementations already handle special attribute lookup efficiently. Looking up a special attribute is much faster than performing a subclass test on an abstract base class.

But calling a `__deconstruct__` method whenever positional arguments are present will slow down normal class matches, right? I see it as mostly a wash.

Why do you think it will slow it down? `__deconstruct__` is called only once. PEP 634, as written, has to perform a sequence of attribute lookups for *each* class pattern. If a class defines `__deconstruct__`, it is likely to be for a reason. For example, namedtuple's `__deconstruct__` would be: def __deconstruct__(self): return self Which is both simple and efficient.

...

The changes to the semantics can be summarized as: - Selecting the kind of pattern uses `cls.__match_kind__` instead of `issubclass(cls, collections.abc.Mapping)` and `issubclass(cls, collections.abc.Sequence)` and allows classes control over which kinds of pattern they match. - Class matching is via the `__attributes__` attribute and `__deconstruct__` method, rather than the `__match_args__` method, and allows classes more control over how they are deconstructed.

In general, it seems to me that this proposal prioritizes:

- complicated library code - ease of implementation in the interpreter

over:

- simple client code - general usability

It what way does this PEP make client code more complex, or impair usability? Evidence please. "Ease of implementation in the interpreter" improves reliability and performance. Those are important things in a language runtime :)

I can't say I support that, generally. When working on PEP 634, we tossed lot of more complicated, powerful protocols (like `__match__` and several others before it... don't mention the phrase "match proxy" around any of the authors!) in favor of simplicity. This sort of feels like a rehash of that.

If you had published these "more complicated, powerful protocols", you might be able to claim that this is a "rehash". But you didn't. Being vague is not "simplicity", it is merely hiding the complexity. Cheers, Mark.

Hi Tobias, On 19/02/2021 5:57 pm, Tobias Kohn wrote:

Hi Mark,

Quoting Mark Shannon <mark@hotpy.org <mailto:mark@hotpy.org>>:

...

[...]

If you had published these "more complicated, powerful protocols", you might be able to claim that this is a "rehash". But you didn't.

I would say that these ideas have been quite prominently published: https://www.python.org/dev/peps/pep-0622/#custom-matching-protocol

But they are not referenced in PEP 634. I shouldn't have to trawl the internet to find the rejected ideas section.

https://dl.acm.org/doi/10.1145/3426422.3426983

That paper describes a `__match__` method, which is absent from PEP 634. Why? Cheers, Mark.

Hi Oscar Quoting Oscar Benjamin <oscar.j.benjamin@gmail.com>:

On Fri, 19 Feb 2021 at 15:41, Tobias Kohn <kohnt@tobiaskohn.ch> wrote: [...]

It's not easy now to look back over the history of all of this. My recollection of the original version of PEP 622 (if that's the right one...) was that it had an overcomplicated protocol for __match__. It needed to be simplified but in the end it was dropped.

Without further context, I would agree with you: it is difficult to look back over the entire history of the pattern matching PEPs.  However, on the one hand PEP 622 is directly linked in the current PEP 634.  On the other hand, for those who actively particiated in the discussion of both PEP 622 as well as the DLS paper, I find it a bit too easy and 'convenient' to call those resources now 'hard to find'. That being said, I think it is part of your homework to first research about the history and what others have done before proposing your innovation.  Granted, this is hard and very laborious work that often takes a long time and can be frustrating.  But if we want to make progress and move forward, we have to stop running in circles.—To be absolutely clear here: I do not think that Mark's proposal is running in circles and I think it is fair enough to bring up these ideas.  But I equally think it is well possible to acknowledge that one part of it is a discussion of existing ideas, and to have a look at why these ideas have not made it into PEP 634.  

The question now is if it will be straight-forward to retrofit a protocol to PEP 634 after it is released and when backward compatibility constraints kick in. PEP 653 (as discussed here) is precisely an attempt to retrofit a protocol to PEP 634. I think the difficulties involved in achieving that will become harder rather than easier in future.

-- Oscar

There are actually two protocols in PEP 653.  One is about changing the fundamental design of pattern matching as outlined in PEP 634.  This is a part that I reject for reasons presented in one of my last posts.  The second one is the customisation part using `__deconstruct__` or `__match__`.  This is something that I like a lot and would certainly like to see in pattern matching—alas, just being in favour of something is not a strong enough argument for it. Similar to my votum above: if we are going to discuss this, I think we need new arguments or evidence, something to show why the previous decision to drop it (for now) was wrong.  For that it might be worth reading the respective section in deferred ideas of PEP 622.  Some part of our decision was based on the notion that adding such a custom protocol later one will be relatively painless, but if we were wrong, please speak up and show us what we have overlooked.  If you can present, for instance, some real-world use cases where this feature would enormously benefit sympy (or any other popular package), that would be helpful and a good starting point. Kind regards, Tobias