I'm strongly negative on the idea of sealed classes. I've explained my reasoning in this thread: https://mail.python.org/archives/list/typing-sig@python.org/thread/7TB36OWSW.... As you point out at the top of this thread, there's already a good solution to this problem using unions. This solution is already supported by the Python type system, and it works with all type checkers. --- Eric Traut Contributor to Pyright & Pylance Microsoft

From my perspective, the "sealed class" proposal is a no-go. It violates a key tenet that lazy type evaluators like pyright depend upon for efficient type evaluation. Namely, it requires the type analysis of a full file (or at least all of the classes defined within that file) to discover all of the subclasses of the sealed class. If you pursue the sealed class approach, it's quite likely that pyright and pylance will abstain from implementing it. So my recommendation is to pursue options 2 or 3. -- Eric Traut Contributor to Pyright & Pylance Microsoft

I don’t think this was a threat, I think this was more a statement about how we *cannot* support it with a lazy architecture (and the architecture isn’t going to change given that Pylance is first and foremost a language server that needs to react swiftly to edits). On Thu, Jul 28, 2022 at 9:58 AM Guido van Rossum <guido@python.org> wrote:

I think threatening that a type checker won't implement a feature is not a good way of supporting an argument.

That said, I'm curious to learn more about ADTs / Rust-style enums and what shape they would take in Python.

On Thu, Jul 28, 2022 at 9:32 AM Eric Traut <eric@traut.com> wrote:

...

From my perspective, the "sealed class" proposal is a no-go. It violates a key tenet that lazy type evaluators like pyright depend upon for efficient type evaluation. Namely, it requires the type analysis of a full file (or at least all of the classes defined within that file) to discover all of the subclasses of the sealed class. If you pursue the sealed class approach, it's quite likely that pyright and pylance will abstain from implementing it.

So my recommendation is to pursue options 2 or 3.

-- Eric Traut Contributor to Pyright & Pylance Microsoft _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org

-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...> _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: gram@geekraver.com

@Thomas Grainger, that approach would make things even worse for a lazy (just-in-time) type evaluator. The problem is that when the type analyzer is told that a variable is of type `DeprecationWarning`, it needs to somehow determine that `DeprecationWarning` should be treated as though it is a union of other classes that derive from `DeprecationWarning` for purposes of type narrowing and type exhaustion checking. That's the basic problem we're trying to solve here. With the `@sealed` (decorator) approach, a lazy analyzer would at least know that `DeprecationWarning` is sealed and should be treated specially. It would then need to examine all of the class declarations in the file (perhaps hundreds or thousands in the worst case) to determine which of them derive from the sealed base class. That means the type evaluation performance when using a `@sealed` class could be really bad, but type analysis of non-sealed classes would be unaffected. With the context manager approach, the type analyzer wouldn't even know that `DeprecationWarning` is a sealed class to begin with. It would need to assume that _any_ class is potentially sealed and always analyze all of the classes in a file to determine which ones derive from that class. The fact that some of these class declarations are located within a `with` statement body could be used as an optimization clue, but any class that is declared within a file that has other classes declared within a `with` statement would need to be analyzed as though it is potentially sealed.

Hi, I am one of the people cheering for Rust-style/Enum-style ADTs (although, as some people have pointed out to me, the term 'ADTs' is a little too broad; we're talking about sum types in particular). I just want to point out that one of the main benefits to this approach is the ability to have singletons (like in PEP 345 enums) alongside classes. I'd be sad if a proposal would be accepted that doesn't support this. I think currently you can get close with a union of classes and literals but it's very awkward. The sealed approach feels unnatural for this, no? In an PEP 345 enum, all the singletons automatically inherit from the wrapper class. Could we just do a PEP 345 Enum++ that, in addition to singletons, can include classes that also automatically inherit from the wrapper? In general I think taking PEP 345 a step further would be a natural way to introduce better sum types to Python. When you say you needed sum types for your engine, is https://github.com/Kenny2github/mahjong/blob/57a4cc4cc879dae2c0baa6c40a5458e... what you're referring to?

On Fri, Jul 29, 2022 at 11:05 PM AbyxDev <ken@abyx.dev> wrote:

Is PEP 345 the PEP you intended to refer to? That PEP is about Python package metadata and I don't see any references to "singletons" or "enums" there.

Apologies, I had a brain fart. What I'm trying to refer to is PEP 435 (which ultimately became the standard library enum module), not 345.

I have some clarifying questions for your suggestion, assuming I've understood it right. Given the below example:

class Tree(SumEnum, inherit=True): # or whatever class Leaf: ... class Node(Mixin): ...

1. Are you saying that the would Leaf implicitly become class Leaf(Tree)? 2. Is Node allowed? 3. If so, is it implicitly class Node(Tree, Mixin) or class Node(Mixin, Tree)?

I don't know, let's discuss it.

Given a super simple existing enum: ``` class MyEnum(enum.Enum): A = "a" def test(self) -> str: return str(self) ``` it's already the case that `isinstance(MyEnum.A, MyEnum)` and `MyEnum.A.test() == 'MyEnum.A'` so teeeeechnically (if you squint) there's precedent for the answers to 1) to be 'yes'. I could also see 2) being true too, and for 3) my gut feeling would be `Node(Mixin, Tree)`. But this is just me spitballing; I'd love to hear what other folks would think of this.

My only peeve would be the fact that, by its nature, this would be the only case where inheriting from a class inside its body makes sense.

Back in the original thread, I proposed one more solution [1] to the "can't inherit from a class inside its body". Namely, once the outer class is done building, find each class defined in the body, build a copy of that class with the outer class injected into its bases, and overwrite the old class member with the new class. [1] https://mail.python.org/archives/list/typing-sig@python.org/message/HT2Q6GWS...

The consistent application of good design discipline by Jukka, Guido, and other contributors to the original Python static type system makes it possible today for a type checker to determine the minimal set of types that another type depends upon without analyzing a full source file or set of source files. All of the type features that have been added to the Python type system since then have been designed in a way that retains this ability. Pyright relies on this to deliver good performance for language server features like completion suggestions. It performs the minimal type analysis required to deliver an on-demand answer, usually within 10s of milliseconds. The `@sealed` proposal would be the first type feature that deviates from this design principle and would require all classes in a file to be analyzed before the type of the sealed class could be determined. Pylance, the language server built on top of pyright, is actively used by more than 4M Python developers. The vast majority of those developers do not care about static type checking, but they do care about accurate and fast completion suggestions, signature help, semantic highlighting, semantic search and rename, inlined documentation, and other interactive convenience features that rely on fast type analysis. The `@sealed` proposal has the potential to negatively impact the development experience for this large group of developers while providing a small benefit to a small group of developers. When we’re considering new typing features (or any language features, for that matter), we should take into account both the benefits of a proposal but also the harm it could do, and we should optimize for the greatest good. At least that’s my philosophy and my motivation for pushing back so hard on the `@sealed` proposal. The `__sealed__` or `__all__` proposals address the concern I highlight above because they explicitly list the subclasses associated with the sealed base class. However, I find these proposals to be inelegant and cumbersome. They violate a basic rule of object-oriented programming: a base class should not have knowledge of its subclasses. And in effect, they invent a new (and cumbersome) way to define a union type. To review, here is David’s proposal for `__sealed__`: ```python form __future__ import annotations class Tree: __sealed__: Node | Leaf class Node(Tree): ... class Leaf(Tree): ... ``` This is effectively telling a type checker to treat the symbol `Tree` as a union of `Node` and `Leaf`, but it’s using a new and unfamiliar way to declare this intent. The `__sealed__` example above is equivalent to the following code sample, which doesn’t require any new type features for users to understand or for type checkers to implement. It is no more work to maintain, and it doesn’t require any cumbersome stringified type references. ```python class _TreeBase: ... class Node(_TreeBase): ... class Leaf(_TreeBase): … Tree = Node | Leaf ``` Note that I’ve used an underscore for the name of the base class to indicate that it is a private implementation detail of this file and should not be used on its own. Only the union type (`Tree`) should be used for static typing purposes. The enum-based proposals are also problematic in that they invent a new and unfamiliar way to represent inheritance. They also muddy the concept of an enumeration, which has historically been used for an enumerated set of values, not an enumerate set of types. We already have a mechanism in the type system today to represent an enumerated set of types; it's called a union. Several of you have indicated that the union solution doesn’t meet your needs, but the counterproposals seem to be to be equivalent semantically, and the maintenance burden seems equivalent too. I’m struggling to understand why the union solution doesn't meet your needs but these other proposals do. @AbyxDev said:

Using a Union exclusively is a complete no-go for my use case (I need inheritance), and using a Union + base class is confusing, clunky, and difficult to maintain.

Can you elaborate? The union approach cleanly supports inheritance using the standard inheritance mechanisms. I don’t understand why you think it’s clunky or difficult to maintain. I use this union pattern regularly in my code, and I find it neither clunky nor difficult to maintain. Since the union mechanism is already fully supported in today’s type system, I encourage you to try it for your own use cases. Once you’ve tried it, perhaps you can report back on whether you still think it’s cumbersome and why. The union solution has the added benefit that it allows you to define different subsets of types through the use of different unions. Here’s a practical example from pyright's parser implementation (which is written in TypeScript, but the concepts are the same). https://github.com/microsoft/pyright/blob/74d43236f6fad8300fc4304c970857027b.... Here I define a type called `ParseNode` that is a union of all possible parse node types. All of these subtypes inherit from `ParseNodeBase`. You can see that I also define other unions that include a subset of these parse node types. For example, the `ExpressionNode` union (https://github.com/microsoft/pyright/blob/74d43236f6fad8300fc4304c970857027b...) includes only the parse node types that represent expressions. I find this to be both flexible and easy to maintain. It fully supports type exhaustion, which makes my code robust and immune to an entire class of regressions. Let’s back up and make sure that we understand the problem we’re trying to solve and the requirements that we’ve collected so far. The original problem statement (https://mail.python.org/archives/list/typing-sig@python.org/thread/QSCT2N4RF...) called for a mechanism that supports type exhaustion in if/else towers and match statements. Here are the requirements and desirables that I think we've collected so far: 1. The mechanism must produce a type that supports type exhaustion checks in if/else towers and match statements. 2. The mechanism must support inheritance. 3. The mechanism must not require full analysis of a source file to determine the type of a single class. 4. Preferably, the code should be easy to maintain — in particular, if new subclasses are added, they should either be automatically discovered or it should be possible for a type checker to guide the developer to do what is needed. 5. Preferably, the solution should not invent new and unfamiliar ways to express existing concepts in Python like inheritance or union types. Am I missing anything here? -- Eric traut Contributor to Pyright & Pylance Microsoft

[It is] possible today for a type checker to determine the minimal set of types that another type depends upon without analyzing a full source file or set of source files

I will admit that the way sealed works is weird. And I'll take your word for it that pyright can understand a type without processing the file in which that type is defined. That's actually really cool and not something I would expect to be possible with Python.

Only the union type (`Tree`) should be used for static typing purposes.

With your design in pyright, does autocompleting a variable of type `Tree` suggest just methods defined on `_TreeBase`? I ask because I get all the methods of `Node` and `Leaf` in PyCharm, which does not happen when autocompleting a variable of type `_TreeBase`. Theoretically, this is resolvable by finding the common base types of `Node` and `Leaf` (i.e. `_TreeBase`), but making it easier for type checkers to understand this is one of the motivations of mine in pursuing this feature.

Since the union mechanism is already fully supported in today’s type system, I encourage you to try it for your own use cases. Once you’ve tried it, perhaps you can report back on whether you still think it’s cumbersome and why.

The Union type can be made to work for static analysis. But there is no rescuing its total absence of runtime capabilities. Python could be fixed so that Union types work as cases in match, but there is no fixing `Tree` so that it exposes the class methods of `_TreeBase`. For example, I use a serialization library that inspects the type annotations to automatically generator serializers. If I use `Tree`, there is no way to customize the serialization of a `Tree` because there is no where to put the class methods that the serialization library calls.

Mehdi2277 wrote:

...

The Union type can be made to work for static analysis. But there is no rescuing its total absence of runtime capabilities. Python could be fixed so that Union types work as cases in match, but there is no fixing `Tree` so that it exposes the class methods of `_TreeBase`. For example, I use a serialization library that inspects the type annotations to automatically generator serializers. If I use `Tree`, there is no way to customize the serialization of a `Tree` because there is no where to put the class methods that the serialization library calls.

This sounds like an implementation detail of the serialization library you use. I also heavily use a runtime serialization library based on types. It's approach is every type defines a serializer + deserializer. Basic types have parsers defined in the library. Common type constructs like Union/dataclass/namedtuple automatically derive a parser too. But all parsers for types are overridable. In the library I use you can do, [workaround]

I think this reflects the motivation of the pro-sealed and pro-enum camps pretty well. We have some base classes that we would like to use with match. The only current solution is to hide the base class and replace it with a Union. Ok, that makes `match` work, but Tree.to_data and Tree.from_data are now gone. Well, I guess there is a workaround where I strip the methods off and register them with the serialization library like Tree is a builtin type. I hope it is understandable why this some people might consider this a very unsatisfactory solution.

...

With your design in pyright, does autocompleting a variable of type `Tree` suggest just methods defined on `_TreeBase`? I ask because I get all the methods of `Node` and `Leaf` in PyCharm, which does not happen when autocompleting a variable of type `_TreeBase`. Theoretically, this is resolvable by finding the common base types of `Node` and `Leaf` (i.e. `_TreeBase`),

Surely PyCharm is correct here and this is not a problem to be resolved?

If your variable is either a Node or a Leaf (and which is not known until runtime), then the list of possible methods while editing has to be taken from all the methods of Node plus all the methods of Lead, and not just the methods they have in common inherited from _TreeBase.

If the type system is sound, the methods on Leaf are not valid to call on an instance of Tree. Even though a Tree could be a Leaf, it might instead be a Node, which would be an error.

On Sat, Jul 30, 2022, 12:52 PM Eric Traut <eric@traut.com> wrote:

The `@sealed` proposal would be the first type feature that deviates from this design principle and would require all classes in a file to be analyzed before the type of the sealed class could be determined.

Which is why I agree that the decorator approach isn't ideal, though I would find it convenient. However, see below for why I don't think all classes need to be analyzed. Pylance, the language server built on top of pyright, is actively used by

more than 4M Python developers. The vast majority of those developers do not care about static type checking, but they do care about accurate and fast completion suggestions, signature help, semantic highlighting, semantic search and rename, inlined documentation, and other interactive convenience features that rely on fast type analysis. The `@sealed` proposal has the potential to negatively impact the development experience for this large group of developers while providing a small benefit to a small group of developers.

I don't see how any sealed class proposal (decorator or not) negatively impacts the audience you describe. The vast majority of developers do not care about static type checking, so in the vast majority of cases, they would never encounter sealed classes, no? Or, if you consider someone using a library that contains sealed classes, to use the example of my library, if the dev is not performing static type checking, no lookup of all subclasses within the file needs to be made. Autocompletion in each rung of the if-else/match-case isinstance() ladder would easily be able to look up methods from the specific subclass being matched; autocompletion outside the ladder would use the base class. Finding the type of one of the subclasses works as normal; you don't need to know what other subclasses there are to know what the parent of a class is. The only case where all subclasses must be looked up, as far as I can think of off the top of my head, is for an exhaustiveness check at the end of the ladder (not performed if not typechecking, which is the majority of cases), or if revealing the type of the argument to each isinstance() call in the ladder. If that last case is slower, what of it? It's rare, and I have faith that the Pylance/Pyright devs can keep the lookup time unnoticeable. The `__sealed__` or `__all__` proposals address the concern I highlight

above because they explicitly list the subclasses associated with the sealed base class. However, I find these proposals to be inelegant and cumbersome. They violate a basic rule of object-oriented programming: a base class should not have knowledge of its subclasses.

Then why has Java implemented it in precisely this way? That is, by listing the names of the allowed subclasses. (Not a challenge, I just know little about Java's design philosophies. However, I've typically considered Java a very strongly and traditionally object-oriented language.) This is effectively telling a type checker to treat the symbol `Tree` as a

union of `Node` and `Leaf`,

Minor correction: Under my scheme, if Tree is not an ABC, Tree becomes a union of Node, Leaf, and an invariant version of Tree itself, since instantiating Tree directly is allowed. You are entirely correct if Tree is in fact an ABC, or if consensus decides that all sealed classes are automatically abstract. Note that I’ve used an underscore for the name of the base class to

indicate that it is a private implementation detail of this file and should not be used on its own. Only the union type (`Tree`) should be used for static typing purposes.

How do you propose to enable isinstance() checks on Tree in versions of Python that don't accept unions as a class argument, if you intend the real base class (the usual argument of such checks) to be private? The enum-based proposals are also problematic in that they invent a new and

unfamiliar way to represent inheritance.

Yeah, this is essentially what I meant in my last long email when I said "Potentially unnatural to Python devs". They also muddy the concept of an enumeration, which has historically been

used for an enumerated set of values, not an enumerate set of types.

This is why if enums are how we do this, I'm more in favor of making a new Enum subclass that additionally modifies member types, rather than adding the functionality to Enum itself. That would keep the concept of an enumeration the same, while introducing ADTs as a new thing that is "like" an enumeration. I’m struggling to understand why the union solution doesn't meet your needs

but these other proposals do.

That's fair; this is a hairy topic and I'm having probably about as much trouble explaining as you are understanding. @AbyxDev said:

...

- snip -

Can you elaborate? The union approach cleanly supports inheritance using

the standard inheritance mechanisms.

The Union + base class approach supports inheritance, though I would not say it does so "cleanly" (more on this later). When I said "I need inheritance" I was intending to discount the approach of defining multiple classes with no common base class and using only a Union to unify them. Since the union mechanism is already fully supported in today’s type

system, I encourage you to try it for your own use cases. Once you’ve tried it, perhaps you can report back on whether you still think it’s cumbersome and why.

You must have missed the part in my first email to this list that mentioned I was already using the Union + base class approach in my code. I wrote my initial email after finding that approach less than ideal.

I don’t understand why you think it’s clunky or difficult to maintain. I

use this union pattern regularly in my code, and I find it neither clunky nor difficult to maintain.

Let me try again to explain, because evidently I'm doing a poor job of it. The clunkiness arises from the fundamental fact that this method uses two things to represent one. If you use the wrong member of the pair, you end up with non-exhaustiveness errors that are difficult to explain (if you use the base class instead of the union), or actual runtime errors (if you try to check instancehood with a union in the currently most widespread versions of Python). There's also the unexplainable subjective measure of "sealed classes come more naturally; union + base class feels more like a recipe to be memorized". It's difficult to maintain mostly because of the relisting of subclasses, which makes it easy to forget to sync the list with the actual subclasses, but also because the name duality makes refactoring somewhat more tedious (e.g. if you rename the type, you now have two things to rename, not one; you can't avoid renaming the implementation base class by hiding it because users need something to isinstance() against).

The union solution has the added benefit that it allows you to define

different subsets of types through the use of different unions.

Here’s a practical example from pyright's parser implementation (which is

written in TypeScript, but the concepts are the same). https://github.com/microsoft/pyright/blob/74d43236f6fad8300fc4304c970857027b.... Here I define a type called `ParseNode` that is a union of all possible parse node types. All of these subtypes inherit from `ParseNodeBase`. You can see that I also define other unions that include a subset of these parse node types. For example, the `ExpressionNode` union ( https://github.com/microsoft/pyright/blob/74d43236f6fad8300fc4304c970857027b...) includes only the parse node types that represent expressions.

This could just as easily be done with a sealed ParseNode ABC, which the sealed ExpressionNode ABC inherits, which the relevant individual parse node classes then inherit. ParseNode would then be treated like a union of all parse node types, and ExpressionNode would be treated like a union of expression node types, without having to relist the node types, rename both ParseNode and ParseNodeBase if the need arises to rename one of them, or remind oneself of when to use the base class vs the union. Here are the requirements and desirables that I think we've collected so

far:

1. The mechanism must produce a type that supports type exhaustion checks

in if/else towers and match statements.

Yep.

2. The mechanism must support inheritance.

Yep.

3. The mechanism must not require full analysis of a source file to determine the type of a single class.

As described above, I doubt that even the @sealed proposal would fail this requirement. I also think this is more of a desirable, since as I mentioned, cases where all subclasses actually do have to be known for some type operation should be few and far between. 4. Preferably, the code should be easy to maintain — in particular, if new

subclasses are added, they should either be automatically discovered or it should be possible for a type checker to guide the developer to do what is needed.

Yep. Though the latter is not ideal, but it's still acceptable. 5. Preferably, the solution should not invent new and unfamiliar ways to

express existing concepts in Python like inheritance or union types.

Personally agreed, though I feel like the Enum camp would have something to say about the advantages of combining enum singletons with enum classes, as seen in Rust and other languages. Am I missing anything here?

I think following on from the previous statement, 6. Preferably, the mechanism should be familiar to users of languages with existing implementations of sealed classes. Definitely a nice-to-have and not a strict requirement though, given that Python's syntax is almost never familiar to users of other languages anyway. Otherwise, no additional requirements immediately come to mind. Hopefully I've clarified myself better! I spent over an hour writing this on a plane so forgive any typographical mistakes or accidental slights you may perceive. I assure you I respect your expertise and experience. I'll be attending the upcoming typing meetup if you'd like to discuss in real-time. Best regards, AbyxDev; https://abyx.dev

On Sat, Jul 30, 2022 at 9:50 PM Guido van Rossum <guido@python.org> wrote:

I’m sorry, folks. I find the current “he-said-she-said” form of the discussion hard to follow. Maybe we can pick a concrete example, show what it looks like in each proposal (including problems), and then discuss the pros and cons of each version?

Sure, I've compiled a gist of whatever pros and cons I can think of. I did a gist to have full formatting control - hope that's acceptable. https://gist.github.com/Kenny2github/3f475e48ef4ef750359f71e22c158f9a Anyone, please feel free to note items I've missed and I'll do my best to update the gist with your additions. (I'm sure there's a better venue for this, but this is what I could come up with on the spur of the moment I had the idea.) -- Best regards, AbyxDev; https://abyx.dev