On Tue, Feb 23, 2021 at 3:49 PM Damian Shaw <damian.peter.shaw@gmail.com> wrote:

Firstly, if I have a library which supports multiple versions of Python and I need to catch all standard exceptions, what is considered the best practise after this PEP is introduced?

Currently I might have code like this right now: try: ... # Code except Exception as e: ... # Logic to handle exception

Hi Damian, Catching all exceptions in this way is not a common pattern. Typically you have an idea what kind of exceptions you expect to get from an operation, and which of those exceptions you are interested in handling. Most of your try/except blocks will be targeted, like handling KeyError from d[k], an operation that will never raise an ExceptionGroup. ExceptionGroups will only be raised by specific APIs which will advertise themselves as such. We don't expect that there will be a need for blanket migrations of "except T" to "except (T, ExceptionGroup)" to "except *T". Irit

Hi Damian, While I agree that there are a handful of use cases for catching all Exceptions, if you look at some of the 5 million hits you found in github, the vast majority are not such cases. They are mostly try/except blocks that wrap a particular operation (a dict access, a raw_input call, etc). You don't want to automatically translate "except Exception" to something that involves ExceptionGroups, because most of the time it is not needed -- the code in the body can't raise ExceptionGroups. Your first email on this thread showed that you were able to quite quickly, just from what's already in the PEP, think of how to do what you need for your valid "except Exception" use case. My feeling is that people who need it will be able to do it quite easily. I'm more worried about people assuming that they need it when they don't. Irit On Tue, Feb 23, 2021 at 8:00 PM Damian Shaw <damian.peter.shaw@gmail.com> wrote:

Hi Irit,

Catching exceptions like this is an extremely common pattern in the real world, e.g. this pattern has over 5 million GitHub matches: https://github.com/search?l=&q=%22except+Exception%22+language%3APython&type=code

How common it is aside there are also many valid use cases for this pattern, e.g. logging a final unhandled exception of a script, catching exceptions inside an orchestration framework, exploring code where the list of exceptions is unknown, etc.

A description from the PEP on how to handle this kind of pattern, especially for code that must support multiple versions of Python, would be extremely helpful.

Damian

...

On Tue, Feb 23, 2021 at 2:27 PM Damian Shaw <damian.peter.shaw@gmail.com> wrote:

A common example I see from the GitHub search is a catch all exception on some third party API. If the user wants to catch all exceptions from calling a third party API why would they want to let an ExceptionGroup go unhandled in the future? Once ExceptionGroups are introduced then libraries will surely implement them where appropriate and users would want to continue having their board exception handling working, would they not?

See my response to Ethan -- I think libraries should think twice before raising ExceptionGroup. Note that ExceptionGroup is a subclass of BaseException, so if you're catching the latter you don't have to do anything special. OTOH we might reconsider deriving ExceptionGroup from BaseException -- maybe it's better to inherit from Exception? I don't think the PEP currently has a clear reason why it must derive from BaseException. I believe it has to do with the possibility that ExceptionGroup might wrap a BaseException instance (e.g. KeyboardInterrupt or SystemExit). But I believe that this was more important in an earlier revision of the design. Note that there is a section labeled "Should MultiError inherit from BaseException, or from Exception? (Or something else?)" in Trio's MultiError v2 issue (https://github.com/python-trio/trio/issues/611). I think the key argument here is "However, if MultiError is an Exception, then it can only contain Exception subclasses, because otherwise a MultiError(KeyboardInterrupt) could get caught by an except Exception handler, which would be bad." But those notes are from back in 2018, not from the 2020 core dev sprint when we got much closer to the current design. So I'm not sure the reasoning still applies. (Yury? Can you help us out here?)

I don't actually know if the code I wrote in the first email is correct though, would it catch all standard Exception / ExceptionGroups across Python 3.x if this PEP was implemented in the future? And would it be best practice for this pattern? If so it seems it would require many users to rewrite their code as it adds boilerplate and nuance for a pattern that is a relatively simple and very commonly used right now.

I guess summarized my commentary is please don't dismiss this pattern out of hand, which this PEP currently disrupts, it is used widely in Python right now and has many valid use cases. I don't have anything further to add so I won't continue this discussion and I assume anything further you have to add to be considered and more informed than my own opinion.

I expect that if you want to do a good job of emulating `except *ValueError` in Python versions that don't have ExceptionGroup you're going to have to write more code. It's relatively straightforward if you just want to ignore or log the errors collectively, but it'll be more complex if you want to handle each suberror separately, because the PEP-654-aware code will have to iterate over the errors, recursing into ExceptionGroup instances. (Irit, maybe the PEP could use a worked-out example of how to do the latter -- even with `except *` available it seems non-trivial.) Thanks,

Damian

Thanks Damian for getting us to think more about 'except Exception:' -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

By the time a KeyboardInterrupt or SystemExit is being grouped into an ExceptionGroup, it already isn't being treated as an immediate interruption ... it has (presumably) already killed its own execution path, but it should not kill the program as a whole. Whether ExceptionGroup inherits from Exception or BaseException shouldn't matter for deciding what it can group. -jJ

On Thu, Feb 25, 2021 at 5:59 AM Guido van Rossum <guido@python.org> wrote:

Here's a potentially alternative plan, which is also complex, but doesn't require asyncio or other use cases to define special classes. Let's define two exceptions, BaseExceptionGroup which wraps BaseException instances, and ExceptionGroup which only wraps Exception instances. (Names to be bikeshedded.) They could share a constructor (always invoked via BaseExceptionGroup) which chooses the right class depending on whether there are any non-Exception instances being wrapped -- this would do the right thing for split() and subgroup() and re-raising unhandled exceptions.

Then 'except Exception:' would catch ExceptionGroup but not BaseExceptionGroup, so if a group wraps e.g. KeyboardError it wouldn't be caught (even if there's also e.g. a ValueError among the wrapped errors).

Pseudo-code:

class BaseExceptionGroup(BaseException): def __new__(cls, msg, errors): if cls is BaseExceptionGroup and all(isinstance(e, Exception) for e in errors): cls = ExceptionGroup return BaseException.__new__(cls, msg, errors)

class ExceptionGroup(Exception, BaseExceptionGroup): pass

This could be a valid compromise. We keep the ability to wrap any exception, while we lose the "fail-fast if you forget to handle an ExceptionGroup" feature, which was intended as a kindness towards those who abuse "except Exception". If we adopt this solution then letting an ExceptionGroup escape from code that is not supposed to raise it, is not a fatal error, it's just some exception like any other. So there is no longer a distinction between code that raises ExceptionGroups and code that doesn't. Any code can propagate them, like any code can raise any other exception. Does this mean that more code needs to be aware of the possibility of them showing up? Is that a problem? Maybe this a simpler state of affairs overall. What would we have done here if we were building Python from scratch? Irit

We don't call it out but we talked about this at the sprint. I think the reason we didn't come up with this solution then is that at the time ExceptionGroups in our plans were not immutable, so this would not have been possible (you don't know at construction time what it is going to contain). On Thu, Feb 25, 2021 at 10:08 PM Guido van Rossum <guido@python.org> wrote:

So is "fail-fast if you forget to handle an ExceptionGroup" really a feature? (Do we call this out in the PEP?)

We may believe that "except Exception" is an abuse, but it is too common to dismiss out of hand. I think if some app has e.g. a main loop where they repeatedly do something that may fail in many ways (e.g. handle a web request), catch all errors and then just log the error and continue from the top, it's a better experience if it logs "ExceptionGroup: <message> [<list of subexceptions>]" than if it crashes.

There's also code that catches Exception, logs the error and some relevant data, and then re-raises it. The logged error may go to a more specialized destination than the generic traceback, and the log message may contain additional data that's only available in that stack frame.

So I think there are enough serious use cases that we should do what's best for those use cases, and not try to lecture users about abuse of the idiom.

I don't know what we would have done if we were building Python from scratch. Possibly we would not have BaseException at all, and the whole mess would go away. (But there are some good reasons why we introduced BaseException, so I don't know that that would really be a better overall experience.)

--Guido

On Fri, Feb 26, 2021 at 2:00 AM Guido van Rossum <guido@python.org> wrote:

Then we should get some more opinions on this. I think it's the best idea so far for kindness towards code using `except Exception:`.

I agree that it's the best idea so far. If someone says 'except Exception' they better mean it because we'll believe them, and if someone forgets to handle an ExceptionGroup then they have a bug and that's how it is.

OT: Is ExceptionGroup *really* immutable in the current implementation? As long as the 'errors' field is a list, I think one could mutate the list directly.

It's not, but we were going to make it an immutable tuple.

Which brings me to the question, do you have a branch that matches the PEP yet?

The implementation matches the PEP less and less every day because the PEP is developing faster than the implementation. But these aren't differences that are more than a technicality to fix (rename things, move an error from runtime to the parser, things like that). The except* implementation is probably pretty close to the PEP because it's the most recent bit.

On 26Feb2021 02:44, Irit Katriel <iritkatriel@googlemail.com> wrote:

On Fri, Feb 26, 2021 at 2:00 AM Guido van Rossum <guido@python.org> wrote:

...

OT: Is ExceptionGroup *really* immutable in the current implementation? As long as the 'errors' field is a list, I think one could mutate the list directly.

It's not, but we were going to make it an immutable tuple.

Could you say why? Other than wanting to discourage mutation happy code getting out there? The reason I ask is that the scenario which comes to my mind is something like: except *OSError as e: AIUI "e" is an ExceptionGroup containing only OSErrors. So, one common thing in my own code is this: try: do something with a file except OSError as e: if e.errno == ENOENT: # file is missing, but that is ok # because we treat it like an empty file elif ... some other ok situation ... else: raise My natural inclination with an ExceptionGroup would be to winnow the OSErrors I'm handed, and push the _unhandled_ errors back into the original ExceptionGroup. That way, after the various except* clauses, a nonempty ExceptionGroup would remain with the unhandled errors, and it might perhaps be reraised then. Cheers, Cameron Simpson <cs@cskk.id.au>

In earlier versions of the PEP ExceptionGroup was not immutable and split actually removed matching exceptions from it. It was also iterable so you could get a flat list of all the contained leaf exceptions. Then we changed it. ExceptionGroup is a tree of exceptions, and the internal nodes of the tree (which are ExceptionGroups) have metadata on them - context, cause, traceback. If you want the full traceback of a leaf exception you need to concatenate the tracebacks of all the exceptions on the path from the root to the leaf. If you flatten an ExceptionGroup and create a new list ExceptionGroup(list(eg)) you don't lose metadata (unless eg's tree has depth 1). Similarly, if you add an exception it needs to have matching metadata to the ExceptionGroup you are adding it to. It's too easy to get that wrong. split() and subgroup() take care to preserve the correct metadata on all the internal nodes, and if you just use them you only make safe operations. This is why I am hesitating to add iteration utilities to the API. Like we did, people will naturally try that first, and it's not the safest API. We actually have the OSErrors example in the PEP, just above https://www.python.org/dev/peps/pep-0654/#caught-exception-objects: try: low_level_os_operation() except *OSerror as errors: raise errors.subgroup(lambda e: e.errno != errno.EPIPE) from None On Sun, Feb 28, 2021 at 6:30 AM Guido van Rossum <guido@python.org> wrote:

We really don’t want users pushing non-exceptions into the list, nor do we want e.g. KeyboardInterrupt to be added to a (non-Base-) ExceptionGroup.

There’s a pattern for what you propose using the split() method and a lambda, or you can keep the exceptions in a list and re-wrap them at the end.

On Sat, Feb 27, 2021 at 20:41 Cameron Simpson <cs@cskk.id.au> wrote:

...

On 26Feb2021 02:44, Irit Katriel <iritkatriel@googlemail.com> wrote:

...

On Fri, Feb 26, 2021 at 2:00 AM Guido van Rossum <guido@python.org> wrote:

...

OT: Is ExceptionGroup *really* immutable in the current implementation? As long as the 'errors' field is a list, I think one could mutate the list directly.

It's not, but we were going to make it an immutable tuple.

Could you say why? Other than wanting to discourage mutation happy code getting out there?

The reason I ask is that the scenario which comes to my mind is something like:

except *OSError as e:

AIUI "e" is an ExceptionGroup containing only OSErrors. So, one common thing in my own code is this:

try: do something with a file except OSError as e: if e.errno == ENOENT: # file is missing, but that is ok # because we treat it like an empty file elif ... some other ok situation ... else: raise

My natural inclination with an ExceptionGroup would be to winnow the OSErrors I'm handed, and push the _unhandled_ errors back into the original ExceptionGroup. That way, after the various except* clauses, a nonempty ExceptionGroup would remain with the unhandled errors, and it might perhaps be reraised then.

Cheers, Cameron Simpson <cs@cskk.id.au>

-- --Guido (mobile)

On Sun, Feb 28, 2021 at 6:17 PM Guido van Rossum <guido@python.org> wrote:

On Sun, Feb 28, 2021 at 2:40 AM Irit Katriel <iritkatriel@googlemail.com> wrote:

...

In earlier versions of the PEP ExceptionGroup was not immutable and split actually removed matching exceptions from it. It was also iterable so you could get a flat list of all the contained leaf exceptions. Then we changed it.

ExceptionGroup is a tree of exceptions, and the internal nodes of the tree (which are ExceptionGroups) have metadata on them - context, cause, traceback. If you want the full traceback of a leaf exception you need to concatenate the tracebacks of all the exceptions on the path from the root to the leaf. If you flatten an ExceptionGroup and create a new list ExceptionGroup(list(eg)) you don't lose metadata (unless eg's tree has depth 1).

Is this a typo? Did you mean "If you flatten [it] and create a new list... you *do* lose metadata (unless ... depth 1)"?

It is a typo, thanks.

You can pass an arbitrary function to eg.split() and eg.subgroup(), and that function can have a side effect. Suppose you want to log and ignore OSError with errno==ENOENT but re-raise the rest, you can do this: ``` def log_and_ignore(err): if err.errno == ENOENT: log(err) return False else: return True try: . . . except *OSError as eg: eg = eg.subgroup(log_and_ignore) if eg is not None: raise eg ``` On Sun, Feb 28, 2021 at 10:35 AM Jim J. Jewett <jimjjewett@gmail.com> wrote:

Looking at the following PEP example, I'm still not sure what he should do to handle some but not all OSError instances:

... raise ExceptionGroup( ... "eg", ... [ ... ValueError(1), ... TypeError(2), ... OSError(3), ... ExceptionGroup( ... "nested", ... [OSError(4), TypeError(5), ValueError(6)]) ... ]

except *OSError seems to get him an ExceptionGroup that he still has to manually tree-walk to handle the OSErrors that he can actually handle.

Once he has manually tree-walked, he has to manually re-raise the other OSErrors that he hasn't filtered out, and they become a sibling of the original ExceptionGroup, because they're (now) being raised at a different location. (Or, not *really* the original ExceptionGroup, but one that looks like it without any OSErrors.) This new sibling relationship is there to show that a failed or incomplete attempt was made at resolving the OSErrors, but no such attempt was made at the ValueErrors or TypeErrors.

Alternatively, he *might* be able to just grab Exception, then use a complicated callback to subgroup only the OSErrors that he will be able to handle, and raise the complement of that. It will still add another layer of ExceptionGroup, but won't split the remainders.

This seems like an awful lot of work to preserve and elaborate structural relations between different exceptions that were originally accidental. (I am assuming that the original relation was "these all happened, probably in different execution threads" and the new relations are just accidents of what got partially handled along the way, or maybe the order in which different execution threads got merged.)

I can't imagine ever needing to care about the full tree structure unless I'm re-implementing asycio/trio/curio. Even if that structural information is important to my own framework, that just means I should create more intermediate tasks to handle it more quickly, before the next level of aggregation.

I would much prefer an API that let me pattern-match (even just by class of the exception instances, though preferably also by attributes), process each exception (as though it were the only one) within the code block that match selects, and indicate whether that one exception should be raised further or not.

-jJ _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/WTZUGBFW... Code of Conduct: http://python.org/psf/codeofconduct/

-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

Hi Cameron, If you go long, I go longer :) On Sun, Feb 28, 2021 at 10:51 PM Cameron Simpson <cs@cskk.id.au> wrote:

On 28Feb2021 10:40, Irit Katriel <iritkatriel@googlemail.com> wrote:

...

split() and subgroup() take care to preserve the correct metadata on all the internal nodes, and if you just use them you only make safe operations. This is why I am hesitating to add iteration utilities to the API. Like we did, people will naturally try that first, and it's not the safest API.

Wouldn't it be safe if the ExceptionGroup were immutable, as you plan? Or have you another hazard in mind?

Making them immutable won't help the metadata issue. split() and subgroup() copy the (context, cause traceback) from the original ExceptionGroups (root and internal nodes of the tree) to the result trees. If you DIY creating new ExceptionGroups you need to take care of that.

But all that said, being able to iterable the subexceptions seems a natural way to manage that:

unhandled = [] try: ......... except *OSError as eg: for e in eg: if an ok exception: handle it else: unhandled.append(e) if unhandled: raise ExceptionGroup("unhandled", unhandled)

You just lost the metadata of eg. It has no context, cause and its traceback begins here. And the exceptions contained in it, if they came from a deeper tree that you flattened into the list, now look like their traceback jumps straight to here from the place they were actually first inserted into an ExceptionGroup. This may well be an impossible code path. Here's an example:

...

...

import traceback

def flatten(exc): ... if isinstance(exc, ExceptionGroup): ... for e in exc.errors: ... yield from flatten(e) ... else: ... yield exc ... def f(): ... try: ... raise ValueError(42) ... except ValueError as e: ... return e ... def g(): ... try: ... raise ExceptionGroup("g", [f()]) ... except ExceptionGroup as e: ... return e ... def h(): ... try: ... raise ExceptionGroup("h", [g()]) ... except ExceptionGroup as e: ... return e ... def flat_h(): ... try: ... raise ExceptionGroup("flat_h", list(flatten(h()))) ... except ExceptionGroup as e: ... return e ...

traceback.print_exception(h()) Traceback (most recent call last): File "<stdin>", line 3, in h ExceptionGroup: h

Traceback (most recent call last): File "<stdin>", line 3, in g ExceptionGroup: g ------------------------------------------------------------ Traceback (most recent call last): File "<stdin>", line 3, in f ValueError: 42

...

...

traceback.print_exception(flat_h()) Traceback (most recent call last): File "<stdin>", line 3, in flat_h ExceptionGroup: flat_h

Traceback (most recent call last): File "<stdin>", line 3, in f ValueError: 42

...

...

traceback.print_exception(h()) prints a reasonable traceback - h() called g() called f(). But according to traceback.print_exception(flat_h()), flat_h() called f(). You can preserve the metadata (and the nested structure with all its metadata) if you replace the last line with: raise eg.subgroup(lambda e: e in unhandled) And for the part before that, iteration, Guido's pattern showed that you can roll it into the subgroup callback.

There are some immediate shortcomings above. In particular, I have no way of referencing the original ExceptionGroup without surprisingly cumbersome:

try: ......... except ExceptionGroup as eg0: unhandled = [] eg, os_eg = eg0.split(OSError) if os_eg: for e in os_eg: if an ok exception: handle it else: unhandled.append(e) if eg: eg, value_eg = eg.split(ValueError) if value_eg: for e in value_eg: if some_ValueError_we_understand: handle it else: unhandled.append(e) if eg: unhandled.append(eg) if unhandled: raise ExceptionGroup("unhandled", unhandled) from eg0

This is where except* can help: try: ......... except except *OSError as eg: unhandled = [] handled, unhandled = eg.split(lambda e: e is an ok exception) # with side effect to handle e if unhandled: raise unhandled except *ValueError as eg: handled, unhandled = eg.split(lambda e: e is a value error we understand) # with side effect to handle e if unhandled: I have the following concerns with the pattern above:

There's no way to make a _new_ ExceptionGroup with the same __cause__ and __context__ and message as the original group: not that I can't assign to these, but I'd need to enuerate them; what if the exceptions grew new attributes I wanted to replicate?

As I said, split() and subgroup() do that for you.

This cries out for another factory method like .subgroup but which makes a new ExceptionGroup from an original group, containing a new sequence of exceptions but the same message and coontext. Example:

unhandled_eg = eg0.with_exceptions(unhandled)

Why same message and context but not same cause and traceback?

I don't see a documented way to access the group's message.

In the PEP: "The ExceptionGroup class exposes these parameters in the fields message and errors". (In my implementation it's still msg and not message, I need to change that).

I'm quite unhappy about .subgroup (and presumably .split) returning None when the group is empty. The requires me to have the gratuitous "if eg:" and "if value_eg:" if-statements in the example above.

If, instead, ExceptionGroups were like any other container I could just test if they were empty:

if eg:

_and_ even if they were empty, iterate over them. Who cares if the loop iterates zero times? The example code would become:

try: ......... except ExceptionGroup as eg0: unhandled = [] eg, os_eg = eg0.split(OSError) for e in os_eg: if an ok exception: handle it else: unhandled.append(e) eg, value_eg = eg.split(ValueError) for e in value_eg: if some_ValueError_we_understand: handle it else: unhandled.append(e) if eg: unhandled.append(eg) if unhandled: raise eg0.with_exceptions(unhandled)

You'll note that "except*" is not useful in this pattern.

I think it is useful, see above.

However...

If a subgroup had a reference to its parent this gets cleaner again:

unhandled = [] eg0 = None try: ......... except* OSError as os_eg: eg0 = os_eg.__original__ # or __parent__ or something for e in os_eg: if an ok exception: handle it else: except* ValueError as value_eg: eg0 = os_eg.__original__ # or __parent__ or something for e in value_eg: if some_ValueError_we_understand: handle it else: unhandled.append(e) except* Exception as eg: eg0 = os_eg.__original__ # or __parent__ or something unhandled.extend(eg) if unhandled: raise eg0.with_exceptions(unhandled)

Except that here I have no way to get "eg0", the original ExceptionGroup, for the final raise without the additional .__original__ attribute.

Anyway, I'm strongly of the opinion that ExceptionGroups should look like containers, be iterable, be truthy/falsey based on empty/nonempty and that .split and .subgroup should return empty subgroups instead of None.

This would be true if iteration was a useful pattern for working with ExceptionGroup, but I still think subgroup/split is a better tool in most cases.

Cheers, Cameron Simpson <cs@cskk.id.au>

I'm trying to shorten this again... On Sun, Feb 28, 2021 at 5:54 PM Cameron Simpson <cs@cskk.id.au> wrote:

Let's turn this on its head:

- what specific harm comes from giving EGs container truthiness for size testing?

For something that's not a pure container, this is an anti-pattern. No other subclass of BaseException (that I know of) can ever be falsey, so existing code would be forgiven to write "if e:" where "if e is not None:" was meant.

- what specific harm comes from returning an empty EG from split on no match instead of None?

It would be an exception. What is raising an empty EG supposed to do?

- what specific harm comes from supporting iteration with a caveat about metadata in the docstring, and maybe a recommendation of subgroup?

Iteration is such an easy-to-use API that it would become an attractive nuisance -- people would use it regardless of whether it's the right tool for the job.

- if I wanted to override subgroup and split to not return None, is that even possible with the current ? i.e. can I make a clean metadata preserved EG from an empty list? For example:

eg2 = eg.subgroup(lambda e: False)

Does that get me None, or an empty group? If the latter, I can roll my own subclass with my desired features. If not, I can't AFAICT.

It returns None. Writing a subclass that behaves differently than the base class will just confuse users who aren't aware of your overrides.

EGs _have_ a .errors attribute which has all these aspects, why not expand it to the class as a whole?

They are different. The `errors` attribute may contain other EGs. But presumably iteration would recurse into those so as to flatten the tree and yield only leaf nodes. The attribute is necessary so it is possible to write traversal functions (including that iterator you so desperately want -- as a helper function).

You seem very happen to implement 80% of what I want using callbacks (lambda e: ...), but I find explicit iteration much easier to read. I rarely use filter() for example, and often prefer a generator expression of list comprehension.

Our (the PEP authors') belief is that in most cases users are better off not handling the leaf exceptions. If you want to handle e.g. KeyError, the place to do that is in the try/except clause immediately surrounding the offending `__getitem__` operation (which will never raise an EG), not several stack frames below, where you're potentially dealing with an aggregate of many KeyErrors, or perhaps a KeyError and a TypeError. At that point the most common option is to ignore or log some or all exception types. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

On Thu, Feb 25, 2021 at 2:13 PM Guido van Rossum <guido@python.org> wrote:

So is "fail-fast if you forget to handle an ExceptionGroup" really a feature? (Do we call this out in the PEP?)

We may believe that "except Exception" is an abuse, but it is too common to dismiss out of hand. I think if some app has e.g. a main loop where they repeatedly do something that may fail in many ways (e.g. handle a web request), catch all errors and then just log the error and continue from the top, it's a better experience if it logs "ExceptionGroup: <message> [<list of subexceptions>]" than if it crashes.

Yeah, 'except Exception' happens a lot in the wild, and what to do about that has been a major sticking point in the ExceptionGroup debates all along. I wouldn't say that 'except Exception' is an abuse even -- what do you want gunicorn to do if your buggy flask app raises some random exception? Crash your entire web server, or log it and attempt to keep going? (This is almost your example, but adding in the part where gunicorn is reliable and well-respected, and that its whole job is to invoke arbitrarily flaky code written by random users.) Yury/I/others did discuss the idea of a BaseExceptionGroup/ExceptionGroup split a lot, and I think the general feeling is that it could potentially work, but feels like a complicated and awkward hack, so no-one was super excited about it. For a while we also had a compromise design where only BaseExceptionGroup was built-in, but we left it non-final specifically so asyncio could define an ExceptionsOnlyExceptionGroup. Another somewhat-related awkward part of the API is how ExceptionGroup and plain-old 'except' should interact *in general*. The intuition is that if you have 'except ValueError' and you get an 'ExceptionGroup(ValueError)', then the user's code has some kind of problem and we should probably do.... something? to let them know? One idea I had was that we should raise a RuntimeError if this happens, sort of similar to PEP 479. But I could never quite figure out how this would help (gunicorn crashing with a RuntimeError isn't obviously better than gunicorn crashing with an ExceptionGroup). == NEW IDEA THAT MAYBE SOLVES BOTH PROBLEMS == Proposal: - any time an unwinding ExceptionGroup encounters a traditional try/except, then it gets replaced with a RuntimeError whose __cause__ is set to the original ExceptionGroup and whose first traceback entry points to the offending try/except block - CUTE BIT I ONLY JUST THOUGHT OF: this substitution happens right *before* we start evaluating 'except' clauses for this try/except So for example: If an ExceptionGroup hits an 'except Exception': The ExceptionGroup is replaced by a RuntimeError. RuntimeError is an Exception, so the 'except Exception' clause catches it. And presumably logs it or something. This way your log contains both a notification that you might want to switch to except* (from the RuntimeError), *along with* the full original exception details (from the __cause__ attribute). If it was an ExceptionGroup(KeyboardInterrupt), then it still gets caught and that's not so great, but at least you get the RuntimeError to point out that something has gone wrong and tell you where? If an ExceptionGroup(ValueError) hits an 'except ValueError': it doesn't get caught, *but* a RuntimeError keeps propagating out to tell you you have a problem. And when that RuntimeError eventually hits the top of your program or ends up in your webserver logs or whatever, then the RuntimeError's traceback will point you to the 'except ValueError' that needs to be fixed. If you write 'except ExceptionGroup': this clause is a no-op that will never execute, because it's impossible to still have an ExceptionGroup when we start matching 'except' clauses. (We could additionally emit a diagnostic if we want.) If you write bare 'except:', or 'except BaseException': the clause always executes (as before), but they get the RuntimeError instead of the ExceptionGroup. If you really *wanted* the ExceptionGroup, you can retrieve it from the __cause__ attribute. (The only case I can think of where this would be useful is if you're writing code that has to straddle both old and new Python versions *and* wants to do something clever with ExceptionGroups. I think this would happen if you're implementing Trio, or implementing a higher-level backport library for catching ExceptionGroups, something like that. So this only applies to like half a dozen users total, but they are important users :-).) -n -- Nathaniel J. Smith -- https://vorpus.org

On Thu, Feb 25, 2021 at 10:23 PM Glenn Linderman <v+python@g.nevcal.com> wrote:

So then why do you need except* at all? Only to catch unwrapped ExceptionGroup before it gets wrapped?

So why not write except ExceptionGroup, and let it catch unwrapped ExceptionGroup?

That "CUTE BIT" could be done only when hitting an except chain that doesn't include an except ExceptionGroup.

Nope, I didn't read the PEP, and don't understand the motivation, but the discussion sure sounded confusing. This is starting to sound almost reasonable.

I'm not sure what to make of the complaint that not reading the motivation makes the motivation confusing :-). But very briefly: the core reason we need ExceptionGroup is because in concurrent programs, multiple things can go wrong at the same time, so our error handling system needs to have some way to represent and cope with that. This means that in concurrent programs (e.g. anything using asyncio, trio, etc.), it's safest to assume that *any* exception could be wrapped in an ExceptionGroup and use except* for everything. The question is how to make error handling in those programs as ergonomic as Python exceptions are currently in non-concurrent programs, without creating too many issues for existing code. So in programs like this, you have to assume that 'except ExceptionGroup' catches *all* exceptions, or worse, a random/unpredictable subset. Saying that that would be good enough is like saying that bare 'except:' is good enough for regular non-concurrent Python (after all, you can always do an isinstance check inside the 'except' block and re-raise the ones you don't want, right?), and that 'except <some type>' is a pointless frivolity. I think most people would disagree with that :-). -n -- Nathaniel J. Smith -- https://vorpus.org

FTR: nobody on this long thread so far has suggested that there are no valid use cases for `except Exception`. Thank you for turning to what happens with 'except ValueError' when an ExceptionGroup[ValueError] is raised, this is important. I'm not sure it's safe to assume that it is necessarily a programming error, and that the interpreter can essentially break the program in this case. Is this not allowed? try: try: obj.func() # function that raises ExceptionGroups except AttributeError: logger.info("obj doesn't have a func") except *(AttributeError, SyntaxError): logger.info("func had some problems") On Fri, Feb 26, 2021 at 5:40 AM Nathaniel Smith <njs@pobox.com> wrote:

On Thu, Feb 25, 2021 at 2:13 PM Guido van Rossum <guido@python.org> wrote:

...

So is "fail-fast if you forget to handle an ExceptionGroup" really a

feature? (Do we call this out in the PEP?)

...

We may believe that "except Exception" is an abuse, but it is too common

to dismiss out of hand. I think if some app has e.g. a main loop where they repeatedly do something that may fail in many ways (e.g. handle a web request), catch all errors and then just log the error and continue from the top, it's a better experience if it logs "ExceptionGroup: <message> [<list of subexceptions>]" than if it crashes.

Yeah, 'except Exception' happens a lot in the wild, and what to do about that has been a major sticking point in the ExceptionGroup debates all along. I wouldn't say that 'except Exception' is an abuse even -- what do you want gunicorn to do if your buggy flask app raises some random exception? Crash your entire web server, or log it and attempt to keep going? (This is almost your example, but adding in the part where gunicorn is reliable and well-respected, and that its whole job is to invoke arbitrarily flaky code written by random users.) Yury/I/others did discuss the idea of a BaseExceptionGroup/ExceptionGroup split a lot, and I think the general feeling is that it could potentially work, but feels like a complicated and awkward hack, so no-one was super excited about it. For a while we also had a compromise design where only BaseExceptionGroup was built-in, but we left it non-final specifically so asyncio could define an ExceptionsOnlyExceptionGroup.

Another somewhat-related awkward part of the API is how ExceptionGroup and plain-old 'except' should interact *in general*. The intuition is that if you have 'except ValueError' and you get an 'ExceptionGroup(ValueError)', then the user's code has some kind of problem and we should probably do.... something? to let them know? One idea I had was that we should raise a RuntimeError if this happens, sort of similar to PEP 479. But I could never quite figure out how this would help (gunicorn crashing with a RuntimeError isn't obviously better than gunicorn crashing with an ExceptionGroup).

== NEW IDEA THAT MAYBE SOLVES BOTH PROBLEMS ==

Proposal:

- any time an unwinding ExceptionGroup encounters a traditional try/except, then it gets replaced with a RuntimeError whose __cause__ is set to the original ExceptionGroup and whose first traceback entry points to the offending try/except block

- CUTE BIT I ONLY JUST THOUGHT OF: this substitution happens right *before* we start evaluating 'except' clauses for this try/except

So for example:

If an ExceptionGroup hits an 'except Exception': The ExceptionGroup is replaced by a RuntimeError. RuntimeError is an Exception, so the 'except Exception' clause catches it. And presumably logs it or something. This way your log contains both a notification that you might want to switch to except* (from the RuntimeError), *along with* the full original exception details (from the __cause__ attribute). If it was an ExceptionGroup(KeyboardInterrupt), then it still gets caught and that's not so great, but at least you get the RuntimeError to point out that something has gone wrong and tell you where?

If an ExceptionGroup(ValueError) hits an 'except ValueError': it doesn't get caught, *but* a RuntimeError keeps propagating out to tell you you have a problem. And when that RuntimeError eventually hits the top of your program or ends up in your webserver logs or whatever, then the RuntimeError's traceback will point you to the 'except ValueError' that needs to be fixed.

If you write 'except ExceptionGroup': this clause is a no-op that will never execute, because it's impossible to still have an ExceptionGroup when we start matching 'except' clauses. (We could additionally emit a diagnostic if we want.)

If you write bare 'except:', or 'except BaseException': the clause always executes (as before), but they get the RuntimeError instead of the ExceptionGroup. If you really *wanted* the ExceptionGroup, you can retrieve it from the __cause__ attribute. (The only case I can think of where this would be useful is if you're writing code that has to straddle both old and new Python versions *and* wants to do something clever with ExceptionGroups. I think this would happen if you're implementing Trio, or implementing a higher-level backport library for catching ExceptionGroups, something like that. So this only applies to like half a dozen users total, but they are important users :-).)

-n

-- Nathaniel J. Smith -- https://vorpus.org

A side benefit is that if an Exception somehow propagates up where only ExceptionGroup is defined, except *() could just work anyway, though it might take a little magic to make sure they act the same. Like Guido said, I don't think it can be retrofitted into existing *-less APIs, and it'll either need a new API and deprecation, or dumping the old for the new hotness if that's the maintenance strategy, but as long as docs call out that this is now returning collective exceptions, I don't see a problem. I don't think it's been mentioned, but it's a nice synergy with pattern matching, even if it's wholly unrelated, and having both in one release will actually make sense. -Em On Fri, Feb 26, 2021 at 6:45 AM Nathaniel Smith <njs@pobox.com> wrote:

On Fri, Feb 26, 2021 at 5:05 AM Irit Katriel <iritkatriel@googlemail.com> wrote:

...

I'm not sure it's safe to assume that it is necessarily a programming error, and that the interpreter can essentially break the program in this case. Is this not allowed?

try: try: obj.func() # function that raises ExceptionGroups except AttributeError: logger.info("obj doesn't have a func") except *(AttributeError, SyntaxError): logger.info("func had some problems")

I'd be fine with disallowing that. The intuition is that things will be simplest if ExceptionGroup is kept as transparent and meaningless as possible, i.e. ExceptionGroup(ValueError) and ValueError mean exactly the same thing -- "some code inside this block raised ValueError" -- and ideally should be processed in exactly the same way. (Of course we can't quite achieve that due to backcompat issues, but the closer we can get, the better, I think?)

If you need to distinguish between the AttributeError from 'obj.__getattr__("func")' vs the AttributeError from the call to func(), then there's already an obvious way to do that, that works for all functions, not just ones that happen to raise ExceptionGroups:

try: f = obj.func except AttributeError: ... try: f() except ...: # or except *...: ...

-n

-- Nathaniel J. Smith -- https://vorpus.org

Excuse me if I post here. Maybe is a stupid question: why, instead of introducing except*, Python can't extend the functionality of except, so it can do what except* would do?

On Fri, Feb 26, 2021 at 11:43 PM Guido van Rossum <guido@python.org> wrote:

On Fri, Feb 26, 2021 at 3:18 PM Marco Sulla <Marco.Sulla.Python@gmail.com> wrote:

...

Excuse me if I post here. Maybe is a stupid question: why, instead of introducing except*, Python can't extend the functionality of except, so it can do what except* would do?

Good question. Here's an example: ``` try: . . . except OSError as err: if err.errno != ENOENT: raise . . . ``` If this would catch ExceptionGroup(OSError), the `err` variable would be an ExceptionGroup instance, which does not have an `errno` attribute.

(Irit: Does the PEP answer this question? I couldn't quickly find it in the rejected ideas. I think it's a reasonable question and we should answer it, either in the Rationale or in Rejected Ideas.)

Added here: https://github.com/python/peps/pull/1846

Thank you for turning to what happens with 'except ValueError' when an ExceptionGroup[ValueError] is raised, this is important.

I'm not sure it's safe to assume that it is necessarily a > programming error, and that the interpreter can essentially break the program in this case.

I'm betting it means a ValueError was raised, but then something (probably an asynchronous framework) aggregated it. I won't swear you would never want to distinguish the two cases, or to distinguish them both from ExceptionGroup[ExceptionGroup[ExceptionGroup[ValueError]]], but ... normally you wouldn't.

Is this not allowed?

try: try: obj.func() # function that raises ExceptionGroups except AttributeError: logger.info("obj doesn't have a func") except *(AttributeError, SyntaxError): logger.info("func had some problems")

Allowed, but probably in error ... no AttributeError will get through to the except * unless it happened inside the except AttributeError handler. Did you mean: try: try: obj.func # function that raises ExceptionGroups except AttributeError: logger.info("obj doesn't have a func") obj.func() except *(AttributeError, SyntaxError): logger.info("func had some problems") I see this as an argument that the except/except* split is tricky, but I don't think it says anything about whether except* clauses should be able to see into nested ExceptionGroups ... nor am I at all confident that I understood your intent. -jJ

On 27Feb2021 00:54, Irit Katriel <iritkatriel@googlemail.com> wrote:

On Sat, Feb 27, 2021 at 12:47 AM Jim J. Jewett <jimjjewett@gmail.com> wrote:

...

...

Is this not allowed?

...

try: try: obj.func() # function that raises ExceptionGroups except AttributeError: logger.info("obj doesn't have a func") except *(AttributeError, SyntaxError): logger.info("func had some problems")

Allowed, but probably in error ... no AttributeError will get through to the except * unless it happened inside the except AttributeError handler. Did you mean:

If obj.func() raises an ExceptionGroup that contains AttributeError then "except AttributeError" doesn't catch it. So it will get through.

And I, for one, would expect that. And _want_ that: I want the code to do what I said, not have some magic which silently/invisibly intercepts ExceptionGroups which contain something buried deep in their subgroup tree. We already allow "deep" exceptions out, to be caught at an arbitrary outer call stack level. I don't see why ExceptionGroups should be any different. I certainly do not want ExceptionGroup([AttributeError]) conflated with AttributeError. That fills me with horror. Cheers, Cameron Simpson <cs@cskk.id.au>

In fact, if you're happy with RuntimeError here, making ExceptionGroup inherit from Exception (instead of BaseException) would do just as well -- after all RuntimeError is pretty arbitrary

Agreed, and I think it should inherit from Exception.

(in fact, it's wrong, because the problem is a static bug in the code, not something that went wrong at run time).

Something went wrong deep inside an individual task. The only clear static bug is that the async library changed how it reported that, and my except clause hasn't done the make-work to keep in step.

Let's see how this compares to the alternatives. First let's define three key examples.

Example 1:

try: raise ExceptionGroup(ValueError) except Exception:

Example 2:

try: raise ExceptionGroup(ValueError) except ValueError:

Example 2(a) : try: raise ExceptionGroup(ValueError,OtherError) except ValueError:

Example 3:

try: raise ExceptionGroup(asyncio.CancelledError) except Exception:

I would prefer that the except clause be executed in all of the examples, but I admit that 2a is a problem, because it could end up silently losing OtherError. And I admit getting example 2 to do the right thing if 2a doesn't might be too much of a contortion. For example 3, I may be missing a subtle point, but I feel that by the time you get to code which doesn't expect asyncio.CancelledError, then you have already cancelled as far as you should. Cancelling may want to bubble up through several other tasks, but it shouldn't kill the whole server, just because the trampoline got sloppy. -jJ

FWIW, the only situation I can think of where you would care that the enclosed exception instances are BaseException but not regular Exception is interactive debugging, and even then there are enough other ways to kill the whole process that I think most people would use one of them instead of wanting a different BaseExceptionGroup that breaks the server instead of just the servlet. I suppose you could argue that the distinction encourages the "good practice" of defensively wrapping "except Exception" in an "except BaseException" that is itself wrapped in a bare except. I suspect it would actually just push people to replace that "except Exception" with the bare except and give up on the logging, because that is a quicker adjustment.

While I don't particularly mind if we get ExceptionGroup, giving it special magical semantics, and especially new syntax, seems like bringing a massively overpowered weapon to bear on something that will only be used rarely. Handling multiple exceptions from an ExceptionGroup could be done using a loop with pattern matching, so is except* really justified? -- Greg

On Fri, 26 Feb 2021 at 23:36, Jim J. Jewett <jimjjewett@gmail.com> wrote:

Whenever I've used except Exception or stronger, it was a sanitary barrier around code that might well do unpredictable or even stupid things. Adding a new kind of exception that I hadn't predicted -- including ExceptionGroup -- would certainly fit this description, and I want my driver loop to do what I told it. (Probably log an unexpected exception, and continue with the next record. I honestly don't even want a page, let alone a crash, because data from outside that barrier ... is often bad, and almost never in ways the on-call person can safely fix. And if they don't have time to find it in the logs, then it isn't a priority that week.)

This is my biggest concern. Disclaimer: I've yet to read the PEP, because async makes my head hurt, but I am aware of some of the background with Trio. Please take this as the perspective of someone thinking "I don't use async/await in my code, can I assume this doesn't affect me?" For me, the biggest requirement I would have is that if I have code like: def safe_fn(): try: do_something_funky() return True except Exception: print("Well, that was a shame...") return False then I am intending to guarantee that calling safe_fn() will never raise an exception. Obviously, the example is a toy (in reality, I'd log the error, skip a record, whatever) but that's not the point here - the point is that I consider it reasonable to expect `except Exception` to be a hard barrier that allows me to be sure I've covered everything. My worry is that if ExceptionGroup exceptions are *not* trapped by `except Exception`, then code like this is no longer safe. And by making ExceptionGroup a BaseException subclass, that's what will happen. Ultimately, there's a genuine use case for code that says "whatever happens in the code I call, this is my last line of defense and I want to be as near to 100% sure as I can be that I regain control at this point". At the moment, that is spelled `except Exception`. If you're proposing to change that, even if it's just to require that it be spelled differently, you're going to break quite a lot of code - and worse, the code you're breaking will be code that has taken care to ensure it's written safely, so you're penalising people who *tried to get stuff right*, which IMO is the worst type of breakage. I will at some point read the PEP fully, to better understand the async side of the story, but for now please consider this as the perspective of someone who doesn't expect to care about async, and therefore wants to feel safe in assuming this PEP won't affect them. Paul PS If you're thinking "but if you were using async, you'd know about it", consider that I could be using a PyPI library that, for example, returned stock values. That library switches to using async, and has a bug that lets an exception group escape. My `except Exception` code is designed *precisely* to protect me against such bugs, without me needing to know anything about how the library works...

On Fri, 26 Feb 2021 at 23:36, Jim J. Jewett <jimjjewett@gmail.com> wrote:

Whenever I've used except Exception or stronger, it was a sanitary barrier around code that might well do unpredictable or even stupid things. Adding a new kind of exception that I hadn't predicted -- including ExceptionGroup -- would certainly fit this description, and I want my driver loop to do what I told it.

This is my concern as well. I've got plenty of "except Exception" and IMO none consititutes "abuse" - they're there for the same reason Jim cites above: a barrier around _arbitrary_ callback code - so that the outer control facility continues to roll on as expected. All of these are handler frameworks for use by other code - they inherently call an unknown and arbitrary thing at some point and expect to catch an Exception raised and suitably handle it (log it, record it for the caller to see later, whatever). Now _all_ those handlers will need a special except* handler for ExceptionGroup, because they DO NOT KNOW whether the called code might raise it. And "except Exception", the longstanding recommended way to catch "everything (well almost everything)" no longer works. Now, I've long wanted something like MultiError or ExceptionGroup to gather miltiple failures for raising together at a suitable slightly-later point. [...] On 27Feb2021 19:06, Guido van Rossum <guido@python.org> wrote:

Our goal for the PEP is that *unless* you're going to use APIs that are documented to raise ExceptionGroup, you won't have to use `except *` nor will you have to deal with ExceptionGroup otherwise.

But.. this isn't Java, where the (hah!) type annotations document the exceptions it raises. Jim again, catching Exception to protect calling code:

...

def safe_fn(): try: do_something_funky() return True except Exception: print("Well, that was a shame...") return False

then I am intending to guarantee that calling safe_fn() will never raise an exception. [...]

Guido:

The modification I proposed where we have both BaseExceptionGroup and ExceptionGroup will satisfy this need. It basically means that *unless* you are explicitly using an API that is documented to raise [Base]ExceptionGroup (such as a future variant of asyncio.gather()), you don't have to care about it. [...] that's why I am proposing to change the PEP so that your code will remain safe.

That would be welcome to me, too. Cheers, Cameron Simpson <cs@cskk.id.au>

On Tue, Feb 23, 2021 at 11:00 PM Irit Katriel <iritkatriel@googlemail.com> wrote:

Hi Steve,

Thank you for trying out the implementation. Please do let me know about bugs you find.

Thanks you for your response! It seems I had missed the (now) obvious naked exception wrapping logic. Apologies for adding noise to the conversation. I've added a comment to your fork's PR with the error detail. Best of luck with the PEP Steve

On Tue, Feb 23, 2021 at 8:00 AM Ivan Pozdeev via Python-Dev < python-dev@python.org> wrote:

- In https://www.python.org/dev/peps/pep-0654/#programming-without-except, the natural way isn't shown:

try: <smth> except (MultiError, ValueError) as e: def _handle(e): if isinstance(e, ValueError): return None else: return exc MultiError.filter(_handle,e)

So a statement that the current handling is "cumbersome" and "unintuitive" is unconvincing.

If this results in lots of boilerplate code with isinstance(), filter() can be changed to e.g. accept a dict of exception types and handlers. Actually, I wonder why you didn't do that already if handling specific exception types and reraising the rest is the standard procedure! Then the code would be reduced to:

try: <smth> except (MultiError, ValueError) as e: MultiError.filter({ValueError: lambda _: None}, e)

- https://github.com/python-trio/trio/issues/611 says that it somehow causes problems with 3rd-party libraries -- but there's no explanation how -- either there or in the PEP.

If some code doesn't know about MultiError's, it should handle one like any other unknown exception that it cannot do anything intelligent about. If it wishes to handle them, you need to split MultiError into a separate library that anyone could use without having to pull the entire `trio`.

I think the main point our PEP tries to make is that having to define a helper function to handle exceptions (and then having to call a utility to call the helper) feels clumsy. However there may be an omission in the PEP -- what if we want to do something special for each suberror? If we just iterate over `eg.errors` we would have to do something recursive for exceptions wrapped inside multiple nested groups. We could add a helper method to ExceptionGroup that iterates over suberrors, flattening nested groups. But we'd need to do something special to recover the tracebacks there (groups share the common part of the traceback). Or we could say "if you need the traceback you're going to have to write something recursive" -- like we have to do in traceback.py. But we might as well generalize whatever we're doing there. (Irit: I suspect there's a piece of API design we missed here.) -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

[Subthread: handling individual errors]

Rather than iterator, I think we should add a visitor that calls a function for each leaf exception,

I agree that we shouldn't add an `__iter__` method. But what if we added a separate method that returns an iterator? Then you could write ``` for e in eg.something(): print("caught", e) ``` This looks cleaner to me than having to write ``` def handler(e): print("caught", e) eg.something(handler) ``` (Does my bias against callback functions shine through? :-)

plus a utility that returns the traceback of a leaf exception (as a single list, copying frames from the tracebacks of ExceptionGroups so that it's not destructive.)

Hm, what would the arguments for that utility be? Since ExceptionGroups can be nested, if you pass in the root EG and the leaf exception, it would require a full tree search to find the nodes that have all the needed tracebacks. Maybe the API for the iterator-returning method should be that it returns an iterator of (exception, list-of-tracebacks) pairs? Or maybe there should be two iterator-returning methods, one that yields just exceptions and one that yields such pairs, so that if you don't need the tracebacks we don't have to construct them. (Separate methods so the static type doesn't depend on the value of a flag.) If in the end we were to go for something with a handler callback we could do a similar thing. On Wed, Feb 24, 2021 at 1:27 AM Irit Katriel <iritkatriel@googlemail.com> wrote:

On Wed, Feb 24, 2021 at 4:55 AM Guido van Rossum <guido@python.org> wrote:

However there may be an omission in the PEP -- what if we want to do

...

something special for each suberror? If we just iterate over `eg.errors` we would have to do something recursive for exceptions wrapped inside multiple nested groups. We could add a helper method to ExceptionGroup that iterates over suberrors, flattening nested groups. But we'd need to do something special to recover the tracebacks there (groups share the common part of the traceback). Or we could say "if you need the traceback you're going to have to write something recursive" -- like we have to do in traceback.py. But we might as well generalize whatever we're doing there. (Irit: I suspect there's a piece of API design we missed here.)

This is mentioned briefly where we talk about why ExceptionGroup is not iterable: https://www.python.org/dev/peps/pep-0654/#the-exceptiongroup-api We left it out pending a use case.

Rather than iterator, I think we should add a visitor that calls a function for each leaf exception, plus a utility that returns the traceback of a leaf exception (as a single list, copying frames from the tracebacks of ExceptionGroups so that it's not destructive.) This way the expensive traceback construction is happening explicitly when it is needed.

I think accessing one exception at a time to do something other than formatting it is more likely to be overused than actually needed. We did that in an earlier version of the PEP for the "filter OSErrors by type" example, which we did with iteration and now do with subgroup:

try: low_level_os_operation() except *OSerror as errors: raise errors.subgroup(lambda e: e.errno != errno.EPIPE) from None

-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

Good question. The main use case I see for iterating over individual exceptions is logging frameworks that want to format their own exception tracebacks for whatever reason. I know that the traceback.py module has extensive utilities for exactly that, but I betcha there are plenty of people who still want to roll their own, e.g. for compatibility with other tooling on their platform. Presumably the code for traversing a tree of exceptions exists in traceback.py. I honestly don't think it matters much whether we do it as an iterator or using callbacks, as long as it visits all the leaves in the tree. Hm, a different idea: maybe it's simple enough that we can just add an example showing how to do this? Then people can tailor that e.g. to use various traversal orders. (We could just link to the code in traceback.py, but it probably is full of distractions.) On Thu, Feb 25, 2021 at 3:01 AM Irit Katriel <iritkatriel@googlemail.com> wrote:

On Thu, Feb 25, 2021 at 5:19 AM Guido van Rossum <guido@python.org> wrote:

...

[Subthread: handling individual errors]

...

Rather than iterator, I think we should add a visitor that calls a function for each leaf exception,

I agree that we shouldn't add an `__iter__` method. But what if we added a separate method that returns an iterator? Then you could write ``` for e in eg.something(): print("caught", e) ``` This looks cleaner to me than having to write ``` def handler(e): print("caught", e) eg.something(handler) ``` (Does my bias against callback functions shine through? :-)

I hear you. I suggested a visitor to make it a bit awkward to use because I'm not sure why people should iterate over individual exceptions in an ExceptionGroup, and I think that by providing an iteration utility we are implying that this is what you should do. So let me be more direct instead of proposing passive-aggressive APIs.

Can we take a step back and talk about how we think people would want to handle ExceptionGroups?

In the rejected ideas section we discuss this a bit, with a reference to Yury's writeup: https://github.com/python/exceptiongroups/issues/3#issuecomment-716203284

TL;DR: when you get a group of exceptions from asyncio or the like, you may want to query it for exception types is contains (with subgroup()), you may want to format it into a log (with traceback.* methods), but you are unlikely to care whether there are 1, 2 or 300 ValueErrors. Your program will probably do the same thing regardless. If you allowed your ValueError get collected into an ExceptionGroup you already lost the context in which it happened to it's unlikely that you can make a targeted recovery which is relevant to this particular exception.

So, what is the use case for iterating over single exceptions?

...

-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

ValueError("Partner: 3 File: 127 record: 93 is missing field: currency") tells the production support people who to contact and what to say. I'm not sure what additional context would be helpful, let alone how it might have been available "at the time", but lost now that the ValueAttribute is collected into an ExceptionGroup. -jJ

On Wed, Feb 24, 2021 at 5:51 PM Jim J. Jewett <jimjjewett@gmail.com> wrote:

Are you saying that

except *ValueError as e:

will catch

ValueError and ExceptionGroup(ValueError) but miss ExceptionGroup(ExceptionGroup(ValueError)) ?

No, it will catch it regardless of how many levels of ExceptionGroup wrap it (zero or more, in fact). -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

Hi Petr, Thank you for your careful reading and encouragement.

The `ExceptionGroup` class is final, i.e., it cannot be subclassed.

What's the rationale for this?

ExceptionGroup.subgroup()/split() need to create new instances, and subclassing would make that complicated if we want the split results to have the same type.

It is possible to catch the ExceptionGroup type with except, but not with except* because the latter is ambiguous

What about `except *(TypeError, ExceptionGroup):`?

Good question. We should block that as well.

...

Motivation: Errors in wrapper code

This use case sticks out a bit: it's the only one where ExceptionGroup doesn't represent joining equivalent tasks. Consider code similar to bpo-40857:

try: with TemporaryDirectory() as tempdir: os.rmdir(tempdir) n = 1 / 0 except ArithmeticError: # that error can be safely ignored! pass

Instead of a FileNotFoundError with ArithmeticError for context you'd now get an ExceptionGroup. Neither is handled by `except ArithmeticError`. Where is the win?

I agree, if TemporaryDirectory() were ever to adopt ExceptionGroups then it will probably be through a new API (like an opt-in parameter) to avoid breaking current code. Users of such a TemporaryDirectory would need to wrap the calls with try-except* (see the example in my previous reply to Steve).

...

Motivation: Multiple failures when retrying an operation

This is somewhat similar to the TemporaryDirectory, except there's no `with` block that feels like it should be "transparent" w.r.t. user errors. If I currently have:

try: create_connection(*addresses) except (Timeout, NetworkNotConnected): # that's OK, let's try later pass

what should happen after Python 3.10? Apart from adding a new function, I can see two possible changes: - create_connection() starts always raising ExceptionGroup on error, breaking backwards compatibility in the error case - create_connection() starts only raising ExceptionGroup only for 2+ errors, breaking backwards compatibility in the 2+ errors case

Both look like heisenbug magnets. IMO, the second one is worse; "code which is now *potentially* raising ExceptionGroup" (as mentioned in the Backwards Compatibility section; emphasis mine) should be discouraged.

Arguably, this here is a problem with the create_connection function: the PEP adds a better way how it could have been designed, and that is virtuous. Still, having it in Motivation might be misleading.

I agree. Raising ExceptionGroups is an API breaking change, for any library. No function should just start raising ExceptionGroups.

long term plan to replace `except` by `catch`

Whoa! Is that a real plan?

In the rejected ideas section, anything goes! Irit

On 2021-02-23 23:05, Caleb Donovick wrote:

What is the motivation for returning `None` on empty splits?  I feel like this creates an unnecessary asymmetry.  I don't personally have a use case for the this feature so I may be missing something but it seems like it would force an annoying pattern:

``` try:     foo() except ExceptionGroup as eg:     g1, g2 = eg.split(predicate)     for e in g1:         handle_true(e)     for e in g2:         handle_false(e) ``` Would need to be written: ``` try:     foo() except ExceptionGroup as eg:     g1, g2 = eg.split(predicate)     if g1 is not None:         for e in g1:             handle_true(e)     if g2 is not None:         for e in g2:             handle_false(e) ```

Also this creates an subtle difference with subgroup:

``` g1, g2 = eg.split(predicate) h1, h2  = eg.subgroup(predicate), eg.subgroup(lambda e: not predicate(e)) assert g1 == h1 and g2 == h2 # only true if `None not in {g1, g2}` ```

.subgroup returns an ExceptionGroup and .split returns 2 of them. ExceptionGroup isn't iterable (it's mentioned in the "Rejected Ideas" section), so your code wouldn't work anyway.

On Mon, Feb 22, 2021 at 4:47 PM Irit Katriel via Python-Dev <python-dev@python.org <mailto:python-dev@python.org>> wrote:

Hi all,

We would like to request feedback on PEP 654 -- Exception Groups and except*.

https://www.python.org/dev/peps/pep-0654/ <https://www.python.org/dev/peps/pep-0654/>

It proposes language extensions that allow programs to raise and handle multiple unrelated exceptions simultaneously, motivated by the needs of asyncio and other concurrency libraries, but with other use cases as well.

* A new standard exception type, ExceptionGroup, to represent multiple exceptions with   shared traceback. * Updates to the traceback printing code to display (possibly nested) ExceptionGroups. * A new syntax except* for handling ExceptionGroups.

A reference implementation (unreviewed) can be found at: https://github.com/iritkatriel/cpython/pull/10 <https://github.com/iritkatriel/cpython/pull/10>

Thank you for your help

On Tue, Feb 23, 2021 at 7:37 PM Ethan Furman <ethan@stoneleaf.us> wrote:

On 2/22/21 4:24 PM, Irit Katriel via Python-Dev wrote:

...

We would like to request feedback on PEP 654 -- Exception Groups and except*.

https://www.python.org/dev/peps/pep-0654/

It proposes language extensions that allow programs to raise and handle multiple unrelated exceptions simultaneously, motivated by the needs of asyncio and other concurrency libraries, but with other use cases as well.

It sounds like the long-term goal is to move away from `except` and replace it with `except *` -- is that correct?

I don't think so -- if we expected that to happen the extra '*' in the syntax would be a nuisance. The premise of the PEP is rather that raising and catching multiple exceptions at once is always going to be an esoteric hobby. The most common case by far would be in async frameworks, but we don't expect 'async def' to eventually become the standard function definition either (nor 'await f()' the standard call :-). And even in an async framework, I don't expect that every async function would be "upgraded" to raise ExceptionGroup -- only specific APIs like gather() or create_connection(). (Even for those, backwards compatibility concerns will probably mean that we'll have to introduce *new* APIs that can raise ExceptionGroup.) The PEP lists a handful of other use cases, none of which seem to suggest to me that this is going to be a common thing. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

Hi Ethan, See Irit's response to my about why ExceptionGroup doesn't subclass Exception -- I think this is still a little bit of an open issue. Another response (which I also read somewhere in one of Irit's messages) is that a library API shouldn't go from raising subclasses of Exception only to raising ExceptionGroups. That would be a backwards incompatible change to an API. --Guido On Tue, Feb 23, 2021 at 8:52 PM Ethan Furman <ethan@stoneleaf.us> wrote:

On 2/23/21 7:56 PM, Guido van Rossum wrote:

...

On Tue, Feb 23, 2021 at 7:37 PM Ethan Furman wrote:

...

...

It sounds like the long-term goal is to move away from `except` and replace it with `except *` -- is that correct?

I don't think so -- if we expected that to happen the extra '*' in the syntax would be a nuisance. The premise of the PEP is rather that raising and catching multiple exceptions at once is always going to be an esoteric hobby. The most common case by far would be in async frameworks, but we don't expect 'async def' to eventually become the standard function definition either (nor 'await f()' the standard call :-).

And even in an async framework, I don't expect that every async function would be "upgraded" to raise ExceptionGroup -- only specific APIs like gather() or create_connection(). (Even for those, backwards compatibility concerns will probably mean that we'll have to introduce *new* APIs that can raise ExceptionGroup.) The PEP lists a handful of other use cases, none of which seem to suggest to me that this is going to be a common thing.

I can see the value in `except *` for concurrent code; my concern is how it will operate with the standard `try/except` framework. With the new non-Exception-derived ExceptionGroup class, existing try-excepts that catch Exception will fail for every ExceptionGroup that leaks through; the fix for that is a doubly-nested try-except/try-except* -- what do we get for that code churn? What is the advantage in not deriving from Exception? If I recall correctly, we have only three exception types that derive directly from BaseException: Exception, SystemExit, and KeyboardInterrupt -- with SystemExit and KeyboardInterrupt both being concerned with making sure an application can quit.

-- ~Ethan~ _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/445RAHP7... Code of Conduct: http://python.org/psf/codeofconduct/

-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

Hi Jim, Let me echo Irit's response here -- what's your use case for subclassing ExceptionGroup? Requiring that subclasses have a compatible __init__ or __new__ sounds like a bug magnet, since this can't be checked when the subclass is being defined, and it goes against conventional rules for those signatures. --Guido On Wed, Feb 24, 2021 at 6:22 AM Jim J. Jewett <jimjjewett@gmail.com> wrote:

The "no subclasses" seems pretty severe, particularly if you can't even use marker attributes because it isn't clear when a different instance of container ExceptionGroup will be substituted.

The justification for this restriction was that .split() had to be be able to instantiate ... wouldn't it be enough to just say that subclasses need a compatible __init__ and __new__ for that reason, and then leave it to consenting adults?

-jJ _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/5OLFDKJC... Code of Conduct: http://python.org/psf/codeofconduct/

-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

On Wed, Feb 24, 2021 at 5:58 PM Jim J. Jewett <jimjjewett@gmail.com> wrote:

If it (compatible __new__ and __init__) needs to be checked at definition time, just try create an instance passing the same arguments you would pass to the base class. If the creation it doesn't raise an exception, that is good enough.

User code can do that, the interpreter should not use such heuristics.

This isn't about theoretical type safety against malice; it is about defining the minimal protocol for an ExceptionGrouping that has to be supported by someone who wants something other than the default flavor.

You still haven't shown a use case for wanting to subclass ExceptionGroup. It's easy to allow this later if there's a valid use case (it wouldn't require a PEP, just a well-reasoned use case). But if we allow it now and in the future we discover it causes subtle bugs, it would be difficult to roll back (requiring deprecation over several releases). Why would you want a different flavor of ExceptionGroup? -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

Ideally, (at least) trivial subclasses could be declared, and the class itself would serve as the marker. I would prefer regular subclasses, so that they could offer methods as well. Alternatively, at least copy the instance __dict__ to the new ExceptionGroup instance. By compatible __init__ and __new__, I mean "whatever you do in ExceptionGroup.subgroup to create a new instance with fewer contained Exceptions ... do the same with MyExceptionGroup." I'm assuming that "whatever" is to call __new__ and __init__ with "a message string and a sequence of the nested exceptions, for example: ExceptionGroup('issues', [ValueError('bad value'), TypeError('bad type')])." -jJ

On Wed, Feb 24, 2021 at 7:09 PM Emily Bowman <silverbacknet@gmail.com> wrote:

After reading through the PEP and skimming the code (but I didn't build it), something I didn't see: What happens to a currently conforming except check?

try: async with trio.open_nursery() as nursery: # Make two concurrent calls to child() nursery.start_soon(child) nursery.start_soon(child) except ValueError: pass

I've removed the * from the example: Say the interface was built for 3.7, but the "trio" module has been upgraded to use ExceptionGroups which can't fall back to a standalone exception.

Silently hand back the first exception, or the first matching exception? Deprecation warning? Silently skip? Context-specific error? Run the default error handler?

I think that deserves a statement in the PEP.

The ExceptionGroup would bubble up. IIRC this Trio behavior is considered problematic by the Trio developers, because it means that if *two* children fail with ValueError, it will raise MultiError and the except clause will not trigger (see the "MultiError v2" issue linked from the PEP). But changing trio.open_nursery() to raise ExceptionGroup would be considered backwards compatible. I would recommend introducing a new API instead. This is discussed (though maybe not in enough detail?) under Backwards Compatibility in the PEP. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

Hi Barry, On Mon, Mar 1, 2021 at 6:12 PM Barry Scott <barry@barrys-emacs.org> wrote:

Have you considered alternative syntax to the "except*"? I feel that expect* is too easy to confuse with plain except.

We briefly considered things like "except Group[ValueError, TypeError]". Do you (or anyone else) have any suggestions on syntax?

As others have said needing to change all except Exception: handlers would be a breaking change for code that I maintain.

I think its accepted that that idiom should continue to be as reliable as it currently is. Can you confirm that I have understood the conversation on this point?

That's not happening. The PEP has been updated regarding this. It was never going to be all "except Exception", most of them would not be affected. But we have a better plan now so this discussion is moot. As an aside - I found it interesting that the option to wrap BaseException instances by an Exception, which came up a couple of times in this thread, didn't generate any anxiety. In my day job I maintain a proprietary DSL that ~4000 developers use, and my experience has been that a bug that swallows an exception you need is worse than a bug that makes you get an exception you did not expect. The latter can hurt you once (and then you fix it). But the former is (1) harder to detect (2) longer to debug (3) once you know about it there is nothing you can do to work around it. Problems like that cost us years and years of work.

I have been thinking about why I would be getting an ExceptionGroup with, for example, a number of OSError for a handler to deal with. Would it not be a difficult piece of code to write without the context of each OSError?

What I'm think is that the lower level pieces would be catching the OSError closer to the source of the problem and raising a App/Lib specific exception that provides the context. For example AppFailedToSaveConfigError and AppFailedToSaveDataError as oppose to a two permission OSError's.

I think you're right about this. OSError is an example of an exception with a data field that is often inspected for its value, so we picked on it to talk about how you would do something like this. Most of the time this won't be needed, but we still need to know that it's possible and reasonably ergonomic to do it.

With context I can imagine that handling the ExceptionGroup would be a lot easier for the App designer.

If that the pattern that emerges then is the complexity of nested ExceptionGroups going to only rarely used?

I can't imagine people building deep trees of exceptions in practice (at least not on purpose). But some nesting may show up naturally, and we need to support it because otherwise it can get awkward if, for example, asyncio.gather() needs to wrap an exception group that came from a TemporaryDirectory.__exit__(). Irit

On Tue, 2 Mar 2021 at 21:46, Irit Katriel via Python-Dev <python-dev@python.org> wrote:

As an aside - I found it interesting that the option to wrap BaseException instances by an Exception, which came up a couple of times in this thread, didn't generate any anxiety.

Probably because it wasn't clear that was ever being proposed... (or at least the implication wasn't obvious - presumably this is somehow related to BaseExceptions being accumulated in ExceptionGroups?) :-( I would consider it essential that if someone hits Ctrl-C and that generates a KeyboardInterrupt, then: 1. That KeyboardInterrupt will *not* get caught by exception handlers only interested in Exception instances 2. That KeyboardInterrupt *will* get caught by any handler that does an explicit `except KeyboardInterrupt` or an `except BaseException`. To me, that's pretty much essential to correct Ctrl-C handling in any app (never ignore a user's Ctrl-C and always exit cleanly if one is raised). That might mean that BaseException instances shouldn't be "groupable", but I don't want to comment on that until I've properly read the PEP (I've skimmed it now, but only superficially). At a minimum, I'd consider it a bug for library code to manually wrap a KeyboardInterrupt in an exception group (just like code that catches KeyboardInterrupt and re-raises it as a ValueError would be today). Paul

On Wed, 3 Mar 2021 at 12:37, Irit Katriel <iritkatriel@googlemail.com> wrote:

This is covered the PEP, but TL;DR: If we could make "except KeyboardInterrupt" catch BaseExceptionGroup([KeyboardInterrupt]) in a reasonably backwards compatible way then we wouldn't need except*. [...] As you noted, no library is under any obligation to wrap KeyboardInterrupts into the exception groups it raises. You may decide it's a bad idea and not do it. What we are discussing here is what the language should make possible. We agree that wrapping a BaseException by an Exception is something we should definitely block. When it's wrapping a BaseException by another, new BaseException type, in my view that's ok. You may have a bug where you don't catch an exception you want to catch, because you are using a new API incorrectly. But you won't have bugs where you swallow an exception that you didn't swallow before.

Thanks for the explanation. I understand your point here, and I see what you're saying. But I have a couple of questions still: 1. Having now read the PEP, I don't actually see a use case for grouping BaseExceptions. Why would anyone catch KeyboardInterrupt or SystemExit and wrap them in a BaseExceptionGroup anyway? It seems to me that the right thing to do, even in async or parallel code, is to just propogate the KeyboardInterrupt/SystemExit up to the main program. Losing a ValueError that happened at the exact same time as the user pressed Ctrl-C seems like it's not going to be a problem in practice... 2. Given the above, why even have a means of grouping BaseExceptions at all? Why not just have ExceptionGroup that can only catch instances of Exception? If there really *was* a low-level case where some code absolutely had to (temporarily) group KeyboardInterrupt, for example, it could be temporarily wrapped: class IGotInterrupted(Exception): def __init__(self, exc): self.exc = exc def might_be_interrupted(): try: critical_stuff() except KeyboardInterrupt as exc: raise IGotInterrupted(exc) def funky_parallel_stuff(): try: do_in_parallel(might_be_interrupted) except *IGotInterrupted as e: # Please excuse the fact that I don't know the # correct way to re-raise here. Might need raise from. raise e.exc # Do we also need to consider an *ungrouped* IGotInterrupted? # That's for the library to decide, and is why not letting it escape is a good thing... That would be appropriate for really low-level code, and it would be a fairly obvious anti-pattern for an exception class like IGotInterrupted to "escape" so that user code would ever see it. I guess the argument for grouping BaseExceptions is "why make it necessary to do complicated wrapping like this?" To which my answer is "because it should almost never be something you do, and so why add support for it when it can be done *without* needing explicit support in the PEP?" And by not having stdlib support for wrapping BaseExceptions, we're signalling that we don't think people should be doing it casually... Paul

On Wed, 3 Mar 2021 at 14:59, Irit Katriel <iritkatriel@googlemail.com> wrote:

...

2. Given the above, why even have a means of grouping BaseExceptions at all? Why not just have ExceptionGroup that can only catch instances of Exception?

Because the IGotInterrupted alternative involves wrapping a BaseException by an Exception, which is not something we should push people into doing (it's not that different from allowing ExceptionGroup to wrap BaseExceptions directly).

That's a fair point.

What's the harm/complication in offering a BaseExceptionGroup(BaseException) in addition to ExceptionGroup(BaseExceptionGroup, Exception)?

Similar to the argument for "except Exception". Applications that trap KeyboardInterrupt so that they can exit cleanly without an ugly traceback will no longer trap *all* keyboard interrupts, as they could miss grouped ones. If we accept that grouped exceptions should never escape out of a well-defined context, then this wouldn't be such an issue. But there's nothing in the PEP that enforces that, and there *is* code that needs to be prepared for "any sort of result". It's the except Exception argument again. So code that wants to exit cleanly in the face of Ctrl-C will need to be rewritten from: try: main() except KeyboardInterrupt: print("User interrupted the program. Exiting") sys.exit(1) to: try: try: main() except KeyboardInterrupt: print("User interrupted the program. Exiting") sys.exit(1) except *KeyboardInterrupt: print("User interrupted the program. Exiting") sys.exit(1) Did I miss an easier way of writing this code? And worse, how would I write it so that it was portable between Python 3.9 and later versions (which is a common requirement for library code - admittedly library code wouldn't normally be doing this sort of top-level trap, but it could just as easily be "catch Ctrl-C and do a bit of tidy-up and re-raise").

I think the only reason you're comfortable with having to select between the exceptions that were raised and discard some of them is because that's where we are today. The PEP lists several standard library and other APIs that discard errors because they need to pick one. That's what we're trying to fix.

Maybe. But I'm not looking at it as being "comfortable" with the current situation, but rather as "I don't use any of these new features, why am I having to change my code to accommodate stuff I don't use?" If I own the full stack, that's not an issue, but frameworks and libraries typically have to interact with other users' code, and there the contract has changed from "do what you like in your code and I'll cope" to "do what you like in your code as long as you don't let an exception group escape, and I'll cope"... And I have to change *my* code to get the old contract back. But it's a small point in the wider scheme of things, and I'm not going to labour the point any more. Thanks for listening and taking the time to reply. Paul Paul

On Wed, Mar 3, 2021 at 4:37 PM Paul Moore <p.f.moore@gmail.com> wrote:

Similar to the argument for "except Exception". Applications that trap KeyboardInterrupt so that they can exit cleanly without an ugly traceback will no longer trap *all* keyboard interrupts, as they could miss grouped ones.

See below.

If we accept that grouped exceptions should never escape out of a well-defined context, then this wouldn't be such an issue. But there's nothing in the PEP that enforces that, and there *is* code that needs to be prepared for "any sort of result". It's the except Exception argument again.

So code that wants to exit cleanly in the face of Ctrl-C will need to be rewritten from:

try: main() except KeyboardInterrupt: print("User interrupted the program. Exiting") sys.exit(1)

to:

try: try: main() except KeyboardInterrupt: print("User interrupted the program. Exiting") sys.exit(1) except *KeyboardInterrupt: print("User interrupted the program. Exiting") sys.exit(1)

It suffices to do try: main() except *KeyboardInterrupt: print("User interrupted the program. Exiting") sys.exit(1) because "except *T" catches Ts as well.

Did I miss an easier way of writing this code? And worse, how would I write it so that it was portable between Python 3.9 and later versions (which is a common requirement for library code - admittedly library code wouldn't normally be doing this sort of top-level trap, but it could just as easily be "catch Ctrl-C and do a bit of tidy-up and re-raise").

For older Pythons you would have to do something like except KeyboardInterrupt: ... except BaseExceptionGroup: # some stub type in old versions # inspect the contents # if there's a KeyboardInterrupt do what you need to do # reraise the rest

...

I think the only reason you're comfortable with having to select between the exceptions that were raised and discard some of them is because that's where we are today. The PEP lists several standard library and other APIs that discard errors because they need to pick one. That's what we're trying to fix.

Maybe. But I'm not looking at it as being "comfortable" with the current situation, but rather as "I don't use any of these new features, why am I having to change my code to accommodate stuff I don't use?" If I own the full stack, that's not an issue, but frameworks and libraries typically have to interact with other users' code, and there the contract has changed from "do what you like in your code and I'll cope" to "do what you like in your code as long as you don't let an exception group escape, and I'll cope"... And I have to change *my* code to get the old contract back.

"except Exception"/"except BaseException" is the special case of "I don't know what I'm calling and I want to catch everything". And that (to repeat, just in case) will work as you expect. If you are actually handling exceptions selectively, then I can break you already in 3.9 just by raising a different exception type to the one you are catching. How is this different? Raising an ExceptionGroup is an API change. If you call APIs that say they will raise ExceptionGroups you need to update your code accordingly. If a library doesn't document that it raises ExceptionGroups and then one of those escapes, then that library has a bug. Just like with any other exception type.

But it's a small point in the wider scheme of things, and I'm not going to labour the point any more. Thanks for listening and taking the time to reply.

It's an important one though. Thanks for asking good questions. Irit

On Wed, Mar 3, 2021 at 6:57 PM Paul Moore <p.f.moore@gmail.com> wrote:

Sorry, I keep thinking I've finished and you keep making interesting points :-)

On Wed, 3 Mar 2021 at 17:01, Irit Katriel <iritkatriel@googlemail.com> wrote:

...

Raising an ExceptionGroup is an API change. If you call APIs that say they will raise ExceptionGroups you need to update your code accordingly. If a library doesn't document that it raises ExceptionGroups and then one of those escapes, then that library has a bug. Just like with any other exception type.

In my experience, libraries don't document what exceptions they raise very well. You can call that a bug, but it's a fact of life, I'm afraid. The problem here isn't so much that the library code now raises an exception that it used not to raise, but rather that *the user hitting Ctrl-C* can now result in a different exception surfacing in my code than it used to. Libraries don't re-wrap KeyboardInterrupt, as you pointed out in a previous response, so I can currently write code that traps KeyboardInterrupt, safe in the knowledge that by doing so I'll handle that user action properly. But with PEP 654, libraries might well (indeed, some libraries almost certainly will) start wrapping KeyboardInterrupt in an exception group. That's a backward incompatible change from the perspective of my code's interaction with the user, and I need to re-code my application to deal with it (and worse still, writing that new code in a way that is portable between versions is not particularly straightforward).

This is also true for MemoryError, and many other errors. What makes KeyboardInterrupt special?

...

For older Pythons you would have to do something like

except KeyboardInterrupt: ... except BaseExceptionGroup: # some stub type in old versions # inspect the contents # if there's a KeyboardInterrupt do what you need to do # reraise the rest

I'd be inclined to suggest that a complete version of this should be included in the "Backward compatibility" part of the PEP, as I honestly don't really know how I'd write that without doing more research. But such an example would make the KeyboardInterrupt case seem more important than it is. Maybe if it's framed as "how to write calling code that's compatible with older versions of Python but still able to handle called code potentially raising exceptions that you need to trap as part of a group", that would be a useful general example.

Or maybe it's not actually something that will be needed that often. I'm not sure - I'm trying to think in terms of pip, where we can't use new features in our own code until we drop support for older versions, but we might potentially rely on a library that uses exception grouping internally on versions where it's available (and that code lets those exception groups escape). It feels like a stretch to claim this is particularly likely, but conversely it's something I can easily imagine *could* happen...

If a library starts raising ExceptionGroups from version 3.X it should probably do that from a new API so people won't have to worry about it just because they are bumping Python version. So I think the cross-version issue is in the case of "I'm calling a user function and I don't know what it is or what it raises", or the "I just want to write all exceptions to the log and ignore them". So this is the "except Exception" case, which will work.

On 4/03/21 5:37 am, Paul Moore wrote:

frameworks and libraries typically have to interact with other users' code, and there the contract has changed from "do what you like in your code and I'll cope" to "do what you like in your code as long as you don't let an exception group escape, and I'll cope"... And I have to change *my* code to get the old contract back.

Seems to me this whole issue would go away if the ordinary except statement were to look inside ExceptionGroups. In other words, the only difference between except and except* would be that multiple except* clauses can be run, whereas only one except clause will run (the first one that matches something in the ExceptionGroup). Is there any good reason not to do things that way? -- Greg

On 3/3/2021 2:49 PM, Irit Katriel via Python-Dev wrote:

On Wed, Mar 3, 2021 at 10:39 PM Greg Ewing <greg.ewing@canterbury.ac.nz <mailto:greg.ewing@canterbury.ac.nz>> wrote:

On 4/03/21 5:37 am, Paul Moore wrote: > frameworks and libraries typically have to interact with other users' > code, and there the contract has changed from "do what you like in > your code and I'll cope" to "do what you like in your code as long as > you don't let an exception group escape, and I'll cope"... And I have > to change *my* code to get the old contract back.

Seems to me this whole issue would go away if the ordinary except statement were to look inside ExceptionGroups.

In other words, the only difference between except and except* would be that multiple except* clauses can be run, whereas only one except clause will run (the first one that matches something in the ExceptionGroup).

Is there any good reason not to do things that way?

That's an interesting idea.

Do you mean that one exception gets handled and the rest of the group is reraised? Or discarded?

The value of sys.exc_info() (and the e in "except T as e:") needs to be a single naked exception. So if there is more than one match in the group we would need to pick one (let's say the first in DFS order).

If we do this, then we have this situation. Before ExceptionGroups, you got to choose which of the exceptions you have is the most important, and you raised only that one. Now you raise a bunch of them and the order of the except clauses in caller's code determines which one of them counts and which ones are discarded. What do you make of that?

You _could_ implement it as you said, but remember, you that with this idea, you are changing how except clauses work—so instead of making the order of the except clauses determine which one counts most, you could instead do something else. One alternative idea would be to take the "first in DFS order" and see if it matches any of the except clauses, and if so, process that one.  If not, then pick the next, and see if it matches, until one is found that matches, and can be processed.

On 3/4/2021 1:41 AM, Irit Katriel wrote:

On Thu, Mar 4, 2021 at 1:38 AM Glenn Linderman <v+python@g.nevcal.com <mailto:v%2Bpython@g.nevcal.com>> wrote:

On 3/3/2021 2:49 PM, Irit Katriel via Python-Dev wrote:

...

That's an interesting idea.

Do you mean that one exception gets handled and the rest of the group is reraised? Or discarded?

The value of sys.exc_info() (and the e in "except T as e:") needs to be a single naked exception. So if there is more than one match in the group we would need to pick one (let's say the first in DFS order).

If we do this, then we have this situation. Before ExceptionGroups, you got to choose which of the exceptions you have is the most important, and you raised only that one. Now you raise a bunch of them and the order of the except clauses in caller's code determines which one of them counts and which ones are discarded. What do you make of that?

You _could_ implement it as you said, but remember, you that with this idea, you are changing how except clauses work—so instead of making the order of the except clauses determine which one counts most, you could instead do something else.

One alternative idea would be to take the "first in DFS order" and see if it matches any of the except clauses, and if so, process that one.  If not, then pick the next, and see if it matches, until one is found that matches, and can be processed.

Or we could make it explicit:

add an optional arg to ExceptionGroup like ExceptionGroup("eg", list_of_exceptions, singleton=None)

In the example of atexit, where currently it raises only the last exception from your callbacks, it will instead raise

ExceptionGroup("atexit errors", all_exceptions, singleton=last_exception)

Then except* works as before, ignoring the singleton. But except matches the singleton.

And there's no magic where you can be surprised about which exception except chose to look at.

I like explicit, and avoiding magic. And this gives a compatibility story for outer loops that except: Exception, and even for others cases that are not recoded for ExceptionGroup handling. And I guess what you are citing is a precedent from atexit, for raising the last one. And I guess in cases other than atexit, when raising an ExceptionGroup, the coder of the new feature would still get more of a choice about which Exception is more important, rather than the coder of the except clauses.  One could quibble that if ValueError and IndexError were both part of the ExceptionGroup, that if the except clauses expected either might happen, and listed them in ValueError and IndexError order, that the intention might have been that ValueError was more interesting to the except clause coder, whereas if the group is raised with the IndexError as the singleton, that the group coder has the opposite intention. I think it is more likely that the except clause coder simply knew they were mutually exclusive and that the order of the clauses didn't matter. Thinking about the above a bit, the only existing except clause sequence that would matter would be if both a base exception class and a derived class were both listed in the except clauses.  The derived exception class should be listed before the base exception class or it wouldn't get processed. So it is not clear that the order of the except clauses really indicates any priority of interest on the part of the except clause coder?

Le 03/03/2021 à 23:49, Irit Katriel via Python-Dev a écrit :

On Wed, Mar 3, 2021 at 10:39 PM Greg Ewing <greg.ewing@canterbury.ac.nz <mailto:greg.ewing@canterbury.ac.nz>> wrote:

[...] In other words, the only difference between except and except* would be that multiple except* clauses can be run, whereas only one except clause will run (the first one that matches something in the ExceptionGroup).

Is there any good reason not to do things that way?

That's an interesting idea. 

Do you mean that one exception gets handled and the rest of the group is reraised? Or discarded? [...]

Hi, I'll take a shoot at this, just to see how it tastes… So, let's say: When an exception group reaches a set of traditional "except" clauses, those are examined one after the other, in the order they are in the code. That way, exceptions matched by several clauses will cause the first one to run, same as today. A subgroup is built with the subset of exceptions matched by the examined clause, as the PEP specifies for "except*". If this subgroup is None, the clause is not selected, and the next clause, if any, is examined. On the contrary, if the subgroup contains at least one matched exception, the clause is selected and no other clause will run (again, same as today). Exceptions not part of the subgroup are discarded. The clause body is then run just once (so the boss only gets one email about KeyboardInterrupt). If the clause uses the "as" form, the "as" variable is bound to one exception in the subgroup, which one is unspecified (at least for now). The other ones are discarded, except if a bare "raise" is reached (below). If a bare "raise" is reached while executing the body, the selected subgroup propagates out of the "try-except" construct. Justification: the whole group cannot propagate, because today a bare "raise" cannot reraise exceptions of a type not matched by the clause. However, if a single-type exception group is handled similar to a single exception in traditional "except" clauses, it is acceptable to let it propagate. So you would have: try: g=BaseExceptionGroup( [ValueError(), KeyboardInterrupt(), KeyboardInterrupt()]) raise g except RuntimeError: # doesn't match log_the_error() except KeyboardInterrupt as e: # builds s=g.subgroup(KeyboardInterrupt) email_the_boss(e) # tells the boss of any one error raise # reraises s except BaseException: # would match, but doesn't run launch_nuclear_attack() # BaseExceptionGroup([KeyboardInterrupt(), KeyboardInterrupt()]) # propagates further, a traditional "except KeyboardInterrupt" # would catch it. The ValueError is discarded. An interesting feature would be: when the matching clause has no "as", "except" behaves the same as "except*", apart from the fact that only one clause may run. Cheers, Baptiste

Hi, Le 05/03/2021 à 14:46, Irit Katriel via Python-Dev a écrit :

Whether you have "as" or not, the value of sys.exc_info() (which is what would be attached as the context to anything you raise in the except block) is the same. So these are not two different cases -- the only difference is whether or not you have a local variable set to sys.exc_info().

I don't really understand how this contradicts the behavior I proposed (i.e. that bare "raise" raises something else than what is bound to the "as" variable). Specifically, which of the following statements are hard and unbreakable rules, even when exception groups are involved: 1) sys.exc_info() must be the same object that is bound to the "as" variable, if any; 2) sys.exc_info() must be the same object that is attached as __context__; 3) sys.exc_info() must be the same object that is raised by a bare "raise". I'd say 1 is a hard rule because both objects are visible to user code, so backwards compatibility applies. Rules 2 and 3, however, are internal to the exception handling machinery, so I'm not so sure. Cheers, Baptiste

On Thu, Mar 4, 2021 at 4:46 PM Baptiste Carvello <devel2021@baptiste-carvello.net <mailto:devel2021@baptiste-carvello.net>> wrote:

Hi,

I'll take a shoot at this, just to see how it tastes… So, let's say:

When an exception group reaches a set of traditional "except" clauses, those are examined one after the other, in the order they are in the code. That way, exceptions matched by several clauses will cause the first one to run, same as today.

A subgroup is built with the subset of exceptions matched by the examined clause, as the PEP specifies for "except*". If this subgroup is None, the clause is not selected, and the next clause, if any, is examined. On the contrary, if the subgroup contains at least one matched exception, the clause is selected and no other clause will run (again, same as today). Exceptions not part of the subgroup are discarded.

The clause body is then run just once (so the boss only gets one email about KeyboardInterrupt). If the clause uses the "as" form, the "as" variable is bound to one exception in the subgroup, which one is unspecified (at least for now). The other ones are discarded, except if a bare "raise" is reached (below).

If a bare "raise" is reached while executing the body, the selected subgroup propagates out of the "try-except" construct. Justification: the whole group cannot propagate, because today a bare "raise" cannot reraise exceptions of a type not matched by the clause. However, if a single-type exception group is handled similar to a single exception in traditional "except" clauses, it is acceptable to let it propagate.

So you would have:

try:     g=BaseExceptionGroup(         [ValueError(), KeyboardInterrupt(), KeyboardInterrupt()])     raise g except RuntimeError:               # doesn't match     log_the_error() except  KeyboardInterrupt as e: # builds s=g.subgroup(KeyboardInterrupt)     email_the_boss(e)           # tells the boss of any one error     raise                       # reraises s except BaseException:           # would match, but doesn't run     launch_nuclear_attack()

# BaseExceptionGroup([KeyboardInterrupt(), KeyboardInterrupt()]) # propagates further, a traditional "except KeyboardInterrupt" # would catch it. The ValueError is discarded.

An interesting feature would be: when the matching clause has no "as", "except" behaves the same as "except*", apart from the fact that only one clause may run.

Cheers, Baptiste _______________________________________________ Python-Dev mailing list -- python-dev@python.org <mailto:python-dev@python.org> To unsubscribe send an email to python-dev-leave@python.org <mailto:python-dev-leave@python.org> https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/EXC6BQVH... Code of Conduct: http://python.org/psf/codeofconduct/

Just to be on the safe side here, I would want to asked a little bit further. On 02.03.21 12:22, Irit Katriel wrote:

1) What is the right "except pattern" when ExceptionGroup is introduced for the use cases described above (remote or concurrent python processes)?

If you want to catch the whole ExceptionGroup and format its traceback, then you can just do "except ExceptionGroup as eg" and then traceback.print_exception(eg).

The except* syntax is for when you want to handle only certain types of exceptions from the group, selectively.

Just to make sure, I understand this completely. In order to make it more tangible here is an example from the stdlib: https://github.com/python/cpython/blob/bf2e7e55d7306b1e2fce7dce767e8df5ff42c... As you can see, BaseException is caught multiple times as the only except-clause. _sendback_result(...) then used to transfer the return_val/exception back to parent process. How is the supposed way of handling a raised ExceptionGroup? a) will the existing code simply work as it? b) if not what are the changes to this lib specifically as a blueprint example for others Reading from the other subthread for this PEP, I am not 100% sure now that "except BaseException" will suffice if ExceptionGroup inherits from Exception instead of BaseException because it seems that ExceptionGroup somehow is handled specially. So, why I try to approach this very issue with existing code. Once that is clear, it should be easy for everybody to apply the same pattern to their code. Here is the copy from github: try:     r = call_item.fn(*call_item.args, **call_item.kwargs) except BaseException as e:     exc = _ExceptionWithTraceback(e, e.__traceback__)     _sendback_result(result_queue, call_item.work_id, exception=exc) else:     _sendback_result(result_queue, call_item.work_id, result=r)     del r Maybe even _sendback_result could benefit from using ExceptionGroup itself. You can see there another "except BaseException" in case of an error. But that's maybe a different topic.

2) What is the recommended approach of printing the traceback potentially incorporating multiple tracebacks - I couldn't find it in the PEP and tracebacks are a really neat tool for error hunting.

Part of the proposal of the PEP is that we teach the builtin traceback formatting code to display ExceptionGroups with all that information.

As long as there's the possibility to receive the usual (two-line) entry-based list, I guess that is enough for every one to work with it (e.g. hiding unnecessary tb entries).

The reference implementation has this, and the examples in the PEP were produced with it. Some of the examples (the early ones) show exceptions that were never raised so there is no traceback. But if you scroll down to the later examples, you see the output for exceptions with tracebacks, cause, context etc.

Ah I see them now. Thank you. :-)

We didn't put too much effort into making the traceback output pretty, so at the moment it's a draft. If there is an ascii artist out there who has suggestions on improving this, that would be great.

Well, yes. It's not really pretty. I would recommend a tree-based solution such as the following: Traceback (most recent call last):   File "<stdin>", line 3, in <module> ExceptionGroup | +---+   Traceback (most recent call last): |   |     File "<stdin>", line 3, in <module> |   |   ExceptionGroup: eg |   |   | |   |   +---+ ValueError: a |   | |   |   During handling of the above exception, another exception occurred: |   | |   |   Traceback (most recent call last): |   |     File "<stdin>", line 5, in <module> |   |   KeyError: 'x' | | +---+   Traceback (most recent call last): |   |     File "<stdin>", line 3, in <module> |   |   ExceptionGroup: eg |   |   | |   |   +---+ TypeError: b I used a similar pattern for the remote-execution lib and received good user feedback (even requesting color encoding for each layer of the tree (not the text), so people know what belongs together). Besides colors, I used https://en.wikipedia.org/wiki/Box-drawing_character but I guess pipes, dashes and pluses could suffice for CPython. One other remark from my side here: in the example of the PEP there seem to be missing a space before 'File "<stdin>"'. Comparing outer tbs with inner tbs, you can see that the "F" of "File" is not underneath the a of "Traceback" but underneath the "r" of it. Regards, Sven

Hey Irit, find my 3 answers below: On 03.03.21 13:17, Irit Katriel wrote:

Hi Sven,

I like your formatting suggestion, thanks. I will do something like that.

You're welcome.

I'm not sure I understand your question. ExceptionGroup is a subclass of Exception (which is a subclass of BaseException). So ExceptionGroup is caught by "except Exception" or "except BaseException".

1) So I understand "try-except BaseException" (cf. concurrent.futures) will work without fixes (i.e. produce the same results).

BaseExceptionGroup is a subclass only of BaseException so it is caught by "except BaseException" but not "except Exception". And ExceptionGroup is allowed to wrap only Exceptions while BaseException can wrap Exceptions and and BaseExceptions. Makes sense?

2) Can you add motivating examples for "BaseExceptionGroup vs ExceptionGroup" in the PEP? Right now, I only see what the consequences are but not why it was done this way. 3) Can you explain (and show the reasoning behind) this automatic choice in the PEP? Sounds a bit like hidden magic to me. Referring to: "The difference between them is that ExceptionGroup can only wrap Exception subclasses while BaseExceptionGroup can wrap any BaseException subclass. A factory method that inspects the nested exceptions and selects between ExceptionGroup and BaseExceptionGroup makes the choice automatic." Best Sven PS: the reason why I was a bit puzzled by the BaseExceptionGroup/ExceptionGroup issue is that: - if it doesn't matter (so we do it automatically, because we do not want to bother anybody), why do we need ExceptionGroup at all, BaseExceptionGroup seems more flexible? - if it does matter, why is the choice automatic and what if it was the wrong choice?

Hi Irit, makes sense. So, in case of a *mixed-type ExceptionGroup,* SystemExit wins and forces the program to exit. Could you add your reasoning to the PEP? This would help future readers and illustrates the chain of thoughts. It might be obvious to you but from the outside it is really a long journey. I actually liked your wording below and the explicitly writing down the consequence I mentioned makes it very clear why this complication exists. Best, Sven On 03.03.21 21:13, Irit Katriel wrote:

Hi Sven,

This is all about the popularity of "except Exception".

Look at the hierarchy of builtin exceptions: https://docs.python.org/3/library/exceptions.html#exception-hierarchy <https://docs.python.org/3/library/exceptions.html#exception-hierarchy>

Most exceptions are subclasses of Exception. There are a few which are not, because they typically mean "this process should exit" so you should not usually handle them in your code. People use "except Exception" as a way to "catch almost everything, but not the critical stuff like SystemExit".

If we make ExceptionGroup be a BaseException, then "except Exception" doesn't catch it. So we make it a subclass of Exception. But then we can't make it wrap things like SystemExit, which people expect will not be caught by "except Exception".  So we add BaseExceptionGroup, which is a subclass of BaseException and therefore is not caught by "except Exception", so it can wrap SystemExit.

Why is the choice automated?  Because it can be. You look at what you're wrapping. If it's all subclasses of Exception, then it can be ExceptionGroup. If there are BaseExceptions, then it needs to be BaseExceptionGroup. There is no reason to ever do anything else.

I hope that makes sense.

On Wed, Mar 3, 2021 at 7:32 PM Sven R. Kunze <srkunze@mail.de <mailto:srkunze@mail.de>> wrote:

Hey Irit,

find my 3 answers below:

On 03.03.21 13:17, Irit Katriel wrote: > Hi Sven, > > I like your formatting suggestion, thanks. I will do something like that.

You're welcome.

> > I'm not sure I understand your question. ExceptionGroup is a subclass > of Exception (which is a subclass of BaseException). So ExceptionGroup > is caught by "except Exception" or "except BaseException".

1) So I understand "try-except BaseException" (cf. concurrent.futures) will work without fixes (i.e. produce the same results).

> BaseExceptionGroup is a subclass only of BaseException so it is caught > by "except BaseException" but not "except Exception". And > ExceptionGroup is allowed to wrap only Exceptions while BaseException > can wrap Exceptions and and BaseExceptions. Makes sense?

2) Can you add motivating examples for "BaseExceptionGroup vs ExceptionGroup" in the PEP? Right now, I only see what the consequences are but not why it was done this way.

3) Can you explain (and show the reasoning behind) this automatic choice in the PEP? Sounds a bit like hidden magic to me.

Referring to: "The difference between them is that ExceptionGroup can only wrap Exception subclasses while BaseExceptionGroup can wrap any BaseException subclass. A factory method that inspects the nested exceptions and selects between ExceptionGroup and BaseExceptionGroup makes the choice automatic."

Best Sven

PS:

the reason why I was a bit puzzled by the BaseExceptionGroup/ExceptionGroup issue is that: - if it doesn't matter (so we do it automatically, because we do not want to bother anybody), why do we need ExceptionGroup at all, BaseExceptionGroup seems more flexible? - if it does matter, why is the choice automatic and what if it was the wrong choice?