On 2020-04-11 5:40 a.m., Stephen J. Turnbull wrote:
Soni L. writes:
so, for starters, here's everything I'm worried about.
in one of my libraries (yes this is real code. all of this is taken from stuff I'm deploying.) I have the following piece of code:
def _extract(self, obj): try: yield (self.key, obj[self.key]) except (TypeError, IndexError, KeyError): if not self.skippable: raise exceptions.ValidationError
due to the wide range of supported objects, I can't expect the TypeError to always come from my attempt to index into a set, or the IndexError to always come from my attempt to index into a sequence, or the KeyError to always come from my attempt to index into a mapping. those could very well be coming from a bug in someone's weird sequence/mapping/set implementation. I have no way of knowing!
I hope that exclamation point doesn't mean you think anybody doesn't understand the problem by now. I think most serious programmers have run into "expected" Exceptions masking unexpected errors. What we don't understand is (a) how your proposed fix is supposed to work without a tremendous effort by third parties (and Python itself in the stdlib), (b) if you do presume enormous amounts of effort, what is in it for the third parties, and (c) why doesn't asking the third parties to create finely graduated hierarchies of exceptions do as well (if not better) than the syntax change?
The grungy details:
So as far as I can tell, your analysis above is exactly right. You have no way of knowing, and with the syntax change *you* still will have no way of knowing. You will be dependent on the implementer to tell you, ie, the obj's getter must distinguish between an actual "this key is not in the object" Error, and an implementation bug. You're asking everybody else to consistently distinguish between KeyErrors you think are KeyErrors and KeyErrors you think are actually bugs. I don't see how they can know what you think, and of course if you're right that they are actually bugs, by that very token the relevant developer probably doesn't know they are there!
And I don't think they'll care enough to make the effort even if they are aware of the difference. Because, if they did care, they already can make the effort by using a wider variety of builtin exceptions, or by adding attributes to the exceptions they raise, or deriving subclasses of the various exceptions that might be raised, or of RuntimeError (and they might disagree on what exception *should* be the base exception!) But they don't do any of those very often. And I've seen principled objections to the proliferation of specialized exceptions (sorry, no cite; ignore if you like). I don't recall them clearly but I suppose some of them would apply to this syntax.
"exception spaces" would enable me to say "I want your (operator/function/whatnot) to raise some errors in my space, so I don't confuse them with bugs in your space instead". and they'd get me exactly that.
But how do you say that to me? I'm not your subcontractor, I'm an independent developer. I write a library, I *don't know* what "your" exception spaces are. It's the other way around, I guess? That is, I raise exceptions in *my* space and some in builtin space and maybe even some in imported spaces and you have to catch them explicitly to distinguish. I think you're going to have to do a lot of studying and a lot of extra work on exception handling to catch only the exceptions you intend to. I'm not sure it's going to be worth it to do the necessary research to be able to write
# these are KeyErrors you deem to be the real McCoy except KeyError in SpaceX | SpaceY | SpaceZ: handle_spacey_errors() # these are exceptions from spaces that you think are probably bugs except KeyError: raise OutDamnSpotErrorwhenever you're handling potential exceptions from third-party code. And I think you have to do it consistently, because otherwise you may be missing lots of masked bugs. If it's not worth it for *you* to use it *consistently*, why would the tremendous effort imposed on others to lay the foundations for your "occasional" use be worth it?
One of the main points of exceptions as a mechanism is that they "bubble up" until caught. If they occurred deep in the import hierarchy, you may consider that a bug if middle layers don't distinguish, but the developer-in-the-middle may disagree with you if you're handing their library inputs they never intended to handle. I guarantee you lose that argument, unless you are *very* nice to Ms. DITM.
Bottom line: I just don't see how this can work in practice unless we make the "in" clause mandatory, which I suspect is a Python 4.0 compatibility break of Python 3.0 annoyance. I know it would annoy me: my exception handling is almost all of the "terminate processing, discard pending input as possibly corrupt, reinitialize, restart processing" variety. I just don't care about "masked bugs".
for starters: operator variants wherever exceptions are used as part of API contract, and add a kwarg to functions that explicitly define exceptions as part of their API contract.
if missing, defaults to None. all exceptions not raised in a specific channel will not match a specific channel, but will match None.
so e.g. you do:
def foo(espace=None): if thing(): raise Bar in espace
which means if the API user isn't expecting to have to handle an error, they can just do:
x = foo()
and then they won't handle it. but if they want to handle it, they can do:
try: x = foo(espace=my_espace) except Bar in my_espace: ...
and then they'll handle it. alternatively, they can let their caller handle it by having the espace come from the caller.
def bar(espace=None): x = foo(espace=espace)
and then the caller is expected to handle it, *if they so choose*.
by default the None handler works even for exceptions in a channel:
try: x = foo(espace=NotNone) except Bar: ...
this'll also catch it. this is for backwards compatibility so existing still code catches exceptions without caring where they come from, and also so you can still have a "catches everything" construct.
we'll also need operator variants. I was thinking of the following:
x.in foo bar # for attribute access x[in foo bar] # for item access x +in foo bar # for math # etc
(yes, these are very ugly. this is one of the unfortunate things of trying to retrofit this into an existing language. but they're more likely to catch bugs so I'm not worried.)
which would call new methods that take in an espace. (we could retrofit existing methods with an optional espace but then we lose the ability to catch everything and shove it in the appropriate espace anyway. this would significantly reduce the effort required to make the stdlib work with this new thing.)
maybe this should be python 4. I don't think any language's ever done Rust-like explicit error handling (i.e. avoiding the issue of swallowing bugs) using exceptions before. it could even be sugar for "except Foo: if Foo.__channel__ != channel: raise" or something, I guess. but... it's definitely worth it. and I don't think it'll be a python 3-level annoyance, if implemented like this. .
it's also backwards compatible.
I don't think so, if it's to be useful to you or anyone else. See discussion preceding "unless we make it mandatory," above.
anyway, I'm gonna keep pushing for this because it's probably the easiest way to retrofix explicit error handling into python,
You're not going to get that, I'm pretty sure, except to the extent that you already have it in the form of Exception subclasses. Python is not intended to make defects hard to inject the way Rust is. (Those are observations, not policy statements. I don't have much if any influence over policy. ;-) I just know a lot about incentives in my professional capacity.)
Steve