How to pop the interpreter's stack?

[Robert Kern]

On 12/16/10 6:33 PM, Steven D'Aprano wrote:
> On Thu, 16 Dec 2010 10:39:34 -0600, Robert Kern wrote:
>
>> On 12/16/10 10:23 AM, Steven D'Aprano wrote:
>>> On Thu, 16 Dec 2010 07:29:25 -0800, Ethan Furman wrote:
>>>
>>>> Tim Arnold wrote:
>>>>> "Ethan Furman"<ethan at stoneleaf.us>   wrote in message
>>>>> news:mailman.4.1292379995.6505.python-list at python.org...
>>>>>> kj wrote:
>>>>>>> The one thing I don't like about this strategy is that the
>>>>>>> tracebacks of exceptions raised during the execution of __pre_spam
>>>>>>> include one unwanted stack level (namely, the one corresponding to
>>>>>>> __pre_spam itself).
>>> [...]
>>>> A decorator was one of the items kj explicity didn't want.  Also,
>>>> while it would have a shallower traceback for exceptions raised during
>>>> the __pre_spam portion, any exceptions raised during spam itself would
>>>> then be one level deeper than desired... while that could be masked by
>>>> catching and (re-?)raising the exception in the decorator, Steven had
>>>> a very good point about why that is a bad idea -- namely, tracebacks
>>>> shouldn't lie about where the error is.
>>>
>>> True, very true... but many hours later, it suddenly hit me that what
>>> KJ was asking for wasn't *necessarily* such a bad idea. My thought is,
>>> suppose you have a function spam(x) which raises an exception. If it's
>>> a *bug*, then absolutely you need to see exactly where the error
>>> occurred, without the traceback being mangled or changed in any way.
>>>
>>> But what if the exception is deliberate, part of the function's
>>> documented behaviour? Then you might want the exception to appear to
>>> come from the function spam even if it was actually generated inside
>>> some private sub-routine.
>>
>> Obfuscating the location that an exception gets raised prevents a lot of
>> debugging (by inspection or by pdb), even if the exception is
>> deliberately raised with an informative error message. Not least, the
>> code that decides to raise that exception may be buggy. But even if the
>> actual error is outside of the function (e.g. the caller is passing bad
>> arguments), you want to at least see what tests the __pre_spam function
>> is doing in order to decide to raise that exception.
>
> And how do you think you see that from the traceback? The traceback
> prints the line which actually raises the exception (and sometimes not
> even that!), which is likely to be a raise statement:
>
>>>> import example
>>>> example.func(42)
> Traceback (most recent call last):
>    File "<stdin>", line 1, in<module>
>    File "example.py", line 3, in func
>      raise ValueError('bad value for x')
> ValueError: bad value for x
>
> The actual test is:
>
> def func(x):
>      if x>  10 and x%2 == 0:
>          raise ValueError('bad value for x')
>
> but you can't get that information from the traceback.

But I can get the line number and trivially go look it up. If we elide that 
stack frame, I have to go hunting and possibly make some guesses. Depending on 
the organization of the code, I may have to make some guesses anyways, but if I 
keep the decision to raise an exception close to the actual raising of the 
exception, it makes things a lot easier.

> Python's exception system has to handle two different situations: buggy
> code, and bad data. It's not even clear whether there is a general
> distinction to be made between the two, but even if there's not a general
> distinction, there's certainly a distinction which we can *sometimes*
> make. If a function contains a bug, we need all the information we can
> get, including the exact line that causes the fault. But if the function
> deliberately raises an exception due to bad input, we don't need any
> information regarding the internals of the function (assuming that the
> exception is sufficiently detailed, a big assumption I grant you!). If I
> re-wrote the above func() like this:
>
> def func(x):
>      if !(x<= 10):
>          if x%2 != 0:
>              pass
>          else:
>              raise ValueError('bad value for x')
>      return
>
> I would have got the same traceback, except the location of the exception
> would have been different (line 6, in a nested if-block). To the caller,
> whether I had written the first version of func() or the second is
> irrelevant. If I had passed the input validation off to a second
> function, that too would be irrelevant.

The caller doesn't care about tracebacks one way or the other, either. Only 
someone *viewing* the traceback cares as well as debuggers like pdb. Eliding the 
stack frame neither helps nor harms the caller, but it does substantially harm 
the developer viewing tracebacks or using a debugger.

> I don't expect Python to magically know whether an exception is a bug or
> not, but there's something to be said for the ability to turn Python
> functions into black boxes with their internals invisible, like C
> functions already are. If (say) math.atan2(y, x) raises an exception, you
> have no way of knowing whether atan2 is a single monolithic function, or
> whether it is split into multiple pieces. The location of the exception
> is invisible to the caller: all you can see is that atan2 raised an
> exception.

And that has frustrated my debugging efforts more often than I can count. I 
would dearly love to have a debugger that can traverse both Python and C stack 
frames. This is a deficiency, not a benefit to be extended to pure Python functions.

>> Tracebacks are inherently over-verbose. This is necessarily true because
>> no algorithm (or clever programmer) can know all the pieces of
>> information that the person debugging may want to know a priori. Most
>> customizations of tracebacks *add* more verbosity rather than reduce it.
>> Removing one stack level from the traceback barely makes the traceback
>> more readable and removes some of the most relevant information.
>
> Right. But I have thought of a clever trick to get the result KJ was
> asking for, with the minimum of boilerplate code. Instead of this:
>
>
> def _pre_spam(args):
>      if condition(args):
>          raise SomeException("message")
>      if another_condition(args):
>          raise AnotherException("message")
>      if third_condition(args):
>          raise ThirdException("message")
>
> def spam(args):
>      _pre_spam(args)
>      do_useful_work()
>
>
> you can return the exceptions instead of raising them (exceptions are
> just objects, like everything else!), and then add one small piece of
> boilerplate to the spam() function:
>
>
> def _pre_spam(args):
>      if condition(args):
>          return SomeException("message")
>      if another_condition(args):
>          return AnotherException("message")
>      if third_condition(args):
>          return ThirdException("message")
>
> def spam(args):
>      exc = _pre_spam(args)
>      if exc: raise exc
>      do_useful_work()

And that makes post-mortem pdb debugging into _pre_spam impossible. Like I said, 
whether the bug is inside _pre_spam or is in the code that is passing the bad 
argument, being able to navigate stack frames to where the code is deciding that 
there is an exceptional condition is important.

Kern's First Maxim: Raise exceptions close to the code that decides to raise an 
exception.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco