[Python-Dev] Traceback problem

[Guido van Rossum]

(Picking up an old thread.)

[Guido]
> > Watch out though.  There are situations where an exception needs
> > to be stored but no frame is available (when executing purely in
> > C).  There is always a thread state.

[Christian]
> I've been sitting a while over this puzzle now.
> 
> tstate has two different kinds of exceptions:
> There are tstate->exc_XXX and tstate->curexc_XXX.
> 
> I have been searching through the whole source trunk
> to validate my thought:
> 
> All internal stuff is only concerned with handling
> tstate->curexc_XXX.

Correct.  This is the "hot" exception that is set by PyErr_SetString()
c.s., cleared by PyErr_Clear(), and so on.

> The tstate->exc_XXX is *only* used in ceval.c .

Once an exception is caught by an except clause, it is transferred
from tstate->curexc_XXX to tstate->exc_XXX, from which sys.exc_info()
can pick it up.

> References to tstate->exc_XXX are only in
> pystate.c (clearing stuff) and sysmodule.c (accessing stuff).
> The only place where tstate->exc_XXX is filled with life
> is ceval.c, which indicates that this is purely interpreter-
> -related and has nothing to do with the internal exception
> state. It is eval_frame which checks for exceptions, normalizes
> them and turns them into interpreter-level exceptions,
> around line 2360 of ceval.c .

Correct.

> After stating that, I conclude that tstate.exc_XXX can only
> be in use if there is an existing interpreter with an existing
> frame. Nobody else makes use of this structure.
> So, whenever you have to save this, you can expect a valid
> frame waiting in f_back that will be able to take it.

Right.  Now let me explain the complicated dance with
frame->f_exc_XXX.  Long ago, when none of this existed, there were
just a few globals: one set corresponding to the "hot" exception, and
one set corresponding to sys.exc_type etc.  The problem was that in
code like this:

   try:
      "something that may fail"
   except "some exception":
      "do something else first"
      "print the exception from sys.exc_type etc."

if "do something else first" invoked something that raised and caught
an exception, sys.exc_type etc. were overwritten.  That was a frequent
cause of subtle bugs.  I fixed this by changing the semantics as
follows:

  - Within one frame, sys.exc_XXX will hold the last exception caught
    *in that frame*.

  - But initially, and as long as no exception is caught in a given
    frame, sys.exc_XXX will hold the last exception caught in the
    previous frame (or the frame before that, etc.).

The first bullet fixed the bug in the above example.  The second
bullet was for backwards compatibility: it was (and is) common to
have a function that is called when an exception is caught, and to
have that function access the caught exception via sys.exc_XXX.
(Example: traceback.print_exc()).

At the same time I fixed the problem that sys.exc_type and friends
weren't thread-safe, by introducing sys.exc_info() which gets it from
tstate; but that's really a separate improvement.

The reset_exc_info() function in ceval.c restores the tstate->exc_XXX
variables to what they were before the current frame was called.  The
set_exc_info() function saves them on the frame so that
reset_exc_info() can restore them.  The invariant is that
frame->f_exc_XXX is NULL iff the current frame never caught an
exception (where "catching" an exception applies only to successful
except clauses); and if the current frame ever caught an exception,
frame->f_exc_XXX is the exception that was stored in tstate->exc_XXX
at the start of the current frame.

Now I hope you'll understand why this was never documented
exactly. :-)

> (This all under the maybe false assumption that I'm not wrong).

No; I guess I was wrong in the quoted text at the top. :-)

> Still not proposing a change. But thanks for the time,
> I understood quite a lot more of the internals, now.

Great!  Hope this message has shed some additional light.

Kevin, I'll try to get to your patch next.

--Guido van Rossum (home page: http://www.python.org/~guido/)