[Python-Dev] PEP 342/343 status?

[Guido van Rossum]

[Guido]
> >Yes, the generator does clear its f_back when it's suspended.

[Phillip]
> I realize this won't fix all your worries; I just want to rule out
> this one *particular* form of cycle as a possibility; i.e., to show
> that mere reference to a generator-iterator in a frame does not
> create a cycle:
> 
> 
>      callerframe ---------> traceback2
>         |     ^                |  |
>         |     |                |  |
>         |     +----------------+  |
>         v                         v
>      geniter -> genframe -> traceback1
>                     ^          |
>                     |          |
>                     +----------+
> 
> As you can see, the geniter itself doesn't have a reference to its
> calling frame, so as soon as the highest-level traceback object is
> released, the cycle collector should release the upper cycle,
> allowing the geniter to complete, and releasing the lower cycle.
> 
> The scenario assumes, by the way, that the traceback object
> referenced by a frame includes a pointer to that same frame, which
> I'm not sure is the case.  I was under the impression that the
> current frame is only added to the traceback when the frame is
> exited, in which case the two cycles shown above wouldn't even
> exist; each traceback would be pointing to the *next* frame down,
> and there would be no cycles at all.  It seems to me that this would
> almost have to be the design, since tracebacks existed before cyclic
> GC did.

Alas, your assumption is valid; this would indeed cause a cycle, much
to the despair of early Python programmers.  There used to be a whole
body of literature about the best way to avoid this (never save a
traceback, or if you do, clear it when you're done with it before
exiting the frame).  When you raise and immediately catch an
exception, there is a single traceback object that references the
current frame (the frame where it was raised *and* caught).  So if you
store sys.exc_info() in a local, you have a cycle already:

  try:
      raise Exception
  except:
      x = sys.exc_info()[2]   # save the traceback

Now we have the following cycle (with slightyl more detail than your
diagram):

             tb_frame
    frame <------------- traceback
      |                       ^
      |                       |
      v         'x'           |
   f_locals ------------------+

BTW, note the repercussions this has for Ping's PEP 344 -- because
the Exception instance references the traceback in that proposal, all
code that catches an exception into a variable creates a cycle, like
this:

  try:
      raise Exception
  except Exception, err:
      pass

This would creates the following cycle:

             tb_frame
    frame <------------- traceback
      |                       ^
      |                       | __traceback__
      v                       |
   f_locals ---------------> err

The good news in all this is that none of these objects has a __del__
method in the proposal; only the 'geniter' object would have one, and
getting it involved in a cycle does seem like rather unlikely.  I
hereby declare my worries unwarranted and will happily add language to
the revamped PEP 343 that a geniter object should have a tp_del slot
and a corresponding __del__ attribute.  This further complicates
has_finalizer() in gcmodule.c, to the point where the latter might
have to be turned into an internal slot.

> >Sure. But I still have some reservations, since cycles can pop up
> >in the strangest of places (especially when tracebacks are involved
> >-- tracebacks have been causing problems due to cycles and keeping
> >variables alive almost since Python's inception). I posted about
> >this a while ago, but don't recall seeing a response that took my
> >fear away.
> 
> Well, I can't prove that it's not possible to create such cycles,
> certainly.  But maybe we should finally get rid of the deprecated
> sys.exc_type/value/traceback variables, so that they can't root any
> cycles?

I sort of doubt that these are the main source of live cycles.  After
all, they are reset whenever a frame is popped (grep the sources for
reset_exc_info).

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)