[pypy-dev] The fate of gc-del

[Armin Rigo]

Hi all,

I'm trying again to look at the gc-del branch, but I'm kind of
failing.  The current state is:

    http://buildbot.pypy.org/summary?branch=gc-del

The failures are all more or less obscure.  The issues all may have to
do with subtly broken things in the finalizer ordering code.  But now
I believe that the attempted implementation was a bad idea.

For reference: you call "rgc.register_finalizer(method)" on objects
where you want a finalizer to be called (instead of "def __del__()",
which is now limited to "lightweight" destructors).  It's a good idea
by itself because (1) you can register a finalizer only after the
object is ready, and not have to care for seeing half-initialized
objects in the finalizer; and (2) you can register a finalizer at any
time, or not at all, on various instances of the same class, which is
very convenient for implementing app-level objects that may or may not
have a user-level __del__ method.

The issue is that the finalizers (particularly the app-level ones)
should only be called at known points in time, e.g. between bytecodes,
and not from random places.  I tried to do this: if a finalizer raises
"FinalizeLater", then it suspends calling finalizers --- both the
current one and all future ones.  Later, between bytecodes, we call
rgc.progress_through_finalizer_queue() to resume.  This looks innocent
enough, but it's a major mess.  First in test runs: because the
finalizer queue is not attached to any space, and raising
FinalizeLater suspends calling any finalizer from there, in order to
maintain the correct order.  Also, I suspect, in real runs.  We're
getting cases where the app-level __del__() is not called and I'm
really unsure why, but I suspect that some finalizer raises
FinalizeLater without making sure that we'll later call
rgc.progress_through_finalizer_queue().  Basically, this API looks
broken now.

It becomes messy when you consider that some helpers in rpython.rlib
really need rgc.register_finalizer(), like rmmap.  These helpers don't
know about the space.  The hack of the previous paragraph was an
attempt to still have rmmap objects finalized in the globally correct
order.

Help! :-)  Any cleaner ideas welcome!  Would it make sense to *not*
call automatically any finalizer, and instead require that the RPython
program calls rgc.flush_finalizer_queue() regularly?  Maybe with some
minimal signal-like event from the GC to mean "there are finalizers
now"?  It would seem closer to what e.g. Java does, which just puts
finalizable objects on a queue from the GC.


A bientôt,

Armin.