[Python-Dev] need help with frozen module/marshal/gc issue involving sub-interpreters for importlib bootstrapping

[Brett Cannon]

So my grand quest for bootstrapping importlib into CPython is damn close to
coming to fruition; I have one nasty bug blocking my way and I can't figure
out what could be causing it. I'm hoping someone here will either know the
solution off the top of their head or will have the time to have a quick
look to see if they can figure it out as my brain is mush at this point.

First, the bug tracking all of this is http://bugs.python.org/issue2377 and
the repo where I have been doing my work is ssh://
hg at hg.python.org/sandbox/bcannon/#bootstrap_importlib (change as needed if
you want an HTTPS checkout). Everything works fine as long as you don't use
sub-interpreters via test_capi (sans some test failures based on some
assumptions which can easily be fixed; the bug I'm talking about is the
only real showstopper at this point).

Here is the issue: if you run test_capi the code triggers an assertion of
``test_subinterps (__main__.TestPendingCalls) ... Assertion failed:
(gc->gc.gc_refs != 0), function visit_decref, file Modules/gcmodule.c, line
327.``. If you run the test under gdb you will discover that the assertion
is related to ref counts when collecting for a generation (basically the
ref updating is hitting 0 when it shouldn't).

Now the odd thing is that this is happening while importing frozen module
code (something I didn't touch) which is calling marshal (something else I
didn't touch) and while it is in the middle of unmarshaling the frozen
module code it is triggering the assertion.

Does anyone have any idea what is going on? Am I possibly doing something
stupid with refcounts which is only manifesting when using
sub-interpreters? All relevant code for bootstrapping is contained in
Python/pythonrun.c:import_init() (with a little tweaking in the _io module
to delay importing the os module and making import.c always use __import__
instead of using the C code). I'm storing the __import__ function in the
PyInterpreterState to keep separate state from the other interpreters (i.e.
separate sys modules so as to use the proper sys.modules, etc.). But as I
said, this all works in a single interpreter view of the world (the entire
test suite doesn't trigger a nasty error like this).

Thanks for any help people can provide me on this now 5 year quest to get
this work finished.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120206/e8a8a824/attachment.html>