[Python-Dev] need help with frozen module/marshal/gc issue involving sub-interpreters for importlib bootstrapping

[Guido van Rossum]

Usually this means that you're not doing an INCREF in a place where you
should, and the object is kept alive by something else. Do you know which
object it is? That might really help... Possibly deleting the last
subinterpreter makes the refcount of that object go to zero. Of course it
could also be that you're doing a DECREF you shouldn't be doing... But the
identity of the object seems key in any case.

--Guido

On Mon, Feb 6, 2012 at 6:57 AM, Brett Cannon <brett at python.org> wrote:

> So my grand quest for bootstrapping importlib into CPython is damn close
> to coming to fruition; I have one nasty bug blocking my way and I can't
> figure out what could be causing it. I'm hoping someone here will either
> know the solution off the top of their head or will have the time to have a
> quick look to see if they can figure it out as my brain is mush at this
> point.
>
> First, the bug tracking all of this is http://bugs.python.org/issue2377and the repo where I have been doing my work is ssh://
> hg at hg.python.org/sandbox/bcannon/#bootstrap_importlib (change as needed
> if you want an HTTPS checkout). Everything works fine as long as you don't
> use sub-interpreters via test_capi (sans some test failures based on some
> assumptions which can easily be fixed; the bug I'm talking about is the
> only real showstopper at this point).
>
> Here is the issue: if you run test_capi the code triggers an assertion of
> ``test_subinterps (__main__.TestPendingCalls) ... Assertion failed:
> (gc->gc.gc_refs != 0), function visit_decref, file Modules/gcmodule.c, line
> 327.``. If you run the test under gdb you will discover that the assertion
> is related to ref counts when collecting for a generation (basically the
> ref updating is hitting 0 when it shouldn't).
>
> Now the odd thing is that this is happening while importing frozen module
> code (something I didn't touch) which is calling marshal (something else I
> didn't touch) and while it is in the middle of unmarshaling the frozen
> module code it is triggering the assertion.
>
> Does anyone have any idea what is going on? Am I possibly doing something
> stupid with refcounts which is only manifesting when using
> sub-interpreters? All relevant code for bootstrapping is contained in
> Python/pythonrun.c:import_init() (with a little tweaking in the _io module
> to delay importing the os module and making import.c always use __import__
> instead of using the C code). I'm storing the __import__ function in the
> PyInterpreterState to keep separate state from the other interpreters (i.e.
> separate sys modules so as to use the proper sys.modules, etc.). But as I
> said, this all works in a single interpreter view of the world (the entire
> test suite doesn't trigger a nasty error like this).
>
> Thanks for any help people can provide me on this now 5 year quest to get
> this work finished.
>
> -Brett
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120206/01ceeaa9/attachment.html>