need help with frozen module/marshal/gc issue involving sub-interpreters for importlib bootstrapping
So my grand quest for bootstrapping importlib into CPython is damn close to coming to fruition; I have one nasty bug blocking my way and I can't figure out what could be causing it. I'm hoping someone here will either know the solution off the top of their head or will have the time to have a quick look to see if they can figure it out as my brain is mush at this point. First, the bug tracking all of this is http://bugs.python.org/issue2377 and the repo where I have been doing my work is ssh:// hg@hg.python.org/sandbox/bcannon/#bootstrap_importlib (change as needed if you want an HTTPS checkout). Everything works fine as long as you don't use sub-interpreters via test_capi (sans some test failures based on some assumptions which can easily be fixed; the bug I'm talking about is the only real showstopper at this point). Here is the issue: if you run test_capi the code triggers an assertion of ``test_subinterps (__main__.TestPendingCalls) ... Assertion failed: (gc->gc.gc_refs != 0), function visit_decref, file Modules/gcmodule.c, line 327.``. If you run the test under gdb you will discover that the assertion is related to ref counts when collecting for a generation (basically the ref updating is hitting 0 when it shouldn't). Now the odd thing is that this is happening while importing frozen module code (something I didn't touch) which is calling marshal (something else I didn't touch) and while it is in the middle of unmarshaling the frozen module code it is triggering the assertion. Does anyone have any idea what is going on? Am I possibly doing something stupid with refcounts which is only manifesting when using sub-interpreters? All relevant code for bootstrapping is contained in Python/pythonrun.c:import_init() (with a little tweaking in the _io module to delay importing the os module and making import.c always use __import__ instead of using the C code). I'm storing the __import__ function in the PyInterpreterState to keep separate state from the other interpreters (i.e. separate sys modules so as to use the proper sys.modules, etc.). But as I said, this all works in a single interpreter view of the world (the entire test suite doesn't trigger a nasty error like this). Thanks for any help people can provide me on this now 5 year quest to get this work finished. -Brett
Usually this means that you're not doing an INCREF in a place where you should, and the object is kept alive by something else. Do you know which object it is? That might really help... Possibly deleting the last subinterpreter makes the refcount of that object go to zero. Of course it could also be that you're doing a DECREF you shouldn't be doing... But the identity of the object seems key in any case. --Guido On Mon, Feb 6, 2012 at 6:57 AM, Brett Cannon <brett@python.org> wrote:
So my grand quest for bootstrapping importlib into CPython is damn close to coming to fruition; I have one nasty bug blocking my way and I can't figure out what could be causing it. I'm hoping someone here will either know the solution off the top of their head or will have the time to have a quick look to see if they can figure it out as my brain is mush at this point.
First, the bug tracking all of this is http://bugs.python.org/issue2377and the repo where I have been doing my work is ssh:// hg@hg.python.org/sandbox/bcannon/#bootstrap_importlib (change as needed if you want an HTTPS checkout). Everything works fine as long as you don't use sub-interpreters via test_capi (sans some test failures based on some assumptions which can easily be fixed; the bug I'm talking about is the only real showstopper at this point).
Here is the issue: if you run test_capi the code triggers an assertion of ``test_subinterps (__main__.TestPendingCalls) ... Assertion failed: (gc->gc.gc_refs != 0), function visit_decref, file Modules/gcmodule.c, line 327.``. If you run the test under gdb you will discover that the assertion is related to ref counts when collecting for a generation (basically the ref updating is hitting 0 when it shouldn't).
Now the odd thing is that this is happening while importing frozen module code (something I didn't touch) which is calling marshal (something else I didn't touch) and while it is in the middle of unmarshaling the frozen module code it is triggering the assertion.
Does anyone have any idea what is going on? Am I possibly doing something stupid with refcounts which is only manifesting when using sub-interpreters? All relevant code for bootstrapping is contained in Python/pythonrun.c:import_init() (with a little tweaking in the _io module to delay importing the os module and making import.c always use __import__ instead of using the C code). I'm storing the __import__ function in the PyInterpreterState to keep separate state from the other interpreters (i.e. separate sys modules so as to use the proper sys.modules, etc.). But as I said, this all works in a single interpreter view of the world (the entire test suite doesn't trigger a nasty error like this).
Thanks for any help people can provide me on this now 5 year quest to get this work finished.
-Brett
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido)
On Mon, Feb 6, 2012 at 11:32, Benjamin Peterson <benjamin@python.org> wrote:
2012/2/6 Brett Cannon <brett@python.org>:
Thanks for any help people can provide me on this now 5 year quest to get this work finished.
Fixed. (_PyExc_Init was behaving badly.)
That did it! Thanks, Benjamin! Doing one more -uall test run before I declare the bootstrap working.
On Mon, 6 Feb 2012 09:57:56 -0500 Brett Cannon <brett@python.org> wrote:
Thanks for any help people can provide me on this now 5 year quest to get this work finished.
Do you have any plan to solve the performance issue? $ ./python -m timeit -s "import sys; mod='struct'" \ "__import__(mod); del sys.modules[mod]" 10000 loops, best of 3: 75.3 usec per loop $ ./python -m timeit -s "import sys; mod='struct'; from importlib import __import__" \ "__import__(mod); del sys.modules[mod]" 1000 loops, best of 3: 421 usec per loop Startup time is already much worse in 3.3 than in 2.7. With such a slowdown in importing fresh modules, applications using many batteries (third-party or not) will be heavily impacted. Regards Antoine.
On Mon, 6 Feb 2012 20:49:48 +0100 Antoine Pitrou <solipsis@pitrou.net> wrote:
On Mon, 6 Feb 2012 09:57:56 -0500 Brett Cannon <brett@python.org> wrote:
Thanks for any help people can provide me on this now 5 year quest to get this work finished.
Do you have any plan to solve the performance issue?
$ ./python -m timeit -s "import sys; mod='struct'" \ "__import__(mod); del sys.modules[mod]" 10000 loops, best of 3: 75.3 usec per loop $ ./python -m timeit -s "import sys; mod='struct'; from importlib import __import__" \ "__import__(mod); del sys.modules[mod]" 1000 loops, best of 3: 421 usec per loop
The culprit for the overhead is likely to be PathFinder.find_module: $ ./python -m timeit -s "import sys; mod='struct'; from importlib._bootstrap import _DefaultPathFinder; finder=_DefaultPathFinder" "finder.find_module('struct')" 1000 loops, best of 3: 355 usec per loop $ ./python -S -m timeit -s "import sys; mod='struct'; from importlib._bootstrap import _DefaultPathFinder; finder=_DefaultPathFinder" "finder.find_module('struct')" 10000 loops, best of 3: 176 usec per loop Note how it's dependent on sys.path length. On an installed Python with many additional sys.path entries (e.g. because of distribute-based module installs), import times will be much worse. Regards Antoine.
On Mon, Feb 6, 2012 at 14:49, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Mon, 6 Feb 2012 09:57:56 -0500 Brett Cannon <brett@python.org> wrote:
Thanks for any help people can provide me on this now 5 year quest to get this work finished.
Do you have any plan to solve the performance issue?
I have not even looked at performance or attempted to profile the code, so I suspect there is room for improvement.
$ ./python -m timeit -s "import sys; mod='struct'" \ "__import__(mod); del sys.modules[mod]" 10000 loops, best of 3: 75.3 usec per loop $ ./python -m timeit -s "import sys; mod='struct'; from importlib import __import__" \ "__import__(mod); del sys.modules[mod]" 1000 loops, best of 3: 421 usec per loop
Startup time is already much worse in 3.3 than in 2.7. With such a slowdown in importing fresh modules, applications using many batteries (third-party or not) will be heavily impacted.
I have a benchmark suite for importing modules directly at importlib.test.benchmark, but it doesn't explicitly cover searching far down sys.path. I will see if any of the existing tests implicitly do that and if not add it.
participants (4)
-
Antoine Pitrou -
Benjamin Peterson -
Brett Cannon -
Guido van Rossum