On Thu, Feb 9, 2012 at 11:23 AM, Sturla Molden <sturla@molden.no> wrote:

On 09.02.2012 20:05, Guido van Rossum wrote:

I'm curious about the module loader problem. Did someone ever analyze
the cause and come up with a fix? Is it the import lock? Maybe it's
something for the bug tracker.

See this:

http://mail.scipy.org/pipermail/numpy-discussion/2012-January/059801.html

The offender is actually imp.find_module, which results in huge number of failed open() calls when used concurrently from many processes.

So a solution is to have one process locate the modules and then broadcast their location to the other processes.

There is even a paper on the issue. Here they suggest importing from ramdisk might work on IBM blue gene, but not on Cray.

http://www.cs.uoregon.edu/Research/paracomp/papers/iccs11/iccs_paper_final.pdf

Another solution might be to use sys.meta_path to bypass imp.find_module:

http://mail.scipy.org/pipermail/numpy-discussion/2012-January/059813.html

The best solution would of course be to fix imp.find_module so it scales properly.

Sturla
_______________________________________________
Python-ideas mailing list
Python-ideas@python.org
http://mail.python.org/mailman/listinfo/python-ideas