[IPython-dev] slow startup due to script magic
Jason Grout
jason-sage at creativetrax.com
Sat Sep 22 01:26:36 EDT 2012
I was looking into why aleph.sagemath.org is so slow in responding to
user requests (i.e., it takes a second or so to respond to any new
request), and it turned out that the major slowdown was starting up a
new IPython kernel. We eventually traced it down to the loading of the
script extension, which by default does 6 popen calls to find the paths
of scripts in the _script_magics_default method, which are really slow
(something like 200 ms for the calls on my computer):
https://github.com/ipython/ipython/blob/master/IPython/core/magics/script.py#L92
and diving deeper,
https://github.com/ipython/ipython/blob/master/IPython/utils/_process_posix.py#L35
If I move the tests up into the module level so that they are run when
the module is loaded, rather than every time a new kernel is created,
the kernel creation time drops dramatically, and in the end, our
response time for aleph is cut in 1/2 or 1/3.
What can we do to optimize the guessing of where different commands
live? Here are at least a few ways:
1. explicitly specify a script_paths dictionary (just curious, what key
would I make in the ipython config file to specify that? I'm still a
little lost when it comes to figuring out how keys relate to actual
classes that are invoked.)
2. Move the guessing up into the module level, cached in a module-level
variable. While this doesn't help in the case where you only launch one
kernel, it dramatically speeds up things if you are launching multiple
kernels.
Any other ideas? It seems such a shame that loading a rather esoteric,
but standard, extension should double or so the startup time of an
IPython kernel.
Thanks,
Jason
More information about the IPython-dev
mailing list