[IPython-dev] Launching new kernels by forking existing kernel
Jason Grout
jason-sage at creativetrax.com
Tue May 8 12:16:19 EDT 2012
Hi all,
(I'm resending this since I don't see it appearing on the list since I
sent it yesterday; my apologies if someone gets it twice.)
As part of the effort to rewrite the Sage Cell server [1] to better use
the new IPython capabilities, we are trying to figure out how to emulate
this behavior:
In our current implementation, when a user requests a computation, the
request goes to a worker process manager. The manager creates a
temporary directory, forks off a worker process, and then the worker
process executes the computation and returns results. After the worker
process is done, the manager cleans up the directory, etc. The goal of
our use-case is hundreds of short-lived, constantly regenerated kernels
with almost zero startup cost (which means it is really bad to start a
kernel from scratch, as it takes some time to import all the Sage
modules). Each kernel interacts with a client for a short period of
time (seconds to minutes).
Poking around a bit, I'm wondering if this is pretty similar to some of
the launchers in the parallel/launcher.py file. Would it be natural to
implement our start-a-new-kernel-by-forking procedure when we receive a
message as a launcher like in that file? Also, is it easy to redirect a
kernel's zmq channels (after we fork, I presume we need to hook the new
kernel up to separate channels with the client.)
Thanks,
Jason
[1] aleph.sagemath.org. See also
http://sage.math.washington.edu/home/jason/sagecell/index.html
More information about the IPython-dev
mailing list