[IPython-dev] Launching new kernels by forking existing kernel

[Jason Grout]

Hi all,

(I'm resending this since I don't see it appearing on the list since I 
sent it yesterday; my apologies if someone gets it twice.)

As part of the effort to rewrite the Sage Cell server [1] to better use 
the new IPython capabilities, we are trying to figure out how to emulate 
this behavior:

In our current implementation, when a user requests a computation, the 
request goes to a worker process manager.  The manager creates a 
temporary directory, forks off a worker process, and then the worker 
process executes the computation and returns results.  After the worker 
process is done, the manager cleans up the directory, etc.  The goal of 
our use-case is hundreds of short-lived, constantly regenerated kernels 
with almost zero startup cost (which means it is really bad to start a 
kernel from scratch, as it takes some time to import all the Sage 
modules).  Each kernel interacts with a client for a short period of 
time (seconds to minutes).

Poking around a bit, I'm wondering if this is pretty similar to some of 
the launchers in the parallel/launcher.py file.  Would it be natural to 
implement our start-a-new-kernel-by-forking procedure when we receive a 
message as a launcher like in that file?  Also, is it easy to redirect a 
kernel's zmq channels (after we fork, I presume we need to hook the new 
kernel up to separate channels with the client.)

Thanks,

Jason

[1] aleph.sagemath.org.  See also 
http://sage.math.washington.edu/home/jason/sagecell/index.html