[IPython-dev] Launching new kernels by forking existing kernel

[Brian Granger]

Jason,

Whenever we fork processes using pyzmq, we have to create a new
context and sockets after the fork.  IOW, pyzmq context and sockets
cannot move over  fork boundary.  I think this means that it will be
essentially impossible to do what you are wanting to.

Cheers,

Brian

On Tue, May 8, 2012 at 9:16 AM, Jason Grout <jason-sage at creativetrax.com> wrote:
> Hi all,
>
> (I'm resending this since I don't see it appearing on the list since I
> sent it yesterday; my apologies if someone gets it twice.)
>
> As part of the effort to rewrite the Sage Cell server [1] to better use
> the new IPython capabilities, we are trying to figure out how to emulate
> this behavior:
>
> In our current implementation, when a user requests a computation, the
> request goes to a worker process manager.  The manager creates a
> temporary directory, forks off a worker process, and then the worker
> process executes the computation and returns results.  After the worker
> process is done, the manager cleans up the directory, etc.  The goal of
> our use-case is hundreds of short-lived, constantly regenerated kernels
> with almost zero startup cost (which means it is really bad to start a
> kernel from scratch, as it takes some time to import all the Sage
> modules).  Each kernel interacts with a client for a short period of
> time (seconds to minutes).
>
> Poking around a bit, I'm wondering if this is pretty similar to some of
> the launchers in the parallel/launcher.py file.  Would it be natural to
> implement our start-a-new-kernel-by-forking procedure when we receive a
> message as a launcher like in that file?  Also, is it easy to redirect a
> kernel's zmq channels (after we fork, I presume we need to hook the new
> kernel up to separate channels with the client.)
>
> Thanks,
>
> Jason
>
> [1] aleph.sagemath.org.  See also
> http://sage.math.washington.edu/home/jason/sagecell/index.html
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev



-- 
Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger at calpoly.edu and ellisonbg at gmail.com