[IPython-dev] Launching new kernels by forking existing kernel
MinRK
benjaminrk at gmail.com
Tue May 8 15:04:46 EDT 2012
On Tue, May 8, 2012 at 11:33 AM, Brian Granger <ellisonbg at gmail.com> wrote:
> Jason,
>
> Whenever we fork processes using pyzmq, we have to create a new
> context and sockets after the fork. IOW, pyzmq context and sockets
> cannot move over fork boundary. I think this means that it will be
> essentially impossible to do what you are wanting to.
>
I think that depends on what he means by this:
> Also, is it easy to redirect a kernel's zmq channels (after we fork, I
presume we need to hook the new kernel up to separate channels with the
client.)
If there is some kernel-launching facility that is forking off processes,
and those forked processes then create the context and sockets for the
kernel, then this should work just fine. What won't work is passing
context and/or sockets across the fork boundary.
This is the behavior we currently have:
1. KernelManager writes connection file.
2. KM starts Kernel via subprocess.Popen, passing the connection file via
the command-line.
If you replace step 2 with:
2a. KM sends special request to KernelStarter, including connection file
(either via network, or in-process, I'm not sure of the architecture).
2b. KernelStarter starts kernel via *fork*, passing connection file, and
being careful to avoid the subprocess inheriting zmq objects.
Then I don't see any reason that shouldn't work.
-MinRK
>
> Cheers,
>
> Brian
>
> On Tue, May 8, 2012 at 9:16 AM, Jason Grout <jason-sage at creativetrax.com>
> wrote:
> > Hi all,
> >
> > (I'm resending this since I don't see it appearing on the list since I
> > sent it yesterday; my apologies if someone gets it twice.)
> >
> > As part of the effort to rewrite the Sage Cell server [1] to better use
> > the new IPython capabilities, we are trying to figure out how to emulate
> > this behavior:
> >
> > In our current implementation, when a user requests a computation, the
> > request goes to a worker process manager. The manager creates a
> > temporary directory, forks off a worker process, and then the worker
> > process executes the computation and returns results. After the worker
> > process is done, the manager cleans up the directory, etc. The goal of
> > our use-case is hundreds of short-lived, constantly regenerated kernels
> > with almost zero startup cost (which means it is really bad to start a
> > kernel from scratch, as it takes some time to import all the Sage
> > modules). Each kernel interacts with a client for a short period of
> > time (seconds to minutes).
> >
> > Poking around a bit, I'm wondering if this is pretty similar to some of
> > the launchers in the parallel/launcher.py file. Would it be natural to
> > implement our start-a-new-kernel-by-forking procedure when we receive a
> > message as a launcher like in that file? Also, is it easy to redirect a
> > kernel's zmq channels (after we fork, I presume we need to hook the new
> > kernel up to separate channels with the client.)
> >
> > Thanks,
> >
> > Jason
> >
> > [1] aleph.sagemath.org. See also
> > http://sage.math.washington.edu/home/jason/sagecell/index.html
> > _______________________________________________
> > IPython-dev mailing list
> > IPython-dev at scipy.org
> > http://mail.scipy.org/mailman/listinfo/ipython-dev
>
>
>
> --
> Brian E. Granger
> Cal Poly State University, San Luis Obispo
> bgranger at calpoly.edu and ellisonbg at gmail.com
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20120508/933a6c95/attachment.html>
More information about the IPython-dev
mailing list