[IPython-dev] MultiKernelManager vs. ipcluster

Brian Granger ellisonbg at gmail.com
Mon Jun 4 23:18:23 EDT 2012


Jason,

Sorry it has taken me so long to get back to you...

On Fri, Jun 1, 2012 at 2:07 PM, Jason Grout <jason-sage at creativetrax.com> wrote:
> Hi everyone,
>
> As mentioned yesterday, we've been exploring ways to implement the Sage
> cell server using the IPython capabilities better.  This message is a
> call for comments, as well as a few questions about the directions of
> things like ipcluster.
>
> Our goals are to:
>
> * start multiple kernels/engines very quickly (e.g., through forking a
> master kernel)
> * connect single kernels to single web sessions
> * be able to transfer files back and forth between the executing
> environment and the web browser
> * do things as securely as is reasonable (e.g., workers are running on a
> firewalled virtual instance, etc.)
> * be able to use an enterprise-level web server, such as Google App Engine.

Honestly I am not sure it makes sense to use IPython at all.  The
design requirements for IPython are very different from this.  I worry
that you are always going to be fighting the design.  ZeroMQ/PyZMQ
makes so much of this type of thing really easy.

>
> We've explored two approaches, and have somewhat-barely-working partial
> proofs-of-concepts of each:
>
> IPCLUSTER
> ---------
>
> We implemented a forking engine factory which is run on the worker
> computer.  A cluster can start up new engines by sshing into the worker
> computer and sending a message to the forking engine factory.  A new
> engine starts and then registers with the controller.  We're thinking it
> might be best to implement a new scheduler that would take messages,
> check to see if the session id matches up with one of the
> currently-running engines, and send the message to the appropriate
> engine if so.  If the message id does not match up to a currently
> running engine, then the scheduler would start up a new engine (can the
> scheduler do this?)
>
> The client would basically be the zmq/socket.io bridge, translating
> messages to/from the controller and the browser.  I guess we could
> implement one client per session, or we could implement one overall
> client that would route cluster output to the appropriate browser.

ipcluster is a very specialized tool for starting engines in various
environments.  If I had your design constraints I would not try to use
ipcluster.

>
> MULTI FORKING KERNEL MANAGER
> ----------------------------
>
> This approach stays much closer to the current IPython notebook
> architecture.  A MultiKernelManager starts up kernels using a
> ForkingKernelManager (based on Min's gist the other day).  Each kernel
> sets up a connection to a specific set of websocket channels through a
> tornado-based bridge.  We still have to implement things like separation
> of the forked kernels (on a separate ssh account somewhere) and the
> tornado handler, and things like that.

You will have to be very careful about the security aspects of this.
You should assume that the kernel running process is entirely hostile
and allow no traffic that originates from it to reach the tornado web
server.

>
> THOUGHTS
> --------
> It seems that the multikernel manager is much lighter-weight, but we'd
> have to implement a lot of the enterprise-level functionality that the
> cluster already has.  On the other hand, the ipcluster approach really
> does more than we need, so, in a sense, we need to trim it back.  We're
> not asking you to make a decision for us, obviously, but it would be
> valuable to hear any comments or suggestions.

If security is important I think you will have to roll your own.
If security is important you will have to come up with a system level
solution - not something that is simply a python library.

> SPECIFIC QUESTIONS
> ------------------
>
> 1. Brian, why did you decide to make a new multikernel manager instead
> of trying to leverage the ipcluster functionality to execute multiple
> engines?

1) ipcluster just does too much that is irrelevant to starting kernels
for the notebook.
2) ipcluster doesn't do any of the more advanced things that you need
for starting kernels across many systems.

> 2. Some time ago, on this list or on a pull request, there was some
> discussion about the difference between kernels and engines (or the lack
> of difference).  Are there some plans to simplify the ipcluster
> architecture a bit to merge the concepts of engines and kernels?

I think Min has answered this, but the answer is now that engine = kernel.

> 3. Any idea about how much overhead the cluster introduces for lots of
> short computations involving lots of output?  We'll test this too, but
> I'm curious if there was thought towards this use-case in the design.

This one is tough to answer.  In general the overhead of
PyZMQ/ZeroMQ/IPython is extremely low.  The overheads you will run
into will be completely determined by things like network throughput,
etc.

Cheers,

Brian

> Thanks,
>
> Jason
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev



-- 
Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger at calpoly.edu and ellisonbg at gmail.com



More information about the IPython-dev mailing list