[IPython-dev] Using Google App Engine channels instead of zmq in new IPython

Brian Granger ellisonbg at gmail.com
Mon Feb 13 17:32:12 EST 2012

At this point, ZeroMQ is not a minor implementation detail of IPython,
it is really our secret sauce.  ZeroMQ has abstractions, such as
queues and messaging patterns, that go beyond simple point to point
messaging and we rely on those throughout the code base.  I don't see
how ZeroMQ could be replaced without rewriting good parts of IPython.
I do think some of the patterns we use could be adapted for use in GAE
though - such as our message protocol.

On Mon, Feb 13, 2012 at 11:15 AM, MinRK <benjaminrk at gmail.com> wrote:
> On Mon, Feb 13, 2012 at 10:44, Jason Grout <jason-sage at creativetrax.com>
> wrote:
>> Over the Sage camp, we're starting to experiment with using Google App
>> Engine to manage communication between browser frontends and worker
>> (kernel) backends.  The idea is to use GAE to host the notebook, while
>> browsers and backend kernels communicate to send and retrieve work and
>> results through the Google App Engine "channel" architecture [1].  The
>> result would be a massively more scalable notebook running in the cloud.
> Scalability / Cloud are great, and certainly an ultimate goal, but my
> perspective is that if replacing zmq is part of the GAE solution, then it is
> just not the place for IPython.
>> We'd (or at least I'd) also like to eventually switch to IPython 0.12,
>> and try to leverage the new IPython architecture and communication in
>> the Sage notebook.  So my question is: how well is the zmq parts
>> encapsulated, and do you think it would be hard to write a google app
>> engine "channel"-based solution that plugs in the place of the zmq
>> communication (and websocket for the html notebook) communication?
> zmq is used principally in two places:
> * The Session object, which abstracts network communication.  You would have
> to write a new version of this object
> * The event/polling loops everywhere (specifically Kernel, KernelManager,
> NotebookApp).  If GoogleAppEngine channels can be made to work with the
> tornado IOLoop, then this shouldn't be too tricky.  Otherwise, you would
> have to rewrite the eventloop of every application/thread we have.
> -MinRK
>> I've CCd both the ipython list and the sage-notebook list---hopefully
>> the discussion can be carried on both lists.  Also a disclaimer: I have
>> about zero experience with GAE; I'm just trying to throw the idea out
>> there and understand a little more about the possible work required.
>> Thanks,
>> Jason
>> [1] http://code.google.com/appengine/docs/python/channel/
>> _______________________________________________
>> IPython-dev mailing list
>> IPython-dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-dev
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev

Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger at calpoly.edu and ellisonbg at gmail.com

More information about the IPython-dev mailing list