[IPython-dev] api between kernel, web server and frontend

[Ondrej Certik]

On Thu, May 13, 2010 at 12:04 PM, Ondrej Certik <ondrej at certik.cz> wrote:
> Hi,
>
> we were in Berkeley with Mateusz and we discussed the API between the
> computational kernel, web server and the javascript frontend (and
> other frontends), so I wanted to share our thoughts.
>
> The main workhorse is a kernel, that has two connections:
>
> 1) feed, that publishes all changes to a given session, anyone (who
> knows the session UID) can subscribe to it and observe the whole
> session (e.g. ipython session)
> 2) request/response channel, which is used for sending python code for
> evaluation and some other minor things. The results are published in
> the feed.
>
> The API is done using json, and it is described here:
>
> http://github.com/ellisonbg/pyzmq/blob/master/examples/kernel/message_spec.rst
>
> and demo implementation of this is here:
>
> http://github.com/ellisonbg/pyzmq/tree/master/examples/kernel/
>
> This demo implementation is using 0MQ (http://www.zeromq.org/) for the
> transport layer (i.e. for sending the json messages). 0MQ is probably
> the best library for sending/receiving messages over the net, but a
> big disadvantage is that it probably can't be used on the google app
> engine and it creates a dependency. But it's just the transport layer,
> in principle it can be replaced with anything else.
>
>
>
> So the above is the main logic for handling computational sessions,
> with many people (frontends) doing simultaneous calculations in the
> same session and one kernel (that however can dispatch the actual
> little calculations to many cores, e.g. do it in parallel). Note that
> this kernel has no database, no user accounts, nothing. All that it
> has is a session with UID (so you need to create the session somehow,
> but that's it) and it has a namespace for the running session, that's
> it.
>
> Now we need a web server, that would use the above API to communicate
> with the kernel, and this web server would have a database with user
> accounts, it would also have a database with user worksheets and cells
> and it would expose this over HTTP. The problem is that using HTTP and
> the current web technologies pose some limitations currently to what
> can be done. We talked with Mateusz how to do that in the car to Reno
> and we agreed that the best way is to use a JSONRPC, so the web server
> would define a Python class with some functionality, and then the
> javascript frontend (running in the browser) would use methods of this
> class to do things (create worksheets, get users, create cells,
> evaluate cells, get the feed from the kernel, ...). The JS frontend
> would have to periodically call some method on the server to get the
> feed, as we don't know how to "subscribe" to it using AJAX. Maybe in
> couple years, this would be simple using web sockets.
>
> I would be interested in any feedback and discussion.
>
> We now need to write a demo implementation of the above web server and
> a client that uses JSONRPC to do the above. The client can (should) be
> command line based and it will only use JSONRPC for all the
> communication. It will then be quite trivial (trivial in terms of the
> logic) to anyone to write a javascript client, as JSONRPC works nice
> with javascript, or adapt codenode or the sage notebook frontends to
> use this JSONRPC.
>
>
> For testing purposes, I would really like to have a demo
> implementation that runs on the google app engine (both the web server
> and the kernel). Then anyone can run a simple client, that connects to
> it (and we will also have a simple javascript client, so you would
> just go to the demo page and it would work). For that to happen, we
> need to figure out some other transport layer than 0MQ, something that
> works on the app engine. This can also be the only option for people
> that don't want/can't to install 0MQ for some reason. Any ideas?

Well, one way to do that would be that the kernel and the web server
would be in the same program and I would be passing messages to the
kernel by simply calling a Python method, or by somehow emulating the
0MQ api in Python in the same process.
That should work, to get it working both with 0MQ and on the app
engine with minimal modifications. Also, I can imagine that ipython
itself could use this emulation by default for people that just want
the good old ipython, and only use 0MQ if you want to run it over the
net.

Ondrej