[IPython-dev] javascript / Python communication for image viewer
Matthias Bussonnier
bussonniermatthias at gmail.com
Thu Aug 28 04:46:36 EDT 2014
Hi all,
You probably don’t want to use widget framework directly but more comms object.
Binary data exchange will be possible with https://github.com/ipython/ipython/pull/6110
—
M
Le 28 août 2014 à 08:45, James Booth <jabooth at gmail.com> a écrit :
> Hey Matthew, Cyrille,
>
> I’m developing a three.js-based web app for image/object annotation called landmarker.io (I work in computer vision/machine learning, getting high quality annotations of 3D and 2D data is often critical to training models). It’s located here
>
> www.landmarker.io (tool, launches in demo by default)
> https://github.com/menpo/landmarker.io (code)
>
> I’m also one of the developer’s of Menpo (www.menpo.io), which is a Python package for building and testing deformable models. I’m going to be developing an IPython widget-version of the landmarker.io tool, so we can correct annotations as we sport problems in the notebook.
>
> I mention all this as:
>
> 1. I’m also interested in understanding how we can transfer large arrays efficiently to IPython widgets
> 2. I have a little experience in how best to handle this in a traditional client/server model, but I don’t know how well this will translate to the comm interface.
>
> Basically, landmarker.io expects to be able to talk to a RESTful interface that is implemented currently by:
>
> https://github.com/menpo/landmarkerio-server
>
> Originally, I sent meshes as large JSON objects, but this was pretty slow as:
>
> 1. It’s more work in JS to parse the JSON into an array
> 2. It’s more work to turn this into the most efficient JS type for numerical arrays (which WebGL loves) which is an ArrayBuffer.
>
> https://developer.mozilla.org/en-US/docs/Web/API/ArrayBuffer
> https://developer.mozilla.org/en-US/docs/Web/API/Float32Array
>
> This meant that even with a server on localhost, skipping through faces to annotate felt a little sluggish.
>
> Instead now I just directly build a pure ArrayBuffer in Python on the server and ship it to the client. That happens here in the server:
>
> https://github.com/menpo/landmarkerio-server/blob/master/landmarkerio/cache.py#L165
>
> (I’m a little lazy here and literally save the file to disk, then gzip it. It could be done in memory with StringIO through, but I do this as a caching process so it’s going to disk anyway).
>
> On the client side, I make an XMLHttpRequest and demand an array buffer from the server:
> https://github.com/menpo/landmarker.io/blob/master/src/js/app/lib/get.js#L9
>
> For completion, the actual parsing of the array buffer is done here:
> https://github.com/menpo/landmarker.io/blob/master/src/js/app/model/mesh.js#L104
>
> Because three.js supports ArrayBuffers, this is really fast. I just point three to the array and we are away. With this implementation, browsing through subjects to annotate with a server on localhost (akin to moving between slices in your brain scan I imagine) is very fast.
>
> I’m afraid I don’t have much knowledge of the comm interface - is there a document I could be pointed out that lays out the protocol? Does it sound possible to send a pure array to JS in the way I’m doing in the landmarkerio-server?
>
> Best,
> James
>
>
> —
> Sent from Mailbox
>
>
> On Wed, Aug 27, 2014 at 9:53 PM, Cyrille Rossant <cyrille.rossant at gmail.com> wrote:
>
> We have pretty similar requirements for Vispy, as we target fast
> visualization of big datasets using OpenGL in the IPython notebook. In
> particular, we can't use Canvas or SVG because it's just too slow for
> big data, so we have to use WebGL. This lets us leverage the GPU for
> visualization.
>
> I would be very interested in seeing the answers to your questions.
> Notably, is it possible to use binary sockets for transferring large
> amounts of binary data between Python and JavaScript?
>
> Cyrille
>
> 2014-08-27 20:52 GMT+01:00 Matthew Brett <matthew.brett at gmail.com>:
> > Guys / gals,
> >
> > I want to ask for advice about writing a brain image display widget for
> > IPython.
> >
> > I would like to make an IPython widget that can take an in-memory numpy array
> > and do an interactive display of orthogonal slices from the array. The
> > display will look something like Papaya:
> >
> > http://rii.uthscsa.edu/mango/papaya
> >
> > where clicking or moving the mouse causes matching slices to be displayed
> > through the three axes of the numpy array (here a brain image).
> >
> > Papaya is pure javascript, so I am assuming that it loads the whole array
> > (brain image) into a javascript variable and takes slices from that.
> >
> > What I would like to do, is to be able to keep the whole 3D array only in
> > Python, and pass the slices as needed to a javascript viewer.
> >
> > In my ignorance, I am not sure which approach to go for first.
> >
> > Should I use the comm / widget interface for this? In that case I guess the
> > procedure would be:
> >
> > * mouse movement generates a 'need slice' message from javascript
> > * python kernel accepts 'need slice' message, takes slice from array, base64
> > encodes into JSON, sends 'here is your slice' message back to javascript
> > with the data
> > * javascript accepts message, unpacks base64'ed JSONed slice into variable and
> > displays slice variable
> >
> > Is that right? I guess that involves Is there any chance that this
> > will be fast enough for
> > satisfying interactive movement through the image, which would likely require
> > something like 20 slices per second, each of say 64 x 64 floating point?
> >
> > If not - is there something else I should look at instead?
> >
> > Another question for more thanks - should I use a Canvas or SVG element to
> > display the images? The fabric.js README [1] seems to imply the
> > Canvas element is
> > faster for some interactive stuff, does anyone have relevant experience to
> > share?
> >
> > Thanks a lot,
> >
> > Matthew
> >
> > [1] https://github.com/kangax/fabric.js/#history
> > _______________________________________________
> > IPython-dev mailing list
> > IPython-dev at scipy.org
> > http://mail.scipy.org/mailman/listinfo/ipython-dev
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20140828/8496a9ee/attachment.html>
More information about the IPython-dev
mailing list