[IPython-dev] javascript / Python communication for image viewer

Matthias Bussonnier bussonniermatthias at gmail.com
Thu Aug 28 04:46:36 EDT 2014


Hi all, 


You probably don’t want to use widget framework directly but more comms object. 
Binary data exchange will be possible with https://github.com/ipython/ipython/pull/6110

— 
M
Le 28 août 2014 à 08:45, James Booth <jabooth at gmail.com> a écrit :

> Hey Matthew, Cyrille,
> 
> I’m developing a three.js-based web app for image/object annotation called landmarker.io (I work in computer vision/machine learning, getting high quality annotations of 3D and 2D data is often critical to training models). It’s located here
> 
> www.landmarker.io (tool, launches in demo by default)
> https://github.com/menpo/landmarker.io (code)
> 
> I’m also one of the developer’s of Menpo (www.menpo.io), which is a Python package for building and testing deformable models. I’m going to be developing an IPython widget-version of the landmarker.io tool, so we can correct annotations as we sport problems in the notebook.
> 
> I mention all this as:
> 
> 1. I’m also interested in understanding how we can transfer large arrays efficiently to IPython widgets
> 2. I have a little experience in how best to handle this in a traditional client/server model, but I don’t know how well this will translate to the comm interface.
> 
> Basically, landmarker.io expects to be able to talk to a RESTful interface that is implemented currently by:
> 
> https://github.com/menpo/landmarkerio-server
> 
> Originally, I sent meshes as large JSON objects, but this was pretty slow as:
> 
> 1. It’s more work in JS to parse the JSON into an array
> 2. It’s more work to turn this into the most efficient JS type for numerical arrays (which WebGL loves) which is an ArrayBuffer.
> 
> https://developer.mozilla.org/en-US/docs/Web/API/ArrayBuffer
> https://developer.mozilla.org/en-US/docs/Web/API/Float32Array
> 
> This meant that even with a server on localhost, skipping through faces to annotate felt a little sluggish.
> 
> Instead now I just directly build a pure ArrayBuffer in Python on the server and ship it to the client. That happens here in the server:
> 
> https://github.com/menpo/landmarkerio-server/blob/master/landmarkerio/cache.py#L165
> 
> (I’m a little lazy here and literally save the file to disk, then gzip it. It could be done in memory with StringIO through, but I do this as a caching process so it’s going to disk anyway).
> 
> On the client side, I make an XMLHttpRequest and demand an array buffer from the server:
> https://github.com/menpo/landmarker.io/blob/master/src/js/app/lib/get.js#L9
> 
> For completion, the actual parsing of the array buffer is done here:
> https://github.com/menpo/landmarker.io/blob/master/src/js/app/model/mesh.js#L104
> 
> Because three.js supports ArrayBuffers, this is really fast. I just point three to the array and we are away. With this implementation, browsing through subjects to annotate with a server on localhost (akin to moving between slices in your brain scan I imagine) is very fast.
> 
> I’m afraid I don’t have much knowledge of the comm interface - is there a document I could be pointed out that lays out the protocol? Does it sound possible to send a pure array to JS in the way I’m doing in the landmarkerio-server?
> 
> Best,
> James
> 
> 
>> Sent from Mailbox
> 
> 
> On Wed, Aug 27, 2014 at 9:53 PM, Cyrille Rossant <cyrille.rossant at gmail.com> wrote:
> 
> We have pretty similar requirements for Vispy, as we target fast 
> visualization of big datasets using OpenGL in the IPython notebook. In 
> particular, we can't use Canvas or SVG because it's just too slow for 
> big data, so we have to use WebGL. This lets us leverage the GPU for 
> visualization. 
> 
> I would be very interested in seeing the answers to your questions. 
> Notably, is it possible to use binary sockets for transferring large 
> amounts of binary data between Python and JavaScript? 
> 
> Cyrille 
> 
> 2014-08-27 20:52 GMT+01:00 Matthew Brett <matthew.brett at gmail.com>: 
> > Guys / gals, 
> > 
> > I want to ask for advice about writing a brain image display widget for 
> > IPython. 
> > 
> > I would like to make an IPython widget that can take an in-memory numpy array 
> > and do an interactive display of orthogonal slices from the array. The 
> > display will look something like Papaya: 
> > 
> > http://rii.uthscsa.edu/mango/papaya 
> > 
> > where clicking or moving the mouse causes matching slices to be displayed 
> > through the three axes of the numpy array (here a brain image). 
> > 
> > Papaya is pure javascript, so I am assuming that it loads the whole array 
> > (brain image) into a javascript variable and takes slices from that. 
> > 
> > What I would like to do, is to be able to keep the whole 3D array only in 
> > Python, and pass the slices as needed to a javascript viewer. 
> > 
> > In my ignorance, I am not sure which approach to go for first. 
> > 
> > Should I use the comm / widget interface for this? In that case I guess the 
> > procedure would be: 
> > 
> > * mouse movement generates a 'need slice' message from javascript 
> > * python kernel accepts 'need slice' message, takes slice from array, base64 
> > encodes into JSON, sends 'here is your slice' message back to javascript 
> > with the data 
> > * javascript accepts message, unpacks base64'ed JSONed slice into variable and 
> > displays slice variable 
> > 
> > Is that right? I guess that involves Is there any chance that this 
> > will be fast enough for 
> > satisfying interactive movement through the image, which would likely require 
> > something like 20 slices per second, each of say 64 x 64 floating point? 
> > 
> > If not - is there something else I should look at instead? 
> > 
> > Another question for more thanks - should I use a Canvas or SVG element to 
> > display the images? The fabric.js README [1] seems to imply the 
> > Canvas element is 
> > faster for some interactive stuff, does anyone have relevant experience to 
> > share? 
> > 
> > Thanks a lot, 
> > 
> > Matthew 
> > 
> > [1] https://github.com/kangax/fabric.js/#history 
> > _______________________________________________ 
> > IPython-dev mailing list 
> > IPython-dev at scipy.org 
> > http://mail.scipy.org/mailman/listinfo/ipython-dev 
> _______________________________________________ 
> IPython-dev mailing list 
> IPython-dev at scipy.org 
> http://mail.scipy.org/mailman/listinfo/ipython-dev 
> 
> 
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20140828/8496a9ee/attachment.html>


More information about the IPython-dev mailing list