[IPython-dev] Interactive Matplotlib in the browser

[Brian Granger]

On Fri, Oct 12, 2012 at 12:21 PM, Michael Droettboom <mdroe at stsci.edu> wrote:
> On 10/12/2012 01:06 PM, Brian Granger wrote:
>> On Thu, Oct 11, 2012 at 11:39 PM, Jason Grout
>> <jason-sage at creativetrax.com> wrote:
>>> On 10/11/12 10:45 PM, Brian Granger wrote:
>>>> It is not clear to me that the stream of PNGs will win in the end.
>>> What do you see as alternatives?  Reimplementing matplotlib in the
>>> browser?  SVG-based rendering, where we send over a complete svg image
>>> at each step (or just the changed svd elements)?  I'm curious what other
>>> ideas you have.
>> There are two cases:
>>
>> * Small data sets and relatively simple plots types.  For these, I do
>> think it makes sense to send the raw JSON data to the browser and
>> implement a Javascript based plotting library that consumes the JSON
>> and has an extremely high level of interactivity.  I don't think we
>> need to do *all* possible plot types with the pixel perfection,
>> embedded math, infinite configurability.  When people are working
>> interactively they want: i) plots that look spectacular - pixelized
>> images don't cut it and ii) high levels of interactivity.
>> I am not suggesting that we reimplement all of Matplotlib.
>
> I agree about the reduced interactivity -- but there is an
> interactivity/functionality tradeoff, as you point out.  I used to just
> think that it was a matter of implementing the missing functionality in
> Javascript, but I have since come to realize there are just things it
> can't do in a reasonable way (multiple projection types, multiple image
> interpolation modes) without another quantum leap in browser functionality.

Absolutely, and i don't think it makes sense to try and do those things in JS.

> As for image quality, the approach outlined here uses the native pixel
> resolution, so it shouldn't be any more pixelized than anything else
> displayed on screen.  I understand that Safari on Macs with a retina
> display does upsampling of images by default, but with the appropriate
> CSS we should be able to address that.  Is it the lack of subpixel
> sampling that is bothersome?  We've always eschewed that in matplotlib
> because it makes the PNGs non-portable (different displays have
> different RGB orderings), but we could flip it on in this case.
>
>>
>> Keep in mind that current situation with Matplotlib and the IPython
>> Notebook is not very ideal.  The pngs that are the default don't look
>> that great and svg is not yet good enough to use as the default.  We
>> really do need in browser plotting that *looks* better.
>
> What doesn't look great about the PNGs?  A lot of effort has gone in to
> antialiasing and hinting the fonts in the best possible way and rounding
> lines to pixel centers etc to make it look as good as possible.  If
> there's specific shortcomings, they can be addressed, since matplotlib
> controls the entire rendering stack.

They simply don't look at clean at vector based plots, especially now
that users can resize the images dynamically.  We could increase the
resolution of the PNGs, but because the PNGs are saved in the notebook
files, those files could get large.  We have tried to choose a
resolution/size that looks decent, but that isn't overly big.
Remember our user create notebook with up to many dozens of plots.  It
is not a shortcoming of matplotlib, just of non-vector image formats
in general.

>>
>> Another important aspect of the JSON data approach.  We want to enable
>> Notebooks to be viewed and used in contexts outside the actual
>> notebook interface (nbviewer for example) where there is *no* python
>> code that is run.  Because we save the JSON data in the notebook, JSON
>> based plotting can be done interactively in contexts like nbviewer.
>> For sharing computational results with a wide audience that is
>> critical.
>>
>> * Large data sets
>>
>> I don't really have any alternatives - this is simply a really hard
>> problem.  In some contexts downsampling is possible.  In other
>> contexts it is possible to stream subsets of JSON data at are required
>> for the current view.  It is my impression that the folks at Continuum
>> are working on this problem.  Maybe they have some insight into this
>> problem.
>
> I was actually speaking to Travis Oliphant at length about this last
> week, and the upshot of that conversation was this present approach.

Nice, I will hopefully have a chance to look at the code soon.

> Mike
>
>>
>>
>>> If
>>>> you make a single static plot of a large data set, that is way better
>>>> than trying to send the data to the browser and rendering it there.
>>>> But if you have to send hundreds or thousands of PNGs to get
>>>> interactivity, that benefit may be washed out.  Especially if you have
>>>> multiple users interacting with plots - the server could quickly grind
>>>> to a halt.  I think we should do tests to see how bad it gets, taking
>>>> into account the multiple user question.  The one performance benefit
>>>> that I can think of is that you can tune the level of interactivity to
>>>> limit the data that comes back.  For large data sets, users might be
>>>> willing to settle for less interactivity.  That option doesn't exist
>>>> when you send all the data back.
>>> I implemented server-side rate-limiting (see my most recent
>>> modifications at https://gist.github.com/3875846, for example).  Or are
>>> you saying that the server might not be able to render pngs for hundreds
>>> or thousands of users?  I wonder if we can get a better protocol than
>>> just a "Motion PNG" [1].
>>>
>>> A disappointing thing is that it seems to take about 100ms to render the
>>> example png image; it would sure be nice if that was a lot faster.
>>>
>>>
>>> [1] http://en.wikipedia.org/wiki/Motion_JPEG
>>>
>>> _______________________________________________
>>> IPython-dev mailing list
>>> IPython-dev at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>>
>>
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev



-- 
Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger at calpoly.edu and ellisonbg at gmail.com