Fwd: [IPython-dev] Data viz in the notebook with JS
Hi all (including Fernando, who I forgot was subscribed when I forwarded that message a few days ago!), This may also be of interest here. It's more from IPython mailing lists about implementing widgets like what we have in Reason, using their notebook engine. As my work on Reason lately has been refactoring it for client-side maintainability and to work with ExtJS4, I haven't thought about the python-side of it lately, and this would have to touch that a bit. If you want to play with the most recently released IPython notebook, it's available when yt is installed with the INST_0MQ option set to 1. The command "ipython notebook" will get you started. Any enterprising devs that want to work on integration should definitely speak up; my time is otherwise mostly occupied for the near/mid-future, but I'd be happy to lend a hand where I can. I'd want this to be part of the ExtJS4 port, as I don't see any point in having two major, independent sets of changes to Reason. Advantages to using IPython-notebook: * Using websockets will reduce latency between actions and responses by a considerable amount * Outsource all of the kernel launching (MPI/Queue/local), which would probably ease the development of interactive in-browser parallel VR * Outsource dealing with process handling * Payloads sent from yt widget code would not *necessarily* need to be received within Reason, but could possibly be displayed in a vanilla Notebook Disadvantages: * Several additional external dependencies (zeromq, pyzeromq, jinja2, tornado). Three of these are currently optional dependencies in the install script. * Payload system will have to be rewritten * Unclear amount of our REPL/Notebook code will have to be rewritten * The IPython notebook's storage format (SQLite) may not play nicely with all Lustre systems * Possible impedance mismatch between JS libraries used in Notebook and ExtJS4 * My understanding of how WebSockets work doesn't cover how they work when SSH tunneling to bind to a socket, so I am putting potential glitches from this approach under "disadvantage" For those interested, my ExtJS4 work is in my fork at https://bitbucket.org/MatthewTurk/yt-extjs4 . The code is considerably cleaned up from the first generation of Reason, and I think more inviting for development. As an example, compare the code for these sets of files between the old Reason and the new: New: yt/gui/reason/html/app/controller/widgets/PlotWindow.js yt/gui/reason/html/app/view/widgets/PlotWindow.js Old: yt/gui/reason/html/js/widget_plotwindow.js View & handler code has been completely separated, and I believe there's just enough magic to make developing new widgets easy and straightforward. I'm still porting existing widgets and bringing basic functionality up to spec, but the architecture is complete. -Matt ---------- Forwarded message ---------- From: Fernando Perez <fperez.net@gmail.com> Date: Sat, Jun 2, 2012 at 5:09 PM Subject: [IPython-dev] Data viz in the notebook with JS To: IPython Development list <ipython-dev@scipy.org>, IPython User list <ipython-user@scipy.org> Hi folks Here's a quick preview of what we can start having now: http://imgur.com/vk2Q6 courtesy of Cameron Bates: https://github.com/crbates/ipython-flot/blob/master/IPython/frontend/html/no... I suggest simply to grab the single flot.py file and put it in your working dir for testing. We still have a ways to go with this kind of setup, because the user experience upon saving is a bit awkward: these javascript objects are only rendered when their code executes, so the saved page shows empty placeholders. We'll have to think more about all this, but it's great to have a concrete point to start from. Now that all the JavaScript refactoring has been merged, writing dynamic code with JS in the notebook will be much cleaner. If anyone is looking for toys of this nature to play with, here's a great summary page of viz JS libraries that could all be very elegantly plugged into the notebook: http://selection.datavisualization.ch Let us know what you come up with! Cheers, f _______________________________________________ IPython-dev mailing list IPython-dev@scipy.org http://mail.scipy.org/mailman/listinfo/ipython-dev
Howdy, On Sat, Jun 2, 2012 at 2:47 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Disadvantages:
* Unclear amount of our REPL/Notebook code will have to be rewritten * The IPython notebook's storage format (SQLite) may not play nicely with all Lustre systems
You mean JSON? IPython uses sqlite only for the user history, and that happens even in the terminal. The notebook makes no additional use of SQLite, the notebook files themselves are JSON files stored directly on disk via regular filesystem mechanisms.
* Possible impedance mismatch between JS libraries used in Notebook and ExtJS4
Very possible.
* My understanding of how WebSockets work doesn't cover how they work when SSH tunneling to bind to a socket, so I am putting potential glitches from this approach under "disadvantage"
That should be fine. We've run qtcosoles and entire clusters with zeromq communications forwarded over SSH. That's not to say there couldn't be some issues with peculiar network/firewall setups (we've recently seen a hard to diagnose such case at a supercomputing facility), but by and large things have been OK on that front. Cheers, f
Hi Fernando, On Mon, Jun 4, 2012 at 12:34 AM, Fernando Perez <fperez.net@gmail.com> wrote:
Howdy,
On Sat, Jun 2, 2012 at 2:47 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Disadvantages:
* Unclear amount of our REPL/Notebook code will have to be rewritten * The IPython notebook's storage format (SQLite) may not play nicely with all Lustre systems
You mean JSON? IPython uses sqlite only for the user history, and that happens even in the terminal. The notebook makes no additional use of SQLite, the notebook files themselves are JSON files stored directly on disk via regular filesystem mechanisms.
Ah, I guess I was referring to Doug's issue here: http://mail.scipy.org/pipermail/ipython-user/2012-May/010107.html While Doug doesn't use yt, his issue was on a system we deploy on at SLAC. We've also had trouble in the past with SQLite on Lustre machines because of file locking, and so I guess what I was getting at was that we'll need to be aware of this possibly stumbling block if IPython becomes part of a piece of core functionality.
* Possible impedance mismatch between JS libraries used in Notebook and ExtJS4
Very possible.
* My understanding of how WebSockets work doesn't cover how they work when SSH tunneling to bind to a socket, so I am putting potential glitches from this approach under "disadvantage"
That should be fine. We've run qtcosoles and entire clusters with zeromq communications forwarded over SSH. That's not to say there couldn't be some issues with peculiar network/firewall setups (we've recently seen a hard to diagnose such case at a supercomputing facility), but by and large things have been OK on that front.
Okay, that's very good to hear! Thanks very much for clearing these things up. I've now had time to study the way the widget code and the new refactored JS in the IPython html notebook are set up, and I think they're both pretty awesome and would be relatively easy to integrate our stuff into, with probably quite large payoffs in terms of maintainability and functionality. If anybody wants to work on this, I think it would be straightforward enough: * Subclass or reimplement the notebook launching in IPython to launch slightly different HTML templates for the notebook itself * Replace our notebook/cell handling code with the CodeMirror-based JS+HTML from IPython (which would be a big step up in functionality by itself) * Make the notebook menus and display play nice, display-wise * Swap out RPC calls to the yt engine with kernel evaluation calls -Matt
Cheers,
f _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
participants (2)
-
Fernando Perez -
Matthew Turk