[IPython-dev] Embedding the HTML notebook

Tue Apr 10 16:39:31 EDT 2012

Hi Brian,

thanks to you and Matthew for kicking off this discussion. It is something
I am interested in as well. I have looked into the codebase as well,
thinking about customizing it, and ran into the same conclusions.

Brian, do you have any (tentative, approximate) timeline for those efforts?

Thanks to all for building these awesome tools.
Jonathan

On Tue, Apr 10, 2012 at 1:38 PM, Brian Granger <ellisonbg at gmail.com> wrote:

> On Tue, Apr 10, 2012 at 5:41 AM, Matthew Turk <matthewturk at gmail.com>
> wrote:
> > Hi Brian,
> >
> > Thanks for your reply.  Over the last couple days I've spent some time
> > investigating, and I was able to make this work with minimal changes
> > -- in advance of going any further, seeing as this is on your radar, I
> > wanted to reply with what I've found to see if it fits in with your
> > vision, or if I should scrap it.
>
> OK great.
>
> > On Mon, Apr 9, 2012 at 6:21 PM, Brian Granger <ellisonbg at gmail.com>
> wrote:
> >> Matthew,
> >>
> >> On Wed, Apr 4, 2012 at 3:46 PM, Matthew Turk <matthewturk at gmail.com>
> wrote:
> >>> Hi there,
> >>>
> >>> After something of a protracted discussion on Fernando's Google+
> >>> stream about embedding the IPython notebook, I started digging into
> >>> the notebook manager code.  As something of an introduction, I'm
> >>> relatively new to this aspect of IPython, but I work on a project (
> >>> yt-project.org ) that currently has a web GUI with a
> >>> (execution-blocking, much less fancy, somewhat clunky) web notebook.
> >>> On top of this we've built a number of display widgets, which really
> >>> are what we spend the most time on and what we're most proud of, but
> >>> more importantly they're what we want to focus our development energy
> >>> on moving forward.
> >>>
> >>> Anyway, this is a long way around of saying we're looking really hard
> >>> at ditching our current cell-REPL system and instead building in the
> >>> IPython notebook, since it's awesome.  I'm trying to come up to speed
> >>> with how the web notebook works, and I have a couple questions about
> >>> the broad feasibility.
> >>>
> >>> To embed as is, it looks like (as shown with NINJA-ide) one can embed
> >>> an iframe to link to a running notebook server.  But rather than doing
> >>> something precisely like that, since we're looking at providing
> >>> additional items such as widgets (although Fernando mentioned that
> >>> interoperation with callbacks is still underway) is it possible to
> >>> extend the templates from the API?
> >>
> >> Right now the answer is no.  We are currently using tornado as the
> >> templating library and it doesn't support loading templates from
> >> multiple directories.  However, we have plans to switch over the
> >> jinja2, which supports this fine.  As part of that change we will also
> >> be reorganizing the templates and javascript code to make it much
> >> easier for people to customize things and reuse things as part of
> >> their own web apps.  This is relatively high on my list of things
> >> todo, but still might take a while to get to.  Part of the problem is
> >> that this stuff is significant enough that it has to wait until after
> >> the 0.13.
> >
> > The change to Jinja2 for the multiple template directories makes sense
> > (although the additional C dependency, with possible pure python
> > fallback, is a tradeoff), although after looking into how
> > RequestHandler is implemented it seems that subclassing handles this
> > very nicely.
>
> Yes, jinja2 is a non-trivial dep, but I think we need this power moving
> forward.
>
> >>
> >>> Inside IPython/frontend/html/handlers.py, the render methods are code
> >>> with calls to specific template names, which then render out HTML
> >>> templates from a path that can be overridden with settings_override.
> >>> I *think* from reading the tornado docs that template_path has to be a
> >>> single string and not an iterable of strings,
> >>
> >> Yes, and this is exactly the weakness of tornado's template renderer
> >> and why we are moving away from it.
> >>
> >>>and the handlers
> >>> themselves are enumerated as classes in the NotebookWebApplication
> >>> handlers.  It was when I got this deep that I started to wonder if
> >>> perhaps there was an obvious, more straightforward way that I was
> >>> missing; if not, I'm definitely willing to do what I can to dive in
> >>> and contribute, if embedding is something of interest to the
> >>> community, too.
> >>>
> >>> If we take as a given that an IPython engine is currently running,
> >>> what would the best route toward embedding a view -- that doesn't
> >>> fully duplicate the contents of static/ and templates/ -- into that
> >>> engine?
> >>
> >> Today there is not easy way of doing this.  You would basically have
> >> to duplicate everything.  After the refactoring, it should be much
> >> easier.
> >
> > What I was able to implement, without the additional dependency of
> > Jinja2, was a single-notebook embedded instance with default and
> > sub-default handlers, allowing for the two basically independent web
> > apps to be served by the same tornado instance.  However, it required
> > a handful of changes to the IPython source which I've put into a
> > (readily-destroyable) fork on github under my username MatthewTurk.  I
> > attempted to comply with the coding practices in the file and in the
> > IPython developer guide.
> >
> > It accomplished this with a couple main changes to the IPython
> > codebase.  The main weakness remains that the HTML fragment that
> > specifically corresponds to the notebook components is not
> > customizable, but it also allows the embedding user to access the
> > IPython components, so setting up their own full HTML fragment should
> > also work.  This is accomplished through:
> >
> > 1) Adding EmbeddedNotebookApp and EmbeddedNotebookWebApplication
> > classes.  These subclass the appropriate classes in the existing
> > implementation.  Both expose a subset of functionality (for instance,
> > no notebook manager), but most importantly the
> > EmbeddedNotebookWebApplication accepts an additional_handlers
> > argument.  These additional handlers will not be prepended with the
> > base_project_url.
> > 2) page.html has been modified to have {%%} blocks around every
> > component of the HTML.  This allows the initial and final HTML tags to
> > be stripped, along with body, script, etc.  embedded_notebook.html
> > subclasses notebook.html and overrides these tags, so that using
> > ExtJS's autoLoad functionality we can load it as a snippet (served by
> > Tornado) and insert it directly into an existing DOM.
> > 3) To accommodate our existing widgets, I extended
> > IPython.Notebook.prototype with an external_handlers attribute.  Any
> > payloads that come through with the type "external_payload" will get
> > passed through to any callback that has been registered with the
> > content['payload_type'].
> >
> > Setting up an embedded application is then just a matter of creating a
> > set of handlers which are fed in as additional_handlers to an instance
> > of EmbeddedNotebookApp.  Passing through the new notebook ID was
> > somewhat tricky, but disallowing access to the notebook manager and
> > redirecting from the base URL /notebook (not
> > base_project_url/notebook) to the correct notebook worked around that.
>
> >From what I can tell, this approach sounds reasonable for your usage
> case and the present condition of the IPython code base.  I don't
> think these changes are appropriate to go into IPython properly,
> especially given the fact that all of this code will be refactored in
> a non-compatible manner.
>
> I can say a bit more about our vision of what the code base will look
> like after refactoring:
>
> * Each page in the web application will have its own directory.
> * Each web service will also have its own directory.
> * Each directory will have its own template, javascript and handlers.
> * Users who want to build a custom app will need to:
>   - write their main program (the equivalent of notebookapp.py) that
> pull in all the right handlers and templates (possibly their own
> subclassed templates).
>  - subclass any handlers
>  - specify their own url scheme.
>
> What I don't think is reasonable is for the main IPython notebook app
> to be a super customizable meta app that covers everyones usage cases.
>  I am strongly -1 on that.  Our philosophy is to give developers well
> isolated components they can use to build whatever they want.  The
> range of options developers will want is simply too broad to try and
> cover them all from a single main program.  Even in IPython, it is
> likely we will have different versions of the main program that
> configure the notebook in different ways.  If we build the components
> well, it will be relatively straightforward to build custom apps using
> them.
>
> > While these changes worked for me to get something going to prototype
> > and experiment with layout inside our app, I understand that they may
> > not fit into the broader vision for the HTML notebook.  If they are of
> > some use I would be more than happy to work on them until they are up
> > to the quality necessary for contribution.  In my conversations with
> > Fernando the interest of the IPython development community in
> > collaborating with other sub-communities in scientific python has been
> > pretty clear, and I would like to do what I can to help with this
> > effort.
>
> We absolutely want to build collaborations with other communities.  In
> this case, I think we need to just get the refactoring done and have
> you try it out as we do the work so we make sure it will work well for
> your usage cases.
>
> The tricky bit is that 1) we have to wait until post 0.13 and 2) the
> changes to the codebase will be so significant that all other
> development on the notebook will have to be frozen.  Because the
> notebook is going to be the foundation of so many things, it is really
> important that we develop a solid architecture from the beginning, and
> that takes time.  But, in our experience, it is well worth it in the
> long run.  It just means things move more slowly at times.  Hope you
> can be patient :)
>
>
> >>
> >>>  Would prepending an additional path to the base_project_url
> >>> work, so that any calls to/from the engine would require that
> >>> base_project_url?  And then, post-instantiation, appending new
> >>> handlers to the NotebookWebApplication, for the "embedding" app's
> >>> templates, which perhaps have to be rendered from strings?
> >>>
> >>> Any ideas would be greatly appreciated, and again, thanks for all your
> >>> hard work on IPython.
> >>
> >> Unfortunately, for now I think the best answer is to wait.  We will
> >> try our best to get to this ASAP though.
> >
> > Ah, I guess I missed that there's a refactor coming.  I will dig
> > further through the IPython-dev archives to try to find out more
> > information about this before we modify any of our code to comply with
> > IPython's current API; most of our users upgrade dependencies
> > relatively slowly, so perhaps it is premature for us to start looking
> > at interoperation with or utilization of the new 0MQ engine system and
> > the HTML notebook.
> >
> > Again, thanks for your reply, and all of your amazing hard work on
> IPython.
>
> Thanks, let's keep in touch.
>
> Cheers,
>
> Brian
>
> > -Matt
> >
> >>
> >> Cheers,
> >>
> >> Brian
> >>
> >>> -Matt
> >>> _______________________________________________
> >>> IPython-dev mailing list
> >>> IPython-dev at scipy.org
> >>> http://mail.scipy.org/mailman/listinfo/ipython-dev
> >>
> >>
> >>
> >> --
> >> Brian E. Granger
> >> Cal Poly State University, San Luis Obispo
> >> bgranger at calpoly.edu and ellisonbg at gmail.com
> >> _______________________________________________
> >> IPython-dev mailing list
> >> IPython-dev at scipy.org
> >> http://mail.scipy.org/mailman/listinfo/ipython-dev
> > _______________________________________________
> > IPython-dev mailing list
> > IPython-dev at scipy.org
> > http://mail.scipy.org/mailman/listinfo/ipython-dev
>
>
>
> --
> Brian E. Granger
> Cal Poly State University, San Luis Obispo
> bgranger at calpoly.edu and ellisonbg at gmail.com
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>

-- 
Jonathan Rocher, PhD
Scientific software developer
Enthought, Inc.
jrocher at enthought.com
1-512-536-1057
http://www.enthought.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20120410/347ec1cf/attachment.html>