[IPython-dev] Embedding the HTML notebook

Brian Granger ellisonbg at gmail.com
Tue Apr 10 17:45:50 EDT 2012 

On Tue, Apr 10, 2012 at 1:39 PM, Jonathan Rocher <jrocher at enthought.com> wrote:
> Hi Brian,
>
> thanks to you and Matthew for kicking off this discussion. It is something I
> am interested in as well. I have looked into the codebase as well, thinking
> about customizing it, and ran into the same conclusions.
>
> Brian, do you have any (tentative, approximate) timeline for those efforts?

We won't even start this until post 0.13, which will be at least 4-6
weeks.  It might take a month after that.  So maybe 3 months total,
being somewhat conservative.

Cheers,

Brian

> Thanks to all for building these awesome tools.
> Jonathan
>
>
> On Tue, Apr 10, 2012 at 1:38 PM, Brian Granger <ellisonbg at gmail.com> wrote:
>>
>> On Tue, Apr 10, 2012 at 5:41 AM, Matthew Turk <matthewturk at gmail.com>
>> wrote:
>> > Hi Brian,
>> >
>> > Thanks for your reply.  Over the last couple days I've spent some time
>> > investigating, and I was able to make this work with minimal changes
>> > -- in advance of going any further, seeing as this is on your radar, I
>> > wanted to reply with what I've found to see if it fits in with your
>> > vision, or if I should scrap it.
>>
>> OK great.
>>
>> > On Mon, Apr 9, 2012 at 6:21 PM, Brian Granger <ellisonbg at gmail.com>
>> > wrote:
>> >> Matthew,
>> >>
>> >> On Wed, Apr 4, 2012 at 3:46 PM, Matthew Turk <matthewturk at gmail.com>
>> >> wrote:
>> >>> Hi there,
>> >>>
>> >>> After something of a protracted discussion on Fernando's Google+
>> >>> stream about embedding the IPython notebook, I started digging into
>> >>> the notebook manager code.  As something of an introduction, I'm
>> >>> relatively new to this aspect of IPython, but I work on a project (
>> >>> yt-project.org ) that currently has a web GUI with a
>> >>> (execution-blocking, much less fancy, somewhat clunky) web notebook.
>> >>> On top of this we've built a number of display widgets, which really
>> >>> are what we spend the most time on and what we're most proud of, but
>> >>> more importantly they're what we want to focus our development energy
>> >>> on moving forward.
>> >>>
>> >>> Anyway, this is a long way around of saying we're looking really hard
>> >>> at ditching our current cell-REPL system and instead building in the
>> >>> IPython notebook, since it's awesome.  I'm trying to come up to speed
>> >>> with how the web notebook works, and I have a couple questions about
>> >>> the broad feasibility.
>> >>>
>> >>> To embed as is, it looks like (as shown with NINJA-ide) one can embed
>> >>> an iframe to link to a running notebook server.  But rather than doing
>> >>> something precisely like that, since we're looking at providing
>> >>> additional items such as widgets (although Fernando mentioned that
>> >>> interoperation with callbacks is still underway) is it possible to
>> >>> extend the templates from the API?
>> >>
>> >> Right now the answer is no.  We are currently using tornado as the
>> >> templating library and it doesn't support loading templates from
>> >> multiple directories.  However, we have plans to switch over the
>> >> jinja2, which supports this fine.  As part of that change we will also
>> >> be reorganizing the templates and javascript code to make it much
>> >> easier for people to customize things and reuse things as part of
>> >> their own web apps.  This is relatively high on my list of things
>> >> todo, but still might take a while to get to.  Part of the problem is
>> >> that this stuff is significant enough that it has to wait until after
>> >> the 0.13.
>> >
>> > The change to Jinja2 for the multiple template directories makes sense
>> > (although the additional C dependency, with possible pure python
>> > fallback, is a tradeoff), although after looking into how
>> > RequestHandler is implemented it seems that subclassing handles this
>> > very nicely.
>>
>> Yes, jinja2 is a non-trivial dep, but I think we need this power moving
>> forward.
>>
>> >>
>> >>> Inside IPython/frontend/html/handlers.py, the render methods are code
>> >>> with calls to specific template names, which then render out HTML
>> >>> templates from a path that can be overridden with settings_override.
>> >>> I *think* from reading the tornado docs that template_path has to be a
>> >>> single string and not an iterable of strings,
>> >>
>> >> Yes, and this is exactly the weakness of tornado's template renderer
>> >> and why we are moving away from it.
>> >>
>> >>>and the handlers
>> >>> themselves are enumerated as classes in the NotebookWebApplication
>> >>> handlers.  It was when I got this deep that I started to wonder if
>> >>> perhaps there was an obvious, more straightforward way that I was
>> >>> missing; if not, I'm definitely willing to do what I can to dive in
>> >>> and contribute, if embedding is something of interest to the
>> >>> community, too.
>> >>>
>> >>> If we take as a given that an IPython engine is currently running,
>> >>> what would the best route toward embedding a view -- that doesn't
>> >>> fully duplicate the contents of static/ and templates/ -- into that
>> >>> engine?
>> >>
>> >> Today there is not easy way of doing this.  You would basically have
>> >> to duplicate everything.  After the refactoring, it should be much
>> >> easier.
>> >
>> > What I was able to implement, without the additional dependency of
>> > Jinja2, was a single-notebook embedded instance with default and
>> > sub-default handlers, allowing for the two basically independent web
>> > apps to be served by the same tornado instance.  However, it required
>> > a handful of changes to the IPython source which I've put into a
>> > (readily-destroyable) fork on github under my username MatthewTurk.  I
>> > attempted to comply with the coding practices in the file and in the
>> > IPython developer guide.
>> >
>> > It accomplished this with a couple main changes to the IPython
>> > codebase.  The main weakness remains that the HTML fragment that
>> > specifically corresponds to the notebook components is not
>> > customizable, but it also allows the embedding user to access the
>> > IPython components, so setting up their own full HTML fragment should
>> > also work.  This is accomplished through:
>> >
>> > 1) Adding EmbeddedNotebookApp and EmbeddedNotebookWebApplication
>> > classes.  These subclass the appropriate classes in the existing
>> > implementation.  Both expose a subset of functionality (for instance,
>> > no notebook manager), but most importantly the
>> > EmbeddedNotebookWebApplication accepts an additional_handlers
>> > argument.  These additional handlers will not be prepended with the
>> > base_project_url.
>> > 2) page.html has been modified to have {%%} blocks around every
>> > component of the HTML.  This allows the initial and final HTML tags to
>> > be stripped, along with body, script, etc.  embedded_notebook.html
>> > subclasses notebook.html and overrides these tags, so that using
>> > ExtJS's autoLoad functionality we can load it as a snippet (served by
>> > Tornado) and insert it directly into an existing DOM.
>> > 3) To accommodate our existing widgets, I extended
>> > IPython.Notebook.prototype with an external_handlers attribute.  Any
>> > payloads that come through with the type "external_payload" will get
>> > passed through to any callback that has been registered with the
>> > content['payload_type'].
>> >
>> > Setting up an embedded application is then just a matter of creating a
>> > set of handlers which are fed in as additional_handlers to an instance
>> > of EmbeddedNotebookApp.  Passing through the new notebook ID was
>> > somewhat tricky, but disallowing access to the notebook manager and
>> > redirecting from the base URL /notebook (not
>> > base_project_url/notebook) to the correct notebook worked around that.
>>
>> >From what I can tell, this approach sounds reasonable for your usage
>> case and the present condition of the IPython code base.  I don't
>> think these changes are appropriate to go into IPython properly,
>> especially given the fact that all of this code will be refactored in
>> a non-compatible manner.
>>
>> I can say a bit more about our vision of what the code base will look
>> like after refactoring:
>>
>> * Each page in the web application will have its own directory.
>> * Each web service will also have its own directory.
>> * Each directory will have its own template, javascript and handlers.
>> * Users who want to build a custom app will need to:
>>   - write their main program (the equivalent of notebookapp.py) that
>> pull in all the right handlers and templates (possibly their own
>> subclassed templates).
>>  - subclass any handlers
>>  - specify their own url scheme.
>>
>> What I don't think is reasonable is for the main IPython notebook app
>> to be a super customizable meta app that covers everyones usage cases.
>>  I am strongly -1 on that.  Our philosophy is to give developers well
>> isolated components they can use to build whatever they want.  The
>> range of options developers will want is simply too broad to try and
>> cover them all from a single main program.  Even in IPython, it is
>> likely we will have different versions of the main program that
>> configure the notebook in different ways.  If we build the components
>> well, it will be relatively straightforward to build custom apps using
>> them.
>>
>> > While these changes worked for me to get something going to prototype
>> > and experiment with layout inside our app, I understand that they may
>> > not fit into the broader vision for the HTML notebook.  If they are of
>> > some use I would be more than happy to work on them until they are up
>> > to the quality necessary for contribution.  In my conversations with
>> > Fernando the interest of the IPython development community in
>> > collaborating with other sub-communities in scientific python has been
>> > pretty clear, and I would like to do what I can to help with this
>> > effort.
>>
>> We absolutely want to build collaborations with other communities.  In
>> this case, I think we need to just get the refactoring done and have
>> you try it out as we do the work so we make sure it will work well for
>> your usage cases.
>>
>> The tricky bit is that 1) we have to wait until post 0.13 and 2) the
>> changes to the codebase will be so significant that all other
>> development on the notebook will have to be frozen.  Because the
>> notebook is going to be the foundation of so many things, it is really
>> important that we develop a solid architecture from the beginning, and
>> that takes time.  But, in our experience, it is well worth it in the
>> long run.  It just means things move more slowly at times.  Hope you
>> can be patient :)
>>
>>
>> >>
>> >>>  Would prepending an additional path to the base_project_url
>> >>> work, so that any calls to/from the engine would require that
>> >>> base_project_url?  And then, post-instantiation, appending new
>> >>> handlers to the NotebookWebApplication, for the "embedding" app's
>> >>> templates, which perhaps have to be rendered from strings?
>> >>>
>> >>> Any ideas would be greatly appreciated, and again, thanks for all your
>> >>> hard work on IPython.
>> >>
>> >> Unfortunately, for now I think the best answer is to wait.  We will
>> >> try our best to get to this ASAP though.
>> >
>> > Ah, I guess I missed that there's a refactor coming.  I will dig
>> > further through the IPython-dev archives to try to find out more
>> > information about this before we modify any of our code to comply with
>> > IPython's current API; most of our users upgrade dependencies
>> > relatively slowly, so perhaps it is premature for us to start looking
>> > at interoperation with or utilization of the new 0MQ engine system and
>> > the HTML notebook.
>> >
>> > Again, thanks for your reply, and all of your amazing hard work on
>> > IPython.
>>
>> Thanks, let's keep in touch.
>>
>> Cheers,
>>
>> Brian
>>
>> > -Matt
>> >
>> >>
>> >> Cheers,
>> >>
>> >> Brian
>> >>
>> >>> -Matt
>> >>> _______________________________________________
>> >>> IPython-dev mailing list
>> >>> IPython-dev at scipy.org
>> >>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>> >>
>> >>
>> >>
>> >> --
>> >> Brian E. Granger
>> >> Cal Poly State University, San Luis Obispo
>> >> bgranger at calpoly.edu and ellisonbg at gmail.com
>> >> _______________________________________________
>> >> IPython-dev mailing list
>> >> IPython-dev at scipy.org
>> >> http://mail.scipy.org/mailman/listinfo/ipython-dev
>> > _______________________________________________
>> > IPython-dev mailing list
>> > IPython-dev at scipy.org
>> > http://mail.scipy.org/mailman/listinfo/ipython-dev
>>
>>
>>
>> --
>> Brian E. Granger
>> Cal Poly State University, San Luis Obispo
>> bgranger at calpoly.edu and ellisonbg at gmail.com
>> _______________________________________________
>> IPython-dev mailing list
>> IPython-dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
>
>
>
> --
> Jonathan Rocher, PhD
> Scientific software developer
> Enthought, Inc.
> jrocher at enthought.com
> 1-512-536-1057
> http://www.enthought.com
>
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>



-- 
Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger at calpoly.edu and ellisonbg at gmail.com

More information about the IPython-dev mailing list