[IPython-dev] Re: questions about the notebook interface

[Fernando Perez]

Robert Kern wrote:
> Toni Alatalo wrote:
> 
>>Greetings.
>>
>>i have some questions about the notebook interface (as it's described in 
>>http://ipython.scipy.org/google_soc/ipnb_google_soc.pdf). must leave 
>>right now so am trying to be brief, sorry for possible unclarities:
> 
> 
> I've been thinking a bit about this too, since I was going to volunteer 
> some work on this if no one got picked for it by Google. Looks like that 
> won't be necessary. Congratulations!
> 
> 
>>1. reasoning for true Python over special string / comments
>>
>>as e.g. rst is nice for writing documents in itself, and can also be 
>>used to write text, make tables, images etc. in Python files (the 
>>original purpose), i'd be interested in hearing the main arguments from 
>>the 'lengthy discussions' (referred in the spec) for deciding on true 
>>Python instead. am probably bying the result already, but would still 
>>like to hear the reasoning.
> 
> 
> I personally don't agree with it. I wasn't a party to those discussions, 
> so I'm sure most of my complaints have already been addressed, but I'm 
> going to air them out here anyways. Because I can. :-)

And because you should :)  Seriously, while I've spent quite a bit of time
thinking about this, and John Hunter and I discussed it at length, it's a
fairly tricky problem full of potential pitfalls.  I _really_ appreciate the
feedback, and at this point I consider the sketch I wrote up as only a bare
draft, fully open to a complete overhaul.  I want to find the right solution
to this problem, wherever the idea comes from.  So keep the debate coming, by
all means.

> The foremost use cases in my head are the essentially interactive or 
> demonstrative ones, not the literate programming ones. I think that 
> people are going to use notebooks to run interactively or generate 
> static, pretty documents. They are going to use the exported code to 
> import or run on a server or whatever other "headless" activity. If you 
> really want to be able to import from a notebook, you could write an 
> import hook that will export the bare code and import that. (Disclosure: 
> I never met a literate programming system that I liked.)

Well, neither have I, but Mathematica does have some strong points.  One of
its major weaknesses, however, is precisely the impossibility of doing major
coding from within the notebook interface (the editor sucks rocks).

> The markup-via-Python seems to me to be difficult to parse without 
> running the code. Every time I make a markup change or spelling 
> correction, my excessively long calculation will run again just to 
> recreate a plot that I've already embedded in the notebook. I should be 
> able to selectively run the code when I need to (re)generate output.

What I'd thought was that in rendering mode, no code marked as input would
ever be run at all.

> Name collisions. Or else evil hacks to avoid name collisions.

why?  We can simply request that the top-level markup import is unmodified:

from notebook import markup

and the file can then be parsed.  Any markup.foo() calls are considered 
markup, and the rest is code to be executed.  As for the problem of people 
wanting to program their markup, I don't necessarily see that as a bad thing. 
  The code marked as input doesn't need to be re-executed on all rendering 
calls (thus negating the problem of rendering being potentially an expensive 
process), but other non-input code can be executed freely.

> You can't embed these calls in functions because they won't get called 
> until the function is run, not when that line of code is executed. And 
> then they get called every time that function gets called.
> 
> I don't think that any-old-text-editor is a viable target. It just 
> doesn't provide any of the real notebook capabilities. Targetting it as 
> an intermediate platform before a GUI shell is available sacrifices the 
> capabilities that the GUI shell is going to provide.

Well, that approach has worked pretty well for Latex.  You can edit it with 
xemacs/vi/notepad/whatever, and yet the LyX guys managed to build a fancy GUI 
environment on top of it.  Furthermore, lyx actually handles the document in 
its own .lyx format, and exports to latex for the final rendering pass.  But 
if you want, you can embed raw latex into the document via ERT insets.

I don't see why a similar model can't work for us: a simple enough format that 
it's valid python syntax, so it can be edited with normal programming tools. 
A notebook library to render the files, and a GUI on top which enables 
interactive use of such documents, using the ipython engine.

> Allow me to make a suggestion for an intermediate target: Modify ipython 
> to make a notebook file (in whatever format you go forward with) as a 
> kind of log. At minimum, it just writes out the In[NN]'s and Out[NN]'s 
> in the notebook format. Add a %magic to write out notes to the notebook. 
> Nothing fancy or formatted, they'd mostly be just reminders that you 
> would expand and format later, not the final markup. If you're on a 
> roll, add another %magic for pylab mode that will save the current 
> figure as a .png and write out an entry to the notebook for including an 
> image.

For a prototype, a curses-based one could be written fairly easily. It 
wouldn't be portable in the long term, but it might be enough for a lot of 
users on *nix systems.

>>2. usage targets of the library
>>
>>ReST and LaTEX are pretty clear, but how about possible other types of 
>>documents, like ones with dynamic / interactive content -- should that 
>>be addressed by the notebook lib? e.g. Slithy 
>>(http://isotropic.org/uw/slithy/) provides a pretty nice way to make 
>>dynamic/interactive diagrams with python (as functions that receive vars 
>>(user controllable with slides in the player, and/or time for 
>>animations) and output drawings). what (other) usages should be 
>>considered? or only 'print targeted' stuff (like the mathreader things 
>>seem almost like pdf)?
> 
> 
> For this summer, just assume that anything that wants to do something 
> funky will create a context in which to do it. Leave how that happens up 
> to the people writing the GUI. Allow things to fail gracefully when it 
> doesn't make sense (e.g. Pygame animations in HTML).

Yup, fully agreed.

>>one other area here is linking, which 
>>rest/latex/pdf and of course html do support too, and there perhaps 
>>granularity .. like to have support for so-called 'purple numbers' 
>>http://www.eekim.com/software/purple/purple.html .
> 
> 
> Intra-notebook linking gets a big +1 from me. As an offhand thought, 
> every time In[NN] or Out[NN] is used in an expression, it should get a 
> link back to the original when the expression is being syntax colored.

Yes, a labeling system for cross-references should be part of the document 
model.  In latex, these naturally expand to \label and \ref calls, with 
similar stuff for reST or HTML.

>>4. ipython and non-ipython
>>
>>i suppose the basic notebook module would be independent from ipython, 
>>and then some separate glue for it would be made for ipython (in the 
>>ipython cvs), like has been done with gnuplot - right? or is there 
>>something ipython specific about the project i'm missing? indeed the 
>>real-time / interactive aspect is interesting, but does that imply much 
>>for the API design? and in this case, should be a separate project, 
>>hosted on some of the sourceforge forks (berlios, gna.org or something?) 
>>.. would 'notebook' be a good name for the package, and pynotebook for 
>>the project, or what? or is it more tied to ipython and should start there?

Yes, the document-focused project (Toni's) should be mostly 
ipython-independent code-wise, though we'll carry the development using 
ipython's resources (website, SVN, lists, etc.).  At some point, we'll begin 
adding support in ipython for it, so Toni may need to put code into ipython as 
well.  In the meantime, Tzanko's project will  probably need to cut into 
ipython much earlier, as there are internal changes needed to separate the 
engine enough for clean gui embedding.

> 
> Development should probably be done on the scipy.org server. Hopefully 
> the SVN server will be up soon for ipython. Most of the development will 
> probably happen outside of the ipython tree itself, I imagine.

I think the transition to the new scipy server is mostly done, so we should 
have an SVN repo available very soon for all this.

Cheers,

f