[IPython-dev] Re: questions about the notebook interface

[Robert Kern]

Fernando Perez wrote:
> Robert Kern wrote:

>> The foremost use cases in my head are the essentially interactive or 
>> demonstrative ones, not the literate programming ones. I think that 
>> people are going to use notebooks to run interactively or generate 
>> static, pretty documents. They are going to use the exported code to 
>> import or run on a server or whatever other "headless" activity. If 
>> you really want to be able to import from a notebook, you could write 
>> an import hook that will export the bare code and import that. 
>> (Disclosure: I never met a literate programming system that I liked.)
> 
> Well, neither have I, but Mathematica does have some strong points.  One of
> its major weaknesses, however, is precisely the impossibility of doing 
> major
> coding from within the notebook interface (the editor sucks rocks).

This is why I think we need to separate these two cases:

1. Writing modules
2. Interacting with interpreter in a REPL (Read-Eval-Print-Loop, i.e. 
ipython)

The interfaces and formats that are good for one aren't necessarily good 
for the other. Problem 1 is a task for a literate programming tool. Now, 
we could certainly write one, and it would be useful, but there are 
existing, capable LP tools out there. If you want to write another one, 
we should address why the existing ones are inadequate and how we are 
going to redo all of the things that they already do well.

But a graphical REPL is something different. A literate program is 
essentially a static document. You're not running it *while* you're 
editing it. That has consequences which I'm going to get to when I add 
to Hans's user stories. Preferably in Wiki form if that managed to make 
it through the server migration.

As I said before, I never met a LP system that I liked. Comments and 
docstrings have always been enough for me. I've also never read a LP 
"book" that was worth the time. None of them made the code more 
understandable or the documentation better written.

In short, I suggest leaving module-type code alone and focusing on a 
notebook REPL.

>> The markup-via-Python seems to me to be difficult to parse without 
>> running the code. Every time I make a markup change or spelling 
>> correction, my excessively long calculation will run again just to 
>> recreate a plot that I've already embedded in the notebook. I should 
>> be able to selectively run the code when I need to (re)generate output.
> 
> What I'd thought was that in rendering mode, no code marked as input would
> ever be run at all.

Yes, you are correct. I had an incorrect picture in my head of how it 
would work.

>> Name collisions. Or else evil hacks to avoid name collisions.
> 
> why?  We can simply request that the top-level markup import is unmodified:
> 
> from notebook import markup
> 
> and the file can then be parsed.  Any markup.foo() calls are considered 
> markup, and the rest is code to be executed. 

That's an arbitrary limitation on the code that can be contained in my 
program. It's also a restriction that is only visible to the person that 
is looking at the plain text. In the graphical REPL, the restriction 
remains, but there's nothing but the ipython's documentation telling you 
that you should never start a line with "markup."

Now, it's a relatively minor restriction and one that I could probably 
live with, but it increments my ick counter.

> As for the problem of 
> people wanting to program their markup, I don't necessarily see that as 
> a bad thing.  The code marked as input doesn't need to be re-executed on 
> all rendering calls (thus negating the problem of rendering being 
> potentially an expensive process), but other non-input code can be 
> executed freely.

That assumes that they're not trying to mix program and markup.

   def f(x, y):
     return x + y
   markup.text("4 + 5 = %s" % f(4, 5))

or

   markup.text(  "| I am | an empty | table |")
   markup.text(  "+------+----------+-------+")
   for x in range(10):
     markup.text("|      |          |       |")

You just *know* that someone's going to try that. Since the primary 
motivation for Python-syntax markup is that users already know the 
syntax, it is confusing to utterly change the semantics. Furthermore, I 
think that it's going to be difficult or impossible to detect prior to 
running the separated codes. And then, they'll probably fail with an 
inscrutable error message.

The XML approach treats everything as data. Text is data. Markup is data 
(okay, it's actually structure, but you get the idea). Code is different 
data. ipython magic commands are another kind of data. They're kept 
separate, and the assumptions about how you treat them and the contexts 
in which they are so treated are kept separate.

The presumption for, "Python syntax wherever possible," is 
understandable, but it's not a good principle in general, I don't think. 
Where it works, it is usually because it is the application of another 
principle, "Do the simplest thing that could possibly work." Often 
Python syntax is the simplest thing that could possibly work because you 
can just exec it and you have your data. That's why it works for certain 
config files or Skencil graphics or saved Chaco plots.

Unfortunately, that's not the case here. To generate the document or 
even just the data structure in memory, you have to parse the file and 
extract the data from an AST which was not designed for this purpose. 
That is going to take a lot of effort that could profitably be used 
elsewhere. We're not short of work that needs to get done.

It used to be the case that XML was usually far from the simplest thing 
that would work, too. SAX and DOM and other Java-isms suck in the Python 
world. You would only touch XML if you had to interoperate with 
something else that needed XML or you wanted to leverage the various XML 
standards. Recently, however, the XML tools available to Python have 
become much more Pythonic. ElementTree (and to a lesser extent, Amara, 
gnosis.xml, and libxml2) changed the landscape. ElementTree provides not 
only parsing and output but also a convenient data structure for 
manipulating the document. That eliminates quote a lot of work that 
could be spent on more interesting things.

>> You can't embed these calls in functions because they won't get called 
>> until the function is run, not when that line of code is executed. And 
>> then they get called every time that function gets called.

I was wrong on this one, too.

def f(x, y):
   markup.text('Add two numbers together like so:")
   return x + y

could work fine with enough AST-fiddling.

>> I don't think that any-old-text-editor is a viable target. It just 
>> doesn't provide any of the real notebook capabilities. Targetting it 
>> as an intermediate platform before a GUI shell is available sacrifices 
>> the capabilities that the GUI shell is going to provide.
> 
> Well, that approach has worked pretty well for Latex.  You can edit it 
> with xemacs/vi/notepad/whatever, and yet the LyX guys managed to build a 
> fancy GUI environment on top of it.  Furthermore, lyx actually handles 
> the document in its own .lyx format, and exports to latex for the final 
> rendering pass.  But if you want, you can embed raw latex into the 
> document via ERT insets.
 >
> I don't see why a similar model can't work for us: a simple enough 
> format that it's valid python syntax, so it can be edited with normal 
> programming tools. A notebook library to render the files, and a GUI on 
> top which enables interactive use of such documents, using the ipython 
> engine.

LaTeX does not have a REPL. It's the static document versus, well, REPL 
distinction again. Python-as-markup and this editing model might work 
for LP, but it greatly restricts the capabilities of a REPL.

>> Allow me to make a suggestion for an intermediate target: Modify 
>> ipython to make a notebook file (in whatever format you go forward 
>> with) as a kind of log. At minimum, it just writes out the In[NN]'s 
>> and Out[NN]'s in the notebook format. Add a %magic to write out notes 
>> to the notebook. Nothing fancy or formatted, they'd mostly be just 
>> reminders that you would expand and format later, not the final 
>> markup. If you're on a roll, add another %magic for pylab mode that 
>> will save the current figure as a .png and write out an entry to the 
>> notebook for including an image.
> 
> For a prototype, a curses-based one could be written fairly easily. It 
> wouldn't be portable in the long term, but it might be enough for a lot 
> of users on *nix systems.

Well, I've got the XML system already working to this state through evil 
hooks. I'll look into dropping into an SVN repository somewhere 
accessible. BTW, whenever you'd complain about how ugly ipython's code 
was and how it was a big mass of tangly code, I'd always think to 
myself, "Nah, it can't be *that* bad!"

It's that bad.  ;-)

Anyways, I think the next easiest target should probably go straight to 
wxScintilla or Py using just plain text (or reST) for markup. There's a 
reason curses is called curses.

Or there's always wxMozilla ....

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter