From Fernando.Perez at colorado.edu  Fri Jul  1 02:41:46 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Fri, 01 Jul 2005 00:41:46 -0600
Subject: [IPython-dev] Re: questions about the notebook interface
In-Reply-To: <d9qcpe$7as$1@sea.gmane.org>
References: <42C00157.2020808@an.org> <d9qcpe$7as$1@sea.gmane.org>
Message-ID: <42C4E5AA.90504@colorado.edu>

Robert Kern wrote:
> Toni Alatalo wrote:
> 
>>Greetings.
>>
>>i have some questions about the notebook interface (as it's described in 
>>http://ipython.scipy.org/google_soc/ipnb_google_soc.pdf). must leave 
>>right now so am trying to be brief, sorry for possible unclarities:
> 
> 
> I've been thinking a bit about this too, since I was going to volunteer 
> some work on this if no one got picked for it by Google. Looks like that 
> won't be necessary. Congratulations!
> 
> 
>>1. reasoning for true Python over special string / comments
>>
>>as e.g. rst is nice for writing documents in itself, and can also be 
>>used to write text, make tables, images etc. in Python files (the 
>>original purpose), i'd be interested in hearing the main arguments from 
>>the 'lengthy discussions' (referred in the spec) for deciding on true 
>>Python instead. am probably bying the result already, but would still 
>>like to hear the reasoning.
> 
> 
> I personally don't agree with it. I wasn't a party to those discussions, 
> so I'm sure most of my complaints have already been addressed, but I'm 
> going to air them out here anyways. Because I can. :-)

And because you should :)  Seriously, while I've spent quite a bit of time
thinking about this, and John Hunter and I discussed it at length, it's a
fairly tricky problem full of potential pitfalls.  I _really_ appreciate the
feedback, and at this point I consider the sketch I wrote up as only a bare
draft, fully open to a complete overhaul.  I want to find the right solution
to this problem, wherever the idea comes from.  So keep the debate coming, by
all means.

> The foremost use cases in my head are the essentially interactive or 
> demonstrative ones, not the literate programming ones. I think that 
> people are going to use notebooks to run interactively or generate 
> static, pretty documents. They are going to use the exported code to 
> import or run on a server or whatever other "headless" activity. If you 
> really want to be able to import from a notebook, you could write an 
> import hook that will export the bare code and import that. (Disclosure: 
> I never met a literate programming system that I liked.)

Well, neither have I, but Mathematica does have some strong points.  One of
its major weaknesses, however, is precisely the impossibility of doing major
coding from within the notebook interface (the editor sucks rocks).

> The markup-via-Python seems to me to be difficult to parse without 
> running the code. Every time I make a markup change or spelling 
> correction, my excessively long calculation will run again just to 
> recreate a plot that I've already embedded in the notebook. I should be 
> able to selectively run the code when I need to (re)generate output.

What I'd thought was that in rendering mode, no code marked as input would
ever be run at all.

> Name collisions. Or else evil hacks to avoid name collisions.

why?  We can simply request that the top-level markup import is unmodified:

from notebook import markup

and the file can then be parsed.  Any markup.foo() calls are considered 
markup, and the rest is code to be executed.  As for the problem of people 
wanting to program their markup, I don't necessarily see that as a bad thing. 
  The code marked as input doesn't need to be re-executed on all rendering 
calls (thus negating the problem of rendering being potentially an expensive 
process), but other non-input code can be executed freely.

> You can't embed these calls in functions because they won't get called 
> until the function is run, not when that line of code is executed. And 
> then they get called every time that function gets called.
> 
> I don't think that any-old-text-editor is a viable target. It just 
> doesn't provide any of the real notebook capabilities. Targetting it as 
> an intermediate platform before a GUI shell is available sacrifices the 
> capabilities that the GUI shell is going to provide.

Well, that approach has worked pretty well for Latex.  You can edit it with 
xemacs/vi/notepad/whatever, and yet the LyX guys managed to build a fancy GUI 
environment on top of it.  Furthermore, lyx actually handles the document in 
its own .lyx format, and exports to latex for the final rendering pass.  But 
if you want, you can embed raw latex into the document via ERT insets.

I don't see why a similar model can't work for us: a simple enough format that 
it's valid python syntax, so it can be edited with normal programming tools. 
A notebook library to render the files, and a GUI on top which enables 
interactive use of such documents, using the ipython engine.

> Allow me to make a suggestion for an intermediate target: Modify ipython 
> to make a notebook file (in whatever format you go forward with) as a 
> kind of log. At minimum, it just writes out the In[NN]'s and Out[NN]'s 
> in the notebook format. Add a %magic to write out notes to the notebook. 
> Nothing fancy or formatted, they'd mostly be just reminders that you 
> would expand and format later, not the final markup. If you're on a 
> roll, add another %magic for pylab mode that will save the current 
> figure as a .png and write out an entry to the notebook for including an 
> image.

For a prototype, a curses-based one could be written fairly easily. It 
wouldn't be portable in the long term, but it might be enough for a lot of 
users on *nix systems.

>>2. usage targets of the library
>>
>>ReST and LaTEX are pretty clear, but how about possible other types of 
>>documents, like ones with dynamic / interactive content -- should that 
>>be addressed by the notebook lib? e.g. Slithy 
>>(http://isotropic.org/uw/slithy/) provides a pretty nice way to make 
>>dynamic/interactive diagrams with python (as functions that receive vars 
>>(user controllable with slides in the player, and/or time for 
>>animations) and output drawings). what (other) usages should be 
>>considered? or only 'print targeted' stuff (like the mathreader things 
>>seem almost like pdf)?
> 
> 
> For this summer, just assume that anything that wants to do something 
> funky will create a context in which to do it. Leave how that happens up 
> to the people writing the GUI. Allow things to fail gracefully when it 
> doesn't make sense (e.g. Pygame animations in HTML).

Yup, fully agreed.

>>one other area here is linking, which 
>>rest/latex/pdf and of course html do support too, and there perhaps 
>>granularity .. like to have support for so-called 'purple numbers' 
>>http://www.eekim.com/software/purple/purple.html .
> 
> 
> Intra-notebook linking gets a big +1 from me. As an offhand thought, 
> every time In[NN] or Out[NN] is used in an expression, it should get a 
> link back to the original when the expression is being syntax colored.

Yes, a labeling system for cross-references should be part of the document 
model.  In latex, these naturally expand to \label and \ref calls, with 
similar stuff for reST or HTML.

>>4. ipython and non-ipython
>>
>>i suppose the basic notebook module would be independent from ipython, 
>>and then some separate glue for it would be made for ipython (in the 
>>ipython cvs), like has been done with gnuplot - right? or is there 
>>something ipython specific about the project i'm missing? indeed the 
>>real-time / interactive aspect is interesting, but does that imply much 
>>for the API design? and in this case, should be a separate project, 
>>hosted on some of the sourceforge forks (berlios, gna.org or something?) 
>>.. would 'notebook' be a good name for the package, and pynotebook for 
>>the project, or what? or is it more tied to ipython and should start there?

Yes, the document-focused project (Toni's) should be mostly 
ipython-independent code-wise, though we'll carry the development using 
ipython's resources (website, SVN, lists, etc.).  At some point, we'll begin 
adding support in ipython for it, so Toni may need to put code into ipython as 
well.  In the meantime, Tzanko's project will  probably need to cut into 
ipython much earlier, as there are internal changes needed to separate the 
engine enough for clean gui embedding.

> 
> Development should probably be done on the scipy.org server. Hopefully 
> the SVN server will be up soon for ipython. Most of the development will 
> probably happen outside of the ipython tree itself, I imagine.

I think the transition to the new scipy server is mostly done, so we should 
have an SVN repo available very soon for all this.

Cheers,

f



From Fernando.Perez at colorado.edu  Fri Jul  1 02:46:00 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Fri, 01 Jul 2005 00:46:00 -0600
Subject: [FWD] Re: [IPython-dev] questions about the notebook interface
In-Reply-To: <mailman.6278.1120121394.917.ipython-dev@scipy.net>
References: <mailman.6278.1120121394.917.ipython-dev@scipy.net>
Message-ID: <42C4E6A8.8050605@colorado.edu>

[I'm forwarding this manually to the list, please subscribe for future 
postings as the ipython lists unfortunately are off limits for 
non-subscribers, due to excessive spam]

> Subject: Re: [IPython-dev] questions about the notebook interface
> From: Hans Meine <meine at kogs1.informatik.uni-hamburg.de>
> Date: Thu, 30 Jun 2005 11:49:00 +0200
> To: ipython-dev at scipy.net
> 
> Hi!
> 
> On Tuesday 28 June 2005 17:36, Toni Alatalo wrote:
> 
>>>I've thought that XEmacs could be the first environment to target for
>>>interactive use, since it already almost has everything we need in its
>>>python-mode/ipython combination.  In the meantime, Tsanko can work on
>>>the more ambitious GUI-oriented project he applied for.
>>
>>fine with me, as emacs is what i always use for programming otherwise too.
>>
>>.. looking forward to hearing the GUI ideas too
> 
> 
> I just want to stress the part of "the notebook interface" that is of greatest 
> importance to me, since most of these discussion did not seem to be concerned 
> with it:  The interactive programming aspect.
> 
> Actually, I don't care much about the final layouting; the biggest feature of 
> a "notebook interface" in my eyes would be that I can incrementally write and 
> debug a program, correcting my mistakes and ending up with a "clean history".
> 
> Right now, ipython is the best environment for my work, but I have to manually 
> extract "good" code fragments.  The most important thing in my eyes would be 
> a possibility to edit and re-run my code without *appending* to the history, 
> but just correcting and extending the existing "In [xx]: " paragraphs.  That 
> would be especially handy in combination with loops, too.
> 
> Maybe I should propose developing / collecting proper user stories first, then 
> you could decide on their priorities and start creating solutions for 
> existing problems (there may well be people focused on layout, which could 
> give another user story).
> 
> Some short user stories(*):
> ------------------------------ >8 snip 8< ----------------------------------
> I am planning to analyse a multidimensional data structure.  I am writing a 
> loop, collecting data but then realize that I need to initialize a variable 
> *before* the loop.  In ipython, I can only cancel the current edit, type the 
> initialization and fetch all lines from the history (or retype them) 
> one-by-one.  What I'd like to be able to do is to just press cursor up, 
> insert a line, and continue further down, then execute the whole loop.
> 
> I successfully analyzed some data derived from images.  Now I want to change 
> some preprocessing parameters and re-run the analysis.  Again, I want to 
> insert / change some code at the beginning and re-execute several blocks of 
> commands that use the now changed variables.
> 
> It would be useful if some sort of dependency graph could be generated.  E.g. 
> when I change a "cell" (piece of code) which (re-)assigns variables, all code 
> fragments (cells) that depend on these results (use that variable) should be 
> marked as "dirty".  Automatic execution of these cells should not be the 
> goal, since that could be unwanted by the user, need a long time, and/or 
> destroy/overwrite precious results.
> 
> For image processing purposes, I want to be able to have image objects that 
> result from python expressions shown as embedded images in the notebook.
> 
> Similarly, I'd like to have graphs (e.g. from Gnuplot.py) be embedded in the 
> notebook.
> 
> A friend of mine wants to be able to write longer Mathematic-like notebooks, 
> and needs proper namespacing for chapters/sections, so that variable names 
> can be re-used.
> ------------------------------ >8 snip 8< ----------------------------------
> 
> Actually, the xemacs approach seems to be promising to me, I am looking 
> forward to further information / results / input.



From Fernando.Perez at colorado.edu  Fri Jul  1 03:40:40 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Fri, 01 Jul 2005 01:40:40 -0600
Subject: [FWD] Re: [IPython-dev] questions about the notebook interface
In-Reply-To: <42C4E6A8.8050605@colorado.edu>
References: <mailman.6278.1120121394.917.ipython-dev@scipy.net>
	<42C4E6A8.8050605@colorado.edu>
Message-ID: <42C4F378.3000005@colorado.edu>

>>From: Hans Meine <meine at kogs1.informatik.uni-hamburg.de>

>>I just want to stress the part of "the notebook interface" that is of greatest 
>>importance to me, since most of these discussion did not seem to be concerned 
>>with it:  The interactive programming aspect.
>>
>>Actually, I don't care much about the final layouting; the biggest feature of 
>>a "notebook interface" in my eyes would be that I can incrementally write and 
>>debug a program, correcting my mistakes and ending up with a "clean history".
>>
>>Right now, ipython is the best environment for my work, but I have to manually 
>>extract "good" code fragments.  The most important thing in my eyes would be 
>>a possibility to edit and re-run my code without *appending* to the history, 
>>but just correcting and extending the existing "In [xx]: " paragraphs.  That 
>>would be especially handy in combination with loops, too.

Indeed, the interactive aspect is pretty dear to my heart (I wouldn't have 
spent all this time developing ipython if I dindn't care about that :).  And I 
think that Tzanko has a good shot at making headway pretty quickly on this 
front with ipython+pycrust.  A while ago I tried to embed ipython in IDLE and 
pycrust; while it didn't quite work, it seemed to me like the remaining 
problems were easy to solve.  It needs a bit of time, since the right solution 
(a quick hack could be done in a few hours) should involve some code 
reorganization.

>>
>>Maybe I should propose developing / collecting proper user stories first, then 
>>you could decide on their priorities and start creating solutions for 
>>existing problems (there may well be people focused on layout, which could 
>>give another user story).
>>
>>Some short user stories(*):
>>------------------------------ >8 snip 8< ----------------------------------
>>I am planning to analyse a multidimensional data structure.  I am writing a 
>>loop, collecting data but then realize that I need to initialize a variable 
>>*before* the loop.  In ipython, I can only cancel the current edit, type the 
>>initialization and fetch all lines from the history (or retype them) 
>>one-by-one.  What I'd like to be able to do is to just press cursor up, 
>>insert a line, and continue further down, then execute the whole loop.

Yup, this is really annoying.  For much of this, Xemacs+ipython.el can be a 
better solution, as you can keep a scratch.py buffer where you do multi-line 
edits, and run them in the ipython buffer with C-c C-|.


>>I successfully analyzed some data derived from images.  Now I want to change 
>>some preprocessing parameters and re-run the analysis.  Again, I want to 
>>insert / change some code at the beginning and re-execute several blocks of 
>>commands that use the now changed variables.

This one is already doable in ipython: just call

macro mm 12:30 34 39:50

and then typing mm is like re-executing all those lines.  All code is executed 
clean, so variable changes are recognized:

In [1]: print 'x is:',x
---------------------------------------------------------------------------
exceptions.NameError                                 Traceback (most recent 
call last)

/usr/local/home/fperez/<console>

NameError: name 'x' is not defined
x is:
In [2]: x=99

In [3]: macro mm 1
Macro `mm` created. To execute, type its name (without quotes).
Macro contents:
print 'x is:',x

In [4]: mm
Out[4]: Executing Macro...
x is: 99

In [5]: x='hi'

In [6]: mm
Out[6]: Executing Macro...
x is: hi

See macro? for further details.

>>It would be useful if some sort of dependency graph could be generated.  E.g. 
>>when I change a "cell" (piece of code) which (re-)assigns variables, all code 
>>fragments (cells) that depend on these results (use that variable) should be 
>>marked as "dirty".  Automatic execution of these cells should not be the 
>>goal, since that could be unwanted by the user, need a long time, and/or 
>>destroy/overwrite precious results.

Very ambitious, but an idea worth keeping in mind for the future.

>>For image processing purposes, I want to be able to have image objects that 
>>result from python expressions shown as embedded images in the notebook.
>>
>>Similarly, I'd like to have graphs (e.g. from Gnuplot.py) be embedded in the 
>>notebook.

My current favorite idea is for the notebook format to have some kind of 
include directive like:

markup.graphic('foo.png',options...)

Once a fancy GUI is up and running which can display text and inline graphics, 
they could be shown on-screen.  But at least rendering them to PS/PDF should 
be easy.  Note that the GUI for this is not trivial: I think that only Qt4 
(released yesterday) has a rich text widget with inline graphics support.  I 
may be wrong, but I think that none of the text widgets in GTK, Tk or WX have 
this capability.  That's why I'm concentrating on having proper backend 
support for now, along with  a basic GUI which can at least display the text 
parts and provide the interactive support.

This would give us the possibility to:

1. Edit the notebooks 'by hand' with plain editors, even if interactive 
support isn't there.

2. Have a gui with interactive support that can provide the ability to re-edit 
the same input group multiple times without re-generating new input cells.

If both systems can operate on the same files, one could switch back and forth 
between the editor and the interactive environment, depending on the task at 
hand (heavy-duty coding or multiple tests).  This might even allow a system 
like Mozilla's composer (now Nvu), which has both a WYSIWIG tab and a raw html 
tab for the same document, and one can switch back and forth between them.

>>A friend of mine wants to be able to write longer Mathematic-like notebooks, 
>>and needs proper namespacing for chapters/sections, so that variable names 
>>can be re-used.

This can be done by issuing a 'del' statement at the start, much like in 
Mathematica you need to call Clear[] on symbols.

>>------------------------------ >8 snip 8< ----------------------------------
>>
>>Actually, the xemacs approach seems to be promising to me, I am looking 
>>forward to further information / results / input.

Thanks for your input!

Best,

f



From tsanko at gmail.com  Fri Jul  1 12:42:03 2005
From: tsanko at gmail.com (Tzanko Matev)
Date: Fri, 01 Jul 2005 19:42:03 +0300
Subject: [IPython-dev] Notebook format idea
Message-ID: <1120236123.5582.33.camel@tzanko>

Hi Toni,

I have a question about the notebook format. How do you plan to
implement python code? I guess that something like

code = Code("""
   #some python code
""")

is not a good idea, because no text editor with python support would
recognize the string as python code. Probably something like

Code.start()
#some python code
code = Code.end()

would work. When the file is converted to a printable format you first
parse it using the parse module and convert all the code segments of the
above form to something like the first example, and then you run the
result to produce the file you need.

Right now I'm using my own very simple notebook format. For now I will
keep on using it, because the internal representation of data does not
depend much on the file format.

Best regards,
Tzanko

P.S. Sorry for the private email



From tsanko at gmail.com  Fri Jul  1 14:13:53 2005
From: tsanko at gmail.com (Tzanko Matev)
Date: Fri, 01 Jul 2005 21:13:53 +0300
Subject: [IPython-dev] Proposed IPython changes
Message-ID: <1120241633.5582.73.camel@tzanko>

Hi Fernando,

I'm going to start implementing the python code plug-in in the notebook
interface. I plan on using the Shell class from PyCrust. For that I need
an interpreter class which mimics the behaviour of PyCrust's own. I will
try to make the needed modifications to InteractiveShell. At first,
however, I plan to convert make_IPython into a number of smaller
functions. Then I will start making methods for configuring an
InteractiveShell instance, that these functions will use instead of
directly messing with the internals. And after that I will implement
what is needed for the Shell class of PyCrust.

>From new_design.pdf I understand that part of the cleanup is a new class
structure which has a better API. My idea is this: make a class Kernel,
which will contain all the IPython functionality. Then the class
InteractiveShell will simply be a terminal interface to Kernel, and the 
notebook program will be a graphical interface. Kernel should contain
methods for running code, loading configuration, loading logs, etc. I'm
not that familiar with the current code base so I don't know how long
this rewrite would take. For now I will only implement parts that are
needed for the GUI. Please tell me if you like this idea, and what do
you think the new class design should be. I want to make the GUI in such
a way that the transition to the new design be as easy as possible.

Best regards,
Tzanko



From Fernando.Perez at colorado.edu  Fri Jul  1 14:27:27 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Fri, 01 Jul 2005 12:27:27 -0600
Subject: [IPython-dev] Proposed IPython changes
In-Reply-To: <1120241633.5582.73.camel@tzanko>
References: <1120241633.5582.73.camel@tzanko>
Message-ID: <42C58B0F.4040603@colorado.edu>

Hi Tzanko,

Tzanko Matev wrote:
> Hi Fernando,
> 
> I'm going to start implementing the python code plug-in in the notebook
> interface. I plan on using the Shell class from PyCrust. For that I need
> an interpreter class which mimics the behaviour of PyCrust's own. I will
> try to make the needed modifications to InteractiveShell. At first,
> however, I plan to convert make_IPython into a number of smaller
> functions. Then I will start making methods for configuring an
> InteractiveShell instance, that these functions will use instead of
> directly messing with the internals. And after that I will implement
> what is needed for the Shell class of PyCrust.
> 
>>From new_design.pdf I understand that part of the cleanup is a new class
> structure which has a better API. My idea is this: make a class Kernel,
> which will contain all the IPython functionality. Then the class
> InteractiveShell will simply be a terminal interface to Kernel, and the 
> notebook program will be a graphical interface. Kernel should contain
> methods for running code, loading configuration, loading logs, etc. I'm
> not that familiar with the current code base so I don't know how long
> this rewrite would take. For now I will only implement parts that are
> needed for the GUI. Please tell me if you like this idea, and what do
> you think the new class design should be. I want to make the GUI in such
> a way that the transition to the new design be as easy as possible.

Yes, that's roughly what I have in mind for the long-term structure.  Having a 
kernel which only takes commands over a specified channel and returns output 
will also allow in the future the possibility to restart the kernel while 
keeping the user interface active, running a kernel on a remote machine, 
controlling multiple kernels from a single front-end, etc.

Please note that the make_ipython routine is just one monster hack for 
historical reasons.  There is absolutely no clean design reason to have it, 
it's really nothing more than an initializer for the ipython instance, because 
when I started writing ipython, I had my own $PYTHONSTARTUP code which I 
merged with a class-based system (IPP).  The quickest way to merge the two 
codes was to keep the IPP __init__, and then run my $PYTHONSTARTUP script 
after that, having renamed it to make_ipython.  But in reality, the class 
constructor should be just one.

The publicly visible classes live in Shell.py, and currently (because of this 
make_ipython mess) they hold an internal reference to the ipython instance 
returned by make_ipython.  Ultimately all that should go away.

There are two other things which should be cleaned up in this process, so they 
don't get in the way.  These are also historical artifacts from the original 
IPP code, they are just gross misfeatures in the current code:

1. (easy) - the logging system is a class mixin.  It shoulde be a standalone 
system, and iypthon should simply instantiate it once and hold a reference to 
this instance at runtime.  Making it a mixin just creates a namespace mess.

2. (not so easy) - the magics are also a mixin.  The namespace mess is worse 
here, though I've tried to prepare things for separation since long ago.  Now 
all magics refer back to the shell via a self.shell attribute, rather than 
relying on self directly.  This means that once separated, as long as the 
standalone magic system is given a valid self.shell pointer to the ipython 
shell, they should continue to run.

Let me know how this all sounds to you.  I know it will take a bit of time to 
familiarize yourself with the tangled hairball that is the ipython engine, but 
nothing here is particularly hard.

Regards,

f



From rkern at ucsd.edu  Fri Jul  1 17:48:45 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Fri, 01 Jul 2005 14:48:45 -0700
Subject: [IPython-dev] Re: questions about the notebook interface
In-Reply-To: <42C4E5AA.90504@colorado.edu>
References: <42C00157.2020808@an.org> <d9qcpe$7as$1@sea.gmane.org>
	<42C4E5AA.90504@colorado.edu>
Message-ID: <da4d9g$jpn$1@sea.gmane.org>

Fernando Perez wrote:
> Robert Kern wrote:

>> The foremost use cases in my head are the essentially interactive or 
>> demonstrative ones, not the literate programming ones. I think that 
>> people are going to use notebooks to run interactively or generate 
>> static, pretty documents. They are going to use the exported code to 
>> import or run on a server or whatever other "headless" activity. If 
>> you really want to be able to import from a notebook, you could write 
>> an import hook that will export the bare code and import that. 
>> (Disclosure: I never met a literate programming system that I liked.)
> 
> Well, neither have I, but Mathematica does have some strong points.  One of
> its major weaknesses, however, is precisely the impossibility of doing 
> major
> coding from within the notebook interface (the editor sucks rocks).

This is why I think we need to separate these two cases:

1. Writing modules
2. Interacting with interpreter in a REPL (Read-Eval-Print-Loop, i.e. 
ipython)

The interfaces and formats that are good for one aren't necessarily good 
for the other. Problem 1 is a task for a literate programming tool. Now, 
we could certainly write one, and it would be useful, but there are 
existing, capable LP tools out there. If you want to write another one, 
we should address why the existing ones are inadequate and how we are 
going to redo all of the things that they already do well.

But a graphical REPL is something different. A literate program is 
essentially a static document. You're not running it *while* you're 
editing it. That has consequences which I'm going to get to when I add 
to Hans's user stories. Preferably in Wiki form if that managed to make 
it through the server migration.

As I said before, I never met a LP system that I liked. Comments and 
docstrings have always been enough for me. I've also never read a LP 
"book" that was worth the time. None of them made the code more 
understandable or the documentation better written.

In short, I suggest leaving module-type code alone and focusing on a 
notebook REPL.

>> The markup-via-Python seems to me to be difficult to parse without 
>> running the code. Every time I make a markup change or spelling 
>> correction, my excessively long calculation will run again just to 
>> recreate a plot that I've already embedded in the notebook. I should 
>> be able to selectively run the code when I need to (re)generate output.
> 
> What I'd thought was that in rendering mode, no code marked as input would
> ever be run at all.

Yes, you are correct. I had an incorrect picture in my head of how it 
would work.

>> Name collisions. Or else evil hacks to avoid name collisions.
> 
> why?  We can simply request that the top-level markup import is unmodified:
> 
> from notebook import markup
> 
> and the file can then be parsed.  Any markup.foo() calls are considered 
> markup, and the rest is code to be executed. 

That's an arbitrary limitation on the code that can be contained in my 
program. It's also a restriction that is only visible to the person that 
is looking at the plain text. In the graphical REPL, the restriction 
remains, but there's nothing but the ipython's documentation telling you 
that you should never start a line with "markup."

Now, it's a relatively minor restriction and one that I could probably 
live with, but it increments my ick counter.

> As for the problem of 
> people wanting to program their markup, I don't necessarily see that as 
> a bad thing.  The code marked as input doesn't need to be re-executed on 
> all rendering calls (thus negating the problem of rendering being 
> potentially an expensive process), but other non-input code can be 
> executed freely.

That assumes that they're not trying to mix program and markup.

   def f(x, y):
     return x + y
   markup.text("4 + 5 = %s" % f(4, 5))

or

   markup.text(  "| I am | an empty | table |")
   markup.text(  "+------+----------+-------+")
   for x in range(10):
     markup.text("|      |          |       |")

You just *know* that someone's going to try that. Since the primary 
motivation for Python-syntax markup is that users already know the 
syntax, it is confusing to utterly change the semantics. Furthermore, I 
think that it's going to be difficult or impossible to detect prior to 
running the separated codes. And then, they'll probably fail with an 
inscrutable error message.

The XML approach treats everything as data. Text is data. Markup is data 
(okay, it's actually structure, but you get the idea). Code is different 
data. ipython magic commands are another kind of data. They're kept 
separate, and the assumptions about how you treat them and the contexts 
in which they are so treated are kept separate.

The presumption for, "Python syntax wherever possible," is 
understandable, but it's not a good principle in general, I don't think. 
Where it works, it is usually because it is the application of another 
principle, "Do the simplest thing that could possibly work." Often 
Python syntax is the simplest thing that could possibly work because you 
can just exec it and you have your data. That's why it works for certain 
config files or Skencil graphics or saved Chaco plots.

Unfortunately, that's not the case here. To generate the document or 
even just the data structure in memory, you have to parse the file and 
extract the data from an AST which was not designed for this purpose. 
That is going to take a lot of effort that could profitably be used 
elsewhere. We're not short of work that needs to get done.

It used to be the case that XML was usually far from the simplest thing 
that would work, too. SAX and DOM and other Java-isms suck in the Python 
world. You would only touch XML if you had to interoperate with 
something else that needed XML or you wanted to leverage the various XML 
standards. Recently, however, the XML tools available to Python have 
become much more Pythonic. ElementTree (and to a lesser extent, Amara, 
gnosis.xml, and libxml2) changed the landscape. ElementTree provides not 
only parsing and output but also a convenient data structure for 
manipulating the document. That eliminates quote a lot of work that 
could be spent on more interesting things.

>> You can't embed these calls in functions because they won't get called 
>> until the function is run, not when that line of code is executed. And 
>> then they get called every time that function gets called.

I was wrong on this one, too.

def f(x, y):
   markup.text('Add two numbers together like so:")
   return x + y

could work fine with enough AST-fiddling.

>> I don't think that any-old-text-editor is a viable target. It just 
>> doesn't provide any of the real notebook capabilities. Targetting it 
>> as an intermediate platform before a GUI shell is available sacrifices 
>> the capabilities that the GUI shell is going to provide.
> 
> Well, that approach has worked pretty well for Latex.  You can edit it 
> with xemacs/vi/notepad/whatever, and yet the LyX guys managed to build a 
> fancy GUI environment on top of it.  Furthermore, lyx actually handles 
> the document in its own .lyx format, and exports to latex for the final 
> rendering pass.  But if you want, you can embed raw latex into the 
> document via ERT insets.
 >
> I don't see why a similar model can't work for us: a simple enough 
> format that it's valid python syntax, so it can be edited with normal 
> programming tools. A notebook library to render the files, and a GUI on 
> top which enables interactive use of such documents, using the ipython 
> engine.

LaTeX does not have a REPL. It's the static document versus, well, REPL 
distinction again. Python-as-markup and this editing model might work 
for LP, but it greatly restricts the capabilities of a REPL.

>> Allow me to make a suggestion for an intermediate target: Modify 
>> ipython to make a notebook file (in whatever format you go forward 
>> with) as a kind of log. At minimum, it just writes out the In[NN]'s 
>> and Out[NN]'s in the notebook format. Add a %magic to write out notes 
>> to the notebook. Nothing fancy or formatted, they'd mostly be just 
>> reminders that you would expand and format later, not the final 
>> markup. If you're on a roll, add another %magic for pylab mode that 
>> will save the current figure as a .png and write out an entry to the 
>> notebook for including an image.
> 
> For a prototype, a curses-based one could be written fairly easily. It 
> wouldn't be portable in the long term, but it might be enough for a lot 
> of users on *nix systems.

Well, I've got the XML system already working to this state through evil 
hooks. I'll look into dropping into an SVN repository somewhere 
accessible. BTW, whenever you'd complain about how ugly ipython's code 
was and how it was a big mass of tangly code, I'd always think to 
myself, "Nah, it can't be *that* bad!"

It's that bad.  ;-)

Anyways, I think the next easiest target should probably go straight to 
wxScintilla or Py using just plain text (or reST) for markup. There's a 
reason curses is called curses.

Or there's always wxMozilla ....

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From hans_meine at gmx.net  Sun Jul  3 12:35:19 2005
From: hans_meine at gmx.net (Hans Meine)
Date: Sun, 3 Jul 2005 18:35:19 +0200
Subject: [IPython-dev] questions about the notebook interface
In-Reply-To: <42C16E88.901@an.org>
References: <42C00157.2020808@an.org> <42C055DF.9040505@colorado.edu>
	<42C16E88.901@an.org>
Message-ID: <200507031835.19199.hans_meine@gmx.net>

Hi!

On Tuesday 28 June 2005 17:36, Toni Alatalo wrote:
> > I've thought that XEmacs could be the first environment to target for
> > interactive use, since it already almost has everything we need in its
> > python-mode/ipython combination.  In the meantime, Tsanko can work on
> > the more ambitious GUI-oriented project he applied for.
>
> fine with me, as emacs is what i always use for programming otherwise too.
>
> .. looking forward to hearing the GUI ideas too

I just want to stress the part of "the notebook interface" that is of greatest 
importance to me, since most of these discussion did not seem to be concerned 
with it:  The interactive programming aspect.

Actually, I don't care much about the final layouting; the biggest feature of 
a "notebook interface" in my eyes would be that I can incrementally write and 
debug a program, correcting my mistakes and ending up with a "clean history".

Right now, ipython is the best environment for my work, but I have to manually 
extract "good" code fragments.  The most important thing in my eyes would be 
a possibility to edit and re-run my code without *appending* to the history, 
but just correcting and extending the existing "In [xx]: " paragraphs.  That 
would be especially handy in combination with loops, too.

Maybe I should propose developing / collecting proper user stories first, then 
you could decide on their priorities and start creating solutions for 
existing problems (there may well be people focused on layout, which could 
give another user story).

Some short user stories(*):
------------------------------ >8 snip 8< ----------------------------------
I am planning to analyse a multidimensional data structure.  I am writing a 
loop, collecting data but then realize that I need to initialize a variable 
*before* the loop.  In ipython, I can only cancel the current edit, type the 
initialization and fetch all lines from the history (or retype them) 
one-by-one.  What I'd like to be able to do is to just press cursor up, 
insert a line, and continue further down, then execute the whole loop.

I successfully analyzed some data derived from images.  Now I want to change 
some preprocessing parameters and re-run the analysis.  Again, I want to 
insert / change some code at the beginning and re-execute several blocks of 
commands that use the now changed variables.

It would be useful if some sort of dependency graph could be generated.  E.g. 
when I change a "cell" (piece of code) which (re-)assigns variables, all code 
fragments (cells) that depend on these results (use that variable) should be 
marked as "dirty".  Automatic execution of these cells should not be the 
goal, since that could be unwanted by the user, need a long time, and/or 
destroy/overwrite precious results.

For image processing purposes, I want to be able to have image objects that 
result from python expressions shown as embedded images in the notebook.

Similarly, I'd like to have graphs (e.g. from Gnuplot.py) be embedded in the 
notebook.

A friend of mine wants to be able to write longer Mathematic-like notebooks, 
and needs proper namespacing for chapters/sections, so that variable names 
can be re-used.
------------------------------ >8 snip 8< ----------------------------------

Actually, the xemacs approach seems to be promising to me, I am looking 
forward to further information / results / input.

-- 
Ciao, /  /
     /--/
    /  / ANS

*: DISCLAIMER: I am no professional software engineer, so my notion of "user 
stories" could be a little bit simplified, but hopefully still useful.



From rkern at ucsd.edu  Sun Jul  3 23:59:29 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Sun, 03 Jul 2005 20:59:29 -0700
Subject: [IPython-dev] Notebook roadmap
Message-ID: <daac75$hlt$1@sea.gmane.org>

Fernando and I hashed out a few things on the phone today. Short 
version: XML wins for now, even if I had to threaten to kidnap 
Fernando's cat to do it. Long version:

   http://www.scipy.org/wikis/featurerequests/NoteBook

There's a page there for adding user stories. I've already added the 
ones Hans has posted and some of my own. If you would like to add your 
own stories (aka "feature requests"), please make yourself an account 
and edit the Wiki to your liking.

Documentation on the XML format and API will be forthcoming. Hopefully, 
we will have SVN space by Tuesday or Wednesday.

P.S. No I didn't threaten to kidnap his cat. Fluffy is perfectly safe.

For now....

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From antont at an.org  Mon Jul  4 07:18:52 2005
From: antont at an.org (Toni Alatalo)
Date: Mon, 04 Jul 2005 14:18:52 +0300
Subject: [IPython-dev] Notebook roadmap
In-Reply-To: <daac75$hlt$1@sea.gmane.org>
References: <daac75$hlt$1@sea.gmane.org>
Message-ID: <42C91B1C.503@an.org>

Robert Kern wrote:

> Fernando and I hashed out a few things on the phone today.

Great!

BTW: Fernando mentioned the meeting possibilities in one early post -- 
perhaps VOIP using Skype or something would be good at some point, but 
FYI also that I'm usually hanging on Freenode IRC as antont and we could 
set a channel there too (#ipython #notebook #scipy or something..)

> Short version: XML wins for now, even if I had to threaten to kidnap 
> Fernando's cat to do it. Long version:
>   http://www.scipy.org/wikis/featurerequests/NoteBook

ok. as the last thing last week i also took a brief look at ElementTree, 
and actually had decided now to continue the work by looking into it 
more if there would be no info guiding otherwise, so good to get this 
confirmation.

> There's a page there for adding user stories. I've already added the 
> ones Hans has posted and some of my own. If you would like to add your 
> own stories (aka "feature requests"), please make yourself an account 
> and edit the Wiki to your liking.

thanks - actually the main thing i had in mind for the start of this 
project was to ask for use cases / user stories, 'cause the requirements 
haven't been too clear at all. but it was good also to do little warm-up 
coding too to get an idea.

was also fun to see that the Scipy tutorial was suggested as the target, 
'cause was reading just that last week to further study an example 
notebook :) .. IIRC the current one is written in LaTEX, i suppose the 
source is somewhere too.

> Documentation on the XML format and API will be forthcoming. 
> Hopefully, we will have SVN space by Tuesday or Wednesday.

ok. i'll study the Docutils DTD and the workings (perhaps internals) of 
ElementTree before that, so can comment on those then. btw, i joined 
docutils-dev and have been wondering how they see ElementTree .. as 
docutils documents are element trees too, perhaps it would make sense 
even for docutils to use the elementree lib (am not sure yet about what 
it provides to) .. if it would be written today (probably is not worth a 
rewrite tho).

~Toni

P.S. to Tzanko's question / idea about code in the books -- i have no 
thoughts on that yet. will try to keep Robert's REPL arguments in mind 
for that, and the remarks from Hans Meine and others.



From hans_meine at gmx.net  Mon Jul  4 07:36:44 2005
From: hans_meine at gmx.net (Hans Meine)
Date: Mon, 4 Jul 2005 13:36:44 +0200
Subject: [IPython-dev] Re: questions about the notebook interface
In-Reply-To: <42C4E5AA.90504@colorado.edu>
References: <42C00157.2020808@an.org> <d9qcpe$7as$1@sea.gmane.org>
	<42C4E5AA.90504@colorado.edu>
Message-ID: <200507041336.44271.hans_meine@gmx.net>

On Friday 01 July 2005 08:41, Fernando Perez wrote:
> > Toni Alatalo wrote:
> > I don't think that any-old-text-editor is a viable target. It just
> > doesn't provide any of the real notebook capabilities. Targetting it as
> > an intermediate platform before a GUI shell is available sacrifices the
> > capabilities that the GUI shell is going to provide.
>
> Well, that approach has worked pretty well for Latex.  You can edit it with
> xemacs/vi/notepad/whatever, and yet the LyX guys managed to build a fancy
> GUI environment on top of it.  Furthermore, lyx actually handles the
> document in its own .lyx format, and exports to latex for the final
> rendering pass.  But if you want, you can embed raw latex into the document
> via ERT insets.

Again, I want to stress the importance of the interactive programming aspect 
of the interface.  The LaTeX-way has the disadvantage that LyX basically 
contains a "LaTeX light" rendering engine, since calling LaTeX is not 
feasible for interactive feedback.

For programming, writing the program in an editor and running it from a shell 
(or even via a keyboard shortcut from within your IDE) is already possible 
today, but that's where ipython jumps in as an improvement.  I would be 
interested in a notebook frontend building on that, basically expanding the 
(already quite powerful) editing possibilities.

> I don't see why a similar model can't work for us: a simple enough format
> that it's valid python syntax, so it can be edited with normal programming
> tools. A notebook library to render the files, and a GUI on top which
> enables interactive use of such documents, using the ipython engine.

I understand that you're more interested in the rendering part, but I am much 
more interested in the interactive GUI.  Without a powerful, comfortable GUI, 
I won't be interested in created the documents I would be able to print out.  
(Admittedly, even then I would seldomly use the printing..)

> For a prototype, a curses-based one could be written fairly easily. It
> wouldn't be portable in the long term, but it might be enough for a lot of
> users on *nix systems.

That would be a good start AFAICS.  Anybody interested in working on sth. like 
that?

Ciao, /  /
     /--/
    /  / ANS



From hans_meine at gmx.net  Mon Jul  4 08:43:17 2005
From: hans_meine at gmx.net (Hans Meine)
Date: Mon, 4 Jul 2005 14:43:17 +0200
Subject: [FWD] Re: [IPython-dev] questions about the notebook interface
In-Reply-To: <42C4F378.3000005@colorado.edu>
References: <mailman.6278.1120121394.917.ipython-dev@scipy.net>
	<42C4E6A8.8050605@colorado.edu> <42C4F378.3000005@colorado.edu>
Message-ID: <200507041443.18054.hans_meine@gmx.net>

On Friday 01 July 2005 09:40, Fernando Perez wrote:
> >>From: Hans Meine <meine at kogs1.informatik.uni-hamburg.de>
BTW: Sorry for reposting my mail, I did not see/read your forward due to time 
constraints..

> Indeed, the interactive aspect is pretty dear to my heart (I wouldn't have
> spent all this time developing ipython if I dindn't care about that :). 
> And I think that Tzanko has a good shot at making headway pretty quickly on
> this front with ipython+pycrust.  A while ago I tried to embed ipython in
> IDLE and pycrust; while it didn't quite work, it seemed to me like the
> remaining problems were easy to solve.

Cool.  That's what I dreamt of.. ;-)

> Yup, this is really annoying.  For much of this, Xemacs+ipython.el can be a
> better solution, as you can keep a scratch.py buffer where you do
> multi-line edits, and run them in the ipython buffer with C-c C-|.

I did not find ipython.el that impressive; it surely has advantages 
(multi-line editing), but also disadvantages (crude shell feeling).

> >>I successfully analyzed some data derived from images.  Now I want to
> >> change some preprocessing parameters and re-run the analysis.  Again, I
> >> want to insert / change some code at the beginning and re-execute
> >> several blocks of commands that use the now changed variables.
>
> This one is already doable in ipython: just call
>
> macro mm 12:30 34 39:50
>
> and then typing mm is like re-executing all those lines.

Ah yes, I repeatedly made mental notes to myself to try that out, but did not 
use it yet.  Indeed, that seems to fill a gap.  Thanks for your pointers!

> My current favorite idea is for the notebook format to have some kind of
> include directive like:
>
> markup.graphic('foo.png',options...)
>
> Once a fancy GUI is up and running which can display text and inline
> graphics, they could be shown on-screen.  But at least rendering them to
> PS/PDF should be easy.  Note that the GUI for this is not trivial: I think
> that only Qt4 (released yesterday) has a rich text widget with inline
> graphics support.  I may be wrong, but I think that none of the text
> widgets in GTK, Tk or WX have this capability.

I know Qt3 well and it "supports"(*) it, and I think I saw some 
graphics-capable richtext in another toolkit-demo, too.  Hmm, gtk-demo does 
not have it (2.4.9).

*: Qt4 will have a *much* nicer richtext API though.  The 3.3 one is hard to 
extend anyways (e.g. by subclassing, since quite some stuff is private).

> That's why I'm 
> concentrating on having proper backend support for now, along with  a basic
> GUI which can at least display the text parts and provide the interactive
> support.

Good plan.

> This would give us the possibility to:
>
> 1. Edit the notebooks 'by hand' with plain editors, even if interactive
> support isn't there.
>
> 2. Have a gui with interactive support that can provide the ability to
> re-edit the same input group multiple times without re-generating new input
> cells.
>
> If both systems can operate on the same files, one could switch back and
> forth between the editor and the interactive environment, depending on the
> task at hand (heavy-duty coding or multiple tests).

I love that idea.

> >>A friend of mine wants to be able to write longer Mathematic-like
> >> notebooks, and needs proper namespacing for chapters/sections, so that
> >> variable names can be re-used.
>
> This can be done by issuing a 'del' statement at the start, much like in
> Mathematica you need to call Clear[] on symbols.

Hmm.  Does that still allow to re-run code in the previous section that refers 
to the deleted variables?  My impression is that 'del' is not enough, but 
there need to be separate namespaces, no?

Ciao, /  /
     /--/
    /  / ANS



From shlomme at gmx.net  Mon Jul  4 14:08:42 2005
From: shlomme at gmx.net (Torsten Marek)
Date: Mon, 04 Jul 2005 20:08:42 +0200
Subject: [IPython-dev] Some notebook thoughts
Message-ID: <42C97B2A.7090809@gmx.net>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi all,

I read with great pleasure that XML has been taken instead of Python as the
markup language for Notebook. It's not that the same goal could not be reached
with Python, but XML gives you a whole lot of stuff for free when it comes to
parsing.

I added some user stories to the wiki and I'd like to elaborate on them more
generally:

When I thought about the notebook GUI, I doubted that any existing text editor
widget might be flexible enough to be used for what I had in mind. There are
"normal" text areas (In [xx]s and Out [xx]s), but there are a lot of other
widgets (I just call them like that, but a fancy term like notebooklets might be
better), for any kind of picture, tables, comments, rich texts, code fragment
editors, pretty-printed data structures, other widgets, MathML ... you name it.

For each widget type (and In and Out would also be widgets), there is a
node-type in the markup file and a handler in the renderer (be it a static one
to HTML, LaTeX, PDF, SVG or a GUI interface). The mininum interface would be
handlers for In, Out and a handler for stuff from print statements.

For a GUI that supports all kinds of widgets, a canvas widget like OGL (from Wx)
or QCanvas (from Qt) might be a better choice. However, for a small
implementation, Scintilla might be enough, as long as the underlying
architecture is flexible enough.

Am I making any sense with that or is that too far out in the woods?

greetings

Torsten

- --
Torsten Marek <shlomme at gmx.net>
ID: A244C858 -- FP: 1902 0002 5DFC 856B F146  894C 7CC5 451E A244 C858
Keyserver: subkeys.pgp.net

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFCyXsqfMVFHqJEyFgRApGAAJ0c+tqadZnFC3L8y5kADAy4oP0+ZACgoW9M
QSNz7ibWvRkozh+ac0HIJ5o=
=ZeN5
-----END PGP SIGNATURE-----



From rkern at ucsd.edu  Mon Jul  4 17:19:20 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Mon, 04 Jul 2005 14:19:20 -0700
Subject: [IPython-dev] Re: Some notebook thoughts
In-Reply-To: <42C97B2A.7090809@gmx.net>
References: <42C97B2A.7090809@gmx.net>
Message-ID: <dac94r$tmq$1@sea.gmane.org>

Torsten Marek wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Hi all,
> 
> I read with great pleasure that XML has been taken instead of Python as the
> markup language for Notebook. It's not that the same goal could not be reached
> with Python, but XML gives you a whole lot of stuff for free when it comes to
> parsing.
> 
> I added some user stories to the wiki and I'd like to elaborate on them more
> generally:
> 
> When I thought about the notebook GUI, I doubted that any existing text editor
> widget might be flexible enough to be used for what I had in mind. There are
> "normal" text areas (In [xx]s and Out [xx]s), but there are a lot of other
> widgets (I just call them like that, but a fancy term like notebooklets might be
> better), for any kind of picture, tables, comments, rich texts, code fragment
> editors, pretty-printed data structures, other widgets, MathML ... you name it.
> 
> For each widget type (and In and Out would also be widgets), there is a
> node-type in the markup file and a handler in the renderer (be it a static one
> to HTML, LaTeX, PDF, SVG or a GUI interface). The mininum interface would be
> handlers for In, Out and a handler for stuff from print statements.
> 
> For a GUI that supports all kinds of widgets, a canvas widget like OGL (from Wx)
> or QCanvas (from Qt) might be a better choice. However, for a small
> implementation, Scintilla might be enough, as long as the underlying
> architecture is flexible enough.
> 
> Am I making any sense with that or is that too far out in the woods?

A little too far into the woods, I think.  :-)

I think that we can do everything that we need to do with a decent rich 
text editor that shows images. Tables, comments, rich texts, code 
fragment editors, and pretty-printed data structures are all text. 
Pictures, SVG, embedded LaTeX formulae, MathML, movies, Kiva contexts, 
etc. can all be shown as static pictures for display. Clicking on them 
could activate editors or viewers in other windows. It doesn't have to 
look gorgeous on-screen; it just has to be functional.

That said, Scintilla ultimately won't be the widget of choice. It 
doesn't show images, and it really doesn't do HTML-like rich text well 
either. It is really only suited for syntax highlighted source code 
editors. For my personal use, that actually won't be too bad. If I can 
write reST as my text between In/Out cells, using indentation to control 
the nesting of those cells in the document, I will really be quite 
happy. But we, as a whole, really do want to be able to show embedded 
images and have real rich text.

It's the nesting of those cells within rich text that kills the idea of 
implementing it all on a QCanvas or similar object. We *don't* want to 
rewrite a rich text editor widget. One idea is to simply have a linear 
arrangement of cells: Rich Text, In, In, Out, In, Rich Text, In, Out, 
etc. each cell getting it's own widget layed out in, e.g. a 
wx.ScrolledWindow. That model fails, however, when (In , In, Out, In) 
are actually supposed to be three levels down:

   <document>
     <section>
       <section>
         <paragraph>
           Stuff
         <In>
         <In>
         <Out>
         <In>
         <paragraph>
           More stuff.

The situation might be recoverable, but it's going to be easier on us if 
we can shove everything into a single rich text editor widget.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From hans_meine at gmx.net  Tue Jul  5 07:26:45 2005
From: hans_meine at gmx.net (Hans Meine)
Date: Tue, 5 Jul 2005 13:26:45 +0200
Subject: [IPython-dev] Re: Some notebook thoughts
In-Reply-To: <dac94r$tmq$1@sea.gmane.org>
References: <42C97B2A.7090809@gmx.net> <dac94r$tmq$1@sea.gmane.org>
Message-ID: <200507051326.45657.hans_meine@gmx.net>

On Monday 04 July 2005 23:19, Robert Kern wrote:
> It's the nesting of those cells within rich text that kills the idea of
> implementing it all on a QCanvas or similar object. We *don't* want to
> rewrite a rich text editor widget.
Right.  I also think rich-text widgets are more appropriate than general 
canvas widgets.

> One idea is to simply have a linear 
> arrangement of cells: Rich Text, In, In, Out, In, Rich Text, In, Out,
> etc. each cell getting it's own widget layed out in, e.g. a
> wx.ScrolledWindow. That model fails, however, when (In , In, Out, In)
> are actually supposed to be three levels down: [...]

Good point.  I want to add another little one; we fiddled around with several 
setups for a (Qt-based) python shell, and one problem was indentation. The 
terminal's main author really wants the prompts to be left-aligned, but 
output to be indented, so that you can immediately recognize the beginnings 
of paragraphs.  Let me sketch it:

>>> a = [1,2,3]
>>> print a
    [1, 2, 3]
>>> foo
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    NameError: name 'foo' is not defined
>>> for i in range(3):
...     b = i
...
>>>

That'd be another user story for a GUI (anybody cares to add it for me?).  
Actually I don't care much about it, and there will probably be other means 
to quickly differentiate between the different cells, but anyhow.

Oh, and another one:

One might want to have the output colored differently depending on whether it 
came through stdout or stderr; e.g. red for errors/warnings.

Ciao, /  /
     /--/
    /  / ANS



From edreamleo at charter.net  Tue Jul  5 10:11:39 2005
From: edreamleo at charter.net (Edward K. Ream)
Date: Tue, 5 Jul 2005 09:11:39 -0500
Subject: [IPython-dev] Leo, IPython and the summer of code
Message-ID: <003d01c5816b$765c2040$6500a8c0@edwardx330hgzb>

There are slight inaccuracies in section 2.2 (complexity level) and section 
1 (overview) of the description of the interactive notebooks project at 
http://IPython.scipy.org/google_soc/ipnb_google_soc.pdf

Section 2.2 states, "This project is relatively complicated.it will require 
a fair amount of thinking in terms of overall library architecture."

Section 1 states, "The full-blown implementation of a graphical shell 
supporting this kind of work model is probably too ambitious for a summer 
project. Simultaneous support for rich-text editing, embedded graphics and 
syntax-highlighted code is extremely complex, and likely to require far more 
effort than can be mustered by an individual developer for a short-term 
project."

Actually, this project is already complete.  Leo, 
http://webpages.charter.net/edreamleo/front.html, already _is_ an 
interactive notebook for Python and IPython.  Any remaining work on this 
project, if done within the framework of Leo, would likely involve tasks of 
one hour to one day in duration.  I can state this with confidence because I 
am Leo's author.

What is Leo you ask?  Leo is full featured outlining editor.  Leo is Open 
Software (Python license), written in 100% pure Python using Tkinter.  Leo 
has an extremely active group of users.  Many are wildly enthusiastic. Leo 
is presently one of the top 100 SourceForge sites.  Leo is 
production-quality software; bugs are typically fixed in a few days.

More importantly, Leo, right now, implements everything envisioned in the 
project description:

- Leo works well with IPython. See below for details.
- Leo has a powerful plugin mechanism.  An ipython plugin would be the 
natural way to do larger summer of code projects in Leo.
- Leo outlines are far more capable than typical notebooks.  Clones allow 
multiple views of data to exist within a single outline.
- Leo's rst2 plugin already supports rst markup, including embedded 
graphics.
- Leo plugins can define extensions to markup and Leo directives.  Such 
extensions might be useful in larger summer of code projects.
- Leo can create various external files (called derived files) from a single 
outline.  Derived files can be edited in any editor, and changes made to 
derived files in others editors automatically appear in an outline when Leo 
reopens it. The format of derived files involves 10 years of design and 
implementation.
- Leo shows that a simple markup language called noweb, 
http://www.eecs.harvard.edu/~nr/noweb/, when used in an outline context, is 
much more powerful than a more complex markup in a 'flat' context.  There is 
absolutely no need for yet another markup language.
- noweb markup is language neutral: it can be used in combination with any 
other language.
- If markup other than noweb were really needed, the ipython plugin could 
define a syntax colorer for that markup.
- Leo supports syntax coloring for LaTeX, cweb and many other languages.  In 
the near future, this colorer will become even more flexible.
- Several flavors of htmlize scripts already exist. Writing other custom 
converters would be dead easy. (For those who understand, just put the 
script of the converter in an @button node.)
- Leo is a superb scripting environment for Python.  See 
http://webpages.charter.net/edreamleo/WhatIsLeo.html In this environment, 
all manner of customizations become very easy.  For example, @button nodes 
would be the way to do smaller summer of code projects.
- As shown below, scripts run from Leo (or from IPython invoked from Leo) 
have full access to all data in Leo outlines. This crucial element is 
missing from the project description.
- Leo works well with editors such as vim and emacs.
- Leo can silently run a script in batch mode.  Just execute:
  leo.py --batch yourScript.py <arguments to Leo>

In short, Leo already goes far beyond the capabilities described in the 
summer of code proposal.  The following section describes how IPython 
presently works with Leo.


Using IPython and Leo

To embed IPython into Leo simply create a node in a Leo outline whose body 
text contains the standard embedding code:

from IPython.Shell import IPShellEmbed
ipshell = IPShellEmbed()
ipshell()


To run IPython from Leo, do the following:

1.  Make sure that Leo is running with a console window visible.
2.  Select the node containing the above script and do Leo's Execute Script 
command.  Better yet, you can create a 'script button' in Leo's icon area 
that will run this script when pressed.  (The scripting plugin must be 
enabled in order to create script buttons.)

After executing this script, (either using the Execute Script command or by 
pressing the 'ipython' script button) Leo continues to run while IPython is 
running.  Way cool.

Important:  When started from Leo using the script above, the c, g and p 
variables are defined in IPython.  c is Leo's present commander (window), g 
is the leoGlobals module and p is the present position in the outline. 
These three variables give you access to all the data in the present Leo 
outline!  This is essentially perfect integration with Leo and IPython, 
right out of the box.

For example, the following script, **when executed by IPython invoked from 
Leo**, prints all the headlines of the present outline, properly indented:

for p in c.allNodes_iter():
    print p.level()*' ',p.headString()


Conclusions

Leo already is a complete interactive notebook for IPython, with all the 
features described in the proposal.  Furthermore, Leo shows how the proposal 
could be improved :-)  Please tell me if I have misunderstood the proposal 
in any way.

Please look at Leo carefully: as far as the proposal is concerned, you can 
probably accomplish more in one week using Leo than you could in six months 
without Leo.  After using Leo for a bit you will certainly have ideas for 
improving how Leo and IPython work together.  Implementing those ideas would 
likely take just a few hours in the context of Leo.  Use plugins for larger 
projects; use @button nodes for smaller (instant) projects.

I am willing to do whatever it takes to make Leo work better with IPython. 
I would love for the IPython community to get what Leo is all about.

Edward

P.S. At the EuroPython conference there was unanimous agreement that Leo 
needs better documentation for newcomers, and that Leo must be able to 
simulate emacs and vim key bindings so Leo can be used more easily without a 
mouse.  These are now top priority items and I hope to finish both within a 
month.  Please don't let these defects get in the way of learning Leo.  In 
particular, please do ask questions about Leo either here or at Leo's help 
forum: http://sourceforge.net/forum/forum.php?forum_id=10227

EKR
--------------------------------------------------------------------
Edward K. Ream   email:  edreamleo at charter.net
Leo: Literate Editor with Outlines
Leo: http://webpages.charter.net/edreamleo/front.html
-------------------------------------------------------------------- 




From edreamleo at charter.net  Tue Jul  5 10:39:53 2005
From: edreamleo at charter.net (Edward K. Ream)
Date: Tue, 5 Jul 2005 09:39:53 -0500
Subject: [IPython-dev] Leo and xml
Message-ID: <004a01c5816f$68236430$6500a8c0@edwardx330hgzb>

I forgot to mention in the long post that .leo files are xml files.  The xml 
contains no domain-specific information--just enough to represent outlines, 
including clones.

10 years of experience with Leo shows that this general approach works very 
well.  Leo supports so-call user attributes (uA's) in the <t> elements.  In 
practice, this allows for unlimited user-domain information to be added. 
For sure, Leo outlines are all you are ever going to need to represent any 
kind of notebook, whatever the data it contains.

I am a huge fan of xml, but there is no need for xml outside of the .leo 
file that represents the overall outline.  In Leo's terms, derived files 
should be free of xml, unless the derived file is itself an xml or html 
file.  This clean split between outlines and the files derived from outlines 
has taken a huge amount of work to get right.  You aren't going to recreate 
it in a summer :-)

I guess my real point is that Leo solved all the conceptual problems in the 
proposal a long time ago.  Every thing just works.  Rather than parsing 
complex markup, conversion scripts handle nodes that may possibly contain 
Leo directives.  Since plugins can define their own directives, this does 
not contrain the user or the tools in any way.

I suspect these remarks may seem a bit oracular at present.  All this will 
become clearer once you start using Leo.  Put another way, a lot of design 
problems are suddenly going to get a lot easier when cast in terms of Leo 
outlines.

Edward
--------------------------------------------------------------------
Edward K. Ream   email:  edreamleo at charter.net
Leo: Literate Editor with Outlines
Leo: http://webpages.charter.net/edreamleo/front.html
--------------------------------------------------------------------




From hans_meine at gmx.net  Tue Jul  5 13:55:55 2005
From: hans_meine at gmx.net (Hans Meine)
Date: Tue, 5 Jul 2005 19:55:55 +0200
Subject: [IPython-dev] Leo, IPython and the summer of code
In-Reply-To: <003d01c5816b$765c2040$6500a8c0@edwardx330hgzb>
References: <003d01c5816b$765c2040$6500a8c0@edwardx330hgzb>
Message-ID: <200507051955.55220.hans_meine@gmx.net>

On Tuesday 05 July 2005 16:11, Edward K. Ream wrote:
> There are slight inaccuracies in section 2.2 (complexity level) and section
..this is a nice introduction for what follows..

> [...] In short, Leo already goes far beyond the capabilities described in
> the summer of code proposal. [...]

My main question is: Why the heck did nobody mention this before??  How come 
that Leo can be used with IPython but not even the IPython author knows this?

How come that we tried to implement a python GUI at our lab in the last 
months, now I got to know (and love) IPython, and after some weeks somebody 
tells me that there's even more?

I must be dreaming.  I *will* investigate what Leo offers.  It feels strange 
to me that there are such large gaps (/missing links) in the world wide 
"web", that things like IPython and Leo can remain undiscovered.
OTOH it's the same with research: Every other day you find that people have 
already though your thoughts, and most often years before you did.

Ciao, /  /
     /--/
    /  / ANS



From rkern at ucsd.edu  Tue Jul  5 15:36:04 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Tue, 05 Jul 2005 12:36:04 -0700
Subject: [IPython-dev] Re: Leo, IPython and the summer of code
In-Reply-To: <003d01c5816b$765c2040$6500a8c0@edwardx330hgzb>
References: <003d01c5816b$765c2040$6500a8c0@edwardx330hgzb>
Message-ID: <daenf8$lq6$1@sea.gmane.org>

Edward K. Ream wrote:

> More importantly, Leo, right now, implements everything envisioned in 
> the project description:
> 
> - Leo works well with IPython. See below for details.

It does not interact with IPython in the way we need. The proposal is a 
bit unclear as to the precise goal of this project. It gives undue 
emphasis on the file format which gives readers the impression that it 
is proposing a new literate programming tool. If that were the case, 
then Leo would satisfy the proposal just fine.

However, we are not looking for a literate programming tool. We are 
looking for a GUI Read-Eval-Print Loop that is embedded in a rich text 
document. The way that Leo interacts with IPython is to run the REPL in 
a console divorced from the document. This is not what we want. To see a 
static example of what we want, please look at this rendered example 
from my current codebase:

http://starship.python.net/~kernr/tut-2.3.5-html.html

That is essentially the format that we want to be editing within the 
GUI. There are two important points here:

   1. We want the shell as part of the GUI.
   2. We don't want the user to have to do any markup unless he is doing 
something special.

I do not believe that using Leo as a base for this project is justified 
at this time. Further development to interface with Leo is an option and 
something to explore later. At this time, I feel that focusing on our 
core requirements and developing outside of a framework (Leo, or any 
others like Envisage, Eclipse, Blackadder, and the like) will serve us 
best in the end.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From hans_meine at gmx.net  Tue Jul  5 16:36:14 2005
From: hans_meine at gmx.net (Hans Meine)
Date: Tue, 5 Jul 2005 22:36:14 +0200
Subject: [IPython-dev] Leo, IPython and the summer of code
In-Reply-To: <200507051955.55220.hans_meine@gmx.net>
References: <003d01c5816b$765c2040$6500a8c0@edwardx330hgzb>
	<200507051955.55220.hans_meine@gmx.net>
Message-ID: <200507052236.19061.hans_meine@gmx.net>

On Tuesday 05 July 2005 19:55, Hans Meine wrote:
> I must be dreaming.  I *will* investigate what Leo offers.  It feels
> strange to me that there are such large gaps (/missing links) in the world
> wide "web", that things like IPython and Leo can remain undiscovered.

OK, as Robert already pointed out, Leo is not the solution to all our 
problems.  I digged through the first tutorials (awfully slow server AFAICS, 
and little text with screenies for dummies), this is my summary in case 
someone wants to pick up where I leave it at for now (maybe you get more 
ideas for the "ideal" notebook interface):

Leo is an "outline editor" (unfortunately using TkInter, which means I had to 
install TK, TCL and python again).  To the upper left, you see the outline, 
an editable tree structure.  You can add, rename and delete nodes as you 
wish, or even "clone" them; the nodes have reference counts > 2 then and 
become regular nodes again when you delete all but one instance.

Each node can have text associated with it, which is displayed in the editor 
pane below.  This text can be syntax-highlighted code, too.

Press Ctrl+Shift+F ("Import to @file") to create a special node which imports 
a source file (e.g. Python, the tutorial uses Java); leo will create a large 
subtree with classes and methods from that source.  When saving, the tree 
struture is flattened and saved to the file again.  (In addition to the 
native .leo files.)  If you change the structure within Leo (e.g. create 
sub-nodes that do not correspond to classes or functions), the result will 
contain special comments that Leo can use to re-create the structure from the 
file.

Much relies on @directives like the @file above or a @language directive that 
switches on syntax highlighting for all sub-nodes and tells Leo how to handle 
the sourcecode stuff mentioned above.

It's a literate programming tool, and although it looks interesting, it does 
not seem to be what I hoped it was either.  I think Edward was a little 
unprecise, too. ;-)  Interesting (domain) name though.. "edream". :-)

-- 
Ciao, /  /                                                    .o.
     /--/                                                     ..o
    /  / ANS                                                  ooo
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20050705/a85fe5cc/attachment.sig>

From Fernando.Perez at colorado.edu  Tue Jul  5 17:26:04 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Tue, 05 Jul 2005 15:26:04 -0600
Subject: [IPython-dev] Leo, IPython and the summer of code
In-Reply-To: <200507052236.19061.hans_meine@gmx.net>
References: <003d01c5816b$765c2040$6500a8c0@edwardx330hgzb>
	<200507051955.55220.hans_meine@gmx.net>
	<200507052236.19061.hans_meine@gmx.net>
Message-ID: <42CAFAEC.10308@colorado.edu>

[ aside: Hans, are you subscribed to ipython-dev?  I'm still getting 
auto-discard notifications from your emails.  I want to make sure your setup 
is Ok, just contact me off-list if you keep having problems ]

Hans Meine wrote:
> On Tuesday 05 July 2005 19:55, Hans Meine wrote:
> 
>>I must be dreaming.  I *will* investigate what Leo offers.  It feels
>>strange to me that there are such large gaps (/missing links) in the world
>>wide "web", that things like IPython and Leo can remain undiscovered.

Well, a response to this is simply that there is only a finite amount of time 
to be spent learning about tools out there :)  I was aware of leo's existence, 
but I thought it was mostly an 'outliner on steroids'.  From Ed's email, it 
seems it also cures AIDS and solves world hunger, so I must have missed some 
of its capabilities :)

Joking aside, Robert points out to the key point, which you confirm: as 
scientists (these tools are undeniably biased by our own needs and background 
with tools like Mathematica, though we hope they'll be useful to 
non-scientists), the interactive read/eval loop is critical.

> OK, as Robert already pointed out, Leo is not the solution to all our 
> problems.  I digged through the first tutorials (awfully slow server AFAICS, 
> and little text with screenies for dummies), this is my summary in case 
> someone wants to pick up where I leave it at for now (maybe you get more 
> ideas for the "ideal" notebook interface):
> 
> Leo is an "outline editor" (unfortunately using TkInter, which means I had to 
> install TK, TCL and python again).  To the upper left, you see the outline, 
> an editable tree structure.  You can add, rename and delete nodes as you 
> wish, or even "clone" them; the nodes have reference counts > 2 then and 
> become regular nodes again when you delete all but one instance.
> 
> Each node can have text associated with it, which is displayed in the editor 
> pane below.  This text can be syntax-highlighted code, too.
> 
> Press Ctrl+Shift+F ("Import to @file") to create a special node which imports 
> a source file (e.g. Python, the tutorial uses Java); leo will create a large 
> subtree with classes and methods from that source.  When saving, the tree 
> struture is flattened and saved to the file again.  (In addition to the 
> native .leo files.)  If you change the structure within Leo (e.g. create 
> sub-nodes that do not correspond to classes or functions), the result will 
> contain special comments that Leo can use to re-create the structure from the 
> file.
> 
> Much relies on @directives like the @file above or a @language directive that 
> switches on syntax highlighting for all sub-nodes and tells Leo how to handle 
> the sourcecode stuff mentioned above.
> 
> It's a literate programming tool, and although it looks interesting, it does 
> not seem to be what I hoped it was either.  I think Edward was a little 
> unprecise, too. ;-)  Interesting (domain) name though.. "edream". :-)

Many thanks for this summary, Hans.  I will download Leo and play with it 
later, as I'm sure it may have a number of interesting things to offer, and we 
may well steal/collaborate/integrate with it in the future.

Cheers,

f



From Fernando.Perez at colorado.edu  Tue Jul  5 17:44:50 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Tue, 05 Jul 2005 15:44:50 -0600
Subject: [IPython-dev] Re: Leo, IPython and the summer of code
In-Reply-To: <daenf8$lq6$1@sea.gmane.org>
References: <003d01c5816b$765c2040$6500a8c0@edwardx330hgzb>
	<daenf8$lq6$1@sea.gmane.org>
Message-ID: <42CAFF52.2030905@colorado.edu>

Robert Kern wrote:
> Edward K. Ream wrote:
> 
> 
>>More importantly, Leo, right now, implements everything envisioned in 
>>the project description:
>>
>>- Leo works well with IPython. See below for details.
> 
> 
> It does not interact with IPython in the way we need. The proposal is a 
> bit unclear as to the precise goal of this project. It gives undue 
> emphasis on the file format which gives readers the impression that it 
> is proposing a new literate programming tool. If that were the case, 
> then Leo would satisfy the proposal just fine.

I take blame for that lack of clarity, and sorry if it confused anyone.  It 
was mostly a poorly-worded attempt at providing a code sketch that would give 
potential applicants something to read and gauge their own interest in 
participating.

In my defense, part of the confusion comes from the fact that even the basic 
design had to be hashed out, as it's not completely trivial (at least, it has 
taken a lot of email and two long telephone conversations with John Hunter 
--matplotlib-- and Robert Kern, people whom I respect tremendously, so it's 
not just a 5 minute thing.  Either that or John, Robert and I are just dumb 
beyond salvation :)

I jumped on the opportunity to grab the google summer of code students to 
participate in this, though it's clear that part of the project will involve 
discussing/understanding the best approach.  I think though that by now we've 
converged fairly well on an that part, and we can start moving on in 
implementation work.

Cheers,

f



From Fernando.Perez at colorado.edu  Tue Jul  5 17:55:01 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Tue, 05 Jul 2005 15:55:01 -0600
Subject: [IPython-dev] Notebook roadmap
In-Reply-To: <daac75$hlt$1@sea.gmane.org>
References: <daac75$hlt$1@sea.gmane.org>
Message-ID: <42CB01B5.703@colorado.edu>

Robert Kern wrote:
> Fernando and I hashed out a few things on the phone today. Short 
> version: XML wins for now, even if I had to threaten to kidnap 
> Fernando's cat to do it. Long version:
> 
>    http://www.scipy.org/wikis/featurerequests/NoteBook

Many thanks for this, Robert.  For those who didn't click on the link, Robert 
has joined in as co-mentor at my pestering.  I'm super grateful to him for 
that, and it will be very useful to have two of us available on this.

And besides the threats on Ilona (our lovely Russian blue), Robert's arguments 
for XML were quite good, esp. in light of how ElementTree serves to placate 
the gag reflex that all programmers with good taste have when confronted with XML.

> There's a page there for adding user stories. I've already added the 
> ones Hans has posted and some of my own. If you would like to add your 
> own stories (aka "feature requests"), please make yourself an account 
> and edit the Wiki to your liking.
> 
> Documentation on the XML format and API will be forthcoming. Hopefully, 
> we will have SVN space by Tuesday or Wednesday.

Yes, the scipy admin is in the process of adding the necessary infrastructure. 
  I can already log in, but can't create users yet.  This should be 
operational within a day or two, and I'll make an SVN repo for this, where 
Toni and Tzanko can keep their respective codes.  IPython itself will also 
live in SVN soon, so when Tzanko needs to directly patch ipython, he will be 
able to work there as well.

Robert, is your code available anywhere?  This could be used as a starter by 
Toni on the document work, as well as providing Tzanko a mockup target to load.

Cheers,

f



From edreamleo at charter.net  Tue Jul  5 18:02:05 2005
From: edreamleo at charter.net (Edward K. Ream)
Date: Tue, 5 Jul 2005 17:02:05 -0500
Subject: [IPython-dev] Re: Leo, IPython and the summer of code
References: <003d01c5816b$765c2040$6500a8c0@edwardx330hgzb>
	<daenf8$lq6$1@sea.gmane.org>
Message-ID: <003f01c581ad$2ea55680$6500a8c0@edwardx330hgzb>

> It does not interact with IPython in the way we need.

I implied in my original post that more work would be needed.  Suppose you 
do have some code that you need.  Embedding that code in Leo should be 
straightforward.

> However, we are not looking for a literate programming tool.

Leo is not _just_ a literate programming tool.  The literate programming 
features are optional, though I suspect that noweb will, in fact, be quite 
useful for this project.  In this discussion what is important is that Leo 
is a very fine notebook, with many general useful features that have nothing 
to do with LP.

> We are looking for a GUI Read-Eval-Print Loop that is embedded in a rich 
> text document.

What prevents you from putting that loop in Leo?  Leo's body pane, that is, 
the body text of each node, is in fact a rich text document.

> 1. We want the shell as part of the GUI.
> 2. We don't want the user to have to do any markup unless he is doing 
> something special.

I see no reason why these goals are incompatible with Leo.

> I do not believe that using Leo as a base for this project is justified at 
> this time.

Isn't it a bit hasty to dismiss Leo out of hand?  Without a code base 
similar to Leo you are likely to spend years of work on the notebook.

Edward
--------------------------------------------------------------------
Edward K. Ream   email:  edreamleo at charter.net
Leo: http://webpages.charter.net/edreamleo/front.html
--------------------------------------------------------------------




From edreamleo at charter.net  Tue Jul  5 18:09:06 2005
From: edreamleo at charter.net (Edward K. Ream)
Date: Tue, 5 Jul 2005 17:09:06 -0500
Subject: [IPython-dev] Leo, IPython and the summer of code
References: <003d01c5816b$765c2040$6500a8c0@edwardx330hgzb>
	<200507051955.55220.hans_meine@gmx.net>
Message-ID: <006701c581ae$2941adf0$6500a8c0@edwardx330hgzb>

> My main question is: Why the heck did nobody mention this before??  How 
> come
> that Leo can be used with IPython but not even the IPython author knows 
> this?

Well, not even the Leo author knew about this until yesterday :-)

Edward
--------------------------------------------------------------------
Edward K. Ream   email:  edreamleo at charter.net
Leo: http://webpages.charter.net/edreamleo/front.html
--------------------------------------------------------------------




From rkern at ucsd.edu  Tue Jul  5 18:09:39 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Tue, 05 Jul 2005 15:09:39 -0700
Subject: [IPython-dev] Re: Notebook roadmap
In-Reply-To: <42CB01B5.703@colorado.edu>
References: <daac75$hlt$1@sea.gmane.org> <42CB01B5.703@colorado.edu>
Message-ID: <daf0f7$f7a$1@sea.gmane.org>

Fernando Perez wrote:

> Robert, is your code available anywhere?  This could be used as a 
> starter by Toni on the document work, as well as providing Tzanko a 
> mockup target to load.

The current snaphsot is available here:

http://starship.python.net/~kernr/notebook.tar.gz

It requires Python 2.4 (can be backported by moving @classmethod down to 
"foo = classmethod(foo)" and changing some generator expressions to list 
comprehensions) and ElementTree. It will use cElementTree if available.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From edreamleo at charter.net  Tue Jul  5 18:14:26 2005
From: edreamleo at charter.net (Edward K. Ream)
Date: Tue, 5 Jul 2005 17:14:26 -0500
Subject: [IPython-dev] Leo, IPython and the summer of code
References: <003d01c5816b$765c2040$6500a8c0@edwardx330hgzb><200507051955.55220.hans_meine@gmx.net><200507052236.19061.hans_meine@gmx.net>
	<42CAFAEC.10308@colorado.edu>
Message-ID: <008501c581ae$e83335d0$6500a8c0@edwardx330hgzb>

> I will download Leo and play with it  later, as I'm sure it may have a 
> number of interesting things to offer, and we may well 
> steal/collaborate/integrate with it in the future.

Excellent.  It appears that I may have offended some by claiming that Leo 
solves (or can solve) all the proposals requirements.  Perhaps I let my 
excitement that IPython already works well with Leo get in the way of due 
decorum.  I apologize for any hurt feelings.  Nevertheless, I think that Leo 
and IPython have something to offer each other, and I repeat my offer to do 
whatever it takes to help Leo support IPython.

Edward
--------------------------------------------------------------------
Edward K. Ream   email:  edreamleo at charter.net
Leo: http://webpages.charter.net/edreamleo/front.html
--------------------------------------------------------------------




From rkern at ucsd.edu  Tue Jul  5 18:32:01 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Tue, 05 Jul 2005 15:32:01 -0700
Subject: [IPython-dev] Re: Leo, IPython and the summer of code
In-Reply-To: <003f01c581ad$2ea55680$6500a8c0@edwardx330hgzb>
References: <003d01c5816b$765c2040$6500a8c0@edwardx330hgzb>
	<daenf8$lq6$1@sea.gmane.org>
	<003f01c581ad$2ea55680$6500a8c0@edwardx330hgzb>
Message-ID: <daf1p5$ihr$1@sea.gmane.org>

Edward K. Ream wrote:
>> It does not interact with IPython in the way we need.
> 
> I implied in my original post that more work would be needed.  Suppose 
> you do have some code that you need.  Embedding that code in Leo should 
> be straightforward.

You also implied that that work would be a matter of hours or a day. I'm 
sorry, but that is simply not the case. The bulk of the work, making a 
REPL that interleaves editable rich text and input/output cells, would 
still have to be done. Using Leo for the rest of the infrastructure is 
as likely to hinder as to help.

>> However, we are not looking for a literate programming tool.
> 
> Leo is not _just_ a literate programming tool.  The literate programming 
> features are optional, though I suspect that noweb will, in fact, be 
> quite useful for this project.  In this discussion what is important is 
> that Leo is a very fine notebook, with many general useful features that 
> have nothing to do with LP.

I think we're talking past each other. Leo does not provide a "notebook" 
in the sense that we are using the term. There are other senses of the 
term in which Leo does in fanct provide a very fine notebook.

>> We are looking for a GUI Read-Eval-Print Loop that is embedded in a 
>> rich text document.
> 
> What prevents you from putting that loop in Leo?  Leo's body pane, that 
> is, the body text of each node, is in fact a rich text document.

Nothing prevents us from putting that loop in Leo. What I am suggesting 
is that Leo does not currently provide such a thing. It will not be a 
matter of a day to put it in.

>> 1. We want the shell as part of the GUI.
>> 2. We don't want the user to have to do any markup unless he is doing 
>> something special.
> 
> I see no reason why these goals are incompatible with Leo.

They aren't, but I don't think that what Leo provides is going to be as 
much help in achieving these goals as you are claiming.

>> I do not believe that using Leo as a base for this project is 
>> justified at this time.
> 
> Isn't it a bit hasty to dismiss Leo out of hand?  Without a code base 
> similar to Leo you are likely to spend years of work on the notebook.

I disagree. The project actually has a fairly limited scope.

Another point arguing against using Leo's infrastructure is that we are 
then tied to having this notebook embedded in Leo. That might be fine if 
all we wanted was a standalone application, but I hope that the code we 
develop here can be useful for embedding into other projects as well as Leo.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From rkern at ucsd.edu  Tue Jul  5 20:16:38 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Tue, 05 Jul 2005 17:16:38 -0700
Subject: [IPython-dev] Re: Notebook roadmap
In-Reply-To: <42CB01B5.703@colorado.edu>
References: <daac75$hlt$1@sea.gmane.org> <42CB01B5.703@colorado.edu>
Message-ID: <daf7ta$via$1@sea.gmane.org>

Fernando Perez wrote:

> Robert, is your code available anywhere?  This could be used as a 
> starter by Toni on the document work, as well as providing Tzanko a 
> mockup target to load.

And some documentation to boot:

http://www.scipy.org/wikis/featurerequests/NoteBookXML

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From mtobis at gmail.com  Wed Jul  6 12:24:09 2005
From: mtobis at gmail.com (Michael Tobis)
Date: Wed, 6 Jul 2005 11:24:09 -0500
Subject: [IPython-dev] notebook composition UI thoughts
Message-ID: <c3aa5555050706092460896631@mail.gmail.com>

John Hunter has mentioned the IPython notebook effort to me a couple
of times. I've been thinking about the Mathematica user interface for
years, and I think it is flawed. I hope that eventually IPython can do
better, and I think the steps being plotted out for the short term
move in the right direction.

A Mathematica notebook serves two primary purposes, which are at odds
with each other. The first is as an interactive environment for
exploration of mathematical and algorithmic ideas, and the second is
as an environment for the development and presentation of formal
exposition of the ideas so developed.

Consider this Mathematica session, obtained by simply moving the
insertion point between the second and third inputs:

==> 
In[1]:= x = 5

Out[1]= 5

--

In[3] = x * x

Out[3]= 36

--

In[2]:= x = 6

Out[2]= 6 
<==

I save it and get the notebook

==> 
x = 5

5

--

x * x

36

--

x = 6

6 
<==

Of course, when I recalculate the cells, the order of execution
changes and the second result is the expected 25. There are far more
confusing results that are similar in concept but much more confusing
in practice. There is also the fact that the notebook inherits 
context from the entire Mathematica session, including all open 
notebooks, that leads to similar flavors of confusion.

One way to look at the problem with the Mathematica model of notebooks
is that it conflates the purposes of authoring and presentation. As a
presentation mechanism, the organizing principle is sequential in the
file space, but as an interactive execution model, the organizing
principle is sequential in time.

The quandary is that explorations are developed sequentially in time
and read and executed sequentially in text.

In practice what I will want if I am developing an interactive
educational, um, thing (I'm not sure "notebook" is the right name) is
not one but two Python sessions: first an interactive environment not
dramatically different from our current iPython, for developing the
concepts, and second a live editable interface into which I can paste
those bits of text, code and other widgets as I determine that they
enhance the presentation.

The order in which I edit a document is not always sequential.
I need to be able to insert executable statements into the middle of
the document. For repeatability, I also want each document to be able
to provide all of its own context.

For this to make sense I need a new python session (or at least a
fresh namespace)  invoked every time I make a change in the
middle of a document. I need anything after the insertion point that
changes to be highlighted and flagged, and I need multiple undos.

In the case of a very long document this will cause quite a lot of
thrashing around as everything gets recalculated. Perhaps the user
interface needs to permit "molecular" rather than atomic changes
before enforcing a recalculation. That is, I would insert multiple
input lines from my live session into various places in the notebook
document before forcing a recalculation of the document.

It seems clear to me that some sort of computer mediated presentation
mechanism will eventually take over from textbooks in mathematics,
engineering and physical science. I think Python deserves to be a big
player in this. 

Mathematica notebooks show us that a great deal of care needs to be
put into the authoring interface.

The postprocessing model described as the summer project seems to me
to support this idea rather than thwart it, so there is fortunately no
great urgency in considering these ideas.

Basically, my proposed notebook development UI is not one Python
session but two, one a conventional iPython session, and the other a
rather more ethereal beast that ensures that everything is executed in
the order I as author intend and that all context is provided within
the notebook and not accidentally in the session.

The overlap between these problems and those addressed by Knuth in his
original Literate Programming work is, to me, striking. I'm agnostic
on the question as to whether the extant Leo code base is the right
approach, but the ideas of literate programming are very germane to
the ideas of developing scientific "notebooks". 

Knuth observes that the order in which the reader wants to read the
code does not necessarily match the order in which the compiler wants
to process it. The flaws in the Mathematica noteboook model show us
(among other things) that the order in which an author writes code and
documentation is not necessarily the same as either of them.

--
mt
--
Michael Tobis
Geophysical Sciences
University of Chicago



From rkern at ucsd.edu  Wed Jul  6 14:22:08 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Wed, 06 Jul 2005 11:22:08 -0700
Subject: [IPython-dev] Re: notebook composition UI thoughts
In-Reply-To: <c3aa5555050706092460896631@mail.gmail.com>
References: <c3aa5555050706092460896631@mail.gmail.com>
Message-ID: <dah7gl$915$1@sea.gmane.org>

Michael Tobis wrote:
> John Hunter has mentioned the IPython notebook effort to me a couple
> of times. I've been thinking about the Mathematica user interface for
> years, and I think it is flawed. I hope that eventually IPython can do
> better, and I think the steps being plotted out for the short term
> move in the right direction.
> 
> A Mathematica notebook serves two primary purposes, which are at odds
> with each other. The first is as an interactive environment for
> exploration of mathematical and algorithmic ideas, and the second is
> as an environment for the development and presentation of formal
> exposition of the ideas so developed.
> 
> Consider this Mathematica session, obtained by simply moving the
> insertion point between the second and third inputs:
> 
> ==> 
> In[1]:= x = 5
> 
> Out[1]= 5
> 
> --
> 
> In[3] = x * x
> 
> Out[3]= 36
> 
> --
> 
> In[2]:= x = 6
> 
> Out[2]= 6 
> <==
> 
> I save it and get the notebook
> 
> ==> 
> x = 5
> 
> 5
> 
> --
> 
> x * x
> 
> 36
> 
> --
> 
> x = 6
> 
> 6 
> <==
> 
> Of course, when I recalculate the cells, the order of execution
> changes and the second result is the expected 25. There are far more
> confusing results that are similar in concept but much more confusing
> in practice. There is also the fact that the notebook inherits 
> context from the entire Mathematica session, including all open 
> notebooks, that leads to similar flavors of confusion.

The current document model has two separate sections: a log and a sheet. 
The log contains all of the text from the In[NN] and Out[NN] prompts and 
other information. Each element is tagged with the number NN. The sheet 
contains the rich text that you write and references to those elements 
in the log. The sheet can contain any, all, or none of the elements in 
the log in any order and with any number of repetitions. When executed, 
the notebook will only consider the elements in the log in order. The 
formatting of the sheet is irrelevant to execution.

At the moment, we are focusing on creating an interactive environment, 
not a code development environment. It is not expected that you will 
write modules in this environment. For that we leave you to your own 
editor and/or literate programming tools.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From hans_meine at gmx.net  Wed Jul  6 14:40:44 2005
From: hans_meine at gmx.net (Hans Meine)
Date: Wed, 6 Jul 2005 20:40:44 +0200
Subject: [IPython-dev] notebook composition UI thoughts
In-Reply-To: <c3aa5555050706092460896631@mail.gmail.com>
References: <c3aa5555050706092460896631@mail.gmail.com>
Message-ID: <200507062040.44406.hans_meine@gmx.net>

On Wednesday 06 July 2005 18:24, Michael Tobis wrote:
> [...] Of course, when I recalculate the cells, the order of execution
> changes and the second result is the expected 25. There are far more
> confusing results that are similar in concept but much more confusing
> in practice. There is also the fact that the notebook inherits
> context from the entire Mathematica session, including all open
> notebooks, that leads to similar flavors of confusion.

Right. That's an important problem to keep in mind.

> One way to look at the problem with the Mathematica model of notebooks
> is that it conflates the purposes of authoring and presentation. As a
> presentation mechanism, the organizing principle is sequential in the
> file space, but as an interactive execution model, the organizing
> principle is sequential in time.

Good point.

> In practice what I will want if I am developing an interactive
> educational, um, thing (I'm not sure "notebook" is the right name) is
> not one but two Python sessions: [...]

I think in this part you just make an assumption on the solution, which might 
not be the right one.  I argue that when developing an interactive, 
educational thing, you do not necessarily want two sessions, but you want a 
repeatable, understandable log.

> [...] For this to make sense I need a new python session (or at least a
> fresh namespace)  invoked every time I make a change in the
> middle of a document. I need anything after the insertion point that
> changes to be highlighted and flagged, and I need multiple undos.

This goes into the direction I pointed out in one of my user stories: What 
you'd really like to have is not to flag everything behind the point, but 
just anything that depends on what you've changed.  *Ideally*, you'd want 
only those parts to be recalculated that actually depend on the changes.

> In the case of a very long document this will cause quite a lot of
> thrashing around as everything gets recalculated.

Ideally, one could just let a dirty flag be passed down a dependency graph, 
and highlight / gray out dirty cells, which would mean only minimal effort

> Perhaps the user 
> interface needs to permit "molecular" rather than atomic changes
> before enforcing a recalculation. That is, I would insert multiple
> input lines from my live session into various places in the notebook
> document before forcing a recalculation of the document.

Right.  The dirty cells should be recalculated only on demand/manual 
activation.

> Knuth observes that the order in which the reader wants to read the
> code does not necessarily match the order in which the compiler wants
> to process it. The flaws in the Mathematica noteboook model show us
> (among other things) that the order in which an author writes code and
> documentation is not necessarily the same as either of them.

OK, then the minimal, nearly trivial first fix for the problem you initially 
presented would be to store the execution order of cells?!
(This corresponds to "the Leo way" BTW, where one could have a sequence of 
cells that make up a program, and then rearrange clones of all those in a 
presentation branch of the outline (in a different order).)
Note that this does not yet solve the problem that code that initialized 
variables could have been overwritten in the meantime, although the variables 
stay in the environment.

Those are interesting problems, OTOH it's hard to save the user from shooting 
him/herself in the foot - he/she will always need to have some kind of idea 
of what happens behind the UI I guess.

Nice greetings,
  Hans



From hans_meine at gmx.net  Wed Jul  6 14:54:07 2005
From: hans_meine at gmx.net (Hans Meine)
Date: Wed, 6 Jul 2005 20:54:07 +0200
Subject: [IPython-dev] notebook composition UI thoughts
In-Reply-To: <200507062040.44406.hans_meine@gmx.net>
References: <c3aa5555050706092460896631@mail.gmail.com>
	<200507062040.44406.hans_meine@gmx.net>
Message-ID: <200507062054.07757.hans_meine@gmx.net>

(FYI: At first, I sent my answer to Michael directly by accident, I am quoting 
his answer.  I wish I were able to send mails to this list on the first 
try... ;-) )

On Wednesday 06 July 2005 19:17, Michael Tobis wrote:
> > Ideally, one could just let a dirty flag be passed down a dependency
> > graph, and highlight / gray out dirty cells, which would mean only
> > minimal effort
>
> Agreed; this would be nicest. I thought of this (the gray-out
> affordance is a nice touch that I missed) and considered it a bit too
> much to include in my first message.
>
> However, the document should vigorously complain about being saved in
> such a state.
>
> It's rather a complicated objective, especially if multiple undo
> operations are also supported.

My collegue got another quite cool idea; he just asked "ah, you want Python to 
do that automatically?".  Of course that'd be cool, but there's another 
possibility:

By default, each cell should be it's own independent module (I think modules 
are quite cheap, right?).  In order to use stuff from another cell, "from 
cell_foo import *" could be used.  This way, dependencies would be more 
clear, and one could even use a special syntax / convention to make the 
extraction of a dependency graph an automatic procedure.

There could also be means to make several cells share a module.  I think I 
like that idea very much, because it re-uses existing technology (python 
modules).  One could also use "import foobar" and use fully-qualified 
references to results from the foobar cell.

Additionally, he still wants chapters to be totally independent of each other 
(maybe separate interpreter instances altogether).

> > OK, then the minimal, nearly trivial first fix for the problem you
> > initially presented would be to store the execution order of cells?!
>
> This replaces an inconsistent presentation with one that is
> consistently incorrect, which I suppose is an imporvement of a sort.
> (giving the "wrong" answer of 36 instead of 25 in the example)

It's not even really wrong; the reason for the answer is (more or less) 
clearly visible in the numbers.

> > Those are interesting problems, OTOH it's hard to save the user from
> > shooting him/herself in the foot - he/she will always need to have some
> > kind of idea of what happens behind the UI I guess.
>
> We can assume that anyone with an interest in authoring or modifying a
> document of this sort is not a fool, but on the other hand the user
> interface should not actively endeavor to make a fool of the author
> either.

That's what I wanted to say.

Greetings,
  Hans



From rkern at ucsd.edu  Wed Jul  6 15:49:21 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Wed, 06 Jul 2005 12:49:21 -0700
Subject: [IPython-dev] Re: notebook composition UI thoughts
In-Reply-To: <200507062040.44406.hans_meine@gmx.net>
References: <c3aa5555050706092460896631@mail.gmail.com>
	<200507062040.44406.hans_meine@gmx.net>
Message-ID: <dahck6$o44$1@sea.gmane.org>

Hans Meine wrote:
> On Wednesday 06 July 2005 18:24, Michael Tobis wrote:

>>[...] For this to make sense I need a new python session (or at least a
>>fresh namespace)  invoked every time I make a change in the
>>middle of a document. I need anything after the insertion point that
>>changes to be highlighted and flagged, and I need multiple undos.
> 
> This goes into the direction I pointed out in one of my user stories: What 
> you'd really like to have is not to flag everything behind the point, but 
> just anything that depends on what you've changed.  *Ideally*, you'd want 
> only those parts to be recalculated that actually depend on the changes.

I think that drastically changing the execution semantics from what 
ipython currently does is out of scope for this summer's project. We are 
going to, wherever we can, do the simplest thing that could possibly 
work. In this case, I think it means the following stories (which happen 
consecutively):

   1. I just made a typo at "In [34]:". I place my cursor back at the 
typo and correct it. I press Enter to reexecute. The cell is visually 
replaced by an "In [35]:" cell with the new code. The "Out[34]:" cell 
has been replaced by "Out[35]:" with the results of executing the new code.

   2. I now want to change a value in "In [33]:"'s code and reexecute 
the cells that follow it, but only if they are still visible in the 
sheet. Thus, "In [33]:" --> "In [36]:" and "In [35]:" --> "In [37]:", etc.

   3. I add an "In [38]:" that references "Out[37]:". I change a value 
in "In [36]:" and reexecute cells 36--38. "Out[41]:" (was "Out[38]:") 
doesn't change; I realize I am being silly and actually assign the value 
I need to a variable and use that instead.

   4. Since I find that I am doing this dance quite often, I assign the 
appropriate cells to a macro that references the variable I keep 
changing. Now I just reassign that variable and use that macro over and 
over again.

   5. I start a new sheet and specify that I want a fresh interpreter 
instance. My first input cell is "In [1]:".

This is not to say that other execution semantics would not be better 
suited, but I think that they are much more difficult to implement and 
really aren't what Toni and Tzanko signed on to do. If you wish to 
contribute code that does implement these fancier semantics, we will 
consider using it in addition to the Simplest Way Kern Can Think Of 
semantics.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From Fernando.Perez at colorado.edu  Wed Jul  6 16:34:50 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Wed, 06 Jul 2005 14:34:50 -0600
Subject: [IPython-dev] Re: notebook composition UI thoughts
In-Reply-To: <dahck6$o44$1@sea.gmane.org>
References: <c3aa5555050706092460896631@mail.gmail.com>
	<200507062040.44406.hans_meine@gmx.net> <dahck6$o44$1@sea.gmane.org>
Message-ID: <42CC406A.1070205@colorado.edu>

Robert Kern wrote:
> Hans Meine wrote:
> 
>>On Wednesday 06 July 2005 18:24, Michael Tobis wrote:
> 
> 
>>>[...] For this to make sense I need a new python session (or at least a
>>>fresh namespace)  invoked every time I make a change in the
>>>middle of a document. I need anything after the insertion point that
>>>changes to be highlighted and flagged, and I need multiple undos.
>>
>>This goes into the direction I pointed out in one of my user stories: What 
>>you'd really like to have is not to flag everything behind the point, but 
>>just anything that depends on what you've changed.  *Ideally*, you'd want 
>>only those parts to be recalculated that actually depend on the changes.
> 
> 
> I think that drastically changing the execution semantics from what 
> ipython currently does is out of scope for this summer's project. We are 
> going to, wherever we can, do the simplest thing that could possibly 
> work.

This is mostly a confirmation of Robert's position.  Module-per-cell, 
multiple-interpreters, auto-dependency analysis, etc. are all potentially 
worthwhile ideas for certain contexts, but certainly not for a first shot.  If 
we can get an interactive environment which enhances the ipython command-line 
experience with full-screen editing (to address the problem of endless 
appending to a session), and basic functionality for supporting rich-text 
documents (even if not rendered directly in real-time in the working window), 
we'll have a useful system for many of us.  This will also be an attainable 
goal for Toni and Tzanko in the time allotted.

I should note that things like automatic dependency analysis are notoriously 
difficult (I'd actually guess nearly impossible) in python.  The language is 
far too dynamic for that, and this is _exactly_ the same problem which has 
made all restricted execution systems for python fail (to the point where 
they've been removed from the standard library).   The discussions on 
python-dev on this issue, both in the context of security and of dynamic type 
inference for optimization purposes, contain all the gory details.

But the basic point is that python has 'eval' and 'exec', both of which are 
enough to thwart just about any reasonable attempt at dependency analysis.  It 
is true that in mathematica, the top-down flow of the text can be misleading, 
as the time-flow of execution (only recorded by the prompt numbers) is what 
really defines the in-memory status of the kernel.  There are systems with 
automatic dependency tracking and recalculation: spreadsheets like excel do 
this, as well as systems like Mathcad.  But those systems have much more 
limited assignment semantics than python, so they can keep such a dependency 
tree consistent at all times with reasonable effort.

I'd rather not provide such a system if it is going to have random 
reliability.  That's been exactly the position of python-dev on restricted 
execution: since we can't guarantee it will work, and it can break almost 
anywhere, we simply won't ship it (until someone can come along with a 
solution shown to be robust).  The extremely dynamic nature of python is what 
makes it so flexible for many things, but it also carries a price.

On the multiple interpreters front, that's more likely in the near term. 
There's a good possibility that the gui will be built with the ipython 
'kernel' living in a separate process, to allow it to be restarted without 
destroying the gui.  If that is done, the basic infrastructure would be in 
place for supporting multiple interpreters connected to different tabs in a 
session or something similar.  Not necessarily something to be implemented 
initially, but it the scaffolding would be in place to make it possible.

And thanks to all for the feedback!  It shows at least that there is a lot of 
interest in such a system, though everyone's idea of their ideal toy is 
obviously a bit different :)

Cheers,

f



From hans_meine at gmx.net  Wed Jul  6 16:51:37 2005
From: hans_meine at gmx.net (Hans Meine)
Date: Wed, 6 Jul 2005 22:51:37 +0200
Subject: [IPython-dev] Re: notebook composition UI thoughts
In-Reply-To: <42CC406A.1070205@colorado.edu>
References: <c3aa5555050706092460896631@mail.gmail.com>
	<dahck6$o44$1@sea.gmane.org> <42CC406A.1070205@colorado.edu>
Message-ID: <200507062251.38149.hans_meine@gmx.net>

On Wednesday 06 July 2005 22:34, Fernando Perez wrote:
> This is mostly a confirmation of Robert's position.  Module-per-cell,
> multiple-interpreters, auto-dependency analysis, etc. are all potentially
> worthwhile ideas for certain contexts, but certainly not for a first shot.

Point taken and agreed.  Still, I'd like to have a "future works" section in 
this discussion and the Wiki. ;-p

> If we can get an interactive environment which enhances the ipython
> command-line experience with full-screen editing (to address the problem of
> endless appending to a session), and basic functionality for supporting
> rich-text documents (even if not rendered directly in real-time in the
> working window), we'll have a useful system for many of us.

Believe me, I am looking forward to it.

> I should note that things like automatic dependency analysis are
> notoriously difficult (I'd actually guess nearly impossible) in python. 
> The language is far too dynamic for that, and this is _exactly_ the same
> problem which has made all restricted execution systems for python fail (to
> the point where they've been removed from the standard library). [...]

You're right, it's difficult, but I am not taking the rest of your arguments; 
I think that our notebook interface is very much different from a restricted 
execution system, in the sense that it's feasible to just tell the user 
"don't use import-in-an-eval, but just clearly write it out as an 
unconditional statement (maybe even at the beginning of the cell), if you 
want it to work".  Some users *want* it to work, and it's no big restriction 
to use import w/o exec or eval.

> It is true that in mathematica, the top-down flow of the text can be
> misleading, as the time-flow of execution (only recorded by the prompt
> numbers) is what really defines the in-memory status of the kernel.

Furthermore, it's about saving and re-executing a freshly opened notebook if I 
understood that correctly.  That is doable with a stored execution order, and 
important IMO.

> There 
> are systems with automatic dependency tracking and recalculation:
> spreadsheets like excel do this, as well as systems like Mathcad.  But
> those systems have much more limited assignment semantics than python, so
> they can keep such a dependency tree consistent at all times with
> reasonable effort.

Oh, these systems often also contain eval-like things, sometimes even allow 
(indirect) cyclic references, and often their dep. analysis fails.  However, 
I still find the basic analysis very useful.

> I'd rather not provide such a system if it is going to have random
> reliability.

It's not really "random", right? ;-)  Then I wouldn't like to have it either.

> That's been exactly the position of python-dev on restricted 
> execution: since we can't guarantee it will work, and it can break almost
> anywhere, we simply won't ship it (until someone can come along with a
> solution shown to be robust).

This is very much understandable because it's about security, but IMO that's a 
different world/topic.  I probably would not develop critical 
spaceship-controlling software in any kind of IPython notebook sessions. ;-)

> On the multiple interpreters front, that's more likely in the near term.
> There's a good possibility that the gui will be built with the ipython
> 'kernel' living in a separate process, to allow it to be restarted without
> destroying the gui.

We did the same in our own "PyTerm", but mainly to make Ctrl-C work.

-- 
Ciao, /  /                                                    .o.
     /--/                                                     ..o
    /  / ANS                                                  ooo



From rkern at ucsd.edu  Wed Jul  6 16:59:25 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Wed, 06 Jul 2005 13:59:25 -0700
Subject: [IPython-dev] Re: notebook composition UI thoughts
In-Reply-To: <200507062251.38149.hans_meine@gmx.net>
References: <c3aa5555050706092460896631@mail.gmail.com>
	<dahck6$o44$1@sea.gmane.org> <42CC406A.1070205@colorado.edu>
	<200507062251.38149.hans_meine@gmx.net>
Message-ID: <dahgne$52b$1@sea.gmane.org>

Hans Meine wrote:
> On Wednesday 06 July 2005 22:34, Fernando Perez wrote:
> 
>>This is mostly a confirmation of Robert's position.  Module-per-cell,
>>multiple-interpreters, auto-dependency analysis, etc. are all potentially
>>worthwhile ideas for certain contexts, but certainly not for a first shot.
> 
> Point taken and agreed.  Still, I'd like to have a "future works" section in 
> this discussion and the Wiki. ;-p

So make one! It's a Wiki. You're allowed to.  :-)

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From Fernando.Perez at colorado.edu  Wed Jul  6 17:04:15 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Wed, 06 Jul 2005 15:04:15 -0600
Subject: [IPython-dev] Re: notebook composition UI thoughts
In-Reply-To: <dahgne$52b$1@sea.gmane.org>
References: <c3aa5555050706092460896631@mail.gmail.com>
	<dahck6$o44$1@sea.gmane.org> <42CC406A.1070205@colorado.edu>
	<200507062251.38149.hans_meine@gmx.net> <dahgne$52b$1@sea.gmane.org>
Message-ID: <42CC474F.3010608@colorado.edu>

Robert Kern wrote:
> Hans Meine wrote:
> 
>>On Wednesday 06 July 2005 22:34, Fernando Perez wrote:
>>
>>
>>>This is mostly a confirmation of Robert's position.  Module-per-cell,
>>>multiple-interpreters, auto-dependency analysis, etc. are all potentially
>>>worthwhile ideas for certain contexts, but certainly not for a first shot.
>>
>>Point taken and agreed.  Still, I'd like to have a "future works" section in 
>>this discussion and the Wiki. ;-p
> 
> 
> So make one! It's a Wiki. You're allowed to.  :-)

Just a note: we're currently using the main scipy wiki for this.  I'm working 
on getting a proper ipython wiki on the new infrastructure being set up by the 
Enthought folks.  We already have our own SVN repo and a bunch of new toys, 
but this is work in progress, so give it a couple of days...

Best,

f



From edreamleo at charter.net  Thu Jul  7 08:25:24 2005
From: edreamleo at charter.net (Edward K. Ream)
Date: Thu, 7 Jul 2005 07:25:24 -0500
Subject: [IPython-dev] Why .py files?
Message-ID: <001201c582ee$f3a9b560$6500a8c0@edwardx330hgzb>

Could someone please explain the fundamental requirement of section 3.1 
(files), namely:



"A basic requirement of this project will be that the Python notebooks shall 
be valid Python source files."



I think of a notebook as similar to a Mathematica notebook, namely



a) the notebook app itself (Mathematica), and

b) notebook files (.nb files), for use primarily by the notebook app.



In this context, I would suppose the following:



- What counts is the user experience in the notebook app.



- The natural format for notebook files today would be xml.



- Most 'computing' (or scripting) would be done in the notebook app.  Why 
are separate .py files essential?



- The notebook app will have filters to produce .pdf, etc.  Why not have a 
filter to produce .py files?



Thanks.


Edward
--------------------------------------------------------------------
Edward K. Ream   email:  edreamleo at charter.net
Leo: http://webpages.charter.net/edreamleo/front.html
--------------------------------------------------------------------




From jdhunter at ace.bsd.uchicago.edu  Thu Jul  7 09:40:04 2005
From: jdhunter at ace.bsd.uchicago.edu (John Hunter)
Date: Thu, 07 Jul 2005 08:40:04 -0500
Subject: [IPython-dev] Why .py files?
In-Reply-To: <001201c582ee$f3a9b560$6500a8c0@edwardx330hgzb> ("Edward K.
	Ream"'s message of "Thu, 7 Jul 2005 07:25:24 -0500")
References: <001201c582ee$f3a9b560$6500a8c0@edwardx330hgzb>
Message-ID: <87r7eau43v.fsf@peds-pc311.bsd.uchicago.edu>

>>>>> "Edward" == Edward K Ream <edreamleo at charter.net> writes:

    Edward> Could someone please explain the fundamental requirement
    Edward> of section 3.1 (files), namely:

    Edward> "A basic requirement of this project will be that the
    Edward> Python notebooks shall be valid Python source files."

Some of the arguments for using *.py files, and the eventual rejection
of these arguments in favor of XML, as well as Robert Kern's prototype
implementation which uses XML as a storage format, have been discussed
exhaustively on this list in the last few weeks

  http://www.scipy.net/pipermail/ipython-dev/2005-June/thread.html
  http://www.scipy.net/pipermail/ipython-dev/2005-July/thread.html

JDH



From edreamleo at charter.net  Thu Jul  7 12:58:48 2005
From: edreamleo at charter.net (Edward K. Ream)
Date: Thu, 7 Jul 2005 11:58:48 -0500
Subject: [IPython-dev] Why .py files?
References: <001201c582ee$f3a9b560$6500a8c0@edwardx330hgzb>
	<87r7eau43v.fsf@peds-pc311.bsd.uchicago.edu>
Message-ID: <000801c58315$25274690$6500a8c0@edwardx330hgzb>

> Some of the arguments for using *.py files, and the eventual rejection
> of these arguments in favor of XML, as well as Robert Kern's prototype
> implementation which uses XML as a storage format, have been discussed
> exhaustively on this list in the last few weeks

So notebooks are xml files and not .py files?  In that case, what, if any, 
of the original proposal remains valid?  Is there a summary of the present 
proposal?

Thanks.

Edward
--------------------------------------------------------------------
Edward K. Ream   email:  edreamleo at charter.net
Leo: http://webpages.charter.net/edreamleo/front.html
--------------------------------------------------------------------




From Fernando.Perez at colorado.edu  Thu Jul  7 13:11:36 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Thu, 07 Jul 2005 11:11:36 -0600
Subject: [IPython-dev] Why .py files?
In-Reply-To: <000801c58315$25274690$6500a8c0@edwardx330hgzb>
References: <001201c582ee$f3a9b560$6500a8c0@edwardx330hgzb>
	<87r7eau43v.fsf@peds-pc311.bsd.uchicago.edu>
	<000801c58315$25274690$6500a8c0@edwardx330hgzb>
Message-ID: <42CD6248.2060309@colorado.edu>

Edward K. Ream wrote:
>>Some of the arguments for using *.py files, and the eventual rejection
>>of these arguments in favor of XML, as well as Robert Kern's prototype
>>implementation which uses XML as a storage format, have been discussed
>>exhaustively on this list in the last few weeks
> 
> 
> So notebooks are xml files and not .py files?  In that case, what, if any, 
> of the original proposal remains valid?  Is there a summary of the present 
> proposal?

I haven't (and I'm not going to) write a new formal proposal.  The original 
one was a working draft, and in the discussions John pointed to, we hashed out 
a new direction.  Robert has put up this info here:

http://www.scipy.org/wikis/featurerequests/NoteBook

There are links to the prototype code he has written using ElementTree there. 
  Toni and Tzanko are finishing up some school-related things, so I expect 
that we'll begin moving into production by next week.

Cheers,

f



From Fernando.Perez at colorado.edu  Thu Jul  7 14:04:40 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Thu, 07 Jul 2005 12:04:40 -0600
Subject: [IPython-dev] Re: notebook composition UI thoughts
In-Reply-To: <200507062251.38149.hans_meine@gmx.net>
References: <c3aa5555050706092460896631@mail.gmail.com>
	<dahck6$o44$1@sea.gmane.org> <42CC406A.1070205@colorado.edu>
	<200507062251.38149.hans_meine@gmx.net>
Message-ID: <42CD6EB8.4080605@colorado.edu>

Hans Meine wrote:
> On Wednesday 06 July 2005 22:34, Fernando Perez wrote:
> 
>>This is mostly a confirmation of Robert's position.  Module-per-cell,
>>multiple-interpreters, auto-dependency analysis, etc. are all potentially
>>worthwhile ideas for certain contexts, but certainly not for a first shot.
> 
> 
> Point taken and agreed.  Still, I'd like to have a "future works" section in 
> this discussion and the Wiki. ;-p

As Robert indicated, by all means please post these ideas there.  Once we get 
a proper ipython wiki (the tools are being currently installed) I'll take care 
of migrating the material over.

>>I should note that things like automatic dependency analysis are
>>notoriously difficult (I'd actually guess nearly impossible) in python. 
>>The language is far too dynamic for that, and this is _exactly_ the same
>>problem which has made all restricted execution systems for python fail (to
>>the point where they've been removed from the standard library). [...]
> 
> 
> You're right, it's difficult, but I am not taking the rest of your arguments; 
> I think that our notebook interface is very much different from a restricted 
> execution system, in the sense that it's feasible to just tell the user 
> "don't use import-in-an-eval, but just clearly write it out as an 
> unconditional statement (maybe even at the beginning of the cell), if you 
> want it to work".  Some users *want* it to work, and it's no big restriction 
> to use import w/o exec or eval.

I dislike half-working stuff.  For example, ipython has a dreload() command, 
which tries to do 'deep reloading', to properly update a module chain.  This 
code wasn't written by me, it's tricky and uncommented, and it came as part of 
one of the original projects I merged into ipython when I started.  I have 
received singing emails of people saying 'dreload() rocks, I'd use ipython if 
only to have access to it, etc.'.  And yet, I hardly publicize it, and I don't 
let it override the default reload() command (which it used to do in the 
original LazyPython).  The reason is that I've seen it fail in subtle cases, 
and I've received reports of puzzled users about its behavior.

Since I haven't taken the effort to fix it (I tried once, and failed), and in 
general it can't be 100% fixed (extension modules can't be reloaded as far as 
I know, regardless of what tricks you play), I keep it in a dark corner.

The moment you make a feature reasonably central to a framework, it HAS to be 
as robust as humanly possible.  Because people _will_ find ways of breaking 
it, either directly or because of third-party code they may be using which 
does funky things.  So I am very, very reluctant to put in a feature such as 
this, which I know is basically impossible to make really robust.

> This is very much understandable because it's about security, but IMO that's a 
> different world/topic.  I probably would not develop critical 
> spaceship-controlling software in any kind of IPython notebook sessions. ;-)

Well, you should talk to Frederic Mantegazza: he uses ipython as the control 
console for the largest neutron scattering source in Europe (at the Institut 
Laue-Langevin in Grenoble) :)

Cheers,

f



From Fernando.Perez at colorado.edu  Thu Jul  7 14:16:16 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Thu, 07 Jul 2005 12:16:16 -0600
Subject: [IPython-dev] Re: notebook composition UI thoughts
In-Reply-To: <42CD6EB8.4080605@colorado.edu>
References: <c3aa5555050706092460896631@mail.gmail.com>
	<dahck6$o44$1@sea.gmane.org> <42CC406A.1070205@colorado.edu>
	<200507062251.38149.hans_meine@gmx.net> <42CD6EB8.4080605@colorado.edu>
Message-ID: <42CD7170.3020209@colorado.edu>

Fernando Perez wrote:

> Well, you should talk to Frederic Mantegazza: he uses ipython as the control 
> console for the largest neutron scattering source in Europe (at the Institut 
> Laue-Langevin in Grenoble) :)

Oops, hit send a bit early before the urls:

http://ipython.scipy.org/misc/PyMAD_description.html
http://www.scipy.net/pipermail/ipython-dev/2005-April/000428.html

So 24/7 reliability _is_ a goal for ipython :)

Best,

f



From mtobis at gmail.com  Fri Jul  8 18:12:32 2005
From: mtobis at gmail.com (Michael Tobis)
Date: Fri, 8 Jul 2005 17:12:32 -0500
Subject: [IPython-dev] Re: notebook composition UI thoughts
In-Reply-To: <dah7gl$915$1@sea.gmane.org>
References: <c3aa5555050706092460896631@mail.gmail.com>
	<dah7gl$915$1@sea.gmane.org>
Message-ID: <c3aa5555050708151214de2532@mail.gmail.com>

On 7/6/05, Robert Kern <rkern at ucsd.edu> wrote:
> The current document model has two separate sections: a log and a sheet.
> The log contains all of the text from the In[NN] and Out[NN] prompts and
> other information. Each element is tagged with the number NN. The sheet
> contains the rich text that you write and references to those elements
> in the log. The sheet can contain any, all, or none of the elements in
> the log in any order and with any number of repetitions.

If I understand you correctly, then the plan does NOT adequately
address my concerns with the Mathematica approach, concerns which I
believe Hans acknowledged.

A second notebook with the same visible inputs in the same order as a
first notebook should, in my opinion, be required to yield the same
results in both cases.

If the sequence of operations is set in a log file normally invisible
to the user, then, as in Mathematica, two notebooks can have the same
sequence of inputs and different sequences of outputs.

This will tend to frustrate and drive away a developer of serious
content for the medium. Invisible statements and invisible or at least
unemphasized changes in the sequence of operations will cause the
notebook to fail to behave in the way that the content developer wants
or naturally expects.

>  When executed,
> the notebook will only consider the elements in the log in order. The
> formatting of the sheet is irrelevant to execution.

If I understand you correctly, then you are replacing Mathematica's
inconsistency with a method that is simple, consistent, and wrong.
 
> At the moment, we are focusing on creating an interactive environment,
> not a code development environment. 

I find this distinction not especially helpful. I am using an editor
to construct a document to tell a computer what to do. How is this not
code development?

Sorry to be so strong about this, but I hate to see this unfortunate
road traveled again.

Michael Tobis



From rkern at ucsd.edu  Fri Jul  8 19:32:52 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Fri, 08 Jul 2005 16:32:52 -0700
Subject: [IPython-dev] Re: notebook composition UI thoughts
In-Reply-To: <c3aa5555050708151214de2532@mail.gmail.com>
References: <c3aa5555050706092460896631@mail.gmail.com>
	<dah7gl$915$1@sea.gmane.org>
	<c3aa5555050708151214de2532@mail.gmail.com>
Message-ID: <dan2f6$rjr$1@sea.gmane.org>

Michael Tobis wrote:
> On 7/6/05, Robert Kern <rkern at ucsd.edu> wrote:
> 
>>The current document model has two separate sections: a log and a sheet.
>>The log contains all of the text from the In[NN] and Out[NN] prompts and
>>other information. Each element is tagged with the number NN. The sheet
>>contains the rich text that you write and references to those elements
>>in the log. The sheet can contain any, all, or none of the elements in
>>the log in any order and with any number of repetitions.
> 
> If I understand you correctly, then the plan does NOT adequately
> address my concerns with the Mathematica approach, concerns which I
> believe Hans acknowledged.
> 
> A second notebook with the same visible inputs in the same order as a
> first notebook should, in my opinion, be required to yield the same
> results in both cases.
 >
> If the sequence of operations is set in a log file normally invisible
> to the user, then, as in Mathematica, two notebooks can have the same
> sequence of inputs and different sequences of outputs.

Every input and output cell is visually marked with its number, just as 
in current ipython. The placement of the input cells in the document may 
not correlate with the order they would be reexecuted. The numbers will 
determine the order of execution. The same sequence of inputs (ordering 
  by the numbers) will give the same outputs. If I rearrange the 
position of the input cells in the document, the order of execution 
remains the same. Whether you actually do rearrange the positions, or 
visually remove cells, is up to you. As you say, "the order in which the 
reader wants to read the code does not necessarily match the order in 
which the compiler wants to process it." The order the interpreter wants 
to process the code is provided by the numbers of the input cells. The 
order the reader wants to read the code is given by the placement of 
those cells in the document.

There, of course, is the option of manually changing the numbers in 
order to exercise a finer control over how the document might be 
reexecuted. Additionally, we can enable more automatic renumberings as 
well. For example, suppose that I don't want any of the cells I visually 
removed from the document to show up in the log, and I want the cells 
renumbered to be contiguous (e.g. instead of skipping from 10 to 14 if I 
deleted cells 11, 12, and 13). We can provide a command that will clean 
the log and make the appropriate changes to the cells in the visual 
document.

> This will tend to frustrate and drive away a developer of serious
> content for the medium. Invisible statements and invisible or at least
> unemphasized changes in the sequence of operations will cause the
> notebook to fail to behave in the way that the content developer wants
> or naturally expects.

They are emphasized by the numbers attached to the cells. We can make an 
option (via a menu, keyboard shortcut, or toolbar button) that will 
display the full log.

>> When executed,
>>the notebook will only consider the elements in the log in order. The
>>formatting of the sheet is irrelevant to execution.
> 
> If I understand you correctly, then you are replacing Mathematica's
> inconsistency with a method that is simple, consistent, and wrong.

Quite possibly. However, it a method that is achievable. If you can 
contribute code that supports more complicated semantics, we will 
consider it.

I think, however, that I am not explaining myself quite clearly enough. 
I do believe that the approach I have described does address most of 
your concerns.

>>At the moment, we are focusing on creating an interactive environment,
>>not a code development environment. 
> 
> I find this distinction not especially helpful. I am using an editor
> to construct a document to tell a computer what to do. How is this not
> code development?

It's the difference between writing a module and typing at the ipython 
prompt. I, at least, approach these activities very differently.

> Sorry to be so strong about this, but I hate to see this unfortunate
> road traveled again.
> 
> Michael Tobis

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From mtobis at gmail.com  Sun Jul 10 22:42:22 2005
From: mtobis at gmail.com (Michael Tobis)
Date: Sun, 10 Jul 2005 21:42:22 -0500
Subject: Fwd: [IPython-dev] Re: notebook composition UI thoughts
In-Reply-To: <c3aa555505071015375615f1c3@mail.gmail.com>
References: <c3aa5555050706092460896631@mail.gmail.com>
	<dah7gl$915$1@sea.gmane.org>
	<c3aa5555050708151214de2532@mail.gmail.com>
	<dan2f6$rjr$1@sea.gmane.org>
	<c3aa555505071015375615f1c3@mail.gmail.com>
Message-ID: <c3aa5555050710194225eeeba5@mail.gmail.com>

On 7/8/05, Robert Kern <rkern at ucsd.edu> wrote:

> The placement of the input cells in the document may
> not correlate with the order they would be reexecuted. The numbers will
> determine the order of execution. The same sequence of inputs (ordering
>   by the numbers) will give the same outputs. If I rearrange the
> position of the input cells in the document, the order of execution
> remains the same.

Understood. This is sufficiently close to Mathematica's approach that
it seems to me to fail in the same way to live up to an important
potential of the medium, the one which interests me most. That is, as
a publication mechanism that can enforce the validity of a formal
argument.

> There, of course, is the option of manually changing the numbers in
> order to exercise a finer control over how the document might be
> reexecuted. Additionally, we can enable more automatic renumberings as
> well. For example, suppose that I don't want any of the cells I visually
> removed from the document to show up in the log, and I want the cells
> renumbered to be contiguous (e.g. instead of skipping from 10 to 14 if I
> deleted cells 11, 12, and 13). We can provide a command that will clean
> the log and make the appropriate changes to the cells in the visual
> document.

Indeed, this sort of functionality is exactly what I am looking for. I
am perfectly OK with it being optional. My purpose in injecting myself
into the conversation at this point is to ensure that options of this
sort are not foreclosed.

> > If I understand you correctly, then you are replacing Mathematica's
> > inconsistency with a method that is simple, consistent, and wrong.
>
> Quite possibly. However, it a method that is achievable. If you can
> contribute code that supports more complicated semantics, we will
> consider it.

I am a bit loathe to commit myself to this but I am indeed inclined to
participate in exactly this way.

I am not a deep enough Python magician to maintain the dependency
trees that Hans is imagining, but there are more pedestrian ways to
proceed that are within my capacities. They may lack something in
elegance and  performance but may be easy to implement and immediately
useful in many circumstances.

> >>At the moment, we are focusing on creating an interactive environment,
> >>not a code development environment.
> >
> > I find this distinction not especially helpful. I am using an editor
> > to construct a document to tell a computer what to do. How is this not
> > code development?
>
> It's the difference between writing a module and typing at the ipython
> prompt. I, at least, approach these activities very differently.

I think that the boundary between coder and user has become
artificially sharp.  I suspect this is in part a result of commercial
interests that aren't in the best interests of human progress. I
suppose that's a topic for another venue, though, and I should take
that particular axe and grind it elsewhere.

For present purposes, let me just emphasize that some of the best
designs are the ones that end up with utility far from their original
conceptions. Here's to the prospect of IPython being in that class!

regards
Michael Tobis



From rkern at ucsd.edu  Mon Jul 11 00:41:22 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Sun, 10 Jul 2005 21:41:22 -0700
Subject: [IPython-dev] Re: Fwd: Re: notebook composition UI thoughts
In-Reply-To: <c3aa5555050710194225eeeba5@mail.gmail.com>
References: <c3aa5555050706092460896631@mail.gmail.com>
	<dah7gl$915$1@sea.gmane.org>	<c3aa5555050708151214de2532@mail.gmail.com>
	<dan2f6$rjr$1@sea.gmane.org>	<c3aa555505071015375615f1c3@mail.gmail.com>
	<c3aa5555050710194225eeeba5@mail.gmail.com>
Message-ID: <dast9m$h3p$1@sea.gmane.org>

[Michael originally sent the previous message privately to me and not 
the list, and I replied privately as well. We seem to be converging on 
the opinion that the need for more complicated execution semantics (like 
Hans's dependency graph) can, for the most part, be ameliorated by 
layering actions on top of the simple, ipython-like execution semantics. 
Future work on those more complicated semantics may be worth considering 
after this Summer of Code project has been completed.]

Michael Tobis wrote:

 > On 7/8/05, Robert Kern <rkern at ucsd.edu> wrote:
 >
 >> The placement of the input cells in the document may
 >> not correlate with the order they would be reexecuted. The numbers will
 >> determine the order of execution. The same sequence of inputs (ordering
 >>  by the numbers) will give the same outputs. If I rearrange the
 >> position of the input cells in the document, the order of execution
 >> remains the same.
 >
 > Understood. This is sufficiently close to Mathematica's approach that
 > it seems to me to fail in the same way to live up to an important
 > potential of the medium, the one which interests me most. That is, as
 > a publication mechanism that can enforce the validity of a formal
 > argument.

I understand your concern, but at the moment, we are not interested in 
living up to a lofty potential, but to produce useful tools by September 
1st. I agree that the semantics I have proposed are not ideal, but they 
are simple to implement. We hope to build components that can be later 
reassembled with different semantics as they become implementable.

 >> There, of course, is the option of manually changing the numbers in
 >> order to exercise a finer control over how the document might be
 >> reexecuted. Additionally, we can enable more automatic renumberings as
 >> well. For example, suppose that I don't want any of the cells I visually
 >> removed from the document to show up in the log, and I want the cells
 >> renumbered to be contiguous (e.g. instead of skipping from 10 to 14 if I
 >> deleted cells 11, 12, and 13). We can provide a command that will clean
 >> the log and make the appropriate changes to the cells in the visual
 >> document.
 >
 > Indeed, this sort of functionality is exactly what I am looking for. I
 > am perfectly OK with it being optional. My purpose in injecting myself
 > into the conversation at this point is to ensure that options of this
 > sort are not foreclosed.

We are not foreclosing the development of anything. We are, however, 
making choices as to what we will be developing for this Summer of Code 
project. Especially, we are not foreclosing on things that can be 
readily built on top of the simple semantics I have proposed.

 >>> If I understand you correctly, then you are replacing Mathematica's
 >>> inconsistency with a method that is simple, consistent, and wrong.
 >>
 >> Quite possibly. However, it a method that is achievable. If you can
 >> contribute code that supports more complicated semantics, we will
 >> consider it.
 >
 > I am a bit loathe to commit myself to this but I am indeed inclined to
 > participate in exactly this way.

Then you can understand why I am loathe to commit myself or others to do 
something that I think is probably impossible.

 > I am not a deep enough Python magician to maintain the dependency
 > trees that Hans is imagining, but there are more pedestrian ways to
 > proceed that are within my capacities. They may lack something in
 > elegance and  performance but may be easy to implement and immediately
 > useful in many circumstances.

It is possible that the specific capabilities that you and Hans are 
imagining are achievable, but I have my doubts. Working code, of course, 
relieves all doubts.

 >>>> At the moment, we are focusing on creating an interactive environment,
 >>>> not a code development environment.
 >>>
 >>> I find this distinction not especially helpful. I am using an editor
 >>> to construct a document to tell a computer what to do. How is this not
 >>> code development?
 >>
 >> It's the difference between writing a module and typing at the ipython
 >> prompt. I, at least, approach these activities very differently.
 >
 > I think that the boundary between coder and user has become
 > artificially sharp.  I suspect this is in part a result of commercial
 > interests that aren't in the best interests of human progress. I
 > suppose that's a topic for another venue, though, and I should take
 > that particular axe and grind it elsewhere.

I am not making a distinction between coder and user. I am mentioning a 
difference that I see in the kinds of activities that I do. I write 
reusable modules one way; I write interactive code another way. The 
mechanisms that I want to support the one activity are different from 
the mechanisms that I want to support the other.

I would also note that literate programming, I think, satisfies the 
"notebook" niche for the "writing reusable code" activity, and that 
there is little point in writing a new literate programming tool. Leo, 
CWEB, et al. do that job just fine. There's no point in trying to 
reinvent them in one summer. However, I don't think they do the job for 
interactive work like what we use ipython for, hence we are focusing our 
summer efforts on creating the best interactive environment.

 > For present purposes, let me just emphasize that some of the best
 > designs are the ones that end up with utility far from their original
 > conceptions. Here's to the prospect of IPython being in that class!
 >
 > regards
 > Michael Tobis

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From antont at an.org  Fri Jul 15 08:14:28 2005
From: antont at an.org (Toni Alatalo)
Date: Fri, 15 Jul 2005 15:14:28 +0300
Subject: [IPython-dev] Re: Notebook roadmap
In-Reply-To: <daf7ta$via$1@sea.gmane.org>
References: <daac75$hlt$1@sea.gmane.org> <42CB01B5.703@colorado.edu>
	<daf7ta$via$1@sea.gmane.org>
Message-ID: <42D7A8A4.8090207@an.org>

Robert Kern wrote:

Hello,

> Fernando Perez wrote:
>
>> Robert, is your code available anywhere?  This could be used as a 
>> starter by Toni on the document work, as well as providing Tzanko a 
>> mockup target to load.
>
> And some documentation to boot:
> http://www.scipy.org/wikis/featurerequests/NoteBookXML


That and the discussions here have been something to digest indeed, but 
the picture is pretty clear for me now (as in understanding the targets 
and the Robert's implementation).

Where next? Tzanko, can you interface with the 'notabene' API, is there 
something to discuss about it, or things missing I could start to add?

Some random remarks from reading the code:

- i also like the target of ReST output, and now we perhaps don't need 
to write that ourselves, 'cause a xhtml2rest 'renderer' was recenlty 
posted to docutils-dev. added a comment about that to the wiki page with 
a link to the code and discussions at 
http://permalink.gmane.org/gmane.text.docutils.devel/3076

- don't know how important writing notebooks as ReST will be (as 
sketched in sketch.nbk), but there some docutils directive would be nice 
for adding the code snippets (cells). .. python:  ..even? as XML is not 
for humans to write.. (and hardly even to read). and of course the 
ipython log would be also nicer to read in a non-XML format, e.g. like 
it originally shows in the ipython prompt or something (this is a bit 
strangely inverse, 'cause rst was originally planned to be written in 
.py files, but here it's vice versa..?)

- the 'ipython monkeypatch' seems to work (except that unlike the 
example shows a name attribute for the Notebook is required - would it 
be better to add some default name for notebooks, or keep requiring that 
and fix the docstring?). am happy that didn't have to write that :o

Hm, i feel like am missing all the important issues that do lurk 
somewhere in the back of the mind, but those are at least some concrete 
points -- i guess when we get actually going with experimenting with a 
prorotype etc. they surface.

So ideas about what to tackle next would be very welcome. Still this 
morning I was not totally convinced if Robert's approach was the way to 
go, but now it seems like that again. Thanks a lot for putting it together!

~Toni



From rkern at ucsd.edu  Fri Jul 15 10:53:38 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Fri, 15 Jul 2005 07:53:38 -0700
Subject: [IPython-dev] Re: Notebook roadmap
In-Reply-To: <42D7A8A4.8090207@an.org>
References: <daac75$hlt$1@sea.gmane.org> <42CB01B5.703@colorado.edu>
	<daf7ta$via$1@sea.gmane.org> <42D7A8A4.8090207@an.org>
Message-ID: <db8ilo$7r5$1@sea.gmane.org>

Toni Alatalo wrote:

> That and the discussions here have been something to digest indeed, but 
> the picture is pretty clear for me now (as in understanding the targets 
> and the Robert's implementation).
> 
> Where next? Tzanko, can you interface with the 'notabene' API, is there 
> something to discuss about it, or things missing I could start to add?
> 
> Some random remarks from reading the code:
> 
> - i also like the target of ReST output, and now we perhaps don't need 
> to write that ourselves, 'cause a xhtml2rest 'renderer' was recenlty 
> posted to docutils-dev. added a comment about that to the wiki page with 
> a link to the code and discussions at 
> http://permalink.gmane.org/gmane.text.docutils.devel/3076

I didn't really have in mind a reST target. Embedding the
<ipython-block> tags in text was just the simplest thing I could think
of at the time. There are better targets now.

> - don't know how important writing notebooks as ReST will be (as 
> sketched in sketch.nbk), but there some docutils directive would be nice 
> for adding the code snippets (cells). .. python:  ..even?

http://bob.pythonmac.org/archives/2003/11/10/making-docutils-do-python-syntax-highlighting-for-html-output/

It still needs a LaTeX handler.

> as XML is not 
> for humans to write.. (and hardly even to read). and of course the 
> ipython log would be also nicer to read in a non-XML format, e.g. like 
> it originally shows in the ipython prompt or something (this is a bit 
> strangely inverse, 'cause rst was originally planned to be written in 
> .py files, but here it's vice versa..?)

Well, outputing just a log to any number of formats has shown itself to
be pretty trivial.

> - the 'ipython monkeypatch' seems to work (except that unlike the 
> example shows a name attribute for the Notebook is required - would it 
> be better to add some default name for notebooks, or keep requiring that 
> and fix the docstring?). am happy that didn't have to write that :o

Tell me about it. Actually, that bit needs to be refactored out to an
intermediary object with some other stuff to leave the notebook object
pure. Its docstring is the least of the things wrong with it.

> Hm, i feel like am missing all the important issues that do lurk 
> somewhere in the back of the mind, but those are at least some concrete 
> points -- i guess when we get actually going with experimenting with a 
> prorotype etc. they surface.

I've been fiddling around trying to settle on an XML markup to use. We
could try a subset of XHTML, but that is a little less semantic and more
presentational than I would prefer. For example, it uses

  <h1>Title</h1>
  <p>Something</p>
  <h2>Lower title</h2>
  <p>Something else in a sub-section</p>

where Docutils/DocBook uses

   <section>
     <title>Title</title>
     <para>Something</para>
     <section>
       <title>Lower title</title>
       <para>Something else in a sub-section</para>
     </section>
   </section>

On the plus side, a subset of XHTML is what twisted.lore uses, and
twisted.lore seems to be well-designed for plugins for custom tags. It
outputs both XHTML and LaTeX.

I haven't been so thrilled with prospect of using Docutils as a document
generation library. It seems to be very much focused on the input
reST-format. Ignoring that bit and generating documents programmatically
is apparently not so straightforward. Plugging in custom content is even 
less so.

By far the easiest approach so far has been to use DocBook's XML,
transforming the <ipython-block> into a <programlisting> (with syntax
highlighting tags inside), and getting HTML and LaTeX out via the XSL
stylesheets. Using lxml instead of ElementTree (same basic API but adds
full XPath and XSLT support), this all happens in Python. The output is
very configurable and looks quite nice.

Incidentally, Docutils' model seems to be mostly borrowed from DocBook,
so if we were to choose an XML subset from Docutils, we'd end up with a
subset of DocBook, too. I have a feeling that we'll settle on a
DocBook subset even if we handle the document generation ourselves or
massage it slightly to pass to some other library and don't
bother with the XSL stylesheets.

> So ideas about what to tackle next would be very welcome. Still this 
> morning I was not totally convinced if Robert's approach was the way to 
> go, but now it seems like that again. Thanks a lot for putting it together!

For the document side, one thing we're going to need no matter what is
an object(s) to hold style information. I made a stupid stab at it in my
SVN sandbox.

On the GUI side, we should try to get to the point of rendering 
something as simple as this:

<sheet>
   <para>I am a paragraph.</para>
   <ipython-block>
      ...
   </ipython-block>
   <para>I am another paragraph.</para>
   <ipython-block>
      ...
   </ipython-block>
   <para>Oh yes, it's me again.</para>
</sheet>

On the ipython side, we need clean ways to do all kinds of things. I'll 
get around to writing them down later.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From Fernando.Perez at colorado.edu  Fri Jul 15 19:27:42 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Fri, 15 Jul 2005 17:27:42 -0600
Subject: [IPython-dev] [FWD] An interesting take on the Notebook Problem
In-Reply-To: <mailman.718.1121443578.29351.ipython-dev@scipy.net>
References: <mailman.718.1121443578.29351.ipython-dev@scipy.net>
Message-ID: <42D8466E.1010509@colorado.edu>

[Chris, note that ipytho-dev discards non-subscriber posts (too much spam). 
I've manually forwarded your message ]

> ------------------------------------------------------------------------
> 
> Subject: An interesting take on the Notebook Problem
> From: Christopher Hart <christopher.e.hart at gmail.com>
> Date: Fri, 15 Jul 2005 13:05:02 -0400
> To: ipython-dev at scipy.net
> 
> To: ipython-dev at scipy.net
> 
> 
> Hi,
> 
> I've been lurking and casually reading the notebook discussion during
> the past month or so.  I'm quite excited about the progress that seems
> to be going on, and the ideas that are being flushed out.
> 
> In the bioinformatics community the problem of recording ones analysis
> for both later inspection and sharing is a large problem.  This paper
> takes, although not precisely focused on the notion of an interactive
> notebook, might still be an interesting read to anyone involved in
> creating a notebook.  I think it highlights nicely some of the
> problems and the desires of at least one active community that would
> benefit greatly from a ipython notebook.  Personally, the particular
> implementation they provide doesn't appeal to me - but that's largely
> because I prefer (i)python ;)  Aside from the methods section - little
> actually knowledge of bioinformatics is a prereq for reading.
> 
> Just thought I'd share....
> 
> Robert Gentleman (2005) "Reproducible Research: A Bioinformatics Case
> Study", Statistical Applications in Genetics and Molecular Biology:
> Vol. 4: No. 1, Article 2.
> http://www.bepress.com/sagmb/vol4/iss1/art2
> 
> """
> Reproducible Research: A Bioinformatics Case Study
> 
> Robert Gentleman, Harvard University
> 
> ABSTRACT:
> While scientific research and the methodologies involved have gone
> through substantial technological evolution the technology involved in
> the publication of the results of these endeavors has remained
> relatively stagnant. Publication is largely done in the same manner
> today as it was fifty years ago. Many journals have adopted electronic
> formats, however, their orientation and style is little different from
> a printed document. The documents tend to be static and take little
> advantage of computational resources that might be available. Recent
> work, Gentleman and Temple Lang (2003), suggests a methodology and
> basic infrastructure that can be used to publish documents in a
> substantially different way. Their approach is suitable for the
> publication of papers whose message relies on computation. Stated
> quite simply, Gentleman and Temple Lang (2003) propose a paradigm
> where documents are mixtures of code and text. Such documents may be
> self-contained or they may be a component of a compendium which
> provides the infrastructure needed to provide access to data and
> supporting software. These documents, or compendiums, can be processed
> in a number of different ways. One transformation will be to replace
> the code with its output -- thereby providing the familiar, but
> limited, static document. <p /> In this paper we apply these concepts
> to a seminal paper in bioinformatics, namely The Molecular
> Classification of Cancer, Golub et al (1999). The authors of that
> paper have generously provided data and other information that have
> allowed us to largely reproduce their results. Rather than reproduce
> this paper exactly we demonstrate that such a reproduction is possible
> and instead concentrate on demonstrating the usefulness of the
> compendium concept itself.
> """
> 
> --chris



From Fernando.Perez at colorado.edu  Sun Jul 17 23:13:39 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Sun, 17 Jul 2005 21:13:39 -0600
Subject: [IPython-dev] IPython moved over to SVN...
Message-ID: <42DB1E63.8090803@colorado.edu>

Hi all,

this is just a heads up for those of you who follow ipython's CVS tree: it is 
no more (well, it remains up, but it's not being updated anymore).

We finally have subversion support for ipython, so those of you who are 
interested should do a fresh svn checkout:

svn co http://ipython.scipy.org/svn/ipython/ipython/trunk ipython

That's the trunk checkout; if we start doing very disruptive work we'll open a 
branch for that and will announce it accordingly.

One of the benefits of the new SVN setup is the flexibility to have new users 
and host other work right there.  In particular, Robert Kern, Toni, Tzanko and 
I are all working on the Summer projects at the same repo.  Anyone interested 
can either check out the respective trunks here:

svn co http://ipython.scipy.org/svn/ipython/nbdoc/trunk nbdoc
svn co http://ipython.scipy.org/svn/ipython/nbshell/trunk nbshell
svn co http://ipython.scipy.org/svn/ipython/google-rkern/trunk google-rkern

Or, if you are interested in the whole thing, you can just pull the entire 
repo in one call:

svn co http://ipython.scipy.org/svn/ipython

Note that this brings the whole repo, with all branches and tags, not just the 
trunks for each project.

All new work, including recent bugfixes, is now taking place in the SVN tree.

I'd like to thank Joe Cooper, the Enthought admin, for his unending patience 
with all my little questions as I was transitioning to the new environment. 
The new setup is extremely nice, with a lot more flexibility and control than 
we had before, and I should also thank Enthought for their generous hosting 
with such nice, speedy resources.

Regards,

f



From tsanko at gmail.com  Mon Jul 18 06:18:43 2005
From: tsanko at gmail.com (Tzanko Matev)
Date: Mon, 18 Jul 2005 13:18:43 +0300
Subject: [IPython-dev] Changes to Notebook Format
Message-ID: <1121681923.3210.18.camel@tzanko>

Hi all,

I would like to suggest a few changes to the ipython-log part of the
notebook format. I think that the following structure might be better:

<ipython-log logid="default-log">

<element id="1">
<input type="normal">
def foo(x, y):
    return x + y
</input>
</element>

<element id="2">   
<input type="normal">
foo(3, 4)
</input>
<output type="normal">
7
</output>
</element>

</ipython-log>

The input tags can have types "normal" and "special" which correspond to
the tags <input> and <special-input> of the current format. The output
tags can have types "normal", "stdout", "figure", etc. One input tag can
have several corresponding output tags with different types. I think
that the resulting ElementTree object would be more useful that way. In
order to parse a file of the current format I need to create its
corresponding ElementTree object and then I must make my own data
representation from that object, because searching for tags with XPath
is slow and with the current format there is no easier way to retrieve
the text I need. With the proposed format I don't really need another
internal data rpresentation, which would make my code cleaner.

Also with the new format we won't really need the id argument which will
make the format less error-prone.

Tzanko



From rkern at ucsd.edu  Mon Jul 18 07:17:37 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Mon, 18 Jul 2005 04:17:37 -0700
Subject: [IPython-dev] Re: Changes to Notebook Format
In-Reply-To: <1121681923.3210.18.camel@tzanko>
References: <1121681923.3210.18.camel@tzanko>
Message-ID: <dbg34q$41i$1@sea.gmane.org>

Tzanko Matev wrote:
> Hi all,
> 
> I would like to suggest a few changes to the ipython-log part of the
> notebook format. I think that the following structure might be better:
> 
> <ipython-log logid="default-log">
> 
> <element id="1">

As a sidenote: conventionally, the "id" attribute on any tag must be 
unique document-wide, so we'll stick with "number" here.

> <input type="normal">
> def foo(x, y):
>     return x + y
> </input>
> </element>
> 
> <element id="2">   
> <input type="normal">
> foo(3, 4)
> </input>
> <output type="normal">
> 7
> </output>
> </element>
> 
> </ipython-log>
> 
> The input tags can have types "normal" and "special" which correspond to
> the tags <input> and <special-input> of the current format. The output
> tags can have types "normal", "stdout", "figure", etc. One input tag can
> have several corresponding output tags with different types.

Every "special" input will also have a "normal" input that keeps the 
transformed-into-valid-Python command, too.

> I think
> that the resulting ElementTree object would be more useful that way. In
> order to parse a file of the current format I need to create its
> corresponding ElementTree object and then I must make my own data
> representation from that object, because searching for tags with XPath
> is slow and with the current format there is no easier way to retrieve
> the text I need. With the proposed format I don't really need another
> internal data rpresentation, which would make my code cleaner.
> 
> Also with the new format we won't really need the id argument which will
> make the format less error-prone.

I think that it's entirely possible that the log might have holes in it. 
"Remove unused log entries" and "Renumber entries to restore contiguity" 
are two different operations. One might want to perform the former but 
not the latter if one references In[NN] or Out[NN].

But that may be something we can worry about later.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From rkern at ucsd.edu  Mon Jul 18 09:20:50 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Mon, 18 Jul 2005 06:20:50 -0700
Subject: [IPython-dev] Re: Changes to Notebook Format
In-Reply-To: <1121681923.3210.18.camel@tzanko>
References: <1121681923.3210.18.camel@tzanko>
Message-ID: <dbgabr$p97$1@sea.gmane.org>

Tzanko Matev wrote:
> Hi all,
> 
> I would like to suggest a few changes to the ipython-log part of the
> notebook format. I think that the following structure might be better:
> 
> <ipython-log logid="default-log">
> 
> <element id="1">
> <input type="normal">
> def foo(x, y):
>     return x + y
> </input>
> </element>
> 
> <element id="2">   
> <input type="normal">
> foo(3, 4)
> </input>
> <output type="normal">
> 7
> </output>
> </element>
> 
> </ipython-log>
> 
> The input tags can have types "normal" and "special" which correspond to
> the tags <input> and <special-input> of the current format. The output
> tags can have types "normal", "stdout", "figure", etc.

In trying to convert some code to this scheme, I'm finding that it's 
rather cumbersome. Since the number is attached to the ancestor of 
<input> and <output>, I can't just get all normal inputs or all special 
inputs. Nor can I easily tell when a an particular entry also has a 
special input. For example, when I want to export the plain code, I used 
to use a single XPath expression to get all normal inputs and another 
XPath expression to all special inputs, which I stick in a dictionary 
keyed by the number. Then I go through the normal inputs and replace 
them with the special inputs if they are in the dictionary. With the 
above scheme, I iterate over the <cell> elements (I changed <element> to 
<cell>), check for the presence of a special input with an XPath 
expression, use it if found or if not, then I get the normal input with 
an XPath expression. That's not ideal.

Can you describe the operations that you need to do? How often do you 
have to access this data such that XPath is observably too slow? I think 
it's probable that this rearrangement may *increase* the number of XPath 
expressions you need to evaluate.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From Fernando.Perez at colorado.edu  Mon Jul 18 12:09:01 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Mon, 18 Jul 2005 10:09:01 -0600
Subject: [FWD] Re: [IPython-dev] Re: Changes to Notebook Format
Message-ID: <42DBD41D.9030809@colorado.edu>

[Toni, I've added you to the whitelist, but if you use multiple email aliases, 
remember that the ipython lists discard non-subscriber posts]
-------------- next part --------------
An embedded message was scrubbed...
From: Toni Alatalo <antont at kyperjokki.fi>
Subject: Re: [IPython-dev] Re: Changes to Notebook Format
Date: Mon, 18 Jul 2005 14:36:32 +0300 (EEST)
Size: 3483
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20050718/68ae10e7/attachment.eml>

From rkern at ucsd.edu  Mon Jul 18 13:02:47 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Mon, 18 Jul 2005 10:02:47 -0700
Subject: [IPython-dev] Re: [FWD] Re: Re: Changes to Notebook Format
In-Reply-To: <42DBD41D.9030809@colorado.edu>
References: <42DBD41D.9030809@colorado.edu>
Message-ID: <dbgnbv$3pg$1@sea.gmane.org>

Fernando Perez wrote:
> [Toni, I've added you to the whitelist, but if you use multiple email 
> aliases, remember that the ipython lists discard non-subscriber posts]
> 
> ------------------------------------------------------------------------
> 
> Subject:
> Re: [IPython-dev] Re: Changes to Notebook Format
> From:
> Toni Alatalo <antont at kyperjokki.fi>
> Date:
> Mon, 18 Jul 2005 14:36:32 +0300 (EEST)
> To:
> ipython-dev at scipy.net
> 
> To:
> ipython-dev at scipy.net
> 
> On Mon, 18 Jul 2005, Robert Kern wrote:
> 
>>><ipython-log logid="default-log">
>>><element id="1">
>>
>>As a sidenote: conventionally, the "id" attribute on any tag must be 
>>unique document-wide, so we'll stick with "number" here.
> 
> ok.
> 
>>>The input tags can have types "normal" and "special" which correspond to
>>>the tags <input> and <special-input> of the current format. The output
>>>tags can have types "normal", "stdout", "figure", etc. One input tag can
>>>have several corresponding output tags with different types.
>>
>>Every "special" input will also have a "normal" input that keeps the 
>>transformed-into-valid-Python command, too.
> 
> hm didn't get all of this yet but ok i guess.

When one does a "special" input (a %magic, a !system command, etc.), 
IPython will transform it to valid Python, either to a method invocation 
on the interpreter object or to a comment. I think we need to store both 
because the transformation may be dependent on what has previously been 
executed.

>>>that the resulting ElementTree object would be more useful that way. In
>>>order to parse a file of the current format I need to create its
>>>corresponding ElementTree object and then I must make my own data
>>>representation from that object, because searching for tags with XPath
>>>is slow and with the current format there is no easier way to retrieve
>>>the text I need. With the proposed format I don't really need another
>>>internal data rpresentation, which would make my code cleaner.
> 
> i definitely agree that you should not have to do any parsing / 
> structuring of the data for the GUI work -- it's my task to keep the 
> underlying lib in such a shape that it facilitates your work (of course we 
> don't have to keep this separation artificially strict, but that's the 
> idea in principle, right?)
> 
> so please voice out all the demands for the API - like methods you'd like 
> to have for the notebook objects etc.
> 
>>are two different operations. One might want to perform the former but 
>>not the latter if one references In[NN] or Out[NN].
>>But that may be something we can worry about later.
> 
> ok
> 
>>Robert Kern
> 
> btw Robert, did you implement / plan to implement those changes already, 
> or would it be for me to do? a chance to get it going with your codebase, 
> perhaps..

http://ipython.scipy.org/svn/ipython/google-rkern/branches/cell

I resolved my issues by making a Cell object that takes the element and 
assigns the contents of its children to Python attributes. I've left the 
figures out of the cells because they have a somewhat different 
structure. We can change that later when we start doing images in the 
GUI and think that it's necessary.

Also, check out the sample output in the test/ directory.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From tsanko at gmail.com  Tue Jul 19 05:01:34 2005
From: tsanko at gmail.com (Tzanko Matev)
Date: Tue, 19 Jul 2005 12:01:34 +0300
Subject: [IPython-dev] Python version
Message-ID: <1121763694.4785.5.camel@tzanko>

Hi all,

What Python version should we require for the notebook packages to run?
Right now I am using Python 2.3 but I think that my code should work on
Python 2.2 as I don't use any new features. Robert Kern's code uses some
Python 2.4 features like class decorators, for example. So which is the
lowest version we should support?

Best,
tzanko



From Fernando.Perez at colorado.edu  Tue Jul 19 12:54:31 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Tue, 19 Jul 2005 10:54:31 -0600
Subject: [IPython-dev] Python version
In-Reply-To: <1121763694.4785.5.camel@tzanko>
References: <1121763694.4785.5.camel@tzanko>
Message-ID: <42DD3047.7030501@colorado.edu>

Tzanko Matev wrote:
> Hi all,
> 
> What Python version should we require for the notebook packages to run?
> Right now I am using Python 2.3 but I think that my code should work on
> Python 2.2 as I don't use any new features. Robert Kern's code uses some
> Python 2.4 features like class decorators, for example. So which is the
> lowest version we should support?

I try to keep ipython 2.2-compatible, though to be honest I haven't actually 
_checked_ with 2.2 in a long time.  My default python runtime is 2.3, and 
there are still a lot of 2.3 installations out there.  I think it would be 
nice to keep 2.3 compatibility, but if it becomes a real burden, it's OK to go 
for 2.4 (I have it installed as well).  By the time this code becomes really 
stable and widely usable, 2.4 will have been out for long enough that it will 
be reasonable to use it as a baseline.

In summary, if @decorators and such really are helping the readability and 
clarity of the code, keep 2.4.  I'll adapt.

Cheers,

f



From rkern at ucsd.edu  Tue Jul 19 13:15:38 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Tue, 19 Jul 2005 10:15:38 -0700
Subject: [IPython-dev] Re: Python version
In-Reply-To: <42DD3047.7030501@colorado.edu>
References: <1121763694.4785.5.camel@tzanko> <42DD3047.7030501@colorado.edu>
Message-ID: <dbjcg4$d5j$1@sea.gmane.org>

Fernando Perez wrote:

> In summary, if @decorators and such really are helping the readability 
> and clarity of the code, keep 2.4.  I'll adapt.

2.4 features I use:

* @classmethod: easily replaced
* builtin set(): easily replaced
* generator expressions: [easily replaced], but oh, how I'll miss them
* keyed sorts and sorted(): less easily replaced

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From shlomme at gmx.net  Tue Jul 19 13:25:11 2005
From: shlomme at gmx.net (Torsten Marek)
Date: Tue, 19 Jul 2005 19:25:11 +0200
Subject: [IPython-dev] Re: Python version
In-Reply-To: <dbjcg4$d5j$1@sea.gmane.org>
References: <1121763694.4785.5.camel@tzanko> <42DD3047.7030501@colorado.edu>
	<dbjcg4$d5j$1@sea.gmane.org>
Message-ID: <42DD3777.6070305@gmx.net>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Robert Kern schrieb:
> Fernando Perez wrote:
> 
>> In summary, if @decorators and such really are helping the readability
>> and clarity of the code, keep 2.4.  I'll adapt.
> 
> 
> 2.4 features I use:
> 
> * @classmethod: easily replaced
> * builtin set(): easily replaced
> * generator expressions: [easily replaced], but oh, how I'll miss them
> * keyed sorts and sorted(): less easily replaced
> 
I think I there are "backports" of sorted() to Python 2.3, but I just can't find
them on google right now.

greetings
Torsten

- --
Torsten Marek <shlomme at gmx.net>
ID: A244C858 -- FP: 1902 0002 5DFC 856B F146  894C 7CC5 451E A244 C858
Keyserver: subkeys.pgp.net

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFC3TdlfMVFHqJEyFgRAlo2AJ9IabMm15NDADtAmtFbTl6i7IvPdwCgm68A
nsAfzgaQrCVEufoErTwXWxs=
=Qij6
-----END PGP SIGNATURE-----



From Fernando.Perez at colorado.edu  Tue Jul 19 13:28:24 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Tue, 19 Jul 2005 11:28:24 -0600
Subject: [IPython-dev] Re: Python version
In-Reply-To: <dbjcg4$d5j$1@sea.gmane.org>
References: <1121763694.4785.5.camel@tzanko> <42DD3047.7030501@colorado.edu>
	<dbjcg4$d5j$1@sea.gmane.org>
Message-ID: <42DD3838.2000409@colorado.edu>

Robert Kern wrote:
> Fernando Perez wrote:
> 
> 
>>In summary, if @decorators and such really are helping the readability 
>>and clarity of the code, keep 2.4.  I'll adapt.
> 
> 
> 2.4 features I use:
> 
> * @classmethod: easily replaced
> * builtin set(): easily replaced
> * generator expressions: [easily replaced], but oh, how I'll miss them
> * keyed sorts and sorted(): less easily replaced

Nah, keep 2.4 then.  I'll survive.

Let's keep the mainline ipython 2.2-safe, though.  That one is in real use out 
there, so I don't want to break things for existing users.  I've stuffed all 
2.2 compatibility symbols in genutils.py, so put in there anything you need 
which may be a backport from 2.3/4, if you touch ipython itself (this goes 
especially to Tzanko).

cheers,

f



From antont at an.org  Fri Jul 22 10:05:18 2005
From: antont at an.org (Toni Alatalo)
Date: Fri, 22 Jul 2005 17:05:18 +0300
Subject: [IPython-dev] [FWD] An interesting take on the Notebook Problem
In-Reply-To: <42D8466E.1010509@colorado.edu>
References: <mailman.718.1121443578.29351.ipython-dev@scipy.net>
	<42D8466E.1010509@colorado.edu>
Message-ID: <42E0FD1E.6000807@an.org>

Fernando Perez wrote:

> [Chris, note that ipytho-dev discards non-subscriber posts (too much 
> spam). I've manually

thanks for posting this - was quite a good read, and something that 
makes the notebook project even more interesting

>> takes, although not precisely focused on the notion of an interactive
>> notebook, might still be an interesting read to anyone involved in
>> (..)
>> Robert Gentleman (2005) "Reproducible Research: A Bioinformatics Case
>> Study", Statistical Applications in Genetics and Molecular Biology:
>> Vol. 4: No. 1, Article 2.
>> http://www.bepress.com/sagmb/vol4/iss1/art2
>
in fact, the question i was left pondering, regarding the work on python 
notebooks and the discussions about literate programming vs. the 'REPL' 
(Read-Eval-Print-Loop, i.e. ipython) modes of working, was how the 
interactive use fits the picture there.

in that paper, they define the so-called Compendiums 'navigable 
documents', referring to the reader being able to 'explore and 
reproduce' the content. is such exploration of e.g. a dataset or a model 
a case of 'REPL', when e.g. changing variables or perhaps defining 
functions to explore te data/model in different cases? i would think so. 
that way such compendiums and ipython would be a perfect match, 
something that literate programming (which if of course one of the 
underlying basic ideas, as Gentleman also mentions) perhaps does not 
facilitate in itself? (although checking out Leo etc. is still in my 
to-do list so can't really tell).

installed R the other day to get a feel of that environment, seemed like 
a rich world and the interactive use was pretty nice (though i still 
think it is & will be better with Python), but unfortunately could not 
get the GolubRR package to work (yet). seems to be some version issue: 
perhaps that GolubRR is packaged for R < 2.0, while the current (in e.g. 
Debian) is > 2.0. a question about it had been posted to the 
Bioconductor mailing list, but i could not find any answer (nor haven't 
gotten one from the poster of that mail) - 
https://stat.ethz.ch/pipermail/bioconductor/2004-November/006906.html .

Besides the SciPy tutorial, perhaps that seminal paper Golub and the R 
compendium should also be among the 'test cases' for Python notebooks 
(hopefully the dataset is not too tricky to convert to whatever the 
format in our system would be).
 
oh and one technical issue relating reproducibility (although that is 
probably not among the top questions for us yet): as Gentleman notes, 
some research methods use randomness, and to be able to reproduce the 
exact same computations, and a heavy way is to include the random table 
used in the compendium. quick Googling now didn't tell me if the Python 
random generator is guaranteed to give the same results with the same 
seed and in what conditions (i know from working on procedural modelling 
that it at least works on the same computer..), but that can be looked 
at later.

ah and one simple issue: how in our format do we include values of 
variables in a text? Gentleman shows on p. 11 how a result from a 
computation is included in a  sentence using the Sexpr command in their 
system. in Python it would of course be, supposing that 'genes' would be 
the list of selected genes, simply:

".. filtering process selected %d genes" % len(genes)
OR
"selected " + len(genes) + " genes"
OR
"selected", len(genes), "genes" -- if we somehow support the form that 
the print statement handles

, but how in the XML? (i guess it's trivial but just don't know yet).

there may be other issues in the paper that would be important to note, 
but those are what i was left thinking.

~Toni



From antont at an.org  Fri Jul 22 10:05:33 2005
From: antont at an.org (Toni Alatalo)
Date: Fri, 22 Jul 2005 17:05:33 +0300
Subject: [IPython-dev] Re: Changes to Notebook Format
In-Reply-To: <dbgabr$p97$1@sea.gmane.org>
References: <1121681923.3210.18.camel@tzanko> <dbgabr$p97$1@sea.gmane.org>
Message-ID: <42E0FD2D.2080607@an.org>

Robert Kern wrote:

> Tzanko Matev wrote:
>
>> <element id="1">
>> <input type="normal">
>> def foo(x, y):
>>     return x + y
>> </input>
>> </element>
>
> In trying to convert some code to this scheme, I'm finding that it's 
> rather cumbersome. Since the number is attached to the ancestor of 
> <input> and <output>, I can't just get all normal inputs or all 
> special inputs. Nor can I easily tell when a an particular entry also 
> has a special input. For example, 

hm.

> when I want to export the plain code, I used to use a single XPath 
> expression to get all normal inputs and another XPath expression to 
> all special inputs, which I stick in a dictionary keyed by the number. 
> Then I go through the normal inputs and replace them with the special 
> inputs if they are in the dictionary. With the above scheme, I iterate 
> over the <cell> elements (I changed <element> to <cell>), check for 
> the presence of a special input with an XPath expression, use it if 
> found or if not, then I get the normal input with an XPath expression. 
> That's not ideal.

thanks for writing up this info on needed operations, btw.

> Can you describe the operations that you need to do? How often do you 
> have to access this data such that XPath is observably too slow? I 
> think it's probable that this rearrangement may *increase* the number 
> of XPath expressions you need to evaluate.

any info on how this is turning out?

i suppose this is not a case of "using xml to solve a problem? now you 
have two problems.." like the saying goes. yet i wonder, would this be 
different for you if we were talking about Python? (dunno if the 
question really warrants an answer)

as already told in a private message for the mentors, i'm off-line next 
week - back to work on this after that, looking forward to looking into 
these issues then.

~Toni

BTW, if i may suggest a style issue: the nbshell code seems to include 
'getters' that would be naturally properties in current Python (they 
came already in 2.2 so no version prob.), e.g. these in PythonPluginFactory:

   def GetString(self):
        """ Returns the type string of the plugin. This is used when a 
notebook
        file is loaded. See notebookformat.txt for more info"""
        return "python"
   
    def GetType(self):
        """ Returns the way data should be passed to the plugin. Currently
        supported types are "raw" and "encoded". See notebookformat.txt for
        more info"""
        return "raw" #Probably only the python code plugin should be raw

those are currently unnecessary, but could be just .string and .type (or 
perhaps .type and .input_type?). then later if they need to wrapped to 
some method just make them properties ( 
http://www.python.org/2.2/descrintro.html#property ).

in one other project i've tried using properties even for something 
really dynamic, in fact for an exactly similar thing as this in 
ipnDocument.py:

    def IsModified(self):
        """returns if the file has been modified since last save"""
        #TODO: Right now there is no point in doing anything useful
        #here. When things are done fix it
        return True

there am using a property called 'changed', which has such a getter that 
it resets the change flag in the object. so this 'IsModified()' might 
also be better just as a property called 'modified', but that is already 
a bit more controversial issue (as ppl are perhaps used to such dynamic 
things being exposed as methods?)



From jdhunter at ace.bsd.uchicago.edu  Fri Jul 22 10:10:00 2005
From: jdhunter at ace.bsd.uchicago.edu (John Hunter)
Date: Fri, 22 Jul 2005 09:10:00 -0500
Subject: [IPython-dev] Re: Python version
In-Reply-To: <42DD3838.2000409@colorado.edu> (Fernando Perez's message of
	"Tue, 19 Jul 2005 11:28:24 -0600")
References: <1121763694.4785.5.camel@tzanko> <42DD3047.7030501@colorado.edu>
	<dbjcg4$d5j$1@sea.gmane.org> <42DD3838.2000409@colorado.edu>
Message-ID: <87wtnjszif.fsf@peds-pc311.bsd.uchicago.edu>

>>>>> "Fernando" == Fernando Perez <Fernando.Perez at colorado.edu> writes:

    Fernando> Nah, keep 2.4 then.  I'll survive.

When starting new projects, my approach is to use the latest of
everything, even CVS if you want.  Since it will likely be months
before anything is usable, might as well go with the latest-greatest,
since you'll be stuck with it for some time.

JDH



From Fernando.Perez at colorado.edu  Fri Jul 22 14:42:42 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Fri, 22 Jul 2005 12:42:42 -0600
Subject: [IPython-dev] Re: Changes to Notebook Format
In-Reply-To: <42E0FD2D.2080607@an.org>
References: <1121681923.3210.18.camel@tzanko> <dbgabr$p97$1@sea.gmane.org>
	<42E0FD2D.2080607@an.org>
Message-ID: <42E13E22.3030207@colorado.edu>

Toni Alatalo wrote:

> BTW, if i may suggest a style issue: the nbshell code seems to include 
> 'getters' that would be naturally properties in current Python (they 
> came already in 2.2 so no version prob.), e.g. these in PythonPluginFactory:
> 
>    def GetString(self):
>         """ Returns the type string of the plugin. This is used when a 
> notebook
>         file is loaded. See notebookformat.txt for more info"""
>         return "python"
>    
>     def GetType(self):
>         """ Returns the way data should be passed to the plugin. Currently
>         supported types are "raw" and "encoded". See notebookformat.txt for
>         more info"""
>         return "raw" #Probably only the python code plugin should be raw
> 
> those are currently unnecessary, but could be just .string and .type (or 
> perhaps .type and .input_type?). then later if they need to wrapped to 
> some method just make them properties ( 
> http://www.python.org/2.2/descrintro.html#property ).

+1 on properties.  I _hate_ code littered with getX/setX everywhere.  This is 
not Java.  I've been convinced to use XML, but that's about as far as you'll 
get me to go ;)

In fact, make X a regular attribute unless it really requires extra 
validation, synchronization, or work to read/write.  But none of this, please:

def getX(self):
   return _X

If that's all we need, then X can just be read as foo.X, set as foo.X='bar', 
and we're done.  As Guido likes to say, python is a language for adults :)  If 
in the future things evolve to the point where X needs fancier access 
controls, it can be turned into a property, and user code (which still 
respects the validation constraints) will be unaffected.

Cheers,

f



From tsanko at gmail.com  Sat Jul 23 14:44:31 2005
From: tsanko at gmail.com (Tzanko Matev)
Date: Sat, 23 Jul 2005 21:44:31 +0300
Subject: [IPython-dev] Re: Changes to Notebook Format
In-Reply-To: <42E0FD2D.2080607@an.org>
References: <1121681923.3210.18.camel@tzanko> <dbgabr$p97$1@sea.gmane.org>
	<42E0FD2D.2080607@an.org>
Message-ID: <1122144272.4157.55.camel@tzanko>

> Robert Kern wrote:
> 
> > Tzanko Matev wrote:
> >
> >> <element id="1">
> >> <input type="normal">
> >> def foo(x, y):
> >>     return x + y
> >> </input>
> >> </element>
> >
> > In trying to convert some code to this scheme, I'm finding that it's 
> > rather cumbersome. Since the number is attached to the ancestor of 
> > <input> and <output>, I can't just get all normal inputs or all 
> > special inputs. Nor can I easily tell when a an particular entry also 
> > has a special input. For example, 
> 
> hm.
> 
> > when I want to export the plain code, I used to use a single XPath 
> > expression to get all normal inputs and another XPath expression to 
> > all special inputs, which I stick in a dictionary keyed by the number. 
> > Then I go through the normal inputs and replace them with the special 
> > inputs if they are in the dictionary. With the above scheme, I iterate 
> > over the <cell> elements (I changed <element> to <cell>), check for 
> > the presence of a special input with an XPath expression, use it if 
> > found or if not, then I get the normal input with an XPath expression. 
> > That's not ideal.
> 
> thanks for writing up this info on needed operations, btw.
> 
> > Can you describe the operations that you need to do? How often do you 
> > have to access this data such that XPath is observably too slow? I 
> > think it's probable that this rearrangement may *increase* the number 
> > of XPath expressions you need to evaluate.
> 
> any info on how this is turning out?
> 

I attach my response to Robert Kern, and his response. I must have
missed to cc: my letter to the mailing list for which I'm sorry. Basicly
the idea is that often I will need to find all the outputs that a given
input generates and that was slow with the first implementation because
I had to use XPath to do it.


> BTW, if i may suggest a style issue: the nbshell code seems to include 
> 'getters' that would be naturally properties in current Python (they 
> came already in 2.2 so no version prob.), e.g. these in PythonPluginFactory:
> 
>    def GetString(self):
>         """ Returns the type string of the plugin. This is used when a 
> notebook
>         file is loaded. See notebookformat.txt for more info"""
>         return "python"
>    
>     def GetType(self):
>         """ Returns the way data should be passed to the plugin. Currently
>         supported types are "raw" and "encoded". See notebookformat.txt for
>         more info"""
>         return "raw" #Probably only the python code plugin should be raw
> 
> those are currently unnecessary, but could be just .string and .type (or 
> perhaps .type and .input_type?). then later if they need to wrapped to 
> some method just make them properties ( 
> http://www.python.org/2.2/descrintro.html#property ).
> 
> in one other project i've tried using properties even for something 
> really dynamic, in fact for an exactly similar thing as this in 
> ipnDocument.py:
> 
>     def IsModified(self):
>         """returns if the file has been modified since last save"""
>         #TODO: Right now there is no point in doing anything useful
>         #here. When things are done fix it
>         return True
> 
> there am using a property called 'changed', which has such a getter that 
> it resets the change flag in the object. so this 'IsModified()' might 
> also be better just as a property called 'modified', but that is already 
> a bit more controversial issue (as ppl are perhaps used to such dynamic 
> things being exposed as methods?)
> 

Point taken. I've written a lot of C++ code so I have learned some bad
habits from it :) I changed some of the get/set methods to properties or
variables. A lot of my code was written before it was decided to go with
xml so a lot of unnecessary stuff needs to be cleaned up. Right now, I
will leave it as it is, because there are more important things to do.

Best,
tzanko
-------------- next part --------------
An embedded message was scrubbed...
From: Tzanko Matev <tsanko at gmail.com>
Subject: Re: [IPython-dev] Re: Changes to Notebook Format
Date: Mon, 18 Jul 2005 23:22:16 +0300
Size: 3583
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20050723/abe21f8d/attachment.mht>
-------------- next part --------------
An embedded message was scrubbed...
From: Robert Kern <rkern at ucsd.edu>
Subject: Re: [IPython-dev] Re: Changes to Notebook Format
Date: Mon, 18 Jul 2005 13:54:07 -0700
Size: 6023
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20050723/abe21f8d/attachment-0001.mht>

From tsanko at gmail.com  Tue Jul 26 06:40:21 2005
From: tsanko at gmail.com (Tzanko Matev)
Date: Tue, 26 Jul 2005 13:40:21 +0300
Subject: [IPython-dev] Question about IPython behaviour
Message-ID: <1122374421.4682.13.camel@tzanko>

Hi all,

The prefilter() method turns strings consisting only of whitespace to
empty strings. This means that in IPython I cannot write the following
code:

In[1]: if 1==1:
   ... ___print 1
   ... ___   
   ... ___print2
   ...

(here I have replaced spaces with underscore). prefilter() will turn the
third line into an empty string and the interpreter will decide that the
block ends there. This differs from the behaviour of the standard
interpreter where the code would run as expected.  So is this a bug or a
feature? I think that empty lines should be allowed in the GUI.

Best,
tzanko



From tsanko at gmail.com  Tue Jul 26 08:50:44 2005
From: tsanko at gmail.com (Tzanko Matev)
Date: Tue, 26 Jul 2005 15:50:44 +0300
Subject: [IPython-dev] Re: Changes to Notebook Format
In-Reply-To: <1121681923.3210.18.camel@tzanko>
References: <1121681923.3210.18.camel@tzanko>
Message-ID: <1122382245.4682.53.camel@tzanko>

Hi all,

I found other problems with the current format. It is entirely possible
that a single input has several outputs, and prints to stdout and/or
stderr several times. Take for examlpe the following code. 

In [1]: if True:
   ...:     1
   ...:     print 'foo'
   ...:     2
   ...:     print 'bar'
   ...:
Out[1]: 1
foo
Out[1]: 2
bar

To fix that we should allow several elements of types <output>, <stdout>
and <stderr> to be included in a cell. Also we want to keep the order in
which they were produced. I can see two ways to do this. 

One way is to add an 'order' attribute to all the <output>, <stdout> and
<stderr> elements in the cell. Also in the <ipython-block> element all
the <ipython-output>, <ipython-stdout> and <ipython-stderr> elements
should have this attribute. Then we won't need separate tags in
<ipython-block> for all types of  output, because the 'number' and
'order' attributes will be enough to decide which output element should
be displayed at the given position.

Another way is to use the order in which the elements in the cell are
written. We can state that the first one or two elements must be of type
<input> and <special-input>. The elements after them will be of types
<output>, <stdout>, <stderr>, <figure>, etc. and the order in which they
were produced by the input corresponds to the order in which they are
written in the cell. Then in <ipython-block> we can get rid of all the
different tags and replace them with tags like <element number='1'
cellnumber='2'/>  which will point to the second element in the first
cell in the log.

Right now, I'm not really sure which one is better. The second way will
probably be faster to run, because I will be able to work with a cell
like a list and not use the find() method or XPath to retrieve elements.
What do you think?

Best,
tzanko



From rkern at ucsd.edu  Tue Jul 26 13:41:44 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Tue, 26 Jul 2005 10:41:44 -0700
Subject: [IPython-dev] Re: Question about IPython behaviour
In-Reply-To: <1122374421.4682.13.camel@tzanko>
References: <1122374421.4682.13.camel@tzanko>
Message-ID: <dc5sks$d58$1@sea.gmane.org>

Tzanko Matev wrote:
> Hi all,
> 
> The prefilter() method turns strings consisting only of whitespace to
> empty strings. This means that in IPython I cannot write the following
> code:
> 
> In[1]: if 1==1:
>    ... ___print 1
>    ... ___   
>    ... ___print2
>    ...
> 
> (here I have replaced spaces with underscore). prefilter() will turn the
> third line into an empty string and the interpreter will decide that the
> block ends there. This differs from the behaviour of the standard
> interpreter where the code would run as expected.  So is this a bug or a
> feature? I think that empty lines should be allowed in the GUI.

So code to a different interface like the one in notabene.executor. The 
GUI decides when the input is complete, sends it whole to the 
interpreter, and gets a response back.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From rkern at ucsd.edu  Tue Jul 26 13:55:22 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Tue, 26 Jul 2005 10:55:22 -0700
Subject: [IPython-dev] Re: Changes to Notebook Format
In-Reply-To: <1122382245.4682.53.camel@tzanko>
References: <1121681923.3210.18.camel@tzanko> <1122382245.4682.53.camel@tzanko>
Message-ID: <dc5tee$ftn$1@sea.gmane.org>

Tzanko Matev wrote:
> Hi all,
> 
> I found other problems with the current format. It is entirely possible
> that a single input has several outputs, and prints to stdout and/or
> stderr several times. Take for examlpe the following code. 
> 
> In [1]: if True:
>    ...:     1
>    ...:     print 'foo'
>    ...:     2
>    ...:     print 'bar'
>    ...:
> Out[1]: 1
> foo
> Out[1]: 2
> bar

I've never run into a case where I actually want to see multiple 
Out[1]'s. It's usually in the middle of a for loop and I've forgotten 
that a function returns an object that I don't really need to see (e.g. 
pylab.plot()). Since only the last one actually goes into the output 
cache (that is, is accessible for later use via Out[1]), I'm not 
convinced that printing multiple Out[1]'s is useful. If I really want to 
see something in the middle of a loop, I'll print it.

As for having stdout cells interleaved with other types of output cells, 
I think that's going to be pretty hard. You'd have to collect the output 
from each statement separately. I'm of the opinion that you should just 
collect one each of stdout and stderr.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From Fernando.Perez at colorado.edu  Wed Jul 27 01:44:40 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Tue, 26 Jul 2005 23:44:40 -0600
Subject: [IPython-dev] Re: Changes to Notebook Format
In-Reply-To: <dc5tee$ftn$1@sea.gmane.org>
References: <1121681923.3210.18.camel@tzanko>
	<1122382245.4682.53.camel@tzanko> <dc5tee$ftn$1@sea.gmane.org>
Message-ID: <42E71F48.6020205@colorado.edu>

Robert Kern wrote:
> Tzanko Matev wrote:
> 
>>Hi all,
>>
>>I found other problems with the current format. It is entirely possible
>>that a single input has several outputs, and prints to stdout and/or
>>stderr several times. Take for examlpe the following code. 
>>
>>In [1]: if True:
>>   ...:     1
>>   ...:     print 'foo'
>>   ...:     2
>>   ...:     print 'bar'
>>   ...:
>>Out[1]: 1
>>foo
>>Out[1]: 2
>>bar
> 
> 
> I've never run into a case where I actually want to see multiple 
> Out[1]'s. It's usually in the middle of a for loop and I've forgotten 
> that a function returns an object that I don't really need to see (e.g. 
> pylab.plot()). Since only the last one actually goes into the output 
> cache (that is, is accessible for later use via Out[1]), I'm not 
> convinced that printing multiple Out[1]'s is useful. If I really want to 
> see something in the middle of a loop, I'll print it.

Here's an even worse example:

In [3]: for i in range(3):
    ...:     print 'passing...'
    ...:     i
    ...:
passing...
Out[3]: 0
passing...
Out[3]: 1
passing...
Out[3]: 2

The multiple Out[] behavior is really mostly an implementation accident, not a 
design feature.  I'd be willing to consider it a bug, in the sense that naked 
expressions in compound statements should not really be treated as return 
valuues.  I think the example above should simply print 'passing...' three 
times, and NOT generate an Out[] cell at all.

> As for having stdout cells interleaved with other types of output cells, 
> I think that's going to be pretty hard. You'd have to collect the output 
> from each statement separately. I'm of the opinion that you should just 
> collect one each of stdout and stderr.

Yup.

Cheers,

f



From Fernando.Perez at colorado.edu  Wed Jul 27 01:48:37 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Tue, 26 Jul 2005 23:48:37 -0600
Subject: [IPython-dev] Re: Question about IPython behaviour
In-Reply-To: <dc5sks$d58$1@sea.gmane.org>
References: <1122374421.4682.13.camel@tzanko> <dc5sks$d58$1@sea.gmane.org>
Message-ID: <42E72035.20309@colorado.edu>

Robert Kern wrote:
> Tzanko Matev wrote:
> 
>>Hi all,
>>
>>The prefilter() method turns strings consisting only of whitespace to
>>empty strings. This means that in IPython I cannot write the following
>>code:
>>
>>In[1]: if 1==1:
>>   ... ___print 1
>>   ... ___   
>>   ... ___print2
>>   ...
>>
>>(here I have replaced spaces with underscore). prefilter() will turn the
>>third line into an empty string and the interpreter will decide that the
>>block ends there. This differs from the behaviour of the standard
>>interpreter where the code would run as expected.  So is this a bug or a
>>feature? I think that empty lines should be allowed in the GUI.
> 
> 
> So code to a different interface like the one in notabene.executor. The 
> GUI decides when the input is complete, sends it whole to the 
> interpreter, and gets a response back.

I second Robert's suggestion.  The reason for ipython's behavior is partly to 
mimic the default '>>>' interpreter, partly because it's tricky to do 
otherwise in a line-oriented environment.  The default python shell simply 
uses the convention 'a single empty line signals the end of input'.  IPython 
inherits from code.InteractiveConsole, which maintains this behavior.

While not ideal, I think that's not terrible for a simple line-oriented system 
which doesn't really offer any serious multiline editing.  But now that we are 
migrating to a GUI, we should most certainly offer a way to have blank lines.

Cheers,

f



From rkern at ucsd.edu  Wed Jul 27 02:04:20 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Tue, 26 Jul 2005 23:04:20 -0700
Subject: [IPython-dev] Re: Question about IPython behaviour
In-Reply-To: <42E72035.20309@colorado.edu>
References: <1122374421.4682.13.camel@tzanko> <dc5sks$d58$1@sea.gmane.org>
	<42E72035.20309@colorado.edu>
Message-ID: <dc7858$g09$1@sea.gmane.org>

Fernando Perez wrote:

> While not ideal, I think that's not terrible for a simple line-oriented 
> system which doesn't really offer any serious multiline editing.  But 
> now that we are migrating to a GUI, we should most certainly offer a way 
> to have blank lines.

We should take a look at the design of pyrepl_. It has real multi-line 
editing and probably a design more suited to our endeavors here. We 
should stea^H^H^H^Hbe inspired by how it solves these problems.

.. _pyrepl: http://codespeak.net/pyrepl/

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From Fernando.Perez at colorado.edu  Wed Jul 27 02:21:10 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Wed, 27 Jul 2005 00:21:10 -0600
Subject: [IPython-dev] Re: Question about IPython behaviour
In-Reply-To: <dc7858$g09$1@sea.gmane.org>
References: <1122374421.4682.13.camel@tzanko> <dc5sks$d58$1@sea.gmane.org>
	<42E72035.20309@colorado.edu> <dc7858$g09$1@sea.gmane.org>
Message-ID: <42E727D6.2020603@colorado.edu>

Robert Kern wrote:
> Fernando Perez wrote:
> 
> 
>>While not ideal, I think that's not terrible for a simple line-oriented 
>>system which doesn't really offer any serious multiline editing.  But 
>>now that we are migrating to a GUI, we should most certainly offer a way 
>>to have blank lines.
> 
> 
> We should take a look at the design of pyrepl_. It has real multi-line 
> editing and probably a design more suited to our endeavors here. We 
> should stea^H^H^H^Hbe inspired by how it solves these problems.
> 
> .. _pyrepl: http://codespeak.net/pyrepl/

I talked to M. Hudson (pyrepl's author) ages ago about this: pyrepl wasn't 
suitable for ipython because it's curses-based.  That's how they get 
multi-line capabilities.  It is certainly a possibility to build a 
curses-enabled version of ipython which could run on top of pyrepl.

But I think the decision to keep ipython line-oriented was a good one: curses 
is *nix only, so had I gone with pyrepl early on, we would have lost all the 
win32 users.  And I know that many people in the win32 world do make heavy use 
of ipython, so I don't want to break things for them.

An _optional_ pyrepl-enhanced ipython would be great, and we certainly can 
look into how they tackle a number of issues.  I just wanted to fill you in a 
bit on the history behind ipython/pyrepl.

Cheers,

f



From rkern at ucsd.edu  Wed Jul 27 02:49:23 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Tue, 26 Jul 2005 23:49:23 -0700
Subject: [IPython-dev] Re: Question about IPython behaviour
In-Reply-To: <42E727D6.2020603@colorado.edu>
References: <1122374421.4682.13.camel@tzanko> <dc5sks$d58$1@sea.gmane.org>
	<42E72035.20309@colorado.edu> <dc7858$g09$1@sea.gmane.org>
	<42E727D6.2020603@colorado.edu>
Message-ID: <dc7apn$ljd$1@sea.gmane.org>

Fernando Perez wrote:

> I talked to M. Hudson (pyrepl's author) ages ago about this: pyrepl 
> wasn't suitable for ipython because it's curses-based.  That's how they 
> get multi-line capabilities.  It is certainly a possibility to build a 
> curses-enabled version of ipython which could run on top of pyrepl.

I'm not so interested in that part of pyrepl. I haven't delved through 
the code enough, but it does do multiline editing pretty much as we want 
to do in the GUI. To enable that, the component that executes the inputs 
probably doesn't use the code.InteractiveConsole line-based API. 
*That's* the part we should steal (if indeed, it does exist and I'm not 
just thinking wishfully). The curses bit is fluff that is obviated by 
going GUI.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From Fernando.Perez at colorado.edu  Wed Jul 27 12:50:55 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Wed, 27 Jul 2005 10:50:55 -0600
Subject: [IPython-dev] Re: Question about IPython behaviour
In-Reply-To: <dc7apn$ljd$1@sea.gmane.org>
References: <1122374421.4682.13.camel@tzanko> <dc5sks$d58$1@sea.gmane.org>
	<42E72035.20309@colorado.edu> <dc7858$g09$1@sea.gmane.org>
	<42E727D6.2020603@colorado.edu> <dc7apn$ljd$1@sea.gmane.org>
Message-ID: <42E7BB6F.7090500@colorado.edu>

Robert Kern wrote:
> Fernando Perez wrote:
> 
> 
>>I talked to M. Hudson (pyrepl's author) ages ago about this: pyrepl 
>>wasn't suitable for ipython because it's curses-based.  That's how they 
>>get multi-line capabilities.  It is certainly a possibility to build a 
>>curses-enabled version of ipython which could run on top of pyrepl.
> 
> 
> I'm not so interested in that part of pyrepl. I haven't delved through 
> the code enough, but it does do multiline editing pretty much as we want 
> to do in the GUI. To enable that, the component that executes the inputs 
> probably doesn't use the code.InteractiveConsole line-based API. 
> *That's* the part we should steal (if indeed, it does exist and I'm not 
> just thinking wishfully). The curses bit is fluff that is obviated by 
> going GUI.

I was just trying to fill the history in vs ipython/pyrepl.  We should 
certainly see if they have that part built in a reasonably curses-independent 
way which we can reuse/steal.

Cheers,

f



From antont at an.org  Fri Jul 29 10:05:16 2005
From: antont at an.org (Toni Alatalo)
Date: Fri, 29 Jul 2005 17:05:16 +0300
Subject: [IPython-dev] small interim info
In-Reply-To: <Pine.LNX.4.44.0507281513210.29737-100000@pallas.utanet.fi>
References: <Pine.LNX.4.44.0507281513210.29737-100000@pallas.utanet.fi>
Message-ID: <42EA379C.4030900@an.org>

Toni Alatalo wrote:

(got married last weekend :) and had a little holiday after that, 
discussed related things in private with mentors, but bringing the 
discussion to public now that got back to work - first just to tell a 
little where am heading)

>read the messages to get myself back on the map, but am not all clear yet 
>what to tackle next. probably can get going by trying out Tzanko's app and 
>studying it again to see how it uses Robert's backend, and finding out 
>what is missing/should be improved there.
>  
>
did that today - most of the time unfortunately went into fighting with 
getting a right version of wxwidgets & wxpython working for the right 
python version (had to modify the 2.6 in debian experimental to compile 
for 2.4, as default is still 2.3, and then had a strange prob. with 
imports but all those are finally solved now)

studied how nbshell ipnDocument uses notabene.notebook, and know from 
before most of how notabene 'writers' work. unfortunately couldn't 
experiment with using the 'live in the GUI ' notabene.notebook object 
interactively from the interpreter, i wonder if there's a way? i mean to 
do something like this:

antont at vesuri:~/development/ipython/nbshell/trunk/nbshell$ python2.4
Python 2.4.1 (#2, May  5 2005, 11:32:06)
[GCC 3.3.5 (Debian 1:3.3.5-12)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
 >>> import Main
 >>> Main.start()

(type things in the editor)

and then:

Main.app.notebook.my_experimental_notebook_method()

?

currently if i e.g. quit the wx app, the prompt returns ok, but the 
interpreter namespace is in a pretty strange state:

 >>> Main
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
NameError: name 'Main' is not defined
 >>> import Main
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
ImportError: __import__ not found

that is (again) pretty irrelevant, but would be just a nice way to try 
new things with the document objects without adding buttons to the GUI, 
but still being able to edit the actual docs in the GUI. anyhow testing 
the notebook api without the GUI interactively  of course works 
normally, and  saving test files from the GUI seems to work, so no 
showstoppers there.

>but if there are suggestions / TODOs for what i should focus on, starting 
>latest on Monday, please do tell.
>  
>
unless adviced otherwise, will continue with the the document output 
(e.g. latex) components.

>~Toni
>  
>
same.



From tsanko at gmail.com  Sun Jul 31 07:06:21 2005
From: tsanko at gmail.com (Tzanko Matev)
Date: Sun, 31 Jul 2005 14:06:21 +0300
Subject: [IPython-dev] NBShell update
Message-ID: <1122807981.4724.34.camel@tzanko>

Hi all,

The Notebook GUI can already be usable. Currently you can mainly use it
as a simple python shell. It lacks history, autocompletion, etc. Its
only advantage is better multiline support. You can edit all the lines
of your last input until it is complete. Also I have added a variable
__app in the namespace which allows you to play with the internals of
NBShell. The svn repository is

    http://ipython.scipy.org/svn/ipython/nbshell/trunk

The instalation instructions are in the README file. I would appreciate
it if you played with it and tell me what bugs you find and what
features you would like me to add.

Best,
tzanko