[IPython-dev] ipython1 and synchronous printing of stdout

[Barry Wark]

As always, I think working code is better than good theory, so please
read the following with that in mind. I think that the opinion you
express below may put us on the road to an architectural mistake in
the redesign of ipython. My understanding of the historical context is
that the current ipython0 interpreter is direclty tied to the frontend
in a way that makes new frontends or stand-alone interpreters
difficult. The solution, as embodied by the the ipython1 codebase and
the evolving IPython.frontend package is to establish a separation
between interpreter and frontend to prevent the type of dependency
that exists in the current code base.

Now, back to the issue at hand. In developing frontends using the new
architecture, you have noticed that there are a lot of things that the
new system is missing. In general, these fall into the category of the
frontend wanting to monitor and/or modify execution in the interpreter
at a more fine-grained level than entire code blocks (the only option
currently). There is a standard pattern for handling this type of
notification, the Observer pattern, and this type of fine-grained
modification, the Delegate pattern. Propper implementation of either
produces minimal overhead (and it's worth remembering that by writing
IPython in python instead of a lower level language, we've all already
stated our preference for abstraction over raw efficiency) [1]. The
alternative, that I believe you are proposing below and in the current
thread on adding a callback to the exception handling code in the
interpreter, is to add callback hooks to the interpreter so that one
(or more?) callbacks can be attached to specific events in the
interpreter's execution flow.

In fact, I think these are not really different solutions, just
different implementations of a similar idea. You propose handling
observer notification from within the shell, whereas I propose
handling it in an intermediary entity (the notification center). The
callback approach is more explicit -- the interpreter must explicitly
expose a callback hook and the frontend must explicitly register *with
the interpreter* for that callback -- but less general. Unless we
create a callback registry etc. in the shell, essentially duplicating
the notification center idea, then when we want to add a new
notification (e.g. for exceptions as in the currently running thread),
we will need to duplicate the callback calling code in the shell
rather than just marking that event for notification by the
notification center. The observer/delegate combination allows the
interpreter to remain agnostic wrt the interface of the frontend and
vice-versa (and huge win, in my opinion). By adding an intermediary --
the notification center and/or delegate dispatcher -- we avoid the
issue of frontends and interpreters being closely tied by contract and
make it easier to provide the same interface directly (between
frontend and shell as in your Wx frontend) or locally or remotely via
a twisted service (as between and engine and the frontend in my Cocoa
frontend).

I know that you're on a deadline and need to write code, but I would
urge a little bit of restraint and planning before you commit to the
callback-only approach. I think avoiding this type of dependency
between shell and frontend will benefit us in the long run.

Barry

[1] For example, firing a notification when no observers are
registered for that notification should not incur significant overhead
beyond a couple of method calls, and perhaps a dictionary lookup, even
in a naive implementation.

On Sun, Jul 20, 2008 at 9:50 PM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> On Sun, Jul 20, 2008 at 09:36:43PM -0700, Barry Wark wrote:
>> There are, I think, several "events" for which a frontend might want a
>> callback. Opening files,
>
> Hum, in what sens? What would trigger this event?
>
>> stdout/stderr output,
>
> Sure, this is why I came up with SyncOutputTrap, that allows me to pass a
> callback to write. I add a wx.Yield() in this calback and my UI can
> refresh each time you write something to stdout/stdin.
>
>> and modifications to the user_ns are some that come immediately to
>> mind. Opening files might be of interest if the frontend wants to
>> notify the user that there is file output to be retrieved from the
>> engine, stdout/stderr for the resason Gael outlines, and modifications
>> to the user_ns to allow the frontend to keep an up-to-date view of the
>> user_ns (ala Matlab's Workspace browser) without querying the the
>> entire engine user_ns after every executed block.
>
> Hum, I am not too excited about firing events when the namespace gets
> updated. This could slow things down a lot. I general I am not too
> excited about having events fired all over the place. This can lead to
> tedious code with getters and setters and a udge performance loss. I am
> much more interested in using objects with a standard interface (such as
> a file, for instance) and then, if you want getters and setters, or
> the possibility to add callabcks, you do it in these objects.
>
>> Apple's NSDistributedNotificationCenter API
>> (http://developer.apple.com/documentation/Cocoa/Reference/Foundation/Classes/NSDistributedNotificationCenter_Class/Reference/Reference.html)
>> seems like a good pattern for this task (I'm sure there are equivalent
>> APIs in other frameworks). Basically, the frontend can register for
>> notifications by name. The interpreter then fires notifications for all
>> events and they are passed on, by the notification center, to all
>> registered observers.
>
> Well, I don't really like this kind of code. I find it clunky, and you
> end up have notification registrations all over the place (reminds me of
> wx). I actually prefer a lot the traits model, or the VTK model, AFAIK,
> where objects can be there own observers. As a result you do not have
> notifications blending in objects where they do not belong. For the core
> frontends, obviously, we do not want to add a dependence to traits, but I
> must imagine I cannot imagine going much further without it. To take your
> example of updating the namespace view, all you have to do is to pass a
> traited dict, instead of a dict, to the interpreter as a namespace. Then
> you can have your namespace update automatically (AFAIK Robert is
> actually working on a branch where he does that).
>
> If we start adding callbacks whereever we need them, I am worried we'll
> end up with callbacks everywhere. I think we need to add callback only
> where they are absolutely necessary.
> My two cents,
>
> Gaël
>