[IPython-dev] ipython1 and synchronous printing of stdout

[Brian Granger]

> As always, I think working code is better than good theory, so please
> read the following with that in mind. I think that the opinion you
> express below may put us on the road to an architectural mistake in
> the redesign of ipython. My understanding of the historical context is
> that the current ipython0 interpreter is direclty tied to the frontend
> in a way that makes new frontends or stand-alone interpreters
> difficult. The solution, as embodied by the the ipython1 codebase and
> the evolving IPython.frontend package is to establish a separation
> between interpreter and frontend to prevent the type of dependency
> that exists in the current code base.

One thing that is absolutely critical is loose coupling.  All the
parts of IPython need to depend only very weakly on each other, and
interact through clean, standardized interfaces (I don't mean this
word in the script zope.interface sense though).  The big problem with
the traditional IPython codebase is that everything is welded
together.  The question is what types of design patterns will make
this possible.

>From this thread, I see a few options:

1.  Traits.  While I really like the Traits model and I think it is
highly appropriate for GUI programming, I think it would lead to too
tightly coupled code.  Also, we can't use Traits in the core of
ipython because it has compiled code.

2.  Plain callbacks.  At least a couple of us seem to think that an
interface that has lots of callbacks becomes very cumbersome.

3.  The Observer/Delegate patterns as are typical is Cocoa.  I know
that Gael is not fund of this, but I do think this pattern is much
better than plain callbacks, as it give a nice formal structure to
everything.  Also, I agree with Barry that this pattern is pretty much
what you get when you try to do plain callbacks properly.

Thus, my own conclusion is that the Observer/Delegate pattern is
probably the best option for us and will lead to the loose coupling
that we need.

Barry, what would it take to put together an example of this pattern?
Will we need some amount of infrastructure to make this possible?

Cheers,

Brian

> Now, back to the issue at hand. In developing frontends using the new
> architecture, you have noticed that there are a lot of things that the
> new system is missing. In general, these fall into the category of the
> frontend wanting to monitor and/or modify execution in the interpreter
> at a more fine-grained level than entire code blocks (the only option
> currently). There is a standard pattern for handling this type of
> notification, the Observer pattern, and this type of fine-grained
> modification, the Delegate pattern. Propper implementation of either
> produces minimal overhead (and it's worth remembering that by writing
> IPython in python instead of a lower level language, we've all already
> stated our preference for abstraction over raw efficiency) [1]. The
> alternative, that I believe you are proposing below and in the current
> thread on adding a callback to the exception handling code in the
> interpreter, is to add callback hooks to the interpreter so that one
> (or more?) callbacks can be attached to specific events in the
> interpreter's execution flow.
>
> In fact, I think these are not really different solutions, just
> different implementations of a similar idea. You propose handling
> observer notification from within the shell, whereas I propose
> handling it in an intermediary entity (the notification center). The
> callback approach is more explicit -- the interpreter must explicitly
> expose a callback hook and the frontend must explicitly register *with
> the interpreter* for that callback -- but less general. Unless we
> create a callback registry etc. in the shell, essentially duplicating
> the notification center idea, then when we want to add a new
> notification (e.g. for exceptions as in the currently running thread),
> we will need to duplicate the callback calling code in the shell
> rather than just marking that event for notification by the
> notification center. The observer/delegate combination allows the
> interpreter to remain agnostic wrt the interface of the frontend and
> vice-versa (and huge win, in my opinion). By adding an intermediary --
> the notification center and/or delegate dispatcher -- we avoid the
> issue of frontends and interpreters being closely tied by contract and
> make it easier to provide the same interface directly (between
> frontend and shell as in your Wx frontend) or locally or remotely via
> a twisted service (as between and engine and the frontend in my Cocoa
> frontend).
>
> I know that you're on a deadline and need to write code, but I would
> urge a little bit of restraint and planning before you commit to the
> callback-only approach. I think avoiding this type of dependency
> between shell and frontend will benefit us in the long run.
>
> Barry
>
> [1] For example, firing a notification when no observers are
> registered for that notification should not incur significant overhead
> beyond a couple of method calls, and perhaps a dictionary lookup, even
> in a naive implementation.
>
> On Sun, Jul 20, 2008 at 9:50 PM, Gael Varoquaux
> <gael.varoquaux at normalesup.org> wrote:
>> On Sun, Jul 20, 2008 at 09:36:43PM -0700, Barry Wark wrote:
>>> There are, I think, several "events" for which a frontend might want a
>>> callback. Opening files,
>>
>> Hum, in what sens? What would trigger this event?
>>
>>> stdout/stderr output,
>>
>> Sure, this is why I came up with SyncOutputTrap, that allows me to pass a
>> callback to write. I add a wx.Yield() in this calback and my UI can
>> refresh each time you write something to stdout/stdin.
>>
>>> and modifications to the user_ns are some that come immediately to
>>> mind. Opening files might be of interest if the frontend wants to
>>> notify the user that there is file output to be retrieved from the
>>> engine, stdout/stderr for the resason Gael outlines, and modifications
>>> to the user_ns to allow the frontend to keep an up-to-date view of the
>>> user_ns (ala Matlab's Workspace browser) without querying the the
>>> entire engine user_ns after every executed block.
>>
>> Hum, I am not too excited about firing events when the namespace gets
>> updated. This could slow things down a lot. I general I am not too
>> excited about having events fired all over the place. This can lead to
>> tedious code with getters and setters and a udge performance loss. I am
>> much more interested in using objects with a standard interface (such as
>> a file, for instance) and then, if you want getters and setters, or
>> the possibility to add callabcks, you do it in these objects.
>>
>>> Apple's NSDistributedNotificationCenter API
>>> (http://developer.apple.com/documentation/Cocoa/Reference/Foundation/Classes/NSDistributedNotificationCenter_Class/Reference/Reference.html)
>>> seems like a good pattern for this task (I'm sure there are equivalent
>>> APIs in other frameworks). Basically, the frontend can register for
>>> notifications by name. The interpreter then fires notifications for all
>>> events and they are passed on, by the notification center, to all
>>> registered observers.
>>
>> Well, I don't really like this kind of code. I find it clunky, and you
>> end up have notification registrations all over the place (reminds me of
>> wx). I actually prefer a lot the traits model, or the VTK model, AFAIK,
>> where objects can be there own observers. As a result you do not have
>> notifications blending in objects where they do not belong. For the core
>> frontends, obviously, we do not want to add a dependence to traits, but I
>> must imagine I cannot imagine going much further without it. To take your
>> example of updating the namespace view, all you have to do is to pass a
>> traited dict, instead of a dict, to the interpreter as a namespace. Then
>> you can have your namespace update automatically (AFAIK Robert is
>> actually working on a branch where he does that).
>>
>> If we start adding callbacks whereever we need them, I am worried we'll
>> end up with callbacks everywhere. I think we need to add callback only
>> where they are absolutely necessary.
>> My two cents,
>>
>> Gaël
>>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev
>