[IPython-dev] ipython1 and synchronous printing of stdout
barrywark at gmail.com
Wed Jul 23 12:17:46 EDT 2008
On Jul 22, 2008, at 3:25 PM, "Brian Granger" <ellisonbg.net at gmail.com>
>> As always, I think working code is better than good theory, so please
>> read the following with that in mind. I think that the opinion you
>> express below may put us on the road to an architectural mistake in
>> the redesign of ipython. My understanding of the historical context
>> that the current ipython0 interpreter is direclty tied to the
>> in a way that makes new frontends or stand-alone interpreters
>> difficult. The solution, as embodied by the the ipython1 codebase and
>> the evolving IPython.frontend package is to establish a separation
>> between interpreter and frontend to prevent the type of dependency
>> that exists in the current code base.
> One thing that is absolutely critical is loose coupling. All the
> parts of IPython need to depend only very weakly on each other, and
> interact through clean, standardized interfaces (I don't mean this
> word in the script zope.interface sense though). The big problem with
> the traditional IPython codebase is that everything is welded
> together. The question is what types of design patterns will make
> this possible.
> From this thread, I see a few options:
> 1. Traits. While I really like the Traits model and I think it is
> highly appropriate for GUI programming, I think it would lead to too
> tightly coupled code. Also, we can't use Traits in the core of
> ipython because it has compiled code.
> 2. Plain callbacks. At least a couple of us seem to think that an
> interface that has lots of callbacks becomes very cumbersome.
> 3. The Observer/Delegate patterns as are typical is Cocoa. I know
> that Gael is not fund of this, but I do think this pattern is much
> better than plain callbacks, as it give a nice formal structure to
> everything. Also, I agree with Barry that this pattern is pretty much
> what you get when you try to do plain callbacks properly.
> Thus, my own conclusion is that the Observer/Delegate pattern is
> probably the best option for us and will lead to the loose coupling
> that we need.
> Barry, what would it take to put together an example of this pattern?
> Will we need some amount of infrastructure to make this possible?
I'd be happy to write up an example and a draft implementation. I'm
going to be mostly offline, traveling, until the weekend. I'm most
familiar with the Cocoa implementation so I'd like to do a bit of
research of alternatives before I write the spec. Therefore next week
is the realistic timeline.
Off the top of my head, I think a synchronous notification center
implementation in I.k.core and an addition to the engineservice API
along with an asynchronous notification center impl in I.kernel would
cover the use cases. I will write up a functional/technical spec and
solicit feedback before I make any decisions though.
Besides the notificatipn center implementation(s), I don't think any
more infrastructure is required beyond modifying the interpreter to
actually fire the appropriate notifications (support for obvious
callbacks such as Gael's use cases should be easy; the notifying
worskspace ala Robert's work will obviously require more effort). As
you pointed out, this is really just carefully implemented callbacks
with a loose coupling.
I'm glad we're dealing with this issue early. I think getting it right
will pay big dividends in the future.
>> Now, back to the issue at hand. In developing frontends using the new
>> architecture, you have noticed that there are a lot of things that
>> new system is missing. In general, these fall into the category of
>> frontend wanting to monitor and/or modify execution in the
>> at a more fine-grained level than entire code blocks (the only option
>> currently). There is a standard pattern for handling this type of
>> notification, the Observer pattern, and this type of fine-grained
>> modification, the Delegate pattern. Propper implementation of either
>> produces minimal overhead (and it's worth remembering that by writing
>> IPython in python instead of a lower level language, we've all
>> stated our preference for abstraction over raw efficiency) . The
>> alternative, that I believe you are proposing below and in the
>> thread on adding a callback to the exception handling code in the
>> interpreter, is to add callback hooks to the interpreter so that one
>> (or more?) callbacks can be attached to specific events in the
>> interpreter's execution flow.
>> In fact, I think these are not really different solutions, just
>> different implementations of a similar idea. You propose handling
>> observer notification from within the shell, whereas I propose
>> handling it in an intermediary entity (the notification center). The
>> callback approach is more explicit -- the interpreter must explicitly
>> expose a callback hook and the frontend must explicitly register
>> the interpreter* for that callback -- but less general. Unless we
>> create a callback registry etc. in the shell, essentially duplicating
>> the notification center idea, then when we want to add a new
>> notification (e.g. for exceptions as in the currently running
>> we will need to duplicate the callback calling code in the shell
>> rather than just marking that event for notification by the
>> notification center. The observer/delegate combination allows the
>> interpreter to remain agnostic wrt the interface of the frontend and
>> vice-versa (and huge win, in my opinion). By adding an intermediary
>> the notification center and/or delegate dispatcher -- we avoid the
>> issue of frontends and interpreters being closely tied by contract
>> make it easier to provide the same interface directly (between
>> frontend and shell as in your Wx frontend) or locally or remotely via
>> a twisted service (as between and engine and the frontend in my Cocoa
>> I know that you're on a deadline and need to write code, but I would
>> urge a little bit of restraint and planning before you commit to the
>> callback-only approach. I think avoiding this type of dependency
>> between shell and frontend will benefit us in the long run.
>>  For example, firing a notification when no observers are
>> registered for that notification should not incur significant
>> beyond a couple of method calls, and perhaps a dictionary lookup,
>> in a naive implementation.
>> On Sun, Jul 20, 2008 at 9:50 PM, Gael Varoquaux
>> <gael.varoquaux at normalesup.org> wrote:
>>> On Sun, Jul 20, 2008 at 09:36:43PM -0700, Barry Wark wrote:
>>>> There are, I think, several "events" for which a frontend might
>>>> want a
>>>> callback. Opening files,
>>> Hum, in what sens? What would trigger this event?
>>>> stdout/stderr output,
>>> Sure, this is why I came up with SyncOutputTrap, that allows me to
>>> pass a
>>> callback to write. I add a wx.Yield() in this calback and my UI can
>>> refresh each time you write something to stdout/stdin.
>>>> and modifications to the user_ns are some that come immediately to
>>>> mind. Opening files might be of interest if the frontend wants to
>>>> notify the user that there is file output to be retrieved from the
>>>> engine, stdout/stderr for the resason Gael outlines, and
>>>> to the user_ns to allow the frontend to keep an up-to-date view
>>>> of the
>>>> user_ns (ala Matlab's Workspace browser) without querying the the
>>>> entire engine user_ns after every executed block.
>>> Hum, I am not too excited about firing events when the namespace
>>> updated. This could slow things down a lot. I general I am not too
>>> excited about having events fired all over the place. This can
>>> lead to
>>> tedious code with getters and setters and a udge performance loss.
>>> I am
>>> much more interested in using objects with a standard interface
>>> (such as
>>> a file, for instance) and then, if you want getters and setters, or
>>> the possibility to add callabcks, you do it in these objects.
>>>> Apple's NSDistributedNotificationCenter API
>>>> seems like a good pattern for this task (I'm sure there are
>>>> APIs in other frameworks). Basically, the frontend can register for
>>>> notifications by name. The interpreter then fires notifications
>>>> for all
>>>> events and they are passed on, by the notification center, to all
>>>> registered observers.
>>> Well, I don't really like this kind of code. I find it clunky, and
>>> end up have notification registrations all over the place (reminds
>>> me of
>>> wx). I actually prefer a lot the traits model, or the VTK model,
>>> where objects can be there own observers. As a result you do not
>>> notifications blending in objects where they do not belong. For
>>> the core
>>> frontends, obviously, we do not want to add a dependence to
>>> traits, but I
>>> must imagine I cannot imagine going much further without it. To
>>> take your
>>> example of updating the namespace view, all you have to do is to
>>> pass a
>>> traited dict, instead of a dict, to the interpreter as a
>>> namespace. Then
>>> you can have your namespace update automatically (AFAIK Robert is
>>> actually working on a branch where he does that).
>>> If we start adding callbacks whereever we need them, I am worried
>>> end up with callbacks everywhere. I think we need to add callback
>>> where they are absolutely necessary.
>>> My two cents,
>> IPython-dev mailing list
>> IPython-dev at scipy.org
More information about the IPython-dev