[IPython-dev] frontend plans

Brian Granger ellisonbg.net at gmail.com
Wed Jun 25 18:59:04 EDT 2008


> Gaël and I had a very nice talk this morning on future directions for
> the frontend package and IPython frontends in general and I'd like to
> let you all know what we were thinking.  I appologize that this
> summary is somewhat terse. Writing about software development is not
> (yet) one of my strengths. I don't have much time to dedicate to this
> today and thought that a brief summary was better than a smoother––but
> unsent–write-up.

Glad you two were able to talk.

> Our discussion centered around Gaël's use case. He's planning to write
> a Wx frontend for IPython that has some concurency (or lack thereof)
> guarantees. Namely, he wants everything to happen synchronously.
> Furthermore, he would like to remove the Twisted dependency from
> IPython.frontend if possible—if there's no asynchronous results in his
> frontend, there appears to be no need for Twisted.  By way of
> contrast, my Cocoa frontend is written to be fully asynchronous,
> ultimately looking more like a Mathematica notebook than a terminal,
> and Twisted is an acceptable dependency. Gaël's requirements are very
> similar to those of the basic terminal frontend, so I think it's in
> everyone's interest to think about how to best meet those
> requirements.

I think this is very true that a terminal based frontend and a
synchronous GUI frontend will be very similar in design.

> There seem to be two ways to get synchronous, Twisted-free frontends:
>
> 1 We could write an IPython.frontend.frontendbase.IFrontEnd
> implementation that talks directly with IPython.kernel.core. A little
> bit of trickery would easily remove the twisted.python.failure.Failure
> dependency from IPython.frontend.frontendbase.

I think that we could have a single base class that implements the
actual frontend logic and then have multiple subclasses that handle
the calls to the various types of backends (kernel.core or
kernel.engineservice).  I need to look more at how the frontend is
implemented, but Barry, do you think this would work.

The other idea would be to play around with using adaptors to handle
the impedance mismatches in the interfaces.

> 2, Write an implementation of IEngineCore that does not depend on
> Twisted and returns results synchronously. Because the IEngine*
> interface specifies that all methods return t.i.d.Deferred results, we
> will need a mock Deferred object. The Synchronous-deferred project
> (lp;syncrhonous-deferred) appears to fit the bill. License is public
> domain, and it's two pure python files (one implementation, one test).
> The project appears to be effectively "complete" -- no activity in
> several months. I propose we include the syncrhonous deferred as an
> external dependency in the IPython distribution. We could then write
> an IEngineBase implementation that returns SynchronousDeferreds. This
> engine implementation could be passed to FrontEndBase's constructor
> and we get synchronous, twisted-free frontends [1]

I am not sure about this one.  The dependency is not just twisted, but
also zope.interface and these two can't really be separated from each
other or things in i.k.  Because zope.interface has C code, we can't
depend on it in the core of IPython or the terminal based IPython (we
want it to work on non CPython implementations).  This can be worked
around by simply not having the implementation declare that it
implements the z.i IEngineCore interface.  But, in either case, that
version of the IEngineCore should not live in i.k as there are twisted
imports _everywhere_.  [as an aside, I think we really need to use
subpackages to encapsulate dependencies like twisted.]

Also, the Synchronous deferred does not present the exact same
interface as a true deferred or make the same promises about when
things will happen.  Thus, I my mind, the new IEngineCore
implementation _couldn't_ possible really implement the interface.  In
fact, if I remember correctly, we have a test that can be run on any
IEngineCore implementer.  It actually calls all the methods of the
interface and checks:

self.assert_(isinstance(return_value, t.i.Deferred)

A fake deferred won't pass such tests.

So, in summary, I think more discussion is needed before commiting to
#2.  I am crazy busy, but I will try to have a look at IFrontEnd
tonight.


> Since the IEngine* interface is much more stable than the IFrontEnd
> interface, I propose going with solution #2. Although it adds an
> additional python file as external dependency, it seems conceptually
> cleaner (all frontends go through the engine interface) and allows
> frontends to decide between synchronous and asynchronous behavior
> without any code changes.

I think it is conceptually convoluted because the only significant way
that IEngireCore different from the underlying core is that its
methods return deferreds (and Failures).  Thus to use IEngineCore, but
then try to get rid of the (true) Deferreds by using fake ones seems
like double work (why not just call the class whose methods don't
return deferreds in the first place).

> If anyone has comments, let's have 'em. If folks think it's a good
> idea, I'll create a branch and start making the changes outlined in #2
> above.
>
> The second issue Gaël and I discussed was code completion in the front
> end. As I understand it, Brian and Fernando want completion to happen
> on the frontend to avoid network latency (and possible blocking engine
> latency). In order to do this, however, the frontend needs a complete
> copy of the user_ns. This means a large memory overhead and a lot of
> synchronization issues. We don't have a very good solution at this
> point. In the Cocoa frontend, I've played with mirroring the user_ns
> top-level keys for completion but going to the engine after that, not
> a very satisfying solution. We need ideas.

Actually, I think we are in agreement that tab completion (the part
that actually looks up things in the users namespace) needs to be done
in the engine/core through its complete method.  There is just no way
it is reasonable to mirror the user_ns in the frontend for the reasons
you mention.  Sorry if we have said anything confusion on this front.
So I guess the complete method of the frontend should just call the
complete method of the engine/core?

Cheers,

Brian

> Barry
>
> [1] Removing the Twisted dependence completely will require moving the
> IEngine* interfaces to a separate module from the implementations. I
> propose moving them to IPython.kernel.engineinterface.py.
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev
>



More information about the IPython-dev mailing list