[IPython-dev] frontend plans

Barry Wark barrywark at gmail.com
Thu Jun 26 16:54:52 EDT 2008

On Wed, Jun 25, 2008 at 3:59 PM, Brian Granger <ellisonbg.net at gmail.com> wrote:
>> Gaël and I had a very nice talk this morning on future directions for
>> the frontend package and IPython frontends in general and I'd like to
>> let you all know what we were thinking.  I appologize that this
>> summary is somewhat terse. Writing about software development is not
>> (yet) one of my strengths. I don't have much time to dedicate to this
>> today and thought that a brief summary was better than a smoother––but
>> unsent–write-up.
> Glad you two were able to talk.
>> Our discussion centered around Gaël's use case. He's planning to write
>> a Wx frontend for IPython that has some concurency (or lack thereof)
>> guarantees. Namely, he wants everything to happen synchronously.
>> Furthermore, he would like to remove the Twisted dependency from
>> IPython.frontend if possible—if there's no asynchronous results in his
>> frontend, there appears to be no need for Twisted.  By way of
>> contrast, my Cocoa frontend is written to be fully asynchronous,
>> ultimately looking more like a Mathematica notebook than a terminal,
>> and Twisted is an acceptable dependency. Gaël's requirements are very
>> similar to those of the basic terminal frontend, so I think it's in
>> everyone's interest to think about how to best meet those
>> requirements.
> I think this is very true that a terminal based frontend and a
> synchronous GUI frontend will be very similar in design.
>> There seem to be two ways to get synchronous, Twisted-free frontends:
>> 1 We could write an IPython.frontend.frontendbase.IFrontEnd
>> implementation that talks directly with IPython.kernel.core. A little
>> bit of trickery would easily remove the twisted.python.failure.Failure
>> dependency from IPython.frontend.frontendbase.
> I think that we could have a single base class that implements the
> actual frontend logic and then have multiple subclasses that handle
> the calls to the various types of backends (kernel.core or
> kernel.engineservice).  I need to look more at how the frontend is
> implemented, but Barry, do you think this would work.

Yes, I think this would work. I thought zope.interface was pure
python. Since we can't require zope.interface in the stripped-down
ipython, I now think option 1 is the best approach. I'll work on that.

> The other idea would be to play around with using adaptors to handle
> the impedance mismatches in the interfaces.
>> 2, Write an implementation of IEngineCore that does not depend on
>> Twisted and returns results synchronously. Because the IEngine*
>> interface specifies that all methods return t.i.d.Deferred results, we
>> will need a mock Deferred object. The Synchronous-deferred project
>> (lp;syncrhonous-deferred) appears to fit the bill. License is public
>> domain, and it's two pure python files (one implementation, one test).
>> The project appears to be effectively "complete" -- no activity in
>> several months. I propose we include the syncrhonous deferred as an
>> external dependency in the IPython distribution. We could then write
>> an IEngineBase implementation that returns SynchronousDeferreds. This
>> engine implementation could be passed to FrontEndBase's constructor
>> and we get synchronous, twisted-free frontends [1]
> I am not sure about this one.  The dependency is not just twisted, but
> also zope.interface and these two can't really be separated from each
> other or things in i.k.  Because zope.interface has C code, we can't
> depend on it in the core of IPython or the terminal based IPython (we
> want it to work on non CPython implementations).

I didn't realize this. I thought zope.interface was pure Python. That
seems to be a show-stopper for option 2.

This can be worked
> around by simply not having the implementation declare that it
> implements the z.i IEngineCore interface.  But, in either case, that
> version of the IEngineCore should not live in i.k as there are twisted
> imports _everywhere_.  [as an aside, I think we really need to use
> subpackages to encapsulate dependencies like twisted.]

By moving the interfaces to a separate module, we could import
i.k.engineinterfaces.py without needing twisted, though obviously
zope.interface would still be required.

> Also, the Synchronous deferred does not present the exact same
> interface as a true deferred or make the same promises about when
> things will happen.

What is your understanding of the differences?

Thus, I my mind, the new IEngineCore
> implementation _couldn't_ possible really implement the interface.  In
> fact, if I remember correctly, we have a test that can be run on any
> IEngineCore implementer.  It actually calls all the methods of the
> interface and checks:
> self.assert_(isinstance(return_value, t.i.Deferred)
> A fake deferred won't pass such tests.

I don't think this is an appropriate test then. The interface should
specify the behavior of the returned object, not its implementation.
If the Synchronous Deferred behaves the same as a t.i.defer.Deferred,
then it should pass the test. The fact that it may not behave the same
is a bigger issue.

> So, in summary, I think more discussion is needed before commiting to
> #2.  I am crazy busy, but I will try to have a look at IFrontEnd
> tonight.

>> Since the IEngine* interface is much more stable than the IFrontEnd
>> interface, I propose going with solution #2. Although it adds an
>> additional python file as external dependency, it seems conceptually
>> cleaner (all frontends go through the engine interface) and allows
>> frontends to decide between synchronous and asynchronous behavior
>> without any code changes.
> I think it is conceptually convoluted because the only significant way
> that IEngireCore different from the underlying core is that its
> methods return deferreds (and Failures).  Thus to use IEngineCore, but
> then try to get rid of the (true) Deferreds by using fake ones seems
> like double work (why not just call the class whose methods don't
> return deferreds in the first place).

It feels weird that there are two ways to interact with ipython—engine
and core directly. It seemed that making a common interface was
cleaner in that it allowed users of that interface to be able to use
engine or core without modification of the frontend. Since that's not
really an option anymore (because of zope.interface), I'm happy to
give up that opinion and move on.
>> If anyone has comments, let's have 'em. If folks think it's a good
>> idea, I'll create a branch and start making the changes outlined in #2
>> above.
>> The second issue Gaël and I discussed was code completion in the front
>> end. As I understand it, Brian and Fernando want completion to happen
>> on the frontend to avoid network latency (and possible blocking engine
>> latency). In order to do this, however, the frontend needs a complete
>> copy of the user_ns. This means a large memory overhead and a lot of
>> synchronization issues. We don't have a very good solution at this
>> point. In the Cocoa frontend, I've played with mirroring the user_ns
>> top-level keys for completion but going to the engine after that, not
>> a very satisfying solution. We need ideas.
> Actually, I think we are in agreement that tab completion (the part
> that actually looks up things in the users namespace) needs to be done
> in the engine/core through its complete method.  There is just no way
> it is reasonable to mirror the user_ns in the frontend for the reasons
> you mention.  Sorry if we have said anything confusion on this front.
> So I guess the complete method of the frontend should just call the
> complete method of the engine/core?

Yes. I agree. frontend.complete() calls engine/core.complete().

> Cheers,
> Brian
>> Barry
>> [1] Removing the Twisted dependence completely will require moving the
>> IEngine* interfaces to a separate module from the implementations. I
>> propose moving them to IPython.kernel.engineinterface.py.
>> _______________________________________________
>> IPython-dev mailing list
>> IPython-dev at scipy.org
>> http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev

More information about the IPython-dev mailing list