[IPython-dev] frontend plans

[Barry Wark]

Gaël and I had a very nice talk this morning on future directions for
the frontend package and IPython frontends in general and I'd like to
let you all know what we were thinking.  I appologize that this
summary is somewhat terse. Writing about software development is not
(yet) one of my strengths. I don't have much time to dedicate to this
today and thought that a brief summary was better than a smoother––but
unsent–write-up.

Our discussion centered around Gaël's use case. He's planning to write
a Wx frontend for IPython that has some concurency (or lack thereof)
guarantees. Namely, he wants everything to happen synchronously.
Furthermore, he would like to remove the Twisted dependency from
IPython.frontend if possible—if there's no asynchronous results in his
frontend, there appears to be no need for Twisted.  By way of
contrast, my Cocoa frontend is written to be fully asynchronous,
ultimately looking more like a Mathematica notebook than a terminal,
and Twisted is an acceptable dependency. Gaël's requirements are very
similar to those of the basic terminal frontend, so I think it's in
everyone's interest to think about how to best meet those
requirements.

There seem to be two ways to get synchronous, Twisted-free frontends:

1 We could write an IPython.frontend.frontendbase.IFrontEnd
implementation that talks directly with IPython.kernel.core. A little
bit of trickery would easily remove the twisted.python.failure.Failure
dependency from IPython.frontend.frontendbase.

2, Write an implementation of IEngineCore that does not depend on
Twisted and returns results synchronously. Because the IEngine*
interface specifies that all methods return t.i.d.Deferred results, we
will need a mock Deferred object. The Synchronous-deferred project
(lp;syncrhonous-deferred) appears to fit the bill. License is public
domain, and it's two pure python files (one implementation, one test).
The project appears to be effectively "complete" -- no activity in
several months. I propose we include the syncrhonous deferred as an
external dependency in the IPython distribution. We could then write
an IEngineBase implementation that returns SynchronousDeferreds. This
engine implementation could be passed to FrontEndBase's constructor
and we get synchronous, twisted-free frontends [1]

Since the IEngine* interface is much more stable than the IFrontEnd
interface, I propose going with solution #2. Although it adds an
additional python file as external dependency, it seems conceptually
cleaner (all frontends go through the engine interface) and allows
frontends to decide between synchronous and asynchronous behavior
without any code changes.

If anyone has comments, let's have 'em. If folks think it's a good
idea, I'll create a branch and start making the changes outlined in #2
above.

The second issue Gaël and I discussed was code completion in the front
end. As I understand it, Brian and Fernando want completion to happen
on the frontend to avoid network latency (and possible blocking engine
latency). In order to do this, however, the frontend needs a complete
copy of the user_ns. This means a large memory overhead and a lot of
synchronization issues. We don't have a very good solution at this
point. In the Cocoa frontend, I've played with mirroring the user_ns
top-level keys for completion but going to the engine after that, not
a very satisfying solution. We need ideas.

Barry

[1] Removing the Twisted dependence completely will require moving the
IEngine* interfaces to a separate module from the implementations. I
propose moving them to IPython.kernel.engineinterface.py.