[IPython-dev] frontend plans

[Barry Wark]

On Thu, Jun 26, 2008 at 4:03 PM, Brian Granger <ellisonbg.net at gmail.com> wrote:
> Barry,
>
>>> I think that we could have a single base class that implements the
>>> actual frontend logic and then have multiple subclasses that handle
>>> the calls to the various types of backends (kernel.core or
>>> kernel.engineservice).  I need to look more at how the frontend is
>>> implemented, but Barry, do you think this would work.
>>
>> Yes, I think this would work. I thought zope.interface was pure
>> python. Since we can't require zope.interface in the stripped-down
>> ipython, I now think option 1 is the best approach. I'll work on that.
>
> Great, I think that will be best, even though it has its own downsides.

We're in agreement then. I'll try option 1 and see how that shakes out.

>
>>> Also, the Synchronous deferred does not present the exact same
>>> interface as a true deferred or make the same promises about when
>>> things will happen.
>>
>> What is your understanding of the differences?
>
> For Deferreds, I think the interface is not a complete specification
> of the objects behavior.  For Deferreds, there are subtle cases that
> arise when you chain them together.  The behavior exhibited in complex
> Deferred chaining _can_ depend on whether or a given Deferred has
> fired or not when the chaining is setup.  We have in various places
> that handles these odd edge cases.  Also, we sometimes access the
> semi-private attributes of Deferreds.  Another thing is that if you
> have multiple unfired Deferreds, the order in which they will fire is
> not deterministic.  For the synchronous deferreds, I think the
> ordering has to be deterministic.  But in my mind, this constitues a
> different behavior and thus "interface" in the board sense of the
> word.
>
> We might be able to get the synchronous deferreds to work, but I am a
> little hesitant to go down that route simply because all of these
> things are _super_ subtle and nearly impossible to debug.

I see the issues. Too bad; the Twisted universe seems to want to
swallow its victims whole. I should have really seen this coming--I've
worked on a Cocoa framework that uses OS X's NSOperationQueue to
emulate the Twisted reactor/Deferred system. You're exactly right that
the Deferred behavior is _very_ complicated when you dig into it. Oh,
well. As long as we're willing to skip the engineservice layer for
synchronous interface to core, it's not an issue and we can forget
about Synchronous Deferreds.

>
>> Thus, I my mind, the new IEngineCore
>>> implementation _couldn't_ possible really implement the interface.  In
>>> fact, if I remember correctly, we have a test that can be run on any
>>> IEngineCore implementer.  It actually calls all the methods of the
>>> interface and checks:
>>>
>>> self.assert_(isinstance(return_value, t.i.Deferred)
>>>
>>> A fake deferred won't pass such tests.
>>
>> I don't think this is an appropriate test then. The interface should
>> specify the behavior of the returned object, not its implementation.
>> If the Synchronous Deferred behaves the same as a t.i.defer.Deferred,
>> then it should pass the test. The fact that it may not behave the same
>> is a bigger issue.
>
> True, I could write a better test for this.  The current test is only
> a super weak test behavior wise, but given the fact that there are no
> other objects that truly act like a Deferred, the test suffices.
>
>>>
>>> So, in summary, I think more discussion is needed before commiting to
>>> #2.  I am crazy busy, but I will try to have a look at IFrontEnd
>>> tonight.
>>
>>>
>>>
>>>> Since the IEngine* interface is much more stable than the IFrontEnd
>>>> interface, I propose going with solution #2. Although it adds an
>>>> additional python file as external dependency, it seems conceptually
>>>> cleaner (all frontends go through the engine interface) and allows
>>>> frontends to decide between synchronous and asynchronous behavior
>>>> without any code changes.
>>>
>>> I think it is conceptually convoluted because the only significant way
>>> that IEngireCore different from the underlying core is that its
>>> methods return deferreds (and Failures).  Thus to use IEngineCore, but
>>> then try to get rid of the (true) Deferreds by using fake ones seems
>>> like double work (why not just call the class whose methods don't
>>> return deferreds in the first place).
>>
>> It feels weird that there are two ways to interact with ipython—engine
>> and core directly. It seemed that making a common interface was
>> cleaner in that it allowed users of that interface to be able to use
>> engine or core without modification of the frontend. Since that's not
>> really an option anymore (because of zope.interface), I'm happy to
>> give up that opinion and move on.
>
> I do sort of agree with this, but I think having two interfaces is
> appropriate because they have vastly different behaviors (synchronous
> vs asynchronous).

Fair enough. I'm convinced. I had it wrong in my head-- I.kernel.core
is still the 'real' IPython. I.kernel is just a Twised wrapper around
that core. Naturally anything that doesn't want what Twisted has to
offer should talk directly with I.kernel.core and the price they pay
is that they loose all the goodies that come with the I.kernel layer.
Now that I just wrote this out, it seems wrong that I.kernel.core is a
subpackage of I.kernel. I know that it's there to isolate the ipython1
stuff from ipython0 stuff, but before too many people start writing
code using I.kernel.core, is it worth discussing if there's a better
spot for it in the IPython tree?

>
>>> Actually, I think we are in agreement that tab completion (the part
>>> that actually looks up things in the users namespace) needs to be done
>>> in the engine/core through its complete method.  There is just no way
>>> it is reasonable to mirror the user_ns in the frontend for the reasons
>>> you mention.  Sorry if we have said anything confusion on this front.
>>> So I guess the complete method of the frontend should just call the
>>> complete method of the engine/core?
>>
>> Yes. I agree. frontend.complete() calls engine/core.complete().
>
> Cool,
>
> Cheers,
>
> Brian
>
>>>
>>> Cheers,
>>>
>>> Brian
>>>
>>>> Barry
>>>>
>>>> [1] Removing the Twisted dependence completely will require moving the
>>>> IEngine* interfaces to a separate module from the implementations. I
>>>> propose moving them to IPython.kernel.engineinterface.py.
>>>> _______________________________________________
>>>> IPython-dev mailing list
>>>> IPython-dev at scipy.org
>>>> http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev
>>>>
>>>
>>
>