[IPython-dev] question about decision to use twisted

[Brian Granger]

> I was thinking of using twisted directly, but I guess I really need to dig
> into ipython1. I don't have a sense of how I would use ipython1 in a
> non-interactive way, is there some discussion in the docs or some examples of
> scripting with ipython1? Is there a callback mechanism in ipython1 to respond
> to non-blocking events?

This is one aspect of our marketing that has been bad.  We always talk
about "interactive parallel computing", but IPython (because ipython1
has been merged into IPython, I will just talk about IPython from here
on out) fully supports non-interactive usage.

We have a full set of non-interactive examples that can be found here:


Warning: these examples have not been merged from ipython1->IPython.
Fernando and I are about to do that this week.

As far as callbacks, we have two interfaces for everything:

1.  A fully asynchronous version that uses twisted and all method
return deferreds.  This doesn't work interactively, but it what you
want to use if you need to respond to non-blocking events.

2.  A regular blocking interface that can be used interactively or
not.  This interface doesn't really block - it just lets you poll for
asynchronous results.

But, we fully supports both approaches and option 1 will fully plug
into a regular Twisted based app.

>> Unless
>> you really want the treading API that processing provides, I think
>> IPython is by far the best solution [I fully acknowledge my bias here]
>> for high-level parallelism.
>>
>> > I would like to know what the ipython developers concerns were at the
>> > time the decision was made to use twisted in ipython1, was there some
>> > discussion on the ipython mailing lists? Any advice or comments would be
>> > greatly appreciated.
>>
>> I don't think we really discussed this on the mailing list, it was
>> probably private discussion between Fernando and myself.  We went with
>> Twisted as:
>>
>> 1) we didn't want to reinvent Twisted
>>
>> 2) we really need what Twisted provides, so the temptation to reinvent
>> it was great.
>>
>> 3) Twisted has the right abstractions that force you to write correct
>> an robust networking code.  Of course you can do this without Twisted,
>> but you are going to work much harder and probably get it wrong in
>> subtle ways that are hard to figure out.
>>
>> We have been extremely happy about going with Twisted - it is our secret
>> sauce.
>>
>> Are there any specific questions you have about Twisted?
>
> After reading some positive things about twisted, I went looking for a couple
> negative reviews. I've read that twisted is a bit of a moving target, and the
> twisted jargon is pretty unfamiliar to me. So finally I was hoping to temper
> all this information with the informed opinion from a group I know and
> respect. This is just the kind of feedback I was hoping to get.

The Twisted dev team is borderline OCD about their code.  Everything
is tested or else removed, they do code review and have a great policy
about API changes.  In summary, while there are parts that are
unstable, most of Twisted is rock solid and really robust.  There are
not many other open source projects that I would recommend as highly
as Twisted in this respect.  I am not a Twisted dev.

Some people complain about the documentation, but I have never really
found it to be a problem.  With that said, it does take a while to
really understand the model.  But it is time well spent.

Cheers,

Brian

> Thank you,
> Darren
>