[IPython-dev] Ipython1 architecture...

[Glenn H Tarbox, PhD]

I'm starting to dig through the Ipython1 code... first thoughts are
there's been some serious thinking done here and I'm impressed.  I have
a lot of ideas driven by past-life experiences but tempered by
recent-life reality and am interested in beginning a dialog on very high
level architectural approaches.

In particular, as I'm sure is very clear to all, its very important that
we keep things as orthogonal as possible.  For example, I'm very
interested in the general needs of parallel computing but also require
elements of more conventional peer-to-peer and client-server
architecture.  From what I see, there shouldn't be any problems in that
regard.

I'm guessing that there are principles relating to non-generic resources
(e.g. live data sources or heterogeneous compute resources) which might
need to be addressed... this might be a simple matter of labeling
convention and a couplea data structures... but some discussion might be
fruitful.

But, the first thing I came across in digging through the code was
reference to threads on the client side.  My concern is basically that
virtually all the code I write avoids threads.  Threads are fine, of
course, but are often the single biggest source of difficulty in a
project for reasons I don't need to go into.

So, I'm interested in how and why threading is used in the client side
of Ipython1.  Is it simply to provide support for non-blocking gui
activity (like using -q4thread on the ipython command line) or is there
some other reason.

Given twisted, I'd posit that its not really necessary to use threads to
support non-blocking command line behavior or anything else on the
client.  The reactor (or select in general) can get around this... but
might require a rewrite of pyreadline which is planned but would take
time... which is why the threading is used...  

I'd probably go farther and say that threading might really only have a
proper place in the architecture for compute bound problems.  Given that
this is a parallel computing activity, that would make sense, but it
would really need to be thought through (and it looks like it might very
have been very carefully thought through).  Fortunately, twisted
provides very nice integration between the "main" thread and compute
threads, so I'm sure that all is well... but...

On the client side, it appears as though the reactor itself is spawned
in a thread.  Given that my gui code will be entirely reactor driven,
I'm probably fine... but how is the thread coordination planned?  From
previous posts its clear that the use of the new twisted blocking thread
stuff is heavily used (can't recall the proper names right now).  Again,
great... but it would be good to know what the intent is and, more
importantly, if I'm gonna get bit.

>From a more researchy / advanced / cool stuff perspective, I'd be
interested in any thinking thats been done WRT coroutines and the wild
stuff that can go on in that space.  Erlang (and I'm no Erlang expert)
has a very thorough architecture which provides all kinds of fanciness
including the persisting / migration of tasklets and levels of
architectural constructs answering questions I haven't yet asked (not
unlike twisted).  There's been some sniffing around by the stackless
folks to see what the right approach to nailing stackless and twisted
might be.  I see some of the concepts proposed for Ipython1 potentially
fitting nicely with things like tasklet migration.  Of course, there are
greenlets etc.

and its on the above that orthogonality need be maintained.

I could digress into domain specific nastiness... but will spare the
larger group... I guess that I hope for some increased radiation on the
Ipython1 front.

-glenn

-- 
Glenn H. Tarbox, PhD    | Don’t worry about people stealing your ideas. If your ideas
206-494-0819            | are any good, you’ll have to ram them down people’s throats
glenn at tarbox.org (gtalk) + ghtdak on aim/freenode