[IPython-dev] Ipython1 architecture...

[Glenn H Tarbox, PhD]

On Fri, 2008-04-11 at 11:23 -0600, Brian Granger wrote:

> >  In particular, as I'm sure is very clear to all, its very important that
> >  we keep things as orthogonal as possible.  For example, I'm very
> >  interested in the general needs of parallel computing but also require
> >  elements of more conventional peer-to-peer and client-server
> >  architecture.  From what I see, there shouldn't be any problems in that
> >  regard.
> 
> Do you mean that the parallel computing stuff is or should be
> orthogonal to p2p/client/server stuff?

I should clarify because I think my particular needs are likely
generalizable to the broader community.  In my case, I have feeds and
clients which will need to "do their thing" independent of IPython1.
However, in looking at the underlying constructs / capabilities of
IPython1, there are really nice synergies which are usable right off the
bat and provide a path to get where I want to be in the parallel
computing space

So, I see that I could suck in feeds, distribute them as I do now, but
also populate IPython1 constructs.  The next step is to invoke IPython1
computations and extend the client side capabilities incrementally...
this might mean that the cognoscenti can just go ahead and do IPython1
things as currently intended... with subsets of the capability exposed
through "pointy-clicky" gui presentation layers.

The reason I brought it up is I think this problem is probably
general... This doesn't require any major thinking at this point because
the core infrastructure is more important, but acknowledging this type
of use case and addressing it, perhaps just with examples, maybe more,
might be useful.

> >
> >  So, I'm interested in how and why threading is used in the client side
> >  of Ipython1.  Is it simply to provide support for non-blocking gui
> >  activity (like using -q4thread on the ipython command line) or is there
> >  some other reason.
> 
> Ah yes, glad you asked.  From my perspective, threads are used in two
> broad ways:
> 
> 1) As a model of parallelism/concurrency.  From this perspective,
> threads are a way of handling concurrency in a shared memory context.
> I personally think that threads are _not_ a good solution in this
> arena.  They are simply too low level and difficult to get right.

Agreed... and at multiple levels... difficult to get right by
infrastructure people... virtually impossible as a general capability
exposed to the masses.  I think threads are great in the kernel, for
example.  Fortunately, very few people are cleared for that kind of work

> 
> 2) As an implementation details for building other models of
> parallelism/concurrency.  For example, Erlang uses an actor based,
> shared nothing, message passing approach to concurreny.  But,
> underneath the hood Erlang uses threads to implement their runtime
> system.  I am all for this type of usage of threads.

Here we descend into the definition of threads...  At one level, the
abstraction is the same... but, really, all we're saying is that its
useful to maintain the stack frames... 

Most languages (Python included) provide threads which maintain state
but also "help" with scheduling, context switching etc... but without
much control.... which means synchronization of shared state and, um,
proper architecture... again, a place that most users shouldn't be
playing.

Erlang / stackless / greenlets provide more control of what a thread
does and how execution is controlled... and this is where we've gone way
way beyond what folks understand... which is unfortunate because this
could, with proper architecture, provide straightforward code writing
without too much concern about this kind of nastiness... note that I
haven't even gotten into the real nastiness which is typically exposed
inside libraries which aren't thread safe

> 
> In my mind, our usage of threads in the ipython1 falls into category
> 2).  We are not advocating that users know or care about threads.
> But, we do need them in our implementation.  Here is why:
> 
> Long ago, when we first started ipython1, we did not use Twisted.  In
> fact, we used plain old blocking sockets for our client.  The reason
> is that we have always wanted an API that your average scientists
> could understand.  And in our experience (we are scientists),
> scientists do not think in terms of event loops or deferreds.  They
> think in very blocking terms.  Thus, we have always felt like our
> design goals require a blocking (possibly polling) user-facing
> interface.

I've been using twisted for quite a while, and still get major brain
damage from time to time trying to figure out how to invert code to be
twisted compliant... so I'm very much in agreement...

> 
> Eventually, we discovered Twisted and wanted to use Twisted for our
> client networking code.  But, we have a particularly odd set of
> constraints:
> 
> 1) We need our client code to run on normal python/ipython sessions
> that re not event driven.
> 
> 2) We want to use Twisted.
> 
> 3) We need to have a truly blocking interface.

Wait... on 3)... Where is the blocking?  This is one of the things I
want to get a handle on... 

> 
> So, we asked on the Twisted lists, "how can we block on a deferred?"
> There was much laughter and blanket statements like "you don't want to
> do that" and "you can't do that."  

I love the twisted guys and think they're onto something really huge...
But, they are very very opinionated on stuff like this (more often then
not, they're also exactly right)...

However, Im looking at Kamaelia etc. and I think there's more to this
tasklet / coroutine / frame maintenance thing than they do.  I think we
can beat this one... not sure how just yet... but I think we can.

> In spite of those statements we
> knew that in fact we did want to do that.  So we spent time playing
> around with various approaches.  The end of these explorations was
> this conclusion:
> 
> If you only have one thread, it is true, "you can't block on a
> deferred."  But also began to see that if there are multiple threads
> around, you could do it.  Then came along blockingCallFromThread, that
> does exactly that:  it allows blocking on a deferred in another
> thread.  Another option is coroutines, and we haven't explored that
> yet.  Coroutines are attractive, but only if they don't require a
> custom python version (like stackless).

but, we're into dangerous territory here.  Coordinating with the "main"
thread is one thing... but then we have all the libraries that somebody
is gonna just go ahead and call.  and away we go...

> 
> >  Given twisted, I'd posit that its not really necessary to use threads to
> >  support non-blocking command line behavior or anything else on the
> >  client.  The reactor (or select in general) can get around this... but
> >  might require a rewrite of pyreadline which is planned but would take
> >  time... which is why the threading is used...
> 
> Ironically, writing non-blocking interfaces is relatively easy using
> threads or twisted.  The challenge is putting a blocking layer on top
> of a non-blocking one.  I should mention that we do have two types of
> clients:
> 
> 1) asynclient:  these clients don't use threads.  Rather they use
> twisted and return deferreds.  These are very new and most people
> don't know about them.  I really like them.
> 
> 2) client: these are a thin wrapper around the asynclient clients that
> use blockingCallFromThread to block.

ah, so I'm a bit behind on my code reading... but this is good to know.

But, I still haven't seen where the need for blocking comes in....

> 
> >  I'd probably go farther and say that threading might really only have a
> >  proper place in the architecture for compute bound problems.  Given that
> >  this is a parallel computing activity, that would make sense, but it
> >  would really need to be thought through (and it looks like it might very
> >  have been very carefully thought through).  Fortunately, twisted
> >  provides very nice integration between the "main" thread and compute
> >  threads, so I'm sure that all is well... but...
> 
> I still think threads are useful in implementing higher level
> concurrency constructs - even though I agree with you that i don't
> think they are a good solution many other things.

I also believe threads are useful... but the care required is very much
under appreciated...  There is no better way to introduce
non-reproducible, non-deterministic, non-debuggable, spurious errors
into infrastructure code than with threads (other than particle
emissions from deep space, something Sun can speak to)  

I'll make a completely unsupportable statement that in aggregate we'd
get more productivity without threads than with... of course, what a
waste that would be because done properly, threads make trivial work of
difficult problems... but too few people have enough of a clue or are
willing to spend the time to understand the issues.

> 
> One thing that is really nice about twisted is that it does play well
> with threads.  Our entire model is really message based (shared
> nothing) and if python didn't have the GIL, we could do some very cool
> single process things.

If we didn't have the GIL, Python would save 50% of mailing list traffic
and have something realistic to say WRT the multicore world we're
already in.  I have a feeling this one will get fixed... there are way
too many smart people involved and its too important not to get fixed.
Good thing we beat that cold virus too huh? :-)

> 
> >  On the client side, it appears as though the reactor itself is spawned
> >  in a thread.  Given that my gui code will be entirely reactor driven,
> >  I'm probably fine... but how is the thread coordination planned?  From
> >  previous posts its clear that the use of the new twisted blocking thread
> >  stuff is heavily used (can't recall the proper names right now).  Again,
> >  great... but it would be good to know what the intent is and, more
> >  importantly, if I'm gonna get bit.
> 
> The interaction between the twisted reactors thread and GUI threads is
> something we do need to think more about.  We would love help when we
> get around to that.  We need to make sure that people don't get bit
> because they choose to use our stuff.

So, why do we have multiple threads here?  The twisted gui reactors,
including the qtreactor I maintain, do everything in one thread.  So,
gtk and qt are single threaded already.  the wxreactor used a separate
thread because of wierd behavior with wx which may no longer be valid...
but if it is valid, we need to get wx fixed (it can't be that hard)...
but there is no reason that the gui needs to be in a different thread.

The difficulty I've run into is, most likely, with pyreadline which
blocks during scrolling... so the -q4thread (or other thread switches)
are provided... but this is probably easily fixed by adding stdin/stdout
to the reactor's main event loop and digging into pyreadline to replace
the blocking code with something asynchronous.  I don't know all the
issues here but I can't imagine this is all that hard either and avoids
all this nastiness

> Coroutines:  I would love to see a demonstration of using coroutines
> and twisted, but using stackless or another custom python version is
> not an option.  Another option is greenlet and corotwine:
> 
> http://codespeak.net/py/dist/greenlet.html
> https://launchpad.net/corotwine/
> 
> These run with the standard Cpython.  I haven't played with them, but
> would love it if someone wanted to scope this out and and report back.

Me too... and perhaps I'll do some of this... but we're into a different
client base here also.  While coroutines provide extremely useful
abstractions and "can" avoid typical threading difficulties (as well as
providing thousands vs. tens of concurrent frames) one is faced with
coroutine semantics.  

I guess the question which needs to be answered is when discussing
coroutines are we outside IPython1 and into IPythonCoroutine or
scipyCoroutine expecting that folks understand that they've crossed
over.

> 
> Erlang:  I _really_ like Erlang.  I have been spending some of my free
> time messing with Erlang and as far as I am concerned it is the best
> language for concurrent/distributed programming.  What do I mean by
> this.  It is what I would like Twisted to be.  Twisted is fantastic,
> but I really need single process twisted apps to scale to multiple
> cores.  An erlang app (at least in principle) can scale across both
> threads and processes and that is really amazing.  I should also say
> that I am amazed at how far Twisted goes in this direction.  It is
> quite amazing!

Hmmm... so, we're mixing models here and this is the kind of thing which
we need to get framed in order to make useful progress... 

coroutines are about maintaining execution state independent of
execution resources in order to provide the kind of control needed to
work properly at scale and to minimize the waste of resources not really
designed for all applications.  So, Erlang is about a lot of parallel
threads but not necessarily about compute bound processing...  Whats
nice is that state machine management is "implicitly" encoded in a
top-down manner (the stack frames are automatically managed by the
language).  Of course, Erlang also provids all the other stuff which
does lend itself to compute bound processing across cores etc...

but, what are we trying to do?  Is it MPI type work with python, or
coroutines... are we doing huge petri-nets or CFD?  There are many
similarities WRT infrastructure... but CFD cares a lot less about state
machines than massive petri-nets.

> 
> But, I should say, I think Python is the best language for scientific,
> mathematical and numerical computing.  Thus, I don't ever see a day
> when I would recommend that a scientist use erlang to solve their
> favorite diff-eq.

agreed.  Python also plays nice with lower level code so wrapping nasty
compute bound funcitonality is easy enough...  I think one of the best
examples is matplotlib (over scipy/numpy)... just in that you're most of
the way to what a lot of folks use matlab for with a much better
language

> 
> But, the combination of Erlang and Python is very interesting.  Here
> is what I have been thinking.  There is a not a Twisted implementation
> of the Erlang node protocol:
> 
> https://launchpad.net/twotp
> 
> I have played around with this and it is very encouraging.  More work
> needs to be done, but there is a great foundation.  This lets
> Python+Twisted processes talk seamlessly to Erlang nodes.  You can
> imagine the possibilities.  My basic vision would be to have an
> implementation of ipython1, where the controller is an erlang node,
> and the engines/clients are python.  The advantages would be:
> 
> 1) Better performance in terms of the networking stuff.

Hmmm... I'm with you on the first part of this... not sure that
performance is ever enhanced by python... python is about programmer
productivity and very rich semantic constructs... but i doubt you're
gonna see overall higher performance

> 
> 2) The controller is currently a huge bottleneck for us.  It simple
> has to do to much.  If we used erlang, it would be much easier to
> scale the controller itself to multiple cores or even multiple hosts.

not sure about this either.  First, I can't imagine the controller has a
huge amount to do but I say this without knowing much about it... seems
like a twisted process living on a multi-core cpu would work fine... the
other cores would provide all the support services (I/O etc) while a
single twisted process can handle coordination messaging...  If the
controller is doing a bunch of compute heavy work, I think I'd need to
know more... but I'd be surprised

> 
> 3) The fault tolerance of the controller would be easier to address
> (OTP,Mnesia, etc.)

this is true... but does it really matter... we need to determine our
domain... some of the best articles on this subject are on Google's
map-reduce infrastructure where they don't deal with this particular
issue... if a compute node goes down, no problem... the statistical
probability of the controller going down is low enough that its not
worth going through all this effort...

the above is not an option for the original intent of Erlang which is
telephone switch management... an entirely different problem.

> 
> Downside of using Erlang:
> 
> 1) Users would have to install Erlang and Python

yea, but if this is scipyCoroutines, they're already in... they either
have this kind of problem or not... if so, we're facilitators...

> 
> Bottom line:  there are lots of really interesting directions to go
> in.  We just need good people with vision and ideas.

The set intersection problem again... :-)

-glenn

> 
> Cheers,
> 
> Brian
> 
> >  and its on the above that orthogonality need be maintained.
> >
> >  I could digress into domain specific nastiness... but will spare the
> >  larger group... I guess that I hope for some increased radiation on the
> >  Ipython1 front.
> >
> >  -glenn
> >
> >  --
> >  Glenn H. Tarbox, PhD    | Don't worry about people stealing your ideas. If your ideas
> >  206-494-0819            | are any good, you'll have to ram them down people's throats
> >  glenn at tarbox.org (gtalk) + ghtdak on aim/freenode
> >
> >  _______________________________________________
> >  IPython-dev mailing list
> >  IPython-dev at scipy.org
> >  http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev
> >
-- 
Glenn H. Tarbox, PhD    | Don’t worry about people stealing your ideas. If your ideas
206-494-0819            | are any good, you’ll have to ram them down people’s throats
glenn at tarbox.org (gtalk) + ghtdak on aim/freenode     | ^ Howard Aiken, IBM engineer