[IPython-dev] Ipython1 architecture...

[Brian Granger]

Back to this thread after a long break...

>  I should clarify because I think my particular needs are likely
>  generalizable to the broader community.  In my case, I have feeds and
>  clients which will need to "do their thing" independent of IPython1.
>  However, in looking at the underlying constructs / capabilities of
>  IPython1, there are really nice synergies which are usable right off the
>  bat and provide a path to get where I want to be in the parallel
>  computing space

OK

>  So, I see that I could suck in feeds, distribute them as I do now, but
>  also populate IPython1 constructs.  The next step is to invoke IPython1
>  computations and extend the client side capabilities incrementally...
>  this might mean that the cognoscenti can just go ahead and do IPython1
>  things as currently intended... with subsets of the capability exposed
>  through "pointy-clicky" gui presentation layers.

If I am understanding this usage case correctly, I think IPython1
would work in this context with its current capabilities.  But you
would definitely want to use our asynchronous client in asynclient.py.

>  The reason I brought it up is I think this problem is probably
>  general... This doesn't require any major thinking at this point because
>  the core infrastructure is more important, but acknowledging this type
>  of use case and addressing it, perhaps just with examples, maybe more,
>  might be useful.

Very true about the examples.  The more the better.

>
>  > >
>  > >  So, I'm interested in how and why threading is used in the client side
>  > >  of Ipython1.  Is it simply to provide support for non-blocking gui
>  > >  activity (like using -q4thread on the ipython command line) or is there
>  > >  some other reason.
>  >
>  > Ah yes, glad you asked.  From my perspective, threads are used in two
>  > broad ways:
>  >
>  > 1) As a model of parallelism/concurrency.  From this perspective,
>  > threads are a way of handling concurrency in a shared memory context.
>  > I personally think that threads are _not_ a good solution in this
>  > arena.  They are simply too low level and difficult to get right.
>
>  Agreed... and at multiple levels... difficult to get right by
>  infrastructure people... virtually impossible as a general capability
>  exposed to the masses.  I think threads are great in the kernel, for
>  example.  Fortunately, very few people are cleared for that kind of work

I agree with this.

>
>  >
>  > 2) As an implementation details for building other models of
>  > parallelism/concurrency.  For example, Erlang uses an actor based,
>  > shared nothing, message passing approach to concurreny.  But,
>  > underneath the hood Erlang uses threads to implement their runtime
>  > system.  I am all for this type of usage of threads.
>
>  Here we descend into the definition of threads...  At one level, the
>  abstraction is the same... but, really, all we're saying is that its
>  useful to maintain the stack frames...
>
>  Most languages (Python included) provide threads which maintain state
>  but also "help" with scheduling, context switching etc... but without
>  much control.... which means synchronization of shared state and, um,
>  proper architecture... again, a place that most users shouldn't be
>  playing.
>
>  Erlang / stackless / greenlets provide more control of what a thread
>  does and how execution is controlled... and this is where we've gone way
>  way beyond what folks understand... which is unfortunate because this
>  could, with proper architecture, provide straightforward code writing
>  without too much concern about this kind of nastiness... note that I
>  haven't even gotten into the real nastiness which is typically exposed
>  inside libraries which aren't thread safe

Yep, I agree that the models of erlang/stackless/greenlets are both
more complicated that most users will tolerate, and also a potentially
nice way of abstracting concurrency.  But, concurrency is very
different things (although they are related).  For example
stackless/greenlets express concurrency, but NOT parallelism.  Erlang
on the other hand handles both concurrency and parallelism.

>
>  >
>  > In my mind, our usage of threads in the ipython1 falls into category
>  > 2).  We are not advocating that users know or care about threads.
>  > But, we do need them in our implementation.  Here is why:
>  >
>  > Long ago, when we first started ipython1, we did not use Twisted.  In
>  > fact, we used plain old blocking sockets for our client.  The reason
>  > is that we have always wanted an API that your average scientists
>  > could understand.  And in our experience (we are scientists),
>  > scientists do not think in terms of event loops or deferreds.  They
>  > think in very blocking terms.  Thus, we have always felt like our
>  > design goals require a blocking (possibly polling) user-facing
>  > interface.
>
>  I've been using twisted for quite a while, and still get major brain
>  damage from time to time trying to figure out how to invert code to be
>  twisted compliant... so I'm very much in agreement...

Yep, I love Twisted, but sometimes it hurts.

>
>  >
>  > Eventually, we discovered Twisted and wanted to use Twisted for our
>  > client networking code.  But, we have a particularly odd set of
>  > constraints:
>  >
>  > 1) We need our client code to run on normal python/ipython sessions
>  > that re not event driven.
>  >
>  > 2) We want to use Twisted.
>  >
>  > 3) We need to have a truly blocking interface.
>
>  Wait... on 3)... Where is the blocking?  This is one of the things I
>  want to get a handle on...

Here is an example:

In [1]: from ipython1.kernel import client

In [2]: mec = client.MultiEngineClient(('localhost',10105))

In [3]: mec.push(dict(a=10))
Out[3]: [None, None, None, None, None]

In [4]: mec.pull('a')
Out[4]: [10, 10, 10, 10, 10]

All of these commands are fully blocking!  But the implementation uses
twisted!  It seems like you don't want/need a blocking interface, but
our main target audience (working scientists who don't know a thing
about asynchronous things) absolutely need a blocking interface.  At
most they neeed to be able to poll for results:

In [5]: mec.block= False

In [6]: mec.pull('a')
Out[6]: <ipython1.kernel.multiengineclient.PendingResult object at 0x15f3870>

In [7]: _.r
Out[7]: [10, 10, 10, 10, 10]

But, we also have a fully asynchronous client interface
(asynclient.py) that 1) doesn't use threads anywhere and 2) every
method returns a deferred.

Bottom line: we need to support both modes with an implementation that
is Twisted based.  To get the blocking mode with Twisted, you _must_
use threads like we do (with blockingCallFromThread).

>
>  >
>  > So, we asked on the Twisted lists, "how can we block on a deferred?"
>  > There was much laughter and blanket statements like "you don't want to
>  > do that" and "you can't do that."
>
>  I love the twisted guys and think they're onto something really huge...
>  But, they are very very opinionated on stuff like this (more often then
>  not, they're also exactly right)...

Yep absolutely.

>  However, Im looking at Kamaelia etc. and I think there's more to this
>  tasklet / coroutine / frame maintenance thing than they do.  I think we
>  can beat this one... not sure how just yet... but I think we can.

Yes, coroutines open up other routes for handling these things other
than threads.  But, I am not yet ready to jump on that until true
coroutines have been implemented well without a hacked python
interpreter.

>
>  > In spite of those statements we
>  > knew that in fact we did want to do that.  So we spent time playing
>  > around with various approaches.  The end of these explorations was
>  > this conclusion:
>  >
>  > If you only have one thread, it is true, "you can't block on a
>  > deferred."  But also began to see that if there are multiple threads
>  > around, you could do it.  Then came along blockingCallFromThread, that
>  > does exactly that:  it allows blocking on a deferred in another
>  > thread.  Another option is coroutines, and we haven't explored that
>  > yet.  Coroutines are attractive, but only if they don't require a
>  > custom python version (like stackless).
>
>  but, we're into dangerous territory here.  Coordinating with the "main"
>  thread is one thing... but then we have all the libraries that somebody
>  is gonna just go ahead and call.  and away we go...

True, at some level, we can't prevent users from calling libraries at
will.  But, I am not quite sure what things a user could call that
would cause problems with our approach.  The only thing that would be
problematic is if they want to call Twisted.  But then they should
simply use our non-threaded asynclient.py stuff.

>
>  >
>  > >  Given twisted, I'd posit that its not really necessary to use threads to
>  > >  support non-blocking command line behavior or anything else on the
>  > >  client.  The reactor (or select in general) can get around this... but
>  > >  might require a rewrite of pyreadline which is planned but would take
>  > >  time... which is why the threading is used...
>  >
>  > Ironically, writing non-blocking interfaces is relatively easy using
>  > threads or twisted.  The challenge is putting a blocking layer on top
>  > of a non-blocking one.  I should mention that we do have two types of
>  > clients:
>  >
>  > 1) asynclient:  these clients don't use threads.  Rather they use
>  > twisted and return deferreds.  These are very new and most people
>  > don't know about them.  I really like them.
>  >
>  > 2) client: these are a thin wrapper around the asynclient clients that
>  > use blockingCallFromThread to block.
>
>  ah, so I'm a bit behind on my code reading... but this is good to know.
>
>  But, I still haven't seen where the need for blocking comes in....

Does my above example clarify this point?

>
>  >
>  > >  I'd probably go farther and say that threading might really only have a
>  > >  proper place in the architecture for compute bound problems.  Given that
>  > >  this is a parallel computing activity, that would make sense, but it
>  > >  would really need to be thought through (and it looks like it might very
>  > >  have been very carefully thought through).  Fortunately, twisted
>  > >  provides very nice integration between the "main" thread and compute
>  > >  threads, so I'm sure that all is well... but...
>  >
>  > I still think threads are useful in implementing higher level
>  > concurrency constructs - even though I agree with you that i don't
>  > think they are a good solution many other things.
>
>  I also believe threads are useful... but the care required is very much
>  under appreciated...  There is no better way to introduce
>  non-reproducible, non-deterministic, non-debuggable, spurious errors
>  into infrastructure code than with threads (other than particle
>  emissions from deep space, something Sun can speak to)
>
>  I'll make a completely unsupportable statement that in aggregate we'd
>  get more productivity without threads than with... of course, what a
>  waste that would be because done properly, threads make trivial work of
>  difficult problems... but too few people have enough of a clue or are
>  willing to spend the time to understand the issues.

At some level I completely agree with these statements.  If we had
written IPython1 using low-level threads rather than Twisted, it would
be buggy nasty code.  With Twisted, it is clean robust code.

>
>  >
>  > One thing that is really nice about twisted is that it does play well
>  > with threads.  Our entire model is really message based (shared
>  > nothing) and if python didn't have the GIL, we could do some very cool
>  > single process things.
>
>  If we didn't have the GIL, Python would save 50% of mailing list traffic
>  and have something realistic to say WRT the multicore world we're
>  already in.  I have a feeling this one will get fixed... there are way
>  too many smart people involved and its too important not to get fixed.
>  Good thing we beat that cold virus too huh? :-)

Here to hoping :-)

>
>  >
>  > >  On the client side, it appears as though the reactor itself is spawned
>  > >  in a thread.  Given that my gui code will be entirely reactor driven,
>  > >  I'm probably fine... but how is the thread coordination planned?  From
>  > >  previous posts its clear that the use of the new twisted blocking thread
>  > >  stuff is heavily used (can't recall the proper names right now).  Again,
>  > >  great... but it would be good to know what the intent is and, more
>  > >  importantly, if I'm gonna get bit.
>  >
>  > The interaction between the twisted reactors thread and GUI threads is
>  > something we do need to think more about.  We would love help when we
>  > get around to that.  We need to make sure that people don't get bit
>  > because they choose to use our stuff.
>
>  So, why do we have multiple threads here?  The twisted gui reactors,
>  including the qtreactor I maintain, do everything in one thread.  So,
>  gtk and qt are single threaded already.  the wxreactor used a separate
>  thread because of wierd behavior with wx which may no longer be valid...
>  but if it is valid, we need to get wx fixed (it can't be that hard)...
>  but there is no reason that the gui needs to be in a different thread.

Yes, getting things in one thread is one step.  But it doesn't solve
all of the problems:

1) Assume the GUI+Twisted reactor are running in the main thread.

2) Assume that user code running in that process needs to block on the
result of a deferred.

3) This will require that user code runs in a different thread and
that blockingCallFromThread is used.

4) But changes are, user code will also want to work with GUI objects,
thus user code must run in the main thread.

But, this is a simple contradiction - user code can't run in the main
thread and the non-main thread.  To beat the contradiction, you have
to get rid of either assumption 1 or assumption 2.  I imagine that you
would like to get rid of 2, but our design criteria don't allow for
that.  All of this will become relevant when IPython itself is running
in an event loop.  I don't know what to do about this.  Any ideas?

>  The difficulty I've run into is, most likely, with pyreadline which
>  blocks during scrolling... so the -q4thread (or other thread switches)
>  are provided... but this is probably easily fixed by adding stdin/stdout
>  to the reactor's main event loop and digging into pyreadline to replace
>  the blocking code with something asynchronous.  I don't know all the
>  issues here but I can't imagine this is all that hard either and avoids
>  all this nastiness

This is beyond my understanding of these things (I know very little
about readline).

>
>  > Coroutines:  I would love to see a demonstration of using coroutines
>  > and twisted, but using stackless or another custom python version is
>  > not an option.  Another option is greenlet and corotwine:
>  >
>  > http://codespeak.net/py/dist/greenlet.html
>  > https://launchpad.net/corotwine/
>  >
>  > These run with the standard Cpython.  I haven't played with them, but
>  > would love it if someone wanted to scope this out and and report back.
>
>  Me too... and perhaps I'll do some of this... but we're into a different
>  client base here also.  While coroutines provide extremely useful
>  abstractions and "can" avoid typical threading difficulties (as well as
>  providing thousands vs. tens of concurrent frames) one is faced with
>  coroutine semantics.

I am not proposing we expose these semantics to users though - only
that we could use them in implementations that are hidden from the
user.  Similar to how we currently use threads.

>  I guess the question which needs to be answered is when discussing
>  coroutines are we outside IPython1 and into IPythonCoroutine or
>  scipyCoroutine expecting that folks understand that they've crossed
>  over.

I don't think that most users will want to touch or think about co-routines.

>
>  >
>  > Erlang:  I _really_ like Erlang.  I have been spending some of my free
>  > time messing with Erlang and as far as I am concerned it is the best
>  > language for concurrent/distributed programming.  What do I mean by
>  > this.  It is what I would like Twisted to be.  Twisted is fantastic,
>  > but I really need single process twisted apps to scale to multiple
>  > cores.  An erlang app (at least in principle) can scale across both
>  > threads and processes and that is really amazing.  I should also say
>  > that I am amazed at how far Twisted goes in this direction.  It is
>  > quite amazing!
>
>  Hmmm... so, we're mixing models here and this is the kind of thing which
>  we need to get framed in order to make useful progress...
>
>  coroutines are about maintaining execution state independent of
>  execution resources in order to provide the kind of control needed to
>  work properly at scale and to minimize the waste of resources not really
>  designed for all applications.  So, Erlang is about a lot of parallel
>  threads but not necessarily about compute bound processing...  Whats
>  nice is that state machine management is "implicitly" encoded in a
>  top-down manner (the stack frames are automatically managed by the
>  language).  Of course, Erlang also provids all the other stuff which
>  does lend itself to compute bound processing across cores etc...
>
>  but, what are we trying to do?  Is it MPI type work with python, or
>  coroutines... are we doing huge petri-nets or CFD?  There are many
>  similarities WRT infrastructure... but CFD cares a lot less about state
>  machines than massive petri-nets.

Again, I am only interested in Erlang for implementation purposes, not
for user-facing code.  Our users don't want to implement a distributed
asynchronous system - they want to use the one that we build.  But the
tools and models that we use in the implementation can be completely
different from what the users see.


>
>  >
>  > But, I should say, I think Python is the best language for scientific,
>  > mathematical and numerical computing.  Thus, I don't ever see a day
>  > when I would recommend that a scientist use erlang to solve their
>  > favorite diff-eq.
>
>  agreed.  Python also plays nice with lower level code so wrapping nasty
>  compute bound funcitonality is easy enough...  I think one of the best
>  examples is matplotlib (over scipy/numpy)... just in that you're most of
>  the way to what a lot of folks use matlab for with a much better
>  language
>
>
>  >
>  > But, the combination of Erlang and Python is very interesting.  Here
>  > is what I have been thinking.  There is a not a Twisted implementation
>  > of the Erlang node protocol:
>  >
>  > https://launchpad.net/twotp
>  >
>  > I have played around with this and it is very encouraging.  More work
>  > needs to be done, but there is a great foundation.  This lets
>  > Python+Twisted processes talk seamlessly to Erlang nodes.  You can
>  > imagine the possibilities.  My basic vision would be to have an
>  > implementation of ipython1, where the controller is an erlang node,
>  > and the engines/clients are python.  The advantages would be:
>  >
>  > 1) Better performance in terms of the networking stuff.
>
>  Hmmm... I'm with you on the first part of this... not sure that
>  performance is ever enhanced by python... python is about programmer
>  productivity and very rich semantic constructs... but i doubt you're
>  gonna see overall higher performance

Sorry if I wasn't clear on this - Erlang has much better performance
that Python+Twisted for asynchronous networking stuff.  Bob Ippolito
has done this comparison and has mentioned to me that the difference
was "unbelievable"

>
>  >
>  > 2) The controller is currently a huge bottleneck for us.  It simple
>  > has to do to much.  If we used erlang, it would be much easier to
>  > scale the controller itself to multiple cores or even multiple hosts.
>
>  not sure about this either.  First, I can't imagine the controller has a
>  huge amount to do but I say this without knowing much about it... seems
>  like a twisted process living on a multi-core cpu would work fine... the
>  other cores would provide all the support services (I/O etc) while a
>  single twisted process can handle coordination messaging...  If the
>  controller is doing a bunch of compute heavy work, I think I'd need to
>  know more... but I'd be surprised

The controller has a _massive_ amount to do under heavy load.  It is
the main bottleneck in IPython1 by a long short.  The difficult part
is that the problems don't come from a few CPU bound things - it comes
from lots of small extra overheads that come when you scale to many
engines, many tasks, etc.  It is the overhead of using Python+Twisted.

>
>  >
>  > 3) The fault tolerance of the controller would be easier to address
>  > (OTP,Mnesia, etc.)
>
>  this is true... but does it really matter... we need to determine our
>  domain... some of the best articles on this subject are on Google's
>  map-reduce infrastructure where they don't deal with this particular
>  issue... if a compute node goes down, no problem... the statistical
>  probability of the controller going down is low enough that its not
>  worth going through all this effort...

True, as of yet, they have punted on this issue.  It is not nearly as
important as the fault tolerance of the engines.

>  the above is not an option for the original intent of Erlang which is
>  telephone switch management... an entirely different problem.

True


>
>  >
>  > Downside of using Erlang:
>  >
>  > 1) Users would have to install Erlang and Python
>
>  yea, but if this is scipyCoroutines, they're already in... they either
>  have this kind of problem or not... if so, we're facilitators...

Yep

>
>  >
>  > Bottom line:  there are lots of really interesting directions to go
>  > in.  We just need good people with vision and ideas.
>
>  The set intersection problem again... :-)

Cheers,

Brian

>  -glenn
>
>
>  >
>  > Cheers,
>  >
>  > Brian
>  >
>  > >  and its on the above that orthogonality need be maintained.
>  > >
>  > >  I could digress into domain specific nastiness... but will spare the
>  > >  larger group... I guess that I hope for some increased radiation on the
>  > >  Ipython1 front.
>  > >
>  > >  -glenn
>  > >
>  > >  --
>  > >  Glenn H. Tarbox, PhD    | Don't worry about people stealing your ideas. If your ideas
>  > >  206-494-0819            | are any good, you'll have to ram them down people's throats
>  > >  glenn at tarbox.org (gtalk) + ghtdak on aim/freenode
>  > >
>  > >  _______________________________________________
>  > >  IPython-dev mailing list
>  > >  IPython-dev at scipy.org
>  > >  http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev
>  > >
>  --
>
> Glenn H. Tarbox, PhD    | Don't worry about people stealing your ideas. If your ideas
>  206-494-0819            | are any good, you'll have to ram them down people's throats
>  glenn at tarbox.org (gtalk) + ghtdak on aim/freenode     | ^ Howard Aiken, IBM engineer
>
>
>