[Baypiggies] Guido's Blog: It isn't Easy to Remove the

[Warren Stringer]

I have an application that could use many cores, a visual music synth. Here
are two snapshots of the ontology:

   Local view: www.muse.com/vj/sky_less.jpg and a somewhat more
   Global view: www.muse.com/vj/sky_mess.jpg  

The way that I dealt with locking was to cache events. These are implemented
within on the green arcs (in the above jpgs). For asynchronous calls, one
way to speed updates is by using a 6 semaphores circular queue with sender
write-only, receiver read-only, and queue manager read-write semaphores. I
tried this technique out on audio streams with around a doubling in
throughput. This works for my application, but I haven't thought through the
more general case. 

The clustering of nodes, in my two examples, uses Newman's algorithm. The
"community" slider, at the bottom left, can change the granularity of the
clustering. If you prefer, a live JAR file can be found here:
www.muse.com/tr3/viz.html 

Why is this relevant? Well think of each shaded area as a possible core. I
would like to run my application on an affordable 80-core TerraFlop chip in
five years. A scheduler could auto-shift the community slider between 1 and
80 cores. 

C libraries are an interesting problem and have both fine-grained and
loose-grain solutions. 

One fine-grain solution is to decompose each C extension into a call graph.
Global mutable state is usually NOT globally used throughout the C
extension. So, a clustering algorithm can decompose an extension into
smaller chunks with function calls marshaled between cluster boundaries.

One loose-grain solution is to provide an interpreter for each extension.
But then, is there a GIL to manage all the little IL's - a LIL GIL?

I like the fine-grained version better because the relevant portion of an
extension may co-reside with byte-code on the same core. Messages are passed
between clusters via non-locking queues. 

Warren Stringer

> On Behalf Of Guido van Rossum
> 
> Well, it would require rewriting all existing C extensions, except for
> the ones that have zero mutable global data. Worse, some C extensions
> wrap around C libraries that have global mutable data -- for those
> there is no fix. Realistically, most C code uses global mutable state,
> and realistically, most Python programs use C extensions.
> 
> On 9/14/07, Warren DeLano <warren at delsci.com> wrote:
...
> > In other words, it is not the interpreter lock that is so limiting,
> > rather, it is the fact that the interpreter acts like a global singleton
> > with respect to the C/API and other native code extensions.
...
> > That simple workaround could preserve compatibility will existing C/API
> > code, while still freeing us from this crushing global singularity...
> >
> > So is this just crazy talk?  Or might something like this actually be
> > worth a try?
...
> > warren at delsci.com
...
> --Guido van Rossum (home page: http://www.python.org/~guido/)