[Baypiggies] Guido's Blog: It isn't Easy to Remove the

Alec Flett alecf at flett.org
Sat Sep 15 02:40:33 CEST 2007

I must admit, something along these lines is very attractive to me.

Here's the pattern of concurrency that will work within my app:
- request comes in
- fire off 'threads' for each independent operation
  - each thread goes and does some work, maybe queries some databases, and
returns its results
  - some threads spawn off additional tasks
- main thread collects all the results
- return them to the caller

Most of this can be accomplished fairly easily with Stackless python - you
can just fire up a tasklet for each operation and they're pretty darn
lightweight and fast - channels are what allow the synchronization at the
end of the request.

The problem is that stackless is only ever going to use one CPU - for a I/O
bound application, this is no problem at all but I've got an application
that is CPU heavy in addition to some I/O. Personally what I'd love to see
is for stackless to get one thread per CPU that it can run arbitrary
tasklets on - I don't care where my tasklets are running because they've
pretty much got all their own local state and only need to synchronize at
the end.

Ultimately my app is most often embedded in a (web) application server, so
as long as I fork a python for each CPU and just run one Stackless thread
per python interpreter I'm good - the load should be evenly distributed
amongst the CPUs assuming a reasonably consistent unit of work per http
request. But for some batch operations not inside a web server, I'd much
rather distribute the work without trying to serialize and synchronize data
between processes.

I am green with envy for Erlang because I think its concurrency is very well
suited to this pattern, but I'm working with 10,000 lines of Python and an
organization built around python developers... not to mention I am in love
with Python itself :)


On 9/14/07, Warren DeLano <warren at delsci.com> wrote:
> Hmm...
> Call me crazy, but it seems to me that the problem isn't so much the
> GIL, but rather, it is our inability to simultaneously run multiple
> Python interpreters within a single process, with each interpreter
> playing nicely with native C extensions and libraries.
> In other words, it is not the interpreter lock that is so limiting,
> rather, it is the fact that the interpreter acts like a global singleton
> with respect to the C/API and other native code extensions.
> To restate this yet another way:  In my perfect Python-embedded world,
> the Python interpreter would itself become an object you can instantiate
> many times, and the bulk of the C/API would then become methods called
> on that object.  Each interpreter instance would still have its own GIL
> -- you wouldn't need to deal with microscopic locking, and you wouldn't
> lose any of Python's existing performance.
> Of course, most C/API methods would however need to take "self" as an
> argument:
> PyObject *PySome_Action(PythonInterpreterInstance *pythonSelf, ...);
> However, we could use TLS (thread-local-state) to dynamically bind a
> specific interpreter instance to a specific thread.
> PythonInterpreterInstance *P = ...some extant interpeter instance...;
> if(PyInstanceViaTLS_BindThread(P)) {
>   PyGILState_STATE gstate;
>   gstate = PyGILState_Ensure();
>   /* global calls within this thread are now directed at a specific
> interpreter */
>   PySome_Action (...);
>   /* meanwhile, other threads could simultaneously be messaging other
> interpreters running in parallel within the same process */
>   PyGILState_Release(gstate);
>   PyInstanceViaTLS_UnbindThread();
> }
> That simple workaround could preserve compatibility will existing C/API
> code, while still freeing us from this crushing global singularity...
> So is this just crazy talk?  Or might something like this actually be
> worth a try?
> Cheers,
> Warren
> warren at delsci.com
> _______________________________________________
> Baypiggies mailing list
> Baypiggies at python.org
> To change your subscription options or unsubscribe:
> http://mail.python.org/mailman/listinfo/baypiggies
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/baypiggies/attachments/20070914/6485c078/attachment-0001.htm 

More information about the Baypiggies mailing list