I must admit, something along these lines is very attractive to me.<br><br>Here's the pattern of concurrency that will work within my app:<br>- request comes in<br>- fire off 'threads' for each independent operation
<br> - each thread goes and does some work, maybe queries some databases, and returns its results<br> - some threads spawn off additional tasks<br>- main thread collects all the results<br>- return them to the caller<br>
<br>Most of this can be accomplished fairly easily with Stackless python - you can just fire up a tasklet for each operation and they're pretty darn lightweight and fast - channels are what allow the synchronization at the end of the request.
<br><br>The problem is that stackless is only ever going to use one CPU - for a I/O bound application, this is no problem at all but I've got an application that is CPU heavy in addition to some I/O. Personally what I'd love to see is for stackless to get one thread per CPU that it can run arbitrary tasklets on - I don't care where my tasklets are running because they've pretty much got all their own local state and only need to synchronize at the end.
<br><br>
Ultimately my app is most often embedded in a (web) application server, so as long as I fork a python for each CPU and just run one Stackless thread per python interpreter I'm good - the load should be evenly distributed amongst the CPUs assuming a reasonably consistent unit of work per http request. But for some batch operations not inside a web server, I'd much rather distribute the work without trying to serialize and synchronize data between processes.
<br><br>I am green with envy for Erlang because I think its concurrency is very well suited to this pattern, but I'm working with 10,000 lines of Python and an organization built around python developers... not to mention I am in love with Python itself :)
<br><br><br>Alec<br><br><div><span class="gmail_quote">On 9/14/07, <b class="gmail_sendername">Warren DeLano</b> <<a href="mailto:warren@delsci.com">warren@delsci.com</a>> wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Hmm...<br><br>Call me crazy, but it seems to me that the problem isn't so much the<br>GIL, but rather, it is our inability to simultaneously run multiple<br>Python interpreters within a single process, with each interpreter
<br>playing nicely with native C extensions and libraries.<br><br>In other words, it is not the interpreter lock that is so limiting,<br>rather, it is the fact that the interpreter acts like a global singleton<br>with respect to the C/API and other native code extensions.
<br><br>To restate this yet another way: In my perfect Python-embedded world,<br>the Python interpreter would itself become an object you can instantiate<br>many times, and the bulk of the C/API would then become methods called
<br>on that object. Each interpreter instance would still have its own GIL<br>-- you wouldn't need to deal with microscopic locking, and you wouldn't<br>lose any of Python's existing performance.<br><br>Of course, most C/API methods would however need to take "self" as an
<br>argument:<br><br>PyObject *PySome_Action(PythonInterpreterInstance *pythonSelf, ...);<br><br>However, we could use TLS (thread-local-state) to dynamically bind a<br>specific interpreter instance to a specific thread.<br>
<br>PythonInterpreterInstance *P = ...some extant interpeter instance...;<br><br>if(PyInstanceViaTLS_BindThread(P)) {<br><br> PyGILState_STATE gstate;<br> gstate = PyGILState_Ensure();<br><br> /* global calls within this thread are now directed at a specific
<br>interpreter */<br><br> PySome_Action (...);<br><br> /* meanwhile, other threads could simultaneously be messaging other<br>interpreters running in parallel within the same process */<br><br> PyGILState_Release(gstate);
<br><br> PyInstanceViaTLS_UnbindThread();<br>}<br><br>That simple workaround could preserve compatibility will existing C/API<br>code, while still freeing us from this crushing global singularity...<br><br>So is this just crazy talk? Or might something like this actually be
<br>worth a try?<br><br>Cheers,<br>Warren<br><a href="mailto:warren@delsci.com">warren@delsci.com</a><br><br><br><br>_______________________________________________<br>Baypiggies mailing list<br><a href="mailto:Baypiggies@python.org">
Baypiggies@python.org</a><br>To change your subscription options or unsubscribe:<br><a href="http://mail.python.org/mailman/listinfo/baypiggies">http://mail.python.org/mailman/listinfo/baypiggies</a><br></blockquote></div>
<br>