Thread State and the Global Interpreter Lock

[Aahz]

In article <3ef8b9b2$1 at news.hks.com>, Ryjek  <ryjek at findmeifyou.can> wrote:
>
>Our application currently consists of two processes: one is a database 
>and one is a windowing program with a user interface. Both embed a 
>Python interpreter for customization purposes (i.e. you can write your 
>own GUI extensions, and your own database procedures).
>
>I would like to merge both processes into one process with two threads 
>(mostly for performance purposes), but keep both threads' data separated 
>the same way as they are now. Python practically does not allow me to do 
>it because it insists that only one thread can execute Python bytecodes 
>at a time. I would like to be able to run both interpreters in 
>completely separate environments, where no state and no objects are 
>shared. I believe the restriction of "one thread at a time" is too 
>strict in such cases. What is the purpose of Py_NewInterpreter() if you 
>cannot run it concurrently ..?

It was a mistake, essentially, kept around for the rare occasions when
the caveats below don't apply.

I would say that if performance is your primary goal, multi-threading
your application would be a poor idea.  Threads work best in two
contexts: breaking up similar types of work for performance (and in the
case of Python, that pretty much needs to be I/O work), and increasing
the responsiveness of user-centered applications.  Unless you're sharing
huge amounts of data, it's quite likely that even outside of Python you
wouldn't see much (if any) performance increase.

>I understand that making Python completely thread safe w.respect to 
>native threads may not be appropriate, but it shouldn't be that 
>difficult to allow for real concurrent execution in isolated 
>environments. At least, this is what they did in Perl: 
>http://www.icewalkers.com/Perl/5.8.0/lib/threads.html

The problem is that Python relies heavily on DLLs, both internally and
third-party libraries.  It's pretty much impossible to set things up so
that random DLLs can be correctly loaded by multiple interpreters in a
single process.

Because threading has existed in core Python for many years, a lot of
bugs and issues have been hashed out -- it's not at all clear to me the
extent to which Perl has done the same.  One reason why Python sticks
with the GIL model is that multi-threading does diddly-squat for
computational threads on a single-CPU box.  It's actually *more*
efficient the way Python does it.
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

Usenet is not a democracy.  It is a weird cross between an anarchy and a
dictatorship.