[Python-Dev] GIL, Python 3, and MP vs. UP

[Bob Ippolito]

On Sep 20, 2005, at 5:43 PM, Guido van Rossum wrote:

> On 9/20/05, John J Lee <jjl at pobox.com> wrote:
>
>> threading is not the only, nor the best, concurrency model.
>> But maybe these chips designed with threading in mind blow that  
>> argument
>> out of the water.  I don't know enough to know whether that's true or
>> not...
>>
>
> I don't know that any chips are designed with threading in mind. Fast
> threading benefits from fast context switches which benefits from
> small register sets. I believe the trend is towards ever large
> register sets. Also, multiple processors with shared memory don't
> scall all that well; multiple processors with explicit IPC channels
> scale much better. All arguments for multi-processing and against
> multi-threading.

Well, in many operating systems, there isn't a whole lot of  
difference between threads and processes (both are kernel threads).   
When using threads, you can typically still use IPC, so you could  
scale similarly.

I think the biggest argument for threading is simply that lots of  
existing C/C++ code wants to use threads.  What we have now works OK,  
but you can't reliably use Python in a real-time thread (e.g. a  
CoreAudio callback) or (reliably) in a signal handler because it  
might block too long because of something else going on.  Of course,  
a lot of other design decisions in Python would prevent you from  
using it in that context too, so GIL-free threading wouldn't be a  
panacea.

Personally my biggest issue with the way the CPython VM works is that  
you can't reliably do multiple interpreters in a single process.  If  
that worked well, you could start an independent interpreter per  
thread and with a per-interpreter GIL you'd have pretty much  
everything you needed... but this would horribly break the C API and  
may be a performance hit.

My use case for this isn't so much about threads, but plug-ins.   
Writing multiple Python-based plug-ins for an application is always a  
mess, because they share too much (sys.modules, sys.path, etc.).   
PyObjC would benefit greatly from this feature, because you can write  
Python-based plug-ins for any Cocoa app that supports plug-ins, even  
if they're otherwise unaware of Python's existence.  There are  
workarounds, of course, with import hooks and similar hacks.  I think  
that mod_python would also benefit from this, and probably other such  
systems.

-bob