[Python-3000] the future of the GIL

[Josiah Carlson]

Talin <talin at acm.org> wrote:
> Giovanni Bajo wrote:
> > On 05/05/2007 15.29, tomer filiba wrote:
> > 
> >> however, running a threaded python script over an 8-core
> >> machine, where you can utilize at most 12.5% of the horsepower,
> >> seems like too large a sacrifice to me.
> > 
> > You seem to believe that the only way to parallelize your programs is to use 
> > threads. IMHO, threads is just the most common and absolutely the worst, under 
> > many points of views.
> 
> I think it's a case of wanting the most general mechanism for doing 
> parallel computation. Any algorithm that can be efficiently parallelized 
> using processes can also be done with threads (assuming that the 
> infrastructure for threading is there), but the converse is not true.

The proposals to remove the GIL have been under the assumption that
shared memory processing using multiple threads is desired.  They also
presume that there will be some sort of locking mechanism on a
per-object basis so that objects won't be clobbered.

By going multi-process rather than multi-threaded, one generally removes
shared memory from the equasion.  Note that this has the same effect as
using queues with threads, which is generally seen as the only way of
making threads "easy".  If one *needs* shared memory, we can certainly
create an mmap-based shared memory subsystem with fine-grained object
locking, or emulate it via a server process as the processing package
has done.

Seriously, give the processing package a try.  It's much faster than one
would expect.


 - Josiah