[Python-Dev] Fwd: Removal of GIL through refcounting removal.

[Josiah Carlson]

On Tue, Nov 4, 2008 at 5:31 AM, Leonardo Santagada <santagada at gmail.com> wrote:
>
> On Nov 4, 2008, at 12:11 AM, Josiah Carlson wrote:
>
>> There is no shortage of algorithms (such as matrix multiplication) that
>> are parallelizable but not particularly good candidates for an IPC-based
>> multiprocessing paradigm.
>>
>> Ahh, but those algorithms aren't going to be written in Python; they are
>> going to be written in C, and are going to manipulate memory directly.  With
>> such things, you can use standard Python threads, call into your C runtime,
>> and release the GIL.  Alternatively, you can use the mmap module to store
>> your data, shared across multiple processes, using the same direct-memory
>> access as you would with multiple threads and GIL release.
>
>
> That's the thing, right now you can't even use python to sketch one of those
> algorithms and for example compare performance with a non parallel version.

Why not?  The data needs to come from *somewhere*, it doesn't appear
from the aether into your RAM.  And even if it *does* appear from
nowhere, you can always use synthetic data to compare single process
vs. multiprocess - in pure Python.

> Also with four cores on common intel processor you might be happy with your
> algo being 4x faster while being quite easy to develop (not needing to use
> c).

Indeed, but it's not uncommon that using psyco you can get 2-4x (if
not occasionally 10-100x), with similar (though generally superior)
performance gains from using Pyrex/Cython.  And really, after
implementing your base algorithm in Python, translating it to C/C++ is
generally pretty straightforward for anything not involving complex
data structures or string parsing.

> I personally think this will be too much work on

And that's up to every programmer to decide for themselves.  I've done
a few of the things I've mentioned (shared mmaps, C code controlled by
Python, etc.), and have been very pleased with the speedups I
attained.  With better tools (Python, multiprocessing, Pyrex/Cython,
...) now available, I can't help but say, "back in my day, we just
wrote it in C and were *happy* that we didn't have to use Fortran". ;)

 - Josiah