[Python-Dev] Fwd: Removal of GIL through refcounting removal.
josiah.carlson at gmail.com
Tue Nov 4 23:18:06 CET 2008
On Tue, Nov 4, 2008 at 5:31 AM, Leonardo Santagada <santagada at gmail.com> wrote:
> On Nov 4, 2008, at 12:11 AM, Josiah Carlson wrote:
>> There is no shortage of algorithms (such as matrix multiplication) that
>> are parallelizable but not particularly good candidates for an IPC-based
>> multiprocessing paradigm.
>> Ahh, but those algorithms aren't going to be written in Python; they are
>> going to be written in C, and are going to manipulate memory directly. With
>> such things, you can use standard Python threads, call into your C runtime,
>> and release the GIL. Alternatively, you can use the mmap module to store
>> your data, shared across multiple processes, using the same direct-memory
>> access as you would with multiple threads and GIL release.
> That's the thing, right now you can't even use python to sketch one of those
> algorithms and for example compare performance with a non parallel version.
Why not? The data needs to come from *somewhere*, it doesn't appear
from the aether into your RAM. And even if it *does* appear from
nowhere, you can always use synthetic data to compare single process
vs. multiprocess - in pure Python.
> Also with four cores on common intel processor you might be happy with your
> algo being 4x faster while being quite easy to develop (not needing to use
Indeed, but it's not uncommon that using psyco you can get 2-4x (if
not occasionally 10-100x), with similar (though generally superior)
performance gains from using Pyrex/Cython. And really, after
implementing your base algorithm in Python, translating it to C/C++ is
generally pretty straightforward for anything not involving complex
data structures or string parsing.
> I personally think this will be too much work on
And that's up to every programmer to decide for themselves. I've done
a few of the things I've mentioned (shared mmaps, C code controlled by
Python, etc.), and have been very pleased with the speedups I
attained. With better tools (Python, multiprocessing, Pyrex/Cython,
...) now available, I can't help but say, "back in my day, we just
wrote it in C and were *happy* that we didn't have to use Fortran". ;)
More information about the Python-Dev