The future of "frozen" types as the number of CPU cores increases
nagle at animats.com
Tue Feb 23 19:35:38 CET 2010
> sjdevnull at yahoo.com wrote:
>> On Feb 20, 9:58 pm, John Nagle <na... at animats.com> wrote:
>>> sjdevn... at yahoo.com wrote:
>>>> On Feb 18, 2:58 pm, John Nagle <na... at animats.com> wrote:
>>>>> Multiple processes are not the answer. That means loading
>>>>> copies of the same code into different areas of memory. The cache
>>>>> miss rate goes up accordingly.
>>>> A decent OS will use copy-on-write with forked processes, which should
>>>> carry through to the cache for the code.
>>> That doesn't help much if you're using the subprocess module. The
>>> C code of the interpreter is shared, but all the code generated from
>>> Python is not.
>> Of course. Multithreading also fails miserably if the threads all try
>> to call exec() or the equivalent.
>> It works fine if you use os.fork().
> What about just using subprocess module to run system commands in worker
> threads? Is this likely to have problems?
The discussion above was about using "fork" to avoid duplicating the
entire Python environment for each subprocess. If you use the subprocess
module, you load a new program, so you don't get much memory sharing.
This increases the number of cache misses, which is a major bottleneck
in many-CPU systems with shared caches.
The issue being discussed was scaling Python for CPUs with many cores.
With Intel shipping 4 cores/8 hyperthread CPUs, the 6/12 part working,
and the 8/16 part coming along, this is more than a theoretical
More information about the Python-list