The future of "frozen" types as the number of CPU cores increases

[John Nagle]

mk wrote:
> sjdevnull at yahoo.com wrote:
>> On Feb 20, 9:58 pm, John Nagle <na... at animats.com> wrote:
>>> sjdevn... at yahoo.com wrote:
>>>> On Feb 18, 2:58 pm, John Nagle <na... at animats.com> wrote:
>>>>>     Multiple processes are not the answer.  That means loading 
>>>>> multiple
>>>>> copies of the same code into different areas of memory.  The cache
>>>>> miss rate goes up accordingly.
>>>> A decent OS will use copy-on-write with forked processes, which should
>>>> carry through to the cache for the code.
>>>     That doesn't help much if you're using the subprocess module.  The
>>> C code of the interpreter is shared, but all the code generated from
>>> Python is not.
> 
>> Of course.  Multithreading also fails miserably if the threads all try
>> to call exec() or the equivalent.
> 
>> It works fine if you use os.fork().
> 
> What about just using subprocess module to run system commands in worker 
> threads? Is this likely to have problems?
> 
> Regards,
> mk

    The discussion above was about using "fork" to avoid duplicating the
entire Python environment for each subprocess.  If you use the subprocess
module, you load a new program, so you don't get much memory sharing.
This increases the number of cache misses, which is a major bottleneck
in many-CPU systems with shared caches.

    The issue being discussed was scaling Python for CPUs with many cores.
With Intel shipping 4 cores/8 hyperthread CPUs, the 6/12 part working,
and the 8/16 part coming along, this is more than a theoretical
issue.

				John Nagle