The future of "frozen" types as the number of CPU cores increases

[John Nagle]

sjdevnull at yahoo.com wrote:
> On Feb 20, 9:58 pm, John Nagle <na... at animats.com> wrote:
>> sjdevn... at yahoo.com wrote:
>>> On Feb 18, 2:58 pm, John Nagle <na... at animats.com> wrote:
>>>>     Multiple processes are not the answer.  That means loading multiple
>>>> copies of the same code into different areas of memory.  The cache
>>>> miss rate goes up accordingly.
>>> A decent OS will use copy-on-write with forked processes, which should
>>> carry through to the cache for the code.
>>     That doesn't help much if you're using the subprocess module.  The
>> C code of the interpreter is shared, but all the code generated from
>> Python is not.
> 
> Of course.  Multithreading also fails miserably if the threads all try
> to call exec() or the equivalent.
> 
> It works fine if you use os.fork().

    Forking in multithreaded programs is iffy.  What happens depends
on the platform, and it's usually not what you wanted to happen.
Solaris before version 10 forks all threads, and both new processes
have all the threads running.  POSIX semantics are to fork only the
thread making the request.  The problem is that this leaves the other
Python threads in the new process with Python state, locks and reference
counts set, but no OS thread to run them.  See

"http://bugs.python.org/issue1683"
"http://bugs.python.org/issue874900"
"http://bugs.python.org/issue6721"
"http://bugs.python.org/issue7242"

There are at least two open bug reports in this area.

If you fork and "exec", which discards the state of Python in the child
process, there's less trouble. But if you're actually forking a
Python environment and running it, you're going to have at least a
memory leak, and probably further difficulties. 	

				John Nagle