[Python-ideas] solving multi-core Python

[Nathaniel Smith]

On Mon, Jun 22, 2015 at 10:56 PM,  <random832 at fastmail.us> wrote:
> On Sun, Jun 21, 2015, at 18:41, Antoine Pitrou wrote:
>> It's actually already the case in POSIX that most things are illegal
>> between fork() and exec(). However, to make fork() practical, many
>> libraries or frameworks tend to ignore those problems deliberately.
>
> I'm not _entirely_ sure that this applies to single-threaded programs,
> or even to multi-threaded programs that don't use constructs that will
> cause problems.
>
> The text is: "A process shall be created with a single thread. If a
> multi-threaded process calls fork(), the new process shall contain a
> replica of the calling thread and its entire address space, possibly
> including the states of mutexes and other resources. Consequently, to
> avoid errors, the child process may only execute async-signal-safe
> operations until such time as one of the exec functions is called. Fork
> handlers may be established by means of the pthread_atfork() function in
> order to maintain application invariants across fork() calls."
>
> Note that it uses "may only" (which is ambiguous) rather than "shall
> only". It could be read that "only [stuff] until exec" is a suggestion
> of what the child process "may" do, under the circumstances described,
> to avoid the particular problems being discussed, rather than as a
> general prohibition.

Yeah, basically the way this works out is: (a) in practice on
mainstream systems you can get away with forking and then doing
whatever, so long as none of the threads in the parent process were
holding any crucial locks, and the child is prepared for them to have
all disappeared. (b) But, if something does break, then system
builders reserve the right to laugh in your face. You can argue about
things being technically ambiguous or whatever, but that's how it
works.

E.g. if you have a single-threaded program that does a matrix
multiply, then forks, and then the child does a matrix multiply, and
you run it on OS X linked to Apple's standard libraries, then the
child will lock up, and if you report this to Apple they will close it
as not-a-bug.

> And the next paragraph is "When the application calls fork() from a
> signal handler and any of the fork handlers registered by
> pthread_atfork() calls a function that is not async-signal-safe, the
> behavior is undefined." suggesting that the behavior is _not_ likewise
> undefined when it was not called from a signal handler.

I wouldn't read anything into this. pthread_atfork registers three
handlers, and two of them are run in the parent process, where
normally they'd be allowed to call any functions they like.

-n

-- 
Nathaniel J. Smith -- http://vorpus.org