Python or OS forking/threading problem?

[Snorri Gylfason]

Thanks everybody for looking into this problem. I was
working with Naris on this (Naris posted the original 
question).


I have a few comments to your discussion that might be helpful.

1) According to Posix when doing fork inside a thread only
   that thread is cloned in the new process. 

   a) Thats exactly why in the code below none of
      the threads are cloned (only the main thread).

   b) This is known problem for the child process. 
      If the cloned thread needs to acquire a lock that
      at the time of the fork some other thread had, it
      will deadlock (since in the child process there
      is no thread to release that lock).
      pthread_atfork is there to help the child process
      to handle this problem.

2) The strange thing in our case is that it is the parent
   thread that starts behaving badly. And as I will soon
   point out, it depends on what the child process is doing.

3) We ran into this problem because we were using popen in
   a thread. Popen is basically only doing fork and exec plus
   some filehandling stuff. The popen module is implemented
   in python. I replaced that module with one in C and the
   problem went away  (so I have a workaround:)
   I did some more experiments and it seems that if the child
   process leaves an atomic operation (enters Python again) 
   our problem may occur.
   I don't know the implementation of Python but everything
   I have seen indicates that the child process is still sharing 
   some lock with the parent process. 

Hope this will help

Snorri

Florian Weimer wrote:
> 
> "Neil Schemenauer" <nascheme at enme.ucalgary.ca> writes:
> 
> > Not for my problem on Linux.  My code didn't call sleep().  It
> > may be a bug with the pthreads in libc6 for Linux.  I can't
> > reproduce it with C code though.
> 
> I think I can explain what happens.  Look at the following code:
> 
> #include <stdio.h>
> #include <string.h>
> 
> #include <unistd.h>
> #include <pthread.h>
> 
> void *
> thread(void * v)
> {
>   for (;;) {
>     char buf[40];
>     int len = sprintf(buf, "%lu\n", (unsigned long) getpid());
>     write(1, buf, len);
>     sleep(1);
>   }
> }
> 
> main()
> {
>   pthread_t t;
>   pthread_create (&t, NULL, &thread, NULL);
>   fork();
>   sleep(3600);
> }
> 
> This clearly shows that the thread is not duplicated on fork().
> 
> Now look at the Python source code:
> 
> static PyObject *
> posix_fork(self, args)
>         PyObject *self;
>         PyObject *args;
> {
>         int pid;
>         if (!PyArg_ParseTuple(args, ":fork"))
>                 return NULL;
>         pid = fork();
>         if (pid == -1)
>                 return posix_error();
>         PyOS_AfterFork();
>         return PyInt_FromLong((long)pid);
> }
> 
> void
> PyOS_AfterFork()
> {
> #ifdef WITH_THREAD
>         main_thread = PyThread_get_thread_ident();
>         main_pid = getpid();
> #endif
> }
> 
> long PyThread_get_thread_ident _P0()
> {
>         volatile pthread_t threadid;
>         if (!initialized)
>                 PyThread_init_thread();
>         /* Jump through some hoops for Alpha OSF/1 */
>         threadid = pthread_self();
>         return (long) *(long *) &threadid;
> }
> 
> In the child process, all threads have disappeared, but the code doesn't
> seem to be prepared to handle this.