On Wed, Dec 17, 2003 at 09:14:50AM -0600, Jeff Epler wrote:

There are also problems on redhat9, where signals in a fork+exec'd subprocess are blocked, for instance. This seemed to be a consequence of blocking all signals in thread_pthread.h:PyThread_start_new_thread().

To follow up on my own message, here is a program that demonstrates the problem I ran into with threads and signals. doit() uses system() to create a process that should kill itself with a signal nearly instantly, otherwise it sleeps for one second. If doit() is called from the main thread or before any threads are created, it prints a time well under 1 second. If it is run from a thread, it takes just over a second, because the delivery of the signal in the subprocess is blocked. Typical output: $ python thread-signal-problem.py not threaded Elapsed time 0.00420594215393 subthread Elapsed time 1.00974297523 main thread Elapsed time 0.00419104099274 import thread, time, os def doit(): t = time.time() os.system("kill $$; sleep 1") t1 = time.time() print "Elapsed time", t1-t print "not threaded" doit() print "subthread" thread.start_new(doit, ()) time.sleep(2) print "main thread" doit()

On Thu, Dec 18, 2003 at 10:22:34PM +0100, Martin v. Löwis wrote:

Can you find out what $$ is, and what the PIDs and thread IDs of all participating threads are?

I'm not sure what all information I should try to gather for you. Let me know if you think this is enough to file a bug report with... I changed the example to make it clearer that it's the subprocess ignoring the signal that is the problem, not anything in Python that is taking time to notice the death of a child process. Typical output (of course, pids and ppids change from run to run. I didn't find gettid(), I think thread.get_ident() is not what you were asking for): not threaded os.getpid() -> 6444 os.getppid() -> 6332 thread.get_ident() -> 1074152064 shell process 6445 thread ppid 6444 Elapsed time 0.00640296936035 subthread os.getpid() -> 6444 os.getppid() -> 6332 thread.get_ident() -> 1082547504 shell process 6447 thread ppid 6444 kill failed Elapsed time 1.01621508598 main thread os.getpid() -> 6444 os.getppid() -> 6332 thread.get_ident() -> 1074152064 shell process 6449 thread ppid 6444 Elapsed time 0.00641894340515 :r threadbug.py import thread, time, os def doit(): print "os.getpid() ->", os.getpid() print "os.getppid() ->", os.getppid() print "thread.get_ident() ->", thread.get_ident() t = time.time() os.system("echo shell process $$; echo thread ppid $PPID; kill $$; echo kill failed; sleep 1") t1 = time.time() print "Elapsed time", t1-t print print print "not threaded" doit() print "subthread" thread.start_new(doit, ()) time.sleep(2) print "main thread" doit()

martin@v.loewis.de (Martin v. Löwis) writes:

Jeff Epler <jepler@unpythonic.net> writes:

...

...

Can you find out what $$ is, and what the PIDs and thread IDs of all participating threads are?

I'm not sure what all information I should try to gather for you. Let me know if you think this is enough to file a bug report with... I changed the example to make it clearer that it's the subprocess ignoring the signal that is the problem, not anything in Python that is taking time to notice the death of a child process.

That is an important observation; signals that are blocked in the parent process will be blocked in the child process as well.

I'm not sure what to do about this: We apparently *want* the signals blocked in the thread, but we don't want them to be blocked in the process invoked through system(). Proposals are welcome.

Does pthread_atfork() help? Cheers, mwh -- We've had a lot of problems going from glibc 2.0 to glibc 2.1. People claim binary compatibility. Except for functions they don't like. -- Peter Van Eynde, comp.lang.lisp

On Mon, Dec 22, 2003 at 02:08:34AM +0100, Martin v. Loewis wrote:

With this fork() implementation, atfork handlers won't be invoked, which clearly looks like a bug to me. You might want to upgrade glibc to glibc-2.3.2-27.9.7.i686.rpm and nptl-devel-2.3.2-27.9.7.i686.rpm. In this version, the definition of FORK is changed to

I also tested a fedora machine with glibc-2.3.2-101.1 nptl-devel-2.3.2-101.1 and the pthread_atfork handler is still not called for system(). Opengroup's documentation for system() says [...] The environment of the executed command shall be as if a child process were created using fork(), and the child process invoked the sh utility using execl() [...] http://www.opengroup.org/onlinepubs/007904975/functions/system.html so this looks like a glibc/nptl bug. I filed it in redhat's bugzilla: https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=112517 Given all this, does it make sense to adopt a patch similar to the one I posted earlier, and ignore the bug in system() on any particular OS? system() and popen() are easy to replace in Python if anybody is really bothered by the problem. Jeff

On Mon, Dec 22, 2003 at 03:46:14AM +0100, Martin v. Loewis wrote:

I'd think so, yes. The patch needs to be elaborated, and I'd delay integration to wait for a Redhat response.

Well, unlike the last two bugs I filed in redhat bugzilla, this one has already seen some activity. I hope they don't take the line that this is not a bug :( ------- Additional Comments From jakub@redhat.com 2003-12-22 03:53 ------- Why? There is no word about system() in http://www.opengroup.org/onlinepubs/007904975/functions/pthread_atfork.html and there doesn't seem to be anything in http://www.opengroup.org/onlinepubs/007904975/functions/system.html that would mandate system() to be implemented using fork() function. ------- Additional Comments From jepler@unpythonic.net 2003-12-22 08:01 ------- I base my claim that pthread_atfork() should be called by system() from reading http://www.opengroup.org/onlinepubs/007904975/functions/system.html The third paragraph under the heading "description" says this: "[CX] The environment of the executed command shall be as if a child process were created using fork(), and the child process invoked the sh utility using execl() as follows [...]" The rationale section expands on what the "environment of the executed command" is: "IEEE Std 1003.1-2001 places additional restrictions on system(). It requires that if there is a command language interpreter, the environment must be as specified by fork() and exec. This ensures, for example, that close-on- exec works, that file locks are not inherited, and that the process ID is different." The pthread_atfork web page says it "provides multi-threaded application programs with a standard mechanism for protecting themselves from fork() calls in a library routine or the application itself." I believe that system() is one such library routine, since it is specified to create an environment "as specified by fork()". This issue arose in the context of Python. For various reasons, signals are blocked in all threads besides the main thread. This means that in processes generated by fork() from subthreads, all signals are blocked, which leads to undesired behavior. Using the child argument of pthread_atfork() allows signals to be unblocked when using fork()+exec() to execute an external program, but not when system() is used.

Unfortunately, this is redhat's position.

------- Additional Comments From roland@redhat.com 2003-12-22 16:37 ------- I think it is clear that the specification refers to the elements of the child process state that survive exec, so that the executed command can perceive them as part of its "environment". You could submit an interpretation request, but I think the committee would concur with my reading. The specification of pthread_atfork refers to calls to fork, not to other parts of the POSIX.1 implementation. If your application calls system, and not fork, those clauses do not apply to it.

How hard would it be to reimplement our own system() and popen() using only POSIX calls, for POSIX systems? I've always thought of these to be pretty simple combinations of fork() and exec(), with an assumption of a working /bin/sh. Without error checking: int system(char *cmd) { pid = fork(); if (!pid) { /* child */ execl("/bin/sh", "-c", cmd, NULL); } else { /* parent */ int sts; waitpid(pid, *sts, 0); return sts; } --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum <guido@python.org> writes:

...

Unfortunately, this is redhat's position.

------- Additional Comments From roland@redhat.com 2003-12-22 16:37 ------- I think it is clear that the specification refers to the elements of the child process state that survive exec, so that the executed command can perceive them as part of its "environment". You could submit an interpretation request, but I think the committee would concur with my reading. The specification of pthread_atfork refers to calls to fork, not to other parts of the POSIX.1 implementation. If your application calls system, and not fork, those clauses do not apply to it.

How hard would it be to reimplement our own system() and popen() using only POSIX calls, for POSIX systems? I've always thought of these to be pretty simple combinations of fork() and exec(), with an assumption of a working /bin/sh. Without error checking:

I think it's a bit harder than what you post, but there's code in APUE for it... Cheers, mwh -- I'd certainly be shocked to discover a consensus. ;-) -- Aahz, comp.lang.python

On Tue, Dec 23, 2003 at 10:01:58AM +0100, Martin v. Loewis wrote:

I would be concerned that we bypass magic that the system vendor put into system(), which is essential for proper operation. For example, on Linux, system() blocks SIGINT in the parent process while the child is running. Also, the shell executable that system() uses may not be /bin/sh.

This behavior (blocking SIGINT and SIGQUIT in the parent) is part of the specification of system(): The system() function shall ignore the SIGINT and SIGQUIT signals, and shall block the SIGCHLD signal, while waiting for the command to terminate. If this might cause the application to miss a signal that would have killed it, then the application should examine the return value from system() and take whatever action is appropriate to the application if the command terminated due to receipt of a signal. Jeff

Jeff Epler wrote:

You could submit an interpretation request, but I think the committee would concur with my reading.

Hmm. This is what I now did: I submitted an interpretation request to PASC: Edition of Specification (Year): 2003 Defect code : 3. Clarification required It is unclear whether calling system(3) invokes atfork handlers in a conforming implementation. system(3) specifies "The environment of the executed command shall be as if a child process were created using fork(), and the child process invoked the sh utility using execl() as follows:" In particular, usage of the word "environment" is confusing here. It may refer just to environment variables, however, the "Application usage" sections indicates that also signal handlers should be arranged as if the process was created through fork() and execl(). This still makes not clear whether handlers installed through pthread_atfork() are invoked. Action: The description of system() should change to "system() behaves as if a new process was created using fork(), and the child process invoked the sh utility using execl() ..." In addition, the application usage section should make it clear that atfork handlers are invoked. Regards, Martin