[Python-bugs-list] [ python-Bugs-488730 ] os.popen() seg faults on FreeBSD 5

noreply@sourceforge.net noreply@sourceforge.net
Thu, 13 Dec 2001 21:02:53 -0800


Bugs item #488730, was opened at 2001-12-03 21:14
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=488730&group_id=5470

Category: Extension Modules
Group: Platform-specific
>Status: Closed
Resolution: None
Priority: 5
Submitted By: Marc Culler (marcculler)
Assigned to: Nobody/Anonymous (nobody)
Summary: os.popen() seg faults on FreeBSD 5 

Initial Comment:
Here is the crash with Python 2.1.1 on my FreeBSD 
5.0-current system:

[ace:culler]$ python
Python 2.1.1 (#5, Aug 18 2001, 17:41:43) 
[GCC 2.95.3 20010315 (release)] on freebsd5
Type "copyright", "credits" or "license" for more
information.
>>> import os
>>> mypipe = os.popen('ls /')
Segmentation fault (core dumped)

No problem occurs when I do the same thing in 
Python 2.0, which was compiled under FreeBSD 4,
but is running here under FreeBSD 5:

[ace:culler]$ python2.0
Python 2.0 (#2, Nov 16 2000, 22:22:48) 
[GCC 2.95.2 19991024 (release)] on freebsd4
Type "copyright", "credits" or "license" for more
information.
>>> import os
>>> mypipe = os.popen('ls /')
>>> mypipe.readlines()
['COPYRIGHT\012', 'bin\012', 'boot\012', 'cdrom\012',
'compat\012', 'dev\012', 'dist\012', 'entropy\012',
'etc\012', 'home\012', 'kernel\012',
'kernel.GENERIC\012', 'kernel.old\012',
'kernel.prev\012', 'mnt\012', 'modules\012',
'modules.old\012', 'proc\012', 'root\012', 'sbin\012',
'shared\012', 'stand\012', 'sys\012', 'tmp\012',
'usr\012', 'var\012']
>>> 


----------------------------------------------------------------------

Comment By: Marc Culler (marcculler)
Date: 2001-12-13 21:02

Message:
Logged In: YES 
user_id=392704

Thank you, Martin.

The situation with incompatible C libraries is not quite as
bad as you suggest.  It turned out that not only was the
-ltk82 dragging in libc, it was dragging in libc.so.4.  I
recompiled the tcl and tk ports and the seg faults went
away.  The stupidest part was that the compiler had tried to
warn me about this, and I wasn't paying attention.

Perhaps you could add a warning in the README file that the
FreeBSD 4 versions of libtk82 and libtcl82 cannot be used
when building pyton under FreeBSD 5.

Thank you for taking the time to advise me on this.

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2001-12-12 22:38

Message:
Logged In: YES 
user_id=21627

It would be best if you ask a BSD expert for advise. I
consider the notion of incompatible C libraries broken
beyond repair (the only other system that does that is MS
VC++, for debug and non-debug C libraries), so you really
need to dig out some documentation that elaborates the
problem and proposes work-arounds.

Please do ldd on the resulting Python binary to see whether
it directly links with libc. If it does, add the option -v
to the  linker line; most likely, gcc implicitly passes a
-lc option. You may experiment with passing -pthread to the
linker line (and all compiler lines) in this case.

Another potential cause is that other libraries (like
-ltk82) drag in libc.so. In that case, I'd say it is
hopeless: you have than the following options
1. do not build a multi-threaded python,
2. do not build a Python that incorporates _tkinter (not
sure that building tkinter as a shared module would help;
you really need that expert to answer that question)
3. rebuild Tcl/Tk to link with -lc_r.

If you ask that we do something about it: We could either
add some text to README, or refuse to build --with-threads
on FreeBSD.

----------------------------------------------------------------------

Comment By: Marc Culler (marcculler)
Date: 2001-12-12 20:13

Message:
Logged In: YES 
user_id=392704

We are getting closer!  Yes, it does seem to be related to
linking against both libc and libc_r.  If I compile the test
program as follows:
gcc -o test test.c -lc -lc_r
I get a seg fault at the same location in _fcntl (and the
backtrace shows that the libc_r version of _fcntl is being
called from the libc version of fdopen).

If I use just -lc, or just -lc_r, or even -lc_r -lc then the
test program runs fine.

So let's assume that the libc and libc_r versions of _fcntl
use different structures for file descriptor table entries,
or some such thing.  The next question is how do I prevent
python from being linked against both libraries?  I looked
through the output of make and nowhere did I see an explicit
-lc flag.  In the makefile it says

LIBS=           -lc_r -lutil

and the linker command being used to link python is 

gcc  -Wl,--export-dynamic -o python  Modules/python.o 
libpython2.1.a -lc_r -lutil -L/usr/local/lib -ltk82 -ltcl82
-L/usr/X11R6/lib -lX11  -lm

So why is it linking against libc and how do I make it stop?

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2001-12-12 04:43

Message:
Logged In: YES 
user_id=6380

Good point. The experiment with a little C program should be
repeated, compiling and linking it with exactly the same
flags as Python is compiled and linked. (Statically, since
the posix module is statically linked into Python.)

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2001-12-11 23:57

Message:
Logged In: YES 
user_id=21627

Could that be one of those "it's BSD, never mix threading
and non-threading C libraries" bugs? I find it surprising
that it crashes inside the C library; the only known causes
(besides genuine bugs) are:

- memory management got corrupted. Without seeing the source
of _fcntl, I think this is unlikely.

- caller and callee have different ideas of how structures
are layed out, due to being compiled with different
definitions of those structures.

Are you sure it is correct to link with both libc and libc_r
simultaneously? Are there additional variants of the C
library which only work in certain combinations? Is this
documented anywhere?

----------------------------------------------------------------------

Comment By: Marc Culler (marcculler)
Date: 2001-12-11 15:43

Message:
Logged In: YES 
user_id=392704

As to the subtlety of this bug, I meant to point out that
Python 2.0, which presumably has a nearly identical posix
module, has a working os.popen().  On the other hand,
Python2.0 loads libc_r.so.4 rather than libc_r.so.5.  So
maybe that is where the secret lies.  Interestingly,
the size of that library has gone from 664784 to 108988.
That suggests that there were some rather major changes
there, doesn't it?

----------------------------------------------------------------------

Comment By: Marc Culler (marcculler)
Date: 2001-12-11 15:11

Message:
Logged In: YES 
user_id=392704

Yes, I thought it looked like it was a FreeBSD bug too.
Unfortunately, though, my little test program produced
a nice listing of my / directory.  I still think it is
FreeBSD's problem, but it is at least a little bit subtle.
I suppose I should build the latest FreeBSD sources before
proceeding any further with this.

#include <stdio.h>
int main(){
int c;
FILE *fp;
fp = popen("ls /", "r");
if (fp == NULL) {
  fprintf(stderr, "popen returned NULL\n");
  exit(-1);
}
while ((c = getc(fp)) != EOF) {
  putchar(c);
}
printf("\nDone\n");
exit(0);
}


----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2001-12-11 14:28

Message:
Logged In: YES 
user_id=6380

Marc, the Python popen code (if you can see through the
#ifdefs for non-Unix platforms :-) is a pretty
straightforward call of the C library popen(). The crash you
receive suggests that it has nothing to do with Python. Can
you try a C program containing the same call? Something like

#include <stdio.h>
main()
{
  popen("ls -l", "r");
}

Does this crash in the same way? If so, you may close this
bug report and report it to the FreeBSD folks instead...

----------------------------------------------------------------------

Comment By: Marc Culler (marcculler)
Date: 2001-12-11 14:14

Message:
Logged In: YES 
user_id=392704

All I know so far is that, according to the stack trace, the
crash seems to occur in _fcntl in libc_r.so:

(gdb) run
Starting program: /usr/local/src/Python211/python 
Python 2.1.1 (#14, Dec 11 2001, 15:55:41) 
[GCC 2.95.3 20010315 (release)] on freebsd5
Type "copyright", "credits" or "license" for more
information.
>>> import os
>>> x = os.popen('ls /')

Program received signal SIGSEGV, Segmentation fault.
0x280e32ee in _fcntl () from /usr/lib/libc_r.so.5
(gdb) backtrace
#0  0x280e32ee in _fcntl () from /usr/lib/libc_r.so.5
#1  0x28391c99 in fdopen () from /usr/lib/libc.so.5
#2  0x2834bf30 in popen () from /usr/lib/libc.so.5
#3  0x80735b1 in posix_popen (self=0x0, args=0x80f488c)
    at ./Modules/posixmodule.c:2936
#4  0x805921e in call_cfunction (func=0x80d0820,
arg=0x80f488c, kw=0x0)
    at Python/ceval.c:2845
#5  0x8057cb1 in eval_code2 (co=0x8159ac0,
globals=0x80e0ccc, locals=0x80e0ccc, 
    args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0,
defcount=0, closure=0x0)
    at Python/ceval.c:1948
#6  0x8055171 in PyEval_EvalCode (co=0x8159ac0,
globals=0x80e0ccc, 
    locals=0x80e0ccc) at Python/ceval.c:341
#7  0x806ca8f in run_node (n=0x8158280, filename=0x80a202d
"<stdin>", 
    globals=0x80e0ccc, locals=0x80e0ccc, flags=0xbfbff70c)
    at Python/pythonrun.c:1045
#8  0x806bc86 in PyRun_InteractiveOneFlags (fp=0x80c93a0, 
    filename=0x80a202d "<stdin>", flags=0xbfbff70c) at
Python/pythonrun.c:570
#9  0x806bae6 in PyRun_InteractiveLoopFlags (fp=0x80c93a0, 
    filename=0x80a202d "<stdin>", flags=0xbfbff70c) at
Python/pythonrun.c:510
#10 0x806b9db in PyRun_AnyFileExFlags (fp=0x80c93a0, 
    filename=0x80a202d "<stdin>", closeit=0,
flags=0xbfbff70c)
    at Python/pythonrun.c:473
#11 0x805222c in Py_Main (argc=0, argv=0xbfbff784) at
Modules/main.c:320
#12 0x8051d04 in main (argc=1, argv=0xbfbff784) at
Modules/python.c:10
#13 0x8051c35 in _start ()


----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2001-12-11 12:58

Message:
Logged In: YES 
user_id=21627

Can't reproduce this on FreeBSD 4.4, either, using the
current CVS.

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2001-12-08 09:18

Message:
Logged In: YES 
user_id=6380

I can't reproduce this, and I don't have access to a FreeBSD
5 machine. Can you debug this yourself? We'd welcome a
patch.

----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=488730&group_id=5470