[ python-Bugs-1758146 ] Crash in PyObject_Malloc
SourceForge.net
noreply at sourceforge.net
Tue Jul 24 15:01:53 CEST 2007
Bugs item #1758146, was opened at 2007-07-21 18:12
Message generated for change (Comment added) made by timbishop
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1758146&group_id=5470
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Interpreter Core
Group: Python 2.5
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Tim Bishop (timbishop)
Assigned to: Nobody/Anonymous (nobody)
Summary: Crash in PyObject_Malloc
Initial Comment:
I'm running the following on Solaris 9 SPARC:
python 2.5
apache 2.2
mod_python 3.3.1
subversion 1.4.4
trac 0.11dev
Trac is a web application that's written in python and is running through apache using mod_python. It also uses the subversion python libraries.
After an undetermined amount of clicks (usually in the order of a minute or two of randomly clicking around) the apache child process dies:
[Sat Jul 21 17:47:27 2007] [error] [client myip] mod_python (pid=15138, interpreter='my.site.com', phase='PythonHandler', handler='trac.web.modpython_frontend'): Application error, referer: http://my.site.com/
[Sat Jul 21 17:47:27 2007] [error] [client myip] ServerName: 'my.site.com', referer: http://my.site.com/
[Sat Jul 21 17:47:27 2007] [error] [client myip] DocumentRoot: '/path/to/docroot', referer: http://my.site.com/
[Sat Jul 21 17:47:27 2007] [error] [client myip] URI: '/trac/', referer: http://my.site.com/
[Sat Jul 21 17:47:27 2007] [error] [client myip] Location: '/trac', referer: http://my.site.com/
[Sat Jul 21 17:47:27 2007] [error] [client myip] Directory: None, referer: http://my.site.com/
[Sat Jul 21 17:47:27 2007] [error] [client myip] Filename: '/path/to/docroot', referer: http://my.site.com/
[Sat Jul 21 17:47:27 2007] [error] [client myip] PathInfo: '/trac/', referer: http://my.site.com/
It's dumped a core file. Examining that with gdb shows a Bus error here:
Core was generated by `/usr/local/sbin/httpd -DLocalConfig -k start'.
Program terminated with signal 10, Bus error.
#0 PyObject_Malloc (nbytes=16) at Objects/obmalloc.c:747
747 if ((pool->freeblock = *(block **)bp) != NULL) {
(gdb) l
742 * Pick up the head block of its free list.
743 */
744 ++pool->ref.count;
745 bp = pool->freeblock;
746 assert(bp != NULL);
747 if ((pool->freeblock = *(block **)bp) != NULL) {
748 UNLOCK();
749 return (void *)bp;
750 }
751 /*
(gdb)
Full gdb output is attached.
I've tried disabling pymalloc when building python, but the problem just moves elsewhere. However, with pymalloc enabled it's consistently on this line.
Do you have advice on how to debug this further?
----------------------------------------------------------------------
>Comment By: Tim Bishop (timbishop)
Date: 2007-07-24 14:01
Message:
Logged In: YES
user_id=25567
Originator: YES
Hi,
That patch is included in 2.5.1, so I don't think it's that causing the
problem.
Tim.
----------------------------------------------------------------------
Comment By: Neal Norwitz (nnorwitz)
Date: 2007-07-24 06:29
Message:
Logged In: YES
user_id=33168
Originator: NO
This gives me an idea. Are you using threads and generators, perhaps?
Could this be related to Bug #1579370? http://python.org/sf/1579370 You
can find a short discussion on it from python-dev around Jan 22-23. Martin
checked in rev 53531 on trunk. This change might be in 2.5.1, I don't
remember when it came out. Try rebuilding Python with this patch installed
and see if it fixes your problem.
----------------------------------------------------------------------
Comment By: Tim Bishop (timbishop)
Date: 2007-07-23 18:21
Message:
Logged In: YES
user_id=25567
Originator: YES
And another followup. It fails with the below error on the most basic
mod_python test cases:
http://www.modpython.org/live/current/doc-html/inst-testing.html
http://www.modpython.org/live/current/doc-html/inst-trouble.html
I've also removed a large proportion of apache modules.
So it seems there's very little left externally that could be causing
this.
----------------------------------------------------------------------
Comment By: Tim Bishop (timbishop)
Date: 2007-07-23 14:09
Message:
Logged In: YES
user_id=25567
Originator: YES
Further info about what's causing the "Invalid thread state for this
thread".
#0 0xfefa0218 in _lwp_kill () from /usr/lib/libc.so.1
(gdb) f 4
#4 0xfe2a8990 in PyThreadState_Swap (newts=0x296c58) at
Python/pystate.c:320
320 Py_FatalError("Invalid thread state for
this thread");
(gdb) l
315 to it, we need to ensure errno doesn't change.
316 */
317 int err = errno;
318 PyThreadState *check =
PyGILState_GetThisThreadState();
319 if (check && check->interp == newts->interp &&
check != newts)
320 Py_FatalError("Invalid thread state for
this thread");
321 errno = err;
322 }
323 #endif
324 return oldts;
(gdb) p check
$1 = (PyThreadState *) 0x19c880
(gdb) p *check
$2 = {next = 0x0, interp = 0x199f90, frame = 0x0, recursion_depth = 0,
tracing = 0, use_tracing = 0, c_profilefunc = 0, c_tracefunc = 0,
c_profileobj = 0x0, c_traceobj = 0x0, curexc_type = 0x0, curexc_value =
0x0, curexc_traceback = 0x0, exc_type = 0xfe3457c4,
exc_value = 0x0, exc_traceback = 0x0, dict = 0x0, tick_counter = 40,
gilstate_counter = 1, async_exc = 0x0, thread_id = 1}
(gdb) p newts
$3 = (PyThreadState *) 0x296c58
(gdb) p *newts
$4 = {next = 0x19c880, interp = 0x199f90, frame = 0x0, recursion_depth =
0, tracing = 0, use_tracing = 0, c_profilefunc = 0,
c_tracefunc = 0, c_profileobj = 0x0, c_traceobj = 0x0, curexc_type =
0x0, curexc_value = 0x0, curexc_traceback = 0x0, exc_type = 0x0,
exc_value = 0x0, exc_traceback = 0x0, dict = 0x0, tick_counter = 0,
gilstate_counter = 1, async_exc = 0x0, thread_id = 1}
(gdb) p oldts
$5 = (PyThreadState *) 0x0
(gdb)
----------------------------------------------------------------------
Comment By: Tim Bishop (timbishop)
Date: 2007-07-23 13:46
Message:
Logged In: YES
user_id=25567
Originator: YES
Ok, I was guessing it was memory corruption, or something else nasty.
I've managed to eliminate subversion, so this just leaves apache,
mod_python and python. I've rebuild with --with-pydebug and it's made the
crash happen earlier. It appears to die now in some checking code that only
happens when pydebug is used:
Program terminated with signal 6, Aborted.
#0 0xfefa0218 in _lwp_kill () from /usr/lib/libc.so.1
(gdb) bt
#0 0xfefa0218 in _lwp_kill () from /usr/lib/libc.so.1
#1 0xfef50c88 in raise () from /usr/lib/libc.so.1
#2 0xfef36e60 in abort () from /usr/lib/libc.so.1
#3 0xfe2af8d8 in Py_FatalError (msg=0xfe309618 "Invalid thread state for
this thread") at Python/pythonrun.c:1559
#4 0xfe2a8990 in PyThreadState_Swap (newts=0x1e8b78) at
Python/pystate.c:320
#5 0xfe2593b0 in PyEval_AcquireThread (tstate=0x1e8b78) at
Python/ceval.c:252
#6 0xfe644b78 in get_interpreter () from
/usr/local/libexec/apache2/mod_python.so
#7 0xfe648f28 in python_cleanup_handler () from
/usr/local/libexec/apache2/mod_python.so
#8 0xff2a3e48 in run_cleanups () from /usr/local/lib/libapr-1.so.0
#9 0xff2a4674 in apr_pool_destroy () from /usr/local/lib/libapr-1.so.0
#10 0x0004581c in ap_process_http_connection ()
#11 0x000415e4 in ap_run_process_connection ()
#12 0x0004c7f8 in child_main ()
#13 0x0004cb18 in make_child ()
#14 0x0004cc14 in startup_children ()
#15 0x0004d778 in ap_mpm_run ()
#16 0x00027110 in main ()
(gdb) f 5
#5 0xfe2593b0 in PyEval_AcquireThread (tstate=0x1e8b78) at
Python/ceval.c:252
252 if (PyThreadState_Swap(tstate) != NULL)
(gdb) l
247 if (tstate == NULL)
248 Py_FatalError("PyEval_AcquireThread: NULL new
thread state");
249 /* Check someone has called PyEval_InitThreads() to create
the lock */
250 assert(interpreter_lock);
251 PyThread_acquire_lock(interpreter_lock, 1);
252 if (PyThreadState_Swap(tstate) != NULL)
253 Py_FatalError(
254 "PyEval_AcquireThread: non-NULL old thread
state");
255 }
256
(gdb) f 4
#4 0xfe2a8990 in PyThreadState_Swap (newts=0x1e8b78) at
Python/pystate.c:320
320 Py_FatalError("Invalid thread state for
this thread");
(gdb) l
315 to it, we need to ensure errno doesn't change.
316 */
317 int err = errno;
318 PyThreadState *check =
PyGILState_GetThisThreadState();
319 if (check && check->interp == newts->interp &&
check != newts)
320 Py_FatalError("Invalid thread state for
this thread");
321 errno = err;
322 }
323 #endif
324 return oldts;
(gdb) f 3
#3 0xfe2af8d8 in Py_FatalError (msg=0xfe309618 "Invalid thread state for
this thread") at Python/pythonrun.c:1559
1559 abort();
(gdb) l
1554 OutputDebugString("\n");
1555 #ifdef _DEBUG
1556 DebugBreak();
1557 #endif
1558 #endif /* MS_WINDOWS */
1559 abort();
1560 }
1561
1562 /* Clean up and exit */
1563
(gdb)
So your guess about threads might well be right. I am compiling with
--with-threads. I guess the next step would be to disable threads?
Tim.
----------------------------------------------------------------------
Comment By: Neal Norwitz (nnorwitz)
Date: 2007-07-21 22:47
Message:
Logged In: YES
user_id=33168
Originator: NO
Firstly, take a deep breath. This probably isn't going to be easy. This
seems like memory corruption, possibly due to threads. That's a guess,
since I don't know if you are running with threads.
The first (probably faster alternative) is to build python and all the C
extension modules --with-pydebug (passed when you run ./configure). That
might catch the error slightly earlier. It won't necessarily point to the
cause of the problem, but may help narrow down the potential causes.
The second alternative is to try running this under valgrind or purify or
with some memory debugger. It seems like there is memory corruption (I'm
guessing from mod_python). This will be more robust at finding the problem
nearer to the root cause. Unfortunately, it might prove too slow to be
useful or find the problem.
If you can narrow down the test case that would help a lot. For example,
could you eliminate subversion (presumably using the C extension module) or
mod_python? If you could eliminate one and remove the problem, that would
help a lot.
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1758146&group_id=5470
More information about the Python-bugs-list
mailing list