[ python-Bugs-1758146 ] Crash in PyObject_Malloc

[SourceForge.net]

Bugs item #1758146, was opened at 2007-07-21 10:12
Message generated for change (Comment added) made by nnorwitz
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1758146&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Interpreter Core
Group: Python 2.5
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Tim Bishop (timbishop)
Assigned to: Nobody/Anonymous (nobody)
Summary: Crash in PyObject_Malloc

Initial Comment:
I'm running the following on Solaris 9 SPARC:

python 2.5
apache 2.2
mod_python 3.3.1
subversion 1.4.4
trac 0.11dev

Trac is a web application that's written in python and is running through apache using mod_python. It also uses the subversion python libraries.

After an undetermined amount of clicks (usually in the order of a minute or two of randomly clicking around) the apache child process dies:

[Sat Jul 21 17:47:27 2007] [error] [client myip] mod_python (pid=15138, interpreter='my.site.com', phase='PythonHandler', handler='trac.web.modpython_frontend'): Application error, referer: http://my.site.com/
[Sat Jul 21 17:47:27 2007] [error] [client myip] ServerName: 'my.site.com', referer: http://my.site.com/
[Sat Jul 21 17:47:27 2007] [error] [client myip] DocumentRoot: '/path/to/docroot', referer: http://my.site.com/
[Sat Jul 21 17:47:27 2007] [error] [client myip] URI: '/trac/', referer: http://my.site.com/
[Sat Jul 21 17:47:27 2007] [error] [client myip] Location: '/trac', referer: http://my.site.com/
[Sat Jul 21 17:47:27 2007] [error] [client myip] Directory: None, referer: http://my.site.com/
[Sat Jul 21 17:47:27 2007] [error] [client myip] Filename: '/path/to/docroot', referer: http://my.site.com/
[Sat Jul 21 17:47:27 2007] [error] [client myip] PathInfo: '/trac/', referer: http://my.site.com/

It's dumped a core file. Examining that with gdb shows a Bus error here:

Core was generated by `/usr/local/sbin/httpd -DLocalConfig -k start'.
Program terminated with signal 10, Bus error.
#0  PyObject_Malloc (nbytes=16) at Objects/obmalloc.c:747
747                             if ((pool->freeblock = *(block **)bp) != NULL) {
(gdb) l
742                              * Pick up the head block of its free list.
743                              */
744                             ++pool->ref.count;
745                             bp = pool->freeblock;
746                             assert(bp != NULL);
747                             if ((pool->freeblock = *(block **)bp) != NULL) {
748                                     UNLOCK();
749                                     return (void *)bp;
750                             }
751                             /*
(gdb) 

Full gdb output is attached.

I've tried disabling pymalloc when building python, but the problem just moves elsewhere. However, with pymalloc enabled it's consistently on this line.

Do you have advice on how to debug this further?

----------------------------------------------------------------------

>Comment By: Neal Norwitz (nnorwitz)
Date: 2007-07-21 14:47

Message:
Logged In: YES 
user_id=33168
Originator: NO

Firstly, take a deep breath.  This probably isn't going to be easy.  This
seems like memory corruption, possibly due to threads.  That's a guess,
since I don't know if you are running with threads.

The first (probably faster alternative) is to build python and all the C
extension modules --with-pydebug (passed when you run ./configure).  That
might catch the error slightly earlier.  It won't necessarily point to the
cause of the problem, but may help narrow down the potential causes.

The second alternative is to try running this under valgrind or purify or
with some memory debugger.  It seems like there is memory corruption (I'm
guessing from mod_python).  This will be more robust at finding the problem
nearer to the root cause.  Unfortunately, it might prove too slow to be
useful or find the problem.

If you can narrow down the test case that would help a lot.  For example,
could you eliminate subversion (presumably using the C extension module) or
mod_python?  If you could eliminate one and remove the problem, that would
help a lot.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1758146&group_id=5470