[ python-Bugs-1413192 ] bsddb: segfault on db.associate call with Txn and large data

Tue Jan 24 20:40:17 CET 2006

Bugs item #1413192, was opened at 2006-01-23 12:35
Message generated for change (Comment added) made by nnorwitz
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1413192&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Extension Modules
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Alex Roitman (rshura)
Assigned to: Neal Norwitz (nnorwitz)
Summary: bsddb: segfault on db.associate call with Txn and large data

Initial Comment:
Problem confirmed on Python2.3.5/bsddb4.2.0.5 and
Python2.4.2/bsddb4.3.0 on Debian sid and Ubuntu Breezy.

It appears, that the associate call, necessary to
create a secondary index, segfaults when:
1. There is a large amount of data
2. Environment is transactional.

The
http://www.gramps-project.org/files/bsddb/testcase.tar.gz
 contains the example code and two databases, pm.db and
pm_ok.db -- both have the same number of keys and each
data item is a pickled tuple with two elements. The
second index is created over the unpickled data[1]. The
pm.db segfaults and the pm_ok.db does not. The second
db has much smaller data items in data[0].

If the environment is set up and opened without TXN
then pm.db is also fine. Seems like a problem in
associate call in a TXN environment, that is only seen
with large enough data.

Please let me know if I can be of further assistance.
This is a show-stopper issue for me, I would go out of
my way to help resolving this or finding a work-around.

Thanks!
Alex

P.S. I could not attach the large file, probably due to
the size limit on the upload, hence a link to the testcase.

----------------------------------------------------------------------

>Comment By: Neal Norwitz (nnorwitz)
Date: 2006-01-24 11:40

Message:
Logged In: YES 
user_id=33168

Could you pull the version of Modules/_bsddb.c out of SVN
and then apply my patch?  I believe your new problem was
fixed recently.  You can look in the Misc/NEWS file to find
the exact patch.

----------------------------------------------------------------------

Comment By: Alex Roitman (rshura)
Date: 2006-01-24 11:37

Message:
Logged In: YES 
user_id=498357

Done same tests on another Debian sid machine, exact same
results (up to one line number, due to my extra fprintf
statement):

(gdb) run test2.py
Starting program: /usr/bin/python2.4-dbg test2.py
[Thread debugging using libthread_db enabled]
[New Thread -1210390848 (LWP 5865)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1210390848 (LWP 5865)]
0xb7b01eb4 in DB_associate (self=0xb7d63df0, args=0xb7d67234,
    kwargs=0xb7d5ee94) at
/home/shura/src/python2.4-2.4.2/Modules/_bsddb.c:1218
1218            Py_DECREF(self->associateCallback);
(gdb) 

----------------------------------------------------------------------

Comment By: Alex Roitman (rshura)
Date: 2006-01-24 11:31

Message:
Logged In: YES 
user_id=498357

OK, built and installed all kinds of python packages with
the patch. All tests are fine. Here goes:

1. Your testcase goes just fine, no segfault with the
patched version.
2. Mine still segfaults.
3. I ran mine under gdb with python2.4-dbg package, here's
the output (printed "shurafine" is my addition, to make sure
that the correct code is being run):

$ gdb python2.4-dbg
GNU gdb 6.4-debian
Copyright 2005 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public
License, and you are
welcome to change it and/or distribute copies of it under
certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show
warranty" for details.
This GDB was configured as "i486-linux-gnu"...Using host
libthread_db library "/lib/tls/i686/cmov/libthread_db.so.1".

(gdb) run test2.py
Starting program: /usr/bin/python2.4-dbg test2.py
[Thread debugging using libthread_db enabled]
[New Thread -1210038592 (LWP 29629)]
shurafine

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1210038592 (LWP 29629)]
0xb7b57f3e in DB_associate (self=0xb7db9f58, args=0xb7dbd3b4,
    kwargs=0xb7db5e94) at
/home/shura/src/python2.4-2.4.2/Modules/_bsddb.c:1219
1219            Py_DECREF(self->associateCallback);
(gdb)

Please let me know if I can be of further assistance.

----------------------------------------------------------------------

Comment By: Alex Roitman (rshura)
Date: 2006-01-24 10:50

Message:
Logged In: YES 
user_id=498357

Thanks for a quick response!

OK, first thing first: your simpler testcase seems to expose
yet another problem, not the one I had. In particular, your
testcase segfaults for me on python2.4.2/bsddb4.3.0 but
*does not* segfault with python2.3.5/bsddb4.2.0.5

In my testcase, I can definitely blame the segfault on the
associate call, not on open. I can demonstrate it by either
commenting out the associate call (no segfault) or by
inserting a print statement right before the associate.

So your testcase does not seem to have an exact same problem
than my testcase. In my testcase nothing seems to depend on
variable names (as one would expect). I am rebuilding
python2.4 with your patch, will post results soon.

----------------------------------------------------------------------

Comment By: Neal Norwitz (nnorwitz)
Date: 2006-01-23 23:03

Message:
Logged In: YES 
user_id=33168

I spoke too soon.  The attached patch works for me or your
original test case and my pared down version.  It also
passes the tests.  It also fixes a potential memory leak. 
Let me know if this works for you.

----------------------------------------------------------------------

Comment By: Neal Norwitz (nnorwitz)
Date: 2006-01-23 22:45

Message:
Logged In: YES 
user_id=33168

I've got a much simpler test case.  The problem seems to be
triggered when the txn is deleted after the env (in
Modules/_bsddb.c 917 vs 966).  If I change the variable
names in python, I don't get the same behaviour (ie, it
doesn't crash).

I removed the original data file, but if you change the_txn
to txn, that might "fix" the problem.  If not, try playing
with different variable names and see if you can get it to
not crash.  Obviously there needs to be a real fix in C
code, but I'm not sure what needs to happen.  It doesn't
look like we keep enough info to do this properly.

----------------------------------------------------------------------

Comment By: Alex Roitman (rshura)
Date: 2006-01-23 12:41

Message:
Logged In: YES 
user_id=498357

Attaching test3.py containing same code without
transactions. Works fine with either pm.db or pm_ok.db

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1413192&group_id=5470