[Patches] [ python-Patches-967763 ] fix bsddb memory leaks

[SourceForge.net]

Patches item #967763, was opened at 2004-06-06 13:51
Message generated for change (Comment added) made by greg
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=967763&group_id=5470

Category: None
>Group: Python 2.3
Status: Open
>Resolution: Remind
Priority: 5
Submitted By: Neal Norwitz (nnorwitz)
Assigned to: Gregory P. Smith (greg)
Summary: fix bsddb memory leaks

Initial Comment:
only one of the changes fixed a leak reported by
valgrind, but i think all the changes are necessary.

----------------------------------------------------------------------

>Comment By: Gregory P. Smith (greg)
Date: 2004-06-27 16:34

Message:
Logged In: YES 
user_id=413

fix committed.  Modules/_bsddb.c 1.32 and  Lib/bsddb/test/test_recno.py 1.8

patch attached.  marking as python 2.3 as this should be committed to the python 2.3 maintenance branch before 2.3.5 is released.


----------------------------------------------------------------------

Comment By: Gregory P. Smith (greg)
Date: 2004-06-27 15:51

Message:
Logged In: YES 
user_id=413

DB_get and DBC_set_range were the two leaks that showed up in valgrind using the existing testsuite.  however there were plenty more possible leak cases (most of them in odd situations like out of memory or passing a non DBTxn object in the txn parameter).  

The most common trigger for a leak in various places would be an exception exit from a many functions when using integer keys (queue & recno databases).  I have a patch which i believe fixes all of these and more after a lot of code inspection that i'll commit once my valgrind run finishes and reveals no bsddb related leaks or double frees. (ugh running valgrind -v --leak-check=yes ./python Lib/bsddb/test/test_all.py on a 5 year old laptop is slow...)

"manual memory management... how quaint."

----------------------------------------------------------------------

Comment By: Neal Norwitz (nnorwitz)
Date: 2004-06-15 18:48

Message:
Logged In: YES 
user_id=33168

should FREE_DBT() be called clearTxn() and friends fail? 
one memory leak is caused in DB_get() by test_recno, line 75
(d.get(100)).  if the FREE_DBT()s at line 1392-1393 are
moved outside the if, so they always execute the leak goes
away.  i'm not sure if this is really the correct solution
though.

the other leak is in test_compat, don't know anything more.

----------------------------------------------------------------------

Comment By: Gregory P. Smith (greg)
Date: 2004-06-14 18:47

Message:
Logged In: YES 
user_id=413

Thanks, I hadn't caught up on python-dev in a while.  What
happens if you apply my attached patch and rerun the test
under valgrind?

The 4 bytes leaked sounds exactly like what i was expecting
reading the code (an integer key causes make_key_dbt to
alloc a 32bit value).  in the DBC_set_range case without the
patch i believe it is likely that the integer is allocated
but the REALLOC flag is overridden in the old code with a
MALLOC causing BerkeleyDB to leak the old pointer and
allocate a new key for the return value (btree accessed
using an integer key).

8 bytes in 2 blocks leaked in DB_get is double what i
expected but without debug info showing the DB_get function
call arguments and database flags its hard to say what happened.

----------------------------------------------------------------------

Comment By: Neal Norwitz (nnorwitz)
Date: 2004-06-14 18:36

Message:
Logged In: YES 
user_id=33168

I just run the regression tests.  Did you see my mail to
python-dev?  

http://mail.python.org/pipermail/python-dev/2004-June/045245.html

Here's the bsddb leaks reported by valgrind:

 4 bytes in 1 blocks are definitely lost in loss record 7 of 663
    at 0x3C01DA1D: malloc (vg_replace_malloc.c:109)
    by 0x3C928265: make_key_dbt (_bsddb.c:400)
    by 0x3C92DE4F: DBC_set_range (_bsddb.c:2911)

 8 bytes in 2 blocks are definitely lost in loss record 29
of 663
    at 0x3C01DA1D: malloc (vg_replace_malloc.c:109)
    by 0x3C928265: make_key_dbt (_bsddb.c:400)
    by 0x3C929088: DB_get (_bsddb.c:1349)


----------------------------------------------------------------------

Comment By: Gregory P. Smith (greg)
Date: 2004-06-14 17:02

Message:
Logged In: YES 
user_id=413

make_key_dbt() will never return with an error leaving a
FREE_DBT(key) necessary.

however looking in depth at the code i believe it would have
been possible to have a leak when using integer keys and
exiting early with an error.  a patch (untested) that should
fix that is attached.  i'll test it out later.

do you have any memleak test cases by any chance?


----------------------------------------------------------------------

Comment By: alan johnson (chomo)
Date: 2004-06-14 14:11

Message:
Logged In: YES 
user_id=943591

looks like the ones needed
anyway

----------------------------------------------------------------------

Comment By: Neal Norwitz (nnorwitz)
Date: 2004-06-13 13:56

Message:
Logged In: YES 
user_id=33168

Greg, just wanted to make sure you saw this.  It seems like
you're the only one working on bsddb these days.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=967763&group_id=5470