refleak hunting season
Last time round I found a bunch of reference counting bugs in Python just *after* a major release; I'm trying to do it all before 2.4.0 this time. I've hacked regrtest again (patch attached). Here's my notes file, though as is the case with such things writing stuff down tends to cause thoughts which cause experiments which cause the notes to be out of date more or less immediately... test_codeccallbacks leaked [2, 2, 2, 2] references registries test_copy leaked [82, 82, 82, 82] references copy_reg.dispatch_table -- now taken care of in regrtest.py test_descr leaked [2, 2, 2, 2] references resurrection, still test_descrtut leaked [1, 1, 1, 1] references exec 'print x' in defaultdict() fixed, need to check in (bit subtle though) test_distutils leaked [2, 2, 2, 2] references no idea test_gc leaked [18, 18, 18, 18] references trashcan test_hotshot leaked [111, 111, 111, 111] references there's been a patch on SF for this for about a year! test_minidom leaked [1, 1, 1, 1] references testPickledDocument, don't know more test_mutants leaked [74, -1123, 1092, -521] references randomized test test_pkg leaked [3, 3, 3, 3] references something to do with execfile? or tempfile.mkstemp? the leak is on the dummy key from dictobject.c!! very odd. see bug #808596 for the explanation. test_pkgimport leaked [6, 6, 6, 6] references no idea similar to the above? (but certainly not the same) I think this might be caused by interning like the above. test_sax leaked [1899, 1899, 1899, 1899] references no idea -- scary numbers, though! test_threadedtempfile leaked [0, 1, -21, 9] references randomized test? test_threading_local leaked [9, 0, 7, 0] references not sure what's going on here; not sure it's a leak, but it's definitely odd test_unicode leaked [7, 7, 7, 7] references the codec registry test_urllib2 leaked [120, -80, -70, 120] references CacheFTPHandler test_xrange leaked [1, 1, 1, 1] references fixed in CVS I'd really appreciate someone taking a look at test_sax. The numbers for test_gc and test_descr are artifacts and if someone sees how to fix them, I'd be grateful. test_threading_local behaves very strangely. test_distutils I haven't even begun to look at. Generally, we're in pretty good shape! (Or our tests really suck, your call ;-). Cheers, mwh -- I never disputed the Perl hacking skill of the Slashdot creators. My objections are to the editors' taste, the site's ugly visual design, and the Slashdot community's raging stupidity. -- http://www.cs.washington.edu/homes/klee/misc/slashdot.html#faq
Michael> I've hacked regrtest again (patch attached). Why not just check your regrtest changes into cvs? Skip
Skip Montanaro <skip@pobox.com> writes:
Michael> I've hacked regrtest again (patch attached).
Why not just check your regrtest changes into cvs?
Because I'm not quite happy with them; I hope to make the changes I want after the run I'm doing with -uall finishes (so, erm, Thursday or so :-). Also, there are entirely too many tests that fail when run with -R: test_builtin test_cgi test_decimal test_dircache test_doctest test_getopt test_logging test_multifile test_socket test_warnings test_zipimport (I meant to mention this in the first mail). I'll probably be filing bug reports on or trying to fix some of these -- tests that muck with the environment have a tendency to be a pain under any circumstances. Oh, and test_threaded_import still deadlocks. That's really annoying! Cheers, mwh -- MARVIN: What a depressingly stupid machine. -- The Hitch-Hikers Guide to the Galaxy, Episode 7
Michael Hudson wrote:
Skip Montanaro <skip@pobox.com> writes:
Michael> I've hacked regrtest again (patch attached).
Why not just check your regrtest changes into cvs?
Because I'm not quite happy with them; I hope to make the changes I want after the run I'm doing with -uall finishes (so, erm, Thursday or so :-).
I would prefer it, if the refcounting was done for each test method not only for the complete test. This would help in finding the leak but unfortunately would required changes to unittest. Bye, Walter Dörwald
Walter Dörwald <walter@livinglogic.de> writes:
Michael Hudson wrote:
Skip Montanaro <skip@pobox.com> writes:
Michael> I've hacked regrtest again (patch attached).
Why not just check your regrtest changes into cvs? Because I'm not quite happy with them; I hope to make the changes I want after the run I'm doing with -uall finishes (so, erm, Thursday or so :-).
I would prefer it, if the refcounting was done for each test method not only for the complete test. This would help in finding the leak but unfortunately would required changes to unittest.
Certainly, this is a superior solution. However, it also has the problem that from regrtest you can't tell whether a given test is using unittest, doctest or whatever. It's probably possible to redesign things so this is all elegant and natural, but I actually wanted to get somewhere with this :-) Cheers, mwh -- I have a feeling that any simple problem can be made arbitrarily difficult by imposing a suitably heavy administrative process around the development. -- Joe Armstrong, comp.lang.functional
On Mon, Aug 02, 2004 at 03:39:36PM +0100, Michael Hudson wrote:
test_sax leaked [1899, 1899, 1899, 1899] references no idea -- scary numbers, though!
It looks like the worst (only?) leakers are: test_expat_incomplete test_expat_inpsource_location Both of these cause exceptions to be raised. I don't know the pyexpat code at all. But I think I fixed a few leaks with the attached patch. Some leaks were caught by the tests, but I doubt all were. The patch is a starting point if anyone else wants to give it a shot. If not, I'll add it to SF or review carefully and check in later. Neal
On Tuesday 03 August 2004 12:20 am, Neal Norwitz wrote:
It looks like the worst (only?) leakers are:
test_expat_incomplete test_expat_inpsource_location
Both of these cause exceptions to be raised. I don't know the pyexpat code at all. But I think I fixed a few leaks with the attached patch. Some leaks were caught by the tests, but I doubt all were. The patch is a starting point if anyone else wants to give it a shot. If not, I'll add it to SF or review carefully and check in later.
Please post to SF and assign to me; I'll take a look sometime tomorrow. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org>
Neal Norwitz <neal@metaslash.com> writes:
On Mon, Aug 02, 2004 at 03:39:36PM +0100, Michael Hudson wrote:
test_sax leaked [1899, 1899, 1899, 1899] references no idea -- scary numbers, though!
It looks like the worst (only?) leakers are:
test_expat_incomplete test_expat_inpsource_location
Yes. The rest are all caused by the module level variable items containing it's old value...
Both of these cause exceptions to be raised. I don't know the pyexpat code at all. But I think I fixed a few leaks with the attached patch.
I certainly agree with your patch to set_error_attr (I had the same). Also, I think one needs to decref 'err' in set_error (PyErr_SetObject increfs its arguments).
Some leaks were caught by the tests, but I doubt all were.
Probably not :-/
The patch is a starting point if anyone else wants to give it a shot. If not, I'll add it to SF or review carefully and check in later.
Well, it seems Fred wants a look. I'll attach my diffs to pyexpat.c to the report (there's another fairly obvious leak exposed by test_minidom). Cheers, mwh --
I'm a little confused. That's because you're Australian! So all the blood flows to your head, away from the organ most normal guys think with. -- Mark Hammond & Tim Peters, comp.lang.python
Michael Hudson <mwh@python.net> writes:
Last time round I found a bunch of reference counting bugs in Python just *after* a major release; I'm trying to do it all before 2.4.0 this time.
I think I've now accounted for all the apparent and real refleaks revealed by the test suite (not with -uall, will try that soon). I've checked my regrtest patches in. I've fixed a bunch of tests that failed when repeated and I've filed bugs against those I couldn't figure out. Hopefully we can keep on top of these things! Cheers, mwh -- Also, remember to put the galaxy back when you've finished, or an angry mob of astronomers will come round and kneecap you with a small telescope for littering. -- Simon Tatham, ucam.chat, from Owen Dunn's review of the year
participants (5)
-
Fred L. Drake, Jr. -
Michael Hudson -
Neal Norwitz -
Skip Montanaro -
Walter Dörwald