[Python-Dev] We got leaks!

[Kevin Jacobs]

  This message is in MIME format.  The first part should be readable text,
  while the remaining parts are likely unreadable without MIME-aware tools.
  Send mail to mime@docserver.cac.washington.edu for more info.

--416156269-869349433-1039224705=:12728
Content-Type: TEXT/PLAIN; charset=US-ASCII

On Fri, 6 Dec 2002, Kevin Jacobs wrote:
> Here it is:
> 
>   import sys
>   import cPickle
> 
>   dumped = cPickle.dumps(object())
>   for i in range(10):
>     cPickle.loads(dumped)
>     print '*' * 10, 'total refs:', sys.gettotalrefcount()
> 
> The leak occurs in cPickle.loads, but not in pickle.loads.  It also does not
> depend on binary or non-binary mode of dumps().  It seems to require a
> new-style object to occur, but not one that implements a custom hash
> function as in the previously reported leak.
> 
> This is all I have time for this evening... If the case(s) are not cracked
> tomorrow, I'll take a stab at dissecting the responsible C code.

I lied.  Attached is a patch for this leak and several others in cPickle.c
(the diff around line 3118 to Instance_New is the culprit for the above test
case).  Someone should check over my changes, since I'm getting a little
cross-eyed by now.  This takes care of all leaks from the Python test suite,
though there are still 2 ref-counts leaking per loop when using the C
version of datetime.

Enjoy,
-Kevin

PS: If Tim doesn't apply this patch fairly instantly, I will, of course,
    submit it to sourceforge.  blah... blah... blah...

--
Kevin Jacobs
The OPAL Group - Enterprise Systems Architect
Voice: (216) 986-0710 x 19         E-mail: jacobs@theopalgroup.com
Fax:   (216) 986-0714              WWW:    http://www.theopalgroup.com

--416156269-869349433-1039224705=:12728
Content-Type: TEXT/PLAIN; charset=US-ASCII; name="cpickle-refcnt.diff"
Content-Transfer-Encoding: BASE64
Content-ID: <Pine.LNX.4.44.0212062031450.12728@penguin.theopalgroup.com>
Content-Description: 
Content-Disposition: attachment; filename="cpickle-refcnt.diff"

SW5kZXg6IE1vZHVsZXMvY1BpY2tsZS5jDQo9PT09PT09PT09PT09PT09PT09
PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09
PT09DQpSQ1MgZmlsZTogL2N2c3Jvb3QvcHl0aG9uL3B5dGhvbi9kaXN0L3Ny
Yy9Nb2R1bGVzL2NQaWNrbGUuYyx2DQpyZXRyaWV2aW5nIHJldmlzaW9uIDIu
OTYNCmRpZmYgLXUgLXIyLjk2IGNQaWNrbGUuYw0KLS0tIE1vZHVsZXMvY1Bp
Y2tsZS5jCTE2IFNlcCAyMDAyIDE3OjI2OjIzIC0wMDAwCTIuOTYNCisrKyBN
b2R1bGVzL2NQaWNrbGUuYwk3IERlYyAyMDAyIDAxOjIxOjI4IC0wMDAwDQpA
QCAtMjI1LDEwICsyMjUsNiBAQA0KIAlyZXR1cm4gcjsNCiB9DQogDQotI2Rl
ZmluZSBQREFUQV9BUFBFTkRfKEQsTyxFUikgeyBcDQotICBpZiAoUGRhdGFf
QXBwZW5kKCgoUGRhdGEqKShEKSksIE8pIDwgMCkgcmV0dXJuIEVSOyBcDQot
fQ0KLQ0KICNkZWZpbmUgUERBVEFfQVBQRU5EKEQsTyxFUikgeyAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgIFwNCiAgICAgaWYgKCgoUGRhdGEq
KShEKSktPmxlbmd0aCA9PSAoKFBkYXRhKikoRCkpLT5zaXplICYmICAgICAg
ICBcDQogICAgICAgICBQZGF0YV9ncm93KChQZGF0YSopKEQpKSA8IDApICAg
ICAgICAgICAgICAgICAgICAgICAgICAgXA0KQEAgLTg5Nyw2ICs4OTMsNyBA
QA0KIAkJaWYgKGtleSA9PSBOVUxMKQ0KIAkJCXJldHVybiAwOw0KIAkJaWYg
KFB5RGljdF9HZXRJdGVtKHNlbGYtPmZhc3RfbWVtbywga2V5KSkgew0KKwkJ
CVB5X0RFQ1JFRihrZXkpOw0KIAkJCVB5RXJyX0Zvcm1hdChQeUV4Y19WYWx1
ZUVycm9yLA0KIAkJCQkgICAgICJmYXN0IG1vZGU6IGNhbid0IHBpY2tsZSBj
eWNsaWMgb2JqZWN0cyBpbmNsdWRpbmcgb2JqZWN0IHR5cGUgJXMgYXQgJXAi
LA0KIAkJCQkgICAgIG9iai0+b2JfdHlwZS0+dHBfbmFtZSwgb2JqKTsNCkBA
IC05MDQsOSArOTAxLDExIEBADQogCQkJcmV0dXJuIDA7DQogCQl9DQogCQlp
ZiAoUHlEaWN0X1NldEl0ZW0oc2VsZi0+ZmFzdF9tZW1vLCBrZXksIFB5X05v
bmUpIDwgMCkgew0KKwkJCVB5X0RFQ1JFRihrZXkpOw0KIAkJCXNlbGYtPmZh
c3RfY29udGFpbmVyID0gLTE7DQogCQkJcmV0dXJuIDA7DQogCQl9DQorCQlQ
eV9ERUNSRUYoa2V5KTsNCiAJfQ0KIAlyZXR1cm4gMTsNCiB9DQpAQCAtOTE5
LDggKzkxOCwxMCBAQA0KIAkJaWYgKGtleSA9PSBOVUxMKQ0KIAkJCXJldHVy
biAwOw0KIAkJaWYgKFB5RGljdF9EZWxJdGVtKHNlbGYtPmZhc3RfbWVtbywg
a2V5KSA8IDApIHsNCisJCQlQeV9ERUNSRUYoa2V5KTsNCiAJCQlyZXR1cm4g
MDsNCiAJCX0NCisJCVB5X0RFQ1JFRihrZXkpOw0KIAl9DQogCXJldHVybiAx
Ow0KIH0NCkBAIC0zMTE1LDYgKzMxMTYsNyBAQA0KIAkJCVB5X1hERUNSRUYo
c2FmZSk7DQogCQkJcmV0dXJuIE5VTEw7DQogCQl9DQorCQlQeV9ERUNSRUYo
c2FmZSk7DQogCX0NCiANCiAJaWYgKGFyZ3M9PVB5X05vbmUpIHsNCkBAIC0z
MjE4LDcgKzMyMjAsMTAgQEANCiAJaWYgKCFtb2R1bGVfbmFtZSkgIHJldHVy
biAtMTsNCiANCiAJaWYgKChsZW4gPSAoKnNlbGYtPnJlYWRsaW5lX2Z1bmMp
KHNlbGYsICZzKSkgPj0gMCkgew0KLQkJaWYgKGxlbiA8IDIpIHJldHVybiBi
YWRfcmVhZGxpbmUoKTsNCisJCWlmIChsZW4gPCAyKSB7DQorCQkJUHlfREVD
UkVGKG1vZHVsZV9uYW1lKTsNCisJCQlyZXR1cm4gYmFkX3JlYWRsaW5lKCk7
DQorCQl9DQogCQlpZiAoKGNsYXNzX25hbWUgPSBQeVN0cmluZ19Gcm9tU3Ry
aW5nQW5kU2l6ZShzLCBsZW4gLSAxKSkpIHsNCiAJCQljbGFzcyA9IGZpbmRf
Y2xhc3MobW9kdWxlX25hbWUsIGNsYXNzX25hbWUsIA0KIAkJCQkJICAgc2Vs
Zi0+ZmluZF9jbGFzcyk7DQo=
--416156269-869349433-1039224705=:12728--