[Cython] Hudson pyregr testing takes too long
stefan_ml at behnel.de
Mon Apr 25 09:20:26 CEST 2011
Vitja Makarov, 25.04.2011 08:19:
> 2011/4/25 Stefan Behnel:
>> Stefan Behnel, 07.04.2011 13:52:
>>> Stefan Behnel, 07.04.2011 13:46:
>>>> I just noticed that the CPython pyregr tests have jumped up from ~14
>>>> minutes for a run to ~4 hours when we added generator support.
>>>> I currently have no idea why that is (well, it's likely because we
>>>> more tests now, but Vitja's branch ran the tests in ~30 minutes). It
>>>> be great if someone could find the time to analyse this problem. The
>>>> current run time makes it basically impossible to keep these tests
>>> Ok, it looks like this is mostly an issue with the Py2.6 tests. The Py2.7
>>> tests take 30-45 minutes, which is very long, but not completely out of
>>> bounds. I've disabled the Py2.6 pyregr tests for now.
>> There seems to be a huge memory leak which almost certainly accounts for
>> this. The Python process that runs the pyregr suite ends up with about 50GB
>> of memory at the end, also in the latest Py3k builds.
>> I have no idea where it may be, but it started to show when we merged the
>> generator support. That's where I noticed the instant jump in the runtime.
> That's very strange for my branch it takes about 30 minutes that is ok.
Does your branch leak memory when you run a generator? Using a debug build
of CPython, running a Cython generator for the first time persistently
increases the reference counts for me (using the generators.pyx test):
Python 2.7.1+ (2.7:c821d3d335e8, Apr 22 2011, 18:37:12)
[GCC 4.4.3] on linux2
>>> import generators as G
>>> import gc
[1, 2, 3, 4]
It seems like this leaked five references. And it seems to be related to
the generator being inside of a closure itself:
[5, 10, (5, 10)]
back to the last value here, but:
[2, 3, 4, 5, 6]
Another six references leaked. And it's only the first time the generator
is run, running it a second time doesn't change anything:
[2, 3, 4, 5, 6]
More information about the cython-devel