[Cython] Hudson pyregr testing takes too long

Vitja Makarov vitja.makarov at gmail.com
Mon Apr 25 18:51:34 CEST 2011

2011/4/25 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 25.04.2011 11:04:
>> 2011/4/25 Stefan Behnel<stefan_ml at behnel.de>:
>>> Vitja Makarov, 25.04.2011 08:19:
>>>> 2011/4/25 Stefan Behnel:
>>>>> Stefan Behnel, 07.04.2011 13:52:
>>>>>> Stefan Behnel, 07.04.2011 13:46:
>>>>>>> I just noticed that the CPython pyregr tests have jumped up from ~14
>>>>>>> minutes for a run to ~4 hours when we added generator support.
>>>>>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py26-c/buildTimeTrend
>>>>>>> I currently have no idea why that is (well, it's likely because we
>>>>>>> compile
>>>>>>> more tests now, but Vitja's branch ran the tests in ~30 minutes). It
>>>>>>> would
>>>>>>> be great if someone could find the time to analyse this problem. The
>>>>>>> current run time makes it basically impossible to keep these tests
>>>>>>> enabled.
>>>>>> Ok, it looks like this is mostly an issue with the Py2.6 tests. The
>>>>>> Py2.7
>>>>>> tests take 30-45 minutes, which is very long, but not completely out
>>>>>> of
>>>>>> bounds. I've disabled the Py2.6 pyregr tests for now.
>>>>> There seems to be a huge memory leak which almost certainly accounts
>>>>> for
>>>>> this. The Python process that runs the pyregr suite ends up with about
>>>>> 50GB
>>>>> of memory at the end, also in the latest Py3k builds.
>>>>> I have no idea where it may be, but it started to show when we merged
>>>>> the
>>>>> generator support. That's where I noticed the instant jump in the
>>>>> runtime.
>>>> That's very strange for my branch it takes about 30 minutes that is ok.
>>> There's also a second path that's worth investigating. As part of the
>>> merge,
>>> there was another change that came in: the CythonPyregrTestCase
>>> implementation. This means that the regression tests are now being run
>>> differently than before. The massive memory consumption may simply be due
>>> to
>>> the mass of unit tests being loaded into memory.
>>    def run_test():
>> ..................................
>>         try:
>>             module = __import__(self.module)
>>             if hasattr(module, 'test_main'):
>>                 module.test_main()
>>         except (unittest.SkipTest, support.ResourceDenied):
>>             result.addSkip(self, 'ok')
>> It seems that all the modules stay loaded so may be they should be
>> unloaded with del sys.modules[module_name]?
> (Binary) module unloading isn't really supported in CPython. There's PEP
> 3121 that has the potential to change it, but it's not completely
> implemented, neither in CPython nor in Cython. A major problem is that
> unloading a module deletes its globals but not necessarily the code that
> uses them. For example, instances of types defined in the module can still
> be alive at that point.
> The way runtests.py deals with this is forking before loading a module.
> However, this does not currently work with the "xmlrunner" which we use on
> Hudson, so we let all tests run in a single process there.

Btw when running plain python code with generators total ref counter
doesn't get back to initial value.
I tried to trace scope and generator destructors and they are run as
expected. So I'm not sure about leaks in generators.


More information about the cython-devel mailing list