Hi, I just noticed that the CPython pyregr tests have jumped up from ~14 minutes for a run to ~4 hours when we added generator support. https://sage.math.washington.edu:8091/hudson/view/cython-devel/job/cython-de... I currently have no idea why that is (well, it's likely because we compile more tests now, but Vitja's branch ran the tests in ~30 minutes). It would be great if someone could find the time to analyse this problem. The current run time makes it basically impossible to keep these tests enabled. Stefan
Stefan Behnel, 07.04.2011 13:46:
I just noticed that the CPython pyregr tests have jumped up from ~14 minutes for a run to ~4 hours when we added generator support.
https://sage.math.washington.edu:8091/hudson/view/cython-devel/job/cython-de...
I currently have no idea why that is (well, it's likely because we compile more tests now, but Vitja's branch ran the tests in ~30 minutes). It would be great if someone could find the time to analyse this problem. The current run time makes it basically impossible to keep these tests enabled.
Ok, it looks like this is mostly an issue with the Py2.6 tests. The Py2.7 tests take 30-45 minutes, which is very long, but not completely out of bounds. I've disabled the Py2.6 pyregr tests for now. Stefan
Stefan Behnel, 07.04.2011 13:52:
Stefan Behnel, 07.04.2011 13:46:
I just noticed that the CPython pyregr tests have jumped up from ~14 minutes for a run to ~4 hours when we added generator support.
https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-p...
I currently have no idea why that is (well, it's likely because we compile more tests now, but Vitja's branch ran the tests in ~30 minutes). It would be great if someone could find the time to analyse this problem. The current run time makes it basically impossible to keep these tests enabled.
Ok, it looks like this is mostly an issue with the Py2.6 tests. The Py2.7 tests take 30-45 minutes, which is very long, but not completely out of bounds. I've disabled the Py2.6 pyregr tests for now.
There seems to be a huge memory leak which almost certainly accounts for this. The Python process that runs the pyregr suite ends up with about 50GB of memory at the end, also in the latest Py3k builds. I have no idea where it may be, but it started to show when we merged the generator support. That's where I noticed the instant jump in the runtime. Stefan
2011/4/25 Stefan Behnel <stefan_ml@behnel.de>:
Stefan Behnel, 07.04.2011 13:52:
Stefan Behnel, 07.04.2011 13:46:
I just noticed that the CPython pyregr tests have jumped up from ~14 minutes for a run to ~4 hours when we added generator support.
https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-p...
I currently have no idea why that is (well, it's likely because we compile more tests now, but Vitja's branch ran the tests in ~30 minutes). It would be great if someone could find the time to analyse this problem. The current run time makes it basically impossible to keep these tests enabled.
Ok, it looks like this is mostly an issue with the Py2.6 tests. The Py2.7 tests take 30-45 minutes, which is very long, but not completely out of bounds. I've disabled the Py2.6 pyregr tests for now.
There seems to be a huge memory leak which almost certainly accounts for this. The Python process that runs the pyregr suite ends up with about 50GB of memory at the end, also in the latest Py3k builds.
I have no idea where it may be, but it started to show when we merged the generator support. That's where I noticed the instant jump in the runtime.
That's very strange for my branch it takes about 30 minutes that is ok. -- vitja.
Vitja Makarov, 25.04.2011 08:19:
2011/4/25 Stefan Behnel:
Stefan Behnel, 07.04.2011 13:52:
Stefan Behnel, 07.04.2011 13:46:
I just noticed that the CPython pyregr tests have jumped up from ~14 minutes for a run to ~4 hours when we added generator support.
https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-p...
I currently have no idea why that is (well, it's likely because we compile more tests now, but Vitja's branch ran the tests in ~30 minutes). It would be great if someone could find the time to analyse this problem. The current run time makes it basically impossible to keep these tests enabled.
Ok, it looks like this is mostly an issue with the Py2.6 tests. The Py2.7 tests take 30-45 minutes, which is very long, but not completely out of bounds. I've disabled the Py2.6 pyregr tests for now.
There seems to be a huge memory leak which almost certainly accounts for this. The Python process that runs the pyregr suite ends up with about 50GB of memory at the end, also in the latest Py3k builds.
I have no idea where it may be, but it started to show when we merged the generator support. That's where I noticed the instant jump in the runtime.
That's very strange for my branch it takes about 30 minutes that is ok.
Does your branch leak memory when you run a generator? Using a debug build of CPython, running a Cython generator for the first time persistently increases the reference counts for me (using the generators.pyx test): Python 2.7.1+ (2.7:c821d3d335e8, Apr 22 2011, 18:37:12) [GCC 4.4.3] on linux2 >>> import generators as G [17021 refs] >>> import gc [17462 refs] >>> gc.collect() 0 [17465 refs] >>> gc.collect() 0 [17465 refs] >>> list(G.with_outer(1,2,3,4)()) [1, 2, 3, 4] [17474 refs] >>> gc.collect() 0 [17470 refs] It seems like this leaked five references. And it seems to be related to the generator being inside of a closure itself:
list(G.test_first_assignment()) [5, 10, (5, 10)] [17475 refs] gc.collect() 0 [17470 refs]
back to the last value here, but:
list(G.generator_nonlocal()(5)) [2, 3, 4, 5, 6] [17481 refs] gc.collect() 0 [17476 refs]
Another six references leaked. And it's only the first time the generator is run, running it a second time doesn't change anything:
list(G.generator_nonlocal()(5)) [2, 3, 4, 5, 6] [17481 refs] gc.collect() 0 [17476 refs]
Stefan
Vitja Makarov, 25.04.2011 08:19:
2011/4/25 Stefan Behnel:
Stefan Behnel, 07.04.2011 13:52:
Stefan Behnel, 07.04.2011 13:46:
I just noticed that the CPython pyregr tests have jumped up from ~14 minutes for a run to ~4 hours when we added generator support.
https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-p...
I currently have no idea why that is (well, it's likely because we compile more tests now, but Vitja's branch ran the tests in ~30 minutes). It would be great if someone could find the time to analyse this problem. The current run time makes it basically impossible to keep these tests enabled.
Ok, it looks like this is mostly an issue with the Py2.6 tests. The Py2.7 tests take 30-45 minutes, which is very long, but not completely out of bounds. I've disabled the Py2.6 pyregr tests for now.
There seems to be a huge memory leak which almost certainly accounts for this. The Python process that runs the pyregr suite ends up with about 50GB of memory at the end, also in the latest Py3k builds.
I have no idea where it may be, but it started to show when we merged the generator support. That's where I noticed the instant jump in the runtime.
That's very strange for my branch it takes about 30 minutes that is ok.
There's also a second path that's worth investigating. As part of the merge, there was another change that came in: the CythonPyregrTestCase implementation. This means that the regression tests are now being run differently than before. The massive memory consumption may simply be due to the mass of unit tests being loaded into memory. Stefan
2011/4/25 Stefan Behnel <stefan_ml@behnel.de>:
Vitja Makarov, 25.04.2011 08:19:
2011/4/25 Stefan Behnel:
Stefan Behnel, 07.04.2011 13:52:
Stefan Behnel, 07.04.2011 13:46:
I just noticed that the CPython pyregr tests have jumped up from ~14 minutes for a run to ~4 hours when we added generator support.
https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-p...
I currently have no idea why that is (well, it's likely because we compile more tests now, but Vitja's branch ran the tests in ~30 minutes). It would be great if someone could find the time to analyse this problem. The current run time makes it basically impossible to keep these tests enabled.
Ok, it looks like this is mostly an issue with the Py2.6 tests. The Py2.7 tests take 30-45 minutes, which is very long, but not completely out of bounds. I've disabled the Py2.6 pyregr tests for now.
There seems to be a huge memory leak which almost certainly accounts for this. The Python process that runs the pyregr suite ends up with about 50GB of memory at the end, also in the latest Py3k builds.
I have no idea where it may be, but it started to show when we merged the generator support. That's where I noticed the instant jump in the runtime.
That's very strange for my branch it takes about 30 minutes that is ok.
There's also a second path that's worth investigating. As part of the merge, there was another change that came in: the CythonPyregrTestCase implementation. This means that the regression tests are now being run differently than before. The massive memory consumption may simply be due to the mass of unit tests being loaded into memory.
def run_test(): .................................. try: module = __import__(self.module) if hasattr(module, 'test_main'): module.test_main() except (unittest.SkipTest, support.ResourceDenied): result.addSkip(self, 'ok') It seems that all the modules stay loaded so may be they should be unloaded with del sys.modules[module_name]? -- vitja.
Vitja Makarov, 25.04.2011 11:04:
2011/4/25 Stefan Behnel<stefan_ml@behnel.de>:
Vitja Makarov, 25.04.2011 08:19:
2011/4/25 Stefan Behnel:
Stefan Behnel, 07.04.2011 13:52:
Stefan Behnel, 07.04.2011 13:46:
I just noticed that the CPython pyregr tests have jumped up from ~14 minutes for a run to ~4 hours when we added generator support.
https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-p...
I currently have no idea why that is (well, it's likely because we compile more tests now, but Vitja's branch ran the tests in ~30 minutes). It would be great if someone could find the time to analyse this problem. The current run time makes it basically impossible to keep these tests enabled.
Ok, it looks like this is mostly an issue with the Py2.6 tests. The Py2.7 tests take 30-45 minutes, which is very long, but not completely out of bounds. I've disabled the Py2.6 pyregr tests for now.
There seems to be a huge memory leak which almost certainly accounts for this. The Python process that runs the pyregr suite ends up with about 50GB of memory at the end, also in the latest Py3k builds.
I have no idea where it may be, but it started to show when we merged the generator support. That's where I noticed the instant jump in the runtime.
That's very strange for my branch it takes about 30 minutes that is ok.
There's also a second path that's worth investigating. As part of the merge, there was another change that came in: the CythonPyregrTestCase implementation. This means that the regression tests are now being run differently than before. The massive memory consumption may simply be due to the mass of unit tests being loaded into memory.
def run_test(): .................................. try: module = __import__(self.module) if hasattr(module, 'test_main'): module.test_main() except (unittest.SkipTest, support.ResourceDenied): result.addSkip(self, 'ok')
It seems that all the modules stay loaded so may be they should be unloaded with del sys.modules[module_name]?
(Binary) module unloading isn't really supported in CPython. There's PEP 3121 that has the potential to change it, but it's not completely implemented, neither in CPython nor in Cython. A major problem is that unloading a module deletes its globals but not necessarily the code that uses them. For example, instances of types defined in the module can still be alive at that point. The way runtests.py deals with this is forking before loading a module. However, this does not currently work with the "xmlrunner" which we use on Hudson, so we let all tests run in a single process there. Stefan
2011/4/25 Stefan Behnel <stefan_ml@behnel.de>:
Vitja Makarov, 25.04.2011 11:04:
2011/4/25 Stefan Behnel<stefan_ml@behnel.de>:
Vitja Makarov, 25.04.2011 08:19:
2011/4/25 Stefan Behnel:
Stefan Behnel, 07.04.2011 13:52:
Stefan Behnel, 07.04.2011 13:46: > > I just noticed that the CPython pyregr tests have jumped up from ~14 > minutes for a run to ~4 hours when we added generator support. > > > > > https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-p... > > I currently have no idea why that is (well, it's likely because we > compile > more tests now, but Vitja's branch ran the tests in ~30 minutes). It > would > be great if someone could find the time to analyse this problem. The > current run time makes it basically impossible to keep these tests > enabled.
Ok, it looks like this is mostly an issue with the Py2.6 tests. The Py2.7 tests take 30-45 minutes, which is very long, but not completely out of bounds. I've disabled the Py2.6 pyregr tests for now.
There seems to be a huge memory leak which almost certainly accounts for this. The Python process that runs the pyregr suite ends up with about 50GB of memory at the end, also in the latest Py3k builds.
I have no idea where it may be, but it started to show when we merged the generator support. That's where I noticed the instant jump in the runtime.
That's very strange for my branch it takes about 30 minutes that is ok.
There's also a second path that's worth investigating. As part of the merge, there was another change that came in: the CythonPyregrTestCase implementation. This means that the regression tests are now being run differently than before. The massive memory consumption may simply be due to the mass of unit tests being loaded into memory.
def run_test(): .................................. try: module = __import__(self.module) if hasattr(module, 'test_main'): module.test_main() except (unittest.SkipTest, support.ResourceDenied): result.addSkip(self, 'ok')
It seems that all the modules stay loaded so may be they should be unloaded with del sys.modules[module_name]?
(Binary) module unloading isn't really supported in CPython. There's PEP 3121 that has the potential to change it, but it's not completely implemented, neither in CPython nor in Cython. A major problem is that unloading a module deletes its globals but not necessarily the code that uses them. For example, instances of types defined in the module can still be alive at that point.
The way runtests.py deals with this is forking before loading a module. However, this does not currently work with the "xmlrunner" which we use on Hudson, so we let all tests run in a single process there.
Btw when running plain python code with generators total ref counter doesn't get back to initial value. I tried to trace scope and generator destructors and they are run as expected. So I'm not sure about leaks in generators. -- vitja.
2011/4/25 Vitja Makarov <vitja.makarov@gmail.com>:
2011/4/25 Stefan Behnel <stefan_ml@behnel.de>:
Vitja Makarov, 25.04.2011 11:04:
2011/4/25 Stefan Behnel<stefan_ml@behnel.de>:
Vitja Makarov, 25.04.2011 08:19:
2011/4/25 Stefan Behnel:
Stefan Behnel, 07.04.2011 13:52: > > Stefan Behnel, 07.04.2011 13:46: >> >> I just noticed that the CPython pyregr tests have jumped up from ~14 >> minutes for a run to ~4 hours when we added generator support. >> >> >> >> >> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-p... >> >> I currently have no idea why that is (well, it's likely because we >> compile >> more tests now, but Vitja's branch ran the tests in ~30 minutes). It >> would >> be great if someone could find the time to analyse this problem. The >> current run time makes it basically impossible to keep these tests >> enabled. > > Ok, it looks like this is mostly an issue with the Py2.6 tests. The > Py2.7 > tests take 30-45 minutes, which is very long, but not completely out > of > bounds. I've disabled the Py2.6 pyregr tests for now.
There seems to be a huge memory leak which almost certainly accounts for this. The Python process that runs the pyregr suite ends up with about 50GB of memory at the end, also in the latest Py3k builds.
I have no idea where it may be, but it started to show when we merged the generator support. That's where I noticed the instant jump in the runtime.
That's very strange for my branch it takes about 30 minutes that is ok.
There's also a second path that's worth investigating. As part of the merge, there was another change that came in: the CythonPyregrTestCase implementation. This means that the regression tests are now being run differently than before. The massive memory consumption may simply be due to the mass of unit tests being loaded into memory.
def run_test(): .................................. try: module = __import__(self.module) if hasattr(module, 'test_main'): module.test_main() except (unittest.SkipTest, support.ResourceDenied): result.addSkip(self, 'ok')
It seems that all the modules stay loaded so may be they should be unloaded with del sys.modules[module_name]?
(Binary) module unloading isn't really supported in CPython. There's PEP 3121 that has the potential to change it, but it's not completely implemented, neither in CPython nor in Cython. A major problem is that unloading a module deletes its globals but not necessarily the code that uses them. For example, instances of types defined in the module can still be alive at that point.
The way runtests.py deals with this is forking before loading a module. However, this does not currently work with the "xmlrunner" which we use on Hudson, so we let all tests run in a single process there.
Btw when running plain python code with generators total ref counter doesn't get back to initial value. I tried to trace scope and generator destructors and they are run as expected. So I'm not sure about leaks in generators.
Recently I've found that pyregr.test_dict (test_mutatingiteration) test makes it slow: def test_mutatingiteration(): d = {} d[1] = 1 for i in d: print i d[i+1] = 1 test_mutatingiteration() In CPython this code raises: RuntimeError: dictionary changed size during iteration And in Cython you have infinite loop. So we can disable this test for now. -- vitja.
Vitja Makarov, 05.05.2011 08:41:
Recently I've found that pyregr.test_dict (test_mutatingiteration) test makes it slow:
def test_mutatingiteration(): d = {} d[1] = 1 for i in d: print i d[i+1] = 1
test_mutatingiteration()
In CPython this code raises: RuntimeError: dictionary changed size during iteration And in Cython you have infinite loop. So we can disable this test for now.
Interesting. Yes, if it's a bug in Cython then it's worth disabling the test until the bug is fixed. Anyway, I'll try to come up with a fix quickly. Stefan
2011/5/5 Stefan Behnel <stefan_ml@behnel.de>:
Vitja Makarov, 05.05.2011 08:41:
Recently I've found that pyregr.test_dict (test_mutatingiteration) test makes it slow:
def test_mutatingiteration(): d = {} d[1] = 1 for i in d: print i d[i+1] = 1
test_mutatingiteration()
In CPython this code raises: RuntimeError: dictionary changed size during iteration And in Cython you have infinite loop. So we can disable this test for now.
Interesting. Yes, if it's a bug in Cython then it's worth disabling the test until the bug is fixed.
Anyway, I'll try to come up with a fix quickly.
Cool! It seems that you've fixed the issue and now pyregr takes about 40 minutes ) -- vitja.
Vitja Makarov, 05.05.2011 16:29:
2011/5/5 Stefan Behnel:
Vitja Makarov, 05.05.2011 08:41:
Recently I've found that pyregr.test_dict (test_mutatingiteration) test makes it slow:
def test_mutatingiteration(): d = {} d[1] = 1 for i in d: print i d[i+1] = 1
test_mutatingiteration()
In CPython this code raises: RuntimeError: dictionary changed size during iteration And in Cython you have infinite loop. So we can disable this test for now.
Interesting. Yes, if it's a bug in Cython then it's worth disabling the test until the bug is fixed.
Anyway, I'll try to come up with a fix quickly.
Cool! It seems that you've fixed the issue and now pyregr takes about 40 minutes )
Yes. :) I've also disabled a couple of other pyregr tests that took more than one minute due to different reasons. For example, garbage collection tends to take ages towards the end of the test run, when to many modules are loaded. Or the subprocess tests take a while, although they are everything but interesting for Cython. Stefan
participants (2)
-
Stefan Behnel -
Vitja Makarov