Re: Separation of tests into unit test and functional tests
Hi Holger,
That's a good point.
On Mon, Oct 19, 2009 at 6:55 PM, SirVer sirver@gmx.de wrote:
Hi,
I have something of a lot of importance to mention. We should start to separate our tests into unit and integration tests. For what reason? The 40 tests need 3.256 seconds on my box to run; that's approximately the time it takes to compile the opencv module here. So compiling + tests = 2 * compiling. That's still acceptable, but 40 tests is nothing. 400 tests will need 30 seconds which is too much to run after each edit.
The main thing I think is to be careful with images. The slowest one by far now are the tests for lpi_filter, because they do a lot of things with a *large* image.
This is why I originally put my test images for io under io/tests. I thought data_dir should only contain images for examples, useful for users. Images for testing algorithms can often be small. For the color conversion I now use images of size (4, 2, 3), which means they take no time at all.
But that is was unittests are for. The reason why I come up with this now is that it is important to make this separation while it is still easy and possible. The best way is to group tests into many groups ('opencv', 'fast', 'slow', 'need_camera').
What is lacking when you apply decorators like @opencv_skip and @slow? Note also that if you run nosetests from some folder you run only tests in subfolders. This is a natural separation already.
A short blog post concerning this is:
http://beust.com/weblog/archives/000319.html
That's mostly semantics.
I have no idea how to achieve this with pynose in a simple way though,
Once again: I think scikit_image will grow fast and therefore test will grow fast. Users will stop running the tests if they take to long. Too long for an interactive coding session is > 10s .
True. The @slow decorator should take care of this. What is the approximate limit again when it should be used?
Cheers, Ralf
Cheers, Holger
2009/10/19 Ralf Gommers ralf.gommers@googlemail.com:
The main thing I think is to be careful with images. The slowest one by far now are the tests for lpi_filter, because they do a lot of things with a *large* image.
Also, the kernel size is currently set to the shape of the image, which is not necessary. I'd also like the filtering code to take a kernel as a parameter, instead of a function. I've placed it on the tasks list.
This is why I originally put my test images for io under io/tests. I thought data_dir should only contain images for examples, useful for users. Images for testing algorithms can often be small. For the color conversion I now use images of size (4, 2, 3), which means they take no time at all.
The idea is for each package to use the same pool of images, so that we don't have to include data-files in each sub-package. If needed, we could put them in subdirectories under data to show what they were intended for.
What is lacking when you apply decorators like @opencv_skip and @slow? Note also that if you run nosetests from some folder you run only tests in subfolders. This is a natural separation already.
I see one danger in using @slow: people stop executing anything but the fast tests. In such a case, it may simply be worth speeding up the slow test, if possible.
Ralf gives a good tip for developing: run only the relevant tests, e.g.,
nosetests scikits.image.opencv
Cheers Stéfan
2009/10/20 Stéfan van der Walt stefan@sun.ac.za
2009/10/19 Ralf Gommers ralf.gommers@googlemail.com:
This is why I originally put my test images for io under io/tests. I
thought
data_dir should only contain images for examples, useful for users.
Images
for testing algorithms can often be small. For the color conversion I now use images of size (4, 2, 3), which means they take no time at all.
The idea is for each package to use the same pool of images, so that we don't have to include data-files in each sub-package. If needed, we could put them in subdirectories under data to show what they were intended for.
Sure that makes sense. It would be good if data_dir has only "nice" images.
I can rearrange them today, either in one subdir per module or simply one new data_dir/testdata.
Cheers, Ralf
Cheers Stéfan
Hi,
What is lacking when you apply decorators like @opencv_skip and @slow? Note also that if you run nosetests from some folder you run only tests in subfolders. This is a natural separation already.
Ahh, now i understood how to select which tests to run with the nosetest script. I finally figured out -a and -A
I see one danger in using @slow: people stop executing anything but the fast tests. In such a case, it may simply be worth speeding up the slow test, if possible.
I started to use @dec.slow now for every test that needs > 0.1 s on my system. They are still run by default but can be left out. It is wise to run the complete test suite (of all modules, not only those you worked on) before commiting ,).
Cheers, Holger
participants (3)
-
Ralf Gommers -
SirVer -
Stéfan van der Walt