On 2019-03-06 20:12, Matěj Cepl wrote:

Hi,

I am a lead maintainer of Python packages in OpenSUSE and I can see the pattern of many packagers adding blindly

python setup.py test

to %check section of our SPEC file. The problem is that if the package doesn't use unittest (it actually uses nose, pytest or something), it could lead to zero found tests, which pass and Python returns exit code 0 (success) even though nothing has been tested. It seems from the outside that everything is all right, package is being tested on every build, but actually it is lie.

Would it be possible to change unittest runner, so that when 0 tests pass, whole test suite would end up failing?

Thank you for considering this,

Strictly speaking, it's not a lie because none of the tests have failed.

On 3/6/2019 3:12 PM, Matěj Cepl wrote:

Hi,

I am a lead maintainer of Python packages in OpenSUSE and I can see the pattern of many packagers adding blindly

python setup.py test

to %check section of our SPEC file.

I am not familiar with setup.py, so I don't know how this affects the presence and contents of any particular files.

The problem is that if the package doesn't use unittest (it actually uses nose, pytest or something), it could lead to zero found tests,

Hence I don't know how unittest might be invoked in the situation you describe nor what output you see and whether you mean 0 test file(s) found or 0 test methods found or 0 lines of test code executed.

which pass and Python returns exit code 0 (success) even though nothing has been tested.

0 test methods does not mean 0 code executed in the tested module. Here is a possible minimal test file test_mod that is better than nothing. import mod import unittest class MinTest(unittest.TestCase): def setUp(self): self.instance = mod.MainClass()

It seems from the outside that everything is all right, package is being tested on every build, but actually it is lie.

Unless a test covers 100% of both lines *and* logic, 'success' never means 'everything is all right'.

Would it be possible to change unittest runner, so that when 0 tests pass, whole test suite would end up failing?

Yes, but unless a change were very narrow, and only affected the particular situation presented, it would be a bad idea. The unittest system is premised on 'success' rather than 'failure' being the default. 1. A test file may do better-that-nothing testing without running a test method. See above. Calling a mininal pass a 'fail' would be wrong. 2. A test file should skip everything when running on a system that cannot runs the tests. Several stdlib modules are OS-specific; their test modules skip all tests on some OS. There is no OS that can run every file in the Python test suite. Skipped test modules must not fail the test suite. IDLE and tkinter require graphics hardware and are then optional. IDLE depends on idlelib and tkinter. Tkinter depends on _tkinter and tcl/tk. Tk depends on having a graphic system, which servers and, in particular, *nix buildbots, generally lack. Again, skipped IDLE and tkinter test.test_x files must not fail a test suite. I agree that labeling the result of running a single test file can be problematical. The following could be either a 'SUCCESS' or 'FAIL', depending on what one wanted and expected. So one should read the detail and judge for oneself. 0:00:00 [1/1] test_idle test_idle skipped -- No module named 'idlelib' # or tkinter or ... test_idle skipped == Tests result: SUCCESS == 1 test skipped: test_idle Total duration: 109 ms Tests result: SUCCESS Unittest effectively assumes the context 'test file in test suite'. -- Terry Jan Reedy

On 2019-03-06 20:12, Matěj Cepl wrote:

Hi,

I am a lead maintainer of Python packages in OpenSUSE and I can see the pattern of many packagers adding blindly

python setup.py test

to %check section of our SPEC file. The problem is that if the package doesn't use unittest (it actually uses nose, pytest or something), it could lead to zero found tests, which pass and Python returns exit code 0 (success) even though nothing has been tested. It seems from the outside that everything is all right, package is being tested on every build, but actually it is lie.

Would it be possible to change unittest runner, so that when 0 tests pass, whole test suite would end up failing?

Thank you for considering this,

Strictly speaking, it's not a lie because none of the tests have failed.

On 3/6/2019 3:12 PM, Matěj Cepl wrote:

Hi,

I am a lead maintainer of Python packages in OpenSUSE and I can see the pattern of many packagers adding blindly

python setup.py test

to %check section of our SPEC file.

I am not familiar with setup.py, so I don't know how this affects the presence and contents of any particular files.

The problem is that if the package doesn't use unittest (it actually uses nose, pytest or something), it could lead to zero found tests,

Hence I don't know how unittest might be invoked in the situation you describe nor what output you see and whether you mean 0 test file(s) found or 0 test methods found or 0 lines of test code executed.

which pass and Python returns exit code 0 (success) even though nothing has been tested.

0 test methods does not mean 0 code executed in the tested module. Here is a possible minimal test file test_mod that is better than nothing. import mod import unittest class MinTest(unittest.TestCase): def setUp(self): self.instance = mod.MainClass()

It seems from the outside that everything is all right, package is being tested on every build, but actually it is lie.

Unless a test covers 100% of both lines *and* logic, 'success' never means 'everything is all right'.

Would it be possible to change unittest runner, so that when 0 tests pass, whole test suite would end up failing?

Yes, but unless a change were very narrow, and only affected the particular situation presented, it would be a bad idea. The unittest system is premised on 'success' rather than 'failure' being the default. 1. A test file may do better-that-nothing testing without running a test method. See above. Calling a mininal pass a 'fail' would be wrong. 2. A test file should skip everything when running on a system that cannot runs the tests. Several stdlib modules are OS-specific; their test modules skip all tests on some OS. There is no OS that can run every file in the Python test suite. Skipped test modules must not fail the test suite. IDLE and tkinter require graphics hardware and are then optional. IDLE depends on idlelib and tkinter. Tkinter depends on _tkinter and tcl/tk. Tk depends on having a graphic system, which servers and, in particular, *nix buildbots, generally lack. Again, skipped IDLE and tkinter test.test_x files must not fail a test suite. I agree that labeling the result of running a single test file can be problematical. The following could be either a 'SUCCESS' or 'FAIL', depending on what one wanted and expected. So one should read the detail and judge for oneself. 0:00:00 [1/1] test_idle test_idle skipped -- No module named 'idlelib' # or tkinter or ... test_idle skipped == Tests result: SUCCESS == 1 test skipped: test_idle Total duration: 109 ms Tests result: SUCCESS Unittest effectively assumes the context 'test file in test suite'. -- Terry Jan Reedy