[Python-ideas] unittest: 0 tests pass means failure of the testsuite

[Terry Reedy]

On 3/6/2019 3:12 PM, Matěj Cepl wrote:
> Hi,
> 
> I am a lead maintainer of Python packages in OpenSUSE and I can
> see the pattern of many packagers adding blindly
> 
>      python setup.py test
> 
> to %check section of our SPEC file.

I am not familiar with setup.py, so I don't know how this affects the 
presence and contents of any particular files.

> The problem is that if the
> package doesn't use unittest (it actually uses nose, pytest or
> something), it could lead to zero found tests,

Hence I don't know how unittest might be invoked in the situation you 
describe nor what output you see and whether you mean 0 test file(s) 
found or 0 test methods found or 0 lines of test code executed.

> which pass and
> Python returns exit code 0 (success) even though nothing has been
> tested.

0 test methods does not mean 0 code executed in the tested module.  Here 
is a possible minimal test file test_mod that is better than nothing.

import mod
import unittest

class MinTest(unittest.TestCase):
     def setUp(self):
         self.instance = mod.MainClass()

> It seems from the outside that everything is all right,
> package is being tested on every build, but actually it is lie.

Unless a test covers 100% of both lines *and* logic, 'success' never 
means 'everything is all right'.

> Would it be possible to change unittest runner, so that when 0
> tests pass, whole test suite would end up failing?

Yes, but unless a change were very narrow, and only affected the 
particular situation presented, it would be a bad idea.  The unittest 
system is premised on 'success' rather than 'failure' being the default.

1. A test file may do better-that-nothing testing without running a test 
method.  See above. Calling a mininal pass a 'fail' would be wrong.

2. A test file should skip everything when running on a system that 
cannot runs the tests.  Several stdlib modules are OS-specific; their 
test modules skip all tests on some OS.  There is no OS that can run 
every file in the Python test suite.  Skipped test modules must not fail 
the test suite.

IDLE and tkinter require graphics hardware and are then optional.  IDLE 
depends on idlelib and tkinter.  Tkinter depends on _tkinter and tcl/tk. 
  Tk depends on having a graphic system, which servers and, in 
particular, *nix buildbots, generally lack.  Again, skipped IDLE and 
tkinter test.test_x files must not fail a test suite.

I agree that labeling the result of running a single test file can be 
problematical.  The following could be either a 'SUCCESS' or 'FAIL', 
depending on what one wanted and expected.  So one should read the 
detail and judge for oneself.

0:00:00 [1/1] test_idle
test_idle skipped -- No module named 'idlelib'  # or tkinter or ...
test_idle skipped

== Tests result: SUCCESS ==
1 test skipped:
     test_idle

Total duration: 109 ms
Tests result: SUCCESS

Unittest effectively assumes the context 'test file in test suite'.

-- 
Terry Jan Reedy