[New-bugs-announce] [issue35327] Using skipTest with subTest gives a misleading UI.

[Paul Ganssle]

New submission from Paul Ganssle <p.ganssle at gmail.com>:

It seems that if you call skipTest *anywhere* in a test function, even in a subTest, the *entire* function gets marked as "skipped", even if only one sub-test is skipped.

Example:

    import unittest

    class SomeTest(unittest.TestCase):
        def test_something(self):
            for i in range(1, 3):
                with self.subTest(i):
                    if i > 1:
                        self.skipTest('Not supported')

                    self.assertEqual(i, 1)

If you run `python3.7 -m unittest -v` on this, you get:

    $ python -m unittest -v
    test_something (test_mod.SomeTest) ... skipped 'Not supported'

    ----------------------------------------------------------------------
    Ran 1 test in 0.000s

    OK (skipped=1)


Despite the fact that the test *was* run in the `i == 1` case. Similarly, pytest marks this as a single skipped test:

    ========= test session starts =======
    
    platform linux -- Python 3.7.1, pytest-4.0.1, py-1.7.0, pluggy-0.8.0
    rootdir: /tmp/test_mod, inifile:
    collected 1 item
    
    test_mod.py s                                           [100%]
    
    ===== 1 skipped in 0.00 seconds =====


The solution to this is not obvious, unfortunately. One way to solve it would be to have each subtest considered a "separate test". Another would be to detect whether subTests have been skipped and expose only the *skipped* tests as separate tests. You could also add a "partially skipped" marker, so to say, "Some parts of this were skipped"

I suspect the right answer is some combination of these three answers - possibly adding an extra-verbose mode that splits out all sub tests and having skipped subTests default to being displayed separately.

Somewhat related to issue #30997.

----------
components: Tests
messages: 330527
nosy: ezio.melotti, michael.foord, p-ganssle, rbcollins
priority: normal
severity: normal
status: open
title: Using skipTest with subTest gives a misleading UI.
versions: Python 3.6, Python 3.7, Python 3.8

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue35327>
_______________________________________