[pytest-dev] process for replacing yield tests with modern style

[holger krekel]

Hi Chris, all,

maybe there is a solution ...

On Fri, Oct 21, 2016 at 13:03 +0100, Chris Dent wrote:
> Several years ago, when I started using pytest, the yield tests were
> my favorite thing about it. Super simple: push out functions that
> have assertions in them. That's what pytest was all about then, and
> it was _glorious_. In the intervening years things have become more
> complex. That's the nature of things.

I don't consider the discussion or development fully finished FWIW.

IIRC i came up with the idea of using yield for generating tests (also thus termed "generative tests").  I liked the simplicity of generating parametrized tests with it, each nicely represented by the typical ".", the primary conceptual unit of testing. 

The main problem with yield-generated tests is that we call xUnit-setup/teardown methods which means we need to handle them during collection.  Also it complicates the "Function" item, the builtin test item class which describes a test function, its parameters and handles it execution and failure representation.  The complications could probably be reduced by separating some code out into a "YieldedFunction".  But the former is a more fundamental problem and the reason why documentation for yield was mostly removed and there is a long standing recommendation to move towards the parametrize decorator or pytest_generate_tests if you need more dynamic or complex parametrization.

So today we have three mechanisms for generating tests which don't require a conftest.py:

- test functions using yield to generate func+param (deprecated)

- @pytest.mark.parametrize-decorated test functions which are executed
  repeatedly with multiple argument sets.

- the pytest_generate_tests(metafunc) hook which allows to parametrize function
  arguments or fixtures which will execute a test function repeatedly 
  with multiple argument sets. You can use this hook in modules and classes.

The decorator is btw implemented through a builtin pytest_generate_tests hook implementation and there can only be one per each module or class.  

So let's now get to your original example at
https://github.com/tiddlyweb/tiddlyweb/blob/master/test/http_runner.py

The parametrize-relevant bits read:

    def test_the_TESTS():
        for test_data in tests:
            test = dict(EMPTY_TEST)
            test.update(test_data)
            yield test['name'], _run_test, test

    def _run_test(test):
        ...

I think you could directly rewrite it as:

    def pytest_generate_tests(metafunc):
        if metafunc.function == test_generic:
            l = []
            for test_data in tests:
                test = dict(EMPTY_TEST)
                test.update(test_data)
                l.append(test)
            
            metafunc.parametrize("test", argvalues=l, ids=ids)

    def test_generic(test):
        ...

haven't tested this -- does it work for you?

best,
holger