[pytest-dev] How to repeat tests with a setup-action before each repitition

[Bruno Oliveira]

Hello Emil,

On Fri, Sep 30, 2016 at 6:26 PM Emil Petersen emil.petersen at thalmic.com
<http://mailto:emil.petersen@thalmic.com> wrote:

I am looking to repeat some of the tests in my test suite, and want to call
a certain action before each repetition. To be clear, I want to:

1) First, run all the tests marked with repeat, as-is
2) Then, take a specific action, in this case rebooting a device
3) Then, repeat each test marked with repeat
4) Then, repeat steps 2-3 the amount of times specified.

Using the above as an example, I want to run test_something and
test_something_else, then expensive_setup, then the two tests again, then
expensive_setup, etc. So only one call to expensive_setup for each run
including test_something() and test_something_else()

What would be a sensible way to go about this?

I might not full fill all your requirements, but here are some ideas:

You can use a parametrized fixture which calls the expensive setup after
its first instantiation:

def test_something(device):
    check_something == last_result_from_file()
def test_something_else(device):
    something_else()
@pytest.fixture(params=[0, 1, 2, 3]def expensive_setup(request):
    device = Device('uri_to_device')
    device.connect()
    if request.param > 0:
        device.reboot()
    return device

As an addendum, it is ideal if each test is only reported as -one- test,
rather than one test being generated for each run.

That’s not so simple, you will probably have to override the terminal
plugin and implement your own pytest_runtest_logreport to accumulate
reports.

I suspect this is more trouble than it is worth.

Ideally, if any one call to the functions ends up generating a failure or
error, it would be able to fail, and abort any attempts at subsequently
running (I.e. failures should be final).

For that you can use --max-fail or -x, which will cause pytest to stop at
the first failure.

In case you’re curious about why the tests need to be repeated, it’s to
hunt for race conditions, and other failures which may occur sometimes.

You might want to take a look at pytest-repeat
<https://github.com/pytest-dev/pytest-repeat>, which was created to help
find intermittent failures.

Hope that helps!

Cheers,
Bruno;
​
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20161003/1d9f205e/attachment.html>