[pytest-dev] pytest-xdist-1.11: restarting crash nodes, fixture cache fix

[holger krekel]

Hi Andreas,

On Mon, Sep 22, 2014 at 22:03 +0200, Andreas Pelme wrote:
> Hi Holger,
> 
> On 19 sep 2014, at 09:52, holger krekel <holger at merlinux.eu> wrote:
> > I presume that with module and class scope people don't presume that
> > a fixture survives until the end of a process so eager teardown is less
> > of a problem, there, right?
> 
> 
> I have not fully grasped how fixture teardown currently happens in pytest. To explore it, I came up with this example:
> 
> import pytest
> 
> 
> @pytest.yield_fixture(scope='class')
> def fix():
>     print 'fixture setup'
>     yield
>     print 'fixture teardown'
> 
> 
> 
> class TestFoo():
>     def test_a(self, fix):
>         print 'test_a'
> 
>     def test_b(self):
>         print 'test_b'
> 
>     def test_c(self, fix):
>         print 'test_c'
> 
> 
> It gives me the output (with the latest pytest from PyPI):
> 
> fixture setup
> test_a
> test_b
> test_c
> fixture teardown
> 
> I.e. even though test_b does not actively request the fixture, it is active during the test.

A few addiotional notes:

- test_b does not have a reference to the fixture so can not "use" it

- "py.test -k test_b" would not see an activated "fix"

- it would be a programming mistake for test_b to depend on fix without
  declaring it.

> Is this even considered to be a bug or a feature? :-)

Let me ask back: how exactly is it a problem?

> This behavior may be considered a bug since it makes test suites
> brittle - if the fixture does not contain a value itself, it can
> probably be neglected to actually properly request the fixture, but
> test_b will still accidentally have “fix” available. 

How does "test_b" have "fix" available?

> will only show itself when running a subset of the test suite (and
> with xdist!).
> 
> I would prefer (I think) if all fixtures where torn down when they are not requested (in my example, before test_b is run, to ensure test_b is only run with its own fixtures).
> 
> However, if all teardowns worked like this, the efficiency will be very bad since there will be a lot of teardowns. I think these fixtures that should be allowed to stay alive, should be explicitly declared like that, like the option to pytest.fixture() suggested earlier.

It would be a somewhat backward incompatible change and probably break some
test suites.

> I think this makes sense regardless of whether the fixture is class,
> module or session scoped. Having different semantics depending on the
> scope would be confusing. Fixture authors must be aware and decide
> whether or not a fixture may accidentally be available to wrong tests.
> I have a hard time to see how we can solve this properly without a
> flag or some kind of distinction for this.
> 
> I am probably missing a lot of details here, what are your thoughts on this?

I am not quite sure what the exact real-life problem is that 
we are trying to solve. Is something breaking that shouldn't, currently?

Maybe the next step is to try to document the fixture setup/teardown
logic more precisely because i am not even sure myself about the details.
xdist certainly has less guarantees than single-process pytest.

best,
holger