[pytest-dev] [pytest-rerunfailures] pytest-rerunfailures not using fixtures on reruns (#10)

[holger krekel]

On Tue, Apr 23, 2013 at 09:56 -0700, Leah Klearman wrote:
> Holger,
> 
> The only reason I can think of why py.test would perform teardown /
> finalization differently depending on the nextitem is if class, module, and
> package level fixtures are being used. 

Yes, i tried to say something equivalent in my previous mail.

> If only method level fixtures are
> used, then they should always get created and always get torn down for each
> test case. Is that not how it works?

Effectively it should be like this and probably independently of
the value of nextitem.  If you find issues let us know.
 
> We can't re-run just the 'call' phase because the failure of the previous
> 'call' phase may have left behind a partial situation (see automated
> testing before we had setup and teardown and fixtures).

Sure, good point.

cheers,
holger

> -Leah
> 
> 
> On Tue, Apr 23, 2013 at 12:16 AM, holger krekel <holger at merlinux.eu> wrote:
> 
> > Hey Leah,
> >
> > On Mon, Apr 22, 2013 at 15:08 -0700, Leah Klearman wrote:
> > > Hey Holger,
> > >
> > > Thank you for the timely patch. I haven't seen Bob online today, but
> > > hopefully he'll be able to test it very soon.
> > >
> > > A word of warning: your calling of runtestprotocol() is not quite right
> > > > and might lead to problems.  "nextitem" should really be the item that
> > > > is going to be run next.  So if you re-run three times the first two
> > > > invocations should have nextitem=item.
> > >
> > >
> > > I agree that that would be ideal, but it only re-runs the test if it
> > fails,
> > > so I
> > > would have to know the future in order to know what value to send
> > nextitem=.
> > > Since hopefully most tests pass the first try, sending nextitem instead
> > of
> > > item seems to be the better answer.
> >
> > FYI pytest perform teardown/finalization according to nextitem:  if your
> > next item uses different fixtures, the "item" ones may be torn down
> > (it also depends on the caching scope of the fixtures).
> >
> > Do you generally want to re-run the setup or could you also settle
> > on just re-running the "call" phase of a test?  The latter one would
> > be easier and probably not disrupt fixture mechanics.
> >
> > best,
> > holger
> >
> > > Thanks!
> > > -Leah
> > >
> > >
> > > On Mon, Apr 22, 2013 at 1:27 AM, holger krekel <holger at merlinux.eu>
> > wrote:
> > >
> > > > Hi Leah, Bob,
> > > >
> > > > On Fri, Apr 19, 2013 at 20:55 -0700, Leah Klearman wrote:
> > > > > Hi Holger and other py.test mavens,
> > > > >
> > > > > Bob has reported a problem with my py.test plugin
> > pytest-rerunfailures
> > > > [1]
> > > > > not re-running the setup before rerunning the test.
> > > > >
> > > > > Looking at my code, it has pytest_runtest_protocol() [2] looping
> > > > > on _pytest.runner.runtestprotocol() [3], which in turn runs the
> > setup,
> > > > the
> > > > > test, and the teardown.
> > > > >
> > > > > [1] https://github.com/klrmn/pytest-rerunfailures
> > > > > [2]
> > > > >
> > > >
> > https://github.com/klrmn/pytest-rerunfailures/blob/master/rerunfailures/plugin.py#L46
> > > > > [3]
> > > > >
> > > >
> > https://bitbucket.org/hpk42/pytest/src/fdc28ac2029f1c0be1dac4991c2f1b014c39a03f/_pytest/runner.py?at=default#cl-65
> > > > >
> > > > > I haven't taken the hours needed to get my head fully into py.test
> > plugin
> > > > > development mode, but I'm not sure I can implement a fix at my layer.
> > > > >
> > > > >
> > > > > I'm hoping someone here will have some insight.
> > > >
> > > > It's a bit intricate.  A function item keeps around some fixture state
> > > > and was so far not intended to be run multiple times.  I went ahead and
> > > > tried to improve the behaviour to better allow re-running.   Please try
> > > > with
> > > >
> > > >     pip install -i http://pypi.testrun.org -U pytest
> > > >
> > > > which should give you pytest-2.3.5.dev16 at least.  This is bound to be
> > > > released soon so quick feedback is welcome.  If you still have problems
> > > > please try to send a minimal test file which shows undesired behaviour.
> > > >
> > > > A word of warning: your calling of runtestprotocol() is not quite right
> > > > and might lead to problems.  "nextitem" should really be the item that
> > > > is going to be run next.  So if you re-run three times the first two
> > > > invocations should have nextitem=item.
> > > >
> > > > best,
> > > > holger
> > > >
> > > >
> > > > > Thanks,
> > > > > -Leah
> > > > >
> > > > >
> > > > > On Sun, Apr 14, 2013 at 6:29 PM, Bob Silverberg <
> > > > notifications at github.com>wrote:
> > > > >
> > > > > > I just verified this behaviour myself with a simple test [1]. I
> > see it
> > > > > > with both funcargs and fixtures, but I'm not sure if it has to do
> > with
> > > > the
> > > > > > plugin, or the way py.test works. It does inject the value into the
> > > > test
> > > > > > method, but it doesn't rerun the fixture, so it seems like it is
> > > > caching
> > > > > > the first run of the fixture and using that on subsequent runs.
> > > > > >
> > > > > > I'm not sure if this is something that the plugin can have any
> > effect
> > > > on,
> > > > > > or if it's just the way fixtures work. It is specified for this
> > fixture
> > > > > > that it is scope='function', and perhaps py.test makes that happen
> > by
> > > > > > checking the function name, which is, of course, the same for each
> > > > run. I
> > > > > > did try removing the scope argument from the fixture but that had
> > no
> > > > > > effect.
> > > > > >
> > > > > > Do you have any thoughts about this, @klrmn <
> > https://github.com/klrmn
> > > > >?
> > > > > >
> > > > > > [1] https://gist.github.com/bobsilverberg/5385035
> > > > > >
> > > > > > —
> > > > > > Reply to this email directly or view it on GitHub<
> > > >
> > https://github.com/klrmn/pytest-rerunfailures/issues/10#issuecomment-16363644
> > > > >
> > > > > > .
> > > > > >
> > > >
> > > > > _______________________________________________
> > > > > Pytest-dev mailing list
> > > > > Pytest-dev at python.org
> > > > > http://mail.python.org/mailman/listinfo/pytest-dev
> > > >
> > > >
> >