[pytest-dev] [pytest-rerunfailures] pytest-rerunfailures not using fixtures on reruns (#10)

[Leah Klearman]

Hi Holger,

Bob appears to be on vacation, so I asked Stephen to run the job in it's
natural environment. The run with the dev version of pytest and my plugin
enabled had the same two failures as the project had with the previous
release of pytest and no plugin. An analysis of whether the failures
actually act differently will have to await Bob returning from vacation.

For our reference, the jenkins config: http://pastie.org/7705026

Wish I could have given you better feedback on the patch. Please use
your discretion as to whether or not to include it in the upcoming release.

Thanks for your help.
-Leah



On Tue, Apr 23, 2013 at 1:30 PM, holger krekel <holger at merlinux.eu> wrote:

> On Tue, Apr 23, 2013 at 09:56 -0700, Leah Klearman wrote:
> > Holger,
> >
> > The only reason I can think of why py.test would perform teardown /
> > finalization differently depending on the nextitem is if class, module,
> and
> > package level fixtures are being used.
>
> Yes, i tried to say something equivalent in my previous mail.
>
> > If only method level fixtures are
> > used, then they should always get created and always get torn down for
> each
> > test case. Is that not how it works?
>
> Effectively it should be like this and probably independently of
> the value of nextitem.  If you find issues let us know.
>
> > We can't re-run just the 'call' phase because the failure of the previous
> > 'call' phase may have left behind a partial situation (see automated
> > testing before we had setup and teardown and fixtures).
>
> Sure, good point.
>
> cheers,
> holger
>
> > -Leah
> >
> >
> > On Tue, Apr 23, 2013 at 12:16 AM, holger krekel <holger at merlinux.eu>
> wrote:
> >
> > > Hey Leah,
> > >
> > > On Mon, Apr 22, 2013 at 15:08 -0700, Leah Klearman wrote:
> > > > Hey Holger,
> > > >
> > > > Thank you for the timely patch. I haven't seen Bob online today, but
> > > > hopefully he'll be able to test it very soon.
> > > >
> > > > A word of warning: your calling of runtestprotocol() is not quite
> right
> > > > > and might lead to problems.  "nextitem" should really be the item
> that
> > > > > is going to be run next.  So if you re-run three times the first
> two
> > > > > invocations should have nextitem=item.
> > > >
> > > >
> > > > I agree that that would be ideal, but it only re-runs the test if it
> > > fails,
> > > > so I
> > > > would have to know the future in order to know what value to send
> > > nextitem=.
> > > > Since hopefully most tests pass the first try, sending nextitem
> instead
> > > of
> > > > item seems to be the better answer.
> > >
> > > FYI pytest perform teardown/finalization according to nextitem:  if
> your
> > > next item uses different fixtures, the "item" ones may be torn down
> > > (it also depends on the caching scope of the fixtures).
> > >
> > > Do you generally want to re-run the setup or could you also settle
> > > on just re-running the "call" phase of a test?  The latter one would
> > > be easier and probably not disrupt fixture mechanics.
> > >
> > > best,
> > > holger
> > >
> > > > Thanks!
> > > > -Leah
> > > >
> > > >
> > > > On Mon, Apr 22, 2013 at 1:27 AM, holger krekel <holger at merlinux.eu>
> > > wrote:
> > > >
> > > > > Hi Leah, Bob,
> > > > >
> > > > > On Fri, Apr 19, 2013 at 20:55 -0700, Leah Klearman wrote:
> > > > > > Hi Holger and other py.test mavens,
> > > > > >
> > > > > > Bob has reported a problem with my py.test plugin
> > > pytest-rerunfailures
> > > > > [1]
> > > > > > not re-running the setup before rerunning the test.
> > > > > >
> > > > > > Looking at my code, it has pytest_runtest_protocol() [2] looping
> > > > > > on _pytest.runner.runtestprotocol() [3], which in turn runs the
> > > setup,
> > > > > the
> > > > > > test, and the teardown.
> > > > > >
> > > > > > [1] https://github.com/klrmn/pytest-rerunfailures
> > > > > > [2]
> > > > > >
> > > > >
> > >
> https://github.com/klrmn/pytest-rerunfailures/blob/master/rerunfailures/plugin.py#L46
> > > > > > [3]
> > > > > >
> > > > >
> > >
> https://bitbucket.org/hpk42/pytest/src/fdc28ac2029f1c0be1dac4991c2f1b014c39a03f/_pytest/runner.py?at=default#cl-65
> > > > > >
> > > > > > I haven't taken the hours needed to get my head fully into
> py.test
> > > plugin
> > > > > > development mode, but I'm not sure I can implement a fix at my
> layer.
> > > > > >
> > > > > >
> > > > > > I'm hoping someone here will have some insight.
> > > > >
> > > > > It's a bit intricate.  A function item keeps around some fixture
> state
> > > > > and was so far not intended to be run multiple times.  I went
> ahead and
> > > > > tried to improve the behaviour to better allow re-running.
> Please try
> > > > > with
> > > > >
> > > > >     pip install -i http://pypi.testrun.org -U pytest
> > > > >
> > > > > which should give you pytest-2.3.5.dev16 at least.  This is bound
> to be
> > > > > released soon so quick feedback is welcome.  If you still have
> problems
> > > > > please try to send a minimal test file which shows undesired
> behaviour.
> > > > >
> > > > > A word of warning: your calling of runtestprotocol() is not quite
> right
> > > > > and might lead to problems.  "nextitem" should really be the item
> that
> > > > > is going to be run next.  So if you re-run three times the first
> two
> > > > > invocations should have nextitem=item.
> > > > >
> > > > > best,
> > > > > holger
> > > > >
> > > > >
> > > > > > Thanks,
> > > > > > -Leah
> > > > > >
> > > > > >
> > > > > > On Sun, Apr 14, 2013 at 6:29 PM, Bob Silverberg <
> > > > > notifications at github.com>wrote:
> > > > > >
> > > > > > > I just verified this behaviour myself with a simple test [1]. I
> > > see it
> > > > > > > with both funcargs and fixtures, but I'm not sure if it has to
> do
> > > with
> > > > > the
> > > > > > > plugin, or the way py.test works. It does inject the value
> into the
> > > > > test
> > > > > > > method, but it doesn't rerun the fixture, so it seems like it
> is
> > > > > caching
> > > > > > > the first run of the fixture and using that on subsequent runs.
> > > > > > >
> > > > > > > I'm not sure if this is something that the plugin can have any
> > > effect
> > > > > on,
> > > > > > > or if it's just the way fixtures work. It is specified for this
> > > fixture
> > > > > > > that it is scope='function', and perhaps py.test makes that
> happen
> > > by
> > > > > > > checking the function name, which is, of course, the same for
> each
> > > > > run. I
> > > > > > > did try removing the scope argument from the fixture but that
> had
> > > no
> > > > > > > effect.
> > > > > > >
> > > > > > > Do you have any thoughts about this, @klrmn <
> > > https://github.com/klrmn
> > > > > >?
> > > > > > >
> > > > > > > [1] https://gist.github.com/bobsilverberg/5385035
> > > > > > >
> > > > > > > —
> > > > > > > Reply to this email directly or view it on GitHub<
> > > > >
> > >
> https://github.com/klrmn/pytest-rerunfailures/issues/10#issuecomment-16363644
> > > > > >
> > > > > > > .
> > > > > > >
> > > > >
> > > > > > _______________________________________________
> > > > > > Pytest-dev mailing list
> > > > > > Pytest-dev at python.org
> > > > > > http://mail.python.org/mailman/listinfo/pytest-dev
> > > > >
> > > > >
> > >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20130423/afb12712/attachment.html>