[pytest-dev] pytest-failed: dump/restore failed tests

[Florian Schulze]

On 15 Jul 2024, at 14:03, Bruno Oliveira wrote:

> Hi Florian,
>
> I find this plugin very useful, in fact I think it would make sense for this to be in pytest-core given that `--lf` already exists.

I wouldn't mind having it in core. I just wanted people to easily try it first and maybe improve it before inclusion. Like pytest-warnings was done initially. Using ``config._do_configure`` would also be a non-issue if it is in core.

> What if the --lf option can optionally accept a file name, which when given, will read the set of failed tests from that file instead of reading it from the cache. This way you can just save the `.pytest_cache/lastfailed` file from CI, and reuse it locally:
>
>     pytest --lf=lastfailed-file-from-ci
>
> Seems like the changes would be minimal for that to happen.

Adding a json parser for that file would be trivial and it would also remove my gripe with the ``--restore-lf`` name. I don't think it is possible or wise to overwrite ``--lf`` from the plugin though.

Also changing ``--lf`` to have an optional filename might cause problems with invocation if file paths are involved. Could be a backward compatibility issue.

How would one access ``.pytest_cache/lastfailed`` from GitHub Actions for example? Just copy'n'paste from the log output seems more intuitive for most people, so the parsing should be kept.

I'm not sure how user-friendly it is to explain how to copy ``.pytest_cache/lastfailed``, so I think ``--dump-lf`` would still be useful.

> I know this is not exactly the feedback you wanted, but thought I would throw the idea in there for your consideration.

Totally valid feedback. Thanks!

Regards,
Florian

>
> Cheers,
> Bruno
>
> On Mon, Jul 15, 2024, at 2:29 AM, Florian Schulze wrote:
>> Hi!
>>
>> I wrote https://github.com/fschulze/pytest-failed and would like to get some feedback on a few things before I publish it.
>>
>> What is it?
>>
>> The plugin adds ``--dump-lf`` and ``--restore-lf`` options.
>>
>> My use case is two fold for test suites with runtimes of several minutes:
>>
>> 1. when tests fail in CI, I can copy the summary from the report and paste it into a file like ``failed.txt`` then re-run just those tests with ``--restore-lf=failed.txt``
>>
>> 2. when fixing a bug I often fix only some of tests at the first attempt, then refactor the fix and want to re-run all initially failing tests. With this plugin I only run the full test suite once, then run with ``--dump-lf=failed.txt`` and can restore the initial bunch of tests with ``--restore-lf=failed.txt`` whenever I like, without ever re-running the full test suite.
>>
>> This is also super useful with ``git bisect`` and likely many other use-cases.
>>
>> The ``=`` in the options seems to be required, as otherwise the filename might be interpreted as ``[file_or_dir]`` by pytest in some forms of invokation.
>>
>> My questions:
>>
>> In [``pytest_cmdline_main``](https://github.com/fschulze/pytest-failed/blob/main/src/pytest_failed/plugin.py#L44) I use ``config._do_configure``, which is internal API. Is there a better way?
>>
>> Anyone have suggestions for better names of the plugin itself and its options? I think ``--dump-lf`` is ok, but I don't really like ``--restore-lf``.
>>
>> When reading the list of test names it knows about the pytest report summary syntax, are there any other formats that would be useful to support for cut'n'paste from CI?
>>
>> If you try the plugin, please give feedback on whether you found it useful and understand how it works.
>>
>> Regards,
>> Florian Schulze
>> _______________________________________________
>> pytest-dev mailing list
>> pytest-dev at python.org
>> https://mail.python.org/mailman/listinfo/pytest-dev
>>