Hi! I wrote https://github.com/fschulze/pytest-failed and would like to get some feedback on a few things before I publish it. What is it? The plugin adds ``--dump-lf`` and ``--restore-lf`` options. My use case is two fold for test suites with runtimes of several minutes: 1. when tests fail in CI, I can copy the summary from the report and paste it into a file like ``failed.txt`` then re-run just those tests with ``--restore-lf=failed.txt`` 2. when fixing a bug I often fix only some of tests at the first attempt, then refactor the fix and want to re-run all initially failing tests. With this plugin I only run the full test suite once, then run with ``--dump-lf=failed.txt`` and can restore the initial bunch of tests with ``--restore-lf=failed.txt`` whenever I like, without ever re-running the full test suite. This is also super useful with ``git bisect`` and likely many other use-cases. The ``=`` in the options seems to be required, as otherwise the filename might be interpreted as ``[file_or_dir]`` by pytest in some forms of invokation. My questions: In [``pytest_cmdline_main``](https://github.com/fschulze/pytest-failed/blob/main/src/pytest_failed/plugin...) I use ``config._do_configure``, which is internal API. Is there a better way? Anyone have suggestions for better names of the plugin itself and its options? I think ``--dump-lf`` is ok, but I don't really like ``--restore-lf``. When reading the list of test names it knows about the pytest report summary syntax, are there any other formats that would be useful to support for cut'n'paste from CI? If you try the plugin, please give feedback on whether you found it useful and understand how it works. Regards, Florian Schulze
Hi Florian, I find this plugin very useful, in fact I think it would make sense for this to be in pytest-core given that `--lf` already exists. What if the --lf option can optionally accept a file name, which when given, will read the set of failed tests from that file instead of reading it from the cache. This way you can just save the `.pytest_cache/lastfailed` file from CI, and reuse it locally: pytest --lf=lastfailed-file-from-ci Seems like the changes would be minimal for that to happen. I know this is not exactly the feedback you wanted, but thought I would throw the idea in there for your consideration. Cheers, Bruno On Mon, Jul 15, 2024, at 2:29 AM, Florian Schulze wrote:
Hi!
I wrote https://github.com/fschulze/pytest-failed and would like to get some feedback on a few things before I publish it.
What is it?
The plugin adds ``--dump-lf`` and ``--restore-lf`` options.
My use case is two fold for test suites with runtimes of several minutes:
1. when tests fail in CI, I can copy the summary from the report and paste it into a file like ``failed.txt`` then re-run just those tests with ``--restore-lf=failed.txt``
2. when fixing a bug I often fix only some of tests at the first attempt, then refactor the fix and want to re-run all initially failing tests. With this plugin I only run the full test suite once, then run with ``--dump-lf=failed.txt`` and can restore the initial bunch of tests with ``--restore-lf=failed.txt`` whenever I like, without ever re-running the full test suite.
This is also super useful with ``git bisect`` and likely many other use-cases.
The ``=`` in the options seems to be required, as otherwise the filename might be interpreted as ``[file_or_dir]`` by pytest in some forms of invokation.
My questions:
In [``pytest_cmdline_main``](https://github.com/fschulze/pytest-failed/blob/main/src/pytest_failed/plugin...) I use ``config._do_configure``, which is internal API. Is there a better way?
Anyone have suggestions for better names of the plugin itself and its options? I think ``--dump-lf`` is ok, but I don't really like ``--restore-lf``.
When reading the list of test names it knows about the pytest report summary syntax, are there any other formats that would be useful to support for cut'n'paste from CI?
If you try the plugin, please give feedback on whether you found it useful and understand how it works.
Regards, Florian Schulze _______________________________________________ pytest-dev mailing list pytest-dev@python.org https://mail.python.org/mailman/listinfo/pytest-dev
On 15 Jul 2024, at 14:03, Bruno Oliveira wrote:
Hi Florian,
I find this plugin very useful, in fact I think it would make sense for this to be in pytest-core given that `--lf` already exists.
I wouldn't mind having it in core. I just wanted people to easily try it first and maybe improve it before inclusion. Like pytest-warnings was done initially. Using ``config._do_configure`` would also be a non-issue if it is in core.
What if the --lf option can optionally accept a file name, which when given, will read the set of failed tests from that file instead of reading it from the cache. This way you can just save the `.pytest_cache/lastfailed` file from CI, and reuse it locally:
pytest --lf=lastfailed-file-from-ci
Seems like the changes would be minimal for that to happen.
Adding a json parser for that file would be trivial and it would also remove my gripe with the ``--restore-lf`` name. I don't think it is possible or wise to overwrite ``--lf`` from the plugin though. Also changing ``--lf`` to have an optional filename might cause problems with invocation if file paths are involved. Could be a backward compatibility issue. How would one access ``.pytest_cache/lastfailed`` from GitHub Actions for example? Just copy'n'paste from the log output seems more intuitive for most people, so the parsing should be kept. I'm not sure how user-friendly it is to explain how to copy ``.pytest_cache/lastfailed``, so I think ``--dump-lf`` would still be useful.
I know this is not exactly the feedback you wanted, but thought I would throw the idea in there for your consideration.
Totally valid feedback. Thanks! Regards, Florian
Cheers, Bruno
On Mon, Jul 15, 2024, at 2:29 AM, Florian Schulze wrote:
Hi!
I wrote https://github.com/fschulze/pytest-failed and would like to get some feedback on a few things before I publish it.
What is it?
The plugin adds ``--dump-lf`` and ``--restore-lf`` options.
My use case is two fold for test suites with runtimes of several minutes:
1. when tests fail in CI, I can copy the summary from the report and paste it into a file like ``failed.txt`` then re-run just those tests with ``--restore-lf=failed.txt``
2. when fixing a bug I often fix only some of tests at the first attempt, then refactor the fix and want to re-run all initially failing tests. With this plugin I only run the full test suite once, then run with ``--dump-lf=failed.txt`` and can restore the initial bunch of tests with ``--restore-lf=failed.txt`` whenever I like, without ever re-running the full test suite.
This is also super useful with ``git bisect`` and likely many other use-cases.
The ``=`` in the options seems to be required, as otherwise the filename might be interpreted as ``[file_or_dir]`` by pytest in some forms of invokation.
My questions:
In [``pytest_cmdline_main``](https://github.com/fschulze/pytest-failed/blob/main/src/pytest_failed/plugin...) I use ``config._do_configure``, which is internal API. Is there a better way?
Anyone have suggestions for better names of the plugin itself and its options? I think ``--dump-lf`` is ok, but I don't really like ``--restore-lf``.
When reading the list of test names it knows about the pytest report summary syntax, are there any other formats that would be useful to support for cut'n'paste from CI?
If you try the plugin, please give feedback on whether you found it useful and understand how it works.
Regards, Florian Schulze _______________________________________________ pytest-dev mailing list pytest-dev@python.org https://mail.python.org/mailman/listinfo/pytest-dev
On Mon, Jul 15, 2024, at 9:53 AM, Florian Schulze wrote:
On 15 Jul 2024, at 14:03, Bruno Oliveira wrote:
Hi Florian,
I find this plugin very useful, in fact I think it would make sense for this to be in pytest-core given that `--lf` already exists.
I wouldn't mind having it in core. I just wanted people to easily try it first and maybe improve it before inclusion. Like pytest-warnings was done initially. Using ``config._do_configure`` would also be a non-issue if it is in core.
What if the --lf option can optionally accept a file name, which when given, will read the set of failed tests from that file instead of reading it from the cache. This way you can just save the `.pytest_cache/lastfailed` file from CI, and reuse it locally:
pytest --lf=lastfailed-file-from-ci
Seems like the changes would be minimal for that to happen.
Adding a json parser for that file would be trivial and it would also remove my gripe with the ``--restore-lf`` name. I don't think it is possible or wise to overwrite ``--lf`` from the plugin though.
Oh I would document that file to be "opaque" for users (meaning we can change it at will, including its format), so users are expected to just copy it from one location to another.
Also changing ``--lf`` to have an optional filename might cause problems with invocation if file paths are involved. Could be a backward compatibility issue.
Good point.
How would one access ``.pytest_cache/lastfailed`` from GitHub Actions for example? Just copy'n'paste from the log output seems more intuitive for most people, so the parsing should be kept.
I'm not sure how user-friendly it is to explain how to copy ``.pytest_cache/lastfailed``, so I think ``--dump-lf`` would still be useful.
Seems just a matter of documenting to use the `upload-artifact` action to upload the file unchanged, without users needing to worry about the contents of the file.
I know this is not exactly the feedback you wanted, but thought I would throw the idea in there for your consideration.
Totally valid feedback. Thanks!
Regards, Florian
Cheers, Bruno
On Mon, Jul 15, 2024, at 2:29 AM, Florian Schulze wrote:
Hi!
I wrote https://github.com/fschulze/pytest-failed and would like to get some feedback on a few things before I publish it.
What is it?
The plugin adds ``--dump-lf`` and ``--restore-lf`` options.
My use case is two fold for test suites with runtimes of several minutes:
1. when tests fail in CI, I can copy the summary from the report and paste it into a file like ``failed.txt`` then re-run just those tests with ``--restore-lf=failed.txt``
2. when fixing a bug I often fix only some of tests at the first attempt, then refactor the fix and want to re-run all initially failing tests. With this plugin I only run the full test suite once, then run with ``--dump-lf=failed.txt`` and can restore the initial bunch of tests with ``--restore-lf=failed.txt`` whenever I like, without ever re-running the full test suite.
This is also super useful with ``git bisect`` and likely many other use-cases.
The ``=`` in the options seems to be required, as otherwise the filename might be interpreted as ``[file_or_dir]`` by pytest in some forms of invokation.
My questions:
In [``pytest_cmdline_main``](https://github.com/fschulze/pytest-failed/blob/main/src/pytest_failed/plugin...) I use ``config._do_configure``, which is internal API. Is there a better way?
Anyone have suggestions for better names of the plugin itself and its options? I think ``--dump-lf`` is ok, but I don't really like ``--restore-lf``.
When reading the list of test names it knows about the pytest report summary syntax, are there any other formats that would be useful to support for cut'n'paste from CI?
If you try the plugin, please give feedback on whether you found it useful and understand how it works.
Regards, Florian Schulze _______________________________________________ pytest-dev mailing list pytest-dev@python.org https://mail.python.org/mailman/listinfo/pytest-dev
This is a nice feature that I would definitely use. On Mon, 15 Jul 2024 at 16:04, Bruno Oliveira <bruno@soliv.dev> wrote:
On Mon, Jul 15, 2024, at 9:53 AM, Florian Schulze wrote:
What if the --lf option can optionally accept a file name, which when given, will read the set of failed tests from that file instead of reading it from the cache. This way you can just save the `.pytest_cache/lastfailed` file from CI, and reuse it locally:
pytest --lf=lastfailed-file-from-ci
Seems like the changes would be minimal for that to happen.
I like this idea. I would want it to have behaviour that --lf=failed.txt checks if the file exists and if not populates it with the contents from the pytest ---lf cache. Then you could do: $ pytest < runts tests, some fail > $ pytest --lf=failed=txt < file doesn't exist so save last-failed there (from .pytest_cache/last_failed) and run the failed tests > $ pytest --lf=failed.txt < file exists now so run tests specified in the file >
I'm not sure how user-friendly it is to explain how to copy ``.pytest_cache/lastfailed``, so I think ``--dump-lf`` would still be useful.
I think the behaviour for --lf=failed.txt I suggested is nicer than expecting users to refer to this file directly.
Seems just a matter of documenting to use the `upload-artifact` action to upload the file unchanged, without users needing to worry about the contents of the file.
Using upload-artifact is awkward for this. It would be nicer if you could copy and paste the output from e.g. pytest -rA. -- Oscar
On 16 Jul 2024, at 1:36, Oscar Benjamin wrote:
I like this idea. I would want it to have behaviour that --lf=failed.txt checks if the file exists and if not populates it with the contents from the pytest ---lf cache. Then you could do:
$ pytest < runts tests, some fail > $ pytest --lf=failed=txt < file doesn't exist so save last-failed there (from .pytest_cache/last_failed) and run the failed tests > $ pytest --lf=failed.txt < file exists now so run tests specified in the file >
Doing this based on the existence of the file causes a problem in my opinion. Let's say I ran this before at some point and the file exists with an old set of tests. Now I run the test suite and got new failures I want to store. I forget to remove the file and run with --lf=failed.txt. Now the old set of tests is used, the new set is gone and I have to re-run the whole suite, which is exactly what I wanted to prevent in the first place. Regards, Florian
On Tue, 16 Jul 2024 at 10:09, Florian Schulze <mail@florian-schulze.net> wrote:
On 16 Jul 2024, at 1:36, Oscar Benjamin wrote:
I like this idea. I would want it to have behaviour that --lf=failed.txt checks if the file exists and if not populates it with the contents from the pytest ---lf cache. Then you could do:
$ pytest < runts tests, some fail > $ pytest --lf=failed=txt < file doesn't exist so save last-failed there (from .pytest_cache/last_failed) and run the failed tests > $ pytest --lf=failed.txt < file exists now so run tests specified in the file >
Doing this based on the existence of the file causes a problem in my opinion. Let's say I ran this before at some point and the file exists with an old set of tests. Now I run the test suite and got new failures I want to store. I forget to remove the file and run with --lf=failed.txt. Now the old set of tests is used, the new set is gone and I have to re-run the whole suite, which is exactly what I wanted to prevent in the first place.
There is always the possibility that you forget and run the wrong command. The downside of a stateful interface is that running the wrong command once potentially means you can't just fix the command and rerun. But --lf is already stateful and that is actually useful in practice because it means you can decide whether to use --lf after having seen the failures. You could also say that when run as --lf=failed.txt if the file exists then the cache doesn't get cleared. Then a subsequent run with --lf or --lf=failed2.txt still recovers the failing tests from the previous full run. -- Oscar
On Tue, Jul 16, 2024, at 6:24 AM, Oscar Benjamin wrote:
On Tue, 16 Jul 2024 at 10:09, Florian Schulze <mail@florian-schulze.net> wrote:
On 16 Jul 2024, at 1:36, Oscar Benjamin wrote:
I like this idea. I would want it to have behaviour that --lf=failed.txt checks if the file exists and if not populates it with the contents from the pytest ---lf cache. Then you could do:
$ pytest < runts tests, some fail > $ pytest --lf=failed=txt < file doesn't exist so save last-failed there (from .pytest_cache/last_failed) and run the failed tests > $ pytest --lf=failed.txt < file exists now so run tests specified in the file >
Doing this based on the existence of the file causes a problem in my opinion. Let's say I ran this before at some point and the file exists with an old set of tests. Now I run the test suite and got new failures I want to store. I forget to remove the file and run with --lf=failed.txt. Now the old set of tests is used, the new set is gone and I have to re-run the whole suite, which is exactly what I wanted to prevent in the first place.
There is always the possibility that you forget and run the wrong command. The downside of a stateful interface is that running the wrong command once potentially means you can't just fix the command and rerun. But --lf is already stateful and that is actually useful in practice because it means you can decide whether to use --lf after having seen the failures.
You could also say that when run as --lf=failed.txt if the file exists then the cache doesn't get cleared. Then a subsequent run with --lf or --lf=failed2.txt still recovers the failing tests from the previous full run.
All the discussion made me realize that perhaps we do not need something attached to `--lf` specifically, just a new flag (or something) to collect tests based on the summary output of a `-ra` run. Then the user is free to execute with `--lf` or even `--sw`. Regardless of the plugin and the --lf semantics we are discussing, I think this would be useful to be in the core itself. Created https://github.com/pytest-dev/pytest/issues/12619 to track it. Cheers, Bruno.
-- Oscar _______________________________________________ pytest-dev mailing list pytest-dev@python.org https://mail.python.org/mailman/listinfo/pytest-dev
How would one access ``.pytest_cache/lastfailed`` from GitHub Actions for example? Just copy'n'paste from the log output seems more intuitive for most people, so the parsing should be kept.
I'm not sure how user-friendly it is to explain how to copy ``.pytest_cache/lastfailed``, so I think ``--dump-lf`` would still be useful.
Seems just a matter of documenting to use the `upload-artifact` action to upload the file unchanged, without users needing to worry about the contents of the file.
One would have to set this up beforehand, which is too much friction. So I think supporting the reporting output directly should be kept. Regards, Florian
participants (3)
-
Bruno Oliveira -
Florian Schulze -
Oscar Benjamin