Hi folks.
Are there any tools available for system testing a large, shell-callable python script with many different command line options?
I'm aware of pytest for unit tests, but what about running a shell command with some options, and checking its stdout for appropriate content?
Thanks.
I was looking for something else that pip used to use and I think Doug Hellman wrote but couldn't find it. I did find https://pypi.org/project/pytest-shell/
Sent from my phone with my typo-happy thumbs. Please excuse my brevity
On Thu, Jun 10, 2021, 12:44 Dan Stromberg strombrg@gmail.com wrote:
Hi folks.
Are there any tools available for system testing a large, shell-callable python script with many different command line options?
I'm aware of pytest for unit tests, but what about running a shell command with some options, and checking its stdout for appropriate content?
Thanks.
-- Dan Stromberg _______________________________________________ code-quality mailing list -- code-quality@python.org To unsubscribe send an email to code-quality-leave@python.org https://mail.python.org/mailman3/lists/code-quality.python.org/ Member address: graffatcolmingov@gmail.com
On 6/10/21 11:43 AM, Dan Stromberg wrote:
Hi folks.
Are there any tools available for system testing a large, shell-callable python script with many different command line options?
I'm aware of pytest for unit tests, but what about running a shell command with some options, and checking its stdout for appropriate content?
Thanks.
-- Dan Stromberg
There are probably lots of approaches to this. The project I mainly work on isn't using pytest because it predates it, we'll maybe convert someday but since it's unpaid on my part I have other priorities. It has extensive tests of getting the right results given various combinations of CLI options. The unit tests (they actually do use unittest, so converting those would be easy) live together with the code containing the units, the system/integration/end-to-end/pick-your-name tests are in a separate test directory, and the test runner can be told to choose one set or the other, default both, but with unit tests first - I think the normal scheme (and pytest has support for this) is to not run the integration tests if the unit tests don't pass. Pytest has support for this behavior with decorators ("markers"), the --ignore-path option, etc.
Hope this at least gives some food for thought, though it's hard to call it an "answer".
Maybe pexpect does what you want? https://pypi.org/project/pexpect/
We recently started using it to test make targets in the Open edX development environment, those tests are at https://github.com/edx/devstack/tree/master/tests if they're useful for reference.
Jeremy Bowman
On 2021-06-10 13:43, Dan Stromberg wrote:
Hi folks. Are there any tools available for system testing a large, shell-callable python script with many different command line options?
I'm aware of pytest for unit tests, but what about running a shell command with some options, and checking its stdout for appropriate content?
Thanks.
Dan Stromberg _______________________________________________ code-quality mailing list -- code-quality@python.org To unsubscribe send an email to code-quality-leave@python.org https://mail.python.org/mailman3/lists/code-quality.python.org/ Member address: code-quality@portabase.org
This is probably a little too specific for your particular case, but I'll suggest it anyway, because it's a set-up that I've found has worked well.
We've developed a number of post-processing validators at my employer and they are all Python scripts built with Click (https://click.palletsprojects.com/en/8.0.x/). They have a large number of options and parameters which interact with each other, plus they perform an important role, so we have to test them fully.
We use a 2-layered approach in the build pipeline:
* Firstly, a simple Make recipe calls the script (both plain and with each command) with `--help` and checks the output for some key phrases. This is effectively a smoke test to ensure that Click is wired in correctly.
* Secondly, we have "integration" tests built with pytest and using pytest_click plugin (https://pypi.org/project/pytest-click/). This allows easy access to the `CliRunner` documented here: https://click.palletsprojects.com/en/8.0.x/testing/ . It's in this layer that we test the permutations / combinations of options and parameters with "broad strokes". For example, we have a happy path test which ensures that a valid report passes checking - it looks like this:
def test(cli_runner, data_good_path: pathlib.Path) -> None: """ Set of data in 'all_test' dir is valid, taken from audit snapshots. """ in_dir = data_good_path / "all_test"
result = cli_runner.invoke(main, ["all", str(in_dir), "201901"])
assert result.exit_code == 0, result.output assert "--- Cross checking ---" in result.output assert "🎉 🎉 🎉" in result.output
Hope that's helpful,
James
On Fri, 11 Jun 2021, at 3:49 AM, Jeremy Bowman wrote:
Maybe pexpect does what you want? https://pypi.org/project/pexpect/
We recently started using it to test make targets in the Open edX development environment, those tests are at https://github.com/edx/devstack/tree/master/tests if they're useful for reference.
Jeremy Bowman
On 2021-06-10 13:43, Dan Stromberg wrote:
Hi folks.
Are there any tools available for system testing a large, shell-callable python script with many different command line options?
I'm aware of pytest for unit tests, but what about running a shell command with some options, and checking its stdout for appropriate content?
Thanks.
--
Dan Stromberg
code-quality mailing list -- code-quality@python.org To unsubscribe send an email to code-quality-leave@python.org https://mail.python.org/mailman3/lists/code-quality.python.org/ Member address: code-quality@portabase.org
code-quality mailing list -- code-quality@python.org mailto:code-quality%40python.org To unsubscribe send an email to code-quality-leave@python.org mailto:code-quality-leave%40python.org https://mail.python.org/mailman3/lists/code-quality.python.org/ Member address: me@jamescooke.info mailto:me%40jamescooke.info
You can use purest. It has a built in fixtures that can inspect stdout and stderr.
https://docs.pytest.org/en/6.2.x/reference.html
Bassam
On Thu, Jun 10, 2021 at 1:44 PM Dan Stromberg strombrg@gmail.com wrote:
Hi folks.
Are there any tools available for system testing a large, shell-callable python script with many different command line options?
I'm aware of pytest for unit tests, but what about running a shell command with some options, and checking its stdout for appropriate content?
Thanks.
-- Dan Stromberg _______________________________________________ code-quality mailing list -- code-quality@python.org To unsubscribe send an email to code-quality-leave@python.org https://mail.python.org/mailman3/lists/code-quality.python.org/ Member address: bassam.khouri@gmail.com