System testing a large, shell-callable python script
Hi folks. Are there any tools available for system testing a large, shell-callable python script with many different command line options? I'm aware of pytest for unit tests, but what about running a shell command with some options, and checking its stdout for appropriate content? Thanks. -- Dan Stromberg
I was looking for something else that pip used to use and I think Doug Hellman wrote but couldn't find it. I did find https://pypi.org/project/pytest-shell/ Sent from my phone with my typo-happy thumbs. Please excuse my brevity On Thu, Jun 10, 2021, 12:44 Dan Stromberg <strombrg@gmail.com> wrote:
Hi folks.
Are there any tools available for system testing a large, shell-callable python script with many different command line options?
I'm aware of pytest for unit tests, but what about running a shell command with some options, and checking its stdout for appropriate content?
Thanks.
-- Dan Stromberg _______________________________________________ code-quality mailing list -- code-quality@python.org To unsubscribe send an email to code-quality-leave@python.org https://mail.python.org/mailman3/lists/code-quality.python.org/ Member address: graffatcolmingov@gmail.com
On 6/10/21 11:43 AM, Dan Stromberg wrote:
Hi folks.
Are there any tools available for system testing a large, shell-callable python script with many different command line options?
I'm aware of pytest for unit tests, but what about running a shell command with some options, and checking its stdout for appropriate content?
Thanks.
-- Dan Stromberg
There are probably lots of approaches to this. The project I mainly work on isn't using pytest because it predates it, we'll maybe convert someday but since it's unpaid on my part I have other priorities. It has extensive tests of getting the right results given various combinations of CLI options. The unit tests (they actually do use unittest, so converting those would be easy) live together with the code containing the units, the system/integration/end-to-end/pick-your-name tests are in a separate test directory, and the test runner can be told to choose one set or the other, default both, but with unit tests first - I think the normal scheme (and pytest has support for this) is to not run the integration tests if the unit tests don't pass. Pytest has support for this behavior with decorators ("markers"), the --ignore-path option, etc. Hope this at least gives some food for thought, though it's hard to call it an "answer".
Maybe pexpect does what you want? https://pypi.org/project/pexpect/ We recently started using it to test make targets in the Open edX development environment, those tests are at https://github.com/edx/devstack/tree/master/tests if they're useful for reference. Jeremy Bowman On 2021-06-10 13:43, Dan Stromberg wrote:
Hi folks. Are there any tools available for system testing a large, shell-callable python script with many different command line options?
I'm aware of pytest for unit tests, but what about running a shell command with some options, and checking its stdout for appropriate content?
Thanks. -- Dan Stromberg _______________________________________________ code-quality mailing list -- code-quality@python.org To unsubscribe send an email to code-quality-leave@python.org https://mail.python.org/mailman3/lists/code-quality.python.org/ Member address: code-quality@portabase.org
This is probably a little too specific for your particular case, but I'll suggest it anyway, because it's a set-up that I've found has worked well. We've developed a number of post-processing validators at my employer and they are all Python scripts built with Click (https://click.palletsprojects.com/en/8.0.x/). They have a large number of options and parameters which interact with each other, plus they perform an important role, so we have to test them fully. We use a 2-layered approach in the build pipeline: * Firstly, a simple Make recipe calls the script (both plain and with each command) with `--help` and checks the output for some key phrases. This is effectively a smoke test to ensure that Click is wired in correctly. * Secondly, we have "integration" tests built with pytest and using pytest_click plugin (https://pypi.org/project/pytest-click/). This allows easy access to the `CliRunner` documented here: https://click.palletsprojects.com/en/8.0.x/testing/ . It's in this layer that we test the permutations / combinations of options and parameters with "broad strokes". For example, we have a happy path test which ensures that a valid report passes checking - it looks like this: def test(cli_runner, data_good_path: pathlib.Path) -> None: """ Set of data in 'all_test' dir is valid, taken from audit snapshots. """ in_dir = data_good_path / "all_test" result = cli_runner.invoke(main, ["all", str(in_dir), "201901"]) assert result.exit_code == 0, result.output assert "--- Cross checking ---" in result.output assert "🎉 🎉 🎉" in result.output Hope that's helpful, James On Fri, 11 Jun 2021, at 3:49 AM, Jeremy Bowman wrote:
Maybe pexpect does what you want? https://pypi.org/project/pexpect/
We recently started using it to test make targets in the Open edX development environment, those tests are at https://github.com/edx/devstack/tree/master/tests if they're useful for reference.
Jeremy Bowman
On 2021-06-10 13:43, Dan Stromberg wrote:
Hi folks.
Are there any tools available for system testing a large, shell-callable python script with many different command line options?
I'm aware of pytest for unit tests, but what about running a shell command with some options, and checking its stdout for appropriate content?
Thanks.
--
Dan Stromberg
_______________________________________________ code-quality mailing list -- code-quality@python.org To unsubscribe send an email to code-quality-leave@python.org https://mail.python.org/mailman3/lists/code-quality.python.org/ Member address: code-quality@portabase.org
_______________________________________________ code-quality mailing list -- code-quality@python.org <mailto:code-quality%40python.org> To unsubscribe send an email to code-quality-leave@python.org <mailto:code-quality-leave%40python.org> https://mail.python.org/mailman3/lists/code-quality.python.org/ Member address: me@jamescooke.info <mailto:me%40jamescooke.info>
You can use purest. It has a built in fixtures that can inspect stdout and stderr. https://docs.pytest.org/en/6.2.x/reference.html Bassam On Thu, Jun 10, 2021 at 1:44 PM Dan Stromberg <strombrg@gmail.com> wrote:
Hi folks.
Are there any tools available for system testing a large, shell-callable python script with many different command line options?
I'm aware of pytest for unit tests, but what about running a shell command with some options, and checking its stdout for appropriate content?
Thanks.
-- Dan Stromberg _______________________________________________ code-quality mailing list -- code-quality@python.org To unsubscribe send an email to code-quality-leave@python.org https://mail.python.org/mailman3/lists/code-quality.python.org/ Member address: bassam.khouri@gmail.com
-- --- "What we can or cannot do, what we consider possible or impossible, is rarely a function of our true capability. It is more likely a function of our beliefs about who we are." - Tony Robbins
participants (6)
-
Bassam Khouri -
Dan Stromberg -
Ian Stapleton Cordasco -
James Cooke -
Jeremy Bowman -
Mats Wichmann