Creating a complex PyTest program, wondering about the best way to handle collection errors, etc.
Hello, all. This is my first post to the mailing list, FWIW. I've been a PyTest user for some years, but now I'm working on writing a rather complex PyTest plugin to facilitate testing our service oriented architecture at $work. The purpose of the plugin is to collect additional tests (for which we are using `pytest_collection_modifyitems`) from a specialized file format that PyTest wouldn't normally understand. I've got it working beautifully, but I would love to hammer out some finer details to really make it a clean-running plugin. With that in mind, I'm looking for some suggestions / best practices / advice here on how best to handle these situations: Collection Errors The author of one of these specialized tests, which we call "test plans," could possibly do a few things wrong that cause more than just making their test fail. For example, they could get their file syntax incorrect, which results in a parsing error. At the moment, I catch the parsing error (we're using PyParsing to parse our test plans) and raise a different exception about the problem (and I've tried AssertionError), but everything I do results in a stack trace dumped with "INTERNALERROR>" and the immediate, abnormal termination of the PyTest process. This contrasts to, for an analog example, a syntax error in a regular Python test file, which instead results in a "ERROR collecting" output, while all other unaffected tests still run and the PyTest process terminates normally. What I'm looking for here is the "right way" to make my plugin report an "ERROR collecting" instead of causing an "INTERNALERROR>" when such a problem occurs collecting these specialized tests. Improving the Test Error Report When tests have errors or failures, PyTest prints out a report of those errors, and with each one includes an exhaustive stack trace. For our specialized tests, with some clever manipulation of the test data added to `items` in `pytest_collection_modifyitems`, I've managed to at least get it to display a meaningful name for the test that failed so that it's at least _possible_ for the developer to figure out which test, in which test file, failed. However, almost the entire stack trace is meaningless and full of plugin code instead of containing information useful to debugging the test. Additionally, I would _love_ to be able to add the actual test file and line number for the failure to the report. Would this be possible? Or, at least, would it be possible to filter the plugin code out of the stack trace so that it doesn't distract the developer from simply viewing the assertion error details? What approach should I take to achieve this? Thanks in advance for any help you can provide, Nick
Nicholas, Regarding the stack trace filtering, I have been bitten by this too in my custom plugin. You can read the solution here in the mail archives: https://mail.python.org/pipermail/pytest-dev/2018-March/004399.html If it is not clear enough, just reply and I might be able to help you forward. Ringo On Wed, Apr 18, 2018 at 8:19 PM, Nicholas Williams < nicholas+pytest@nicholaswilliams.net> wrote:
Hello, all. This is my first post to the mailing list, FWIW.
I've been a PyTest user for some years, but now I'm working on writing a rather complex PyTest plugin to facilitate testing our service oriented architecture at $work. The purpose of the plugin is to collect additional tests (for which we are using `pytest_collection_modifyitems`) from a specialized file format that PyTest wouldn't normally understand. I've got it working beautifully, but I would love to hammer out some finer details to really make it a clean-running plugin.
With that in mind, I'm looking for some suggestions / best practices / advice here on how best to handle these situations:
Collection Errors The author of one of these specialized tests, which we call "test plans," could possibly do a few things wrong that cause more than just making their test fail. For example, they could get their file syntax incorrect, which results in a parsing error. At the moment, I catch the parsing error (we're using PyParsing to parse our test plans) and raise a different exception about the problem (and I've tried AssertionError), but everything I do results in a stack trace dumped with "INTERNALERROR>" and the immediate, abnormal termination of the PyTest process. This contrasts to, for an analog example, a syntax error in a regular Python test file, which instead results in a "ERROR collecting" output, while all other unaffected tests still run and the PyTest process terminates normally. What I'm looking for here is the "right way" to make my plugin report an "ERROR collecting" instead of causing an "INTERNALERROR>" when such a problem occurs collecting these specialized tests.
Improving the Test Error Report When tests have errors or failures, PyTest prints out a report of those errors, and with each one includes an exhaustive stack trace. For our specialized tests, with some clever manipulation of the test data added to `items` in `pytest_collection_modifyitems`, I've managed to at least get it to display a meaningful name for the test that failed so that it's at least _possible_ for the developer to figure out which test, in which test file, failed. However, almost the entire stack trace is meaningless and full of plugin code instead of containing information useful to debugging the test. Additionally, I would _love_ to be able to add the actual test file and line number for the failure to the report. Would this be possible? Or, at least, would it be possible to filter the plugin code out of the stack trace so that it doesn't distract the developer from simply viewing the assertion error details? What approach should I take to achieve this?
Thanks in advance for any help you can provide,
Nick
_______________________________________________ pytest-dev mailing list pytest-dev@python.org https://mail.python.org/mailman/listinfo/pytest-dev
-- *Ringo De Smet* ringo.de.smet@ontoforce.com
Hi Nick, On Wed 18 Apr 2018 at 13:19 -0500, Nicholas Williams wrote:
Hello, all. This is my first post to the mailing list, FWIW.
I've been a PyTest user for some years, but now I'm working on writing a rather complex PyTest plugin to facilitate testing our service oriented architecture at $work. The purpose of the plugin is to collect additional tests (for which we are using `pytest_collection_modifyitems`) from a
I think you'll fare better by integrating deeper into the collection mechanism instead of using pytest_collection_modifyitems. How are these test written? You probably should look at the pytest_collect_directory and pytest_collect_file hooks to do this. Unfortunately the documentation on these things is very thin. Finding existing code that does this is probably your best bet. The _pytest/python.py plugin is the canonical example though is very complicated. If you hunt other plugins you may find other examples.
specialized file format that PyTest wouldn't normally understand. I've got it working beautifully, but I would love to hammer out some finer details to really make it a clean-running plugin.
With that in mind, I'm looking for some suggestions / best practices / advice here on how best to handle these situations:
Collection Errors The author of one of these specialized tests, which we call "test plans," could possibly do a few things wrong that cause more than just making their test fail. For example, they could get their file syntax incorrect, which results in a parsing error. At the moment, I catch the parsing error (we're using PyParsing to parse our test plans) and raise a different exception about the problem (and I've tried AssertionError), but everything I do results in a stack trace dumped with "INTERNALERROR>" and the immediate, abnormal termination of the PyTest process. This contrasts to, for an analog example, a syntax error in a regular Python test file, which instead results in a "ERROR collecting" output, while all other unaffected tests still run and the PyTest process terminates normally. What I'm looking for here is the "right way" to make my plugin report an "ERROR collecting" instead of causing an "INTERNALERROR>" when such a problem occurs collecting these specialized tests.
I think these problems would go away if you join the collection mechanism "correctly". Not that "correctly" is necessarily easy or obvious... but the hints above might help you.
Improving the Test Error Report
I think this was already answered. Cheers, Floris
The tests work this way: - There's a test class that inherits from our specialized test class (ServicePlanTestCase), which inherits from `unittest.TestCase` (for several reasons, this detail cannot be altered) - That test class can do setup and teardown like a normal test class, but it has a static attribute that points to a directory - That directory contains one or more files ending in a particular extension, and those files each contain one or more tests defined using a particular syntax Based on the hints you gave me, it sounds like I could do something like this: https://github.com/pytest-dev/pytest/blob/4678cbeb913385f00cc21b79662459a8c9... Only, instead of checking for inheritance from TestCase, I'd check for inheritance from our ServicePlanTestCase, and in that case I would return a new collector object that inherits from _pytest.python.Class, and write that new collector class to collect our tests. Am I barking up the right tree now? Thanks, Nick On Thu, Apr 19, 2018 at 2:18 PM, Floris Bruynooghe <flub@devork.be> wrote:
Hi Nick,
On Wed 18 Apr 2018 at 13:19 -0500, Nicholas Williams wrote:
Hello, all. This is my first post to the mailing list, FWIW.
I've been a PyTest user for some years, but now I'm working on writing a rather complex PyTest plugin to facilitate testing our service oriented architecture at $work. The purpose of the plugin is to collect additional tests (for which we are using `pytest_collection_modifyitems`) from a
I think you'll fare better by integrating deeper into the collection mechanism instead of using pytest_collection_modifyitems. How are these test written? You probably should look at the pytest_collect_directory and pytest_collect_file hooks to do this. Unfortunately the documentation on these things is very thin. Finding existing code that does this is probably your best bet. The _pytest/python.py plugin is the canonical example though is very complicated. If you hunt other plugins you may find other examples.
specialized file format that PyTest wouldn't normally understand. I've got it working beautifully, but I would love to hammer out some finer details to really make it a clean-running plugin.
With that in mind, I'm looking for some suggestions / best practices / advice here on how best to handle these situations:
Collection Errors The author of one of these specialized tests, which we call "test plans," could possibly do a few things wrong that cause more than just making their test fail. For example, they could get their file syntax incorrect, which results in a parsing error. At the moment, I catch the parsing error (we're using PyParsing to parse our test plans) and raise a different exception about the problem (and I've tried AssertionError), but everything I do results in a stack trace dumped with "INTERNALERROR>" and the immediate, abnormal termination of the PyTest process. This contrasts to, for an analog example, a syntax error in a regular Python test file, which instead results in a "ERROR collecting" output, while all other unaffected tests still run and the PyTest process terminates normally. What I'm looking for here is the "right way" to make my plugin report an "ERROR collecting" instead of causing an "INTERNALERROR>" when such a problem occurs collecting these specialized tests.
I think these problems would go away if you join the collection mechanism "correctly". Not that "correctly" is necessarily easy or obvious... but the hints above might help you.
Improving the Test Error Report
I think this was already answered.
Cheers, Floris
On Thu 19 Apr 2018 at 16:50 -0500, Nicholas Williams wrote:
The tests work this way:
- There's a test class that inherits from our specialized test class (ServicePlanTestCase), which inherits from `unittest.TestCase` (for several reasons, this detail cannot be altered) - That test class can do setup and teardown like a normal test class, but it has a static attribute that points to a directory - That directory contains one or more files ending in a particular extension, and those files each contain one or more tests defined using a particular syntax
I'm not even going to ask how this all came to be this way... but thanks for giving enough context.
Based on the hints you gave me, it sounds like I could do something like this: https://github.com/pytest-dev/pytest/blob/4678cbeb913385f00cc21b79662459a8c9...
Only, instead of checking for inheritance from TestCase, I'd check for inheritance from our ServicePlanTestCase, and in that case I would return a new collector object that inherits from _pytest.python.Class, and write that new collector class to collect our tests. Am I barking up the right tree now?
pytest_pycollect_makeitem is supposed to create a single item, while from what you describe you still have two collections nested. You should probably create a custom collection node for your class, which then creates a different kind of collection node for each file which in turn creates the actual test nodes in it's .collect().
Floris, your advice was invaluable. We have created a really fantastic PyTest plugin for our service tests. However, I wanted to share it with you, because it seems to have introduced some type of weird conflict with the PyTest Coverage plugin, and I was hoping maybe you had some insights as to what the problem might be: Normally, when I run PyTest tests with the PyTest Coverage plugin, everything works as expected in regards to the coverage calculation and the coverage reports. And, with this change to our project, everything STILL works properly IF AND ONLY IF I run tests with `coverage run --source=pysoa setup.py test` followed by `coverage report` (so, without the PyTest Coverage plugin). However, with this change to our project, something VERY weird happens to the coverage report if I rely on the PyTest Coverage plugin. Now, ONLY LINES OF CODE WITHIN FUNCTIONS AND METHODS are marked as covered. The following types of code, which were previously marked as covered (and are clearly covered), are now marked as uncovered (again, only when using the PyTest Coverage plugin): - module-level imports - module-level code that executes on import - module docstrings - module-level class definitions (class ClassName(...):), or class definitions within other class definitions - code that executes within a class on definition - class docstrings - function definition lines (def func_name(...):), but not the code within the functions, which is marked as covered - method definition lines within classes (def method_name(...):), but not the code within the functions, which is marked as covered While the entire pull request is large and daunting, the most relevant part is the PyTest plugin itself, which is contained in a single file "pytest_plugin.py" that you can search for on the page. Here it is: https://github.com/eventbrite/pysoa/pull/87/files Unfortunately, I did not notice this problem with coverage until the bulk of the code was written and tested, so determining what part of the code caused it by removing bits of code is ... impracticable. I'm hoping one of the pytest-cov experts in here will have some idea what the heck is going on and can point me in the right direction. Perhaps I'm doing something wrong, but this smells very strongly of a pytest-cov bug somehow. Thanks, Nick On Fri, Apr 20, 2018 at 2:50 PM, Floris Bruynooghe <flub@devork.be> wrote:
On Thu 19 Apr 2018 at 16:50 -0500, Nicholas Williams wrote:
The tests work this way:
- There's a test class that inherits from our specialized test class (ServicePlanTestCase), which inherits from `unittest.TestCase` (for several reasons, this detail cannot be altered) - That test class can do setup and teardown like a normal test class, but it has a static attribute that points to a directory - That directory contains one or more files ending in a particular extension, and those files each contain one or more tests defined using a particular syntax
I'm not even going to ask how this all came to be this way... but thanks for giving enough context.
Based on the hints you gave me, it sounds like I could do something like this: https://github.com/pytest-dev/pytest/blob/4678cbeb913385f00cc21b79662459 a8c9fafa87/_pytest/unittest.py#L14-L22
Only, instead of checking for inheritance from TestCase, I'd check for inheritance from our ServicePlanTestCase, and in that case I would return a new collector object that inherits from _pytest.python.Class, and write that new collector class to collect our tests. Am I barking up the right tree now?
pytest_pycollect_makeitem is supposed to create a single item, while from what you describe you still have two collections nested. You should probably create a custom collection node for your class, which then creates a different kind of collection node for each file which in turn creates the actual test nodes in it's .collect().
Wow. After carefully reading the pytest-cov plugin (only) code and the PyTest code that powers loading plugins, I figured it out. The plugin was getting loaded before pytest-cov started the Coverage analyzer. Since the plugin had module-level imports from other parts of our project, that meant that most of the project was imported (and, so, module-level imports and code and class definitions had already executed) before Coverage started. The fix was actually very simple: I moved the plugin into a different module and then moved its top-level imports for the same project to happen at the last second. Viola. Coverage is fixed. Now just the plugin module itself reports wonky coverage numbers, but I can live with that. Thanks again for your help! Nick On Mon, Apr 23, 2018 at 5:42 PM, Nicholas Williams < nicholas+pytest@nicholaswilliams.net> wrote:
Floris, your advice was invaluable. We have created a really fantastic PyTest plugin for our service tests. However, I wanted to share it with you, because it seems to have introduced some type of weird conflict with the PyTest Coverage plugin, and I was hoping maybe you had some insights as to what the problem might be:
Normally, when I run PyTest tests with the PyTest Coverage plugin, everything works as expected in regards to the coverage calculation and the coverage reports. And, with this change to our project, everything STILL works properly IF AND ONLY IF I run tests with `coverage run --source=pysoa setup.py test` followed by `coverage report` (so, without the PyTest Coverage plugin). However, with this change to our project, something VERY weird happens to the coverage report if I rely on the PyTest Coverage plugin. Now, ONLY LINES OF CODE WITHIN FUNCTIONS AND METHODS are marked as covered. The following types of code, which were previously marked as covered (and are clearly covered), are now marked as uncovered (again, only when using the PyTest Coverage plugin):
- module-level imports - module-level code that executes on import - module docstrings - module-level class definitions (class ClassName(...):), or class definitions within other class definitions - code that executes within a class on definition - class docstrings - function definition lines (def func_name(...):), but not the code within the functions, which is marked as covered - method definition lines within classes (def method_name(...):), but not the code within the functions, which is marked as covered
While the entire pull request is large and daunting, the most relevant part is the PyTest plugin itself, which is contained in a single file "pytest_plugin.py" that you can search for on the page. Here it is: https://github.com/eventbrite/pysoa/pull/87/files
Unfortunately, I did not notice this problem with coverage until the bulk of the code was written and tested, so determining what part of the code caused it by removing bits of code is ... impracticable. I'm hoping one of the pytest-cov experts in here will have some idea what the heck is going on and can point me in the right direction. Perhaps I'm doing something wrong, but this smells very strongly of a pytest-cov bug somehow.
Thanks,
Nick
On Fri, Apr 20, 2018 at 2:50 PM, Floris Bruynooghe <flub@devork.be> wrote:
On Thu 19 Apr 2018 at 16:50 -0500, Nicholas Williams wrote:
The tests work this way:
- There's a test class that inherits from our specialized test class (ServicePlanTestCase), which inherits from `unittest.TestCase` (for several reasons, this detail cannot be altered) - That test class can do setup and teardown like a normal test class, but it has a static attribute that points to a directory - That directory contains one or more files ending in a particular extension, and those files each contain one or more tests defined using a particular syntax
I'm not even going to ask how this all came to be this way... but thanks for giving enough context.
Based on the hints you gave me, it sounds like I could do something like this: https://github.com/pytest-dev/pytest/blob/4678cbeb913385f00c c21b79662459a8c9fafa87/_pytest/unittest.py#L14-L22
Only, instead of checking for inheritance from TestCase, I'd check for inheritance from our ServicePlanTestCase, and in that case I would return a new collector object that inherits from _pytest.python.Class, and write that new collector class to collect our tests. Am I barking up the right tree now?
pytest_pycollect_makeitem is supposed to create a single item, while from what you describe you still have two collections nested. You should probably create a custom collection node for your class, which then creates a different kind of collection node for each file which in turn creates the actual test nodes in it's .collect().
Hi Nick, On Tue 24 Apr 2018 at 07:03 -0500, Nicholas Williams wrote:
Wow. After carefully reading the pytest-cov plugin (only) code and the PyTest code that powers loading plugins, I figured it out. The plugin was getting loaded before pytest-cov started the Coverage analyzer. Since the plugin had module-level imports from other parts of our project, that meant that most of the project was imported (and, so, module-level imports and code and class definitions had already executed) before Coverage started.
Nice you figured this annoying issue out while I was ignoring my mail for a while... :) This is a bit brittle but unavoidable unfortunately.
Thanks again for your help!
Glad my vague pointers proved useful! Cheers, Floris
One thing that concerned me in this whole process is that the vast majority of PyTest code appears to be behind a private API named `_pytest`. I was unable to implement this plugin without accessing these APIs. I could find no way around it. I minimized the imports as much as I could, even copying smaller sections of code instead of importing, but I still had to make these imports: from _pytest.unittest import ( TestCaseFunction, UnitTestCase, ) from _pytest._code.code import TracebackEntry from _pytest._code.source import Source from _pytest.mark import MARK_GEN How much are we going to be able to rely on these staying fairly stable? I'd feel much better if these were public APIs. :-/ Thanks, Nick On Fri, Apr 27, 2018 at 3:06 PM, Floris Bruynooghe <flub@devork.be> wrote:
Hi Nick,
On Tue 24 Apr 2018 at 07:03 -0500, Nicholas Williams wrote:
Wow. After carefully reading the pytest-cov plugin (only) code and the PyTest code that powers loading plugins, I figured it out. The plugin was getting loaded before pytest-cov started the Coverage analyzer. Since the plugin had module-level imports from other parts of our project, that meant that most of the project was imported (and, so, module-level imports and code and class definitions had already executed) before Coverage started.
Nice you figured this annoying issue out while I was ignoring my mail for a while... :) This is a bit brittle but unavoidable unfortunately.
Thanks again for your help!
Glad my vague pointers proved useful!
Cheers, Floris
On Fri 27 Apr 2018 at 15:15 -0500, Nicholas Williams wrote:
One thing that concerned me in this whole process is that the vast majority of PyTest code appears to be behind a private API named `_pytest`. I was unable to implement this plugin without accessing these APIs. I could find no way around it. I minimized the imports as much as I could, even copying smaller sections of code instead of importing, but I still had to make these imports:
from _pytest.unittest import ( TestCaseFunction, UnitTestCase, )
I think the intention was the public API would be pytest.File, pytest.Function, pytest.Collector, pytest.Instance and pytest.Item (listed here in no particular order). Your use case seems to want to customise the existing unittest support which IIRC was never imagined this would happen. Hence you're either stuck re-creating lots of things for the unittest support using the public APIs I've listed or importing the private things I guess. So maybe you have a valid case for wanting pytest to expose a few more of the collecting semantics. Would be best to discuss and examine this in an issue/feature request I guess (if you do so please mention me in the issue as otherwise I'll probably miss it)
from _pytest._code.code import TracebackEntry from _pytest._code.source import Source from _pytest.mark import MARK_GEN
These are maybe somewhat more troubling, I'd have to look into these carefully.
How much are we going to be able to rely on these staying fairly stable? I'd feel much better if these were public APIs. :-/
As I kind of suggest above, the second lot seems a lot scarier to me and more likely to easily break. Regards, Floris
participants (3)
-
Floris Bruynooghe -
Nicholas Williams -
Ringo De Smet