[pytest-dev] Creating a complex PyTest program, wondering about the best way to handle collection errors, etc.

[Nicholas Williams]

Wow. After carefully reading the pytest-cov plugin (only) code and the
PyTest code that powers loading plugins, I figured it out. The plugin was
getting loaded before pytest-cov started the Coverage analyzer. Since the
plugin had module-level imports from other parts of our project, that meant
that most of the project was imported (and, so, module-level imports and
code and class definitions had already executed) before Coverage started.
The fix was actually very simple: I moved the plugin into a different
module and then moved its top-level imports for the same project to happen
at the last second. Viola. Coverage is fixed. Now just the plugin module
itself reports wonky coverage numbers, but I can live with that.

Thanks again for your help!

Nick


On Mon, Apr 23, 2018 at 5:42 PM, Nicholas Williams <
nicholas+pytest at nicholaswilliams.net> wrote:

> Floris, your advice was invaluable. We have created a really fantastic
> PyTest plugin for our service tests. However, I wanted to share it with
> you, because it seems to have introduced some type of weird conflict with
> the PyTest Coverage plugin, and I was hoping maybe you had some insights as
> to what the problem might be:
>
> Normally, when I run PyTest tests with the PyTest Coverage plugin,
> everything works as expected in regards to the coverage calculation and the
> coverage reports. And, with this change to our project, everything STILL
> works properly IF AND ONLY IF I run tests with `coverage run --source=pysoa
> setup.py test` followed by `coverage report` (so, without the PyTest
> Coverage plugin). However, with this change to our project, something VERY
> weird happens to the coverage report if I rely on the PyTest Coverage
> plugin. Now, ONLY LINES OF CODE WITHIN FUNCTIONS AND METHODS are marked as
> covered. The following types of code, which were previously marked as
> covered (and are clearly covered), are now marked as uncovered (again, only
> when using the PyTest Coverage plugin):
>
> - module-level imports
> - module-level code that executes on import
> - module docstrings
> - module-level class definitions (class ClassName(...):), or class
> definitions within other class definitions
> - code that executes within a class on definition
> - class docstrings
> - function definition lines (def func_name(...):), but not the code within
> the functions, which is marked as covered
> - method definition lines within classes (def method_name(...):), but not
> the code within the functions, which is marked as covered
>
> While the entire pull request is large and daunting, the most relevant
> part is the PyTest plugin itself, which is contained in a single file
> "pytest_plugin.py" that you can search for on the page. Here it is:
> https://github.com/eventbrite/pysoa/pull/87/files
>
> Unfortunately, I did not notice this problem with coverage until the bulk
> of the code was written and tested, so determining what part of the code
> caused it by removing bits of code is ... impracticable. I'm hoping one of
> the pytest-cov experts in here will have some idea what the heck is going
> on and can point me in the right direction. Perhaps I'm doing something
> wrong, but this smells very strongly of a pytest-cov bug somehow.
>
> Thanks,
>
> Nick
>
>
> On Fri, Apr 20, 2018 at 2:50 PM, Floris Bruynooghe <flub at devork.be> wrote:
>
>> On Thu 19 Apr 2018 at 16:50 -0500, Nicholas Williams wrote:
>>
>> > The tests work this way:
>> >
>> > - There's a test class that inherits from our specialized test class
>> > (ServicePlanTestCase), which inherits from `unittest.TestCase` (for
>> several
>> > reasons, this detail cannot be altered)
>> > - That test class can do setup and teardown like a normal test class,
>> but
>> > it has a static attribute that points to a directory
>> > - That directory contains one or more files ending in a particular
>> > extension, and those files each contain one or more tests defined using
>> a
>> > particular syntax
>>
>> I'm not even going to ask how this all came to be this way... but thanks
>> for giving enough context.
>>
>> > Based on the hints you gave me, it sounds like I could do something like
>> > this:
>> > https://github.com/pytest-dev/pytest/blob/4678cbeb913385f00c
>> c21b79662459a8c9fafa87/_pytest/unittest.py#L14-L22
>> >
>> > Only, instead of checking for inheritance from TestCase, I'd check for
>> > inheritance from our ServicePlanTestCase, and in that case I would
>> return a
>> > new collector object that inherits from _pytest.python.Class, and write
>> > that new collector class to collect our tests. Am I barking up the right
>> > tree now?
>>
>> pytest_pycollect_makeitem is supposed to create a single item,
>> while from what you describe you still have two collections nested.  You
>> should probably create a custom collection node for your class, which
>> then creates a different kind of collection node for each file which in
>> turn creates the actual test nodes in it's .collect().
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20180424/db04c03d/attachment-0001.html>