Thanks for the suggestion.

I think I tried something similar for tests that involved an environment variable and found that it doesn't play nicely with coverage.py at all.

Also, I will have to solve this problem at some point anyway because the property tests for the module (not currently included in the PR) include tests that have the C and pure Python version running side-by-side, which would be hard to achieve with subinterpreters.

On Wed, May 6, 2020 at 7:52 AM Paul Ganssle <paul@ganssle.io> wrote:

As part of PEP 399, an idiom for testing both C and pure Python versions of a library is suggested making use if import_fresh_module.

Unfortunately, I'm finding that this is not amazingly robust. We have this issue: https://bugs.python.org/issue40058, where the tester for datetime needs to do some funky manipulations to the state of sys.modules for reasons that are now somewhat unclear, and still sys.modules is apparently left in a bad state.

When implementing PEP 615, I ran into similar issues and found it very difficult to get two independent instances of the same module – one with the C extension blocked and one with it intact. I ended up manually importing the C and Python extensions and grafting them onto two "fresh" imports with nothing blocked.

When I've had to deal with similar issues in the past, I've given up
on messing with sys.modules and just had one test spawn a subprocess
to do the import+run the actual tests. It's a big hammer, but the nice
thing about big hammers is that there's no subtle issues, either they
smash the thing or they don't.

But, I don't know how awkward that would be to fit into Python's
unittest system, if you have lots of tests you need to run this way.

-n