On Dec 12, 2014, at 07:52 AM, Andrew Stuart wrote:
I want to run the REST tests against my live Mailman server with its Postfix mail server and Postgres database, via:
nose2 -v -P rest
After many (many) hours of digging deep and figuring out how it works, I suspect at this stage that maybe the tests set up their own mock servers? If so, I’m guessing these mock servers are getting in the way of using the live servers and thus preventing the tests from running?
Assuming it is correct that the tests are setting up their own mock http smtp ltmp responders, is there a way to turn this behaviour off so the tests will run against my live REST/Postgres/Postfix?
Hi Andrew,
You're right that the test suite starts its own http and lmtp servers. Take a look at src/mailman/testing/layers.py for how this is controlled, and review src/mailman/testing/nose.py and this documentation on nose2 support for Zope testrunner-style layers:
https://nose2.readthedocs.org/en/latest/plugins/layers.html
I think even if you could somehow substitute real servers for the testing ones, you'll still have a problem running the test suite against them, to any semblance of success. The tests are dependent on specific arrangements of sample data in order to run successfully. E.g. you'll see that the ConfigLayer, which is the "base class"[1] creates an example.com domain, so that all a test needs to do is create a mailing list in that domain without first having to create the domain.
The test suite is also very careful to restore pristine state between individual tests; look for reset_the_world() which is run after every test, and does exactly what it describes. This ensures that no state from one test can affect the state of another test. In the real world testing you propose above, you would have quite a bit of bleed-through I suspect.
Can you describe why you want to run the test suite against a live Mailman server?
I do think it would be interesting to build an integration test suite that would talk to real MTAs and databases, but that would take a fair bit of work and probably be separate from the unittests, which aim to gather 100% code coverage.
Cheers, -Barry
[1] Although layers don't really support normal Python class hierarchies.