[Mailman-Developers] How can I run the REST tests against live servers?

[Barry Warsaw]

On Dec 12, 2014, at 07:52 AM, Andrew Stuart wrote:

>I want to run the REST tests against my live Mailman server with its Postfix
>mail server and Postgres database, via:
>
>nose2 -v -P rest
>
>After many (many) hours of digging deep and figuring out how it works, I
>suspect at this stage that maybe the tests set up their own mock servers?  If
>so, I’m guessing these mock servers are getting in the way of using the live
>servers and thus preventing the tests from running?
>
>Assuming it is correct that the tests are setting up their own mock http smtp
>ltmp responders, is there a way to turn this behaviour off so the tests will
>run against my live REST/Postgres/Postfix?

Hi Andrew,

You're right that the test suite starts its own http and lmtp servers.  Take a
look at src/mailman/testing/layers.py for how this is controlled, and review
src/mailman/testing/nose.py and this documentation on nose2 support for Zope
testrunner-style layers:

https://nose2.readthedocs.org/en/latest/plugins/layers.html

I think even if you could somehow substitute real servers for the testing
ones, you'll still have a problem running the test suite against them, to any
semblance of success.  The tests are dependent on specific arrangements of
sample data in order to run successfully.  E.g. you'll see that the
ConfigLayer, which is the "base class"[1] creates an example.com domain, so
that all a test needs to do is create a mailing list in that domain without
first having to create the domain.

The test suite is also very careful to restore pristine state between
individual tests; look for reset_the_world() which is run after every test,
and does exactly what it describes.  This ensures that no state from one test
can affect the state of another test.  In the real world testing you propose
above, you would have quite a bit of bleed-through I suspect.

Can you describe why you want to run the test suite against a live Mailman
server?

I do think it would be interesting to build an integration test suite that
would talk to real MTAs and databases, but that would take a fair bit of work
and probably be separate from the unittests, which aim to gather 100% code
coverage.

Cheers,
-Barry

[1] Although layers don't really support normal Python class hierarchies.