Hi Barry
I’m writing an authenticating proxy for the Mailman REST API and want to make sure everything works as expected against real infrastructure (Postgres/Postfix/Sqlalchemy/Falcon).
I was hoping that just pointing the REST tests request port to my proxy would work but no dice - lots of failures - trying to diagnose the failures has led me back to trying to get the test suite to work against the live servers.
I’m wondering if maybe the shortest route here might not be for me to write a new set of tests which purely talk REST and don’t touch the back end directly at all. Except that’s not a short route. :-)
Trying to bend the existing tests to meet my needs is looking harder than I have time to deal with.
Andrew
On 12 Dec 2014, at 8:27 am, Barry Warsaw <barry@list.org> wrote:
On Dec 12, 2014, at 07:52 AM, Andrew Stuart wrote:
I want to run the REST tests against my live Mailman server with its Postfix mail server and Postgres database, via:
nose2 -v -P rest
After many (many) hours of digging deep and figuring out how it works, I suspect at this stage that maybe the tests set up their own mock servers? If so, I’m guessing these mock servers are getting in the way of using the live servers and thus preventing the tests from running?
Assuming it is correct that the tests are setting up their own mock http smtp ltmp responders, is there a way to turn this behaviour off so the tests will run against my live REST/Postgres/Postfix?
Hi Andrew,
You're right that the test suite starts its own http and lmtp servers. Take a look at src/mailman/testing/layers.py for how this is controlled, and review src/mailman/testing/nose.py and this documentation on nose2 support for Zope testrunner-style layers:
https://nose2.readthedocs.org/en/latest/plugins/layers.html
I think even if you could somehow substitute real servers for the testing ones, you'll still have a problem running the test suite against them, to any semblance of success. The tests are dependent on specific arrangements of sample data in order to run successfully. E.g. you'll see that the ConfigLayer, which is the "base class"[1] creates an example.com domain, so that all a test needs to do is create a mailing list in that domain without first having to create the domain.
The test suite is also very careful to restore pristine state between individual tests; look for reset_the_world() which is run after every test, and does exactly what it describes. This ensures that no state from one test can affect the state of another test. In the real world testing you propose above, you would have quite a bit of bleed-through I suspect.
Can you describe why you want to run the test suite against a live Mailman server?
I do think it would be interesting to build an integration test suite that would talk to real MTAs and databases, but that would take a fair bit of work and probably be separate from the unittests, which aim to gather 100% code coverage.
Cheers, -Barry
[1] Although layers don't really support normal Python class hierarchies.
Mailman-Developers mailing list Mailman-Developers@python.org https://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/andrew.stuart%40s...
Security Policy: http://wiki.list.org/x/QIA9