Andrew Stuart writes:
I’m writing an authenticating proxy for the Mailman REST API and want to make sure everything works as expected against real infrastructure (Postgres/Postfix/Sqlalchemy/Falcon).
Determining that "things work as expected against real infrastructure" is not what the test suite does. AFAIK we don't have any such tests, and I don't really see how the tests we do have can be adapted to "tests against real infrastructure". The problem is specifying what to expect of real infrastructure.
You could, for example, write tests that insert users, query for them, and then delete them, but this has several problems. Do you want to do these tests on "live" servers that might actually be in use? -- Probably not, but how do you prevent that? Attributes of the various objects created by the system may depend on context. -- The test needs to ignore any differences due merely to different context. Coverage is necessarily imperfect in the sense that some internal state changes can't be checked since the "before" state is indeterminate in a live system. And so on.