On Dec 12, 2014, at 08:35 AM, Andrew Stuart wrote:

I’m writing an authenticating proxy for the Mailman REST API and want to make sure everything works as expected against real infrastructure (Postgres/Postfix/Sqlalchemy/Falcon).

We definitely want such a proxy. It's always been our intent that the core's REST API is an adminipstrative API with full access. It should only be exposed on localhost or other highly secure interface.

A proxy server such as you describe would be available on a public interface and would allow people to script the management of their list memberships and list administrations.

I was hoping that just pointing the REST tests request port to my proxy would work but no dice - lots of failures - trying to diagnose the failures has led me back to trying to get the test suite to work against the live servers.

I’m wondering if maybe the shortest route here might not be for me to write a new set of tests which purely talk REST and don’t touch the back end directly at all. Except that’s not a short route. :-)

Trying to bend the existing tests to meet my needs is looking harder than I have time to deal with.

Even in this case, I wouldn't expect the standard unittests to work. For example, if a normal user were to try to access the list admin API, you'd expect that to fail.

I think one approach would be to continue to use the test servers, but have a backdoor connection which would let you set up the state before each test and reset the state after each test. Then the tests themselves would use the authenticated API to flex the things it can and shouldn't be able to do. It would be possible for example to expose some helper resources in the admin API which would not normally be exposed in production. Something under a top level /tests resource perhaps.

Cheers, -Barry

Barry - would you like the authenticating proxy to be part of Mailman core or a third party project?

It’s close to being functionally operational (which is to say working but in a raw state).

as

On 12 Dec 2014, at 11:34 am, Barry Warsaw <barry@list.org> wrote:

On Dec 12, 2014, at 08:35 AM, Andrew Stuart wrote:

I’m writing an authenticating proxy for the Mailman REST API and want to make sure everything works as expected against real infrastructure (Postgres/Postfix/Sqlalchemy/Falcon).

We definitely want such a proxy. It's always been our intent that the core's REST API is an adminipstrative API with full access. It should only be exposed on localhost or other highly secure interface.

A proxy server such as you describe would be available on a public interface and would allow people to script the management of their list memberships and list administrations.

I was hoping that just pointing the REST tests request port to my proxy would work but no dice - lots of failures - trying to diagnose the failures has led me back to trying to get the test suite to work against the live servers.

I’m wondering if maybe the shortest route here might not be for me to write a new set of tests which purely talk REST and don’t touch the back end directly at all. Except that’s not a short route. :-)

Trying to bend the existing tests to meet my needs is looking harder than I have time to deal with.

Even in this case, I wouldn't expect the standard unittests to work. For example, if a normal user were to try to access the list admin API, you'd expect that to fail.

I think one approach would be to continue to use the test servers, but have a backdoor connection which would let you set up the state before each test and reset the state after each test. Then the tests themselves would use the authenticated API to flex the things it can and shouldn't be able to do. It would be possible for example to expose some helper resources in the admin API which would not normally be exposed in production. Something under a top level /tests resource perhaps.

Cheers, -Barry

Mailman-Developers mailing list Mailman-Developers@python.org https://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/andrew.stuart%40s...

Security Policy: http://wiki.list.org/x/QIA9

Sure thing that’s fine.

FYI the Mailman REST authentication proxy works in this way:

** Python 3 only ** built using Falcon ** the proxy reads the structure of the Mailman 3 REST API by parsing the Mailman 3 Swagger API specification ** each Mailman 3 REST API URL/HTTP method call combination is defined in the Swagger spec and has a unique OperationID ** the proxy intercepts all inbound requests to the Mailman 3 REST API, this is done through Falcon’s “sink” capability ** the proxy does a regex match on the inbound URL and looks up the operationID to work out which Mailman 3 REST API method is being called ** in the proxy a set of request handlers is defined for each Mailman 3 API operation ** the inbound request is pushed through the request handlers, typically the final request handler proxies the authenticated request to the Mailman REST API, unless one of the request handlers aborts the processing due to some sort of auth issue

A typical example of one API operation mapping to its set of handlers looks something like this:

{‘UsersGet’: [validateJSONAuthToken, refreshJSONAuthToken, userMustbeListAdmin, proxyRequestToMailman]}

In the above example the inbound request goes through a handler that validates its JWT authentication token, a handler that refreshes its JWT token, a handler that checks to ensure that the requesting user (as identified from the JWT token) has the appropriate rights to carry out the requested Mailman 3 operation, then the request is proxied to the Mailman REST API and the result is send back to the client/requestor.

JSON web token authentication is automatically layered on top of the API.

Typically you would want the authenticating proxy to sit behind a reverse proxy such as HAproxy or Nginx which would do gzip compression and SSL.

Most of this is working.

as

On 22 Jan 2015, at 2:57 am, Barry Warsaw <barry@list.org> wrote:

On Jan 22, 2015, at 12:02 AM, Andrew Stuart wrote:

Barry - would you like the authenticating proxy to be part of Mailman core or a third party project?

For now, I think it should not be distributed as part of core. Maybe after the 3.0 release. The code could still be donated to the FSF and managed under the GNU Mailman project in a separate source repository. That's kind of your call; let me know if you'd like to do that.

It’s close to being functionally operational (which is to say working but in a raw state).

Very cool. Thanks for working on this important piece.

Cheers, -Barry

Mailman-Developers mailing list Mailman-Developers@python.org https://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/andrew.stuart%40s...

Security Policy: http://wiki.list.org/x/QIA9

Hi Stephen

Thanks for the guidance - I’ll have to admit I don’t know much about how to do such things right so the more guidance you are willing to give the better. Most of my code so far has been for me and no-one else so my next task is to learn how to write tests properly.

thanks

Andrew

On 12 Dec 2014, at 12:45 pm, Stephen J. Turnbull <stephen@xemacs.org> wrote:

Andrew Stuart writes:

I’m writing an authenticating proxy for the Mailman REST API and want to make sure everything works as expected against real infrastructure (Postgres/Postfix/Sqlalchemy/Falcon).

Determining that "things work as expected against real infrastructure" is not what the test suite does. AFAIK we don't have any such tests, and I don't really see how the tests we do have can be adapted to "tests against real infrastructure". The problem is specifying what to expect of real infrastructure.

You could, for example, write tests that insert users, query for them, and then delete them, but this has several problems. Do you want to do these tests on "live" servers that might actually be in use? -- Probably not, but how do you prevent that? Attributes of the various objects created by the system may depend on context. -- The test needs to ignore any differences due merely to different context. Coverage is necessarily imperfect in the sense that some internal state changes can't be checked since the "before" state is indeterminate in a live system. And so on.

Just to clarify, I didn’t get the tests going against live servers, but I did manage to get my proxy server inserted into the “normal” Mail man REST testing process.

It’s a first step for me in getting authentication stuff going.

Do ASCII diagrams work in email? Your email reader will need to use a fixed width fonr to see this:

              *****normal test configuration*****                         +-------------------+                                      | Mailman test code | +--+ Sends tests (HTTP requests)     +--------+----------+    |                                 |               |                                 Creates HTTP server |               |                                 then tears it down  |               |                                 |               |                                 +---------+-----------+   |                                 |                     |   |                                 | Mailman HTTP server | <-+                                 |                     |                                     +---------------------+                                     *****my test configuration*****                              +-------------------+                                    | Mailman test code | +--+ Sends tests (HTTP requests)   +--------+----------+    |                               |             +-+-----------------------+       Creates HTTP server |             |                         |       then tears it down  |             | My proxy server         |       |             |                         |       +---------+-----------+ +-+-----------------------+       |                     |   |                               | Mailman HTTP server | <-+                               |                     |                                   +---------------------+                                  

On 17 Dec 2014, at 12:08 am, Stephen J. Turnbull <stephen@xemacs.org> wrote:

Andrew Stuart writes:

Thanks for the guidance - I’ll have to admit I don’t know much about how to do such things right so the more guidance you are willing to give the better. Most of my code so far has been for me and no-one else so my next task is to learn how to write tests properly.

Well, since you got the existing tests to run against live servers, I'm pretty impressed. It certainly proves the concept"

It might be worthwhile to provide a guide on using them as the last step of setting up a server, though I'm not sure how easy that would be (the easiest thing is to say "edit the code", but that's not a great way to impress new users).

Did they leave any "residue" in the databases that you can detect? (That's the most important convenience of a one-shot "test server" -- you just tear it down and throw it away.) On a live server that could be bad. Eg, if a test creates a database user with a scripted password and leaves it behind, anybody that knows the script can access (some part of) the database!

Steve