Re: [Mailman-Developers] MM3 REST API testing [was: Python 3]
Am 26.12.2014 um 16:38 schrieb Aurelien Bompard:
And mailman.client must be ported too, since it does import stuff from mailman for testing purposes. I think Postorious is safe, but that must be verified (I believe it imports the testing framework from mailman.client and thus must be ported too).
I think, this is worth another thread.
So far we haven't found a perfect solution for testing packages that rely heavily on the MM3 REST API. As far as mailman.client, Postorius and HK are concerned, testing without *some* core integration doesn't make too much sense IMO, because any changes in the API would go unnoticed.
The first solution for that was to start the core (out of process) with a temporary configuration and on a different port before running the tests and tearing it down afterwards. That way mailman.client/Postorius would be tested with a real MM core underneath it. Compatibility with the API was guaranteed, but it was painfully slow and also a little unstable. It's not fun when every typo is punished with a 10 or 20s penalty!
That's why I was extremely happy when Aurélien came up with a much better idea a while ago. His solution made use of the webtest package, which made it possible to test the MM3 API wsgi app without the HTTP overhead or the need to start/stop the main MM process. Much faster, more stable, a real delight compared to the situation we had before.
Unfortunately, this now means direct "from mailman" imports which will break with incompatible Python versions. Too bad, we didn't think about this (I, for one, was aware of the fact that Barry plans to do a monolingual port at some point in the future. So I could have thought about that earlier. :-/ meh...).
Of course, in order to make our tests work with a py3 MM core, we could go back to the slow and no-fun way of testing that we had before. But maybe there's better way...
Any ideas?
Cheers Flo
On Dec 27, 2014, at 01:16 PM, Florian Fuchs wrote:
So far we haven't found a perfect solution for testing packages that rely heavily on the MM3 REST API. As far as mailman.client, Postorius and HK are concerned, testing without *some* core integration doesn't make too much sense IMO, because any changes in the API would go unnoticed.
I'm playing with a solution that involves the use of the 'vcr' package, which as its name implies (well, at least for us, um, mature programmers ;) is a facility to record and replay HTTP traffic. It's compatible with both Python 2 and 3 and is available on the Cheeseshop.
So the basic idea is to set up an MM3 installation (tox --notest -r
suffices), then start up its REST server with a fresh mailman.db, and run the
test suite against it. This will capture the traffic in a yaml file, which
we could ship with mailman.client.
Then to run the test suite, you'd use vcr in replay mode against the captured yaml file. Any time the API changes, you'd have to re-record the traffic against a fresh database.
I have a branch that's working in this direction and it's promising, but I've also had to rewrite the testing infrastructure for mailman.client, and make some other changes. It also points out what I think are some bugs in Mailman core, so I'm working those out on my py3 branch in parallel.
It's not yet ready for review, but if you want to keep an eye on things, look here: lp:~barry/mailman.client/bilingual
Feel free to provide feedback on the approach, or the changes so far, but I will submit a merge proposal when/if it's ready to go.
Cheers, -Barry
Am 02.01.2015 um 05:40 schrieb Barry Warsaw:
On Dec 27, 2014, at 01:16 PM, Florian Fuchs wrote:
So far we haven't found a perfect solution for testing packages that rely heavily on the MM3 REST API. As far as mailman.client, Postorius and HK are concerned, testing without *some* core integration doesn't make too much sense IMO, because any changes in the API would go unnoticed.
I'm playing with a solution that involves the use of the 'vcr' package, which as its name implies (well, at least for us, um, mature programmers ;) is a facility to record and replay HTTP traffic. It's compatible with both Python 2 and 3 and is available on the Cheeseshop.
I took a quick look at vcr and it really looks promising. Although apparently it doesn't support VHS, which is very sad. But we can still use it for testing I guess... :-)
So the basic idea is to set up an MM3 installation (
tox --notest -rsuffices), then start up its REST server with a fresh mailman.db, and run the test suite against it. This will capture the traffic in a yaml file, which we could ship with mailman.client.Then to run the test suite, you'd use vcr in replay mode against the captured yaml file. Any time the API changes, you'd have to re-record the traffic against a fresh database.
That sounds fantastic! Do you think it might make sense to ship the traffic dump with mailman instead of mailman.client (and work the recording into the test suite somehow)? That way there would only be a single file to observe for changes in the REST API. Sure, a change in the yaml file doesn't necessarily mean a change in the API (might as well be just an additional test case or different test data). But still, if the yaml file stays the same between commits, that would be a nice guarantee that the API stayed stable.
I have a branch that's working in this direction and it's promising, but I've also had to rewrite the testing infrastructure for mailman.client, and make some other changes. It also points out what I think are some bugs in Mailman core, so I'm working those out on my py3 branch in parallel.
It's not yet ready for review, but if you want to keep an eye on things, look here: lp:~barry/mailman.client/bilingual
Feel free to provide feedback on the approach, or the changes so far, but I will submit a merge proposal when/if it's ready to go.
Thanks for starting the port, Barry. Looking forward to seeing that merge proposal (although I don't expect I'll have a too much to add... ;-).
Florian
On Jan 03, 2015, at 10:16 PM, Florian Fuchs wrote:
I took a quick look at vcr and it really looks promising. Although apparently it doesn't support VHS, which is very sad. But we can still use it for testing I guess... :-)
:)
That sounds fantastic! Do you think it might make sense to ship the traffic dump with mailman instead of mailman.client (and work the recording into the test suite somehow)? That way there would only be a single file to observe for changes in the REST API. Sure, a change in the yaml file doesn't necessarily mean a change in the API (might as well be just an additional test case or different test data). But still, if the yaml file stays the same between commits, that would be a nice guarantee that the API stayed stable.
It's possible. One reason I'm shipping it with mailman.client in my merge proposal is because once it's generated, there's no second guessing in mmclient about where the yaml file lives. If we shipped it with core, you'd have to have some way to tell mmclient where to find the yaml file, since different developers may be working in completely different file system locations of course.
But do let me know what you think about the approach in my mp. I'm definitely not opposed to moving it if it makes sense.
Of course we also talked about making mmclient an actual subpackage of core and placing it in core's tree. That wasn't very convenient at the time, but maybe that's worth a revisit at some point.
Cheers, -Barry
FWIW some notes on mitmproxy and ngrep - helpful tools when intercepting and recording HTTP traffic.
mitmproxy/mitmdump: I know doesn’t serve the same purpose as VCR but its helpful to be able to capture and see what is going on between server and client when Mailman tests are running.
https://mitmproxy.org/doc/mitmdump.html
mitmproxy itself seems to have Python 3 as its next major initiative.
I configured Mailman tests to run through a proxy (I sent an email to this list a few weeks back with instructions for how to do this).
Then:
mitmdump -p 7001 -P http://127.0.0.1:9001 -w outfile
Captures traffic to outfile. It can be loaded back into mitmproxy and modified and replayed.
**ngrep ngrep is useful when digging deeper. it will show you traffic in real time on a given port.
for example sudo ngrep -W byline -d lo port 9001 will show what is happening on the Mailman REST API port when configured for port 9001
Dunno if this is useful for anyone but everything helps when digging into HTTP.
as
On 2 Jan 2015, at 3:40 pm, Barry Warsaw <barry@python.org> wrote:
On Dec 27, 2014, at 01:16 PM, Florian Fuchs wrote:
So far we haven't found a perfect solution for testing packages that rely heavily on the MM3 REST API. As far as mailman.client, Postorius and HK are concerned, testing without *some* core integration doesn't make too much sense IMO, because any changes in the API would go unnoticed.
I'm playing with a solution that involves the use of the 'vcr' package, which as its name implies (well, at least for us, um, mature programmers ;) is a facility to record and replay HTTP traffic. It's compatible with both Python 2 and 3 and is available on the Cheeseshop.
So the basic idea is to set up an MM3 installation (tox --notest -r
suffices), then start up its REST server with a fresh mailman.db, and run the
test suite against it. This will capture the traffic in a yaml file, which
we could ship with mailman.client.
Then to run the test suite, you'd use vcr in replay mode against the captured yaml file. Any time the API changes, you'd have to re-record the traffic against a fresh database.
I have a branch that's working in this direction and it's promising, but I've also had to rewrite the testing infrastructure for mailman.client, and make some other changes. It also points out what I think are some bugs in Mailman core, so I'm working those out on my py3 branch in parallel.
It's not yet ready for review, but if you want to keep an eye on things, look here: lp:~barry/mailman.client/bilingual
Feel free to provide feedback on the approach, or the changes so far, but I will submit a merge proposal when/if it's ready to go.
Cheers, -Barry
Mailman-Developers mailing list Mailman-Developers@python.org https://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/andrew.stuart%40s...
Security Policy: http://wiki.list.org/x/QIA9
On Jan 04, 2015, at 09:44 AM, Andrew Stuart wrote:
mitmproxy/mitmdump: I know doesn’t serve the same purpose as VCR but its helpful to be able to capture and see what is going on between server and client when Mailman tests are running.
mitm* look pretty cool. mailman.client needs to be bilingual (Python 2 and 3) so these might end up being interesting once they port mitm* to Python 3 (unless it doesn't matter because they're never imported, I guess).
Cheers, -Barry
My understanding is that VCR will capture the HTTP test request/responses and comparison to some baseline will indicate that the API is behaving as expected.
Forgive me if I’m misunderstanding something fundamental about the VCR based testing approach. Some questions to helop me understand:
Are the HTTP responses going to be the same between runs and between systems?
How deep are the comparisons of the HTTP responses? Are you thinking of doing byte level comparisions of the entire response data, including headers etc?
Often the contents of the HTTP headers vary for a whole bunch of reasons.
Also if there is a proxy in the middle that can easily rewrite the whole request and response. I’m guessing a byte level comparison of the responses would fail.
If it was the content of the responses being compared then that’s likely to work but that’s not really what VCR is about, right?
Am I understanding the VCR approach correctly?
as
On 2 Jan 2015, at 3:40 pm, Barry Warsaw <barry@python.org> wrote:
On Dec 27, 2014, at 01:16 PM, Florian Fuchs wrote:
So far we haven't found a perfect solution for testing packages that rely heavily on the MM3 REST API. As far as mailman.client, Postorius and HK are concerned, testing without *some* core integration doesn't make too much sense IMO, because any changes in the API would go unnoticed.
I'm playing with a solution that involves the use of the 'vcr' package, which as its name implies (well, at least for us, um, mature programmers ;) is a facility to record and replay HTTP traffic. It's compatible with both Python 2 and 3 and is available on the Cheeseshop.
So the basic idea is to set up an MM3 installation (tox --notest -r
suffices), then start up its REST server with a fresh mailman.db, and run the
test suite against it. This will capture the traffic in a yaml file, which
we could ship with mailman.client.
Then to run the test suite, you'd use vcr in replay mode against the captured yaml file. Any time the API changes, you'd have to re-record the traffic against a fresh database.
I have a branch that's working in this direction and it's promising, but I've also had to rewrite the testing infrastructure for mailman.client, and make some other changes. It also points out what I think are some bugs in Mailman core, so I'm working those out on my py3 branch in parallel.
It's not yet ready for review, but if you want to keep an eye on things, look here: lp:~barry/mailman.client/bilingual
Feel free to provide feedback on the approach, or the changes so far, but I will submit a merge proposal when/if it's ready to go.
Cheers, -Barry
Mailman-Developers mailing list Mailman-Developers@python.org https://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/andrew.stuart%40s...
Security Policy: http://wiki.list.org/x/QIA9
On Jan 04, 2015, at 10:06 AM, Andrew Stuart wrote:
Are the HTTP responses going to be the same between runs and between systems?
Yes, they should be. If you think about it, GETs are supposed to be idempotent, so those won't change (even the etags) between requests. As it turns out, if replayed in the same order they were recorded (which I think is guaranteed by a full tox run via nose2, as my mailman.client branch adds), then even non-idempotent requests will have the same responses.
How deep are the comparisons of the HTTP responses? Are you thinking of doing byte level comparisions of the entire response data, including headers etc?
I know that vcr has some support for tweaking the match between requests and responses, but I didn't find those to be necessary so I didn't dig into that very deeply.
Cheers, -Barry
I feel like prefacing all my questions with an apology if it’s a dumb question. Anyway...
What’s the advantage of using VCR over a set of tests that use Kenneth Reitz requests library to send HTTP queries to the API and have a look at the response to see if it contains the expected HTTP codes/data?
thanks
On 2 Jan 2015, at 3:40 pm, Barry Warsaw <barry@python.org> wrote:
On Dec 27, 2014, at 01:16 PM, Florian Fuchs wrote:
So far we haven't found a perfect solution for testing packages that rely heavily on the MM3 REST API. As far as mailman.client, Postorius and HK are concerned, testing without *some* core integration doesn't make too much sense IMO, because any changes in the API would go unnoticed.
I'm playing with a solution that involves the use of the 'vcr' package, which as its name implies (well, at least for us, um, mature programmers ;) is a facility to record and replay HTTP traffic. It's compatible with both Python 2 and 3 and is available on the Cheeseshop.
So the basic idea is to set up an MM3 installation (tox --notest -r
suffices), then start up its REST server with a fresh mailman.db, and run the
test suite against it. This will capture the traffic in a yaml file, which
we could ship with mailman.client.
Then to run the test suite, you'd use vcr in replay mode against the captured yaml file. Any time the API changes, you'd have to re-record the traffic against a fresh database.
I have a branch that's working in this direction and it's promising, but I've also had to rewrite the testing infrastructure for mailman.client, and make some other changes. It also points out what I think are some bugs in Mailman core, so I'm working those out on my py3 branch in parallel.
It's not yet ready for review, but if you want to keep an eye on things, look here: lp:~barry/mailman.client/bilingual
Feel free to provide feedback on the approach, or the changes so far, but I will submit a merge proposal when/if it's ready to go.
Cheers, -Barry
Mailman-Developers mailing list Mailman-Developers@python.org https://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/andrew.stuart%40s...
Security Policy: http://wiki.list.org/x/QIA9
On Jan 04, 2015, at 10:21 AM, Andrew Stuart wrote:
What’s the advantage of using VCR over a set of tests that use Kenneth Reitz requests library to send HTTP queries to the API and have a look at the response to see if it contains the expected HTTP codes/data?
The latter approach requires a running MM3 core REST server. The vcr approach only requires that when the core's REST API changes, i.e. in order to generate a new tape.yaml file. The nice thing about vcr is that it integrates with all kinds of HTTP libraries, including httplib2 which is what mailman.client uses, so the test suite just adds a simple context manager to go through vcr when in replay mode.
Cheers, -Barry
On Jan 01, 2015, at 11:40 PM, Barry Warsaw wrote:
I'm playing with a solution that involves the use of the 'vcr' package
So this branch is now ready for review.
https://code.launchpad.net/~barry/mailman.client/bilingual/+merge/245537
I even added documentation on how to run the test to record the yaml file containing the HTTP traffic. There are tox rules for doing both the recording, and for running the test suite with the recorded traffic against both Python 2 and Python 3. I've only tested 2.7 and 3.4 since I don't currently have good builds of 2.6, 3.2, or 3.3, but the tox.ini file should support those.
There are now no imports of any mailman modules. I did have to add a new
top-level queues resource to the REST API which will allow you to inject
messages, query, and delete messages from the various queues. I thought this
was a pretty handy feature anyway, and it was easy to add, so that's now in
the Mailman 3 trunk. Which btw, as you might notice, I did go ahead and merge
the py3 branch. I think with this mailman.client approach and a REST-based HK
IArchiver implementation (TBD), it was safe enough to do, and I didn't want to
have to continue to implement new features in both branches (e.g. the queues
resource).
Please have a look and let me know what you think.
Cheers, -Barry
Am 06.01.2015 um 04:24 schrieb Barry Warsaw:
On Jan 01, 2015, at 11:40 PM, Barry Warsaw wrote:
I'm playing with a solution that involves the use of the 'vcr' package
So this branch is now ready for review.
https://code.launchpad.net/~barry/mailman.client/bilingual/+merge/245537
I even added documentation on how to run the test to record the yaml file containing the HTTP traffic. There are tox rules for doing both the recording, and for running the test suite with the recorded traffic against both Python 2 and Python 3. I've only tested 2.7 and 3.4 since I don't currently have good builds of 2.6, 3.2, or 3.3, but the tox.ini file should support those.
There are now no imports of any
mailmanmodules. I did have to add a new top-levelqueuesresource to the REST API which will allow you to inject messages, query, and delete messages from the various queues. I thought this was a pretty handy feature anyway, and it was easy to add, so that's now in the Mailman 3 trunk. Which btw, as you might notice, I did go ahead and merge the py3 branch. I think with this mailman.client approach and a REST-based HK IArchiver implementation (TBD), it was safe enough to do, and I didn't want to have to continue to implement new features in both branches (e.g. the queues resource).Please have a look and let me know what you think.
I haven't had the time to look at it today. But will definitely do it tomorrow. Thanks a lot!
Florian
participants (4)
-
Andrew Stuart -
Barry Warsaw -
Barry Warsaw -
Florian Fuchs