[py-dev] Acceptance testing with py.test

[Ronny Pfannschmidt]

On Fri, 2010-02-12 at 22:23 +0100, holger krekel wrote:
> Hi Pontus,
> 
> On Fri, Feb 12, 2010 at 14:32 +0100, Pontus Åström wrote:
> > I'm considering a plugin for doing acceptance testing with py.test. The 
> > core idea is to mimic the behaviour of Concordion 
> > (http://concordion.org) with respect to visual effects and reports, but 
> > use a python engine. The python libs I am considering are py.test for 
> > running tests and genshi for xml instrumentation.
> 
> I think that's a good plan :) 
> 
> > Then, to get a polished plugin some code need to be written; a py.test 
> > plugin for test collection and reporting, and a utility library for 
> > easing genshi xml insturmentation.
> > 
> > Well, the reason for this explanation is to see if anybody out there has 
> > some input regarding the idea, or maybe know of some useful libs or 
> > approaches to achieve the aim, which btw. is to obtain efficient, 
> > readable, executable acceptance tests connected to requirements. Read 
> > more about this aim on the Concordion website.
> 
I have some basic ideas about structuring that kind of test.

A) steped reporting, so each test reports the current step
B) collection of dependend test-items, having each test item as `step`

A requires extending the py.test reporting (but might be easy)
B requires extending the py.test test execution

Ali currently does acceptance testing for webapps using approach B.
The test collector is at
http://bitbucket.org/aafshar/glashammer-testing/src/tip/glashammer/utils/testing.py
however it currently lacks dependend tests support.

We would also like to build acceptance testing tools for pygtk,
but those seem more fit for approach A.
We would need those for more complex applications,
like http://pida.co.uk

Another interesting acceptance test approach is the way mercurial is
tested.
Its using sectioned shell scripts that can be matched with sectioned
expected output files.

I will do some more Work in those Areas after 17.03

-- Ronny

> Myself, i haven't too much acceptance testing with non-python domains. 
> I hear that Ruby is strong with domain-specific testing but haven't
> checked on things myself yet.  As far as giving you some ideas 
> on how the py.test plugin side might look like i hope that Ronny
> Pfannschmidt or Ali Afshar can provide a pointer to their code
> for driving tests from yaml definitions as a starting point. 
> 
> cheers,
> holger