[AstroPy] Testing Guidelines

[Mark Sienkiewicz]

Erik Tollerud wrote:
> On Mon, Aug 1, 2011 at 11:03 AM, Mark Sienkiewicz <sienkiew at stsci.edu> wrote:
>   
>> If the decision goes that way, I can write up a document describing a common
>> subset that covers most of what you need to do in routine testing.  Then you
>> make your standard "use the astrolib-defined nose/py.test subset whenever it
>> is practical".  Only a few unusual cases will need to deviate.
>>     
>
> That seems like a great idea to me.

I'll put something together when I get some time.


>   Unless there are objections, I'm
> inclined to suggest we adopt py.test as the suggested runner based on
> it having the widest compatibility... Or are there still misgivings
> about this? (I'm speculating not given that there's been nothing on
> this topic in the last couple weeks.)
>   

Unless somebody specifically requests a poll, I think we have a consensus.


>>> Except there's one problem: the configuration files are formatted
>>> differently...  We want to use configuration settings for just one of
>>> the packages so that the suite always runs the same.
>>>       
>> What configuration file is that?  I don't recall ever writing a
>> configuration file for either nose or py.test.
>>     
>
> In setup.cfg you can add a [nosetest] or a [pytest] section to
>   

I see - you are talking about a setuptools config file, not a nose 
config or py.test config.  I'll have to read up on what that is about.


> Assuming we adopt py.test, though, the other useful change would be to
> generate the standalone py.test script and hook it into "python
> setup.py test" so that no one has to download py.test to run the test
> suite with py.test.  That's the only thing I can see that would
> definitely require a source code change.
>   

Yes.  Put that in the coding standard.  We should have a script (with a 
standardized name) to update all the generated source code.  It should 
re-gen C interface defintions, and make the py.test standalone script.  
Presumably there may be other things, e.g. if the author uses a parser 
generator or something like that.


>> IMHO, the most interesting feature to me are the new comparison functions.
>>     
>
> Just so we're on the same page, by this do you mean all the new
> assert* methods of the TestCase class?
>   

Yes.

> It's not clear to me that it's possible to use these comparison tool
> outside unittest2 - they seem to all be methods of TestCase and thus
> have to be done inside a unittest TestCase object (which is the whole
> point of using nose or py.test to avoid).  As you say, that's really
> too bad, because they have some nice features.
>   

All you need is to make a TestCase object and call the comparison 
methods.  Here is a proof of concept:

import unittest2

class dummy_ut2( unittest2.TestCase ) :  # not named test_something
    __test__ = False  # just in case
    def runTest( wtf ) :
        raise Exception('Should never be called')
xcmp = dummy_ut2()

def test_a() :
    xcmp.assertAlmostEqual( 1.001, 1.002, places=4 )

In principle, you can bury this kind of thing in a library, then use the 
comparison functions even when you don't want to make classes based on 
unittest2.

The major drawback I see here is that py.test reports a complete listing 
of the function that raises the exception, NOT the function that it 
called to run the test.   So, you get a listing of assertAlmostEqual 
instead of a listing of test_a.  I have in mind various ideas for 
workarounds.

So why would you do this?

- get the nice comparison functions of unittest2 without re-implementing 
them
- make them available to any test system

Of course, nothing here stops anybody from writing
    assert a == b
when that is the convenient thing to do.


> It looks to me like most of this sort of useful contextual information
> is part of in py.test's output, though  (see
> http://pytest.org/latest/example/reportingdemo.html#tbreportdemo).
> The multiline string diff as well as a sequence diff reporting seems
> to be builtin to py.test if you just do "assert a==b" ... I didn't
> realize that until now - pretty neat!
>   

That is nice.  I had not seen that before.