[AstroPy] Testing Guidelines

[Erik Tollerud]

> b.t.w.  I suggest that you differentiate the _purpose_ of a test from
> the _mechanism_ of the test.

This is a very good point - when I think about it more, it makes sense
to have the same test-runner framework.  With either nose or py.test
it seems reasonable to divide the testing into tests which always
depend only on the core modules, and a separate suite that is more
general, potentially using e.g. pysofa to test coordinate conversions
and the like.  As Mark says, there isn't necessarily a need (at least
at this stage) to carefully divide tests into categories, as long as
testing coverage is good for the actual purpose of the module.

> What is "complete coverage" anyway - testing all public methods with all allowed input types (and possibly some that are not allowed to test the correct error handling)? This is probably not even realised for numpy right now, so I generally second that idea. In particular, it would probably needlessly delay acceptance of some largely finished packages that are otherwise ready for inclusion.
> I would probably put a somewhat stronger emphasis than "Unit tests are encouraged..." into the guidelines for any newly  developed or revised code, also to encourage test-driven development practices.

Absolutely agreed - the plan is to make it clear in the example
package that everything should be unit-tested if it can be, as that's
what new packages will presumably use.


Now the nose vs. py.test question... From Mark's summary, it sounds
like it's difficult to choose between them because they're too
similar!  Given Mark and Victoria's comments (and looking over the
docs in more detail myself), I'm beginning to think that py.test has
more potential.  Especially important is that fact that it appears
that py.test can directly use tests written for nose, while the
converse is not true... this would make integrate already-existing
projects with nose much easier and lower the re-learning penalty to
pretty much nothing.  Anyone else have experience with py.test that
may be of relevance?


>
> Cheers,
>                                                        Derek
>
>> On Fri, Jul 8, 2011 at 12:57 PM, Mark Sienkiewicz <sienkiew at stsci.edu> wrote:
>>>
>>>>> As for the testing framework, we were thinking nose
>>>>> (http://somethingaboutorange.com/mrl/projects/nose/1.0.0/).   Nose is
>>>>> very nice for a project like this because it's super easy to run and
>>>>> also easy to write tests with, but is also compatible with the stdlib
>>>>> unittests.
>>>
>>>
>>> "use nose" is a little ambiguous.  numpy "uses nose" but if you look
>>> inside numpy.test() it looks like of like they implemented their own
>>> testing system using nose as a library.  I've never succeeded in using
>>> nosetests to run the numpy tests, for example.
>>>
>>> If you choose nose, my preference is to say:  Tests come in a separate
>>> directory from the source.  You can run all the tests with the command
>>> "nosetests tests/".
>>>
>>> This is not to exclude the possibility of installing some of the tests
>>> or of providing a test() function, but I also want to run the tests
>>> directly with nosetests.
>>>
>>> I actually have an ulterior motive here:  I use a nose plugin that feeds
>>> a report generator.  If you only provide a foo.test() function, I can't
>>> necessarily get it to use the plugin even if you base the whole thing on
>>> nose.  You might not think that was a big deal, but it gets important
>>> when you have the test volume that I regularly deal with: > 100 000
>>> tests per night, sometimes > 10 000 fail, depending what the developers
>>> are up to.
>>>
>>> b.t.w.  This doesn't mean I have any attachment to nose; it just happens
>>> to be one of a small number of test runners that I use right now.
>>>
>>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>



-- 
Erik Tollerud