[AstroPy] Testing Guidelines

[Thomas Robitaille]

Let's keep the discussion going on testing frameworks. In the past,
I've mainly used the standard unittest framework built-in to Python to
write tests. I was wondering if those of you familiar with nose or
py.test could explain what these packages add to the standard
framework? What is it that *we* need on top of the most basic testing
functionality unittest provides? Since py.test and nose are so
similar, what are the *differences*? I think if we know this, it will
be easier for those of us not too familiar with either framework to
weigh in on the issue.

By the way, it looks like there's been a lot of work on unittest in
python 2.7 (aka unittest2) so how does that compare to nose and
py.test?

Cheers,
Tom

On 13 July 2011 18:54, Erik Tollerud <erik.tollerud at gmail.com> wrote:
>> b.t.w.  I suggest that you differentiate the _purpose_ of a test from
>> the _mechanism_ of the test.
>
> This is a very good point - when I think about it more, it makes sense
> to have the same test-runner framework.  With either nose or py.test
> it seems reasonable to divide the testing into tests which always
> depend only on the core modules, and a separate suite that is more
> general, potentially using e.g. pysofa to test coordinate conversions
> and the like.  As Mark says, there isn't necessarily a need (at least
> at this stage) to carefully divide tests into categories, as long as
> testing coverage is good for the actual purpose of the module.
>
>> What is "complete coverage" anyway - testing all public methods with all allowed input types (and possibly some that are not allowed to test the correct error handling)? This is probably not even realised for numpy right now, so I generally second that idea. In particular, it would probably needlessly delay acceptance of some largely finished packages that are otherwise ready for inclusion.
>> I would probably put a somewhat stronger emphasis than "Unit tests are encouraged..." into the guidelines for any newly  developed or revised code, also to encourage test-driven development practices.
>
> Absolutely agreed - the plan is to make it clear in the example
> package that everything should be unit-tested if it can be, as that's
> what new packages will presumably use.
>
>
> Now the nose vs. py.test question... From Mark's summary, it sounds
> like it's difficult to choose between them because they're too
> similar!  Given Mark and Victoria's comments (and looking over the
> docs in more detail myself), I'm beginning to think that py.test has
> more potential.  Especially important is that fact that it appears
> that py.test can directly use tests written for nose, while the
> converse is not true... this would make integrate already-existing
> projects with nose much easier and lower the re-learning penalty to
> pretty much nothing.  Anyone else have experience with py.test that
> may be of relevance?
>
>
>>
>> Cheers,
>>                                                        Derek
>>
>>> On Fri, Jul 8, 2011 at 12:57 PM, Mark Sienkiewicz <sienkiew at stsci.edu> wrote:
>>>>
>>>>>> As for the testing framework, we were thinking nose
>>>>>> (http://somethingaboutorange.com/mrl/projects/nose/1.0.0/).   Nose is
>>>>>> very nice for a project like this because it's super easy to run and
>>>>>> also easy to write tests with, but is also compatible with the stdlib
>>>>>> unittests.
>>>>
>>>>
>>>> "use nose" is a little ambiguous.  numpy "uses nose" but if you look
>>>> inside numpy.test() it looks like of like they implemented their own
>>>> testing system using nose as a library.  I've never succeeded in using
>>>> nosetests to run the numpy tests, for example.
>>>>
>>>> If you choose nose, my preference is to say:  Tests come in a separate
>>>> directory from the source.  You can run all the tests with the command
>>>> "nosetests tests/".
>>>>
>>>> This is not to exclude the possibility of installing some of the tests
>>>> or of providing a test() function, but I also want to run the tests
>>>> directly with nosetests.
>>>>
>>>> I actually have an ulterior motive here:  I use a nose plugin that feeds
>>>> a report generator.  If you only provide a foo.test() function, I can't
>>>> necessarily get it to use the plugin even if you base the whole thing on
>>>> nose.  You might not think that was a big deal, but it gets important
>>>> when you have the test volume that I regularly deal with: > 100 000
>>>> tests per night, sometimes > 10 000 fail, depending what the developers
>>>> are up to.
>>>>
>>>> b.t.w.  This doesn't mean I have any attachment to nose; it just happens
>>>> to be one of a small number of test runners that I use right now.
>>>>
>>>
>> _______________________________________________
>> AstroPy mailing list
>> AstroPy at scipy.org
>> http://mail.scipy.org/mailman/listinfo/astropy
>>
>
>
>
> --
> Erik Tollerud
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>