[AstroPy] Tests for AstroPy (was POLL: vision for a common Astronomy package)

[Victoria G. Laidler]

Ole Streicher wrote:
> BTW, are there ideas for regression tests of astropy? Especially in the
> case you mentioned, this would be *very* helpful to check that
> everything works in the target system.
I was thinking about this as well. I suggest that there be minimum 
testing standards for astropy packages. Packages should come with tests 
that:
- can be run using a standard syntax (what?)
- verify that all significant elements of the package can successfully 
execute
- verify that all significant elements of the package, for some minimum 
set of common use cases, give the right answer when run
- come with documentation that in some way describe test coverage (how 
much of the system do the tests cover) and the source of the test 
answers (ie, how did you determine the "right answer" used in the tests)

This last point is especially important if we want to get people to 
adopt astropy libraries in place of their current favorite tool. In this 
case, testing serves as a confidence builder: astropy package X is at 
least as good as OtherPackage Y.

I'm using two phrases in the above that are meant to have a basic 
definition but also allow some wiggle room for package developers:
- all significant elements of the package: the basic parts that most 
people who use the package at all will use, similar to the dependency 
discussion)
- minimum set of common use cases: the typical use cases that most 
people will encounter

This allows a package to be included in astropy without requiring the 
developer to provide an exhaustive set of tests that verify correct 
behavior in every possible corner of the domain space. This is 
especially important for software whose behavior is data-dependent. Of 
course, more tests are better, and would produce better coverage.

Vicki Laidler