[AstroPy] Testing Guidelines

Erik Tollerud erik.tollerud at gmail.com
Mon Aug 15 03:25:50 EDT 2011

On Mon, Aug 1, 2011 at 11:03 AM, Mark Sienkiewicz <sienkiew at stsci.edu> wrote:
> Erik Tollerud wrote:
>> In principle, the similarities between nose and py.test mean we
>> wouldn't really have to make a decision until we need features that
>> differ between them- developers could run whichever they prefer...
> If the decision goes that way, I can write up a document describing a common
> subset that covers most of what you need to do in routine testing.  Then you
> make your standard "use the astrolib-defined nose/py.test subset whenever it
> is practical".  Only a few unusual cases will need to deviate.

That seems like a great idea to me.  Unless there are objections, I'm
inclined to suggest we adopt py.test as the suggested runner based on
it having the widest compatibility... Or are there still misgivings
about this? (I'm speculating not given that there's been nothing on
this topic in the last couple weeks.)

>> Except there's one problem: the configuration files are formatted
>> differently...  We want to use configuration settings for just one of
>> the packages so that the suite always runs the same.
> What configuration file is that?  I don't recall ever writing a
> configuration file for either nose or py.test.

In setup.cfg you can add a [nosetest] or a [pytest] section to
customize various things like which directories to look into (if not
default), as well as a variety of options setting how the output
appears.  While not necessarily important if you know what you're
doing, this makes it much easier to have random people running the
test suite, because you can be sure the output they are seeing looks
just like the output you expect to be seeing.  I suppose this isn't
necessary if none of the defaults will be changed, but it still makes
sense to specify which we want people to run (and developers to be
familiar with).

Assuming we adopt py.test, though, the other useful change would be to
generate the standalone py.test script and hook it into "python
setup.py test" so that no one has to download py.test to run the test
suite with py.test.  That's the only thing I can see that would
definitely require a source code change.

>> The biggest change I see is that it seems to now include a test
>> discovery tool, but it is much less documented (a few paragraphs
>> rather than many pages each for the other two), and seems to lack most
>> of the nose and py.test runners' customization (like configuring in
>> the setup.cfg file, easy plugins, etc.). It's an improvement, but it's
>> still based around the more complex syntax of unittest, so most of
>> what I said above as far as I can tell is still true.
> IMHO, the most interesting feature to me are the new comparison functions.

Just so we're on the same page, by this do you mean all the new
assert* methods of the TestCase class?

> I never really liked all the weird "fail unless not less than or equal" kind
> of functions, but the nice thing in unittest2 is that some of the comparison
> functions are able to display useful reports about what the nature of the
> failure.  For example, I seem to remember a string compare that will split
> multi-line strings and display a diff-like result.
> It strikes me as poor modularization, though:  there should be a separate
> library of comparison functions that can be used with any test framework,
> not a slightly different implementation in every test framework.
> I think you can write a test function for nose or py.test and still use a
> unittest2 comparison function; you just have to be a little tricky.  It's
> the kind of thing you would hide in a library.  Maybe we provide our own
> library of test helper functions.

It's not clear to me that it's possible to use these comparison tool
outside unittest2 - they seem to all be methods of TestCase and thus
have to be done inside a unittest TestCase object (which is the whole
point of using nose or py.test to avoid).  As you say, that's really
too bad, because they have some nice features.

It looks to me like most of this sort of useful contextual information
is part of in py.test's output, though  (see
The multiline string diff as well as a sequence diff reporting seems
to be builtin to py.test if you just do "assert a==b" ... I didn't
realize that until now - pretty neat!

> The biggest negative of unittest2 is that it is still unittest.  When I
> first read about unittest, I thought "this is crazy complicated for running
> a simple test; I'm not going to use it".  I have a better understanding of
> it now, but I  still only ever use it when I have no other choice, and that
> basically means when I am working on code that somebody else already wrote
> using unittest.

I completely agree on this point - that's why I certainly don't want
to say "only unittest tests will be accepted" or something along those

Erik Tollerud

More information about the AstroPy mailing list