[Python-Dev] Trimming the fat from "make quicktest" (was Re: I am now lost - committed, pulled, merged, what is "collapse"?)

[skip at pobox.com]

    Antoine> Does anyone use "make quicktest" for something useful?

I don't use it at all.

    Antoine> There is a reason the regression test suite has many tests...
    Antoine> "Blacklisting" some of them sounds like a bad thing to do.

+1.

Eliminating tests based on the time it takes to run them suggests that the
resulting smaller test run may have considerably different overall coverage
quality than you might desire.  Some tests (syntax, basic arithmetic, etc)
probably run blazingly fast and will be fully covered by a "make nanotest",
while some really important stuff (anything which forks or creates sockets)
will have very poor nanotest coverage because many of its tests cases won't
be run.  The odds that someone breaks syntax or basic arithmetic
functionality (or even changes those parts of the system) are pretty low, so
repeatedly running those tests simply because they run fast gives a false
sense of security.

Skip