[Python-Dev] devguide: Issue #18871: make it more explicit that the test suite should be run before

Antoine Pitrou solipsis at pitrou.net
Thu Aug 29 08:40:29 CEST 2013

On Wed, 28 Aug 2013 20:05:25 -0400
Terry Reedy <tjreedy at udel.edu> wrote:
> On 8/28/2013 5:51 PM, antoine.pitrou wrote:
> > +Does the test suite still pass?
> There are several assumptions packed in that question.
> 0. The test suite is relevant to the patch.
> Not true for doc patches, some idlelib patches, and probably some others.

That was implicit in the formulation: "You must :ref:`run the whole
test suite <runtests>` to ensure that it passes before pushing any
*code changes*" (not doc changes, spelling corrections, etc.).

The only reason to know that the test suite isn't relevant to a code
change is to... *run the test suite*. Don't assume that you are
omniscient and know that your change won't break something. Developers
not being omniscient is why we have a test suite in the first place.

(btw, if idlelib isn't tested, it should!)

> 1. The test suite runs to completion.
> For at least a couple of years, that was not true on Windows desktops. 
> The popup Runtime Error boxes from now are now gone, but a month ago, a 
> crashing test froze Command Prompt.

But was the test corrected? Did you open an issue at the time?
Did you try to debug the error?

We are collectively responsible for Python's quality. This is not
some third-party software you have no control on.

> 2. The test suite runs without error.
> I have hardly ever seen this on Windows, ever (with sporadic runs over 
> several years). Today, test_email and test_sax failed in successive 
> runs*.
> 3. If there is a new failure, it is due to the patch.
> There have been at least one, but I think more, intermittent failures of 
> Windows tests in the last few months

Same answer as above.

> 4. The gain of answering that question is worth the cost.

Accepting responsibility for one's own changes is part of why we trust
each others as committers. The cost of *you* running the test suite is
smaller than the cost of other developers trying to investigate a
sudden buildbot failure, where it comes from etc.

Having you run the test suite encourages you to be conscious of its
existence, of its imperfections, and to feel responsible in making it
better. Shrugging it off because it sometimes doesn't work isn't
helpful. We are *striving* to make it better (both in coverage and in

> I worry that further emphasizing an overly broad, time-consuming, and 
> sometimes impractical rule will only discourage more participation.

IMO, this is a calculated risk that is worthwhile to take. Times have
changed, and being rigorous with testing is central in most successful
projects nowadays.



More information about the Python-Dev mailing list