[Python-Dev] Status of Python buildbots

[Victor Stinner]

Hi,

I spent last week working on fixing buildbots:

   https://www.python.org/dev/buildbot/

It should now be able again to rely on them to detect regressions. Changes:

* Fix various bugs (I don't even recall which ones)
* Fix multiple random failures
* Fix dozen of warnings

I also enhanced the configuration of buildbots to search for warnings
in the output of the "tests" step (Python test suite). I use a regular
expression to search for patterns:

https://github.com/python/buildmaster-config/blob/master/master/master.cfg#L93-L125

If a warning is found, the buildbot now becomes orange. As I wrote, I
fixed dozens of warnings, but there are still a few ones (and many
unstable tests failing randomly).

The orange state notify for example when a test failed once, but
passed when it was run again in verbose mode. It helps to detect
unstable tests, but also real bugs.


The "Warning -- xxx was modified by test_xxx" warning now always log
the old and new value to ease debugging these warnings. Example on
Python 2.7:

Warning -- files was modified by test_sax
  Before: []
  After:  ['@test_352_tmp']


TODO:

* Create a mailing list and send an email to this list when the state
of a buildbot changes (only from green to orange or red? similar to
what we have on #python-dev IRC channel). I suggest to start with a
whitelist of buildbots known to be stable... hum, like the list of
"stable buildbots" ;-) Let's start with the 3.x branch, and later
enable it on more branches. The mailing list should help to coordinate
when multiple developers work on buildbot issues in parallel.

* Fix FreeBSD buildbots which fail to compile Python because of
http://bugs.python.org/issue23404 I proposed a change to not
regenerate generated files based on file modification time anymore,
but require an explicit action ("make regen-all"). This change solves
many practical issues.

* Fix remaining warnings.

* Fix unstable tests.

* Add more warnings! I proposed to emit ResourceWarning in
multiprocessing.Queue and concurrent.futures executors ;-)
http://bugs.python.org/issue30171 and
http://bugs.python.org/issue30244

* Add more buildbots! Zachary Ware proposed to add a buildbot running
"regen-all" to check that generated files are up to date.

* Repeat ;-)

Victor