Status of Python buildbots
Hi, I spent last week working on fixing buildbots: https://www.python.org/dev/buildbot/ It should now be able again to rely on them to detect regressions. Changes: * Fix various bugs (I don't even recall which ones) * Fix multiple random failures * Fix dozen of warnings I also enhanced the configuration of buildbots to search for warnings in the output of the "tests" step (Python test suite). I use a regular expression to search for patterns: https://github.com/python/buildmaster-config/blob/master/master/master.cfg#L... If a warning is found, the buildbot now becomes orange. As I wrote, I fixed dozens of warnings, but there are still a few ones (and many unstable tests failing randomly). The orange state notify for example when a test failed once, but passed when it was run again in verbose mode. It helps to detect unstable tests, but also real bugs. The "Warning -- xxx was modified by test_xxx" warning now always log the old and new value to ease debugging these warnings. Example on Python 2.7: Warning -- files was modified by test_sax Before: [] After: ['@test_352_tmp'] TODO: * Create a mailing list and send an email to this list when the state of a buildbot changes (only from green to orange or red? similar to what we have on #python-dev IRC channel). I suggest to start with a whitelist of buildbots known to be stable... hum, like the list of "stable buildbots" ;-) Let's start with the 3.x branch, and later enable it on more branches. The mailing list should help to coordinate when multiple developers work on buildbot issues in parallel. * Fix FreeBSD buildbots which fail to compile Python because of http://bugs.python.org/issue23404 I proposed a change to not regenerate generated files based on file modification time anymore, but require an explicit action ("make regen-all"). This change solves many practical issues. * Fix remaining warnings. * Fix unstable tests. * Add more warnings! I proposed to emit ResourceWarning in multiprocessing.Queue and concurrent.futures executors ;-) http://bugs.python.org/issue30171 and http://bugs.python.org/issue30244 * Add more buildbots! Zachary Ware proposed to add a buildbot running "regen-all" to check that generated files are up to date. * Repeat ;-) Victor
On Wed, May 3, 2017 at 10:22 AM, Victor Stinner
Hi,
I spent last week working on fixing buildbots:
Thanks!
* Add more buildbots! Zachary Ware proposed to add a buildbot running "regen-all" to check that generated files are up to date.
* Repeat ;-)
I have been saying for several months now that I would like to set up a Cygwin buildbot--an important step in making that platform supportable again. I now have the infrastructure available to do so (Windows VM on an OpenStack infrastructure at my university). I wanted to wait until the tests on Cygwin were more stable, but since you allow unstable buildbots I could include it among them for now. Is the buildbot setup documentation on the wiki page still accurate? Thanks, Erik
On Wed, May 3, 2017 at 4:16 AM, Erik Bray
I have been saying for several months now that I would like to set up a Cygwin buildbot--an important step in making that platform supportable again. I now have the infrastructure available to do so (Windows VM on an OpenStack infrastructure at my university). I wanted to wait until the tests on Cygwin were more stable, but since you allow unstable buildbots I could include it among them for now.
That would be fine :)
Is the buildbot setup documentation on the wiki page still accurate?
Probably not; the devguide should be more up-to-date: https://docs.python.org/devguide/buildslave.html I don't know what unique challenges you may run into setting up a Cygwin worker, but please submit PRs against that page if you find anything to be lacking. -- Zach
On Wed, 3 May 2017 at 01:24 Victor Stinner
Hi,
I spent last week working on fixing buildbots:
https://www.python.org/dev/buildbot/
It should now be able again to rely on them to detect regressions.
Thanks, Victor!
Changes:
* Fix various bugs (I don't even recall which ones) * Fix multiple random failures * Fix dozen of warnings
I also enhanced the configuration of buildbots to search for warnings in the output of the "tests" step (Python test suite). I use a regular expression to search for patterns:
https://github.com/python/buildmaster-config/blob/master/master/master.cfg#L...
If a warning is found, the buildbot now becomes orange. As I wrote, I fixed dozens of warnings, but there are still a few ones (and many unstable tests failing randomly).
The orange state notify for example when a test failed once, but passed when it was run again in verbose mode. It helps to detect unstable tests, but also real bugs.
The "Warning -- xxx was modified by test_xxx" warning now always log the old and new value to ease debugging these warnings. Example on Python 2.7:
Warning -- files was modified by test_sax Before: [] After: ['@test_352_tmp']
TODO:
* Create a mailing list and send an email to this list when the state of a buildbot changes (only from green to orange or red? similar to what we have on #python-dev IRC channel). I suggest to start with a whitelist of buildbots known to be stable... hum, like the list of "stable buildbots" ;-) Let's start with the 3.x branch, and later enable it on more branches. The mailing list should help to coordinate when multiple developers work on buildbot issues in parallel.
SGTM. -Brett
* Fix FreeBSD buildbots which fail to compile Python because of http://bugs.python.org/issue23404 I proposed a change to not regenerate generated files based on file modification time anymore, but require an explicit action ("make regen-all"). This change solves many practical issues.
* Fix remaining warnings.
* Fix unstable tests.
* Add more warnings! I proposed to emit ResourceWarning in multiprocessing.Queue and concurrent.futures executors ;-) http://bugs.python.org/issue30171 and http://bugs.python.org/issue30244
* Add more buildbots! Zachary Ware proposed to add a buildbot running "regen-all" to check that generated files are up to date.
* Repeat ;-)
Victor _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
participants (4)
-
Brett Cannon -
Erik Bray -
Victor Stinner -
Zachary Ware