[Python-Dev] Most 3.x buildbots are green again, please don't break them and watch them!

Zachary Ware zachary.ware+pydev at gmail.com
Wed Apr 13 16:16:08 EDT 2016

On Wed, Apr 13, 2016 at 6:40 AM, Victor Stinner
<victor.stinner at gmail.com> wrote:
> Hi,
> Last months, most 3.x buildbots failed randomly. Some of them were
> always failing. I spent some time to fix almost all Windows and Linux
> buildbots. There were a lot of different issues.

Thank you for doing this!

> Maybe it's time to move more 3.x buildbots to the "stable" category?
> http://buildbot.python.org/all/waterfall?category=3.x.stable

A few months ago, I put together a list of suggestions for updating
the stable/unstable list, but never got around to implementing it.

> We have many offline buildbots. What's the status of these buildbots?
> Should we expect that they come back soon?

My Windows 8.1 bot is a VM that resides on a machine that has been
disturbingly unstable lately, and it's starting to seem like the
instability is due to that VM.  I hope to have it back up (and stable)
again soon, but have no timetable for it.  My Docs bot was off after
losing power over the weekend, and I just hadn't noticed yet.  It's
back now.

I'll ping the python-buildbots list about other offline bots.

> Or would it be possible to hide them? It would help to check the
> status of all buildbots.

I'm not sure, but that would be a nice feature.

> - the 4 ICC buildbots are failing with stack overflow, segfault, etc.
> Again, I'm not sure that these buildbots are useful since it looks
> like we don't support this compiler yet. Or does it help to work on
> supporting this compiler? Who is working on ICC support?

The Ubuntu ICC bot is generally quite stable.  The OSX ICC bot is
currently offline, but has only a couple of known issues.  The Windows
ICC bot is still a bit experimental, but has inched closer to
producing a working build.  R. David Murray and I have been working
with Intel on ICC support.

> By the way, I'm always surprised by the huge difference of time needed
> to run a build on the different slaves: from a few minutes to more
> than 3 hours. The fatest Windows slave takes 28 minutes (run tests in
> parallel using 4 child processes), whereas the 3 others (run tests
> sequentially and) take between 2 hours and more than 3 hours! Why
> running tests on Windows takes so long?

Most of that is down to debug mode; building Python in debug mode
links with the debug CRT which also enables all manner of extra
checks.  When it's up, the non-debug Windows bot also runs the test
suite in ~28 minutes, running sequentially.


After receiving a suggestion from koobs several months ago, I've been
intermittently thinking about completely redoing our buildmaster setup
such that instead of a single builder per version on each slave, we
instead set up a series of builders with particular 'tags', and each
builder attaches to each slave that satisfies the tags (running each
build only on the first slave available).  This would allow us to test
some of the rarer options (such as --without-threads) significantly
more often than 'never', and generally get a lot more
customization/flexibility of builds.  I haven't had a chance to sit
down and think out all the edge cases of this idea, but what do people
generally think of it?  I think the GitHub switchover will be a good
time to do this if it's generally seen as a decent idea, since there
will need to be some work on the buildmaster to do the switch anyway.


More information about the Python-Dev mailing list