[Python-Dev] buildbot VM exhausted in test_crashers

[David Bolen]

I've run out of virtual memory on my Ubuntu buildbot today (and I
think the FreeBSD buildbots a few days ago).  Those have physically
OOMd/restarted the VM.  The Windows buildbots have been giving me
VM warnings, but appear to have gotten through it.

I believe it's the test_crashers test, which when I observed the
Ubuntu buildbot when trying to bring it back online suddenly spikes to
over 500MB allocated to the Python process (which is the limit of
physical memory on that box), then causes an overall OOM and crash.

The problem is that the failing build is automatically retried when I
try to reconnect the buildbot, so it just get rights into trouble
again.  The comment in that build's changeset looks like it might be
getting rolled back?

Assuming that's the case, is there a way to flush any pending builds
for my slaves before I bring them back online?

Thanks.

-- David

PS: Neither FreeBSD nor my Ubuntu slave are actually up though the master
thinks they are.  The pinging option worked once for the Ubuntu slave but
at the moment I can't seem to get the master to think that any of them
are really disconnected.