[Python-Dev] OpenBSD buildbot has many failures

[Victor Stinner]

Hi,

2015-04-01 12:47 GMT+02:00 Tim Golden <mail at timgolden.me.uk>:
> On the back of Victor's recent emails re buildbots, I've knocked
> something up which can be scheduled to email the status of some or all
> buildbots:
>
>   https://github.com/tjguk/buildbotinfo

Are you aware of this previous project?
https://code.google.com/p/bbreport/

I also wrote two very simple scripts to download and parse buildbot output:

https://bitbucket.org/haypo/misc/src/5929cc110f0352cecb384adceae3647f26fa693e/python/buildbot_download.py?at=default

https://bitbucket.org/haypo/misc/src/5929cc110f0352cecb384adceae3647f26fa693e/python/buildbot_parse.py?at=default

I'm more interested on the parser part. My first goal was to compute a
summary in one line: success, timeout, fatal error, etc. The next step
would be to parse test failures to be able to ignore tests known to
fail, to only detect regressions. It would be better than the current
"red" / "green" status. For example, I want to always ignore
MemoryError just because I'm unable to fix buildbots which are known
to regulary fail with MemoryError (ex: OpenIndiana).

I used Jenkins which uses JUnit reports. It's much more powerful
because it computes statistics on tests to check which tests fail
randomly, which tests are new failures, etc. I don't think that it
would be complex to generate a JUnit report (or another easy to parse
report) in regrtest. But XML is maybe not the best format to handle
timeout and fatal errors :-)

Victor