[core-workflow] PEP 512 - Choosing a CI service

[Darjus Loktevic]

Quick background: A little while ago I've added TravisCI integration to
Jython github mirror. Most recently one of Jython contributors asked
whether we can integrate the xunit XML output of Jython's regrtest with
TravisCI. The answer was no, which led me to researching other options.

*Why not TravisCI?*
TravisCI does not support build artifacts. As simple as that, there's no
builtin support for them. After researching for a while, they promised to
add it but have no public timelines. This is quite limiting in my view as
it would be nice to have more than just a binary pass/fail and manually dig
through regrtest output.
One of the positives for TravisCI is Mac OS X support.
https://travis-ci.org/jythontools/jython

*Codeship*
I have not actually used it, but from reading documentation it feels more
like a deployment service and does not have builtin support for xunit.

*Shippable*
Configuration format is super similar to TravisCI, which made it easy to
try. Has support for both code coverage and xunit test results. Simple UI.
One big negative for me with it is that it is slow, and by slow i mean over
1h versus 25min for TravisCI. Presumably paid version is faster.
https://app.shippable.com/builds/569eebaed3a5e70d00b309c5

*CircleCI* - My preferred
Very fast, free tier gives 4 parallel builds (free tier), supports xunit,
supports Mac OS X (have not tried), and supports debugging via SSH (which i
think is a cool feature for those cases when it works on your machine and
not on the build server and there are no clues as to why). Supports
exporting coverage to several coverage services.
On the negatives, different configuration format to TravisCI, UI could be
simpler/easier.
https://circleci.com/gh/darjus/jython/3#tests

Cheers,
Darjus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/core-workflow/attachments/20160120/0bb82236/attachment.html>