[Cryptography-dev] Dedicated Travis Builders

Sun Mar 29 22:30:52 CEST 2015

(we had some conversation in IRC regarding this, but here is a summary of my remarks)

I would feel terrible if we tell the Travis people we want them to do all this work, then ultimately decide we need to move off Travis because doing combined coverage there is simple. If we do go down this path then in the future we'll need to deploy a coverage service that can aggregate data from both Travis and jenkins. We've talked about that in the past (and Donald wrote https://github.com/dstufft/converge) but nobody has been willing to step up and own that to date. :( Maybe we can get some volunteers for this effort? Anybody out there want to help?

-Paul

On March 29, 2015 at 12:02:29 PM, Alex Gaynor (alex.gaynor at gmail.com) wrote:

Sorry, I'm not following, it seems like you're assuming that "thing that replaces coveralls" wouldn't be able to work with both Travis and Jenkins?

Alex

On Sun, Mar 29, 2015 at 12:59 PM, Paul Kehrer <paul.l.kehrer at gmail.com> wrote:
Travis is (potentially) offering to undertake significant work on our behalf so I believe we need to have an answer to our long term coverage questions before we commit to using this solution. Being faster and having custom docker images doesn't grant us a workable coverage solution so we're going to be discussing moving entirely to jenkins again at some point in the future if we can't resolve that issue.

-Paul

On March 29, 2015 at 11:55:55 AM, Alex Gaynor (alex.gaynor at gmail.com) wrote:

Paul, it seems like coveralls is kind of orthagonal to this porposal, besides the fact that in the short term moving more stuff under Travis would mean coveralls included more stuff?

Alex

On Sun, Mar 29, 2015 at 12:53 PM, Paul Kehrer <paul.l.kehrer at gmail.com> wrote:
(Sorry for the exposition that's about to happen, just wanted to write it all down first)

We use two different CI systems right now:

- Travis, which provides OS X 10.9 builds as well as linux builds (against Ubuntu 12.04 with two different versions of OpenSSL).
- Jenkins, which provides CentOS 5, CentOS 6.4, CentOS 7, Debian Jessie, Debian Wheezy, FreeBSD 10.1, Ubuntu linking cryptography against LibreSSL, Ubuntu linking cryptography against OpenSSL 1.0.2{letter}, OS X 10.7, OS X 10.8, OS X 10.10, Windows Server 2008 (32-bit Pythons), and Windows Server 2012 (64-bit Pythons).

Travis provides us some (large) advantages over Jenkins, including not having to manage our own fleet of builders, far better (albeit less flexible) configuration through .travis.yml, and a difficult to quantify "pleasantness" to the entire experience.

So, what problems do we run into with Travis?

Coverage
------------
At this time we use coveralls (which is now commenting up to 9 times on every PR for reasons that pass human understanding) to gain a view of our combined coverage. Unfortunately this system ties us to reporting coverage exclusively from Travis. This means code paths (like windows only code) cannot have coverage tracked.

Speed/Reliability
----------------------
We have limited (albeit very high at the moment) concurrency available from Travis. On a typical workday it can be 1-3 hours before CI completes, which significantly impairs our ability to quickly review/merge code.  We have had occasions in the past where we've had Travis jobs waiting in the queue for over 8 hours due to reliability problems within the Travis infrastructure.

Flexibility
------------
Travis provides only OS X 10.9 and Ubuntu 12.04.

The openstack builder possibility solves much of the speed issue, and custom docker images would negate the need for our jenkins linux builders (of which there are currently 7). I am, however, concerned about coverage. Without being able to run more than OS X 10.9 and (potentially) various userlands of Linux via docker images we're still significantly handicapped. It is unreasonable to expect to be able to run arbitrary OSes so we'll always run a small jenkins instance for things like alternate OS X versions, FreeBSD, etc, but the lack of Windows support is painful.

We have recently discussed moving entirely off Travis (https://github.com/pyca/cryptography/issues/1729), but if the above comes to fruition and there's a good way to combine coverage from multiple sources (and someone wants to own getting it deployed that isn't me) then I'd be happy to stay with Travis and only use jenkins for the edge cases.

-Paul

On March 28, 2015 at 5:56:31 PM, Donald Stufft (donald at stufft.io) wrote:

I was talking to the Travis CI folks and they mentioned they have the
possibility of using openstack builders now (or in the near future). We talked
a little bit about the possibility of getting PyCA it's own dedicated set of
builders in Travis CI hosted inside of Rackspace on one of our cloud accounts.

Currently the entirety of the PyPA organization on Github gets 10 concurrent
builders (typically this number is 5) across all repositories. Assuming Travis
CI is able to do it (they'd need to check with their ops team, and there would
be some cost associated with things on their side as well) we could essentially
control how much concurrency we want by just throwing more Rackspace VMs at
our builds.

Josh said they use a 2 vCPU machine w/ 4GB of ram, so looking at the Rackspace
Compute1-4 which is similar we'd be looking at the ability to run ~27 of those
VMs full time on the "standard" 2k/month free cloud offering that Rackspace
offers OSS projects. However we'd actually be able to get more than that
probably because Travis starts up the VMs on demand so anytime there aren't
tests to run we won't have any VMs running.

Independent of this, they are also working on the ability to run custom docker
images (which means custom Linux OS images) that we can preinstall different
software into.

Is this something we'd want to explore? Assuming that the Travis CI ops team
and such are OK with it, it could essentially let us scale up our concurrency
on Travis to whatever amount of dollars of Rackspace cloud we want to throw at
it.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

_______________________________________________
Cryptography-dev mailing list
Cryptography-dev at python.org
https://mail.python.org/mailman/listinfo/cryptography-dev

_______________________________________________
Cryptography-dev mailing list
Cryptography-dev at python.org
https://mail.python.org/mailman/listinfo/cryptography-dev

--
"I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084
_______________________________________________
Cryptography-dev mailing list
Cryptography-dev at python.org
https://mail.python.org/mailman/listinfo/cryptography-dev

_______________________________________________
Cryptography-dev mailing list
Cryptography-dev at python.org
https://mail.python.org/mailman/listinfo/cryptography-dev

--
"I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084
_______________________________________________  
Cryptography-dev mailing list  
Cryptography-dev at python.org  
https://mail.python.org/mailman/listinfo/cryptography-dev  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cryptography-dev/attachments/20150329/ed3ed080/attachment.html>