[Cryptography-dev] Dedicated Travis Builders

Sun Mar 29 21:51:36 CEST 2015

One option for coveralls replacement is having each Jenkins job export the .coverage file as an artifact, and then have another job which consumes all of the .coverage files from those jobs and combines them. That would be Jenkins specific if we went with that solution.

> On Mar 29, 2015, at 1:02 PM, Alex Gaynor <alex.gaynor at gmail.com> wrote:
> 
> Sorry, I'm not following, it seems like you're assuming that "thing that replaces coveralls" wouldn't be able to work with both Travis and Jenkins?
> 
> Alex
> 
> On Sun, Mar 29, 2015 at 12:59 PM, Paul Kehrer <paul.l.kehrer at gmail.com <mailto:paul.l.kehrer at gmail.com>> wrote:
> Travis is (potentially) offering to undertake significant work on our behalf so I believe we need to have an answer to our long term coverage questions before we commit to using this solution. Being faster and having custom docker images doesn't grant us a workable coverage solution so we're going to be discussing moving entirely to jenkins again at some point in the future if we can't resolve that issue.
> 
> -Paul
> 
> On March 29, 2015 at 11:55:55 AM, Alex Gaynor (alex.gaynor at gmail.com <mailto:alex.gaynor at gmail.com>) wrote:
> 
>> Paul, it seems like coveralls is kind of orthagonal to this porposal, besides the fact that in the short term moving more stuff under Travis would mean coveralls included more stuff?
>> 
>> Alex
>> 
>> On Sun, Mar 29, 2015 at 12:53 PM, Paul Kehrer <paul.l.kehrer at gmail.com <mailto:paul.l.kehrer at gmail.com>> wrote:
>> (Sorry for the exposition that's about to happen, just wanted to write it all down first)
>> 
>> We use two different CI systems right now:
>> 
>> - Travis, which provides OS X 10.9 builds as well as linux builds (against Ubuntu 12.04 with two different versions of OpenSSL).
>> - Jenkins, which provides CentOS 5, CentOS 6.4, CentOS 7, Debian Jessie, Debian Wheezy, FreeBSD 10.1, Ubuntu linking cryptography against LibreSSL, Ubuntu linking cryptography against OpenSSL 1.0.2{letter}, OS X 10.7, OS X 10.8, OS X 10.10, Windows Server 2008 (32-bit Pythons), and Windows Server 2012 (64-bit Pythons).
>> 
>> Travis provides us some (large) advantages over Jenkins, including not having to manage our own fleet of builders, far better (albeit less flexible) configuration through .travis.yml, and a difficult to quantify "pleasantness" to the entire experience.
>> 
>> So, what problems do we run into with Travis?
>> 
>> Coverage
>> ------------
>> At this time we use coveralls (which is now commenting up to 9 times on every PR for reasons that pass human understanding) to gain a view of our combined coverage. Unfortunately this system ties us to reporting coverage exclusively from Travis. This means code paths (like windows only code) cannot have coverage tracked.
>> 
>> Speed/Reliability
>> ----------------------
>> We have limited (albeit very high at the moment) concurrency available from Travis. On a typical workday it can be 1-3 hours before CI completes, which significantly impairs our ability to quickly review/merge code.  We have had occasions in the past where we've had Travis jobs waiting in the queue for over 8 hours due to reliability problems within the Travis infrastructure.
>> 
>> Flexibility
>> ------------
>> Travis provides only OS X 10.9 and Ubuntu 12.04.
>> 
>> 
>> 
>> The openstack builder possibility solves much of the speed issue, and custom docker images would negate the need for our jenkins linux builders (of which there are currently 7). I am, however, concerned about coverage. Without being able to run more than OS X 10.9 and (potentially) various userlands of Linux via docker images we're still significantly handicapped. It is unreasonable to expect to be able to run arbitrary OSes so we'll always run a small jenkins instance for things like alternate OS X versions, FreeBSD, etc, but the lack of Windows support is painful.
>> 
>> We have recently discussed moving entirely off Travis (https://github.com/pyca/cryptography/issues/1729 <https://github.com/pyca/cryptography/issues/1729>), but if the above comes to fruition and there's a good way to combine coverage from multiple sources (and someone wants to own getting it deployed that isn't me) then I'd be happy to stay with Travis and only use jenkins for the edge cases.
>> 
>> -Paul
>> 
>> On March 28, 2015 at 5:56:31 PM, Donald Stufft (donald at stufft.io <mailto:donald at stufft.io>) wrote:
>> 
>>> I was talking to the Travis CI folks and they mentioned they have the
>>> possibility of using openstack builders now (or in the near future). We talked
>>> a little bit about the possibility of getting PyCA it's own dedicated set of
>>> builders in Travis CI hosted inside of Rackspace on one of our cloud accounts.
>>> 
>>> Currently the entirety of the PyPA organization on Github gets 10 concurrent
>>> builders (typically this number is 5) across all repositories. Assuming Travis
>>> CI is able to do it (they'd need to check with their ops team, and there would
>>> be some cost associated with things on their side as well) we could essentially
>>> control how much concurrency we want by just throwing more Rackspace VMs at
>>> our builds.
>>> 
>>> Josh said they use a 2 vCPU machine w/ 4GB of ram, so looking at the Rackspace
>>> Compute1-4 which is similar we'd be looking at the ability to run ~27 of those
>>> VMs full time on the "standard" 2k/month free cloud offering that Rackspace
>>> offers OSS projects. However we'd actually be able to get more than that
>>> probably because Travis starts up the VMs on demand so anytime there aren't
>>> tests to run we won't have any VMs running.
>>> 
>>> Independent of this, they are also working on the ability to run custom docker
>>> images (which means custom Linux OS images) that we can preinstall different
>>> software into.
>>> 
>>> Is this something we'd want to explore? Assuming that the Travis CI ops team
>>> and such are OK with it, it could essentially let us scale up our concurrency
>>> on Travis to whatever amount of dollars of Rackspace cloud we want to throw at
>>> it.
>>> 
>>> ---
>>> Donald Stufft
>>> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>>> 
>>> _______________________________________________
>>> Cryptography-dev mailing list
>>> Cryptography-dev at python.org <mailto:Cryptography-dev at python.org>
>>> https://mail.python.org/mailman/listinfo/cryptography-dev <https://mail.python.org/mailman/listinfo/cryptography-dev>
>> 
>> _______________________________________________
>> Cryptography-dev mailing list
>> Cryptography-dev at python.org <mailto:Cryptography-dev at python.org>
>> https://mail.python.org/mailman/listinfo/cryptography-dev <https://mail.python.org/mailman/listinfo/cryptography-dev>
>> 
>> 
>> 
>> 
>> --
>> "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
>> "The people's good is the highest law." -- Cicero
>> GPG Key fingerprint: 125F 5C67 DFE9 4084
>> _______________________________________________
>> Cryptography-dev mailing list
>> Cryptography-dev at python.org <mailto:Cryptography-dev at python.org>
>> https://mail.python.org/mailman/listinfo/cryptography-dev <https://mail.python.org/mailman/listinfo/cryptography-dev>
> 
> _______________________________________________
> Cryptography-dev mailing list
> Cryptography-dev at python.org <mailto:Cryptography-dev at python.org>
> https://mail.python.org/mailman/listinfo/cryptography-dev <https://mail.python.org/mailman/listinfo/cryptography-dev>
> 
> 
> 
> 
> --
> "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
> "The people's good is the highest law." -- Cicero
> GPG Key fingerprint: 125F 5C67 DFE9 4084
> _______________________________________________
> Cryptography-dev mailing list
> Cryptography-dev at python.org
> https://mail.python.org/mailman/listinfo/cryptography-dev

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cryptography-dev/attachments/20150329/6deccf92/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/cryptography-dev/attachments/20150329/6deccf92/attachment-0001.sig>