[Cryptography-dev] Interested in your CI Experience

[Paul Kehrer]

Hi Augustina,

cryptography's needs are a bit unusual. Since we depend on the underlying
OS/distribution to provide us with OpenSSL (or commoncrypto on the macOS
side) and we compile against a library that has a somewhat checkered past
with versioning we have to test against numerous configurations. To achieve
the coverage we want we currently use two different CI systems and
aggregate coverage across them.

== Travis CI ==

Recent CI build: https://travis-ci.org/pyca/cryptography/builds/181337155

Travis CI gives us configuration-as-code, reasonable concurrency (we have
10 simultaneous builders, which is a godsend with the number of jobs we
run), and support for both linux and macOS (multiple versions).

== Jenkins ==

Recent CI build:
https://jenkins.cryptography.io/job/cryptography-master/1879/

Jenkins (using github pull request builder and multiconfiguration jobs)
allows us to test unusual configs (like random test order) as well as
against the set of configurations not supported by Travis. This includes
Windows, FreeBSD, various linux distributions, and additional versions of
macOS.

Jenkins also lets us build wheels for distribution during releases using
the same infrastructure. We currently ship 21 wheels, but that number will
grow shortly with Python 3.6.

Our current config is not using jenkinsfile (which has proven challenging
to implement due to a variety of bugs) so the configs are not checked into
CI and we commonly have to spend time working on the cluster since it's
managed by us. However, an advantage this does offer is that our
concurrency scales with our willingness to boot more VMs...


So, in an ideal world what we want out of CI is:

* Many OSes/distributions
* configuration as code
* high concurrency to minimize total CI time
* integration with github for status reporting
* fast and reliable booting of test instances (whether they're full VMs or
containers)

As part of the "we want it to be fast" requirement being able to cache
artifacts we may re-use repeatedly (like Python runtimes) can be extremely
valuable in lowering total CI runtime.

If BonnyCI can offer this to us in the future then we'll be happy to take a
close look!

-Paul Kehrer (reaperhulk)

On December 9, 2016 at 12:28:52 PM, Augustina Ragwitz (
aragwitz.lists at pobox.com) wrote:

I'm working on a project with a group of folks from the Openstack
community called BonnyCI. In a nutshell, we're making the Openstack
Infra CI environment work with Github and making it available for
non-Openstack projects. I've spoken with Donald Stufft privately to see
if you all might want to chat further about your current CI workflow and
he suggested I reach out to the mailing list. Our big challenge with
BonnyCI is while we know the Openstack needs and use cases, we have
little knowledge outside of that, and we want to make sure what we're
building is actually going to help address any issues folks might be
having wih their current setup. As BonnyCI gets more stable, we'll also
be looking for projects who would be willing to try it out!

If anyone would be interested in talking to me further, I'm happy to
talk on this list, via direct email, IRC, hangouts/skype/etc. I've been
lurking in your IRC channel, so feel free to ping me or message me
directly there (nick: auggy). I don't want this to take too much of your
time (or energy) so I'm open to whatever works for anyone who is
interested in talking to me further.

-- 
Augustina Ragwitz
Señora Software Engineer
---
irc: auggy
_______________________________________________
Cryptography-dev mailing list
Cryptography-dev at python.org
https://mail.python.org/mailman/listinfo/cryptography-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cryptography-dev/attachments/20161209/b4d27af9/attachment.html>