On Nov 3, 2016, at 10:17 AM, Barry Warsaw <barry@python.org> wrote:

On Nov 03, 2016, at 12:54 AM, Nick Coghlan wrote:

This is also an area where I'm fine with recommending freemium
solutions if they're the lowest barrier to entry option for new users,
and "Use GitHub + Travis CI" qualifies on that front.

I won't rehash the GitHub/GitLab debate, but in some of my projects (hosted on
GH) I've had to ditch Travis because of limitations on that platform.
Specifically, I needed to run various tests on an exact specification of
various Ubuntu platforms, e.g. does X run on an up-to-date Ubuntu Y.Z?

I originally used Docker for this, but our projects had additional
constraints, such as needing to bind-mount, which aren't supported on the
Travis+Docker platform.  So we ended up ditching the Travis integration and
hooking our test suite into the Ubuntu autopkgtest system (which is nearly
identical to the Debian autopkgtest system but runs on Ubuntu infrastructure).

Python may not be affected by similar constraints, but it is worth keeping in
mind.  Travis isn't a perfect technical solution for all projects, but it may
be good enough for Python.

I think phrasing this in terms of "perfect" and "good enough" presents a highly misleading framing.  Examined in this fashion, of course we may reluctantly use the "good enough" option, but don't we want the best option?

A better way to look at it is cost vs. benefit.

How much does it cost you in terms of time and money to run and administer the full spectrum of "real" operating systems X.Z that you wish to support?  How much does it cost in terms of waiting for all that extra build infrastructure to run all the time?  How much additional confidence and assurance that it will work does that buy you, over the confidence of passing tests within a docker container?  Is that additional confidence worth the investment of resources?

Of course, volunteer-driven projects are not concerned directly with top-level management allocation of ostensibly fungible resources, and so a hard "costly" solution that someone is interested in and committed to is far less expensive than a "cheap" solution that everyone finds boring, so we have to take that into account as well.

As it happens, Twisted has a massive investment in existing Buildbot CI infrastructure _as well as_ Travis and Appveyor.  Travis and Appveyor address something that our CI can't, which is allowing unauthenticated builds from randos issuing their first pull requests.  This gives contributors much faster feedback which is adequate for the majority of changes.

However, many of our ancillary projects, which do not have as many platform-sensitive components, are built using Travis only, and that's a very good compromise for them.  It has allowed us to maintain a much larger and more diverse ecosystem with a much smaller team than we used to be able to.

In the future, we may have to move to a different CI service, but I can tell you that for sure that 90% of the work involved in getting builds to run on Travis is transferrable to any platform that can run a shell script.  There's a bit of YAML configuration we would need to replicate, and we might have to do some fancy dancing with Docker to get other ancillary services run on the backend in some other way, but I would not worry about vendor lock-in at all for this sort of service.  Probably, the amount of time and energy on system maintenance that Travis saves us in a given week is enough to balance out all the possible future migration work.

-glyph