[Tutor] OT: (Continuous) integration testing: Can a solo developer with very limited resources truly do this?

[Albert-Jan Roskam]

> From: dyoo at hashcollision.org
> Date: Sat, 12 Mar 2016 18:34:02 -0800
> To: robertvstepp at gmail.com
> Subject: Re: [Tutor] OT: (Continuous) integration testing: Can a solo developer with very limited resources truly do this?
> CC: tutor at python.org
> 
> On Sat, Mar 12, 2016 at 10:10 AM, boB Stepp <robertvstepp at gmail.com> wrote:
> > From "Practices of an Agile Developer" by Venkat Subramaniam and Andy
> > Hunt, c. 2006, page 90:
> >
> > <quote>
> > You're already writing unit tests to exercise your code.  Whenever you
> > modify or refactor your code, you exercise your test cases before you
> > check in the code.  All you have to do now is exercise your test cases
> > on each supported platform or environment.
> 
> 
> Hi Bob,
> 
> 
> There are some common infrastructure that folks can use.  For example,
> Travis CL is one such continuous integration system that's getting
> traction:
> 
>      https://travis-ci.org/
> 
> I think there's another CI system called Jenkins CI that's also used
> quite a bit.

I have used Jenkins CI. It was designed for/with Java, but works very well with Python. I used a multi-configuration project, where you run all unittests in a matrix of N python versions * M operating systems. There are many, many plugins for Jenkins CI and I believe I used the VirtualBox and the Git plugins. 
You can set things up in such a way that, with each commit, Jenkins tells Virtualbox to fire up the various OSes, then tell Git to make a shallow clone of your repo, then run the tests for each python version (I think this works with tox and virtualenv). It takes a while before everything is up and running, but you can start by setting up Jenkins so it just runs the tests on your host OS. Then you gradually add one OS after another. Hackintosh (unofficial MacOS) was the hardest to get working. Later it got annoying that all these VMs were fired up after each commit. So I manually fired them up whenever I believed it was a good moment. At one point, my tests failed after I updated Jenkins itself, which was very, very annoying. Jenkins, VirtualBox and Git are free software.

Appveyor is a similar service to Travis CI, but for windows. It is free for open source projects. You can set it up such that all unittests (and possibly build steps) are run after each "git push".

Docker is a way to efficiently run multiple Linux instances. I have no experience with this (because I don't want to limit things to Linux).

Vagrant is a disgustingly easy way to fire up VMs (http://www.vagrantbox.es/). You can create your own vagrant boxes with Veewee. I now see that there is a vagrant plugin for virtualbox (https://wiki.jenkins-ci.org/display/JENKINS/Vagrant-plugin)

Hercules is a program to emulate mainframes (http://www.hercules-390.org/). It felt to geeky to try and combine this with Jenkins :-).

One very different point: I once read somewhere that requirements.txt should "pin the versions": http://nvie.com/posts/pin-your-packages/.
That seems a very good idea to me.