[Tutor] OT: (Continuous) integration testing: Can a solo developer with very limited resources truly do this?

[boB Stepp]

>From "Practices of an Agile Developer" by Venkat Subramaniam and Andy
Hunt, c. 2006, page 90:

<quote>
You're already writing unit tests to exercise your code.  Whenever you
modify or refactor your code, you exercise your test cases before you
check in the code.  All you have to do now is exercise your test cases
on each supported platform or environment.

If your application is expected to run on different operating systems
(MacOS, Linux, Windows, etc.), *** you need to test on all of them ***
[My emphasis.].  If you expect your application to work on different
versions of the Java Virtual Machine (VM) or .NET common language
runtime (CLR), *** you need to test that as well *** [My emphasis.].
</quote>

I firmly agree that this -- in principle -- must be done.  But how on
earth can a solo developer, perhaps hoping to make a buck or two, with
very limited resources, accomplish manual (much less continuous,
automated) integration testing on all reasonable environment
possibilities?  Just the possibilities for Windows with their
constant, unending (albeit, necessary) string of security updates
boggles my imagination!

One of the many beauties of Python is its portability across many
platforms, but it is not perfect.  And surprises can occur, even if it
is not Python's fault.  I *thought* I was finally finished with
supporting Python 2.4 at work, and could start using features
available in Python 2.6.  While testing some new code where I had
starting using "with ..." to take care of closing files, I started
receiving erratic crashes.  Sure enough the traceback pointed to the
sections of code using the "with ..." blocks.  Huh?!?  After
investigation, I found that when my IS and commercial vendor friends
installed the second computational server for the planning software we
use, they did *not* install the *exact* same environment.  The OS
version was the same, but not all of the other ancillary software was
the same.  In this instance, they went backwards with Python and
installed Python 2.4 on the new server.  Since which server gets used
is out of the user's control, sometimes my tests ran on one server,
sometimes on the other.  And since I am stuck doing manual testing,
these kinds of things are especially hard to catch!  The only way I
will ever be able to automate testing at work will be to write my own
custom testing framework that can bridge the "automation gap" between
the software we use with its proprietary scripting language which does
not support automated testing of any sort and the Python that I find
myself using more and more, where I have the potential to automate.
But the bottom line is -- at work -- the core data I need is in the
proprietary environment and after the needed processing with Python it
has to be translated back into the proprietary environment's scripting
language and run there.

The above was just a single practical example (Rant?  Sorry!).  But
even for stuff I do at home, even with using some complex combination
of virtual environments to test all relevant OS's (Assuming I could
even afford copies WinXP, WinVista, Win7, Win10, etc., with all their
many permutations.), how can automated, continuous integration testing
be accomplished *in practice* for the severely resource-constrained
solo developer?

-- 
boB