[Tutor] OT: (Continuous) integration testing: Can a solo developer with very limited resources truly do this?

[wolfrage8765 at gmail.com]

> You're already writing unit tests to exercise your code.  Whenever you
> modify or refactor your code, you exercise your test cases before you
> check in the code.  All you have to do now is exercise your test cases
> on each supported platform or environment.
>
> If your application is expected to run on different operating systems
> (MacOS, Linux, Windows, etc.), *** you need to test on all of them ***
> [My emphasis.].  If you expect your application to work on different
> versions of the Java Virtual Machine (VM) or .NET common language
> runtime (CLR), *** you need to test that as well *** [My emphasis.].
> </quote>
>
> I firmly agree that this -- in principle -- must be done.  But how on
> earth can a solo developer, perhaps hoping to make a buck or two, with

I as a solo developer have been unable to achieve this task too; but
once my software is closer to complete that is when I look to do
integration testing.

> very limited resources, accomplish manual (much less continuous,
> automated) integration testing on all reasonable environment
> possibilities?  Just the possibilities for Windows with their
> constant, unending (albeit, necessary) string of security updates
> boggles my imagination!

You can always download the free test versions of windows; that is
what I do and then I throw them on a virtual machine so as to not have
to suck up that cost. But dealing with what security updates break is
a different story all together.

>
> One of the many beauties of Python is its portability across many
> platforms, but it is not perfect.  And surprises can occur, even if it
> is not Python's fault.  I *thought* I was finally finished with
> supporting Python 2.4 at work, and could start using features
> available in Python 2.6.  While testing some new code where I had
> starting using "with ..." to take care of closing files, I started
> receiving erratic crashes.  Sure enough the traceback pointed to the
> sections of code using the "with ..." blocks.  Huh?!?  After
> investigation, I found that when my IS and commercial vendor friends
> installed the second computational server for the planning software we
> use, they did *not* install the *exact* same environment.  The OS
> version was the same, but not all of the other ancillary software was
> the same.  In this instance, they went backwards with Python and
> installed Python 2.4 on the new server.  Since which server gets used
> is out of the user's control, sometimes my tests ran on one server,
> sometimes on the other.  And since I am stuck doing manual testing,
> these kinds of things are especially hard to catch!  The only way I
> will ever be able to automate testing at work will be to write my own
> custom testing framework that can bridge the "automation gap" between
> the software we use with its proprietary scripting language which does
> not support automated testing of any sort and the Python that I find
> myself using more and more, where I have the potential to automate.
> But the bottom line is -- at work -- the core data I need is in the
> proprietary environment and after the needed processing with Python it
> has to be translated back into the proprietary environment's scripting
> language and run there.

Yes in my virtual environments I have over time been diligently
writing a testing automation tool; that starts when the OS boots; then
looks to a specific server for a specific directory to download the
code to be tested. But it is very rough; the key is to start small and
write code that can be re-used. But I am still missing an android and
IOS client so I have to test those manually.

>
> The above was just a single practical example (Rant?  Sorry!).  But
> even for stuff I do at home, even with using some complex combination
> of virtual environments to test all relevant OS's (Assuming I could
> even afford copies WinXP, WinVista, Win7, Win10, etc., with all their
> many permutations.), how can automated, continuous integration testing
> be accomplished *in practice* for the severely resource-constrained
> solo developer?

Good Luck!