[Tutor] Unit testing infinite loops

[James Chapman]

Thanks Steven!

You've raised a few valid points, mostly that the run_forever method
should be broken up. I like the principle of a method doing just one
thing and for whatever reason I didn't apply that thinking to this
method as it's the master loop (even though it does nothing). So for
starters I'll fix that. Breaking everything up makes testing easier,
which in turn makes development easier.

On the note of what doesn't need a test...
The question of coverage always comes up when unit testing is
mentioned and I read an interesting blog article once about it. It
basically said: Assume you have 85% coverage on a program that
consists of 1,000 lines. That's 150 lines which are not tested. If
those lines are print lines, sleep lines, getters etc it's not really
a problem. But what happens when you scale that up. 1,000,000 lines of
code lets say (not unheard of, although in python that would be out of
this world big). You now end up with 150,000 lines of untested code.
While the percentage of code covered is high, there is _a_lot_ of code
there that isn't tested and a lot of room for mistakes to creep in. A
mistake on one of those 150,000 lines could break the build and
possibly cost you hours or even days tracking it down. If those lines
were tested however, your continuous integration build system would
hopefully highlight the fault.

In my experience testing works, saves time down the line, and makes
code easier to come back to.
--
James


On 31 January 2014 13:21, Steven D'Aprano <steve at pearwood.info> wrote:
> On Fri, Jan 31, 2014 at 11:31:49AM +0000, James Chapman wrote:
>> Hello tutors
>>
>> I've constructed an example which shows a problem I'm having testing a real
>> world program and would like to run it past you.
> [...]
>> class Infinite_Loop_Tutor_Question(object):
>>     def run_forever(self):
>>         self.start_A()
>>         time.sleep(0.5)
>>         self.start_B()
>>         try:
>>             while True:
>>                 time.sleep(1)
>>         except KeyboardInterrupt:
>>             print("Caught Keyboard Interrupt...")
>>             sys.exit(0)
> [...]
>> In my example above, testing the everything but the run_forever method is
>> trivial.
>>
>> So on to my question... The run_forever method essentially just fires up a
>> bunch of threads to serve various purposes and then waits for CTRL-C to
>> terminate the entire program. Testing this at the moment is very difficult
>> because the unit test ends up in the infinite loop. So, would a better idea
>> be to create an attribute, set it to True and then do
>>
>> try:
>>     while self.attribute:
>>         time.sleep(1)
>> except KeyboardInterrupt:
>>     ...
>
>
> That probably won't hurt.
>
>
>> My unit test could then set the attribute. However I'd still have the
>> problem of how I get from the unit test line that fires up the method to
>> the next line to change the attribute.
>>
>> So how should the run_forever method be written so that it's testable, or
>> if it's testable as is, how would I test it?
>
> What are you trying to test? You don't just "test" a method, you test
> *something specific* about the method. So what specifically are you
> trying to test?
>
>
>> And please, no comments about syntax, clean exits of threads, thread
>> communication, resources, or even the need for testing the run_forever
>> method. In my test I want to test that it makes the relevant calls and then
>> enters the infinite loop at which point I want to terminate it.
>
> Ah, you see, now you have a problem. Consider this function:
>
> def long_calc():
>     time.sleep(60*60*24*365)
>     return 1
>
>
> How do I test that the function returns 1? As given, I can't
> really, not unless I wait a whole year for the sleep() to return. So
> what I can do is split the function into two pieces:
>
> def _sleep_a_year():
>     time.sleep(60*60*24*365)
>
> def _do_calculation():
>     return 1
>
> def long_calc():
>     _sleep_a_year()
>     return _do_calculation()
>
>
> Now I can unit-test the _do_calculation function, and long_calc() is now
> simple enough that I don't really need to unit-test it. (Unit testing
> should not be treated as a religion. You test what you can. Any testing
> is better than nothing, and if there are some parts of the program which
> are too hard to test automatically, don't test them automatically.)
>
> Or, I can monkey-patch the time.sleep function. Before running my
> test_long_calc unit-test, I do this:
>
> import time
> time.sleep = lambda n: None
>
> and then restore it when I'm done. But like all monkey-patching, that's
> risky -- what if the calculation relies on time.sleep somewhere else?
> (Perhaps it calls a function, which calls another function in a module
> somewhere, which calls a third module, which needs time.sleep.) So
> monkey-patching should be a last resort.
>
> Another alternative is to write the function so it can be
> tested using a mock:
>
> def long_calc(sleeper=time.sleep):
>     sleeper(60*60*24*365)
>     return 1
>
>
> Then I can test it like this:
>
> assert stupid(lambda n: None) == 1
>
> where the lambda acts as a mock-up for the real sleep function.
>
>
> Let's look at your method. As given, it's too hard to test. Maybe you
> could write a unit test which fires off another thread, which then
> sleeps for a few seconds before (somehow!) sending a KeyboardInterrupt
> to the main thread. But that's hard to explain and harder to do, and I
> really wouldn't want to rely on something so fiddly. So let's re-design
> the method with testing in mind.
>
> First, pull out the part that does the infinite loop:
>
>     def do_infinite_loop():
>         # Loop forever. Sleep a bit to avoid hogging the CPU.
>         while True:
>             time.sleep(1)
>
>
> That's *so simple* that it doesn't need a test. The body of the method
> is two short, easy lines, plus a comment. If somebody can read that and
> be unsure whether or not it works correctly, they're in trouble.
>
> But, if you like, you can make it more complicated. Have the method
> check for a magic global variable, or a instance attribute, or
> something:
>
>     def do_infinite_loop():
>         # Loop forever. Sleep a bit to avoid hogging the CPU.
>         # Perhaps not forever.
>         if hasattr(self, 'DONT_LOOP_FOREVER'):
>             x = 10
>             while x > 0:
>                 time.sleep(1)
>                 x -= 1
>         else:
>             while True:
>                 time.sleep(1)
>
>
> Yuck. Now you have added enough complication that it is no longer
> obvious that the method works, and while you can test the non-infinite
> loop part, you still can't test the infinite loop part. No, better to
> stick with the simplest thing that works.
>
> Now for the rest of your method:
>
>
>     def run_forever(self):
>         self.start_A()
>         time.sleep(0.5)
>         self.start_B()
>         try:
>             self.do_infinite_loop()
>         except KeyboardInterrupt:
>             print("Caught Keyboard Interrupt...")
>             sys.exit(0)
>
>
> How does this help us with testing? We can monkey-patch the
> do_infinite_loop method!
>
>
> class MyTest(unittest.TestCase):
>     def test_run_forever_catches_KeyboardInterrupt_and_exits(self):
>         def mock_looper(self):
>             raise KeyboardInterrupt
>         instance = Infinite_Loop_Tutor_Question()
>         # Monkey-patch.
>         instance.do_infinite_loop = mock_looper
>         self.assertRaises(SystemExit, instance.do_infinite_loop)
>
>
> Testing that it prints the appropriate message, I leave as an exercise.
> (Hint: you can re-direct stdout, run the method inside a try...except
> block, then restore stdout.)
>
> (By the way, I haven't tested any of this code.)
>
>
> --
> Steven
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor