[Chicago] The current state of Testing Stuff

[Tal Liron]

This is very much a matter of 1) quality standards in your industry, and 
2) personality.


I have two rules of thumbs:


1) The Scripting Rule: If I find myself having to do the same thing more 
than once, I make it into a script. In the realm of testing, this is 
anything automated, from "unit" tests to Selenium browser scripts. I 
never, ever, write "unit tests" just because.

I've tried the path of religiously testing everything, and found it to 
be a huge waste of my creative juices, with meagre benefits. I think 
that in most cases there are better ways to assure quality, involving a 
combination of responsible humans assisted by automated scripts.

With exceptions. There are definitely industries where any level of 
failure is unacceptable, and anything that could be tested must be 
tested a zillion times. I just haven't worked in these industries for a 
while, because I like sleeping at night.

On that note, I find the name "unit test" to be so completely misused 
that it's lost its meaning to me. What is the "unit"? Far too often, 
programmers and frameworks use it to refer to an "object" or some other 
implementation-specific construct. I think the "unit" should be whatever 
aspect, subsystem, entry point, etc., of your software that could be 
tested. In particular, the paradigm of "unit testing" creates a tunnel 
vision that ignores a unit's deep integration into larger systems. Many 
bugs (especially concurrency bugs) will only appear when run as a whole. 
So, let's just call it "automated testing" and describe the "unit" as 
appropriate. Even the name "regression suite" is a bit annoying to me, 
because it implies a suite of "unit tests." The unit, again, is 
slippery. So, again, rule of thumb: I write a test only when I find 
myself repeating myself. Like I am now. :)


2) I do a lot of work on subtle concurrency stuff, where testing = 
debugging. The only to find concurrency bugs is to throw massive amounts 
of threads, processes, browsers, etc., at your software, and also to 
test on multi- vs. single-core machines. (P.S. I love BrowserMob.) This 
kind of testing always involves robust logging, stack traces, and 
sometimes VM instrumentalization (if your platform supports it). 
Programming concurrency is really, really hard (for me), and no amount 
of poring over the source code can help me find bugs. I can't count how 
many times concurrency tests have discovered major breakage in my code. 
Moreover, they teach me a lot.


The rule of thumb here is to ALWAYS test concurrency, and this does 
contradict my previous rule of thumb.


-Tal


On 12/10/2010 10:48 AM, Brian Ray wrote:

> This is an intentionally vague topic regarding testing from
> acceptability to unit tests. I am suggesting things have changed and
> perhaps it is time to review testing methodologies.  I am sure many of
> you have dealt with this change already. At least this could be an
> interesting topic for some I hope.
>
> I recall in 2005 ChiPy had a plethora of talks on unit testing, Mock
> Objects and fitness testing. Later we talked a lot about nose tests
> and runner tools. More recently we had talks on great tools like tox
> but it still included things like hudson to run. Somewhere in between
> we talked about Selenium.  There was a link (oh here it is) that lists
> tools http://pycheesecake.org/wiki/PythonTestingToolsTaxonomy . I am
> not sure this test is exhaustive.
>
> So here comes my question.... What did we learn from these different
> efforts in testing; what has changed in testing; what is the best
> modern method to implement large scale testing that covers the whole
> stack? How much testing in development should be made available for
> testing in QA and acceptability?
>
> What is the point of writing test on something that will never fail?
> Have testing tools changed to become more restful? Has anyone ever had
> a test failure from an automated suite that actually pointed to
> something useful? It seems testing the smaller lower level items can
> be covered well with Unit Tests. Higher and middle level items perhaps
> are covered well by fitness and maybe browser level? How do we
> automate useful tests? How does one approach testing more complicated
> things like events and threads?  Now things are becoming so mashed...
> what are people doing to test interpolation with thinks like web
> services that someone else maintains?
>
> I guess I am also looking for life cycle configuration ideas.  I could
> see someone saying something like this:  "We write UnitTests we run
> from Nose for TDD, for acceptability in development we use urllib2 to
> test RESTFul stuff, and then on top of that we use Selenium for
> browser testing..."
>