On Tue Nov 25 2014 at 2:27:46 PM Dave Halter <
davidhalter88@gmail.com> wrote:
There is no explicit handling of the Python semantics in Chef (our tool)—this is the major advantage of the technique. The Python interpreter simply runs the code under test inside the symbolic x86 VM. An example may help clarify this (see below).
The output is a set of test cases that comprise an automatically-generated test suite for the program under test. These test cases capture all the execution paths (including corner cases & buggy cases) discovered automatically during execution.
Say we want to test the code in the simplejson package.
Traditionally, one would write a suite of unit tests that exercise a predetermined set of input-output pairs. For instance, you'd have to think of what inputs would best cover the package behavior, i.e., valid JSON, invalid JSON, empty strings, slightly malformed JSON, and so on. This is quite tedious and one may very well miss obscure corner cases.
However, with Chef, we write instead so called "symbolic tests", which use Chef's API. Here is a simple example for simplejson, say simplejson_test.py:
import importlib
import simplejson
import sys
from chef import light
class SimpleJSONTest(light.SymbolicTest):
def setUp(self):
pass
def runTest(self):
simplejson.loads(self.getString("input", '\x00'*15))
if __name__ == "__main__":
light.runFromArgs(SimpleJSONTest, arg_list=sys.argv)
This piece of code does several things:
* It encapsulates the test functionality in a test class that derives from light.SymbolicTest (as opposed to Python's own TestCase).
* Instead of defining a particular JSON string to pass to simplejson, it asks the framework to construct one, according to some specs. In this case, it asks for a string that will be referred to as "input", of 15 characters, with a default value of null characters everywhere.
* When the script is executed, the framework instantiates the symbolic test and runs it.
You can run simplejson_test.py in two modes:
1) In symbolic mode, the test runs inside the symbolic virtual machine. The call to SimpleJSONTest.getString(...) returns a special "symbolic string", which taints the returned variable and causes the execution of the interpreter to "fork" (akin to a process fork) everytime a branch depending on the symbolic string is encountered during execution. This works everywhere inside the interpreter -- either for Python-level branches, or for native branches inside C extension modules -- because the symbolic VM runs at x86 level.
Every time the execution forks, the framework generates a new "test case" -- a concrete assignment to the symbolic string that would cause the interpreter to execute precisely the newly discovered path (e.g., the '{ "key": 1}' string).
2) In replay mode, all the test cases generated in symbolic mode can be replayed via the same simplejson_test.py file. In this mode, the SimpleJSONTest.getString(...) call returns one of the concrete input assignments generated and checks that the replayed output matches the output observed in symbolic mode (this is to weed out nondeterminism).
Oh, this is excellent! Thanks a lot!
Cheers,
Stefan