Hello all,

I'd like to start a discussion (but hopefully a brief one!) about using property testing techniques in Twisted.

For those who aren't familiar with them, a property test is a kind of unit test written such that it can assert a certain invariant holds over a range of inputs. This is in contrast with more conventional unit tests in which a single test asserts a certain invariant holds for a single input and typically many such tests are written to cover the necessary range of inputs.

Twisted mostly uses conventional unit tests though it does include a few property tests written in an ad hoc way. If you search for loops in the test suite you quickly come across them. For example, twisted.test.test_amp.ParsingTests includes several. test_booleanValues does not use a loop but it is a property test. test_ParsingRoundTrip does use a loop and is also a property test. In these cases, the developer knew they wanted to exercise the implementation against more than one input but they were limited in how exhaustive they could be by the tools available.

There is a popular library for property testing in Python called Hypothesis. I've pushed some changes to a branch to demonstrate usage (note: not currently for review) of some of the basic APIs and the tests that result:

The given decorator marks a test as a property test. The consequence of this is usually that the test method runs more than once. The "strategies" passed to given determine how it is called.
Strategies like tuples, lists, integers specify what values are passed to the decorated function. Each will correspond to a parameter the test method accepts. A simple strategy like integers will result in int values being passed. Many strategies can be composed. A strategy like tuples will result in tuple values being passed, with the elements of the value determined by the strategies passed to the tuples strategy.
Helper functions like assume support dealing with corner cases related to invalid inputs which are tricky to avoid in a strategy. If it is hard to constructively define exactly the right set of inputs, you can define a strategy that finds "too many" and filter out the bad ones afterwards.

The test is then written to make an assertion that is correct with respect to the specific values the test is called with. For complex code, it can take a while to get used to writing tests in this style but for simpler code it's often only a matter of using the strategy-provided values instead of some hard-coded values.

Once using given, the test method actually runs many times, each time with different values "drawn" from the given strategies. When the test fails, Hypothesis tries to "shrink" the values - it simplifies them as far as it can while still causing the test to fail. When it has simplified them as much as it can without making the test pass, those simpler values are reported. This means that Hypothesis can find complex inputs that break the code under test but when simple values will do, those are the ones a developer gets to work with.

There are many other benefits to this style of testing, such as reduced effort required to write comprehensive tests and improved code re-use in the test suite. I won't describe them further here but you can learn more at the Hypothesis website - https://hypothesis.works/.

I think property testing is a valuable tool in the testing toolbox and Twisted should consider adopting Hypothesis to employ it for some tests.

Thoughts?

Jean-Paul