Hello all,
I'd like to start a discussion (but hopefully a brief one!) about using
property testing techniques in Twisted.
For those who aren't familiar with them, a property test is a kind of unit
test written such that it can assert a certain invariant holds over a range
of inputs. This is in contrast with more conventional unit tests in which
a single test asserts a certain invariant holds for a single input and
typically many such tests are written to cover the necessary range of
inputs.
Twisted mostly uses conventional unit tests though it does include a few
property tests written in an ad hoc way. If you search for loops in the
test suite you quickly come across them. For example,
twisted.test.test_amp.ParsingTests includes several. test_booleanValues
does not use a loop but it is a property test. test_ParsingRoundTrip does
use a loop and is also a property test. In these cases, the developer knew
they wanted to exercise the implementation against more than one input but
they were limited in how exhaustive they could be by the tools available.
There is a popular library for property testing in Python called
Hypothesis. I've pushed some changes to *a branch*
<https://github.com/twisted/twisted/compare/26c3c04e55d0b25532709e1e1dc5d143…>
to demonstrate usage (note: *not* currently for review) of some of the
basic APIs and the tests that result:
- The given decorator marks a test as a property test. The consequence
of this is usually that the test method runs more than once. The
"strategies" passed to given determine how it is called.
- Strategies like tuples, lists, integers specify what values are passed
to the decorated function. Each will correspond to a parameter the test
method accepts. A simple strategy like integers will result in int
values being passed. Many strategies can be composed. A strategy like
tuples will result in tuple values being passed, with the elements of
the value determined by the strategies passed to the tuples strategy.
- Helper functions like assume support dealing with corner cases related
to invalid inputs which are tricky to avoid in a strategy. If it is hard
to constructively define exactly the right set of inputs, you can define a
strategy that finds "too many" and filter out the bad ones afterwards.
The test is then written to make an assertion that is correct with respect
to the specific values the test is called with. For complex code, it can
take a while to get used to writing tests in this style but for simpler
code it's often only a matter of using the strategy-provided values instead
of some hard-coded values.
Once using given, the test method actually runs many times, each time with
different values "drawn" from the given strategies. When the test fails,
Hypothesis tries to "shrink" the values - it simplifies them as far as it
can while still causing the test to fail. When it has simplified them as
much as it can without making the test pass, those simpler values are
reported. This means that Hypothesis can find complex inputs that break
the code under test but when *simple* values will do, those are the ones a
developer gets to work with.
There are many other benefits to this style of testing, such as reduced
effort required to write comprehensive tests and improved code re-use in
the test suite. I won't describe them further here but you can learn more
at the Hypothesis website - https://hypothesis.works/.
I think property testing is a valuable tool in the testing toolbox and
Twisted should consider adopting Hypothesis to employ it for some tests.
Thoughts?
Jean-Paul
On behalf of the Twisted contributors, I announce the release candidate
of Twisted 22.8.0
This is the first release after we migrated from Trac Tickets to GitHub Issues.
In this release, the API docs were updated to a newer pydoctor version
and the RTD theme is enabled.
The old Trac twistedmatrix.com website is now offline.
Redirections were implemented on a best effort basis :)
If you find any broken link you can report this via GitHub Issues
https://github.com/twisted/twisted/issues/new/choose
`trial --jobs=N --exitfirst` is now supported, for a much better
experience when running parallel tests.
If you haven't try trial parallel test run yet, give it a try.
On local dev computers with many CPUs you get a huge performance
improvement (assuming your tests are CPU bound )
Python 3.6 is no longer supported in this version.
The release and NEWS file is available for review at
https://github.com/twisted/twisted/pull/11622/files
Release candidate documentation is available at
https://twisted--11622.org.readthedocs.build/en/11622/
Wheels for the release candidate are available on PyPI
https://pypi.org/project/Twisted/22.8.0rc1/
python -m pip install Twisted==22.8.0rc1
Please test it and report any issues.
If nothing comes up in one week,
22.8.0 will be released based on the latest release candidate.
Many thanks to everyone who had a part in Twisted development,
the supporters of the Twisted Software Foundation,
the developers, and all the people testing and building great things
with Twisted!
Slava Ukraini!
--
Adi Roiban
