[Python-ideas] [Brainstorm] Testing with Documented ABCs

[Marko Ristin-Kaufmann]

Hi Abe,

I agree. That's why I prefaced this topic with [Brainstorm]. I want to
> explore the solution space to this problem and discuss some of the pros and
> cons of different ideas,  *not* proceed straight to action.


You are right. Please apologize, I was so primed by the discussions we had
in October 2019 that I didn't pay enough attention to "branstorm" in the
subject.

Fuzz testing and stateful testing like that provided by hypothesis might
> work together with contracts in an interesting way.
>

You might want to look at the literature on automatic test generation. A
possible entry point could be:
https://www.research-collection.ethz.ch/handle/20.500.11850/69581

If I had time available, I would start with a tool that analyses a given
module and automatically generates code for the Hypothesis test cases. The
tool needs to select functions which accept primitive data types and for
each one of them translates their contracts into Hypothesis code. If
contracts are not trivially translatable to Hypothesis, the function is
ignored. For readability and speed of development (of the code under test,
not of the tool), I would prefer this tool *not *to be dynamic so that the
developer herself needs to re-run it if the function signatures changed.

The ingredients for such a tool are all there with icontract (similar to
sphinx-icontract, you import the module and analyze its functions; you can
copy/past parts of sphinx-icontract implementation for parsing and listing
the AST of the contracts). (If you'd like to continue discussing this
topic, let's create an issue on icontract github page or switch to private
correspondence in order not to spam this mail list).

There seems like a lot of opportunity for the re-use of contracts, so maybe
> we should consider a mechanism to facilitate that.
>

This was the case for the requests library. @James Lu <jamtlu at gmail.com>
was looking into it -- a lot of functions had very similar contracts.
However, in our code base at the company (including the open-sourced
libraries), there was not a single case where we thought that contracts
re-use would be beneficial. Either it would have hurt the readability and
introduce unnecessary couplings (when the contracts were trivial) or it
made sense to encapsulate more complex contracts in a separate function.


>> *Multiple predicates per decorator. *
>>
> I suppose it may be difficult to implement a clean, *backwards-compatible*
> solution, but yes; going through the arguments in a sequence would be my
> naive solution. Each entry has an optional description, a callable, and an
> optional tag or level to enable toggling (I would follow a simple model
> such as logging levels) *in that order*.
>

I found that to be too error-prone in a larger code base, but that is my
very subjective opinion. Maybe you could make an example?

but without new syntax; each step between icontracts and an Eiffel-esque
> platonic ideal would require significant hackery with diminishing returns
> on investment.
>

I agree. There are also issues with core python interpreter which I expect
to remain open for a long time (see the issues related to retrieving code
text of lambda functions and decorators and tweaking dynamically the
behavior of help(.) for functions).

Cheers,
Marko

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20181207/ef0b2db7/attachment.html>