[Python-ideas] [Brainstorm] Testing with Documented ABCs

Abe Dillon abedillon at gmail.com
Wed Dec 5 21:43:44 EST 2018


[Marko Ristin-Kaufmann]
>
> What we do need at this moment, IMO, is a broad practical experience of
> using contracts in Python. Once you make a change to the language, it's
> impossible to undo. In contrast to what has been suggested in the previous
> discussions (including my own voiced opinions), I actually now don't think
> that introducing a language change would be beneficial *at this precise
> moment*.


I agree. That's why I prefaced this topic with [Brainstorm]. I want to
explore the solution space to this problem and discuss some of the pros and
cons of different ideas,  *not* proceed straight to action. I also wanted
to bring three thoughts to the table:

   1. Fuzz testing and stateful testing like that provided by hypothesis
   might work together with contracts in an interesting way.
   2. Tying tests/contracts to the bits of documentation that they validate
   is a great way to keep documentation in sync with code, but doctest does it
   a bit "backwards".
   Like in icontract-sphinx (or even this) it's better to construct
   documentation (partially) from test code than to write test code within
   documentation.
   In general, I find the relationship between documentation, testing, and
   type-checking interesting. The problems they each address seem to overlap
   quite a bit.
   3. There seems like a lot of opportunity for the re-use of contracts, so
   maybe we should consider a mechanism to facilitate that.


[Marko Ristin-Kaufmann]

> I'd prefer to hear from people who actually use contracts in their
> professional Python programming -- apart from the noisy syntax, how was the
> experience? Did it help you catch bugs (and how many)? Were there big
> problems with maintainability? Could you easily refactor? What were the
> limits of the contracts you encountered? What kind of snapshot mechanism do
> we need? How did you deal with multi-threading? And so on.


That's a good point. I would argue that the concept of contracts isn't new,
so there should be at least a few cases that we can draw on where others
have tread before us (which you've obviously done to a large degree).
That's not to belittle the work you've done on icontracts. It's a great
tool for the reasons you describe.

[Marko Ristin-Kaufmann]

> *Multiple predicates per decorator. *The problem is that you can not deal
> with toggling/describing individual contracts easily. While you can hack
> your way through it (considering the arguments in the sequence, for
> example), we found it clearer to have separate decorators. Moreover,
> tracebacks are much easier to read, which is important when you debug a
> program.


I suppose it may be difficult to implement a clean, *backwards-compatible*
solution, but yes; going through the arguments in a sequence would be my
naive solution. Each entry has an optional description, a callable, and an
optional tag or level to enable toggling (I would follow a simple model
such as logging levels) *in that order*. It makes sense that the text
description come first because that's the most relevant to a reader (like a
doc-string), then the corresponding code, then the toggling flag which will
often be an optimization detail which generally fall behind code
correctness in priority. It may be less straight-forward to parse, but I
wouldn't call it a "hack".

I guess I'm not sure what to say about tracebacks being hard to read.

[Marko Ristin-Kaufmann]

> *Practicality of decorators. *We have retrospective meetings at the
> company and I frequently survey the opinions related to the contracts
> (explicitly asking about the readability and maintainability) -- so far
> nobody had any difficulties and nobody was bothered by the noisy syntax.


That's fair enough. I think the implementation you've come up with is
pretty close to optimally concise given the tools at your disposal. I think
something like Eiffel is a good goal for Python to eventually shoot for,
but without new syntax; each step between icontracts and an Eiffel-esque
platonic ideal would require significant hackery with diminishing returns
on investment.

On Thu, Nov 29, 2018 at 1:05 AM Marko Ristin-Kaufmann <
marko.ristin at gmail.com> wrote:

> Hi Abe,
> Thanks for your suggestions! We actually already considered the two
> alternatives you propose.
>
> *Multiple predicates per decorator. *The problem is that you can not deal
> with toggling/describing individual contracts easily. While you can hack
> your way through it (considering the arguments in the sequence, for
> example), we found it clearer to have separate decorators. Moreover,
> tracebacks are much easier to read, which is important when you debug a
> program.
>
> *AST magic. *The problem with any approach based on parsing (be it
> parsing the code or the description) is that parsing is slow so you end up
> spending a lot of cycles on contracts which might not be enabled (many
> contracts are applied only in the testing environment, not int he
> production). Hence you must have an approach that offers practically zero
> overhead cost to importing a module when its contracts are turned off.
>
> Decoding byte-code does not work as current decoding libraries can not
> keep up with the changes in the language and the compiler hence they are
> always lagging behind.
>
> *Practicality of decorators. *We have retrospective meetings at the
> company and I frequently survey the opinions related to the contracts
> (explicitly asking about the readability and maintainability) -- so far
> nobody had any difficulties and nobody was bothered by the noisy syntax.
> The decorator syntax is simply not beautiful, no discussion about that. But
> when it comes to maintenance,  there's a linter included (
> https://github.com/Parquery/pyicontract-lint), and if you want contracts
> rendered in an appealing way, there's a documentation tool for sphinx (
> https://github.com/Parquery/sphinx-icontract). The linter facilitates the
> maintainability a lot and sphinx tool gives you nice documentation for a
> library so that you don't even have to look into the source code that often
> if you don't want to.
>
> We need to be careful not to mistake issues of aesthetics for practical
> issues. Something might not be beautiful, but can be useful unless it's
> unreadable.
>
> *Conclusion. *What we do need at this moment, IMO, is a broad practical
> experience of using contracts in Python. Once you make a change to the
> language, it's impossible to undo. In contrast to what has been suggested
> in the previous discussions (including my own voiced opinions), I actually
> now don't think that introducing a language change would be beneficial *at
> this precise moment*. We don't know what the use cases are, and there is
> no practical experience to base the language change on.
>
> I'd prefer to hear from people who actually use contracts in their
> professional Python programming -- apart from the noisy syntax, how was the
> experience? Did it help you catch bugs (and how many)? Were there big
> problems with maintainability? Could you easily refactor? What were the
> limits of the contracts you encountered? What kind of snapshot mechanism do
> we need? How did you deal with multi-threading? And so on.
>
> icontract library is already practically usable and, if you don't use
> inheritance, dpcontracts is usable as well.  I would encourage everybody to
> try out programming with contracts using an existing library and just hold
> their nose when writing the noisy syntax. Once we unearthed deeper problems
> related to contracts, I think it will be much easier and much more
> convincing to write a proposal for introducing contracts in the core
> language. If I had to write a proposal right now, it would be only based on
> the experience of writing a humble 100K code base by a team of 5-10 people.
> Not very convincing.
>
>
> Cheers,
> Marko
>
> On Thu, 29 Nov 2018 at 02:26, Abe Dillon <abedillon at gmail.com> wrote:
>
>> Marko, I have a few thoughts that might improve icontract.
>> First, multiple clauses per decorator:
>>
>> @pre(
>>     *lambda* x: x >= 0,
>>     *lambda* y: y >= 0,
>>     *lambda* width: width >= 0,
>>     *lambda* height: height >= 0,
>>     *lambda* x, width, img: x + width <= width_of(img),
>>     *lambda* y, height, img: y + height <= height_of(img))
>> @post(
>>     *lambda* self: (self.x, self.y) in self,
>>     *lambda* self: (self.x+self.width-1, self.y+self.height-1) in self,
>>     *lambda* self: (self.x+self.width, self.y+self.height) not in self)
>> *def* __init__(self, img: np.ndarray, x: int, y: int, width: int,
>> height: int) -> None:
>>     self.img = img[y : y+height, x : x+width].copy()
>>     self.x = x
>>     self.y = y
>>     self.width = width
>>     self.height = height
>>
>> *def* __contains__(self, pt: Tuple[int, int]) -> bool:
>>     x, y = pt
>>     return (self.x <= x < self.x + self.width) and (self.y <= y < self.y +
>> self.height)
>>
>>
>> You might be able to get away with some magic by decorating a method just
>> to flag it as using contracts:
>>
>>
>> @contract  # <- does byte-code and/or AST voodoo
>> *def* __init__(self, img: np.ndarray, x: int, y: int, width: int,
>> height: int) -> None:
>>     pre(x >= 0,
>>         y >= 0,
>>         width >= 0,
>>         height >= 0,
>>         x + width <= width_of(img),
>>         y + height <= height_of(img))
>>
>>     # this would probably be declared at the class level
>>     inv(*lambda* self: (self.x, self.y) in self,
>>         *lambda* self: (self.x+self.width-1, self.y+self.height-1) in
>> self,
>>         *lambda* self: (self.x+self.width, self.y+self.height) not in
>> self)
>>
>>     self.img = img[y : y+height, x : x+width].copy()
>>     self.x = x
>>     self.y = y
>>     self.width = width
>>     self.height = height
>>
>> That might be super tricky to implement, but it saves you some lambda
>> noise. Also, I saw a forked thread in which you were considering some sort
>> of transpiler  with similar syntax to the above example. That also works.
>> Another thing to consider is that the role of descriptors
>> <https://www.smallsurething.com/python-descriptors-made-simple/>
>> overlaps some with the role of invariants. I don't know what to do with
>> that knowledge, but it seems like it might be useful.
>>
>> Anyway, I hope those half-baked thoughts have *some* value...
>>
>> On Wed, Nov 28, 2018 at 1:12 AM Marko Ristin-Kaufmann <
>> marko.ristin at gmail.com> wrote:
>>
>>> Hi Abe,
>>>
>>> I've been pulling a lot of ideas from the recent discussion on design by
>>>> contract (DBC), the elegance and drawbacks
>>>> <https://bemusement.org/doctests-arent-code> of doctests
>>>> <https://docs.python.org/3/library/doctest.html>, and the amazing talk
>>>> <https://www.youtube.com/watch?v=MYucYon2-lk> given by Hillel Wayne at
>>>> this year's PyCon entitled "Beyond Unit Tests: Taking your Tests to the
>>>> Next Level".
>>>>
>>>
>>> Have you looked at the recent discussions regarding design-by-contract
>>> on this list (
>>> https://groups.google.com/forum/m/#!topic/python-ideas/JtMgpSyODTU
>>> and the following forked threads)?
>>>
>>> You might want to have a look at static checking techniques such as
>>> abstract interpretation. I hope to be able to work on such a tool for
>>> Python in some two years from now. We can stay in touch if you are
>>> interested.
>>>
>>> Re decorators: to my own surprise, using decorators in a larger code
>>> base is completely practical including the  readability and maintenance of
>>> the code. It's neither that ugly nor problematic as it might seem at first
>>> look.
>>>
>>> We use our https://github.com/Parquery/icontract at the company. Most
>>> of the design choices come from practical issues we faced -- so you might
>>> want to read the doc even if you don't plant to use the library.
>>>
>>> Some of the aspects we still haven't figured out are: how to approach
>>> multi-threading (locking around the whole function with an additional
>>> decorator?) and granularity of contract switches (right now we use
>>> always/optimized, production/non-optimized and teating/slow, but it seems
>>> that a larger system requires finer categories).
>>>
>>> Cheers Marko
>>>
>>>
>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20181205/f3e6d3cb/attachment-0001.html>


More information about the Python-ideas mailing list