Hi,
(I'd like to fork from a previous thread, "Pre-conditions and
post-conditions", since it got long and we started discussing a couple of
different things. Let's discuss in this thread the implementation of a
library for design-by-contract and how to push it forward to hopefully add
it to the standard library one day.)
For those unfamiliar with contracts and current state of the discussion in
the previous thread, here's a short summary. The discussion started by me
inquiring about the possibility to add design-by-contract concepts into the
core language. The idea was rejected by the participants mainly because
they thought that the merit of the feature does not merit its costs. This
is quite debatable and seems to reflect many a discussion about
design-by-contract in general. Please see the other thread, "Why is
design-by-contract not widely adopted?" if you are interested in that
debate.
We (a colleague of mine and I) decided to implement a library to bring
design-by-contract to Python since we don't believe that the concept will
make it into the core language anytime soon and we needed badly a tool to
facilitate our work with a growing code base.
The library is available at http://github.com/Parquery/icontract. The hope
is to polish it so that the wider community could use it and once the
quality is high enough, make a proposal to add it to the standard Python
libraries. We do need a standard library for contracts, otherwise projects
with *conflicting* contract libraries can not integrate (*e.g., *the
contracts can not be inherited between two different contract libraries).
So far, the most important bits have been implemented in icontract:
- Preconditions, postconditions, class invariants
- Inheritance of the contracts (including strengthening and weakening of
the inherited contracts)
- Informative violation messages (including information about the values
involved in the contract condition)
- Sphinx extension to include contracts in the automatically generated
documentation (sphinx-icontract)
- Linter to statically check that the arguments of the conditions are
correct (pyicontract-lint)
We are successfully using it in our code base and have been quite happy
about the implementation so far.
There is one bit still missing: accessing "old" values in the postcondition
(*i.e., *shallow copies of the values prior to the execution of the
function). This feature is necessary in order to allow us to verify state
transitions.
For example, consider a new dictionary class that has "get" and "put"
methods:
from typing import Optional
from icontract import post
class NovelDict:
def length(self)->int:
...
def get(self, key: str) -> Optional[str]:
...
@post(lambda self, key, value: self.get(key) == value)
@post(lambda self, key: old(self.get(key)) is None and
old(self.length()) + 1 == self.length(),
"length increased with a new key")
@post(lambda self, key: old(self.get(key)) is not None and
old(self.length()) == self.length(),
"length stable with an existing key")
def put(self, key: str, value: str) -> None:
...
How could we possible implement this "old" function?
Here is my suggestion. I'd introduce a decorator "before" that would allow
you to store whatever values in a dictionary object "old" (*i.e. *an object
whose properties correspond to the key/value pairs). The "old" is then
passed to the condition. Here is it in code:
# omitted contracts for brevity
class NovelDict:
def length(self)->int:
...
# omitted contracts for brevity
def get(self, key: str) -> Optional[str]:
...
@before(lambda self, key: {"length": self.length(), "get": self.get(key)})
@post(lambda self, key, value: self.get(key) == value)
@post(lambda self, key, old: old.get is None and old.length + 1 ==
self.length(),
"length increased with a new key")
@post(lambda self, key, old: old.get is not None and old.length ==
self.length(),
"length stable with an existing key")
def put(self, key: str, value: str) -> None:
...
The linter would statically check that all attributes accessed in "old"
have to be defined in the decorator "before" so that attribute errors would
be caught early. The current implementation of the linter is fast enough to
be run at save time so such errors should usually not happen with a
properly set IDE.
"before" decorator would also have "enabled" property, so that you can turn
it off (*e.g., *if you only want to run a postcondition in testing). The
"before" decorators can be stacked so that you can also have a more
fine-grained control when each one of them is running (some during test,
some during test and in production). The linter would enforce that before's
"enabled" is a disjunction of all the "enabled"'s of the corresponding
postconditions where the old value appears.
Is this a sane approach to "old" values? Any alternative approach you would
prefer? What about better naming? Is "before" a confusing name?
Thanks a lot for your thoughts!
Cheers,
Marko
I'm still not sure why all this focus on new syntax or convoluted IDE
enhancements. I presented a very simple utility function that accomplishes
exactly the started goal of DRY in keyword arguments.
Yes, I wrote a first version that was incomplete. And perhaps these 8-9
lines miss some corner case. But the basic goal is really, really easy to
accomplish with existing Python.
>>> import inspect
>>> def reach(name):
... for f in inspect.stack():
... if name in f[0].f_locals:
... return f[0].f_locals[name]
... return None
...
>>> def use(names):
... kws = {}
... for name in names.split():
... kws[name] = reach(name)
... return kws
...
>>> def function(a=11, b=22, c=33, d=44):
... print(a, b, c, d)
...
>>> function(a=77, **use('b d'))
77 None 33 None
>>> def foo():
... a, b, c = 1, 2, 3
... function(a=77, **use('b d'))
...
>>> foo()
77 2 33 None
On Tue, Sep 25, 2018, 6:31 AM Anders Hovmöller <boxed(a)killingar.net> wrote:
> Hi,
>
> I'd like to reopen this discussion if anyone is interested. Some things
> have changed since I wrote my original proposal so I'll first summarize:
>
> 1. People seem to prefer the syntax `foo(=a)` over the syntax I suggested.
> I believe this is even more trivial to implement in CPython than my
> original proposal anyway...
> 2. I have updated my analysis tool:
> https://gist.github.com/boxed/610b2ba73066c96e9781aed7c0c0b25c It will
> now also give you statistics on the number of arguments function calls
> have. I would love to see some statistics for other closed source programs
> you might be working on and how big those code bases are.
> 3. I have made a sort-of implementation with MacroPy:
> https://github.com/boxed/macro-kwargs/blob/master/test.py I think this is
> a dead end, but it was easy to implement and fun to try!
> 4. I have also recently had the idea that a foo=foo type pattern could be
> handled in for example PyCharm as a code folding feature (and maybe as a
> completion feature).
>
> I still think that changing Pythons syntax is the right way to go in the
> long run but with point 4 above one could experience what this feature
> would feel like without running a custom version of Python and without
> changing your code. I admit to a lot of trepidation about wading into
> PyCharms code though, I have tried to do this once before and I gave up.
>
> Any thoughts?
>
> / Anders
> _______________________________________________
> Python-ideas mailing list
> Python-ideas(a)python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
Hmm, I was wrong: there is no reliable way to get the code of a lambda function.
If it was possible to execute all code paths of the function, we could monkey patch the builtins so { } used our own custom set class.
Alternatively, the decorator could also accept a string.
Or maybe we could send a PEP to add the .func_code attribute to lambdas as well as normal functions.
There’s also a technique online where they find the lambda’s source by locating the file the function was defined in and then removing the irrelevant parts, but that just doesn’t sound practical to me.
There’s also MacroPy.
I think the best solution would be to mock the old object and record the operations done to the object, like the other replier gave a PoC of. Proposed syntax
from icontract import post, old
@post(lambda: ...,
key=old.self.key(),
)
Sent from my iPhone
> On Sep 25, 2018, at 1:49 AM, Marko Ristin-Kaufmann <marko.ristin(a)gmail.com> wrote:
>
> Hi James,
> Thanks for the feedback!
>
> I also thought about decompiling the condition to find its AST and figure out what old values are needed. However, this is not so easily done at the moment as all the decompilation libraries I looked at (meta, ucompyle6) are simply struggling to keep up with the development of the python bytecode. In other words, the decompiler needs to be updated with every new version of python which is kind of a loosing race (unless the python devs themselves don't provide this functionality which is not the case as far as I know).
>
> There is macropy (https://github.com/lihaoyi/macropy) which was suggested on the other thread (https://groups.google.com/forum/#!topic/python-ideas/dmXz_7LH4GI) that I'm currently looking at.
>
> Cheers,
> Marko
>
>
>> On Tue, 25 Sep 2018 at 00:35, James Lu <jamtlu(a)gmail.com> wrote:
>> You could disassemble (import dis) the lambda to biew the names of the lambdas.
>>
>> @before(lambda self, key, _, length, get: self.length(), self.get(key))
>>
>> Perhaps you could disassemble the function code and look at all operations or accesses that are done to “old.” and evaluate those expressions before the function runs. Then you could “replace” the expression.
>> @post(lambda self, key, old: old.get is None and old.length + 1 ==
>> self.length())
>>
>> Either the system would grab old.get and old.length or be greedy and grab old.get is None and old.length + 1. It would then replace the old.get and old.length with injects that only respond to is None and +1.
>>
>> Or, a syntax like this
>> @post(lambda self, key, old: [old.get(old.key)] is None and [old.self.length() + 1] ==
>> self.length())
>>
>> Where the stuff inside the brackets is evaluated before the decorated function runs. It would be useful for networking functions or functions that do something ephemeral, where data related to the value being accessed needed for the expression no longer exists after the function.
>>
>> This does conflict with list syntax forever, so maybe either force people to do list((expr,)) or use an alternate syntax like one item set syntax { } or double set syntax {{ }} or double list syntax [[ ]]. Ditto with having to avoid the literals for the normal meaning.
>>
>> You could modify Python to accept any expression for the lambda function and propose that as a PEP. (Right now it’s hardcoded as a dotted name and optionally a single argument list surrounded by parentheses.)
>>
>> I suggest that instead of “@before” it’s “@snapshot” and instead of “old” it’s “snapshot”.
>>
>> Python does have unary plus/minus syntax as well as stream operators (<<, >>) and list slicing syntax and the @ operator and operators & and | if you want to play with syntax. There’s also the line continuation character for crazy lambdas.
>>
>> Personally I prefer
>> @post(lambda self, key, old: {{old.self.get(old.key)}} and {{old.self.length() + 1}} ==
>> self.length())
>>
>> because it’s explicit about what it does (evaluate the expressions within {{ }} before the function runs. I also find it elegant.
>>
>> Alternatively, inside the {{ }} could be a special scope where locals() is all the arguments @pre could’ve received as a dictionary. For either option you can remove the old parameter from the lambda. Example:
>> @post(lambda self, key: {{self.get(key)}} and {{self.length() + 1}} ==
>> self.length())
>>
>> Perhaps the convention should be to write {{ expr }} (with the spaces in between).
>>
>> You’d probably have to use the ast module to inspect it instead of the dis modul. Then find some way to reconstruct the expressions inside the double brackets- perhaps by reconstructing the AST and compiling it to a code object, or perhaps by finding the part of the string the expression is located. dis can give you the code as a string and you can run a carefully crafted regex on it.
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas(a)python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
> Date: Mon, 24 Sep 2018 09:46:16 +0200
> From: Marko Ristin-Kaufmann <marko.ristin(a)gmail.com>
> To: Python-Ideas <python-ideas(a)python.org>
> Subject: Re: [Python-ideas] Why is design-by-contracts not widely
> adopted?
> Message-ID:
> <CAGu4bVB4=Bou+DhoDzayx=i7eQ2Gr8KDDOFONeaB-Qvt8LZo2A(a)mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
[munch]
> Their users would hugely benefit from a more mature
> and standardized contracts library with informative violation messages.
Will respond in another message, because it's a big topic.
> I really don't see how DbC has to do with duck typing (unless you reduce it
> to mere isinstance conditions, which would simply be a straw-man argument)
> -- could you please clarify?
I argue that Design by Contract doesn't make sense for Python and other
dynamically typed, duck typed languages because it's contrary to how the
language, and the programmer, expects to work.
In Python we can write something like:
def foo(x):
x.bar(y)
What's the type of x? What's the type of y? What is the contract of bar?
Don't know, don't care. x, or y, can be an instance, a class, a module, a
proxy for a remote web service. The only "contract" is that object x will
respond to message bar that takes one argument. Object x, do whatever
you want with it.
And that's a feature, not a bug, not bad design. It follows Postel's Law
for Internet protocols of being liberal in what you accept. It follows the
Agile principle of valuing working software over comprehensive doco.
It allows software components to be glued together quickly and easily.
It's a style of programming that has been successful for many years,
not just in Python but also in Lisp and Smalltalk and Perl and JavaScript.
It works.
Not for everything. If I were writing the avionics control routines for a
helicopter gas turbine, I'd use formal notation and static type checking
and preconditions and whatnot. But I wouldn't be using Python either.
> As soon as you need to document your code, and
> this is what most modules have to do in teams of more than one person
> (especially so if you are developing a library for a wider audience), you
> need to write down the contracts. Please see above where I tried to
> explained that 2-5) are inferior approaches to documenting contracts
> compared to 1).
You left off option 6), plain text. Comments. Docstrings. README files.
Web pages. Books. In my experience, this is what most people consider
documentation. A good book, a good blog post, can explain more about
how a library works and what the implementation requirements and
restrictions are than formal contract notation. In particular, contracts in
Eiffel don't explain *why* they're there.
As for 4) reading the code, why not? "Use the source, Luke" is now a
programming cliche because it works. It's particularly appropriate for
Python packages which are usually distributed in source form and, as
you yourself noted, easy to read.
--
cheers,
Hugh Fisher
Hi Marko,
I think there are several ways to approach this problem, though am not
weighing in on whether DbC is a good thing in Python. I wrote a simple
implementation of DbC which is currently a run-time checker. You could,
with the appropriate tooling, validate statically too (as with all
approaches). In my approach, I use a “proxy” object to allow the contract
code to be defined at function definition time. It does mean that some
things are not as pretty as one would like - anything that cannot be hooked
into with magic methods i.e isinstance, but I think this is acceptable as
it makes features like old easier. Also, one hopes that it encourages
simpler contract checks as a side-effect. Feel free to take a look -
https://github.com/agoose77/pyffel
It is by no means well written, but a fun PoC nonetheless.
Regards,
Angus
Have you looked at the built-in AST module, ast? https://docs.python.org/3/library/ast.html
I don’t see anything preventing you from walking the AST Python itself can give you- you’d look for two Set AST nodes if we were to do {{ }}.
There’s also the parser built-in module. You can use it if you first use dis.code_info to get the source then re-parse it. It helps with parse trees. Parse trees are generated before the AST I think. You’d use the parser module’s ST objects with the token module’s constants, for example token.LBRACE or token.RBRACE.
Have you looked at the built-in dis module? You can use dis.code_info(obj) to get the string of the function. Then you could look for your specified syntax with regex and recompile that with the ast module.
Sent from my iPhone
> On Sep 25, 2018, at 1:49 AM, Marko Ristin-Kaufmann <marko.ristin(a)gmail.com> wrote:
>
> Hi James,
> Thanks for the feedback!
>
> I also thought about decompiling the condition to find its AST and figure out what old values are needed. However, this is not so easily done at the moment as all the decompilation libraries I looked at (meta, ucompyle6) are simply struggling to keep up with the development of the python bytecode. In other words, the decompiler needs to be updated with every new version of python which is kind of a loosing race (unless the python devs themselves don't provide this functionality which is not the case as far as I know).
>
> There is macropy (https://github.com/lihaoyi/macropy) which was suggested on the other thread (https://groups.google.com/forum/#!topic/python-ideas/dmXz_7LH4GI) that I'm currently looking at.
>
> Cheers,
> Marko
>
>
>> On Tue, 25 Sep 2018 at 00:35, James Lu <jamtlu(a)gmail.com> wrote:
>> You could disassemble (import dis) the lambda to biew the names of the lambdas.
>>
>> @before(lambda self, key, _, length, get: self.length(), self.get(key))
>>
>> Perhaps you could disassemble the function code and look at all operations or accesses that are done to “old.” and evaluate those expressions before the function runs. Then you could “replace” the expression.
>> @post(lambda self, key, old: old.get is None and old.length + 1 ==
>> self.length())
>>
>> Either the system would grab old.get and old.length or be greedy and grab old.get is None and old.length + 1. It would then replace the old.get and old.length with injects that only respond to is None and +1.
>>
>> Or, a syntax like this
>> @post(lambda self, key, old: [old.get(old.key)] is None and [old.self.length() + 1] ==
>> self.length())
>>
>> Where the stuff inside the brackets is evaluated before the decorated function runs. It would be useful for networking functions or functions that do something ephemeral, where data related to the value being accessed needed for the expression no longer exists after the function.
>>
>> This does conflict with list syntax forever, so maybe either force people to do list((expr,)) or use an alternate syntax like one item set syntax { } or double set syntax {{ }} or double list syntax [[ ]]. Ditto with having to avoid the literals for the normal meaning.
>>
>> You could modify Python to accept any expression for the lambda function and propose that as a PEP. (Right now it’s hardcoded as a dotted name and optionally a single argument list surrounded by parentheses.)
>>
>> I suggest that instead of “@before” it’s “@snapshot” and instead of “old” it’s “snapshot”.
>>
>> Python does have unary plus/minus syntax as well as stream operators (<<, >>) and list slicing syntax and the @ operator and operators & and | if you want to play with syntax. There’s also the line continuation character for crazy lambdas.
>>
>> Personally I prefer
>> @post(lambda self, key, old: {{old.self.get(old.key)}} and {{old.self.length() + 1}} ==
>> self.length())
>>
>> because it’s explicit about what it does (evaluate the expressions within {{ }} before the function runs. I also find it elegant.
>>
>> Alternatively, inside the {{ }} could be a special scope where locals() is all the arguments @pre could’ve received as a dictionary. For either option you can remove the old parameter from the lambda. Example:
>> @post(lambda self, key: {{self.get(key)}} and {{self.length() + 1}} ==
>> self.length())
>>
>> Perhaps the convention should be to write {{ expr }} (with the spaces in between).
>>
>> You’d probably have to use the ast module to inspect it instead of the dis modul. Then find some way to reconstruct the expressions inside the double brackets- perhaps by reconstructing the AST and compiling it to a code object, or perhaps by finding the part of the string the expression is located. dis can give you the code as a string and you can run a carefully crafted regex on it.
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas(a)python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
I have a working implementation for a new syntax which would make using keyword arguments a lot nicer. Wouldn't it be awesome if instead of:
foo(a=a, b=b, c=c, d=3, e=e)
we could just write:
foo(*, a, b, c, d=3, e)
and it would mean the exact same thing? This would not just be shorter but would create an incentive for consistent naming across the code base.
So the idea is to generalize the * keyword only marker from function to also have the same meaning at the call site: everything after * is a kwarg. With this feature we can now simplify keyword arguments making them more readable and concise. (This syntax does not conflict with existing Python code.)
The full PEP-style suggestion is here: https://gist.github.com/boxed/f72221e7e77370be3e5703087c1ba54d
I have also written an analysis tool you can use on your code base to see what kind of impact this suggestion might have. It's available at https://gist.github.com/boxed/610b2ba73066c96e9781aed7c0c0b25c . The results for django and twisted are posted as comments to the gist.
We've run this on our two big code bases at work (both around 250kloc excluding comments and blank lines). The results show that ~30% of all arguments would benefit from this syntax.
Me and my colleague Johan Lübcke have also written an implementation that is available at: https://github.com/boxed/cpython
/ Anders Hovmöller
> Which features of the TC39 committee's ECMAscript (ES) language governance
> model would be helpful to incorporate into the Python language governance
> model?
Having “beta” or “alpha” editions of features, special versions of the interpreter people can test out to see if they prefer the version with the new feature.
To prevent splintering, the main releases would only support the main feature set. In a worst case scenario, people can compile incompatible code to .pyc before running it.
Perhaps it’s because fewer Python functions involve transitioning between states. Web development and statistics don’t involve many state transition. State transitions are where I think I would find it useful to write contracts out explicitly.
You could disassemble (import dis) the lambda to biew the names of the lambdas.
@before(lambda self, key, _, length, get: self.length(), self.get(key))
Perhaps you could disassemble the function code and look at all operations or accesses that are done to “old.” and evaluate those expressions before the function runs. Then you could “replace” the expression.
@post(lambda self, key, old: old.get is None and old.length + 1 ==
self.length())
Either the system would grab old.get and old.length or be greedy and grab old.get is None and old.length + 1. It would then replace the old.get and old.length with injects that only respond to is None and +1.
Or, a syntax like this
@post(lambda self, key, old: [old.get(old.key)] is None and [old.self.length() + 1] ==
self.length())
Where the stuff inside the brackets is evaluated before the decorated function runs. It would be useful for networking functions or functions that do something ephemeral, where data related to the value being accessed needed for the expression no longer exists after the function.
This does conflict with list syntax forever, so maybe either force people to do list((expr,)) or use an alternate syntax like one item set syntax { } or double set syntax {{ }} or double list syntax [[ ]]. Ditto with having to avoid the literals for the normal meaning.
You could modify Python to accept any expression for the lambda function and propose that as a PEP. (Right now it’s hardcoded as a dotted name and optionally a single argument list surrounded by parentheses.)
I suggest that instead of “@before” it’s “@snapshot” and instead of “old” it’s “snapshot”.
Python does have unary plus/minus syntax as well as stream operators (<<, >>) and list slicing syntax and the @ operator and operators & and | if you want to play with syntax. There’s also the line continuation character for crazy lambdas.
Personally I prefer
@post(lambda self, key, old: {{old.self.get(old.key)}} and {{old.self.length() + 1}} ==
self.length())
because it’s explicit about what it does (evaluate the expressions within {{ }} before the function runs. I also find it elegant.
Alternatively, inside the {{ }} could be a special scope where locals() is all the arguments @pre could’ve received as a dictionary. For either option you can remove the old parameter from the lambda. Example:
@post(lambda self, key: {{self.get(key)}} and {{self.length() + 1}} ==
self.length())
Perhaps the convention should be to write {{ expr }} (with the spaces in between).
You’d probably have to use the ast module to inspect it instead of the dis modul. Then find some way to reconstruct the expressions inside the double brackets- perhaps by reconstructing the AST and compiling it to a code object, or perhaps by finding the part of the string the expression is located. dis can give you the code as a string and you can run a carefully crafted regex on it.