Contracts in python -- a report & next steps
Hi, I would like to give you a short report on the development of icontract library following the discussion about the introduction of contracts into Python (e.g., see [1, 2, 3, 4]). *Features* The functionality of icontract library is basically on par with Eiffel, Cobra and other languages supporting contracts except for loop invariants and less readable syntax. The features include: * "require", "ensure" and "invariant" decorators to define the contracts * "snapshot" decorator to capture the values prior to the function invocation to allow postconditions to verify the state transitions * inheritance of the contracts (with strengthening/weakening) * tracing of the argument values on contract violation * informative violation messages automatically parsed from the code of the condition function * The user can specify custom errors to be raised on a specific contract violation * individual toggling of the contracts * linter to check the contract arguments * sphinx extension to render the contracts automatically in the documentation (including a sophisticated matching of logical implications) We covered all the use cases we could find in our code base (at Parquery AG) such as: * functions * instance methods * class and static methods * property getters, setters and deleters * slot wrappers *Roadblocks* During the development, the following roadblocks were encountered: * We wanted to include the contracts in the output of help(). Unfortunately, help() renders the __doc__ of the class and not of the instance. For functions, this is the class "function" which you can not inherit from. See [5] for more details. * We need to inspect the source code of the condition and error lambdas to generate the violation message and infer the error type in the documentation, respectively. inspect.getsource(.) is broken on lambdas defined in decorators in Python 3.5.2+ (see [6]). We circumvented this bug by using inspect.findsource(.), inspect.getsourcefile(.) and examining the local source code of the lambda by searching for other decorators above and other decorators and a function or class definition below. The decorator code is parsed and then we match the condition and error arguments in the AST of the decorator. This is brittle as it prevents us from having partial definitions of contract functions or from sharing the contracts among functions. Here is a short code snippet to demonstrate where the current implementation fails: import icontract require_x_positive = icontract.require( lambda x: x > 0, error=lambda: ValueError("x must be positive")) @require_x_positive def some_func(x: int) -> None: pass However, we haven't faced a situation in the code base where we would do something like the above, so I am unsure whether this is a big issue. As long as decorators are directly applied to functions and classes, everything worked fine on our code base. *Our Experience* We have been using icontract in our team for our production code base (~100K LOC) as well as for a couple of open source projects (each <10K LOC) since August 1st 2018 (~ 3 months). In contrast to points raised during the discussions in [1, 2, 3, 4], we did not have issues with readability and modifying contracts. Thus far, we have not encountered problems with variable renames and major code refactorings. We use Pycharm, so it remains open whether this also applies to development environments such as emacs and vim. We confirm that contracts help us improve the documentation, keep the documentation up-to-date, better reason about the function's input and output and catch bugs early and more easily during the unit and integration tests. *Status* The library interface is currently frozen after the version bump to 2.0.0 and is not expected to change in the next six months. All the reported bugs have been fixed and no bug fixes are pending. *Next Steps?* I personally doubt that we are enough people to form a party to push for a change in the language. A standardized library seems to me like a realizable compromise given the state of the discussion on this mail list. Before we organize a collective to write a proposal to standardize the library, I would suggest that a couple of other interested teams adopt icontract, apply it to their code bases and report their experiences on this mail list. I hope that 2-3 reports would be insightful enough to either convince other people that contracts in python are worth standardizing (and highlight missing features yet to be implemented) or provide solid material to discard the endeavor at the current moment. In the meanwhile, it would be of great help if somebody could vet the documentation and the code of icontract. *Authors* The library was mainly written by me (with some help of my colleague Adam Radomski). We discussed the features within the dev team at Parquery (Zurich, Switzerland) as well as in email correspondence with James Lu. [1] https://groups.google.com/forum/#!topic/python-ideas/mvzLsdwGuww [2] https://groups.google.com/forum/#!topic/python-ideas/JtMgpSyODTU [3] https://groups.google.com/forum/#!topic/python-ideas/mvzLsdwGuww [4] https://groups.google.com/forum/#!topic/python-ideas/dmXz_7LH4GI%5B1-25%5D [5] https://groups.google.com/forum/#!topic/python-ideas/c9ntrVuh6WE [6] https://bugs.python.org/issue21217
P.S. Here is the link to the github repo: https://github.com/Parquery/icontract On Wed, 24 Oct 2018 at 09:40, Marko Ristin-Kaufmann <marko.ristin@gmail.com> wrote:
On Wed, Oct 24, 2018 at 6:41 PM Marko Ristin-Kaufmann <marko.ristin@gmail.com> wrote:
For the sake of those of us who REALLY don't feel like diving back into the extensive threads on this subject, can you please summarize the benefits of having this in the stdlib rather than as a third-party library? Also - have you benchmarked the performance cost of adding contracts? Particularly: if you're planning to contractify the stdlib, what is the impact on startup performance? ChrisA
Hi Chris, For the sake of those of us who REALLY don't feel like diving back
Certainly. We need a standard approach to contracts as opposed to third-party libraries for the following technical reasons: * There is no dependency hell if two packages use different versions of the same contract library. * Two packages need to inherit each other's contracts (*e.g.*, one package defines a class which inherits a base class from a different package and hence needs to inherit its contracts as well). * Third-party libraries for testing and static verification need a standard approach to contracts in order to be usable. Otherwise, the authors of these libraries (*e.g. *Hypothesis) need to support multiple contrat libraries (if they could ever bother to support multiple of them). There are many more, but these are the reasons that I find critical.
I made some preliminary benchmarks. The source code is available at: https://github.com/Parquery/icontract/tree/master/benchmarks/startup Average over ten runs in milliseconds on my machine (Intel(R) Core(TM) i7-4700MQ CPU @ 2.40GHz, 8 cores, 4 GB RAM): Duration to import the module functions_100_with_no_contract : 795.59 ± 10.47 Duration to import the module functions_100_with_1_contract : 919.53 ± 61.22 Duration to import the module functions_100_with_5_contracts : 1075.81 ± 59.87 Duration to import the module functions_100_with_10_contracts : 1290.22 ± 90.04 Duration to import the module functions_100_with_1_disabled_contract : 833.60 ± 32.07 Duration to import the module functions_100_with_5_disabled_contracts : 851.31 ± 66.93 Duration to import the module functions_100_with_10_disabled_contracts : 897.90 ± 143.02 Duration to import the module classes_100_with_no_invariant : 843.61 ± 28.21 Duration to import the module classes_100_with_1_invariant : 3409.71 ± 95.78 Duration to import the module classes_100_with_5_invariants : 4005.93 ± 131.97 Duration to import the module classes_100_with_10_invariants : 4801.82 ± 157.56 Duration to import the module classes_100_with_1_disabled_invariant : 885.88 ± 44.24 Duration to import the module classes_100_with_5_disabled_invariants : 912.53 ± 101.91 Duration to import the module classes_100_with_10_disabled_invariants : 963.77 ± 161.76 Please let me know if these are the benchmarks you had in mind or you would like to see something else. I have neither optimized the code for speed nor investigated why the performance of class invariants differs so much from the functions. The important bit for me was that disabling contracts basically had no impact on import time. Cheers, Marko
On Wed, Oct 24, 2018 at 9:08 PM Marko Ristin-Kaufmann <marko.ristin@gmail.com> wrote:
Wouldn't version differences happen just as much whether it's in the stdlib or a third-party library? The contracts API has to be built to cope with this, regardless. But having it in the stdlib would at least mean that you don't have to worry about incompatible DIFFERENT contracts libraries.
* Two packages need to inherit each other's contracts (e.g., one package defines a class which inherits a base class from a different package and hence needs to inherit its contracts as well).
Ditto. So, small benefit. Enough to be of value, but not (on its own) compelling.
* Third-party libraries for testing and static verification need a standard approach to contracts in order to be usable. Otherwise, the authors of these libraries (e.g. Hypothesis) need to support multiple contrat libraries (if they could ever bother to support multiple of them).
And ditto again. So I would disagree that these benefits are critical, but they are definitely benefits.
The main question is: are you intending for the standard library to be annotated with contracts? If not, all your proposed benefits can be achieved at the level of a single project, by just saying "we're going to use THIS contracts library", unless I'm misunderstanding something here. If so, you suddenly have to figure out how much longer it takes *across all of Python's startup*. How much longer will it take to run "hg help" with annotations set up? That's the kind of benchmark that's going to make or break this proposal. Adding a notable delay to command-line tools like that would seriously hurt. ChrisA
Hi Chris, If not, all your proposed benefits can be achieved at the level of a
single project, by just saying "we're going to use THIS contracts library", unless I'm misunderstanding something here.
I think we are having a big disconnect in the discussion. Please apologize for my vagueness and point me where you would like me to elaborate more. Let me try to have another iteration where I'll try to paraphrase myself and illustrate what I think that you are thinking. I imagine that you conceive contracts purely as an approach to a testing to be applied to a single project. I'm not talking about that. I'm talking about two packages on pypi, both specifying contracts, each developed by a separate team from different organizations. Let's call these packages package_a and package_b, respectively (and team A and B, analogously). Imagine now that the contracts are not standardized. For example, team A uses dpcontracts whereas team B uses icontract. Enter team C working on the package_c. There is a class package_c.SomeClass that needs to inherit both from package_a.some_module.BaseClass and from package_b.another_module.AnotherBaseClass. Imagine the mess that developers in team C have to deal with -- how are contracts supposed to be strengthened and weakened in package_c.SomeClass? A nightmare with different contract libraries. Even if they don't have to work with multiple inheritance, they would need to use different syntaxes for different concrete classes inheriting from the two packages, respectively. And now think of contract groups that can be specified and applied to functions: @require(...) @require(...) @ensure(...) def some_func(...): ... @enforce_contracts_of(some_func) @require(...) def some_other_func(...): ... some_group = [ require(...), ensure(...), ensure(...) ] @enforce_contracts(some_group) @ensure(...) def yet_another_function(...): ... How are these contract groups supposed to play together between package_a and package_b when applied in package_c? (A side remark: the name and implementation of "enforce_contracts_of" and "enforce_contracts" are still in discussion for icontract and have not been implemented yet.) Furthermore, what about toggling contracts? Team C needs to know both how to toggle the contracts of package_a.some_module and also how to toggle the contracts of package_b.another_module. They need to be aware of and probably set two different environment variables with different toggling granularities *etc*. What if the toggling conflicts? Even if they both used, say, icontract. Package_a might require icontract 1.x whereas package_b requires 2.x. This results in a dependency hell. A standardized lib would force both packages to use the same version of contract lib (or not be supported in that given python version at all) -- the compatibility would be dictated by the python version, not by the version of a non-standard contract lib. Let's focus now on the team working on mypy. For example, consider the code: @invariant(lambda self: self.something is not None or self.another is not None) class SomeClass: ... # later in code somewhere def some_func(): s = SomeClass(...) if s.something is None: # mypy knows here that s.another is not None. ... But if there existed multiple non-standard contract libraries, can we really expect that mypy deals with all of them? I couldn't possibly imagine that. If these points are still not clear -- imagine having multiple non-standard, but widely used libraries and approaches for abstract base classes or object-oriented programming. A horrifying thought :) I'd probably even prefer the python world without contracts than with a widespread use of multiple contract libraries. If pythonistas are to use contracts more widely without stomping on each other's feet, there needs to be some form of standardization so that different packages can inter-operate. There needs to be a convention how to toggle the contracts at different levels of granularity (individual contracts, module-level, pypi package-level, only in testing *etc.*). Environment variables? Setting global variables before importing a module? Some mechanism I'm not aware of? Are the contracts on or off by default? There are so many other details that needs to be fleshed out and this needs to be a collective effort if it is to succeed. I don't have the knowledge to figure out all the conceptual details alone and need help from more experienced people. --- Now a completely different point. Whether we write contracts *for* the modules of the standard library is another discussion. It is not directly connected with the standardization of contracts (and also not part of the current thread, IMO). To write contracts or not to write contracts for a standard library is a per-module decision and concerns the authors and users of each module in separation. Some modules might include contracts that are run only in test mode (*e.g., *in case of icontract, that would be the situation when the environment variable ICONTRACT_SLOW is set); other modules might verify only some of them in production; yet other modules might always verify all the contracts. Or standard lib might take the approach that each module has a corresponding environment variable to turn *on *the contracts, while its contracts are disabled by default. In current implementation of icontract, there is a negligible overhead at the moment during the import of a module with disabled contracts (numbers from the previous message; average over 10 runs, in milliseconds): Duration to import the module functions_100_with_no_contract : 795.59 ± 10.47 Duration to import the module functions_100_with_1_disabled_contract : 833.60 ± 32.07 Duration to import the module functions_100_with_5_disabled_contracts : 851.31 ± 66.93 Duration to import the module functions_100_with_10_disabled_contracts : 897.90 ± 143.02 The whole import overhead is attributed to the interpreter having to parse the decorators and the call to the decorator at the import time (when the decorator immediately returns the wrapped function). If this overhead is unacceptable -- then we need to figure out how to circumvent the parsing of contracts in the interpreter directly and ignore the contract decorators which are disabled. Or have a code base which is shipped for production and another one which is shipped for testing (the former would have the contracts automatically stripped out). This overhead is a limitation of *any *decorator -- *e.g.,* if abstract base classes are OK, I don't see why contracts wouldn't be. I could think of many other optimizations in CPython (*e.g., *it could in-line the code to verify the condition in the function body to avoid making two calls, one to the wrapper and another to the wrapped function), but this is really not the topic of the current thread (please feel free to fork to another thread if you'd like to discuss that particular topic). --- My conclusion is at this point is that the best course of action would be that other teams pick up a contract library (now that there is at least icontract library with a full feature set for design-by-contract akin to Eiffel and other languages), and present us how they use contracts and what problems they were facing (*e.g., *were there any problems with the toggling granularity? readability? maintainability?). Then we decide how to proceed. What are the opinions of the other folks that would like to have a standardized contract library? Cheers, Marko
On Wed, Oct 24, 2018 at 11:23 AM, Marko Ristin-Kaufmann <marko.ristin@gmail.com> wrote:
I appreciate your enthusiasm making Python better, and concern for the ecosystem as a whole! But, right now, this particular problem is mostly a theoretical issue, right? Currently 0% of the packages on PyPI use any contracts library (within rounding error), so it'd be very unlikely to have conflicts like this. So in the future, a few different things could happen: Maybe you fail to convince users that contracts are useful, so the usage remains rare. In that case, the problem never happens. Maybe icontracts is super successful, everyone adopts it, and it becomes a de facto standard. Great! Now your problem is solved, because everyone is using icontract. Maybe two different contracts libraries become successful, and people start hitting this problem for real. Something has to be done! But... what? The Python core devs aren't going to know which library is better, or where the pain points are, or how to best mitigate them. The people best placed to figure that out are the developers of the libraries, and their users. And even if we move one of them into the stdlib, that won't magically force people to adopt it. The stdlib has urllib, but most people use requests; the stdlib has ssl, but lots of projects use pyopenssl... In your position, I wouldn't be talking to the core devs; I'd be writing blog posts to proselytize the advantages of contracts, working with popular projects that are interested in adopting them, writing case studies, going to meetups, submitting talks to pycons, that sort of thing. If you want contracts to become a widely-used thing, you have to convince people to use them. The core team doesn't have a magic wand that can short-circuit that process.
But if there existed multiple non-standard contract libraries, can we really expect that mypy deals with all of them? I couldn't possibly imagine that.
mypy currently ships with special support for both dataclasses and attrs, which are basically two implementations of the same idea. If there were multiple popular contract libraries, and there was some benefit to mypy supporting popular contract libraries, then I assume it would support all of them too. Why not? -n -- Nathaniel J. Smith -- https://vorpus.org
On Wed, Oct 24, 2018 at 12:57 AM, Chris Angelico <rosuav@gmail.com> wrote:
I'm not (currently) a fan of design by contract, and don't expect I'll end up using it, whether or not we get a nifty standard library for it.... That being said -- this is analogous to why PEP 484 -- Type Hints was adopted, even though the syntax changes in Python had long been adopted to enable it. That is: the IF folks are going to use DbC in Python, the community is MUCH better off if everyone does it in the same way. And the way to make that happen is to put it in the stdlib. However, I'm not sure it's anywhere near time to actually do that -- before we get there, there needs to be a pretty good community of folks using icontract (or maybe something else?) and ideally some interest from a core developer or two. After all, PEP 484 didn't get rollling until Guido got interested in MyPy. But by all means -- start building that community -- maybe some of us skeptics will be drawn in ....
Contractifying the stdlib is an entirely independent question -- yes, if we wanted to add contract to the stdlib, we'd need a contract library in the stdlib, but there's no need for the other way around to be true. Is the stdlib being fully typhinted since PEP 484 was added? a quick scan of the PEP didn't seem to mention it at all. Performance (particularly start up time) would absolutely be a consideration if and when we get there, but I don't know that it has any impact to anything anyone might do now. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
Chris Barker via Python-ideas writes:
This is historically imprecise, though: function annotations were adopted with an eye to some form of optional typing.
After all, PEP 484 didn't get rollling until Guido got interested in MyPy.
I don't know how much effort Guido put into type hinting himself until the run-up to PEP 484. But he has said that type hinting was the intended use case all along, and he wasn't very sympathetic to arguments that people had found other uses that conflicted. That said, stub files could serve that purpose (with the stub overriding inline annotations, and a flag to mypy to not check the inlines).
That is: the IF folks are going to use DbC in Python, the community is MUCH better off if everyone does it in the same way.
I don't think that's true for icontracts (note, I'm being specific to the implementation here!) in the same way as for type hints, though. Type hints are part of the syntax (even though analyzed by a separate program). You would really need pragmas (function by function, maybe even argument by argument) flags to allow use of multiple type-checkers. Python never liked pragmas at all (this could change with Guido's retirement as BDFL, but of course we're all here because to a great degree we like Guido's taste in such things). With decorator-based contracts, however, you have a function wrapped by code generated by a decorator. It's transparent to the compiler. I just don't see Marko's "dependency hell" being that big a problem. Consider receiving a binary module "contraction" using icontracts. Your project uses bfcontracts (b for banana, f for flavor), also implemented as a set of decorators. As far as I can see, to use bf contracts at top level, all you need to do is import icontracts icontracts.verify_contracts = True # Note: no further mention of icontracts below. import bfcontracts import contraction class inherited(iparent): @bfcontracts.require(additional_contract) def recontract(old_arg_1, ...): return contraction.iparent.recontract(old_arg_1, ...) This would be annoying, but not impossible. You can't weaken the contracts for iparent.recontract as far as I can see in a decorator- based contract library. I haven't thought about it, but with a little bit of trickery it might even be possible in the simple case above to do @bfcontracts.require(additional_contract) @bfcontracts.call_with_inherited_contracts(recontract) def recontract(old_arg_1, ...): pass It's true that you can't replace the implementation of recontract() (ie, without calling contractions.iparent.recontract) and preserve the contracts automatically. This means that contract implementations will have to know something about how other implementations (that do inherit contracts on interfaces rather than function/method implementation) store such contracts. This certainly would be worth standardizing with a dunder attribute or something like that.
And the way to make that happen is to put it in the stdlib.
True, but
+1
Is the stdlib being fully typhinted since PEP 484 was added? a quick scan of the PEP didn't seem to mention it at all.
There was discussion about stub hint files for the stdlib, but I believe it's being implemented outside of the stdlib. Sorry, don't have time to check. -- Associate Professor Division of Policy and Planning Science http://turnbull.sk.tsukuba.ac.jp/ Faculty of Systems and Information Email: turnbull@sk.tsukuba.ac.jp University of Tsukuba Tel: 029-853-5175 Tennodai 1-1-1, Tsukuba 305-8573 JAPAN
A few thoughts: 1) Just as with with PEP 484, we could distinguish between contract specification and checking. To achieve some of the goals that you state, we need to standardise the way people would state contracts in their code (and provide a small implementation in the stdlib, similar to the typing module), but how these contracts are verified (at runtime or statically) or leveraged in the documentation are left to third party projects (similar to mypy & al). 2) Building momentum is paramount. First, since there are many contracts libraries out there, some dating more than 15 years (ex: PyDBC), we'd need to build consensus between the people who wrote and use these libraries. And of course, get feedback from people who use this approach on significant projects. I'm willing to try your library on one of my projects. 3) Having contracts for most of the stdlib would be a worthy goal (for contracts users) but, as noted by some, would come with some speed issues. And of course, would represent a multi-year effort. Hopefully your project (i.e. standardising a library, but not "contracting" the stdlib) would still be valuable if this never happen. 4) Interaction between contracts and typing is an open question. S. On Wed, Oct 24, 2018 at 9:40 AM Marko Ristin-Kaufmann < marko.ristin@gmail.com> wrote:
-- Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/ Chairman, Free&OSS Group @ Systematic Cluster - http://www.gt-logiciel-libre.org/ Co-Chairman, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/ Founder & Organiser, PyParis & PyData Paris - http://pyparis.org/ & http://pydata.fr/
Hi, Stephen J. Turnbull wrote:
Of course you can weaken the contracts for iparent.recontract with a decorator-based contract library. From my first message:
The features include: ...
* inheritance of the contracts (with strengthening/weakening)
...
Please also have a look at the readme on https://github.com/Parquery/icontract/, Section "Inheritance". Chris Barker wrote:
I absolutely agree. In my first message I wrote:
and in my second message (before yours), I wrote:
Stefane Fermigier wrote:
This is a great idea -- though I'm not sure how it could be implemented other than already specifying the implementation (i.e. how the contracts are verified). If you have some time, could you elaborate a bit how this might work? I'm suffering from tunnel vision at this point after implementing the library.
Thanks, I'm looking forward to hearing your experience! I contacted the authors of dpcontracts, but there is still no clear plan of action how to merge icontract and dpcontracts (see https://github.com/deadpixi/contracts/issues/15). Other libraries seem either unmaintained or not applicable except for single-argument checks and involve custom syntax (like https://pypi.org/project/PyContracts/; I only played a bit with PyContracts, so it would be great if anybody could tell their experience with it in a larger code base). We found that all the present libraries were far from usable for a larger code base (so we rolled out our own). To the best of our knowledge, none could deal with inheritance properly (weakening/strengthening) and you always had to write out the condition along the lambda function (unless the code was parsed from the documentation which increases the import time significantly even if you want to disable some of the contracts). As others already mentioned in the previous discussion, part of the reason for not adopting DbC was the lack of tools. I'm not saying that icontract should be that standard tool -- not at all! We need to first find out what such standard tool would need to fulfill*. *Assuming that we don't make language changes, what would we like the standard contract library to look like? What features should it have? What is too much of a niche to be left out?
I agree. Nathaniel Smith wrote:
I thought python-ideas is an open list to generally discuss ideas, not the core dev list (isn't that python-dev)? That's why I wrote here (after being forwarded from python-dev list). I agree that these efforts you mention would be worthwhile. Implementation of a (prototype) library and participating in the discussions on this mail list are the maximum effort I can put up at the moment. Maybe in 2019 I could manage to go to the local Python summit. Cheers, Marko
On Thu, Oct 25, 2018, 14:44 Marko Ristin-Kaufmann <marko.ristin@gmail.com> wrote:
Python-ideas is for ideas for changing the python language or standard library. And the subthread I was replying to was about adding contracts support to the stdlib, so you're in the right place for that discussion. I was trying to explain why trying to add contract support to the stdlib doesn't make much sense right now. And if you switch your focus to trying to recruit users and collaborators, like I recommend, then python-ideas isn't the best place to do that. Most of your potential users aren't here! -n
P.S. Here is the link to the github repo: https://github.com/Parquery/icontract On Wed, 24 Oct 2018 at 09:40, Marko Ristin-Kaufmann <marko.ristin@gmail.com> wrote:
On Wed, Oct 24, 2018 at 6:41 PM Marko Ristin-Kaufmann <marko.ristin@gmail.com> wrote:
For the sake of those of us who REALLY don't feel like diving back into the extensive threads on this subject, can you please summarize the benefits of having this in the stdlib rather than as a third-party library? Also - have you benchmarked the performance cost of adding contracts? Particularly: if you're planning to contractify the stdlib, what is the impact on startup performance? ChrisA
Hi Chris, For the sake of those of us who REALLY don't feel like diving back
Certainly. We need a standard approach to contracts as opposed to third-party libraries for the following technical reasons: * There is no dependency hell if two packages use different versions of the same contract library. * Two packages need to inherit each other's contracts (*e.g.*, one package defines a class which inherits a base class from a different package and hence needs to inherit its contracts as well). * Third-party libraries for testing and static verification need a standard approach to contracts in order to be usable. Otherwise, the authors of these libraries (*e.g. *Hypothesis) need to support multiple contrat libraries (if they could ever bother to support multiple of them). There are many more, but these are the reasons that I find critical.
I made some preliminary benchmarks. The source code is available at: https://github.com/Parquery/icontract/tree/master/benchmarks/startup Average over ten runs in milliseconds on my machine (Intel(R) Core(TM) i7-4700MQ CPU @ 2.40GHz, 8 cores, 4 GB RAM): Duration to import the module functions_100_with_no_contract : 795.59 ± 10.47 Duration to import the module functions_100_with_1_contract : 919.53 ± 61.22 Duration to import the module functions_100_with_5_contracts : 1075.81 ± 59.87 Duration to import the module functions_100_with_10_contracts : 1290.22 ± 90.04 Duration to import the module functions_100_with_1_disabled_contract : 833.60 ± 32.07 Duration to import the module functions_100_with_5_disabled_contracts : 851.31 ± 66.93 Duration to import the module functions_100_with_10_disabled_contracts : 897.90 ± 143.02 Duration to import the module classes_100_with_no_invariant : 843.61 ± 28.21 Duration to import the module classes_100_with_1_invariant : 3409.71 ± 95.78 Duration to import the module classes_100_with_5_invariants : 4005.93 ± 131.97 Duration to import the module classes_100_with_10_invariants : 4801.82 ± 157.56 Duration to import the module classes_100_with_1_disabled_invariant : 885.88 ± 44.24 Duration to import the module classes_100_with_5_disabled_invariants : 912.53 ± 101.91 Duration to import the module classes_100_with_10_disabled_invariants : 963.77 ± 161.76 Please let me know if these are the benchmarks you had in mind or you would like to see something else. I have neither optimized the code for speed nor investigated why the performance of class invariants differs so much from the functions. The important bit for me was that disabling contracts basically had no impact on import time. Cheers, Marko
On Wed, Oct 24, 2018 at 9:08 PM Marko Ristin-Kaufmann <marko.ristin@gmail.com> wrote:
Wouldn't version differences happen just as much whether it's in the stdlib or a third-party library? The contracts API has to be built to cope with this, regardless. But having it in the stdlib would at least mean that you don't have to worry about incompatible DIFFERENT contracts libraries.
* Two packages need to inherit each other's contracts (e.g., one package defines a class which inherits a base class from a different package and hence needs to inherit its contracts as well).
Ditto. So, small benefit. Enough to be of value, but not (on its own) compelling.
* Third-party libraries for testing and static verification need a standard approach to contracts in order to be usable. Otherwise, the authors of these libraries (e.g. Hypothesis) need to support multiple contrat libraries (if they could ever bother to support multiple of them).
And ditto again. So I would disagree that these benefits are critical, but they are definitely benefits.
The main question is: are you intending for the standard library to be annotated with contracts? If not, all your proposed benefits can be achieved at the level of a single project, by just saying "we're going to use THIS contracts library", unless I'm misunderstanding something here. If so, you suddenly have to figure out how much longer it takes *across all of Python's startup*. How much longer will it take to run "hg help" with annotations set up? That's the kind of benchmark that's going to make or break this proposal. Adding a notable delay to command-line tools like that would seriously hurt. ChrisA
Hi Chris, If not, all your proposed benefits can be achieved at the level of a
single project, by just saying "we're going to use THIS contracts library", unless I'm misunderstanding something here.
I think we are having a big disconnect in the discussion. Please apologize for my vagueness and point me where you would like me to elaborate more. Let me try to have another iteration where I'll try to paraphrase myself and illustrate what I think that you are thinking. I imagine that you conceive contracts purely as an approach to a testing to be applied to a single project. I'm not talking about that. I'm talking about two packages on pypi, both specifying contracts, each developed by a separate team from different organizations. Let's call these packages package_a and package_b, respectively (and team A and B, analogously). Imagine now that the contracts are not standardized. For example, team A uses dpcontracts whereas team B uses icontract. Enter team C working on the package_c. There is a class package_c.SomeClass that needs to inherit both from package_a.some_module.BaseClass and from package_b.another_module.AnotherBaseClass. Imagine the mess that developers in team C have to deal with -- how are contracts supposed to be strengthened and weakened in package_c.SomeClass? A nightmare with different contract libraries. Even if they don't have to work with multiple inheritance, they would need to use different syntaxes for different concrete classes inheriting from the two packages, respectively. And now think of contract groups that can be specified and applied to functions: @require(...) @require(...) @ensure(...) def some_func(...): ... @enforce_contracts_of(some_func) @require(...) def some_other_func(...): ... some_group = [ require(...), ensure(...), ensure(...) ] @enforce_contracts(some_group) @ensure(...) def yet_another_function(...): ... How are these contract groups supposed to play together between package_a and package_b when applied in package_c? (A side remark: the name and implementation of "enforce_contracts_of" and "enforce_contracts" are still in discussion for icontract and have not been implemented yet.) Furthermore, what about toggling contracts? Team C needs to know both how to toggle the contracts of package_a.some_module and also how to toggle the contracts of package_b.another_module. They need to be aware of and probably set two different environment variables with different toggling granularities *etc*. What if the toggling conflicts? Even if they both used, say, icontract. Package_a might require icontract 1.x whereas package_b requires 2.x. This results in a dependency hell. A standardized lib would force both packages to use the same version of contract lib (or not be supported in that given python version at all) -- the compatibility would be dictated by the python version, not by the version of a non-standard contract lib. Let's focus now on the team working on mypy. For example, consider the code: @invariant(lambda self: self.something is not None or self.another is not None) class SomeClass: ... # later in code somewhere def some_func(): s = SomeClass(...) if s.something is None: # mypy knows here that s.another is not None. ... But if there existed multiple non-standard contract libraries, can we really expect that mypy deals with all of them? I couldn't possibly imagine that. If these points are still not clear -- imagine having multiple non-standard, but widely used libraries and approaches for abstract base classes or object-oriented programming. A horrifying thought :) I'd probably even prefer the python world without contracts than with a widespread use of multiple contract libraries. If pythonistas are to use contracts more widely without stomping on each other's feet, there needs to be some form of standardization so that different packages can inter-operate. There needs to be a convention how to toggle the contracts at different levels of granularity (individual contracts, module-level, pypi package-level, only in testing *etc.*). Environment variables? Setting global variables before importing a module? Some mechanism I'm not aware of? Are the contracts on or off by default? There are so many other details that needs to be fleshed out and this needs to be a collective effort if it is to succeed. I don't have the knowledge to figure out all the conceptual details alone and need help from more experienced people. --- Now a completely different point. Whether we write contracts *for* the modules of the standard library is another discussion. It is not directly connected with the standardization of contracts (and also not part of the current thread, IMO). To write contracts or not to write contracts for a standard library is a per-module decision and concerns the authors and users of each module in separation. Some modules might include contracts that are run only in test mode (*e.g., *in case of icontract, that would be the situation when the environment variable ICONTRACT_SLOW is set); other modules might verify only some of them in production; yet other modules might always verify all the contracts. Or standard lib might take the approach that each module has a corresponding environment variable to turn *on *the contracts, while its contracts are disabled by default. In current implementation of icontract, there is a negligible overhead at the moment during the import of a module with disabled contracts (numbers from the previous message; average over 10 runs, in milliseconds): Duration to import the module functions_100_with_no_contract : 795.59 ± 10.47 Duration to import the module functions_100_with_1_disabled_contract : 833.60 ± 32.07 Duration to import the module functions_100_with_5_disabled_contracts : 851.31 ± 66.93 Duration to import the module functions_100_with_10_disabled_contracts : 897.90 ± 143.02 The whole import overhead is attributed to the interpreter having to parse the decorators and the call to the decorator at the import time (when the decorator immediately returns the wrapped function). If this overhead is unacceptable -- then we need to figure out how to circumvent the parsing of contracts in the interpreter directly and ignore the contract decorators which are disabled. Or have a code base which is shipped for production and another one which is shipped for testing (the former would have the contracts automatically stripped out). This overhead is a limitation of *any *decorator -- *e.g.,* if abstract base classes are OK, I don't see why contracts wouldn't be. I could think of many other optimizations in CPython (*e.g., *it could in-line the code to verify the condition in the function body to avoid making two calls, one to the wrapper and another to the wrapped function), but this is really not the topic of the current thread (please feel free to fork to another thread if you'd like to discuss that particular topic). --- My conclusion is at this point is that the best course of action would be that other teams pick up a contract library (now that there is at least icontract library with a full feature set for design-by-contract akin to Eiffel and other languages), and present us how they use contracts and what problems they were facing (*e.g., *were there any problems with the toggling granularity? readability? maintainability?). Then we decide how to proceed. What are the opinions of the other folks that would like to have a standardized contract library? Cheers, Marko
On Wed, Oct 24, 2018 at 11:23 AM, Marko Ristin-Kaufmann <marko.ristin@gmail.com> wrote:
I appreciate your enthusiasm making Python better, and concern for the ecosystem as a whole! But, right now, this particular problem is mostly a theoretical issue, right? Currently 0% of the packages on PyPI use any contracts library (within rounding error), so it'd be very unlikely to have conflicts like this. So in the future, a few different things could happen: Maybe you fail to convince users that contracts are useful, so the usage remains rare. In that case, the problem never happens. Maybe icontracts is super successful, everyone adopts it, and it becomes a de facto standard. Great! Now your problem is solved, because everyone is using icontract. Maybe two different contracts libraries become successful, and people start hitting this problem for real. Something has to be done! But... what? The Python core devs aren't going to know which library is better, or where the pain points are, or how to best mitigate them. The people best placed to figure that out are the developers of the libraries, and their users. And even if we move one of them into the stdlib, that won't magically force people to adopt it. The stdlib has urllib, but most people use requests; the stdlib has ssl, but lots of projects use pyopenssl... In your position, I wouldn't be talking to the core devs; I'd be writing blog posts to proselytize the advantages of contracts, working with popular projects that are interested in adopting them, writing case studies, going to meetups, submitting talks to pycons, that sort of thing. If you want contracts to become a widely-used thing, you have to convince people to use them. The core team doesn't have a magic wand that can short-circuit that process.
But if there existed multiple non-standard contract libraries, can we really expect that mypy deals with all of them? I couldn't possibly imagine that.
mypy currently ships with special support for both dataclasses and attrs, which are basically two implementations of the same idea. If there were multiple popular contract libraries, and there was some benefit to mypy supporting popular contract libraries, then I assume it would support all of them too. Why not? -n -- Nathaniel J. Smith -- https://vorpus.org
On Wed, Oct 24, 2018 at 12:57 AM, Chris Angelico <rosuav@gmail.com> wrote:
I'm not (currently) a fan of design by contract, and don't expect I'll end up using it, whether or not we get a nifty standard library for it.... That being said -- this is analogous to why PEP 484 -- Type Hints was adopted, even though the syntax changes in Python had long been adopted to enable it. That is: the IF folks are going to use DbC in Python, the community is MUCH better off if everyone does it in the same way. And the way to make that happen is to put it in the stdlib. However, I'm not sure it's anywhere near time to actually do that -- before we get there, there needs to be a pretty good community of folks using icontract (or maybe something else?) and ideally some interest from a core developer or two. After all, PEP 484 didn't get rollling until Guido got interested in MyPy. But by all means -- start building that community -- maybe some of us skeptics will be drawn in ....
Contractifying the stdlib is an entirely independent question -- yes, if we wanted to add contract to the stdlib, we'd need a contract library in the stdlib, but there's no need for the other way around to be true. Is the stdlib being fully typhinted since PEP 484 was added? a quick scan of the PEP didn't seem to mention it at all. Performance (particularly start up time) would absolutely be a consideration if and when we get there, but I don't know that it has any impact to anything anyone might do now. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
Chris Barker via Python-ideas writes:
This is historically imprecise, though: function annotations were adopted with an eye to some form of optional typing.
After all, PEP 484 didn't get rollling until Guido got interested in MyPy.
I don't know how much effort Guido put into type hinting himself until the run-up to PEP 484. But he has said that type hinting was the intended use case all along, and he wasn't very sympathetic to arguments that people had found other uses that conflicted. That said, stub files could serve that purpose (with the stub overriding inline annotations, and a flag to mypy to not check the inlines).
That is: the IF folks are going to use DbC in Python, the community is MUCH better off if everyone does it in the same way.
I don't think that's true for icontracts (note, I'm being specific to the implementation here!) in the same way as for type hints, though. Type hints are part of the syntax (even though analyzed by a separate program). You would really need pragmas (function by function, maybe even argument by argument) flags to allow use of multiple type-checkers. Python never liked pragmas at all (this could change with Guido's retirement as BDFL, but of course we're all here because to a great degree we like Guido's taste in such things). With decorator-based contracts, however, you have a function wrapped by code generated by a decorator. It's transparent to the compiler. I just don't see Marko's "dependency hell" being that big a problem. Consider receiving a binary module "contraction" using icontracts. Your project uses bfcontracts (b for banana, f for flavor), also implemented as a set of decorators. As far as I can see, to use bf contracts at top level, all you need to do is import icontracts icontracts.verify_contracts = True # Note: no further mention of icontracts below. import bfcontracts import contraction class inherited(iparent): @bfcontracts.require(additional_contract) def recontract(old_arg_1, ...): return contraction.iparent.recontract(old_arg_1, ...) This would be annoying, but not impossible. You can't weaken the contracts for iparent.recontract as far as I can see in a decorator- based contract library. I haven't thought about it, but with a little bit of trickery it might even be possible in the simple case above to do @bfcontracts.require(additional_contract) @bfcontracts.call_with_inherited_contracts(recontract) def recontract(old_arg_1, ...): pass It's true that you can't replace the implementation of recontract() (ie, without calling contractions.iparent.recontract) and preserve the contracts automatically. This means that contract implementations will have to know something about how other implementations (that do inherit contracts on interfaces rather than function/method implementation) store such contracts. This certainly would be worth standardizing with a dunder attribute or something like that.
And the way to make that happen is to put it in the stdlib.
True, but
+1
Is the stdlib being fully typhinted since PEP 484 was added? a quick scan of the PEP didn't seem to mention it at all.
There was discussion about stub hint files for the stdlib, but I believe it's being implemented outside of the stdlib. Sorry, don't have time to check. -- Associate Professor Division of Policy and Planning Science http://turnbull.sk.tsukuba.ac.jp/ Faculty of Systems and Information Email: turnbull@sk.tsukuba.ac.jp University of Tsukuba Tel: 029-853-5175 Tennodai 1-1-1, Tsukuba 305-8573 JAPAN
A few thoughts: 1) Just as with with PEP 484, we could distinguish between contract specification and checking. To achieve some of the goals that you state, we need to standardise the way people would state contracts in their code (and provide a small implementation in the stdlib, similar to the typing module), but how these contracts are verified (at runtime or statically) or leveraged in the documentation are left to third party projects (similar to mypy & al). 2) Building momentum is paramount. First, since there are many contracts libraries out there, some dating more than 15 years (ex: PyDBC), we'd need to build consensus between the people who wrote and use these libraries. And of course, get feedback from people who use this approach on significant projects. I'm willing to try your library on one of my projects. 3) Having contracts for most of the stdlib would be a worthy goal (for contracts users) but, as noted by some, would come with some speed issues. And of course, would represent a multi-year effort. Hopefully your project (i.e. standardising a library, but not "contracting" the stdlib) would still be valuable if this never happen. 4) Interaction between contracts and typing is an open question. S. On Wed, Oct 24, 2018 at 9:40 AM Marko Ristin-Kaufmann < marko.ristin@gmail.com> wrote:
-- Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/ Chairman, Free&OSS Group @ Systematic Cluster - http://www.gt-logiciel-libre.org/ Co-Chairman, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/ Founder & Organiser, PyParis & PyData Paris - http://pyparis.org/ & http://pydata.fr/
Hi, Stephen J. Turnbull wrote:
Of course you can weaken the contracts for iparent.recontract with a decorator-based contract library. From my first message:
The features include: ...
* inheritance of the contracts (with strengthening/weakening)
...
Please also have a look at the readme on https://github.com/Parquery/icontract/, Section "Inheritance". Chris Barker wrote:
I absolutely agree. In my first message I wrote:
and in my second message (before yours), I wrote:
Stefane Fermigier wrote:
This is a great idea -- though I'm not sure how it could be implemented other than already specifying the implementation (i.e. how the contracts are verified). If you have some time, could you elaborate a bit how this might work? I'm suffering from tunnel vision at this point after implementing the library.
Thanks, I'm looking forward to hearing your experience! I contacted the authors of dpcontracts, but there is still no clear plan of action how to merge icontract and dpcontracts (see https://github.com/deadpixi/contracts/issues/15). Other libraries seem either unmaintained or not applicable except for single-argument checks and involve custom syntax (like https://pypi.org/project/PyContracts/; I only played a bit with PyContracts, so it would be great if anybody could tell their experience with it in a larger code base). We found that all the present libraries were far from usable for a larger code base (so we rolled out our own). To the best of our knowledge, none could deal with inheritance properly (weakening/strengthening) and you always had to write out the condition along the lambda function (unless the code was parsed from the documentation which increases the import time significantly even if you want to disable some of the contracts). As others already mentioned in the previous discussion, part of the reason for not adopting DbC was the lack of tools. I'm not saying that icontract should be that standard tool -- not at all! We need to first find out what such standard tool would need to fulfill*. *Assuming that we don't make language changes, what would we like the standard contract library to look like? What features should it have? What is too much of a niche to be left out?
I agree. Nathaniel Smith wrote:
I thought python-ideas is an open list to generally discuss ideas, not the core dev list (isn't that python-dev)? That's why I wrote here (after being forwarded from python-dev list). I agree that these efforts you mention would be worthwhile. Implementation of a (prototype) library and participating in the discussions on this mail list are the maximum effort I can put up at the moment. Maybe in 2019 I could manage to go to the local Python summit. Cheers, Marko
On Thu, Oct 25, 2018, 14:44 Marko Ristin-Kaufmann <marko.ristin@gmail.com> wrote:
Python-ideas is for ideas for changing the python language or standard library. And the subthread I was replying to was about adding contracts support to the stdlib, so you're in the right place for that discussion. I was trying to explain why trying to add contract support to the stdlib doesn't make much sense right now. And if you switch your focus to trying to recruit users and collaborators, like I recommend, then python-ideas isn't the best place to do that. Most of your potential users aren't here! -n
participants (7)
-
Anders Hovmöller -
Chris Angelico -
Chris Barker -
Marko Ristin-Kaufmann -
Nathaniel Smith -
Stephen J. Turnbull -
Stéfane Fermigier