Python-ideas September 2018

python-ideas@python.org

108 participants
94 discussions

by Marko Ristin-Kaufmann

Hi, (I'd like to fork from a previous thread, "Pre-conditions and post-conditions", since it got long and we started discussing a couple of different things. Let's discuss in this thread the implementation of a library for design-by-contract and how to push it forward to hopefully add it to the standard library one day.) For those unfamiliar with contracts and current state of the discussion in the previous thread, here's a short summary. The discussion started by me inquiring about the possibility to add design-by-contract concepts into the core language. The idea was rejected by the participants mainly because they thought that the merit of the feature does not merit its costs. This is quite debatable and seems to reflect many a discussion about design-by-contract in general. Please see the other thread, "Why is design-by-contract not widely adopted?" if you are interested in that debate. We (a colleague of mine and I) decided to implement a library to bring design-by-contract to Python since we don't believe that the concept will make it into the core language anytime soon and we needed badly a tool to facilitate our work with a growing code base. The library is available at http://github.com/Parquery/icontract. The hope is to polish it so that the wider community could use it and once the quality is high enough, make a proposal to add it to the standard Python libraries. We do need a standard library for contracts, otherwise projects with *conflicting* contract libraries can not integrate (*e.g., *the contracts can not be inherited between two different contract libraries). So far, the most important bits have been implemented in icontract: - Preconditions, postconditions, class invariants - Inheritance of the contracts (including strengthening and weakening of the inherited contracts) - Informative violation messages (including information about the values involved in the contract condition) - Sphinx extension to include contracts in the automatically generated documentation (sphinx-icontract) - Linter to statically check that the arguments of the conditions are correct (pyicontract-lint) We are successfully using it in our code base and have been quite happy about the implementation so far. There is one bit still missing: accessing "old" values in the postcondition (*i.e., *shallow copies of the values prior to the execution of the function). This feature is necessary in order to allow us to verify state transitions. For example, consider a new dictionary class that has "get" and "put" methods: from typing import Optional from icontract import post class NovelDict: def length(self)->int: ... def get(self, key: str) -> Optional[str]: ... @post(lambda self, key, value: self.get(key) == value) @post(lambda self, key: old(self.get(key)) is None and old(self.length()) + 1 == self.length(), "length increased with a new key") @post(lambda self, key: old(self.get(key)) is not None and old(self.length()) == self.length(), "length stable with an existing key") def put(self, key: str, value: str) -> None: ... How could we possible implement this "old" function? Here is my suggestion. I'd introduce a decorator "before" that would allow you to store whatever values in a dictionary object "old" (*i.e. *an object whose properties correspond to the key/value pairs). The "old" is then passed to the condition. Here is it in code: # omitted contracts for brevity class NovelDict: def length(self)->int: ... # omitted contracts for brevity def get(self, key: str) -> Optional[str]: ... @before(lambda self, key: {"length": self.length(), "get": self.get(key)}) @post(lambda self, key, value: self.get(key) == value) @post(lambda self, key, old: old.get is None and old.length + 1 == self.length(), "length increased with a new key") @post(lambda self, key, old: old.get is not None and old.length == self.length(), "length stable with an existing key") def put(self, key: str, value: str) -> None: ... The linter would statically check that all attributes accessed in "old" have to be defined in the decorator "before" so that attribute errors would be caught early. The current implementation of the linter is fast enough to be run at save time so such errors should usually not happen with a properly set IDE. "before" decorator would also have "enabled" property, so that you can turn it off (*e.g., *if you only want to run a postcondition in testing). The "before" decorators can be stacked so that you can also have a more fine-grained control when each one of them is running (some during test, some during test and in production). The linter would enforce that before's "enabled" is a disjunction of all the "enabled"'s of the corresponding postconditions where the old value appears. Is this a sane approach to "old" values? Any alternative approach you would prefer? What about better naming? Is "before" a confusing name? Thanks a lot for your thoughts! Cheers, Marko

2 years, 6 months

Fwd: Keyword only argument on function call

by David Mertz

I'm still not sure why all this focus on new syntax or convoluted IDE enhancements. I presented a very simple utility function that accomplishes exactly the started goal of DRY in keyword arguments. Yes, I wrote a first version that was incomplete. And perhaps these 8-9 lines miss some corner case. But the basic goal is really, really easy to accomplish with existing Python. >>> import inspect >>> def reach(name): ... for f in inspect.stack(): ... if name in f[0].f_locals: ... return f[0].f_locals[name] ... return None ... >>> def use(names): ... kws = {} ... for name in names.split(): ... kws[name] = reach(name) ... return kws ... >>> def function(a=11, b=22, c=33, d=44): ... print(a, b, c, d) ... >>> function(a=77, **use('b d')) 77 None 33 None >>> def foo(): ... a, b, c = 1, 2, 3 ... function(a=77, **use('b d')) ... >>> foo() 77 2 33 None On Tue, Sep 25, 2018, 6:31 AM Anders Hovmöller <boxed(a)killingar.net> wrote: > Hi, > > I'd like to reopen this discussion if anyone is interested. Some things > have changed since I wrote my original proposal so I'll first summarize: > > 1. People seem to prefer the syntax `foo(=a)` over the syntax I suggested. > I believe this is even more trivial to implement in CPython than my > original proposal anyway... > 2. I have updated my analysis tool: > https://gist.github.com/boxed/610b2ba73066c96e9781aed7c0c0b25c It will > now also give you statistics on the number of arguments function calls > have. I would love to see some statistics for other closed source programs > you might be working on and how big those code bases are. > 3. I have made a sort-of implementation with MacroPy: > https://github.com/boxed/macro-kwargs/blob/master/test.py I think this is > a dead end, but it was easy to implement and fun to try! > 4. I have also recently had the idea that a foo=foo type pattern could be > handled in for example PyCharm as a code folding feature (and maybe as a > completion feature). > > I still think that changing Pythons syntax is the right way to go in the > long run but with point 4 above one could experience what this feature > would feel like without running a custom version of Python and without > changing your code. I admit to a lot of trepidation about wading into > PyCharms code though, I have tried to do this once before and I gave up. > > Any thoughts? > > / Anders > _______________________________________________ > Python-ideas mailing list > Python-ideas(a)python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ >

2 years, 6 months

Re: [Python-ideas] "old" values in postconditions

by James Lu

Hmm, I was wrong: there is no reliable way to get the code of a lambda function. If it was possible to execute all code paths of the function, we could monkey patch the builtins so { } used our own custom set class. Alternatively, the decorator could also accept a string. Or maybe we could send a PEP to add the .func_code attribute to lambdas as well as normal functions. There’s also a technique online where they find the lambda’s source by locating the file the function was defined in and then removing the irrelevant parts, but that just doesn’t sound practical to me. There’s also MacroPy. I think the best solution would be to mock the old object and record the operations done to the object, like the other replier gave a PoC of. Proposed syntax from icontract import post, old @post(lambda: ..., key=old.self.key(), ) Sent from my iPhone > On Sep 25, 2018, at 1:49 AM, Marko Ristin-Kaufmann <marko.ristin(a)gmail.com> wrote: > > Hi James, > Thanks for the feedback! > > I also thought about decompiling the condition to find its AST and figure out what old values are needed. However, this is not so easily done at the moment as all the decompilation libraries I looked at (meta, ucompyle6) are simply struggling to keep up with the development of the python bytecode. In other words, the decompiler needs to be updated with every new version of python which is kind of a loosing race (unless the python devs themselves don't provide this functionality which is not the case as far as I know). > > There is macropy (https://github.com/lihaoyi/macropy) which was suggested on the other thread (https://groups.google.com/forum/#!topic/python-ideas/dmXz_7LH4GI) that I'm currently looking at. > > Cheers, > Marko > > >> On Tue, 25 Sep 2018 at 00:35, James Lu <jamtlu(a)gmail.com> wrote: >> You could disassemble (import dis) the lambda to biew the names of the lambdas. >> >> @before(lambda self, key, _, length, get: self.length(), self.get(key)) >> >> Perhaps you could disassemble the function code and look at all operations or accesses that are done to “old.” and evaluate those expressions before the function runs. Then you could “replace” the expression. >> @post(lambda self, key, old: old.get is None and old.length + 1 == >> self.length()) >> >> Either the system would grab old.get and old.length or be greedy and grab old.get is None and old.length + 1. It would then replace the old.get and old.length with injects that only respond to is None and +1. >> >> Or, a syntax like this >> @post(lambda self, key, old: [old.get(old.key)] is None and [old.self.length() + 1] == >> self.length()) >> >> Where the stuff inside the brackets is evaluated before the decorated function runs. It would be useful for networking functions or functions that do something ephemeral, where data related to the value being accessed needed for the expression no longer exists after the function. >> >> This does conflict with list syntax forever, so maybe either force people to do list((expr,)) or use an alternate syntax like one item set syntax { } or double set syntax {{ }} or double list syntax [[ ]]. Ditto with having to avoid the literals for the normal meaning. >> >> You could modify Python to accept any expression for the lambda function and propose that as a PEP. (Right now it’s hardcoded as a dotted name and optionally a single argument list surrounded by parentheses.) >> >> I suggest that instead of “@before” it’s “@snapshot” and instead of “old” it’s “snapshot”. >> >> Python does have unary plus/minus syntax as well as stream operators (<<, >>) and list slicing syntax and the @ operator and operators & and | if you want to play with syntax. There’s also the line continuation character for crazy lambdas. >> >> Personally I prefer >> @post(lambda self, key, old: {{old.self.get(old.key)}} and {{old.self.length() + 1}} == >> self.length()) >> >> because it’s explicit about what it does (evaluate the expressions within {{ }} before the function runs. I also find it elegant. >> >> Alternatively, inside the {{ }} could be a special scope where locals() is all the arguments @pre could’ve received as a dictionary. For either option you can remove the old parameter from the lambda. Example: >> @post(lambda self, key: {{self.get(key)}} and {{self.length() + 1}} == >> self.length()) >> >> Perhaps the convention should be to write {{ expr }} (with the spaces in between). >> >> You’d probably have to use the ast module to inspect it instead of the dis modul. Then find some way to reconstruct the expressions inside the double brackets- perhaps by reconstructing the AST and compiling it to a code object, or perhaps by finding the part of the string the expression is located. dis can give you the code as a string and you can run a carefully crafted regex on it. >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas(a)python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/

2 years, 6 months

Why is design-by-contracts not widely adopted?

by Hugh Fisher

> Date: Mon, 24 Sep 2018 09:46:16 +0200 > From: Marko Ristin-Kaufmann <marko.ristin(a)gmail.com> > To: Python-Ideas <python-ideas(a)python.org> > Subject: Re: [Python-ideas] Why is design-by-contracts not widely > adopted? > Message-ID: > <CAGu4bVB4=Bou+DhoDzayx=i7eQ2Gr8KDDOFONeaB-Qvt8LZo2A(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" [munch] > Their users would hugely benefit from a more mature > and standardized contracts library with informative violation messages. Will respond in another message, because it's a big topic. > I really don't see how DbC has to do with duck typing (unless you reduce it > to mere isinstance conditions, which would simply be a straw-man argument) > -- could you please clarify? I argue that Design by Contract doesn't make sense for Python and other dynamically typed, duck typed languages because it's contrary to how the language, and the programmer, expects to work. In Python we can write something like: def foo(x): x.bar(y) What's the type of x? What's the type of y? What is the contract of bar? Don't know, don't care. x, or y, can be an instance, a class, a module, a proxy for a remote web service. The only "contract" is that object x will respond to message bar that takes one argument. Object x, do whatever you want with it. And that's a feature, not a bug, not bad design. It follows Postel's Law for Internet protocols of being liberal in what you accept. It follows the Agile principle of valuing working software over comprehensive doco. It allows software components to be glued together quickly and easily. It's a style of programming that has been successful for many years, not just in Python but also in Lisp and Smalltalk and Perl and JavaScript. It works. Not for everything. If I were writing the avionics control routines for a helicopter gas turbine, I'd use formal notation and static type checking and preconditions and whatnot. But I wouldn't be using Python either. > As soon as you need to document your code, and > this is what most modules have to do in teams of more than one person > (especially so if you are developing a library for a wider audience), you > need to write down the contracts. Please see above where I tried to > explained that 2-5) are inferior approaches to documenting contracts > compared to 1). You left off option 6), plain text. Comments. Docstrings. README files. Web pages. Books. In my experience, this is what most people consider documentation. A good book, a good blog post, can explain more about how a library works and what the implementation requirements and restrictions are than formal contract notation. In particular, contracts in Eiffel don't explain *why* they're there. As for 4) reading the code, why not? "Use the source, Luke" is now a programming cliche because it works. It's particularly appropriate for Python packages which are usually distributed in source form and, as you yourself noted, easy to read. -- cheers, Hugh Fisher

2 years, 6 months

Re: [Python-ideas] Why is design-by-contracts not widely adopted?

by Angus Hollands

Hi Marko, I think there are several ways to approach this problem, though am not weighing in on whether DbC is a good thing in Python. I wrote a simple implementation of DbC which is currently a run-time checker. You could, with the appropriate tooling, validate statically too (as with all approaches). In my approach, I use a “proxy” object to allow the contract code to be defined at function definition time. It does mean that some things are not as pretty as one would like - anything that cannot be hooked into with magic methods i.e isinstance, but I think this is acceptable as it makes features like old easier. Also, one hopes that it encourages simpler contract checks as a side-effect. Feel free to take a look - https://github.com/agoose77/pyffel It is by no means well written, but a fun PoC nonetheless. Regards, Angus

2 years, 6 months

Re: [Python-ideas] "old" values in postconditions

by James Lu

Have you looked at the built-in AST module, ast? https://docs.python.org/3/library/ast.html I don’t see anything preventing you from walking the AST Python itself can give you- you’d look for two Set AST nodes if we were to do {{ }}. There’s also the parser built-in module. You can use it if you first use dis.code_info to get the source then re-parse it. It helps with parse trees. Parse trees are generated before the AST I think. You’d use the parser module’s ST objects with the token module’s constants, for example token.LBRACE or token.RBRACE. Have you looked at the built-in dis module? You can use dis.code_info(obj) to get the string of the function. Then you could look for your specified syntax with regex and recompile that with the ast module. Sent from my iPhone > On Sep 25, 2018, at 1:49 AM, Marko Ristin-Kaufmann <marko.ristin(a)gmail.com> wrote: > > Hi James, > Thanks for the feedback! > > I also thought about decompiling the condition to find its AST and figure out what old values are needed. However, this is not so easily done at the moment as all the decompilation libraries I looked at (meta, ucompyle6) are simply struggling to keep up with the development of the python bytecode. In other words, the decompiler needs to be updated with every new version of python which is kind of a loosing race (unless the python devs themselves don't provide this functionality which is not the case as far as I know). > > There is macropy (https://github.com/lihaoyi/macropy) which was suggested on the other thread (https://groups.google.com/forum/#!topic/python-ideas/dmXz_7LH4GI) that I'm currently looking at. > > Cheers, > Marko > > >> On Tue, 25 Sep 2018 at 00:35, James Lu <jamtlu(a)gmail.com> wrote: >> You could disassemble (import dis) the lambda to biew the names of the lambdas. >> >> @before(lambda self, key, _, length, get: self.length(), self.get(key)) >> >> Perhaps you could disassemble the function code and look at all operations or accesses that are done to “old.” and evaluate those expressions before the function runs. Then you could “replace” the expression. >> @post(lambda self, key, old: old.get is None and old.length + 1 == >> self.length()) >> >> Either the system would grab old.get and old.length or be greedy and grab old.get is None and old.length + 1. It would then replace the old.get and old.length with injects that only respond to is None and +1. >> >> Or, a syntax like this >> @post(lambda self, key, old: [old.get(old.key)] is None and [old.self.length() + 1] == >> self.length()) >> >> Where the stuff inside the brackets is evaluated before the decorated function runs. It would be useful for networking functions or functions that do something ephemeral, where data related to the value being accessed needed for the expression no longer exists after the function. >> >> This does conflict with list syntax forever, so maybe either force people to do list((expr,)) or use an alternate syntax like one item set syntax { } or double set syntax {{ }} or double list syntax [[ ]]. Ditto with having to avoid the literals for the normal meaning. >> >> You could modify Python to accept any expression for the lambda function and propose that as a PEP. (Right now it’s hardcoded as a dotted name and optionally a single argument list surrounded by parentheses.) >> >> I suggest that instead of “@before” it’s “@snapshot” and instead of “old” it’s “snapshot”. >> >> Python does have unary plus/minus syntax as well as stream operators (<<, >>) and list slicing syntax and the @ operator and operators & and | if you want to play with syntax. There’s also the line continuation character for crazy lambdas. >> >> Personally I prefer >> @post(lambda self, key, old: {{old.self.get(old.key)}} and {{old.self.length() + 1}} == >> self.length()) >> >> because it’s explicit about what it does (evaluate the expressions within {{ }} before the function runs. I also find it elegant. >> >> Alternatively, inside the {{ }} could be a special scope where locals() is all the arguments @pre could’ve received as a dictionary. For either option you can remove the old parameter from the lambda. Example: >> @post(lambda self, key: {{self.get(key)}} and {{self.length() + 1}} == >> self.length()) >> >> Perhaps the convention should be to write {{ expr }} (with the spaces in between). >> >> You’d probably have to use the ast module to inspect it instead of the dis modul. Then find some way to reconstruct the expressions inside the double brackets- perhaps by reconstructing the AST and compiling it to a code object, or perhaps by finding the part of the string the expression is located. dis can give you the code as a string and you can run a carefully crafted regex on it. >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas(a)python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/

2 years, 6 months

Keyword only argument on function call

by Anders Hovmöller

I have a working implementation for a new syntax which would make using keyword arguments a lot nicer. Wouldn't it be awesome if instead of: foo(a=a, b=b, c=c, d=3, e=e) we could just write: foo(*, a, b, c, d=3, e) and it would mean the exact same thing? This would not just be shorter but would create an incentive for consistent naming across the code base. So the idea is to generalize the * keyword only marker from function to also have the same meaning at the call site: everything after * is a kwarg. With this feature we can now simplify keyword arguments making them more readable and concise. (This syntax does not conflict with existing Python code.) The full PEP-style suggestion is here: https://gist.github.com/boxed/f72221e7e77370be3e5703087c1ba54d I have also written an analysis tool you can use on your code base to see what kind of impact this suggestion might have. It's available at https://gist.github.com/boxed/610b2ba73066c96e9781aed7c0c0b25c . The results for django and twisted are posted as comments to the gist. We've run this on our two big code bases at work (both around 250kloc excluding comments and blank lines). The results show that ~30% of all arguments would benefit from this syntax. Me and my colleague Johan Lübcke have also written an implementation that is available at: https://github.com/boxed/cpython / Anders Hovmöller

2 years, 6 months

Re: [Python-ideas] JS? governance model is worth inspecting

by James Lu

> Which features of the TC39 committee's ECMAscript (ES) language governance > model would be helpful to incorporate into the Python language governance > model? Having “beta” or “alpha” editions of features, special versions of the interpreter people can test out to see if they prefer the version with the new feature. To prevent splintering, the main releases would only support the main feature set. In a worst case scenario, people can compile incompatible code to .pyc before running it.

2 years, 6 months

Re: [Python-ideas] Why is design-by-contracts not widely adopted?

by James Lu

Perhaps it’s because fewer Python functions involve transitioning between states. Web development and statistics don’t involve many state transition. State transitions are where I think I would find it useful to write contracts out explicitly.

2 years, 6 months

Re: [Python-ideas] "old" values in postconditions

by James Lu

You could disassemble (import dis) the lambda to biew the names of the lambdas. @before(lambda self, key, _, length, get: self.length(), self.get(key)) Perhaps you could disassemble the function code and look at all operations or accesses that are done to “old.” and evaluate those expressions before the function runs. Then you could “replace” the expression. @post(lambda self, key, old: old.get is None and old.length + 1 == self.length()) Either the system would grab old.get and old.length or be greedy and grab old.get is None and old.length + 1. It would then replace the old.get and old.length with injects that only respond to is None and +1. Or, a syntax like this @post(lambda self, key, old: [old.get(old.key)] is None and [old.self.length() + 1] == self.length()) Where the stuff inside the brackets is evaluated before the decorated function runs. It would be useful for networking functions or functions that do something ephemeral, where data related to the value being accessed needed for the expression no longer exists after the function. This does conflict with list syntax forever, so maybe either force people to do list((expr,)) or use an alternate syntax like one item set syntax { } or double set syntax {{ }} or double list syntax [[ ]]. Ditto with having to avoid the literals for the normal meaning. You could modify Python to accept any expression for the lambda function and propose that as a PEP. (Right now it’s hardcoded as a dotted name and optionally a single argument list surrounded by parentheses.) I suggest that instead of “@before” it’s “@snapshot” and instead of “old” it’s “snapshot”. Python does have unary plus/minus syntax as well as stream operators (<<, >>) and list slicing syntax and the @ operator and operators & and | if you want to play with syntax. There’s also the line continuation character for crazy lambdas. Personally I prefer @post(lambda self, key, old: {{old.self.get(old.key)}} and {{old.self.length() + 1}} == self.length()) because it’s explicit about what it does (evaluate the expressions within {{ }} before the function runs. I also find it elegant. Alternatively, inside the {{ }} could be a special scope where locals() is all the arguments @pre could’ve received as a dictionary. For either option you can remove the old parameter from the lambda. Example: @post(lambda self, key: {{self.get(key)}} and {{self.length() + 1}} == self.length()) Perhaps the convention should be to write {{ expr }} (with the spaces in between). You’d probably have to use the ast module to inspect it instead of the dis modul. Then find some way to reconstruct the expressions inside the double brackets- perhaps by reconstructing the AST and compiling it to a code object, or perhaps by finding the part of the string the expression is located. dis can give you the code as a string and you can run a carefully crafted regex on it.

2 years, 6 months

← Newer
1
2
3
4
5
6
7
8
9
10
Older →

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

Python-ideas September 2018