Python-ideas September 2018

python-ideas@python.org

108 participants
93 discussions

Fwd: Keyword only argument on function call

by David Mertz

I'm still not sure why all this focus on new syntax or convoluted IDE enhancements. I presented a very simple utility function that accomplishes exactly the started goal of DRY in keyword arguments. Yes, I wrote a first version that was incomplete. And perhaps these 8-9 lines miss some corner case. But the basic goal is really, really easy to accomplish with existing Python. >>> import inspect >>> def reach(name): ... for f in inspect.stack(): ... if name in f[0].f_locals: ... return f[0].f_locals[name] ... return None ... >>> def use(names): ... kws = {} ... for name in names.split(): ... kws[name] = reach(name) ... return kws ... >>> def function(a=11, b=22, c=33, d=44): ... print(a, b, c, d) ... >>> function(a=77, **use('b d')) 77 None 33 None >>> def foo(): ... a, b, c = 1, 2, 3 ... function(a=77, **use('b d')) ... >>> foo() 77 2 33 None On Tue, Sep 25, 2018, 6:31 AM Anders Hovmöller <boxed(a)killingar.net> wrote: > Hi, > > I'd like to reopen this discussion if anyone is interested. Some things > have changed since I wrote my original proposal so I'll first summarize: > > 1. People seem to prefer the syntax `foo(=a)` over the syntax I suggested. > I believe this is even more trivial to implement in CPython than my > original proposal anyway... > 2. I have updated my analysis tool: > https://gist.github.com/boxed/610b2ba73066c96e9781aed7c0c0b25c It will > now also give you statistics on the number of arguments function calls > have. I would love to see some statistics for other closed source programs > you might be working on and how big those code bases are. > 3. I have made a sort-of implementation with MacroPy: > https://github.com/boxed/macro-kwargs/blob/master/test.py I think this is > a dead end, but it was easy to implement and fun to try! > 4. I have also recently had the idea that a foo=foo type pattern could be > handled in for example PyCharm as a code folding feature (and maybe as a > completion feature). > > I still think that changing Pythons syntax is the right way to go in the > long run but with point 4 above one could experience what this feature > would feel like without running a custom version of Python and without > changing your code. I admit to a lot of trepidation about wading into > PyCharms code though, I have tried to do this once before and I gave up. > > Any thoughts? > > / Anders > _______________________________________________ > Python-ideas mailing list > Python-ideas(a)python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ >

1 year, 9 months

Re: [Python-ideas] "old" values in postconditions

by James Lu

Hmm, I was wrong: there is no reliable way to get the code of a lambda function. If it was possible to execute all code paths of the function, we could monkey patch the builtins so { } used our own custom set class. Alternatively, the decorator could also accept a string. Or maybe we could send a PEP to add the .func_code attribute to lambdas as well as normal functions. There’s also a technique online where they find the lambda’s source by locating the file the function was defined in and then removing the irrelevant parts, but that just doesn’t sound practical to me. There’s also MacroPy. I think the best solution would be to mock the old object and record the operations done to the object, like the other replier gave a PoC of. Proposed syntax from icontract import post, old @post(lambda: ..., key=old.self.key(), ) Sent from my iPhone > On Sep 25, 2018, at 1:49 AM, Marko Ristin-Kaufmann <marko.ristin(a)gmail.com> wrote: > > Hi James, > Thanks for the feedback! > > I also thought about decompiling the condition to find its AST and figure out what old values are needed. However, this is not so easily done at the moment as all the decompilation libraries I looked at (meta, ucompyle6) are simply struggling to keep up with the development of the python bytecode. In other words, the decompiler needs to be updated with every new version of python which is kind of a loosing race (unless the python devs themselves don't provide this functionality which is not the case as far as I know). > > There is macropy (https://github.com/lihaoyi/macropy) which was suggested on the other thread (https://groups.google.com/forum/#!topic/python-ideas/dmXz_7LH4GI) that I'm currently looking at. > > Cheers, > Marko > > >> On Tue, 25 Sep 2018 at 00:35, James Lu <jamtlu(a)gmail.com> wrote: >> You could disassemble (import dis) the lambda to biew the names of the lambdas. >> >> @before(lambda self, key, _, length, get: self.length(), self.get(key)) >> >> Perhaps you could disassemble the function code and look at all operations or accesses that are done to “old.” and evaluate those expressions before the function runs. Then you could “replace” the expression. >> @post(lambda self, key, old: old.get is None and old.length + 1 == >> self.length()) >> >> Either the system would grab old.get and old.length or be greedy and grab old.get is None and old.length + 1. It would then replace the old.get and old.length with injects that only respond to is None and +1. >> >> Or, a syntax like this >> @post(lambda self, key, old: [old.get(old.key)] is None and [old.self.length() + 1] == >> self.length()) >> >> Where the stuff inside the brackets is evaluated before the decorated function runs. It would be useful for networking functions or functions that do something ephemeral, where data related to the value being accessed needed for the expression no longer exists after the function. >> >> This does conflict with list syntax forever, so maybe either force people to do list((expr,)) or use an alternate syntax like one item set syntax { } or double set syntax {{ }} or double list syntax [[ ]]. Ditto with having to avoid the literals for the normal meaning. >> >> You could modify Python to accept any expression for the lambda function and propose that as a PEP. (Right now it’s hardcoded as a dotted name and optionally a single argument list surrounded by parentheses.) >> >> I suggest that instead of “@before” it’s “@snapshot” and instead of “old” it’s “snapshot”. >> >> Python does have unary plus/minus syntax as well as stream operators (<<, >>) and list slicing syntax and the @ operator and operators & and | if you want to play with syntax. There’s also the line continuation character for crazy lambdas. >> >> Personally I prefer >> @post(lambda self, key, old: {{old.self.get(old.key)}} and {{old.self.length() + 1}} == >> self.length()) >> >> because it’s explicit about what it does (evaluate the expressions within {{ }} before the function runs. I also find it elegant. >> >> Alternatively, inside the {{ }} could be a special scope where locals() is all the arguments @pre could’ve received as a dictionary. For either option you can remove the old parameter from the lambda. Example: >> @post(lambda self, key: {{self.get(key)}} and {{self.length() + 1}} == >> self.length()) >> >> Perhaps the convention should be to write {{ expr }} (with the spaces in between). >> >> You’d probably have to use the ast module to inspect it instead of the dis modul. Then find some way to reconstruct the expressions inside the double brackets- perhaps by reconstructing the AST and compiling it to a code object, or perhaps by finding the part of the string the expression is located. dis can give you the code as a string and you can run a carefully crafted regex on it. >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas(a)python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/

1 year, 9 months

Why is design-by-contracts not widely adopted?

by Hugh Fisher

> Date: Mon, 24 Sep 2018 09:46:16 +0200 > From: Marko Ristin-Kaufmann <marko.ristin(a)gmail.com> > To: Python-Ideas <python-ideas(a)python.org> > Subject: Re: [Python-ideas] Why is design-by-contracts not widely > adopted? > Message-ID: > <CAGu4bVB4=Bou+DhoDzayx=i7eQ2Gr8KDDOFONeaB-Qvt8LZo2A(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" [munch] > Their users would hugely benefit from a more mature > and standardized contracts library with informative violation messages. Will respond in another message, because it's a big topic. > I really don't see how DbC has to do with duck typing (unless you reduce it > to mere isinstance conditions, which would simply be a straw-man argument) > -- could you please clarify? I argue that Design by Contract doesn't make sense for Python and other dynamically typed, duck typed languages because it's contrary to how the language, and the programmer, expects to work. In Python we can write something like: def foo(x): x.bar(y) What's the type of x? What's the type of y? What is the contract of bar? Don't know, don't care. x, or y, can be an instance, a class, a module, a proxy for a remote web service. The only "contract" is that object x will respond to message bar that takes one argument. Object x, do whatever you want with it. And that's a feature, not a bug, not bad design. It follows Postel's Law for Internet protocols of being liberal in what you accept. It follows the Agile principle of valuing working software over comprehensive doco. It allows software components to be glued together quickly and easily. It's a style of programming that has been successful for many years, not just in Python but also in Lisp and Smalltalk and Perl and JavaScript. It works. Not for everything. If I were writing the avionics control routines for a helicopter gas turbine, I'd use formal notation and static type checking and preconditions and whatnot. But I wouldn't be using Python either. > As soon as you need to document your code, and > this is what most modules have to do in teams of more than one person > (especially so if you are developing a library for a wider audience), you > need to write down the contracts. Please see above where I tried to > explained that 2-5) are inferior approaches to documenting contracts > compared to 1). You left off option 6), plain text. Comments. Docstrings. README files. Web pages. Books. In my experience, this is what most people consider documentation. A good book, a good blog post, can explain more about how a library works and what the implementation requirements and restrictions are than formal contract notation. In particular, contracts in Eiffel don't explain *why* they're there. As for 4) reading the code, why not? "Use the source, Luke" is now a programming cliche because it works. It's particularly appropriate for Python packages which are usually distributed in source form and, as you yourself noted, easy to read. -- cheers, Hugh Fisher

1 year, 9 months

Re: [Python-ideas] Why is design-by-contracts not widely adopted?

by Angus Hollands

Hi Marko, I think there are several ways to approach this problem, though am not weighing in on whether DbC is a good thing in Python. I wrote a simple implementation of DbC which is currently a run-time checker. You could, with the appropriate tooling, validate statically too (as with all approaches). In my approach, I use a “proxy” object to allow the contract code to be defined at function definition time. It does mean that some things are not as pretty as one would like - anything that cannot be hooked into with magic methods i.e isinstance, but I think this is acceptable as it makes features like old easier. Also, one hopes that it encourages simpler contract checks as a side-effect. Feel free to take a look - https://github.com/agoose77/pyffel It is by no means well written, but a fun PoC nonetheless. Regards, Angus

1 year, 9 months

Re: [Python-ideas] "old" values in postconditions

by James Lu

Have you looked at the built-in AST module, ast? https://docs.python.org/3/library/ast.html I don’t see anything preventing you from walking the AST Python itself can give you- you’d look for two Set AST nodes if we were to do {{ }}. There’s also the parser built-in module. You can use it if you first use dis.code_info to get the source then re-parse it. It helps with parse trees. Parse trees are generated before the AST I think. You’d use the parser module’s ST objects with the token module’s constants, for example token.LBRACE or token.RBRACE. Have you looked at the built-in dis module? You can use dis.code_info(obj) to get the string of the function. Then you could look for your specified syntax with regex and recompile that with the ast module. Sent from my iPhone > On Sep 25, 2018, at 1:49 AM, Marko Ristin-Kaufmann <marko.ristin(a)gmail.com> wrote: > > Hi James, > Thanks for the feedback! > > I also thought about decompiling the condition to find its AST and figure out what old values are needed. However, this is not so easily done at the moment as all the decompilation libraries I looked at (meta, ucompyle6) are simply struggling to keep up with the development of the python bytecode. In other words, the decompiler needs to be updated with every new version of python which is kind of a loosing race (unless the python devs themselves don't provide this functionality which is not the case as far as I know). > > There is macropy (https://github.com/lihaoyi/macropy) which was suggested on the other thread (https://groups.google.com/forum/#!topic/python-ideas/dmXz_7LH4GI) that I'm currently looking at. > > Cheers, > Marko > > >> On Tue, 25 Sep 2018 at 00:35, James Lu <jamtlu(a)gmail.com> wrote: >> You could disassemble (import dis) the lambda to biew the names of the lambdas. >> >> @before(lambda self, key, _, length, get: self.length(), self.get(key)) >> >> Perhaps you could disassemble the function code and look at all operations or accesses that are done to “old.” and evaluate those expressions before the function runs. Then you could “replace” the expression. >> @post(lambda self, key, old: old.get is None and old.length + 1 == >> self.length()) >> >> Either the system would grab old.get and old.length or be greedy and grab old.get is None and old.length + 1. It would then replace the old.get and old.length with injects that only respond to is None and +1. >> >> Or, a syntax like this >> @post(lambda self, key, old: [old.get(old.key)] is None and [old.self.length() + 1] == >> self.length()) >> >> Where the stuff inside the brackets is evaluated before the decorated function runs. It would be useful for networking functions or functions that do something ephemeral, where data related to the value being accessed needed for the expression no longer exists after the function. >> >> This does conflict with list syntax forever, so maybe either force people to do list((expr,)) or use an alternate syntax like one item set syntax { } or double set syntax {{ }} or double list syntax [[ ]]. Ditto with having to avoid the literals for the normal meaning. >> >> You could modify Python to accept any expression for the lambda function and propose that as a PEP. (Right now it’s hardcoded as a dotted name and optionally a single argument list surrounded by parentheses.) >> >> I suggest that instead of “@before” it’s “@snapshot” and instead of “old” it’s “snapshot”. >> >> Python does have unary plus/minus syntax as well as stream operators (<<, >>) and list slicing syntax and the @ operator and operators & and | if you want to play with syntax. There’s also the line continuation character for crazy lambdas. >> >> Personally I prefer >> @post(lambda self, key, old: {{old.self.get(old.key)}} and {{old.self.length() + 1}} == >> self.length()) >> >> because it’s explicit about what it does (evaluate the expressions within {{ }} before the function runs. I also find it elegant. >> >> Alternatively, inside the {{ }} could be a special scope where locals() is all the arguments @pre could’ve received as a dictionary. For either option you can remove the old parameter from the lambda. Example: >> @post(lambda self, key: {{self.get(key)}} and {{self.length() + 1}} == >> self.length()) >> >> Perhaps the convention should be to write {{ expr }} (with the spaces in between). >> >> You’d probably have to use the ast module to inspect it instead of the dis modul. Then find some way to reconstruct the expressions inside the double brackets- perhaps by reconstructing the AST and compiling it to a code object, or perhaps by finding the part of the string the expression is located. dis can give you the code as a string and you can run a carefully crafted regex on it. >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas(a)python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/

1 year, 9 months

Keyword only argument on function call

by Anders Hovmöller

I have a working implementation for a new syntax which would make using keyword arguments a lot nicer. Wouldn't it be awesome if instead of: foo(a=a, b=b, c=c, d=3, e=e) we could just write: foo(*, a, b, c, d=3, e) and it would mean the exact same thing? This would not just be shorter but would create an incentive for consistent naming across the code base. So the idea is to generalize the * keyword only marker from function to also have the same meaning at the call site: everything after * is a kwarg. With this feature we can now simplify keyword arguments making them more readable and concise. (This syntax does not conflict with existing Python code.) The full PEP-style suggestion is here: https://gist.github.com/boxed/f72221e7e77370be3e5703087c1ba54d I have also written an analysis tool you can use on your code base to see what kind of impact this suggestion might have. It's available at https://gist.github.com/boxed/610b2ba73066c96e9781aed7c0c0b25c . The results for django and twisted are posted as comments to the gist. We've run this on our two big code bases at work (both around 250kloc excluding comments and blank lines). The results show that ~30% of all arguments would benefit from this syntax. Me and my colleague Johan Lübcke have also written an implementation that is available at: https://github.com/boxed/cpython / Anders Hovmöller

1 year, 9 months

Re: [Python-ideas] JS? governance model is worth inspecting

by James Lu

> Which features of the TC39 committee's ECMAscript (ES) language governance > model would be helpful to incorporate into the Python language governance > model? Having “beta” or “alpha” editions of features, special versions of the interpreter people can test out to see if they prefer the version with the new feature. To prevent splintering, the main releases would only support the main feature set. In a worst case scenario, people can compile incompatible code to .pyc before running it.

1 year, 9 months

Re: [Python-ideas] Why is design-by-contracts not widely adopted?

by James Lu

Perhaps it’s because fewer Python functions involve transitioning between states. Web development and statistics don’t involve many state transition. State transitions are where I think I would find it useful to write contracts out explicitly.

1 year, 9 months

Re: [Python-ideas] "old" values in postconditions

by James Lu

You could disassemble (import dis) the lambda to biew the names of the lambdas. @before(lambda self, key, _, length, get: self.length(), self.get(key)) Perhaps you could disassemble the function code and look at all operations or accesses that are done to “old.” and evaluate those expressions before the function runs. Then you could “replace” the expression. @post(lambda self, key, old: old.get is None and old.length + 1 == self.length()) Either the system would grab old.get and old.length or be greedy and grab old.get is None and old.length + 1. It would then replace the old.get and old.length with injects that only respond to is None and +1. Or, a syntax like this @post(lambda self, key, old: [old.get(old.key)] is None and [old.self.length() + 1] == self.length()) Where the stuff inside the brackets is evaluated before the decorated function runs. It would be useful for networking functions or functions that do something ephemeral, where data related to the value being accessed needed for the expression no longer exists after the function. This does conflict with list syntax forever, so maybe either force people to do list((expr,)) or use an alternate syntax like one item set syntax { } or double set syntax {{ }} or double list syntax [[ ]]. Ditto with having to avoid the literals for the normal meaning. You could modify Python to accept any expression for the lambda function and propose that as a PEP. (Right now it’s hardcoded as a dotted name and optionally a single argument list surrounded by parentheses.) I suggest that instead of “@before” it’s “@snapshot” and instead of “old” it’s “snapshot”. Python does have unary plus/minus syntax as well as stream operators (<<, >>) and list slicing syntax and the @ operator and operators & and | if you want to play with syntax. There’s also the line continuation character for crazy lambdas. Personally I prefer @post(lambda self, key, old: {{old.self.get(old.key)}} and {{old.self.length() + 1}} == self.length()) because it’s explicit about what it does (evaluate the expressions within {{ }} before the function runs. I also find it elegant. Alternatively, inside the {{ }} could be a special scope where locals() is all the arguments @pre could’ve received as a dictionary. For either option you can remove the old parameter from the lambda. Example: @post(lambda self, key: {{self.get(key)}} and {{self.length() + 1}} == self.length()) Perhaps the convention should be to write {{ expr }} (with the spaces in between). You’d probably have to use the ast module to inspect it instead of the dis modul. Then find some way to reconstruct the expressions inside the double brackets- perhaps by reconstructing the AST and compiling it to a code object, or perhaps by finding the part of the string the expression is located. dis can give you the code as a string and you can run a carefully crafted regex on it.

1 year, 9 months

Asynchronous exception handling around with/try statement borders

by Erik Bray

Hi folks, I normally wouldn't bring something like this up here, except I think that there is possibility of something to be done--a language documentation clarification if nothing else, though possibly an actual code change as well. I've been having an argument with a colleague over the last couple days over the proper way order of statements when setting up a try/finally to perform cleanup of some action. On some level we're both being stubborn I think, and I'm not looking for resolution as to who's right/wrong or I wouldn't bring it to this list in the first place. The original argument was over setting and later restoring os.environ, but we ended up arguing over threading.Lock.acquire/release which I think is a more interesting example of the problem, and he did raise a good point that I do want to bring up. </prologue> My colleague's contention is that given lock = threading.Lock() this is simply *wrong*: lock.acquire() try: do_something() finally: lock.release() whereas this is okay: with lock: do_something() Ignoring other details of how threading.Lock is actually implemented, assuming that Lock.__enter__ calls acquire() and Lock.__exit__ calls release() then as far as I've known ever since Python 2.5 first came out these two examples are semantically *equivalent*, and I can't find any way of reading PEP 343 or the Python language reference that would suggest otherwise. However, there *is* a difference, and has to do with how signals are handled, particularly w.r.t. context managers implemented in C (hence we are talking CPython specifically): If Lock.__enter__ is a pure Python method (even if it maybe calls some C methods), and a SIGINT is handled during execution of that method, then in almost all cases a KeyboardInterrupt exception will be raised from within Lock.__enter__--this means the suite under the with: statement is never evaluated, and Lock.__exit__ is never called. You can be fairly sure the KeyboardInterrupt will be raised from somewhere within a pure Python Lock.__enter__ because there will usually be at least one remaining opcode to be evaluated, such as RETURN_VALUE. Because of how delayed execution of signal handlers is implemented in the pyeval main loop, this means the signal handler for SIGINT will be called *before* RETURN_VALUE, resulting in the KeyboardInterrupt exception being raised. Standard stuff. However, if Lock.__enter__ is a PyCFunction things are quite different. If you look at how the SETUP_WITH opcode is implemented, it first calls the __enter__ method with _PyObjet_CallNoArg. If this returns NULL (i.e. an exception occurred in __enter__) then "goto error" is executed and the exception is raised. However if it returns non-NULL the finally block is set up with PyFrame_BlockSetup and execution proceeds to the next opcode. At this point a potentially waiting SIGINT is handled, resulting in KeyboardInterrupt being raised while inside the with statement's suite, and finally block, and hence Lock.__exit__ are entered. Long story short, because Lock.__enter__ is a C function, assuming that it succeeds normally then with lock: do_something() always guarantees that Lock.__exit__ will be called if a SIGINT was handled inside Lock.__enter__, whereas with lock.acquire() try: ... finally: lock.release() there is at last a small possibility that the SIGINT handler is called after the CALL_FUNCTION op but before the try/finally block is entered (e.g. before executing POP_TOP or SETUP_FINALLY). So the end result is that the lock is held and never released after the KeyboardInterrupt (whether or not it's handled somehow). Whereas, again, if Lock.__enter__ is a pure Python function there's less likely to be any difference (though I don't think the possibility can be ruled out entirely). At the very least I think this quirk of CPython should be mentioned somewhere (since in all other cases the semantic meaning of the "with:" statement is clear). However, I think it might be possible to gain more consistency between these cases if pending signals are checked/handled after any direct call to PyCFunction from within the ceval loop. Sorry for the tl;dr; any thoughts?

1 year, 9 months

← Newer
1
2
3
4
5
6
7
8
9
10
Older →

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

Python-ideas September 2018