Python-ideas March 2018

python-ideas@python.org

88 participants
50 discussions

PEP draft: Unifying function/method classes
by Jeroen Demeyer 03 Apr '18

03 Apr '18

I have prepared a PEP draft for unifying function/method classes. You can find it at https://github.com/jdemeyer/PEP-functions This has not officially been submitted as PEP yet, I want to hear your comments first. Thanks, Jeroen.

7 6

Generalized version of contextlib.close
by Roberto Martínez 02 Apr '18

02 Apr '18

Hi, sometimes I need to use contextlib.close but over methods with a different name, for example stop(), halt(), etc. For those cases I have to write my own contextlib.close specialized version with a hard-coded method name. I think adding a "method" argument to contextlib.close can be very useful: @contextmanager def closing(thing, method="close"): try: yield thing finally: getattr(thing, method)() Or maybe something even more generic: @contextmanager def calling(fn, *args, **kwargs): try: yield finally: fn(*args, **kwargs) Best regards, Roberto

4 3

Re: [Python-ideas] PEP draft: Unifying function/method classes
by Jeroen Demeyer 01 Apr '18

01 Apr '18

On 2018-03-31 18:09, Steven D'Aprano wrote: > It seems like a huge amount of work What is a huge amount of work? Writing the PEP? Implementing the PEP? Using the PEP? Adapting existing Python code to the PEP? > Why isn't the answer to provide a hook to support introspection? That is a lot easier said than done. How would you do that? The problem is that multiple classes are involved, at least "builtin_function_or_method" and "method_descriptor". So you'll end up with at least 3 classes (those 2 plus the existing "function") supporting introspection. With my proposal, this is implemented only once. And do you expect users to implement those hooks or do you expect Python to do it? If you want to do it in Python, you'll need to add a new class anyway. That doesn't look simpler than my PEP. That being said, if there is a general consensus that this is the right thing to do, I'll go for it. However, I'm afraid that people will complain that I'm complicating functions in Python even more. Second: while introspection was the initial motivation, it does make sense to unify built-in functions and Python functions. For example: why are unbound methods just functions in Python classes but not in extension types? If you are going to do some reorganization of function classes anyway, you might as well do it properly. I claim that functions will be *easier* to work with in Python when my PEP is accepted, both for the end user as well as for the implementor of custom function classes. > Seems to me that if you want a fast, exact (no subclasses) check, you > should use "type(obj) is Class" rather than isinstance. If the *only* > reason to prohibit subclassing is to make isinstance a bit faster, > I don't think that's a good enough reason. I didn't really mean "isinstance" literally, I was mostly thinking of the C API. I agree that it's not clear. Do you happen to know why the existing function classes in Python disallow subclassing? I assumed that it was for exactly this reason. Jeroen.

3 2

Re: [Python-ideas] PEP draft: Unifying function/method classes
by Jeroen Demeyer 31 Mar '18

31 Mar '18

On 2018-03-31 21:12, Terry Reedy wrote: > I would be all for more of the builtins and stdlib being converted. > Can't 3rd-party C code use ArgumentClinic? ArgumentClinic stores the signature as text. For default values, only a few specific classes are supported. I want to support arbitrary Python objects. > Sourcefile is meaningless when there is no source file. This is motivated by Cython, in which case there *is* a genuine and useful source file.

1 0

Fixing class scope brainstorm
by Guido van Rossum 30 Mar '18

30 Mar '18

In the PEP 572 threads there's some grumbling about class scopes. Here's a random brainstorm. How about we change the scoping rules so that only functions defined with 'def' (or inside one of those) cannot see the class scope, and comprehensions and lambdas treat the class scope as an outer scope so they can close over class variables? Not sure what should happen to nested classes, they should probably be lumped in with the 'def' scopes. -- --Guido van Rossum (python.org/~guido)

10 25

Sets, Dictionaries
by Julia Kim 30 Mar '18

30 Mar '18

My suggestion is to change the syntax for creating an empty set and an empty dictionary as following. an_empty_set = {} an_empty_dictionary = {:} Compatibility issues could be resolved with a program which takes a Python program (codes) as a text and edits it. Sent from my iPhone

7 8

threading lock and isinstance
by Jacco van Dorp 29 Mar '18

29 Mar '18

Currently, you cannot use isinstance checks on threading locks, because they're created by a factory function instead of being actual classes. Now that we've got __subclasshook__ and __instancecheck__, is there still a reason other than "history" that we can't use isinstance here ? There could exist a thin wrapper class along the lines of: class Lock: def __new__(): return __allocate_lock() # Call factory function @classmethod def __subclasshook__(cls, inst): # code... As far as I can think, no code would be broken by this - even current introspection which we're trying to replace would work just fine. My C experience is rather limited, so i dont know if it's hard to write the subclass /isinstance checks. While probably not that important, would people consider this to be a good idea ? (I found this bug report: https://bugs.python.org/issue3352 which has a post of Nick Coghlan from 2008-07-14 22:25 where this is mentioned - but in respect to multiprocessing. However, that thread is rather old and dead. I could not find any other references to this.)

2 1

PEP 572 version 2: Statement-Local Name Bindings
by Chris Angelico 29 Mar '18

29 Mar '18

After dozens of posts and a wide variety of useful opinions and concerns being raised, here is the newest version of PEP 572 for your debating pleasure. Formatted version: https://www.python.org/dev/peps/pep-0572/ There are now several more examples, greater clarity in edge cases, and improved wording of the actual proposal and specifications. Also, the reference implementation has been significantly enhanced, for those who wish to try this themselves. ChrisA PEP: 572 Title: Syntax for Statement-Local Name Bindings Author: Chris Angelico <rosuav(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 28-Feb-2018 Python-Version: 3.8 Post-History: 28-Feb-2018, 02-Mar-2018 Abstract ======== Programming is all about reusing code rather than duplicating it. When an expression needs to be used twice in quick succession but never again, it is convenient to assign it to a temporary name with small scope. By permitting name bindings to exist within a single statement only, we make this both convenient and safe against name collisions. Rationale ========= When a subexpression is used multiple times in a list comprehension, there are currently several ways to spell this, none of which is universally accepted as ideal. A statement-local name allows any subexpression to be temporarily captured and then used multiple times. Additionally, this syntax can in places be used to remove the need to write an infinite loop with a ``break`` in it. Capturing part of a ``while`` loop's condition can improve the clarity of the loop header while still making the actual value available within the loop body. Syntax and semantics ==================== In any context where arbitrary Python expressions can be used, a named expression can appear. This must be parenthesized for clarity, and is of the form ``(expr as NAME)`` where ``expr`` is any valid Python expression, and ``NAME`` is a simple name. The value of such a named expression is the same as the incorporated expression, with the additional side-effect that NAME is bound to that value in all retrievals for the remainder of the current statement. Just as function-local names shadow global names for the scope of the function, statement-local names shadow other names for that statement. They can also shadow each other, though actually doing this should be strongly discouraged in style guides. Assignment to statement-local names is ONLY through this syntax. Regular assignment to the same name will remove the statement-local name and affect the name in the surrounding scope (function, class, or module). Statement-local names never appear in locals() or globals(), and cannot be closed over by nested functions. Execution order and its consequences ------------------------------------ Since the statement-local name binding lasts from its point of execution to the end of the current statement, this can potentially cause confusion when the actual order of execution does not match the programmer's expectations. Some examples:: # A simple statement ends at the newline or semicolon. a = (1 as y) print(y) # NameError # The assignment ignores the SLNB - this adds one to 'a' a = (a + 1 as a) # Compound statements usually enclose everything... if (re.match(...) as m): print(m.groups(0)) print(m) # NameError # ... except when function bodies are involved... if (input("> ") as cmd): def run_cmd(): print("Running command", cmd) # NameError # ... but function *headers* are executed immediately if (input("> ") as cmd): def run_cmd(cmd=cmd): # Capture the value in the default arg print("Running command", cmd) # Works Some of these examples should be considered *bad code* and rejected by code review and/or linters; they are not, however, illegal. Example usage ============= These list comprehensions are all approximately equivalent:: # Calling the function twice stuff = [[f(x), x/f(x)] for x in range(5)] # External helper function def pair(x, value): return [value, x/value] stuff = [pair(x, f(x)) for x in range(5)] # Inline helper function stuff = [(lambda y: [y,x/y])(f(x)) for x in range(5)] # Extra 'for' loop - see also Serhiy's optimization stuff = [[y, x/y] for x in range(5) for y in [f(x)]] # Iterating over a genexp stuff = [[y, x/y] for x, y in ((x, f(x)) for x in range(5))] # Expanding the comprehension into a loop stuff = [] for x in range(5): y = f(x) stuff.append([y, x/y]) # Wrapping the loop in a generator function def g(): for x in range(5): y = f(x) yield [y, x/y] stuff = list(g) # Using a statement-local name stuff = [[(f(x) as y), x/y] for x in range(5)] If calling ``f(x)`` is expensive or has side effects, the clean operation of the list comprehension gets muddled. Using a short-duration name binding retains the simplicity; while the extra ``for`` loop does achieve this, it does so at the cost of dividing the expression visually, putting the named part at the end of the comprehension instead of the beginning. Statement-local name bindings can be used in any context, but should be avoided where regular assignment can be used, just as ``lambda`` should be avoided when ``def`` is an option. As the name's scope extends to the full current statement, even a block statement, this can be used to good effect in the header of an ``if`` or ``while`` statement:: # Current Python, not caring about function return value while input("> ") != "quit": print("You entered a command.") # Current Python, capturing return value - four-line loop header while True: command = input("> "); if command == "quit": break print("You entered:", command) # Proposed alternative to the above while (input("> ") as command) != "quit": print("You entered:", command) # See, for instance, Lib/pydoc.py if (re.search(pat, text) as match): print("Found:", match.group(0)) while (sock.read() as data): print("Received data:", data) Particularly with the ``while`` loop, this can remove the need to have an infinite loop, an assignment, and a condition. It also creates a smooth parallel between a loop which simply uses a function call as its condition, and one which uses that as its condition but also uses the actual value. Performance costs ================= The cost of SLNBs must be kept to a minimum, particularly when they are not used; the normal case MUST NOT be measurably penalized. SLNBs are expected to be uncommon, and using many of them in a single function should definitely be discouraged. Thus the current implementation uses a linked list of SLNB cells, with the absence of such a list being the normal case. This list is used for code compilation only; once a function's bytecode has been baked in, execution of that bytecode has no performance cost compared to regular assignment. Other Python implementations may choose to do things differently, but a zero run-time cost is strongly recommended, as is a minimal compile-time cost in the case where no SLNBs are used. Open questions ============== 1. What happens if the name has already been used? ``(x, (1 as x), x)`` Currently, prior usage functions as if the named expression did not exist (following the usual lookup rules); the new name binding will shadow the other name from the point where it is evaluated until the end of the statement. Is this acceptable? Should it raise a syntax error or warning? 2. Syntactic confusion in ``except`` statements. While technically unambiguous, it is potentially confusing to humans. In Python 3.7, parenthesizing ``except (Exception as e):`` is illegal, and there is no reason to capture the exception type (as opposed to the exception instance, as is done by the regular syntax). Should this be made outright illegal, to prevent confusion? Can it be left to linters? It may also (and independently) be of value to use a subscope for the normal except clause binding, such that ``except Exception as e:`` will no longer unbind a previous use of the name ``e``. 3. Similar confusion in ``with`` statements, with the difference that there is good reason to capture the result of an expression, and it is also very common for ``__enter__`` methods to return ``self``. In many cases, ``with expr as name:`` will do the same thing as ``with (expr as name):``, adding to the confusion. 4. Should closures be able to refer to statement-local names? Either way, there will be edge cases that make no sense. Assigning to a name will "push through" the SLNB and bind to the regular name; this means that a statement ``x = x`` will promote the SLNB to full name, and thus has an impact. Closing over statement-local names, however, introduces scope and lifetime confusions, as it then becomes possible to have two functions in almost the same context, closing over the same name, referring to two different cells. Alternative proposals ===================== Proposals of this nature have come up frequently on python-ideas. Below are a number of alternative syntaxes, some of them specific to comprehensions, which have been rejected in favour of the one given above. 1. ``where``, ``let``, ``given``:: stuff = [(y, x/y) where y = f(x) for x in range(5)] stuff = [(y, x/y) let y = f(x) for x in range(5)] stuff = [(y, x/y) given y = f(x) for x in range(5)] This brings the subexpression to a location in between the 'for' loop and the expression. It introduces an additional language keyword, which creates conflicts. Of the three, ``where`` reads the most cleanly, but also has the greatest potential for conflict (eg SQLAlchemy and numpy have ``where`` methods, as does ``tkinter.dnd.Icon`` in the standard library). 2. ``with``:: stuff = [(y, x/y) with y = f(x) for x in range(5)] As above, but reusing the `with` keyword. Doesn't read too badly, and needs no additional language keyword. Is restricted to comprehensions, though, and cannot as easily be transformed into "longhand" for-loop syntax. Has the C problem that an equals sign in an expression can now create a name binding, rather than performing a comparison. 3. ``with... as``:: stuff = [(y, x/y) with f(x) as y for x in range(5)] As per option 2, but using ``as`` in place of the equals sign. Aligns syntactically with other uses of ``as`` for name binding, but a simple transformation to for-loop longhand would create drastically different semantics; the meaning of ``with`` inside a comprehension would be completely different from the meaning as a stand-alone statement. 4. ``EXPR as NAME`` without parentheses:: stuff = [[f(x) as y, x/y] for x in range(5)] Omitting the parentheses from this PEP's proposed syntax introduces many syntactic ambiguities. 5. Adorning statement-local names with a leading dot:: stuff = [[(f(x) as .y), x/.y] for x in range(5)] This has the advantage that leaked usage can be readily detected, removing some forms of syntactic ambiguity. However, this would be the only place in Python where a variable's scope is encoded into its name, making refactoring harder. This syntax is quite viable, and could be promoted to become the current recommendation if its advantages are found to outweigh its cost. 6. Allowing ``(EXPR as NAME)`` to assign to any form of name. This is exactly the same as the promoted proposal, save that the name is bound in the same scope that it would otherwise have. Any expression can assign to any name, just as it would if the ``=`` operator had been used. Discrepancies in the current implementation =========================================== 1. SLNBs are implemented using a special (and mostly-invisible) name mangling. They may sometimes appear in globals() and/or locals() with their simple or mangled names (but buggily and unreliably). They should be suppressed as though they were guinea pigs. References ========== .. [1] Proof of concept / reference implementation (https://github.com/Rosuav/cpython/tree/statement-local-variables) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

21 91

Re: [Python-ideas] PEP proposal: unifying function/method classes
by Jeroen Demeyer 28 Mar '18

28 Mar '18

On 2018-03-23 00:36, Antoine Pitrou wrote: > It does make sense, since the proposal sounds ambitious (and perhaps > impossible without breaking compatibility). Well, *some* breakage of backwards compatibility will be unavoidable. My plan (just a plan for now!) is to preserve backwards compatibility in the following ways: * Existing Python attributes of functions/methods should continue to exist and behave the same * The inspect module should give the same results as now (by changing the implementation of some of the functions in inspect to match the new classes) * Everything from the documented Python/C API. This means that I might break compatibility in the following ways: * Changing the classes of functions/methods (this is the whole point of this PEP). So anything involving isinstance() checks might break. * The undocumented parts of the Python/C API, in particular the C structure.

4 4

PEP 572: Statement-Local Name Bindings, take three!
by Cammil Taank 28 Mar '18

28 Mar '18

> > ... From here, > the most important concern and question is: Is there any other syntax > or related proposal that ought to be mentioned here? I am not sure if this is valid, but perhaps this is an alternative syntax which might be simpler: ``name! expr`` So for example, instead of: stuff = [[(f(x) as y), x/y] for x in range(5)] stuff = [[y! f(x), x/y] for x in range(5)] As far as I can tell there would be no conflicts with the current uses of "!". One potential source of ambiguity would be in: x = y! a + b # should y be a or (a + b)? I think this is solved by requiring the target expression to be non-greedy. If you want a larger named expression, you can always use parenthesis. i.e. ``x = y! (z + z)`` I feel brevity and minimised punctuation are important for the adoption of statement-locals, and personally I feel it reads well. I also happen to prefer the name preceding the expression, though I suspect this is quite subjective. Also, apologies if I have grossly misunderstood something. Cammil

1 0

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

Python-ideas March 2018