Mailman 3 October 2021 - Python-ideas

zip(x, y, z, strict=True)
by Ram Rachum Dec. 1, 2022

Dec. 1, 2022

Here's something that would have saved me some debugging yesterday: >>> zipped = zip(x, y, z, strict=True) I suggest that `strict=True` would ensure that all the iterables have been exhausted, raising an exception otherwise. This is useful in cases where you're assuming that the iterables all have the same lengths. When your assumption is wrong, you currently just get a shorter result, and it could take you a while to figure out why it's happening. What do you think?

27 179

disallow assignment to unknown ssl.SSLContext attributes
by Thomas Grainger June 10, 2022

June 10, 2022

I was debugging some code that was using TLSv1.2 when I expected it to only support TLSv1.3, I tracked it down to a call to: context.miunimum_version = ssl.TLSVersion.TLSv1_3 it should have been: context.minimum_version = ssl.TLSVersion.TLSv1_3 I'd like invalid attribute assignment to be prevented at runtime

13 32

Bringing the print statement back
by Guido van Rossum March 2, 2022

March 2, 2022

In Python 3.10 we will no longer be burdened by the old parser (though 3rd party tooling needs to catch up). One thing that the PEG parser makes possible in about 20 lines of code is something not entirely different from the old print statement. I have a prototype: Python 3.10.0a0 (heads/print-statement-dirty:5ed19fcc1a, Jun 9 2020, 16:31:17) [Clang 11.0.0 (clang-1100.0.33.8)] on darwin Type "help", "copyright", "credits" or "license" for more information. Cannot read termcap database; using dumb terminal settings. >>> print 2+2 4 >>> print "hello world" hello world >>> print "hello", input("Name:") Name:Guido hello Guido >>> print 1, 2, 3, sep=", " 1, 2, 3 >>> But wait, there's more! The same syntax will make it possible to call *any* function: >>> len "abc" 3 >>> Or any method: >>> import sys >>> sys.getrefcount "abc" 24 >>> Really, *any* method: >>> class C: ... def foo(self, arg): print arg ... >>> C().foo 2+2 4 >>> There are downsides too, though. For example, you can't call a method without arguments: >>> print <built-in function print> >>> Worse, the first argument cannot start with a parenthesis or bracket: >>> print (1, 2, 3) 1 2 3 >>> C().foo (1, 2, 3) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: C.foo() takes 2 positional arguments but 4 were given >>> print (2+2), 42 4 (None, 42) >>> C().foo [0] Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'method' object is not subscriptable >>> No, it's not April 1st. I am seriously proposing this (but I'll withdraw it if the response is a resounding "boo, hiss"). After all, we currently have a bunch of complexity in the parser just to give a helpful error message to people used to Python 2's print statement: >>> print 1, 2, 3 File "<stdin>", line 1 print 1, 2, 3 ^ SyntaxError: Missing parentheses in call to 'print'. Did you mean print(1, 2, 3)? >>> And IIRC there have been a number of aborted attempts at syntactic hacks to allow people to call functions (like print) without parentheses, although (I think) none of them made it into a PEP. The PEG parser makes this much simpler, because it can simply backtrack -- by placing the grammar rule for this syntax (tentatively called "call statement") last in the list of alternatives for "small statement" we ensure that everything that's a valid expression statement (including print() calls) is still an expression statement with exactly the same meaning, while still allowing parameter-less function calls, without lexical hacks. (There is no code in my prototype that checks for a space after 'print' -- it just checks that there's a name, number or string following a name, which is never legal syntax.) One possible extension I didn't pursue (yet -- dare me!) is to allow parameter-less calls inside other expressions. For example, my prototype does not support things like this: >>> a = (len "abc") File "<stdin>", line 1 a = (len "abc") ^ SyntaxError: invalid syntax >>> I think that strikes a reasonable balance between usability and reduced detection of common errors. I could also dial it back a bit, e.g. maybe it's too much to allow 'C().foo x' and we should only allow dotted names (sufficient to access functions in imported modules and method calls on variables). Or maybe we should only allow simple names (allowing 'len x' but disallowing 'sys.getrefcount x'. Or maybe we should really only bring back print statements. I believe there are some other languages that support a similar grammar (Ruby? R? Raku?) but I haven't investigated. Thoughts? -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-…>

42 66

Adding a thin wrapper class around the functions in stdlib.heapq
by bunslow Feb. 17, 2022

Feb. 17, 2022

Nothing so bombastic this time. The heapq functions are basically all named "heapsomething", and basically all take a "heap" for their first argument, with supplementary args coming after. It's a textbook example of the (hypothetical) Object Oriented Manifesto™ where defining a class increases type safety and programmers' conceptual clarity. There're practically no drawbacks, and the code to be added would be very simple. Updating the tests and docs would probably be harder. In pure Python, such a class would look like this: class Heap(list): def __init__(self, iterable=None): if iterable: super().__init__(iterable) else: super().__init__() self.heapify() push = heapq.heappush pop = heapq.heappop pushpop = heapq.heappushpop replace = heapq.heapreplace heapify = heapq.heapify # This could be a simple wrapper as well, but I had the following thoughts anyways, so here they are def nsmallest(self, n, key=None): # heapq.nsmallest makes a *max* heap of the first n elements, # while we know that self is already a min heap, so we can # make the max heap construction faster self[:n] = reversed(self[:n]) return heapq.nsmallest(n, self, key) # do we define nlargest on a minheap?? Wrapping around the C builtin functions (which aren't descriptors) would be a bit harder, but not much so: from functools import partial class Heap(list): def __init__(self, iterable=None): if iterable: super().__init__(iterable) else: super().__init__() self.heapify = partial(heapq.heapify, self) self.push = partial(heapq.heappush, self) ... self.heapify() Thoughts?

17 32

Allow syntax "func(arg=x if condition)"
by Peter O'Connor Jan. 19, 2022

Jan. 19, 2022

I often find that python lacks a nice way to say "only pass an argument under this condition". (See previous python-list email in "Idea: Deferred Default Arguments?") Example 1: Defining a list with conditional elements include_bd = True current_way = ['a'] + (['b'] if include_bd else [])+['c']+(['d'] if include_bd else []) new_way = ['a', 'b' if include_bd, 'c', 'd' if include_bd] also_new_way = list('a', 'b' if include_bd, 'c', 'd' if include_bd) Example 2: Deferring to defaults of called functions def is_close(a, b, precicion=1e-9): return abs(a-b) < precision def approach(pose, target, step=0.1, precision=None): # Defers to default precision if not otherwise specified: velocity = step*(target-pose) \ if not is_close(pose, target, precision if precision is not None) \ else 0 return velocity Not sure if this has been discussed, but I cannot see any clear downside to adding this, and it has some clear benefits (duplicated default arguments and **kwargs are the scourge of many real world code-bases)

6 9

PEP 671: Syntax for late-bound function argument defaults
by Chris Angelico Nov. 27, 2021

Nov. 27, 2021

Incorporates comments from the thread we just had. Is anyone interested in coauthoring this with me? Anyone who has strong interest in seeing this happen - whether you've been around the Python lists for years, or you're new and interested in getting involved for the first time, or anywhere in between! https://www.python.org/dev/peps/pep-0671/ PEP: 671 Title: Syntax for late-bound function argument defaults Author: Chris Angelico <rosuav(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 24-Oct-2021 Python-Version: 3.11 Post-History: 24-Oct-2021 Abstract ======== Function parameters can have default values which are calculated during function definition and saved. This proposal introduces a new form of argument default, defined by an expression to be evaluated at function call time. Motivation ========== Optional function arguments, if omitted, often have some sort of logical default value. When this value depends on other arguments, or needs to be reevaluated each function call, there is currently no clean way to state this in the function header. Currently-legal idioms for this include:: # Very common: Use None and replace it in the function def bisect_right(a, x, lo=0, hi=None, *, key=None): if hi is None: hi = len(a) # Also well known: Use a unique custom sentinel object _USE_GLOBAL_DEFAULT = object() def connect(timeout=_USE_GLOBAL_DEFAULT): if timeout is _USE_GLOBAL_DEFAULT: timeout = default_timeout # Unusual: Accept star-args and then validate def add_item(item, *optional_target): if not optional_target: target = [] else: target = optional_target[0] In each form, ``help(function)`` fails to show the true default value. Each one has additional problems, too; using ``None`` is only valid if None is not itself a plausible function parameter, the custom sentinel requires a global constant; and use of star-args implies that more than one argument could be given. Specification ============= Function default arguments can be defined using the new ``=>`` notation:: def bisect_right(a, x, lo=0, hi=>len(a), *, key=None): def connect(timeout=>default_timeout): def add_item(item, target=>[]): The expression is saved in its source code form for the purpose of inspection, and bytecode to evaluate it is prepended to the function's body. Notably, the expression is evaluated in the function's run-time scope, NOT the scope in which the function was defined (as are early-bound defaults). This allows the expression to refer to other arguments. Self-referential expressions will result in UnboundLocalError:: def spam(eggs=>eggs): # Nope Multiple late-bound arguments are evaluated from left to right, and can refer to previously-calculated values. Order is defined by the function, regardless of the order in which keyword arguments may be passed. Choice of spelling ------------------ Our chief syntax proposal is ``name=>expression`` -- our two syntax proposals ... ahem. Amongst our potential syntaxes are:: def bisect(a, hi=>len(a)): def bisect(a, hi=:len(a)): def bisect(a, hi?=len(a)): def bisect(a, hi!=len(a)): def bisect(a, hi=\len(a)): def bisect(a, hi=`len(a)`): def bisect(a, hi=@len(a)): Since default arguments behave largely the same whether they're early or late bound, the preferred syntax is very similar to the existing early-bind syntax. The alternatives offer little advantage over the preferred one. How to Teach This ================= Early-bound default arguments should always be taught first, as they are the simpler and more efficient way to evaluate arguments. Building on them, late bound arguments are broadly equivalent to code at the top of the function:: def add_item(item, target=>[]): # Equivalent pseudocode: def add_item(item, target=<OPTIONAL>): if target was omitted: target = [] Open Issues =========== - yield/await? Will they cause problems? Might end up being a non-issue. - annotations? They go before the default, so is there any way an anno could want to end with ``=>``? References ========== Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.

31 292

Unpacking in tuple/list/set/dict comprehensions
by Erik Demaine Nov. 14, 2021

Nov. 14, 2021

Extended unpacking notation (* and **) from PEP 448 gives us great ways to concatenate a few iterables or dicts: ``` (*it1, *it2, *it3) # tuple with the concatenation of three iterables [*it1, *it2, *it3] # list with the concatenation of three iterables {*it1, *it2, *it3} # set with the union of three iterables {**dict1, **dict2, **dict3} # dict with the combination of three dicts # roughly equivalent to dict1 | dict2 | dict3 thanks to PEP 584 ``` I propose (not for the first time) that similarly concatenating an unknown number of iterables or dicts should be possible via comprehensions: ``` (*it for it in its) # tuple with the concatenation of iterables in 'its' [*it for it in its] # list with the concatenation of iterables in 'its' {*it for it in its} # set with the union of iterables in 'its' {**d for d in dicts} # dict with the combination of dicts in 'dicts' ``` The above is my attempt to argue that the proposed notation is natural: `[*it for it in its]` is exactly analogous to `[*its[0], *its[1], ..., *its[len(its)-1]]`. There are other ways to do this, of course: ``` [x for it in its for x in it] itertools.chain(*its) sum(it for it in its, []) functools.reduce(operator.concat, its, []) ``` But none are as concise and (to me, and hopefully others who understand * notation) as intuitive. For example, I recently wanted to write a recursion like so, which accumulated a set of results from within a tree structure: ``` def recurse(node): # base case omitted return {*recurse(child) for child in node.children} ``` In fact, I am teaching a class and just asked a question on a written exam for which several students wrote this exact code in their solution (which inspired writing this message). So I do think it's quite intuitive, even to those relatively new to Python. Now, on to previous proposals. I found this thread from 2016 (!); please let me know if there are others. https://mail.python.org/archives/list/python-ideas@python.org/thread/SBM3LY… There are several arguments for and against this feature in that thread. I'll try to summarize: Arguments for: * Natural extension to PEP 448 (it's mentioned as a variant within PEP 448) * Easy to implement: all that's needed in CPython is to *remove* some code blocking this. Arguments against: * Counterintuitive (to some) * Hard to teach * `[...x... for x in y]` is no longer morally equivalent to `answer = []; for x in y: answer.append(...x...)` (unless `list1.append(a, b)` were equivalent to `list1.extend([a, b])`) Above I've tried to counter the first two "against" arguments. Some counters to the third "against" argument: 1. `[*...x... for x in y]` is equivalent to `answer = []; for x in y: answer.extend(...x...)` (about as easy to teach, I'd say) 2. Maybe `list1.append(a, b)` should be equivalent to `list1.extend([a, b])`? It is in JavaScript (`Array.push`). And I don't see why one would expect it to append a tuple `(a, b)`; that's what `list1.append((a, b))` is for. I think the main argument against this is to avoid programming errors, which is fine, but I don't see why it should hold back an advanced feature involving both unpacking and comprehensions. Erik -- Erik Demaine | edemaine(a)mit.edu | http://erikdemaine.org/

15 30

PEP 671 proof-of-concept: A less invasive implementation
by Jonathan Fine Oct. 31, 2021

Oct. 31, 2021

Hi One of the motives for PEP 671 is that the signature of a function fn, and hence the associated help(fn) is sometimes opaque regarding default values. I won't repeat the excellent examples already given. In the current implementation default values are handled outside the compiled code of the function, which is available at fn.__code__. Instead they are stored in 'metadata' associated with the function. Here's one way to see this. from inspect import signature as sig def fn(a, b=1, c=2): return a, b, c sig(fn) # Gives <Signature (a, b=1, c=2)> fn.__defaults__ = ('hi', 'there') sig(fn) # Gives <Signature (a, b='hi', c='there')> We can also change the __code__ object, but care is needed here. def gn(): return (1, 2, 3) fn.__code__ = gn.__code__ fn() # Gives (1, 2, 3). sig(fn) # Gives <Signature ()> fn.__defaults__ # Gives ('hi', 'there') The signature of fn, together with the arguments actually supplied, is used to initialise the code frame which is put on the top of the stack, and in which fn.__code__ executes. I suggest that an implementation which provides additional flexibility in the manner in which the code frame is initialised would be less invasive. Necessarily, PEP 671 allows programmer supplied code to be used in the 'initialisation phase'. The previous attempts place that code in fn.__code__. I suggest that the implementation adds a new attribute fn.__wibble__ to fn, which can either be None or a code object. And if fn.__wibble__ is not None, then it is used to initialise the code frame in which fn.__code__ executes. It would as before take as input the arguments actually supplied by the user. I stop here, saying nothing for now about two important questions. First, what is the programmer syntax for creating such a 'two-part' function fn. Second, what does the user see as a result of help(fn). Or in other words, how to extend the semantics of inspect.signature. with kind regards Jonathan

3 7

PEP 671 late-bound defaults implementation
by Steven D'Aprano Oct. 31, 2021

Oct. 31, 2021

I have a suggestion for the implementation. I think that Chris' current approach is to compile the late-bound defaults in the function body. So if we have a function like this: def func(a, b=early_expression, @c=late_expression): block the function code looks like this (after compilation): # Pseudocode if c is unbound: c = late_expression block (except in bytecode of course). Chris, do I have that right? If it is wrong, probably everything I say next is irrelevant. There is a strange asymmetry to the way the default for b and c are handled. For b, it is the interpreter's responsibilty to load the default value (pre-evaluated and cached) and bind it to the parameter. But for c, it is the function object's responsibility. I'd like to suggest a different approach which I expect will be more flexible, I hope won't cost too much in performance, and in my opinion much more closely matches the semantics of the feature. I think it should remain the interpreter's responsibility to set up all the parameters before entering the function, including late-bound defaults. That will have the big advantage that disassembling func will only show the code for "block", not the associated code that tests and binds late-bound defaults. (Just like currently, it doesn't show the code for binding early-bound defaults.) This suggests that each late-bound expression should be compiled into a separate code object, all of which are then squirrelled away in the function object (just as the __code__ and __defaults__ currently are). I imagine the process will be something like: * set up a new local namespace for the function call * bind arguments to parameters in that namespace * bind early-bound defaults from the function __defaults__ to parameters * (NEW) evaluate the appropriate late-bound expression code objects, running them in the local namespace, and binding their results to the parameters; * enter the function's code block. Benefits: - the function code block is smaller, since it no longer has to include the "test, evaluate, bind" for every late-bound parameter; - this may improve code locality, which is good for performance (or so I am told); - introspection tools such as dis can disassemble the body of the function independently of the late-bound parameters; - which means we can inspect the late-bound parameters independently by passing their code object to dis; - for testing, we can evaluate the expression code objects using eval (maybe?); - we may be able to include the source code to the expression in the expression's own code block, e.g. in the co_name field(?); - we may be able to replace/modify the defaults' code blocks independently of the main function __code__, e.g. for byte-code hacking, or other function object hacking. Costs: - the function object itself may be a little larger. Thoughts? -- Steve

2 2

Context managers in expressions
by jcg.sturdy＠gmail.com Oct. 30, 2021

Oct. 30, 2021

I've been thinking for a while that it would be good to be able to use context managers in an expression context, and I haven't been able to come up with any arguments against it (although it may not be to everyone's taste) and I can't see any sign that it's come up before (although the words of it don't make very specific search terms). My suggestion is that a "with" clause could be written after an expression (consistent with the conditional operator being written with "if" after the first expression). In general, this would be: a(b) with c(d) as b or it could allow multiple context managers: a(b, c) with d(e) as b, f(g) as c My original motivating example was this: if name.startswith("https://"): data = requests.get(name).json() else: with open(name) as stream: data = json.load(stream) which I'd much rather be able to write with a single expression: data = (requests.get(name).json() if name.startswith("https://") else (json.load(stream) with open(name) as stream)) It would behave just like a "with" statement, but pass a value / values back. I don't think there would be any backward-compatibility issues with this.

4 3