Mailman 3 October 2021 - Python-ideas

zip(x, y, z, strict=True)
by Ram Rachum Dec. 1, 2022

Dec. 1, 2022

Here's something that would have saved me some debugging yesterday: >>> zipped = zip(x, y, z, strict=True) I suggest that `strict=True` would ensure that all the iterables have been exhausted, raising an exception otherwise. This is useful in cases where you're assuming that the iterables all have the same lengths. When your assumption is wrong, you currently just get a shorter result, and it could take you a while to figure out why it's happening. What do you think?

27 179

disallow assignment to unknown ssl.SSLContext attributes
by Thomas Grainger June 10, 2022

June 10, 2022

I was debugging some code that was using TLSv1.2 when I expected it to only support TLSv1.3, I tracked it down to a call to: context.miunimum_version = ssl.TLSVersion.TLSv1_3 it should have been: context.minimum_version = ssl.TLSVersion.TLSv1_3 I'd like invalid attribute assignment to be prevented at runtime

13 32

Bringing the print statement back
by Guido van Rossum March 2, 2022

March 2, 2022

In Python 3.10 we will no longer be burdened by the old parser (though 3rd party tooling needs to catch up). One thing that the PEG parser makes possible in about 20 lines of code is something not entirely different from the old print statement. I have a prototype: Python 3.10.0a0 (heads/print-statement-dirty:5ed19fcc1a, Jun 9 2020, 16:31:17) [Clang 11.0.0 (clang-1100.0.33.8)] on darwin Type "help", "copyright", "credits" or "license" for more information. Cannot read termcap database; using dumb terminal settings. >>> print 2+2 4 >>> print "hello world" hello world >>> print "hello", input("Name:") Name:Guido hello Guido >>> print 1, 2, 3, sep=", " 1, 2, 3 >>> But wait, there's more! The same syntax will make it possible to call *any* function: >>> len "abc" 3 >>> Or any method: >>> import sys >>> sys.getrefcount "abc" 24 >>> Really, *any* method: >>> class C: ... def foo(self, arg): print arg ... >>> C().foo 2+2 4 >>> There are downsides too, though. For example, you can't call a method without arguments: >>> print <built-in function print> >>> Worse, the first argument cannot start with a parenthesis or bracket: >>> print (1, 2, 3) 1 2 3 >>> C().foo (1, 2, 3) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: C.foo() takes 2 positional arguments but 4 were given >>> print (2+2), 42 4 (None, 42) >>> C().foo [0] Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'method' object is not subscriptable >>> No, it's not April 1st. I am seriously proposing this (but I'll withdraw it if the response is a resounding "boo, hiss"). After all, we currently have a bunch of complexity in the parser just to give a helpful error message to people used to Python 2's print statement: >>> print 1, 2, 3 File "<stdin>", line 1 print 1, 2, 3 ^ SyntaxError: Missing parentheses in call to 'print'. Did you mean print(1, 2, 3)? >>> And IIRC there have been a number of aborted attempts at syntactic hacks to allow people to call functions (like print) without parentheses, although (I think) none of them made it into a PEP. The PEG parser makes this much simpler, because it can simply backtrack -- by placing the grammar rule for this syntax (tentatively called "call statement") last in the list of alternatives for "small statement" we ensure that everything that's a valid expression statement (including print() calls) is still an expression statement with exactly the same meaning, while still allowing parameter-less function calls, without lexical hacks. (There is no code in my prototype that checks for a space after 'print' -- it just checks that there's a name, number or string following a name, which is never legal syntax.) One possible extension I didn't pursue (yet -- dare me!) is to allow parameter-less calls inside other expressions. For example, my prototype does not support things like this: >>> a = (len "abc") File "<stdin>", line 1 a = (len "abc") ^ SyntaxError: invalid syntax >>> I think that strikes a reasonable balance between usability and reduced detection of common errors. I could also dial it back a bit, e.g. maybe it's too much to allow 'C().foo x' and we should only allow dotted names (sufficient to access functions in imported modules and method calls on variables). Or maybe we should only allow simple names (allowing 'len x' but disallowing 'sys.getrefcount x'. Or maybe we should really only bring back print statements. I believe there are some other languages that support a similar grammar (Ruby? R? Raku?) but I haven't investigated. Thoughts? -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-…>

42 66

Adding a thin wrapper class around the functions in stdlib.heapq
by bunslow Feb. 17, 2022

Feb. 17, 2022

Nothing so bombastic this time. The heapq functions are basically all named "heapsomething", and basically all take a "heap" for their first argument, with supplementary args coming after. It's a textbook example of the (hypothetical) Object Oriented Manifesto™ where defining a class increases type safety and programmers' conceptual clarity. There're practically no drawbacks, and the code to be added would be very simple. Updating the tests and docs would probably be harder. In pure Python, such a class would look like this: class Heap(list): def __init__(self, iterable=None): if iterable: super().__init__(iterable) else: super().__init__() self.heapify() push = heapq.heappush pop = heapq.heappop pushpop = heapq.heappushpop replace = heapq.heapreplace heapify = heapq.heapify # This could be a simple wrapper as well, but I had the following thoughts anyways, so here they are def nsmallest(self, n, key=None): # heapq.nsmallest makes a *max* heap of the first n elements, # while we know that self is already a min heap, so we can # make the max heap construction faster self[:n] = reversed(self[:n]) return heapq.nsmallest(n, self, key) # do we define nlargest on a minheap?? Wrapping around the C builtin functions (which aren't descriptors) would be a bit harder, but not much so: from functools import partial class Heap(list): def __init__(self, iterable=None): if iterable: super().__init__(iterable) else: super().__init__() self.heapify = partial(heapq.heapify, self) self.push = partial(heapq.heappush, self) ... self.heapify() Thoughts?

17 32

Allow syntax "func(arg=x if condition)"
by Peter O'Connor Jan. 19, 2022

Jan. 19, 2022

I often find that python lacks a nice way to say "only pass an argument under this condition". (See previous python-list email in "Idea: Deferred Default Arguments?") Example 1: Defining a list with conditional elements include_bd = True current_way = ['a'] + (['b'] if include_bd else [])+['c']+(['d'] if include_bd else []) new_way = ['a', 'b' if include_bd, 'c', 'd' if include_bd] also_new_way = list('a', 'b' if include_bd, 'c', 'd' if include_bd) Example 2: Deferring to defaults of called functions def is_close(a, b, precicion=1e-9): return abs(a-b) < precision def approach(pose, target, step=0.1, precision=None): # Defers to default precision if not otherwise specified: velocity = step*(target-pose) \ if not is_close(pose, target, precision if precision is not None) \ else 0 return velocity Not sure if this has been discussed, but I cannot see any clear downside to adding this, and it has some clear benefits (duplicated default arguments and **kwargs are the scourge of many real world code-bases)

6 9

PEP 671: Syntax for late-bound function argument defaults
by Chris Angelico Nov. 27, 2021

Nov. 27, 2021

Incorporates comments from the thread we just had. Is anyone interested in coauthoring this with me? Anyone who has strong interest in seeing this happen - whether you've been around the Python lists for years, or you're new and interested in getting involved for the first time, or anywhere in between! https://www.python.org/dev/peps/pep-0671/ PEP: 671 Title: Syntax for late-bound function argument defaults Author: Chris Angelico <rosuav(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 24-Oct-2021 Python-Version: 3.11 Post-History: 24-Oct-2021 Abstract ======== Function parameters can have default values which are calculated during function definition and saved. This proposal introduces a new form of argument default, defined by an expression to be evaluated at function call time. Motivation ========== Optional function arguments, if omitted, often have some sort of logical default value. When this value depends on other arguments, or needs to be reevaluated each function call, there is currently no clean way to state this in the function header. Currently-legal idioms for this include:: # Very common: Use None and replace it in the function def bisect_right(a, x, lo=0, hi=None, *, key=None): if hi is None: hi = len(a) # Also well known: Use a unique custom sentinel object _USE_GLOBAL_DEFAULT = object() def connect(timeout=_USE_GLOBAL_DEFAULT): if timeout is _USE_GLOBAL_DEFAULT: timeout = default_timeout # Unusual: Accept star-args and then validate def add_item(item, *optional_target): if not optional_target: target = [] else: target = optional_target[0] In each form, ``help(function)`` fails to show the true default value. Each one has additional problems, too; using ``None`` is only valid if None is not itself a plausible function parameter, the custom sentinel requires a global constant; and use of star-args implies that more than one argument could be given. Specification ============= Function default arguments can be defined using the new ``=>`` notation:: def bisect_right(a, x, lo=0, hi=>len(a), *, key=None): def connect(timeout=>default_timeout): def add_item(item, target=>[]): The expression is saved in its source code form for the purpose of inspection, and bytecode to evaluate it is prepended to the function's body. Notably, the expression is evaluated in the function's run-time scope, NOT the scope in which the function was defined (as are early-bound defaults). This allows the expression to refer to other arguments. Self-referential expressions will result in UnboundLocalError:: def spam(eggs=>eggs): # Nope Multiple late-bound arguments are evaluated from left to right, and can refer to previously-calculated values. Order is defined by the function, regardless of the order in which keyword arguments may be passed. Choice of spelling ------------------ Our chief syntax proposal is ``name=>expression`` -- our two syntax proposals ... ahem. Amongst our potential syntaxes are:: def bisect(a, hi=>len(a)): def bisect(a, hi=:len(a)): def bisect(a, hi?=len(a)): def bisect(a, hi!=len(a)): def bisect(a, hi=\len(a)): def bisect(a, hi=`len(a)`): def bisect(a, hi=@len(a)): Since default arguments behave largely the same whether they're early or late bound, the preferred syntax is very similar to the existing early-bind syntax. The alternatives offer little advantage over the preferred one. How to Teach This ================= Early-bound default arguments should always be taught first, as they are the simpler and more efficient way to evaluate arguments. Building on them, late bound arguments are broadly equivalent to code at the top of the function:: def add_item(item, target=>[]): # Equivalent pseudocode: def add_item(item, target=<OPTIONAL>): if target was omitted: target = [] Open Issues =========== - yield/await? Will they cause problems? Might end up being a non-issue. - annotations? They go before the default, so is there any way an anno could want to end with ``=>``? References ========== Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.

31 292

Unpacking in tuple/list/set/dict comprehensions
by Erik Demaine Nov. 14, 2021

Nov. 14, 2021

Extended unpacking notation (* and **) from PEP 448 gives us great ways to concatenate a few iterables or dicts: ``` (*it1, *it2, *it3) # tuple with the concatenation of three iterables [*it1, *it2, *it3] # list with the concatenation of three iterables {*it1, *it2, *it3} # set with the union of three iterables {**dict1, **dict2, **dict3} # dict with the combination of three dicts # roughly equivalent to dict1 | dict2 | dict3 thanks to PEP 584 ``` I propose (not for the first time) that similarly concatenating an unknown number of iterables or dicts should be possible via comprehensions: ``` (*it for it in its) # tuple with the concatenation of iterables in 'its' [*it for it in its] # list with the concatenation of iterables in 'its' {*it for it in its} # set with the union of iterables in 'its' {**d for d in dicts} # dict with the combination of dicts in 'dicts' ``` The above is my attempt to argue that the proposed notation is natural: `[*it for it in its]` is exactly analogous to `[*its[0], *its[1], ..., *its[len(its)-1]]`. There are other ways to do this, of course: ``` [x for it in its for x in it] itertools.chain(*its) sum(it for it in its, []) functools.reduce(operator.concat, its, []) ``` But none are as concise and (to me, and hopefully others who understand * notation) as intuitive. For example, I recently wanted to write a recursion like so, which accumulated a set of results from within a tree structure: ``` def recurse(node): # base case omitted return {*recurse(child) for child in node.children} ``` In fact, I am teaching a class and just asked a question on a written exam for which several students wrote this exact code in their solution (which inspired writing this message). So I do think it's quite intuitive, even to those relatively new to Python. Now, on to previous proposals. I found this thread from 2016 (!); please let me know if there are others. https://mail.python.org/archives/list/python-ideas@python.org/thread/SBM3LY… There are several arguments for and against this feature in that thread. I'll try to summarize: Arguments for: * Natural extension to PEP 448 (it's mentioned as a variant within PEP 448) * Easy to implement: all that's needed in CPython is to *remove* some code blocking this. Arguments against: * Counterintuitive (to some) * Hard to teach * `[...x... for x in y]` is no longer morally equivalent to `answer = []; for x in y: answer.append(...x...)` (unless `list1.append(a, b)` were equivalent to `list1.extend([a, b])`) Above I've tried to counter the first two "against" arguments. Some counters to the third "against" argument: 1. `[*...x... for x in y]` is equivalent to `answer = []; for x in y: answer.extend(...x...)` (about as easy to teach, I'd say) 2. Maybe `list1.append(a, b)` should be equivalent to `list1.extend([a, b])`? It is in JavaScript (`Array.push`). And I don't see why one would expect it to append a tuple `(a, b)`; that's what `list1.append((a, b))` is for. I think the main argument against this is to avoid programming errors, which is fine, but I don't see why it should hold back an advanced feature involving both unpacking and comprehensions. Erik -- Erik Demaine | edemaine(a)mit.edu | http://erikdemaine.org/

15 30

PEP 671 proof-of-concept: A less invasive implementation
by Jonathan Fine Oct. 31, 2021

Oct. 31, 2021

Hi One of the motives for PEP 671 is that the signature of a function fn, and hence the associated help(fn) is sometimes opaque regarding default values. I won't repeat the excellent examples already given. In the current implementation default values are handled outside the compiled code of the function, which is available at fn.__code__. Instead they are stored in 'metadata' associated with the function. Here's one way to see this. from inspect import signature as sig def fn(a, b=1, c=2): return a, b, c sig(fn) # Gives <Signature (a, b=1, c=2)> fn.__defaults__ = ('hi', 'there') sig(fn) # Gives <Signature (a, b='hi', c='there')> We can also change the __code__ object, but care is needed here. def gn(): return (1, 2, 3) fn.__code__ = gn.__code__ fn() # Gives (1, 2, 3). sig(fn) # Gives <Signature ()> fn.__defaults__ # Gives ('hi', 'there') The signature of fn, together with the arguments actually supplied, is used to initialise the code frame which is put on the top of the stack, and in which fn.__code__ executes. I suggest that an implementation which provides additional flexibility in the manner in which the code frame is initialised would be less invasive. Necessarily, PEP 671 allows programmer supplied code to be used in the 'initialisation phase'. The previous attempts place that code in fn.__code__. I suggest that the implementation adds a new attribute fn.__wibble__ to fn, which can either be None or a code object. And if fn.__wibble__ is not None, then it is used to initialise the code frame in which fn.__code__ executes. It would as before take as input the arguments actually supplied by the user. I stop here, saying nothing for now about two important questions. First, what is the programmer syntax for creating such a 'two-part' function fn. Second, what does the user see as a result of help(fn). Or in other words, how to extend the semantics of inspect.signature. with kind regards Jonathan

3 7

PEP 671 late-bound defaults implementation
by Steven D'Aprano Oct. 31, 2021

Oct. 31, 2021

I have a suggestion for the implementation. I think that Chris' current approach is to compile the late-bound defaults in the function body. So if we have a function like this: def func(a, b=early_expression, @c=late_expression): block the function code looks like this (after compilation): # Pseudocode if c is unbound: c = late_expression block (except in bytecode of course). Chris, do I have that right? If it is wrong, probably everything I say next is irrelevant. There is a strange asymmetry to the way the default for b and c are handled. For b, it is the interpreter's responsibilty to load the default value (pre-evaluated and cached) and bind it to the parameter. But for c, it is the function object's responsibility. I'd like to suggest a different approach which I expect will be more flexible, I hope won't cost too much in performance, and in my opinion much more closely matches the semantics of the feature. I think it should remain the interpreter's responsibility to set up all the parameters before entering the function, including late-bound defaults. That will have the big advantage that disassembling func will only show the code for "block", not the associated code that tests and binds late-bound defaults. (Just like currently, it doesn't show the code for binding early-bound defaults.) This suggests that each late-bound expression should be compiled into a separate code object, all of which are then squirrelled away in the function object (just as the __code__ and __defaults__ currently are). I imagine the process will be something like: * set up a new local namespace for the function call * bind arguments to parameters in that namespace * bind early-bound defaults from the function __defaults__ to parameters * (NEW) evaluate the appropriate late-bound expression code objects, running them in the local namespace, and binding their results to the parameters; * enter the function's code block. Benefits: - the function code block is smaller, since it no longer has to include the "test, evaluate, bind" for every late-bound parameter; - this may improve code locality, which is good for performance (or so I am told); - introspection tools such as dis can disassemble the body of the function independently of the late-bound parameters; - which means we can inspect the late-bound parameters independently by passing their code object to dis; - for testing, we can evaluate the expression code objects using eval (maybe?); - we may be able to include the source code to the expression in the expression's own code block, e.g. in the co_name field(?); - we may be able to replace/modify the defaults' code blocks independently of the main function __code__, e.g. for byte-code hacking, or other function object hacking. Costs: - the function object itself may be a little larger. Thoughts? -- Steve

2 2

Context managers in expressions
by jcg.sturdy＠gmail.com Oct. 30, 2021

Oct. 30, 2021

I've been thinking for a while that it would be good to be able to use context managers in an expression context, and I haven't been able to come up with any arguments against it (although it may not be to everyone's taste) and I can't see any sign that it's come up before (although the words of it don't make very specific search terms). My suggestion is that a "with" clause could be written after an expression (consistent with the conditional operator being written with "if" after the first expression). In general, this would be: a(b) with c(d) as b or it could allow multiple context managers: a(b, c) with d(e) as b, f(g) as c My original motivating example was this: if name.startswith("https://"): data = requests.get(name).json() else: with open(name) as stream: data = json.load(stream) which I'd much rather be able to write with a single expression: data = (requests.get(name).json() if name.startswith("https://") else (json.load(stream) with open(name) as stream)) It would behave just like a "with" statement, but pass a value / values back. I don't think there would be any backward-compatibility issues with this.

4 3

PEP 671 proof-of-concept implementation
by Chris Angelico Oct. 30, 2021

Oct. 30, 2021

https://github.com/Rosuav/cpython/tree/pep-671 So uhh... anyone who knows about the internals of CPython and wants to join me on this, I would *really* appreciate coauthors! The implementation ended up a lot more invasive than I originally planned. Some of that is inherent to the problem, but other parts might be able to be done more cleanly. The way I've done it: * Argument defaults (either in __defaults__ or __kwdefaults__) are now tuples of (desc, value) or (desc,) for early-bound and late-bound respectively * Early-bound defaults get mapped as normal. Late-bound defaults are left unbound at time of function call. * For each late-bound default as of the 'def' statement, a check is coded: if the local is unbound, set it based on the given expression. This means that it's possible to replace an early-bound default with a late-bound, but instead of actually evaluating the expression, it just leaves it unbound: >>> def f(x=1): print(x) ... >>> f.__defaults__ ((None, 1),) >>> f.__defaults__ = ((None,),) >>> f() Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 1, in f UnboundLocalError: cannot access local variable 'x' where it is not associated with a value >>> I'm not entirely happy with this, but I also don't want to have too much of a performance impact in the normal case. So far unimplemented is the description of the argument default. My plan is for early-bound defaults to have None there (as they currently do), but late-bound ones get the source code. (In theory, it may be of value to retain the source code for earlies too, which would allow hex or octal integer literals to show up in help() as such, rather than showing the (decimal) repr of the resulting value.) Anyone got good pointers on how to do this, or is that likely to be impractical? Feel free to criticize my code. As you'll see from the commit messages in that branch, I have no idea what I'm doing here :) ChrisA

6 11

`is in`/`not is in` operators
by Jeremiah Vivian Oct. 29, 2021

Oct. 29, 2021

I DO expect this thread to be bombarded with negative replies. Currently, there are `in`/`not in` operators which work like this in Python: > def contains(contains_value, iterable, not_in): > for element in iterable: > if element == contains_value: > return True ^ not_in > return False ^ not_in If I wanted to check if an *exact* object is in an iterable, I would have to loop like this (reverse boolean values for implementation of `not is in`): > is_in = False > for element in iterable: > if element is contains_value: > is_in = True I would want a more *convenient* way to check for this value. So therefore, there should be `is in`/`not is in` operators to do it better. Is this a valid reason?

5 20

Python Shared Objects
by byko3y＠gmail.com Oct. 28, 2021

Oct. 28, 2021

Due to https://www.python.org/dev/peps/pep-0554/ multi-interpreters implementation going really slow, I had the audicity to try an alternative route towards the same objective of implementing multicore support of python: instead of sharing the memory by running multiple threads, I employed an interprocess shared memory with multiple processes. I know there are multiprocessing.sharedctypes and multiprocessing.shared_memory, but I went much deeper into it by solving two problems they failed to solve: sharing of complex dynamic objects and synchronization of data access. I have a working prototype to show: https://github.com/byko3y/python-shared-objects It's a kind of software transactional memory within shared memory. It has a basic support for fundamental python types (bool, int, str, bytes, tuple, list, dict, object), for both atomic and non-atomic transactions via fine-grained RW-locks, has a built-in protection against deadlock and starvation. Most of my code can also be used for cross-interpreter communication after PEP 554 is successfully implemented, since cross-interpreter communication is still an open question.

2 2

Syntax for late-bound arguments
by Chris Angelico Oct. 28, 2021

Oct. 28, 2021

Spun off from PEP 505 thread on python-dev. One of the weaker use-cases for None-coalescing is function default arguments, where the technical default is None but the function wants to use some other value instead. This is a weak argument in favour of None-coalescing because only a small set of such situations are only slightly improved by a "??" operator, and the problem is far larger. Consider the bisect.bisect() function [1]: def bisect(a, x, lo=0, hi=None): if lo < 0: raise ValueError('lo must be non-negative') if hi is None: hi = len(a) It's clear what value lo gets if you omit it. It's less clear what hi gets. And the situation only gets uglier if None is a valid argument, and a unique sentinel is needed; this standard idiom makes help() rather unhelpful: _missing = object() def spaminate(thing, count=_missing): if count is _missing: count = thing.getdefault() Proposal: Proper syntax and support for late-bound argument defaults. def spaminate(thing, count=:thing.getdefault()): ... def bisect(a, x, lo=0, hi=:len(a)): if lo < 0: raise ValueError('lo must be non-negative') This would be approximately equivalent to the _missing idiom, with the following exceptions: 1) Inspecting the function would reveal the source code for the late-bound value 2) There is no value which can be passed as an argument to achieve the same effect 3) All late-bound defaults would be evaluated before any other part of the function executes (ie the "if hi is None" check would now be done prior to "if lo < 0"). The syntax I've chosen is deliberately subtle, since - in many many cases - it won't make any difference whether the argument is early or late bound, so they should look similar. While it is visually similar to the inline assignment operator, neither form is currently valid in a function's parameter list, and thus should not introduce ambiguity. The expression would be evaluated in the function's context, having available to it everything that the function has. Notably, this is NOT the same as the context of the function definition, but this is only rarely going to be significant (eg class methods where a bare name in an early-bound argument default would come from class scope, but the same bare name would come from local scope if late-bound). The purpose of this change is to have the function header define, as fully as possible, the function's arguments. Burying part of that definition inside the function is arbitrary and unnecessary. ChrisA [1] Technically it's bisect_right but I'm using the simpler name here. Also, it recently grew a key parameter, which would only get in the way here. https://docs.python.org/3/library/bisect.html#bisect.bisect_right

15 53

Parameter tuple unpacking in the age of positional-only arguments
by Erik Demaine Oct. 28, 2021

Oct. 28, 2021

On Tue, 26 Oct 2021, Eric V. Smith wrote: > You may or may not recall that a big reason for the removal of "tuple > parameter unpacking" in PEP 3113 was that they couldn't be supported by the > inspect module. Quoting that PEP: "Python has very powerful introspection > capabilities. These extend to function signatures. There are no hidden > details as to what a function's call signature is." > > (Aside: I loved tuple parameter unpacking, and used it all the time! I was > sad to see them go, but I agreed with PEP 3113.) Having recently heard a friend say "the removal of tuple parameter unpacking was one thing that Python 3 got wrong", I read this and PEP 3113 with interest. It seems like another approach would be to treat tuple-unpacking parameters as positional-only, now that this is a thing, or perhaps require that they are explicitly positional-only via in PEP 570: def move((x, y), /): ... # could be valid? def move((x, y)): ... # could remain invalid? Is it worth revisiting parameter tuple-unpacking in the age of positional-only arguments? Or is this still a no-go from the perspective of introspection, because it violates "There are no hidden details as to what a function's call signature is."? (This may be a very short-lived thread.) Erik -- Erik Demaine | edemaine(a)mit.edu | http://erikdemaine.org/

3 2

Add __len__ to ipaddress._BaseNetwork
by nyuszika7h＠gmail.com Oct. 26, 2021

Oct. 26, 2021

Currently, ipaddress._BaseNetwork (and by extension, ipaddress.IPv4Network and ipaddress.IPv6Network) does not have a __len__ method, it only has num_addresses. This makes the following code not work: >>> random.choice(ipaddress.ip_network('6.0.0.0/8')) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/nyuszika7h/.pyenv/versions/3.10.0/lib/python3.10/random.py", line 378, in choice return seq[self._randbelow(len(seq))] TypeError: object of type 'IPv4Network' has no len() The workaround is a bit ugly: >>> (network := ipaddress.ip_network('6.0.0.0/8'))[random.randrange(network.num_addresses)] IPv4Address('6.60.184.142') With a custom subclass, all works well: >>> class MyIPv4Network(ipaddress.IPv4Network): ... def __len__(self): ... return self.num_addresses ... >>> random.choice(MyIPv4Network('6.0.0.0/8')) IPv4Address('6.40.110.63')

2 1

Fwd: Simple curl/wget-like download functionality in urllib (like http offers server)
by Tom P Oct. 26, 2021

Oct. 26, 2021

In the context of building Docker images, it is often required to download stuff. If curl/wget is available, great, but often slim images don't include that. The urllib could provide a very simple download functionality (like http offers a simple server): from urllib.request import urlopen data = urlopen('https://.../install-poetry.py').read() # print or save data

10 16

Re: PEP 671: Syntax for late-bound function argument defaults
by Jon Kiparsky Oct. 25, 2021

Oct. 25, 2021

While I love this idea, and I think this PEP is bound for glory, I don't love any of the proposed spellings for denoting late-binding arguments. It seems to me that a specialized symbol combination indicating that a particular argument has special behavior does not serve readability, regardless of what symbol combination is chosen. I would prefer to build on the fact that arguments already come in two flavors with somewhat different behaviors, and that the ordering of these is determined. Going by this analogy, it would make sense to have late-binding arguments following keyword arguments, set off by some separator, such as a double-pipe: def my_func(positional_arg, some_kwarg="foo" || late_binding_arg=[]]: # body of func or def my_func(positional_arg, some_kwarg="foo" || late_binding_arg=[]): # body of func I'm not massively attached to this particular symbol (which has advantages and disadvantages), but more to the idea that we can easily see that the arguments to the right of it are late-binding, and we can easily teach a beginner what that means. We can also easily scan a function definition and know that there is some specialized behavior going on, which is harder to see with a spelling like def my_func(positional_arg, some_kwarg<="foo", another_kwarg=[]): # body of func and we will probably even notice much more easily that there is a likely bug in that last definition (since we probably meant for the list arg to be late-binding) Best -Jon Kiparsky jon.kiparsky(a)gmail.com

3 2

Return False from __contains__ method if object not hashable for set and dict.
by Kazantcev Andrey Oct. 25, 2021

Oct. 25, 2021

Now if do something like `[] in set()` python raise an exception, but if an object isn't hashable then we know for sure that it isn't in the set. Propose return False for these cases. What do you think?

3 2

def variable = value
by blek blek Oct. 25, 2021

Oct. 25, 2021

Hello people, I was thinking of a "def" statement in Python. Its really weird that it can define only method, since it means "define". It shouldn't be hard to detect if this is a variable or a method, because method needs "()" to have arguments(even if there are zero of them), and defining variable value supposed to have "=" as the definition mark. Hope it would be added someday

5 6

os.workdir() context manager
by Marc-Andre Lemburg Oct. 22, 2021

Oct. 22, 2021

Hello all, I sometimes write Python scripts which need to work in specific work directories. When putting such code into functions, the outer function typically does not expect the current work dir (CWD) to be changed, so wrap the code which need the (possibly) modified CWD using a simply context manager along the lines of: class workdir: def __init__(self, dir): self.dir = dir def __enter__(self): self.curdir = os.getcwd() os.chdir(self.dir) def __exit__(self, *exc): os.chdir(self.curdir) return False Would there be interest in adding something like this to the os module as os.workdir() ? Example: def backup_home_dir(account): with os.workdir(os.path.join('/home', account)): # Create a backup of the account dir, rooted at the account's # home dir restic.backup(repo, '.') Notes: - The context manager is not thread safe. There's no thread safe model for the current work dir. OTOH, scripts usually don't use threads, so not a big deal. - The context manager could be made more elaborate, e.g. adding optional logging for debugging purposes. - The same could be added to pathlib's objects as .workdir() method returning a context manager. Thoughts ? Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Sep 14 2021) >>> Python Projects, Coaching and Support ... https://www.egenix.com/ >>> Python Product Development ... https://consulting.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/

17 37

Fwd: [Python-Dev] Re: Semi-proposal: Tagged None
by David Mertz, Ph.D. Oct. 21, 2021

Oct. 21, 2021

I actually thought this thread was on pyhon-ideas where it is now appropriate. Chris quotes everything I wrote, and his comments are all useful, so let me move starting with that. ---------- Forwarded message --------- From: Chris Angelico <rosuav(a)gmail.com> Date: Thu, Oct 21, 2021, 8:42 PM Subject: [Python-Dev] Re: Semi-proposal: Tagged None To: Python-Dev <python-dev(a)python.org> On Fri, Oct 22, 2021 at 3:23 AM David Mertz, Ph.D. <david.mertz(a)gmail.com> wrote: > > On Thu, Oct 21, 2021 at 2:52 AM Steven D'Aprano <steve(a)pearwood.info> wrote: >> >> On Tue, Oct 19, 2021 at 05:09:42PM -0700, Michael Selik wrote: >> > None and its ilk often conflate too many qualities. For example, is it >> > missing because it doesn't exist, it never existed, or because we never >> > received a value, despite knowing it must exist? > > >> >> 30+ years later, and we cannot easily, reliably or portably use NAN >> payloads. Most people don't care. If we offerred them a dozen or a >> thousand distinct sentinels for all the various kinds of missing data, >> how many people would use them and how many would just stick to plain >> old None? > > > In data science, I have been frustrated by the sparsity of ways of spelling "missing value." Might be worth redirecting this to -ideas. > Besides the distinction Michael points out, and that Steven did in relation to NaNs with payloads, I encounter missingness of various other sorts as well. Crucially, an important kind of missing data is data where the value I received seems unreliable and I have decided to *impute* missingness rather than accept a value I believe is unreliable. > > But there is also something akin to what Michael points out (maybe it's just an example). For example, "middle name" is something that some people simply do not have, other people choose not to provide on a survey, and others still we just don't know anything beyond "it's not there." > And some people have more than one (I have a brother with two of them). Not the best example to use, since names have WAY more complexities than different types of absence, but there are other cases where that sort of thing comes up. For instance, if someone says on a survey that s/he is in Australia, and then you ask for a postcode, then leaving it blank should be recorded as "chose not to provide"; but if the country is listed as Timor-Leste / East Timor, then "not applicable" would be appropriate, since the country doesn't use postal codes. > Of course, when I impute missingness, I can do so at various stages of data cleaning, and for various different reasons or confidences. None (or NaN) are sort of OK, but carrying metadata as to the nature of missingness would be nice. > Right. Using postcodes as an example again, for someone in Australia, a postcode of "E3B 0H8" doesn't make sense, as that isn't the format we use. So you could wipe that out and replace it with "No postal code, malformed data entered". > So my strawman suggestion is tagging None's. I suppose spellings like `None[reason]` or `None(reason)` are appealing. > > An obvious problem that I recognize is that it's not obvious this can "play nice" with the common idiom `if mydata is not None: ...`. None really is a singleton, and a "tagged singleton" or "annotated singleton" probably doesn't work well with Python's object model. > > My goal, of course, would be to have TaggedNone be a kind of subclass of None, in the same way that bool is a subclass of int, and hence True is a kind of 1. However, I'd want a large number of custom None's, with some sort of accessible string or numeric code or something to inspect which one it was. > But this is where I start to disagree. None should remain a singleton, but "no data available" could be its own thing, tied in with the way that you do your data storage and stats. As such, you wouldn't be checking it with 'is', so you wouldn't have that problem (the Python 'is' operator will only ever test for actual object identity). Keep None simple and dependable, and then "Missing Data" can be an entire class of values if you so desire. ChrisA _______________________________________________ Python-Dev mailing list -- python-dev(a)python.org To unsubscribe send an email to python-dev-leave(a)python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/6NY5NQC… Code of Conduct: http://python.org/psf/codeofconduct/

3 4

Making imports callable
by Abdur-Rahmaan Janhangeer Oct. 21, 2021

Oct. 21, 2021

Greetings list, Let's say i import module1 import module1 can we do module1() straight out of the box? If not i propose something like if __name__ == '__main__': __file__.callable = True Kind Regards, Abdur-Rahmaan Janhangeer about <https://compileralchemy.github.io/> | blog <https://www.pythonkitchen.com> github <https://github.com/Abdur-RahmaanJ> Mauritius

2 3

PEP able? : Request new 'fn' format specifier to specify relationship between positional arguments
by Shash Srinivasa Oct. 21, 2021

Oct. 21, 2021

Hi, Draft PEP attached: Proposal: def is_between(n1,n2,n3): fn"is number {n1} between {n2} and {n3}" fn"{n2}<{n1}<{n3}?" # the token "fn" immediately following the def statement, speficy the descriptive text # multipe "fn" specifiers describes multiple ways the funtion can be invoked # the "fn" statement is at the same indentation level as the def statement return (n1 > n2 and n1 < n3) . . . # if "fn" statements are found indented and not immediately following def statement, indicate function invocation. # function can be invoked using descriptive text pecified in "fn" statement. if fn"is number {value} between {valid} and {low_threshold}": print("underflow") if is_between(value,low_threshold,high_threshold): print("in range") if fn"{low_threshold}>{value}>{high_threshold} ?" print("overflow") . . Please comment Thanks Shash

1 0

Implementing string unary operators
by Jeremiah Vivian Oct. 20, 2021

Oct. 20, 2021

I posted a previous thread about overloading the unary `+` operator in strings with `ord`, and that expanded to more than just the unary `+` operator. So I'm saying now, there should be these implementations: > +string - `int(string, 10)` (or just `int(string)`) > -string - `int(string, 8)` > ~string - `int(string, 16)` Or: > +string - `string.lstrip()` > -string - `string.rstrip()` > ~string - `string.strip()` If anyone has better ideas, they can post it here.

15 41

Re: Type-hinting dictionaries for an arbitrary number of arbitrary key/value pairs? Counterpart to PEP 589?
by Alex Waygood Oct. 19, 2021

Oct. 19, 2021

Indeed — we essentially lie to mypy about the method resolution order for list, dict, etc (mypy thinks that list directly inherits from collections.abc.MutableSequence — see the typeshed stub here: https://github.com/python/typeshed/blob/32bc2161a107db20c2ebc85aad31c29730d…), but numeric ABCs are not special-cased by typeshed in the same way as collections ABCs. Fundamentally, mypy has no knowledge about virtual subclassing. Again, I covered this very comprehensively on StackOverflow ;) https://stackoverflow.com/questions/69334475/how-to-hint-at-number-types-i-… Best, Alex > On 16 Oct 2021, at 13:54, Steven D'Aprano <steve(a)pearwood.info> wrote: > > On Sat, Oct 16, 2021 at 09:54:13AM +0100, Alex Waygood wrote: >>>> On 16 Oct 2021, at 06:13, Steven D'Aprano <steve(a)pearwood.info> wrote: >>> Be careful about believing what you are told. >> Indeed, MyPy will correctly raise errors if you assign {None: []} to a >> variable annotated with dict[str, Number]. However, you'll find that >> MyPy also raises an error if you assign {'foo': 4} to a variable >> annotated with dict[str, Number]: > > Hah, serves me right for not testing it for both positive and negative > cases. > > In my very limited testing now, I see that the problem appears to be > with the Number type. mypy correctly accepts this: > > Data = Dict[str, int] > a: Data = {'spam': 42} > > (no errors), but if you change the annotation to use Number instead of > int, it wrongly flags that as a type error. > > Possibly mypy doesn't know that ints and floats are instances of Number? > > _______________________________________________ > Python-ideas mailing list -- python-ideas(a)python.org > To unsubscribe send an email to python-ideas-leave(a)python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/FJY3Q… > Code of Conduct: http://python.org/psf/codeofconduct/

4 4

Real Positional Arguments or OO Currying
by Mathew Elman Oct. 19, 2021

Oct. 19, 2021

I don't know if this has been suggested before, or if this is outlandishly impossible (though I would be surprised if it was), so apologies in advance if so. I have on occasion come across a situation where I use/write a signature like this: def insert_x_into_y(x, y): ... or worse def insert_into(item, container): ... where, despite a driving idea of python syntax being readability in english, the function signature is distinctly not english. "I'll just go and insert into this item that container", is not only never said but is actually ambiguous in english. What would be really cool, is if python let you write function signatures like this: def insert_(item)_into_(container): ... where the arguments dispersed between the function name are positional only argument, and any key word arguments would have to go at the end. It would create a function that could be called as: insert_(1)_into_(my_list) or insert__into_(1, my_list) The purpose of allowing both should be obvious - so that the function can be referenced and called in other places. (Rather than just skipping the brackets the function call with only the end parentheses could have a special stand in character e.g. ., ?, !, _ or other if that was more preferred.) This sort of signature is particularly annoying for boolean checks like `isinstance` (N.B. I am _not_ suggesting changing any builtins), which one could wrap with: def is_(obj)_an_instance_of_(type): return isinstance(obj, type) For precedence in other languages, this is similar to curried functions in functional languages e.g Haskell, especially if each part of a function were to be callable, which would be up for debate. Allowing each part to be called would make sense if each "next" partial function were an attribute on the previous and what it returned, making it a sort of object oriented currying. Then the syntax could be with a `.`: def is_(obj)._an_instance_of_(type): ... is_(1)._an_instance_of_(int) is_._an_instance_of_(1,int)

9 27

Accessing target name at runtime
by Jeremiah Paige Oct. 18, 2021

Oct. 18, 2021

Hello Python community, Earlier this year I put forward a proposal for decorators on variables which I felt was well received in concept, but fell flat on the syntax. One of the responses to that thread was rather than abusing the decorator, to use a new symbol to get the target of an assignment statement as a string on the right hand side. I would call this compile-time reflection of assignment targets, or simply reflection. Although initially continuing on with my original syntax, the idea of using such a token in my own code came up on more than one occasion this year and I wanted to see what it might really look like. From that, this proposal is to add a new token to Python, currently spelled <<<, which, at compile time, is replaced with the string representation of the nearest assignment target, either assignment statement or assignment expression. It does not bind with either keyword arguments or default parameters. Neither does it work in any kind of non-assignment binding. When encountered anywhere other than the right hand side of some assignment, it is an unconditional SyntaxError. Having access to the target at runtime is most often useful in meta programming and the standard library has several factory functions that require the user to repeat the variable name in the function call to get expected behavior. Bellow are some examples of where I believe the reflection token would be used if adopted. >>> Point = namedtuple(<<<, 'x, y, z') >>> Point <class '__main__.Point'> >>> UUIDType = NewType(<<<, str) >>> UUIDType __main__.UUIDType >>> class Colors(Enum): ... Black = <<< ... GRAY = <<< ... WHITE = <<< ... >>> Colors.GRAY.value 'GRAY' >>> HOME = '$' + <<< >>> HOME '$HOME' >>> if error := response.get(<<<): ... if errorcode := error.get(<<<): ... print(f"Can't continue, got {errorcode=}") ... Can't continue, got errorcode=13 To get a feel for using this new token I have created a fork of the 3.11 alpha that implements a *very* incomplete version of this new grammar, just enough to actually produce all of the examples above. It also passes a small new test suite with further examples https://github.com/ucodery/cpython/blob/reflection/Lib/test/test_reflection… .

7 17

Except * formatting
by Rob Cliffe Oct. 17, 2021

Oct. 17, 2021

I'm sending this to python-list because my emails to python-dev keep getting bounced back (after a few days delay). I've no idea why. Instead of `except group ...`, what about `except for ...`? No new keywords. Reads naturally in English. Hints that there is more than one kind of exception coming. Best wishes Rob Cliffe

2 1

Re: Structure Pattern for annotation
by Alex Waygood Oct. 17, 2021

Oct. 17, 2021

I agree with Steven. I very much like Abdulla's proposed syntax for dicts, TypedDicts and sets. But I'm not sure that the idea for `Annotated` is workable, and the proposal for lists seems too prone to ambiguity, given how extensively square brackets are already used in typing syntax. One question about the proposals, however: how would we represent other mappings apart from dicts? Would `collections.abc.Mapping` and `types.MappingProxyType` still be spelled as `Mapping[str, int]` and `MappingProxyType[str, int]`? If so, that would be a slightly annoying inconsistency. I do also quite like the idea of an improved syntax for tuples specifically. Tuples are already very different types to lists/strings/etc in the context of typing, essentially representing heterogeneous structs of fixed length rather than homogenous sequences of unspecified length. As such, I think it "makes sense" to special-case tuples without special-casing other sequences such as list, `collections.abc.Sequence` or `collections.deque`. def foo() -> tuple[list[list[str]], list[list[str]]] would become def foo() -> (list[list[str]], list[list[str]]) That feels quite readable and natural, to me. Best, Alex > On 14 Oct 2021, at 09:25, Steven D'Aprano <steve(a)pearwood.info> wrote: > On Thu, Oct 14, 2021 at 12:32:57AM +0400, Abdulla Al Kathiri wrote: > >> Today I found myself write a function that returns a tuple of list of >> list of strings (tuple[list[list[str]], list[list[str]]]). Wouldn’t it >> easier to read to write it like the following: >> ([[str]], [[str]])? > > Not really. Your first example is explicit and I can get the meaning by > just reading it out loud: > > tuple[list[list[str]], list[list[str]]] > > "tuple (of) list (of) list (of) str, list (of) list (of) str > > Your abbreviated version: > > ([[str]], [[str]]) > > is too terse. I have to stop and think about what it means, not just > read it out loud. Without the hint of named types (tuple and list), my > first reaction to seeing [str] is "is this an optional string?". > > And then I wonder why it's not written: > > ([[""]], [[""]]) > > Why abbreviate list and tuple but not string? > > Code is read more than it is written, and can be too terse as well as > too verbose. > > On the other hand: > >> Similarly for TypedDict, replace the following.. >> class Movie(TypedDict): >> name: str >> year: int >> with >> {‘name’: str, ‘year’: int} > > To my eye, that one does work. As far as I know, curly brackets {} > aren't otherwise used in annotations (unlike square brackets), and they > don't look like "optional" to me. They look like a dict. > > So on first glance at least, I think that: > > {'name': str, 'year': int} > > is better than the class syntax we already have. > > > Likewise: > >> dict[str, int] will be {str: int} >> set[int] will be {int}. > > work for me too. > > >> Also, Annotated[float, “seconds”] can be replaced with something like >> float #seconds indicating whatever comes after the hashtag is just >> extra information similar to a normal comment in the code. > > No, because the # indicates that the rest of the line is a comment. This > is already legal: > > def func(d: {str # this is an actual comment > : int}) -> Any: ... > > so this would be ambiguous between a real comment and an annotation. > > Even if we agreed to change the behaviour of comments, you suggested: > > func(d: {str # product label: [float] # prices from 2000 to 2015}) > > How is the interpreter to know that the first annotation is just > > "product label" > > rather than this? > > "product label: [float] # prices from 2000 to 2015" > > So I don't think this works. > > > -- > Steve > _______________________________________________ > Python-ideas mailing list -- python-ideas(a)python.org > To unsubscribe send an email to python-ideas-leave(a)python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/T7E54… > Code of Conduct: http://python.org/psf/codeofconduct/

7 8

Type-hinting dictionaries for an arbitrary number of arbitrary key/value pairs? Counterpart to PEP 589?
by Sebastian M. Ernst Oct. 16, 2021

Oct. 16, 2021

Hi all, disclaimer: I have no idea on potential syntax or if it is applicable to a wide audience or if there is already a good solution to this. It is more like a "gap" in the type hint spec that I ran across in a project. In function/method signatures, I can hint at dictionaries for example as follows: ```python from numbers import Number from typing import Dict from typeguard import typechecked Data = Dict[str, Number] @typechecked def foo(bar: Data): print(bar) ``` Yes, this is using run-time checks (typeguard), which works just fine. Only strings as keys and Number objects as values are going through. (I was told that MyPy does not get this at the moment.) The issue is that `bar` itself still allows "everything" to go in (and out): ```python @typechecked def foo2(bar: Data): bar[1.0] = b'should not be allowed' ``` PEP 589 introduces typed dictionaries, but for a fixed set of predefined keys (similar to struct-like constructs in other languages). In contrast, I am looking for an arbitrary number of typed keys/value pairs. For reference, related question on SO: https://stackoverflow.com/q/69555006/1672565 Best regards, Sebastian

6 8

Antichecked exceptions
by Soni L. Oct. 16, 2021

Oct. 16, 2021

PEP 479 introduced this idea that basically boils down to raising your exceptions only when you mean it. Specifically: generators only raise StopIteration when they return/exit/terminate. And they prevent the user from raising it under other circumstances. Can we call this "antichecked exceptions"? Generators specifically check for StopIteration raised in undesired circumstances, and make sure they get wrapped. This is a form of runtime checking, so the "checked" part of the name makes sense. But this is unlike "checked exceptions", like those in java, that mostly just encourage uhh silently ignoring the exception? (And those are *statically* checked anyway...) So the prefix "anti" distinguishes it from that. Obviously this name is meant to be more broad than just the implementation of generators. (We made a pypi package that tries to expand on the concept.) But, thoughts? Naming things is powerful. (Is this the wrong list for this? Uhh like it's a name idea for a thing in python? But uhh... it's not so much a change proposal? Except to refer to it by this new name? But anyway...)

2 1

Structure Pattern for annotation
by Abdulla Al Kathiri Oct. 15, 2021

Oct. 15, 2021

Hello, Today I found myself write a function that returns a tuple of list of list of strings (tuple[list[list[str]], list[list[str]]]). Wouldn’t it easier to read to write it like the following: ([[str]], [[str]])? Similarly for TypedDict, replace the following.. class Movie(TypedDict): name: str year: int with {‘name’: str, ‘year’: int} dict[str, int] will be {str: int} set[int] will be {int}. This is similar to the new proposed Callable alternative (i.e., (int, float) -> float) since it mimics the structure of a function. Also, Annotated[float, “seconds”] can be replaced with something like float #seconds indicating whatever comes after the hashtag is just extra information similar to a normal comment in the code. Imagine you could replace the following def func(d: dict[Annotated[str, “product label”], Annotated[list[float], “prices from 2000 to 2015”]])… With def func(d: {str # product label: [float] # prices from 2000 to 2015})… Abdulla Sent from my iPhone

3 3

Re: dict_items.__getitem__?
by Alex Waygood Oct. 14, 2021

Oct. 14, 2021

I think there definitely should be a more obvious way to do this (specifically the first and last keys/values/items of a dictionary — I'm ambivalent about the others, since they won't always be fast, as discussed). An anti-pattern you see quite often on Stack Overflow to get the first key of a dictionary is something like the following: first_key = list(mydict.keys())[0] Whereas obviously, a much better way (especially if it's a very large dictionary) is to do: first_key = next(iter(mydict)) [Christopher Barker] > I'll leave it exercise for the reader to find that thead For reference, the (very long) previous thread is here: https://mail.python.org/archives/list/python-ideas@python.org/thread/S7UMTW…. [Inada Naoki] > I think we can add `itertools.first()` for this idiom, and > `itertools.last()` for `next(iter(reversed(x)))` idiom. I like this idea, a lot. Another possibility I've been wondering about was whether several methods should be added to the dict interface: dict.first_key = lambda self: next(iter(self)) dict.first_val = lambda self: next(iter(self.values())) dict.first_item = lambda self: next(iter(self.items())) dict.last_key = lambda self: next(reversed(self)) dict.last_val = lambda self: next(reversed(self.values())) dict.last_item = lambda self: next(reversed(self.items())) But I think I like a lot more the idea of adding general ways of doing these things to itertools. Best, Alex > On 5 Oct 2021, at 05:30, Christopher Barker <pythonchb(a)gmail.com> wrote: > > On Mon, Oct 4, 2021 at 5:46 PM Erik Demaine <edemaine(a)mit.edu> wrote: >> Have folks thought about allowing indexing dictionary views as in the >> following code, where d is a dict object? >> >> d.keys()[0] >> d.keys()[-1] >> d.values()[0] >> d.values()[-1] >> d.items()[0] >> d.items()[-1] # item that would be returned by d.popitem() > > since dicts were made order-preserving, indexing the keys, items, etc does make some sense. > > > > I've also often wanted to get an arbitrary item/key from a dictionary, and > > This is indeed one of the use cases identified. > >> I found some related discussion in >> https://mail.python.org/archives/list/python-ideas@python.org/thread/QVTGZD… >> but not this exact idea. > > That's a pretty different idea but this exact idea has been discussed on this list relatively recently. I still like it, but there wan't much general support. > > I'll leave it exercise for the read to find that thead, but it is there, and I suggest you look for it if you want to further pursue this idea. > > -CHB > > > -- > Christopher Barker, PhD (Chris) > > Python Language Consulting > - Teaching > - Scientific Software Development > - Desktop GUI and Web Development > - wxPython, numpy, scipy, Cython > _______________________________________________ > Python-ideas mailing list -- python-ideas(a)python.org > To unsubscribe send an email to python-ideas-leave(a)python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/RAEDZ… > Code of Conduct: http://python.org/psf/codeofconduct/

18 63

Overloading unary plus in strings with "ord"
by Jeremiah Vivian Oct. 12, 2021

Oct. 12, 2021

So `ord` is already a really fast function with (last check before this thread was posted) 166 nsec per loop. But I'm wondering... doing `ord(a)` produces this bytecode: > 2 4 LOAD_NAME 1 (ord) > 6 LOAD_NAME 0 (a) > 8 CALL_FUNCTION 1 > 10 POP_TOP > 12 LOAD_CONST 1 (None) > 14 RETURN_VALUE But doing `+a` only produces this: > 2 4 LOAD_NAME 0 (a) > 6 UNARY_POSITIVE > 8 POP_TOP > 10 LOAD_CONST 1 (None) > 12 RETURN_VALUE So an operator has its own bytecode, doesn't need to `LOAD_*` a function, and doesn't have the impact on performance when converting arguments to the format of the (after `POP()`ing every argument) TOS function and then calling that function. And also, the unary `+` of strings only copies strings, which should be redundant in most cases. Maybe `ord` can take the place of unary `+` in strings?

5 18

Adding a Version type
by Finn Mason Oct. 11, 2021

Oct. 11, 2021

Hello all, I was thinking about the following idioms: __version__ = "1.0" import sys if sys.version_info < (3, 6): # Yell at the user I feel like there could be a better way to define versions. There's no real standard way to specify versions in Python, other than "use semantic versioning." This can cause problems such as: * Uncertainty of the type of the version (commence the pattern matching! Or `if... elif` statements if that's your thing) * No real way to specify an alpha, beta, pre, or post version number in a tuple of ints, e.g. `(2, 1, 0, "post0")` doesn't really work. I propose a new `Version` type, a standard way to define versions. The signature would look something like `Version(major: int, minor: int = 0, patch: Optional[int] = None, alpha: Optional[int] = None, beta: Optional[int] = None, pre: Optional[int] = None, post: Optional[int] = None)`. To maintain backwards compatibility, comparisons such as `Version(1, 2) == (1, 2)` and `Version(1, 2) == "1.2"` will return `True`. `str(Version(1, 2))` will return `"1.2"` (to clarify, if `patch` were 0 instead of None, it would return `"1.2.0"`). There will be an `as_tuple()` method to give the version number as a tuple (maybe named tuple?), e.g. `Version(1, 2).as_tuple()` will return `(1, 2)`. A problem is that not all versioning systems are covered by this proposal. For example, I've done some programming in C#, and many C# programs use a four number system instead of the three number system usually used in Python code, i.e. major.minor.patch.bugfix instead of major.minor.patch. (This may not seem very different, but it often is.) Where to place this type is an open question. I honestly have no good ideas. This could just be implemented as a PyPI package, but there may not be a point in adding a whole dependency to use a semantic type that other people probably aren't using (which wouldn't be the case if it were part of core Python). This would be implemented in Python 3.11 or 3.12, but I'd recommend a backport on PyPI if implemented. Questions? Thoughts? Suggestions to improve it?

6 8

socketserver StreamRequestHandler non-blocking read__
by Evan Greenup Oct. 10, 2021

Oct. 10, 2021

For StreamRequetHandler's rfile attribute, it will block when the read readline readlines reach the end. It would be nice to add non-blocking read because the seek() is unavailable for rfile. and there is no other way to detect if it is ended. When the stream is empty and waiting for new input, then it will return None type. This would be a nice feature.

1 0

There must be a better way to do class-attributes -- allow descriptor.__set__ on `type`, not just `object`
by Randolf Scholz Oct. 10, 2021

Oct. 10, 2021

Over the past few weeks/months I have had on an off issues with the way python handles class-attributes. Consider the default way to do it: ``` class A: attr = 2 """Docstring of the attribute""" ``` There are 2 main issues I have with this way of setting class-attributes: 1. It does not allow to set class-attributes that depend on other class-attributes. (Example: setting a path that depends on cls.__name__) 2. It does not allow to set lazily evaluated attributes. If setting the class-attribute involves an expensive computation, it will be done at class definition time. One possible resolution is to use metaclasses, however this comes with its own issues: 1. MetaClasses do not work nice with documentation. Attributes defined my metaclasses will neither show up in help(), dir(), nor will they be in the correct place when documenting the class with tools like sphinx. 2. Who wants to write an additional metaclass whenever they want to write a class and have some minimal class-attribute processing? A new possible resolution, possible since python 3.9, is the combination of @classmethod and @property. However, this has it's own problems and pitfalls, cf. https://bugs.python.org/issue44904, https://bugs.python.org/issue45356. In fact, after lots of trial and error and reading into descriptors I figured that "true" class-properties are not possible in the current versions of python for a very simple reason: ╔══════════════════════════════════════════════════════╗ ║descriptor's __set__ method is never called when setting a class-attribute. (see MWE) ║ ╚══════════════════════════════════════════════════════╝ Therefore, I propose the following: 1. By default `type.__setattr__ ` should check whether the attribute implements `__set__` and if so, use that instead. (it already works like this for object.__setattr__), (cf. PatchedTrig below) 2. There should be a clearer distinction between class-attributes and instances-attributes. Instances should probably by default not be allowed to change class-attributes, since this can affect global state. 3. It would be very nice to have a built-in @attribute decorator, much like @property, but specifically for stetting class attributes. (again, metaclasses are possible but the issues with documentation compatibility seem very bad) (pretty much what @classmethod@property promises but doesn't fulfill.) ## MWE ```python import math class TrigConst: def __init__(self, const=math.pi): self.const = const def __get__(self, obj, objtype=None): return self.const def __set__(self, obj, value): print(f"__set__ called", f"{obj=}") self.const = value class Trig: const = TrigConst() ``` Then ```python Trig().const # calls TrigConst.__get__ Trig().const = math.tau # calls TrigConst.__set__ Trig.const # calls TrigConst.__get__ Trig.const = math.pi # overwrites TrigConst attribute with float. ``` Patched version ```python class PatchedSetattr(type): def __setattr__(cls, key, value): if hasattr(cls, key): obj = cls.__dict__[key] if hasattr(obj, "__set__"): obj.__set__(cls, value) else: super().__setattr__(key, value) class PatchedTrig(metaclass=PatchedSetattr): const = TrigConst() ``` Then ```python cls = Trig print(cls.const, type(cls.__dict__["const"])) cls.const = math.tau print(cls.const, type(cls.__dict__["const"])) ``` gives ``` 3.141592653589793 <class '__main__.TrigConst'> 6.283185307179586 <class 'float'> ``` But ```cls cls = PatchedTrig print(cls.const, type(cls.__dict__["const"])) cls.const = math.tau print(cls.const, type(cls.__dict__["const"])) ``` gives ``` 3.141592653589793 <class '__main__.TrigConst'> __set__ called obj=<class '__main__.PatchedTrig'> 6.283185307179586 <class '__main__.TrigConst'> ```

1 0

Invalidate list iterators after size changes?
by Tyler Hou Oct. 9, 2021

Oct. 9, 2021

Right now, the following code is valid in Python 3.9 (and infinitely loops): ``` lst = [1, 2, 3] for i in lst: lst.append(i) ``` IMO this is a source of bugs, and even though this behavior is technically documented (https://bugs.python.org/issue32767), I don't know if it should belong in the core language. For example, you cannot change the size of a set or dict during iteration; you will get a RuntimeError. Allegedly this is due to the underlying implementation of the dict/set iterators (https://bugs.python.org/issue22084) but I think it would be beneficial this was made part of the public API for list, set, and dict iterators. Concretely, I'm proposing the following set of changes to Python: 1. If the size of a list, set, or dict changes, invalidate all existing iterators to that containers. Document this behavior as part of the public API. 2. Attempting to call next() on an invalidated iterator raises a RuntimeError. 3. Add a key_iter function to itertools (name up for debate), which replicates the existing functionality for lists. It could also make a copy of the keys of a set/dict, so you could use it to mutate those as well during iteration (to allow users to write functions that mutate an arbitrary duck-typed collection) 4. Gate (1, 2) for lists behind a __future__ flag with a long migration window (e.g. ~3.16). Benefits: 1. Reduce a source of bugs (deleting while iterating forward through a list often means that your logic skips elements; appending can lead to infinite loops or unexpected behavior). 2. Make the iterator invalidation consistent between lists, sets, and dicts. 3. More consistent behavior with collections in other languages (C++, Java). Drawbacks: 1. Migration costs. But mutating during iteration should be relatively rare, given that most people discourage mutating while iterating (https://google.github.io/styleguide/pyguide.html#284-decision, https://stackoverflow.com/a/1637875/1625484) __future__ feature flag plus deprecation warnings should go along way to help people migrate their code. We would also offer an easy in-place fix (key_iter). 2. You can't write algorithms that append to a list (e.g. BFS) with the normal list iterator. Again, you could use key_iter (or use a deque with block size == 1; see Open Questions). Open Questions Is (3) necessary? It's not hard to write an implementation yourself. What should be done about deque for block sizes > 1? If this change gets made, should we add way to create a deque with block size == 1 whose iterators are never invalidated?

6 5

String method to check for a float
by dpalit17＠outlook.com Oct. 7, 2021

Oct. 7, 2021

It would be nice to have a string method that checks for a float. Currently there is no support for this, either built-in or in the standard library. There is a thread, dating back to Dec 2020, that proposes a trivial implementation for str.isfloat . I was thinking of a method that did more. Consider the following code. It returns True if the string is a proper float, False if it is an int and None otherwise. def isfloat(s): try: int(s) return False except ValueError: try: float(s) return True except ValueError: return None This will be useful when we want to preserve the type of the number that is in string format. Anywhere a number is input as a string to a method and we want to later on output the original number, we can use the above. If, instead (as suggested in the other thread), the string is simply converted to a float, then the info that the string was an int is lost.

6 20

More efficient list copying
by Steven D'Aprano Oct. 5, 2021

Oct. 5, 2021

This half-baked idea is inspired by this thread here: https://mail.python.org/archives/list/python-ideas@python.org/message/5LGWV… which in turn was inspired by this Stackoverflow post: https://stackoverflow.com/questions/56966429/getting-pairs-of-one-item-and-… Problem: today, more and more people are using Python to work with Big Data, or at least Biggish Data. For some folks, it is not unusual to have lists with a gigabyte or more or data. So imagine you have a list with a hundred million items, and you need a copy of the entire list. Easy: alist = [...] # 100 million items blist = alist[:] And that ought to be pretty efficient too, making the slice is basically just a copy of a bunch of pointers, plus a bit of overhead. A good implementation should have lightning fast list slicing, close to the speed of memcopy in C. Give or take. But now suppose you want a copy of all the items except the item in position (let's say) 1000. Here's a bad way: blist = alist[:] del blist[1000] That's inefficient because the list has to shift almost a hundred million items over, and I think they have to be moved one at a time. Which is slowish, even in C. Here's another way: blist = alist[:1000] + alist[1001:] That has to make a slice (copy 1000 items), a second slice (copy another many million items), then make a third copy to concatenate them, then garbage collect the two temporary slices. And while you might have enough memory for double the original list (alist and blist) you might not have enough for triple (alist, the two slices, blist). There are lots of other variants we could come up with, but ultimately we're doing lots of extra work that has to be thrown away, and that costs time or memory or both. How about if lists had an in-place method for doing a fast copy of pointers from one list to another? We could do this: blist = [None]*(len(alist)-1) # Preallocate. blist.copyfrom(alist, index=0, start=0, end=1000) blist.copyfrom(alist, index=1000, start=1001) No temporary slices are needed. Here's a pure-Python (and therefore slow) implementation: def copyfrom(dest, source, index=0, start=0, end=None): if end is None: end = len(source) if len(dest) - index < end - start: raise ValueError('destination too small') for i in range(start, end): dest[index+i-start] = source[i] In pure Python, copying each item over one by one will save memory but cost a lot of time. Doing it as a method in C should save both memory and time. The implementation could optimize the cases where the source is a list, and fall back on iteration for other sequence types. Now *personally* I don't work with a lot of Big Data, so this doesn't really scratch any of my personal itches, but I thought I'd throw it out there for others to tell me why it's a dumb idea :-) -- Steve

6 6

Syntax Sugar for __name__ == "__main__" boilerplate?
by Jonathan Crall Oct. 5, 2021

Oct. 5, 2021

I was curious if / what sort of proposals have been considered for simplifying the pattern: ``` def main(): ... if __name__ == "__main__": main() ``` I imagine this topic must have come up before, so I'd be interested in any relevant history. But unless I'm missing something, it seems like adding some easier alternative to this cumbersome entrypoint syntax would be worth considering. My motivation for writing this suggestion is in an attempt to stop a common anti-pattern, where instead of defining a `main` function (or a function by any other name) an simply calling that by adding the above two lines, a lot of Python users I work with will just start dumping their logic into the global scope of the module. Needless to say, this can have consequences. If there was some default builtin, let's call it `__main__` for now (open to suggestions), that took a function as an argument and conditionally executed it if `__name__ == "__main__"` in the caller's scope, that would allow us to simplify the above boilerplate to a single line with no extra indentation: ``` def main(): ... __main__(main) ``` In addition to being simpler, it would allow users to avoid the trap of adding logic that impacts the global scope. It would also save me some keystrokes, which I'm always grateful for. Furthermore, it could be used as a decorator (and the use-case wouldn't be unreasonable!), and we all know how much new Python users love decorators when they find out about them. ``` @__main__ def main(): ... ``` Maybe having such a builtin would discourage globals and help new users get the use-decorators-everywhere bug out of their system. -- -Dr. Jon Crall (him)

19 38

dict_items.__getitem__?
by Erik Demaine Oct. 4, 2021

Oct. 4, 2021

Have folks thought about allowing indexing dictionary views as in the following code, where d is a dict object? d.keys()[0] d.keys()[-1] d.values()[0] d.values()[-1] d.items()[0] d.items()[-1] # item that would be returned by d.popitem() I could see value to the last form in particular: you might want to inspect the last item of a dictionary before possibly popping it. I've also often wanted to get an arbitrary item/key from a dictionary, and d.items()[0] seems natural for this. Of course, the universal way to get the first item from an iterable x is item = next(iter(x)) I can't say this is particularly readable, but it is functional and fast. Or sometimes I use this pattern: for item in x: break If you wanted the last item of a dictionary d (the one to be returned from d.popitem()), you could write this beautiful code: last = next(iter(reversed(d.items()))) Given the dictionary order guarantee from Python 3.7, adding indexing (__getitem__) to dict views seems natural. The potential catch is that (I think) it would require linear time to access an item in the middle, because you need to count the dummy elements. But accessing [i] and [-i] should be doable in O(|i|) time. (I've wondered about the possibility of doing binary or interpolation search, but without some stored index signposts, I don't think it's possible.) Python is also full of operations that take linear time to do: list.insert(0, x), list.pop(0), list.index(), etc. But it may be that __getitem__ takes constant time on all built-in data structures, and the apparent symmetry but very different performance between dict()[i] and list()[i] might be confusing. That said, I really just want d[0] and d[-1], which is when these are fast. I found some related discussion in https://mail.python.org/archives/list/python-ideas@python.org/thread/QVTGZD… but not this exact idea. Erik -- Erik Demaine | edemaine(a)mit.edu | http://erikdemaine.org/

3 2

Power Assertions: Is it PEP-able?
by noam＠10ne.org Oct. 4, 2021

Oct. 4, 2021

Hi all, I’d like your comments and feedback on an enhancement that introduces power assertions to the Python language. Proposal -------- This feature is inspired by a similar feature of the Groovy language[1], and is effectively a variant of the `assert` keyword. When an assertion expression evaluates to `False`, the output shows not only the failure, but also a breakdown of the evaluated expression from the inner part to the outer part. For example, a procedure like: ```python class SomeClass: def __init__(self): self.val = {'d': 'e'} def __str__(self): return str(self.val) sc = SomeClass() assert sc.val['d'] == 'f' ``` Will result in the output: ```python Assertion failed: sc.val['d'] == f | | | | e False | {'d': 'e'} ``` See link [2] if the formatting above is screwed up. In the output above we can see the value of every part of the expression from left to right, mapped to their expression fragment with the pipe (`|`). The number of rows that are printed depend on the value of each fragment of the expression. If the value of a fragment is longer than the actual fragment (`{'d': 'e'}` is longer than `sc`), then the next value (`e`) will be printed on a new line which will appear above. Values are appended to the same line until it overflows in length to horizontal position of the next fragment. The information that’s displayed is dictated by the type. If the type is a constant value, it will be displayed as is. If the type implements `__str__`, then the return value of that will be displayed. It is important to note that expressions with side effects are affected by this feature. This is because in order to display this information, we must store references to the instances and not just the values. Rational -------- Every test boils down to the binary statement "Is this true or false?", whether you use the built-in assert keyword or a more advanced assertion method provided by a testing framework. When an assertion fails, the output is binary too — "Expected x, but got y". There are helpful libraries like Hamcrest which give you a more verbose breakdown of the difference and answer the question "What exactly is the difference between x and y?". This is extremely helpful, but it still focuses on the difference between the values. We need to keep in mind that a given state is normally an outcome of a series of states, that is, one outcome is a result of multiple conditions and causes. This is where power assertion comes in. It allows us to better understand what led to the failure. Implementation -------- I’ve already built a fully functional implementation[2] of this feature as part of my Python testing framework - Nimoy[3]. The current implementation uses AST manipulation to remap the expression to a data structure[4] at compile time, so that it can then be evaluated and printed[5] at runtime. [1] http://docs.groovy-lang.org/next/html/documentation/core-testing-guide.html… [2] https://browncoat-ninjas.github.io/nimoy/examples/#power-assertions-beta [3] https://github.com/browncoat-ninjas/nimoy/ [4] https://github.com/browncoat-ninjas/nimoy/blob/develop/nimoy/ast_tools/expr… [5] https://github.com/browncoat-ninjas/nimoy/blob/develop/nimoy/assertions/pow…

8 17

Re: Feature request enumerate_with_rest or enumerate with skip or filter callback
by Laurent Lyaudet Oct. 4, 2021

Oct. 4, 2021

Le dim. 3 oct. 2021 à 16:21, <python-ideas-request(a)python.org> a écrit : > Date: Mon, 4 Oct 2021 01:03:34 +1100 > From: Steven D'Aprano <steve(a)pearwood.info> > Subject: [Python-ideas] Re: Feature request enumerate_with_rest or > enumerate with skip or filter callback > To: python-ideas(a)python.org > Message-ID: <20211003140333.GR16229(a)ando.pearwood.info> > Content-Type: text/plain; charset=us-ascii > > Hi Laurent, Hello Steve, > It is not clear to me what you mean by "filter by indices". > > On Sat, Oct 02, 2021 at 10:25:05PM +0200, Laurent Lyaudet wrote: > > > The idea is to filter a list by indices : > [...] > > Since filter() returns an iterator instead of a list, it could do what > > is needed... if the callback had access to the index like the > > Javascript array filter function. > > Do you mean this? > > https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Ob… Yes exactly, > You shouldn't assume we are all experts on Javascript :-) > No problem, I'll give more links next time :) > If that is what you want, it is easy to get access to the index. We can > just do: > > filter(function, enumerate(items)) > That's exactly what I proposed. Except for the fact that I incorrectly permuted the two arguments of filter. I quote my second email : Le sam. 2 oct. 2021 à 22:35, <python-ideas-request(a)python.org> a écrit : > Date: Sat, 2 Oct 2021 22:25:05 +0200 > From: Laurent Lyaudet <laurent.lyaudet(a)gmail.com> >... > Currently, the following solution is available : > filter(enumerate(my_list), lambda x: x[0] != i) > But it is slightly ugly and unefficient to have two function calls for > such a simple task I think. I feel uncomfortable because it is often the case that I write explicitly something, and I get an answer as if the person answered without reading all my email or reading another version. > If you want something else, I'm afraid you will have to explain in more > detail what you want, sorry. Again for my second email, I would like one of these 3 options. > What would be your prefered way of doing this ? > enumerate(my_list, filter_callback=(lambda x: x != i)) > filter_by_index(my_list, lambda x: x != i) > # à la JS > filter(my_list, lambda _, x: x != i) filter_by_index(lambda x: x != i, my_list) could be defined as follow (I permuted args to have similar order to python's filter()): def filter_by_index(function, my_list): for i, item in enumerate(my_list) if function(i): yield item like filter is just (without the case function=None) filter(function, my_list): for i, item in enumerate(my_list) if function(item): yield item The problem is not that it is hard to code in Python. The problem is that it is a basic building block for an iterators tools library like itertools. enumerate(my_list, filter_callback=(lambda x: x != i)) could be defined in a similar way but there are two variants. It just needs to return one of the two indices of the item: - either the index of the item in the original sequence, - or the index of the item in the filtered sequence (new counter). For my use case, I do not need the index so I have no argument in favor of either one of the two indices. > # à la JS > filter(my_list, lambda _, x: x != i) You provided the link from MDN and an helper function for that. The lambda takes 2 or 3 parameters: item, index, array/list/sequence and returns true for item that must be output. I see nothing more to say. Best regards, Laurent Lyaudet

1 0

Re: Python-ideas Digest, Vol 179, Issue 25
by Laurent Lyaudet Oct. 4, 2021

Oct. 4, 2021

Le dim. 3 oct. 2021 à 16:21, <python-ideas-request(a)python.org> a écrit : > Date: Mon, 4 Oct 2021 01:03:34 +1100 > From: Steven D'Aprano <steve(a)pearwood.info> > Subject: [Python-ideas] Re: Feature request enumerate_with_rest or > enumerate with skip or filter callback > To: python-ideas(a)python.org > Message-ID: <20211003140333.GR16229(a)ando.pearwood.info> > Content-Type: text/plain; charset=us-ascii > > Hi Laurent, Hello Steve, > It is not clear to me what you mean by "filter by indices". > > On Sat, Oct 02, 2021 at 10:25:05PM +0200, Laurent Lyaudet wrote: > > > The idea is to filter a list by indices : > [...] > > Since filter() returns an iterator instead of a list, it could do what > > is needed... if the callback had access to the index like the > > Javascript array filter function. > > Do you mean this? > > https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Ob… Yes exactly, > You shouldn't assume we are all experts on Javascript :-) > No problem, I'll give more links next time :) > If that is what you want, it is easy to get access to the index. We can > just do: > > filter(function, enumerate(items)) > That's exactly what I proposed. Except for the fact that I incorrectly permuted the two arguments of filter. I quote my second email : Le sam. 2 oct. 2021 à 22:35, <python-ideas-request(a)python.org> a écrit : > Date: Sat, 2 Oct 2021 22:25:05 +0200 > From: Laurent Lyaudet <laurent.lyaudet(a)gmail.com> >... > Currently, the following solution is available : > filter(enumerate(my_list), lambda x: x[0] != i) > But it is slightly ugly and unefficient to have two function calls for > such a simple task I think. I feel uncomfortable because it is often the case that I write explicitly something, and I get an answer as if the person answered without reading all my email or reading another version. > If you want something else, I'm afraid you will have to explain in more > detail what you want, sorry. Again for my second email, I would like one of these 3 options. > What would be your prefered way of doing this ? > enumerate(my_list, filter_callback=(lambda x: x != i)) > filter_by_index(my_list, lambda x: x != i) > # à la JS > filter(my_list, lambda _, x: x != i) filter_by_index(lambda x: x != i, my_list) could be defined as follow (I permuted args to have similar order to python's filter()): def filter_by_index(function, my_list): for i, item in enumerate(my_list) if function(i): yield item like filter is just (without the case function=None) filter(function, my_list): for i, item in enumerate(my_list) if function(item): yield item The problem is not that it is hard to code in Python. The problem is that it is a basic building block for an iterators tools library like itertools. enumerate(my_list, filter_callback=(lambda x: x != i)) could be defined in a similar way but there are two variants. It just needs to return one of the two indices of the item: - either the index of the item in the original sequence, - or the index of the item in the filtered sequence (new counter). For my use case, I do not need the index so I have no argument in favor of either one of the two indices. > # à la JS > filter(my_list, lambda _, x: x != i) You provided the link from MDN and an helper function for that. The lambda takes 2 or 3 parameters: item, index, array/list/sequence and returns true for item that must be output. I see nothing more to say. Best regards, Laurent Lyaudet

1 0

Shorthand syntax for lambda functions that have a single parameter
by Dominik Vilsmeier Oct. 4, 2021

Oct. 4, 2021

Lambda functions that have a single parameter are a common thing, e.g. for "key" functions: `sorted(items, key=lambda x: x['key'])`. For these cases however, the rather long word "lambda" together with the repetition of the parameter name, results in more overhead than actual content (the `x['key']`) and thus makes it more difficult to read. This is also a difficulty for `map` and `filter` for example where there's lots of visual overhead when using a lambda function and hence it's difficult to read: `filter(lambda x: x > 0, items)` or `map(lambda x: f'{x:.3f}', items)`. Hence the proposal is to add a new syntax via the new token `?`. For the examples above: * `sorted(items, key=?['key'])` * `filter(? > 0, items)` * `map(f'{?:.3f}', items)` The rules are simple: whenever the token `?` is encountered as part of an expression (at a position where a name/identifier would be legal), the expression is replaced by a lambda function with a single parameter which has that expression as a return value, where any instances of `?` are replaced by the name of that single parameter. For example: * `?['key']` translates to `lambda x: x['key']` * `? > 0` translates to `lambda x: x > 0` * `f'{?:.3f}'` translates to `lambda x: f'{x:.3f}'` * `?*?` translates to `lambda x: x*x` With the difference that the replacement function would use an unnamed parameter, i.e. not shadowing any names from outer scopes. So `? * x` would use `x` from enclosing scopes, not as the lambda parameter. Regarding operator precedence, the rules would be similar as for lambda functions, i.e. it will include as much in the expression as it would for a lambda function. To find more example use cases, I used `grep -rnE 'lambda [_a-zA-Z][_a-zA-Z0-9]*:' Lib/` on the CPython standard library to find single parameter lambda functions and there are various results coming up (although not all are applicable, e.g. because they return a constant value). I'll include a few examples below, but the actual list is much longer: ``` modes[char] = max(items, key=lambda x: x[1]) modes[char] = max(items, key=?[1]) # proposed new syntax obj = unwrap(obj, stop=(lambda f: hasattr(f, "__signature__"))) obj = unwrap(obj, stop=hasattr(?, "__signature__")) s = re.sub(r"-[a-z]\b", lambda m: m.group().upper(), s) s = re.sub(r"-[a-z]\b", ?.group().upper(), s) members.sort(key=lambda t: (t[1], t[0])) members.sort(key=(?[1], ?[0])) ``` Of course, there is no requirement to use `?` as the token, it could be any other character that's currently not used by Python, i.e. `$` or `!` would be possible too.

13 22

Re: Feature request enumerate_with_rest or enumerate with skip or filter callback
by Laurent Lyaudet Oct. 3, 2021

Oct. 3, 2021

Le sam. 2 oct. 2021 à 15:54, <python-ideas-request(a)python.org> a écrit : > Date: Sat, 2 Oct 2021 12:45:33 +0200 > From: Laurent Lyaudet <laurent.lyaudet(a)gmail.com> > Subject: [Python-ideas] Feature request enumerate_with_rest or > enumerate with skip or filter callback > To: python-ideas(a)python.org > Message-ID: > <CAB1LBmsXQJbeLuyK6tG8aP=kb36DAR==VcajO-zHuHJGnfvAJA(a)mail.gmail.com> > Content-Type: text/plain; charset="UTF-8" > > Hello, > > This is a very simple feature request that does not break anything but > I don't know if you may find it interesting. > It would be nice to have a function or method of list objects that does this : > - First idea : > def enumerate_with_rest(my_list): > for i, item in enumerate(my_list): > yield i, item, my_list[:i] + my_list[i + 1:] > > It could be called easily with: > > for i, item, rest in enumerate_with_rest(my_list): > # do something > > or > for i, item, rest in my_list.enumerate_with_rest(): > # do something > > I am not the only one who had the same need : > https://stackoverflow.com/questions/56966429/getting-pairs-of-one-item-and-… > > It would be nice to have an optimized C function for this. > However, it may be less interesting than this : > - Second idea > enumerate_with_rest above has quadratic complexity. > It is probably true that most processes that use it will also have > quadratic complexity. > However, it would be better to return an iterator instead of a list > for the rest, > it would use less space. > For this a param skip to enumerate would do the job > def enumerate_with_rest(my_list): > for i, item in enumerate(my_list): > yield i, item, enumerate(my_list, skip=i) > There could be variants of this idea like : > - enumerate(my_list, skip=i) > - enumerate(my_list, skip=[i]) > - enumerate(my_list, filter_callback=(lambda x: x != i)) > > Please let me know what you think of it :) > > Thanks for your time, best regards, > Laurent Lyaudet Hello, Regarding the last suggestion. - enumerate(my_list, filter_callback=(lambda x: x != i)) The idea is to filter a list by indices : a quick search for that just yields : https://stackoverflow.com/questions/11847491/python-filtering-lists-by-indi… which is not really helpful Since filter() returns an iterator instead of a list, it could do what is needed... if the callback had access to the index like the Javascript array filter function. > def enumerate_with_rest(my_list): > for i, item in enumerate(my_list): > yield i, item, my_list[:i] + my_list[i + 1:] could also be : def enumerate_with_rest(my_list): for i, item in enumerate(my_list): yield i, item, filter_by_index(my_list, lambda x: x != i) I think my feature request for enumerate_with_rest is maybe too specific. However, I think there should definitely be in itertools something to filter indices like JS permits. I have not found a function to do this there: https://docs.python.org/3/library/functions.html https://docs.python.org/3/library/itertools.html What would be your prefered way of doing this ? enumerate(my_list, filter_callback=(lambda x: x != i)) filter_by_index(my_list, lambda x: x != i) # à la JS filter(my_list, lambda _, x: x != i) Currently, the following solution is available : filter(enumerate(my_list), lambda x: x[0] != i) But it is slightly ugly and unefficient to have two function calls for such a simple task I think. I would appreciate any feedback; even if none of my ideas are accepted, I may learn something :) Best regards, Laurent Lyaudet

2 1

Feature request enumerate_with_rest or enumerate with skip or filter callback
by Laurent Lyaudet Oct. 3, 2021

Oct. 3, 2021

Hello, This is a very simple feature request that does not break anything but I don't know if you may find it interesting. It would be nice to have a function or method of list objects that does this : - First idea : def enumerate_with_rest(my_list): for i, item in enumerate(my_list): yield i, item, my_list[:i] + my_list[i + 1:] It could be called easily with: for i, item, rest in enumerate_with_rest(my_list): # do something or for i, item, rest in my_list.enumerate_with_rest(): # do something I am not the only one who had the same need : https://stackoverflow.com/questions/56966429/getting-pairs-of-one-item-and-… It would be nice to have an optimized C function for this. However, it may be less interesting than this : - Second idea enumerate_with_rest above has quadratic complexity. It is probably true that most processes that use it will also have quadratic complexity. However, it would be better to return an iterator instead of a list for the rest, it would use less space. For this a param skip to enumerate would do the job def enumerate_with_rest(my_list): for i, item in enumerate(my_list): yield i, item, enumerate(my_list, skip=i) There could be variants of this idea like : - enumerate(my_list, skip=i) - enumerate(my_list, skip=[i]) - enumerate(my_list, filter_callback=(lambda x: x != i)) Please let me know what you think of it :) Thanks for your time, best regards, Laurent Lyaudet

3 3

Better exception handling hygiene
by Soni L. Oct. 3, 2021

Oct. 3, 2021

So uh, this is a hardly at all fleshed out idea, but one thing we really don't like about python is having to do stuff like this so as to not swallow exceptions: def a_potentially_recursive_function(some, args): """ Does stuff and things. Raises ExceptionWeCareAbout under so and so conditions. """ try: some.user_code() except ExceptionWeCareAbout as exc: raise RuntimeError from exc code_we_assume_is_safe() if args.something and some_condition: raise ExceptionWeCareAbout It'd be nice if there was a way to... make this easier to deal with. Perhaps something where, something like this: def a_potentially_recursive_function(some, args) with ExceptionWeCareAbout: """ Does stuff and things. Raises ExceptionWeCareAbout under so and so conditions. """ some.user_code() code_we_assume_is_safe() if args.something and some_condition: raise ExceptionWeCareAbout becomes: def a_potentially_recursive_function(some, args): """ Does stuff and things. Raises ExceptionWeCareAbout under so and so conditions. """ try: some.user_code() code_we_assume_is_safe() if args.something and some_condition: let_exception_through = True raise ExceptionWeCareAbout except ExceptionWeCareAbout as exc: if let_exception_through: raise else: raise RuntimeError from exc (or equivalent) and something like: def foo() with Bar: try: baz() except Bar: raise becomes: def foo(): try: try: baz() except Bar: allow_exception = True raise except Bar as exc: if allow_exception: raise else: raise RuntimeError from exc (thus allowing explicit exception propagation) Additionally, note that it would only apply to the raise itself - something like `raise may_raise_the_same_error()` would desugar as if: exception = may_raise_the_same_error() allow_exception = True raise exception Obviously this doesn't solve exception handling, doesn't require the caller to catch the exceptions, etc etc. It does, however, encourage better exception hygiene. Chances are something like this would significantly reduce the amount of swallowed exceptions, if it gets widely adopted. We know something like this would've saved us a lot of trouble, so we're sharing the idea in the hopes it can, in the future, help others.

5 28

Allow regex group name redefinitions
by vencik＠razdva.cz Oct. 2, 2021

Oct. 2, 2021

Hello everybody, I've got a suggestion for the std. re module developers: to consider allowing match group name redefinitions, especially in alternatives. While you may not see the point at first glance, let me try to reason such a thing using a real-world example from my practice: Imagine a company, using certain codes for their products/product components (unfortunately, I'm not at liberty to disclose the true nature of them, but bare with me). Let's say that they may have the following forms: r"(?P<type>AB|C|D)[- ](?P<prefix>[A-Z])?(?P<number>\d+)(?P<postfix>[A-Z])?" So far so good. But now, imagine that a particular type of code has a bit different syntax: r"(?P<type>E)[- ](?P<prefix>[A-Za-z])?(?P<number>\d+)[- ](?P<postfix>[A-Za-z])?" As you can see, the prefix & postfix may be lowercase in case of code type E and moreover, a space or dash is required before the postfix. If I merged the definitions, I'd have to allow that syntax even for the AB, C and D code types---but that would've been incorrect and would require post-matching checks. Ideally I'd like to have the opportunity to define the regex as an alternative: r"(?P<type>AB|C|D)[- ](?P<prefix>[A-Z])?(?P<number>\d+)(?P<postfix>[A-Z])?|(?P<type>E)[- ](?P<prefix>[A-Za-z])?(?P<number>\d+)[- ](?P<postfix>[A-Za-z])?" I can't, of course, getting the re.error: redefinition of group name error upon the regex compilation. But is that really a problem, especially in such alternatives? If you imagine the regex as a FSA, the code type branches into completely independent sub-trees of the automaton state transitions. There's no problem with efficiency; the regex might look a bit complex, but the matching is perfectly efficient---definitely more so than if I match multiple expressions. The redefinition of the match group names is IMO technically perfectly possible and note that in such alternatives, re-assignments won't really happen. And finally, even if they would happen, what's the problem with that? Might be a logical error in the regex definition of course, but that's the programmer's lookout in general... So what do you think? If the match group name redefinition was allowed, I could just match a single regex, getting match group dict and read out parsed parts of the codes by name---nice and easy. Currently, my 2 choices are: 1/ Use uniquely named groups, which requires me to do a post-match group name consolidation of sort or 2/ Match multiple reg. expressions, which is unnecessary Therefore, I ask you to reconsider issuing the error, which I deem redundant and unnecessarily limiting a justified use-case, IMO. Also note that doing that won't break any old code---anything that worked before will continue to work with unchanged semantics; so such a change would be perfectly safe. Thanks, Best Regards vasek

3 2

Reusing more builtins for type-hinting
by Matt del Valle Oct. 2, 2021

Oct. 2, 2021

In the spirit of 3.9 allowing builtin collections like `list`, `dict`, `tuple`, and `set` to be subscripted for type-hinting to remove the need for importing their equivalents from `typing`, I'd like to propose the same thing for a few other builtin objects, namely: `type` (for use instead of`typing.Type`) `any` (for use instead of `typing.Any`) `callable` (for use instead of `typing.Callable`) (I'm aware that a better syntax for type-hinting callables using arrow-syntax is potentially currently in the works, so maybe this last one can be ignored) Having to explicitly import objects from `typing` for basic type-hinting use-cases is a not-inconsiderable source of frustration preventing their uptake, especially by beginners. 3.9 made a valuable step forward in reducing this friction, but I think we can go further.

7 14

pathlib update idea: iterable Path that returns subpaths
by Aaron Stacy Oct. 2, 2021

Oct. 2, 2021

I’d like to propose an enhancement to the wonderful pathlib module: make Path instances iterable, where the iterator returns subpaths. This would be functionally very similar to Path(some_directory).rglob(‘*’). Why not just use rglob(‘*’)? While globs are an extremely useful shorthand, in my experience they’re a bit of an artifact of Unix shells that programmers frequently aren’t aware of or don’t understand. Python does a great job of distilling the concept of what it means to iterate over an object. IO objects come to mind, where [line for line in sys.stdin] very intuitively iterates through each line of input. I believe making Path iterable over its sub paths provides a similarly intuitive concept of iterating over a path. Strawman implementation: class Path: # ... def __iter__(self): return self.rglob('*') Questions: If this were added, where in the Path class hierarchy[1] would it belong? [1]: https://docs.python.org/3/library/pathlib.html#module-pathlib

3 2

define an idiomatic way to make a generator that throws StopIteration immediately and add it to PEP8
by Thomas Grainger Oct. 1, 2021

Oct. 1, 2021

the way to make an (aysnc)generator that immediately raises Stop(Async)Iteration immediately is to insert an unreachable `yield` statement, however there's two ways to do it: def example(): if False: yield or: def example(): return yield currently test_asyncgen.py uses both, eg: @types.coroutine def __anext__(self): if False: yield "this is a generator-based coroutine" if self.yielded >= 2: raise StopAsyncIteration() else: self.yielded += 1 return self.yielded or: async def agenfn(): try: await _async_yield(1) except MyError: await _async_yield(2) return yield the `if False: yield` version has a disadvantage in that it can be anywhere in the function, if this one is chosen then an additional caveat is that it should be the first two statements in a function if `return; yield` version has an disadvantage in that the unreachable yield could be inserted after any `return`, if this one is chosen then an additional caveat is that it should be the last two statements in a function and de-dented as much as syntactically possible. Which alternative do you prefer? I prefer `return; yield`

3 3