Mailman 3 October 2021 - Python-ideas

zip(x, y, z, strict=True)
by Ram Rachum Dec. 1, 2022

Dec. 1, 2022

Here's something that would have saved me some debugging yesterday: >>> zipped = zip(x, y, z, strict=True) I suggest that `strict=True` would ensure that all the iterables have been exhausted, raising an exception otherwise. This is useful in cases where you're assuming that the iterables all have the same lengths. When your assumption is wrong, you currently just get a shorter result, and it could take you a while to figure out why it's happening. What do you think?

27 179

disallow assignment to unknown ssl.SSLContext attributes
by Thomas Grainger June 10, 2022

June 10, 2022

I was debugging some code that was using TLSv1.2 when I expected it to only support TLSv1.3, I tracked it down to a call to: context.miunimum_version = ssl.TLSVersion.TLSv1_3 it should have been: context.minimum_version = ssl.TLSVersion.TLSv1_3 I'd like invalid attribute assignment to be prevented at runtime

13 32

Bringing the print statement back
by Guido van Rossum March 2, 2022

March 2, 2022

In Python 3.10 we will no longer be burdened by the old parser (though 3rd party tooling needs to catch up). One thing that the PEG parser makes possible in about 20 lines of code is something not entirely different from the old print statement. I have a prototype: Python 3.10.0a0 (heads/print-statement-dirty:5ed19fcc1a, Jun 9 2020, 16:31:17) [Clang 11.0.0 (clang-1100.0.33.8)] on darwin Type "help", "copyright", "credits" or "license" for more information. Cannot read termcap database; using … [View More]dumb terminal settings. >>> print 2+2 4 >>> print "hello world" hello world >>> print "hello", input("Name:") Name:Guido hello Guido >>> print 1, 2, 3, sep=", " 1, 2, 3 >>> But wait, there's more! The same syntax will make it possible to call *any* function: >>> len "abc" 3 >>> Or any method: >>> import sys >>> sys.getrefcount "abc" 24 >>> Really, *any* method: >>> class C: ... def foo(self, arg): print arg ... >>> C().foo 2+2 4 >>> There are downsides too, though. For example, you can't call a method without arguments: >>> print <built-in function print> >>> Worse, the first argument cannot start with a parenthesis or bracket: >>> print (1, 2, 3) 1 2 3 >>> C().foo (1, 2, 3) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: C.foo() takes 2 positional arguments but 4 were given >>> print (2+2), 42 4 (None, 42) >>> C().foo [0] Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'method' object is not subscriptable >>> No, it's not April 1st. I am seriously proposing this (but I'll withdraw it if the response is a resounding "boo, hiss"). After all, we currently have a bunch of complexity in the parser just to give a helpful error message to people used to Python 2's print statement: >>> print 1, 2, 3 File "<stdin>", line 1 print 1, 2, 3 ^ SyntaxError: Missing parentheses in call to 'print'. Did you mean print(1, 2, 3)? >>> And IIRC there have been a number of aborted attempts at syntactic hacks to allow people to call functions (like print) without parentheses, although (I think) none of them made it into a PEP. The PEG parser makes this much simpler, because it can simply backtrack -- by placing the grammar rule for this syntax (tentatively called "call statement") last in the list of alternatives for "small statement" we ensure that everything that's a valid expression statement (including print() calls) is still an expression statement with exactly the same meaning, while still allowing parameter-less function calls, without lexical hacks. (There is no code in my prototype that checks for a space after 'print' -- it just checks that there's a name, number or string following a name, which is never legal syntax.) One possible extension I didn't pursue (yet -- dare me!) is to allow parameter-less calls inside other expressions. For example, my prototype does not support things like this: >>> a = (len "abc") File "<stdin>", line 1 a = (len "abc") ^ SyntaxError: invalid syntax >>> I think that strikes a reasonable balance between usability and reduced detection of common errors. I could also dial it back a bit, e.g. maybe it's too much to allow 'C().foo x' and we should only allow dotted names (sufficient to access functions in imported modules and method calls on variables). Or maybe we should only allow simple names (allowing 'len x' but disallowing 'sys.getrefcount x'. Or maybe we should really only bring back print statements. I believe there are some other languages that support a similar grammar (Ruby? R? Raku?) but I haven't investigated. Thoughts? -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-…> [View Less]

42 66

Adding a thin wrapper class around the functions in stdlib.heapq
by bunslow Feb. 18, 2022

Feb. 18, 2022

Nothing so bombastic this time. The heapq functions are basically all named "heapsomething", and basically all take a "heap" for their first argument, with supplementary args coming after. It's a textbook example of the (hypothetical) Object Oriented Manifesto™ where defining a class increases type safety and programmers' conceptual clarity. There're practically no drawbacks, and the code to be added would be very simple. Updating the tests and docs would probably be harder. In pure Python, … [View More]

17 32

Allow syntax "func(arg=x if condition)"
by Peter O'Connor Jan. 19, 2022

Jan. 19, 2022

I often find that python lacks a nice way to say "only pass an argument under this condition". (See previous python-list email in "Idea: Deferred Default Arguments?") Example 1: Defining a list with conditional elements include_bd = True current_way = ['a'] + (['b'] if include_bd else [])+['c']+(['d'] if include_bd else []) new_way = ['a', 'b' if include_bd, 'c', 'd' if include_bd] also_new_way = list('a', 'b' if include_bd, 'c', 'd' if include_bd) Example 2: Deferring to … [View More]

6 9

PEP 671: Syntax for late-bound function argument defaults
by Chris Angelico Nov. 27, 2021

Nov. 27, 2021

Incorporates comments from the thread we just had. Is anyone interested in coauthoring this with me? Anyone who has strong interest in seeing this happen - whether you've been around the Python lists for years, or you're new and interested in getting involved for the first time, or anywhere in between! https://www.python.org/dev/peps/pep-0671/ PEP: 671 Title: Syntax for late-bound function argument defaults Author: Chris Angelico <rosuav(a)gmail.com> Status: Draft Type: Standards Track … [View More]Content-Type: text/x-rst Created: 24-Oct-2021 Python-Version: 3.11 Post-History: 24-Oct-2021 Abstract ======== Function parameters can have default values which are calculated during function definition and saved. This proposal introduces a new form of argument default, defined by an expression to be evaluated at function call time. Motivation ========== Optional function arguments, if omitted, often have some sort of logical default value. When this value depends on other arguments, or needs to be reevaluated each function call, there is currently no clean way to state this in the function header. Currently-legal idioms for this include:: # Very common: Use None and replace it in the function def bisect_right(a, x, lo=0, hi=None, *, key=None): if hi is None: hi = len(a) # Also well known: Use a unique custom sentinel object _USE_GLOBAL_DEFAULT = object() def connect(timeout=_USE_GLOBAL_DEFAULT): if timeout is _USE_GLOBAL_DEFAULT: timeout = default_timeout # Unusual: Accept star-args and then validate def add_item(item, *optional_target): if not optional_target: target = [] else: target = optional_target[0] In each form, ``help(function)`` fails to show the true default value. Each one has additional problems, too; using ``None`` is only valid if None is not itself a plausible function parameter, the custom sentinel requires a global constant; and use of star-args implies that more than one argument could be given. Specification ============= Function default arguments can be defined using the new ``=>`` notation:: def bisect_right(a, x, lo=0, hi=>len(a), *, key=None): def connect(timeout=>default_timeout): def add_item(item, target=>[]): The expression is saved in its source code form for the purpose of inspection, and bytecode to evaluate it is prepended to the function's body. Notably, the expression is evaluated in the function's run-time scope, NOT the scope in which the function was defined (as are early-bound defaults). This allows the expression to refer to other arguments. Self-referential expressions will result in UnboundLocalError:: def spam(eggs=>eggs): # Nope Multiple late-bound arguments are evaluated from left to right, and can refer to previously-calculated values. Order is defined by the function, regardless of the order in which keyword arguments may be passed. Choice of spelling ------------------ Our chief syntax proposal is ``name=>expression`` -- our two syntax proposals ... ahem. Amongst our potential syntaxes are:: def bisect(a, hi=>len(a)): def bisect(a, hi=:len(a)): def bisect(a, hi?=len(a)): def bisect(a, hi!=len(a)): def bisect(a, hi=\len(a)): def bisect(a, hi=`len(a)`): def bisect(a, hi=@len(a)): Since default arguments behave largely the same whether they're early or late bound, the preferred syntax is very similar to the existing early-bind syntax. The alternatives offer little advantage over the preferred one. How to Teach This ================= Early-bound default arguments should always be taught first, as they are the simpler and more efficient way to evaluate arguments. Building on them, late bound arguments are broadly equivalent to code at the top of the function:: def add_item(item, target=>[]): # Equivalent pseudocode: def add_item(item, target=<OPTIONAL>): if target was omitted: target = [] Open Issues =========== - yield/await? Will they cause problems? Might end up being a non-issue. - annotations? They go before the default, so is there any way an anno could want to end with ``=>``? References ========== Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. [View Less]

31 292

Unpacking in tuple/list/set/dict comprehensions
by Erik Demaine Nov. 14, 2021

Nov. 14, 2021

Extended unpacking notation (* and **) from PEP 448 gives us great ways to concatenate a few iterables or dicts: ``` (*it1, *it2, *it3) # tuple with the concatenation of three iterables [*it1, *it2, *it3] # list with the concatenation of three iterables {*it1, *it2, *it3} # set with the union of three iterables {**dict1, **dict2, **dict3} # dict with the combination of three dicts # roughly equivalent to dict1 | dict2 | dict3 thanks to PEP 584 ``` I propose (not for the first time) that … [View More]similarly concatenating an unknown number of iterables or dicts should be possible via comprehensions: ``` (*it for it in its) # tuple with the concatenation of iterables in 'its' [*it for it in its] # list with the concatenation of iterables in 'its' {*it for it in its} # set with the union of iterables in 'its' {**d for d in dicts} # dict with the combination of dicts in 'dicts' ``` The above is my attempt to argue that the proposed notation is natural: `[*it for it in its]` is exactly analogous to `[*its[0], *its[1], ..., *its[len(its)-1]]`. There are other ways to do this, of course: ``` [x for it in its for x in it] itertools.chain(*its) sum(it for it in its, []) functools.reduce(operator.concat, its, []) ``` But none are as concise and (to me, and hopefully others who understand * notation) as intuitive. For example, I recently wanted to write a recursion like so, which accumulated a set of results from within a tree structure: ``` def recurse(node): # base case omitted return {*recurse(child) for child in node.children} ``` In fact, I am teaching a class and just asked a question on a written exam for which several students wrote this exact code in their solution (which inspired writing this message). So I do think it's quite intuitive, even to those relatively new to Python. Now, on to previous proposals. I found this thread from 2016 (!); please let me know if there are others. https://mail.python.org/archives/list/python-ideas@python.org/thread/SBM3LY… There are several arguments for and against this feature in that thread. I'll try to summarize: Arguments for: * Natural extension to PEP 448 (it's mentioned as a variant within PEP 448) * Easy to implement: all that's needed in CPython is to *remove* some code blocking this. Arguments against: * Counterintuitive (to some) * Hard to teach * `[...x... for x in y]` is no longer morally equivalent to `answer = []; for x in y: answer.append(...x...)` (unless `list1.append(a, b)` were equivalent to `list1.extend([a, b])`) Above I've tried to counter the first two "against" arguments. Some counters to the third "against" argument: 1. `[*...x... for x in y]` is equivalent to `answer = []; for x in y: answer.extend(...x...)` (about as easy to teach, I'd say) 2. Maybe `list1.append(a, b)` should be equivalent to `list1.extend([a, b])`? It is in JavaScript (`Array.push`). And I don't see why one would expect it to append a tuple `(a, b)`; that's what `list1.append((a, b))` is for. I think the main argument against this is to avoid programming errors, which is fine, but I don't see why it should hold back an advanced feature involving both unpacking and comprehensions. Erik -- Erik Demaine | edemaine(a)mit.edu | http://erikdemaine.org/ [View Less]

15 30

PEP 671 proof-of-concept: A less invasive implementation
by Jonathan Fine Nov. 1, 2021

Nov. 1, 2021

Hi One of the motives for PEP 671 is that the signature of a function fn, and hence the associated help(fn) is sometimes opaque regarding default values. I won't repeat the excellent examples already given. In the current implementation default values are handled outside the compiled code of the function, which is available at fn.__code__. Instead they are stored in 'metadata' associated with the function. Here's one way to see this. from inspect import signature as sig def fn(a, b=… [View More]

3 7

PEP 671 late-bound defaults implementation
by Steven D'Aprano Oct. 31, 2021

Oct. 31, 2021

I have a suggestion for the implementation. I think that Chris' current approach is to compile the late-bound defaults in the function body. So if we have a function like this: def func(a, b=early_expression, @c=late_expression): block the function code looks like this (after compilation): # Pseudocode if c is unbound: c = late_expression block (except in bytecode of course). Chris, do I have that right? If it is wrong, probably everything I say next is … [View More]irrelevant. There is a strange asymmetry to the way the default for b and c are handled. For b, it is the interpreter's responsibilty to load the default value (pre-evaluated and cached) and bind it to the parameter. But for c, it is the function object's responsibility. I'd like to suggest a different approach which I expect will be more flexible, I hope won't cost too much in performance, and in my opinion much more closely matches the semantics of the feature. I think it should remain the interpreter's responsibility to set up all the parameters before entering the function, including late-bound defaults. That will have the big advantage that disassembling func will only show the code for "block", not the associated code that tests and binds late-bound defaults. (Just like currently, it doesn't show the code for binding early-bound defaults.) This suggests that each late-bound expression should be compiled into a separate code object, all of which are then squirrelled away in the function object (just as the __code__ and __defaults__ currently are). I imagine the process will be something like: * set up a new local namespace for the function call * bind arguments to parameters in that namespace * bind early-bound defaults from the function __defaults__ to parameters * (NEW) evaluate the appropriate late-bound expression code objects, running them in the local namespace, and binding their results to the parameters; * enter the function's code block. Benefits: - the function code block is smaller, since it no longer has to include the "test, evaluate, bind" for every late-bound parameter; - this may improve code locality, which is good for performance (or so I am told); - introspection tools such as dis can disassemble the body of the function independently of the late-bound parameters; - which means we can inspect the late-bound parameters independently by passing their code object to dis; - for testing, we can evaluate the expression code objects using eval (maybe?); - we may be able to include the source code to the expression in the expression's own code block, e.g. in the co_name field(?); - we may be able to replace/modify the defaults' code blocks independently of the main function __code__, e.g. for byte-code hacking, or other function object hacking. Costs: - the function object itself may be a little larger. Thoughts? -- Steve [View Less]

2 2

Context managers in expressions
by jcg.sturdy＠gmail.com Oct. 30, 2021

Oct. 30, 2021

I've been thinking for a while that it would be good to be able to use context managers in an expression context, and I haven't been able to come up with any arguments against it (although it may not be to everyone's taste) and I can't see any sign that it's come up before (although the words of it don't make very specific search terms). My suggestion is that a "with" clause could be written after an expression (consistent with the conditional operator being written with "if" after the first … [View More]

4 3

PEP 671 proof-of-concept implementation
by Chris Angelico Oct. 30, 2021

Oct. 30, 2021

https://github.com/Rosuav/cpython/tree/pep-671 So uhh... anyone who knows about the internals of CPython and wants to join me on this, I would *really* appreciate coauthors! The implementation ended up a lot more invasive than I originally planned. Some of that is inherent to the problem, but other parts might be able to be done more cleanly. The way I've done it: * Argument defaults (either in __defaults__ or __kwdefaults__) are now tuples of (desc, value) or (desc,) for early-bound and … [View More]

6 11

`is in`/`not is in` operators
by Jeremiah Vivian Oct. 29, 2021

Oct. 29, 2021

I DO expect this thread to be bombarded with negative replies. Currently, there are `in`/`not in` operators which work like this in Python: > def contains(contains_value, iterable, not_in): > for element in iterable: > if element == contains_value: > return True ^ not_in > return False ^ not_in If I wanted to check if an *exact* object is in an iterable, I would have to loop like this (reverse boolean values for implementation of `not is in`): > … [View More]

5 20

Python Shared Objects
by byko3y＠gmail.com Oct. 28, 2021

Oct. 28, 2021

Due to https://www.python.org/dev/peps/pep-0554/ multi-interpreters implementation going really slow, I had the audicity to try an alternative route towards the same objective of implementing multicore support of python: instead of sharing the memory by running multiple threads, I employed an interprocess shared memory with multiple processes. I know there are multiprocessing.sharedctypes and multiprocessing.shared_memory, but I went much deeper into it by solving two problems they failed to … [View More]

2 2

Syntax for late-bound arguments
by Chris Angelico Oct. 28, 2021

Oct. 28, 2021

Spun off from PEP 505 thread on python-dev. One of the weaker use-cases for None-coalescing is function default arguments, where the technical default is None but the function wants to use some other value instead. This is a weak argument in favour of None-coalescing because only a small set of such situations are only slightly improved by a "??" operator, and the problem is far larger. Consider the bisect.bisect() function [1]: def bisect(a, x, lo=0, hi=None): if lo < 0: raise … [View More]ValueError('lo must be non-negative') if hi is None: hi = len(a) It's clear what value lo gets if you omit it. It's less clear what hi gets. And the situation only gets uglier if None is a valid argument, and a unique sentinel is needed; this standard idiom makes help() rather unhelpful: _missing = object() def spaminate(thing, count=_missing): if count is _missing: count = thing.getdefault() Proposal: Proper syntax and support for late-bound argument defaults. def spaminate(thing, count=:thing.getdefault()): ... def bisect(a, x, lo=0, hi=:len(a)): if lo < 0: raise ValueError('lo must be non-negative') This would be approximately equivalent to the _missing idiom, with the following exceptions: 1) Inspecting the function would reveal the source code for the late-bound value 2) There is no value which can be passed as an argument to achieve the same effect 3) All late-bound defaults would be evaluated before any other part of the function executes (ie the "if hi is None" check would now be done prior to "if lo < 0"). The syntax I've chosen is deliberately subtle, since - in many many cases - it won't make any difference whether the argument is early or late bound, so they should look similar. While it is visually similar to the inline assignment operator, neither form is currently valid in a function's parameter list, and thus should not introduce ambiguity. The expression would be evaluated in the function's context, having available to it everything that the function has. Notably, this is NOT the same as the context of the function definition, but this is only rarely going to be significant (eg class methods where a bare name in an early-bound argument default would come from class scope, but the same bare name would come from local scope if late-bound). The purpose of this change is to have the function header define, as fully as possible, the function's arguments. Burying part of that definition inside the function is arbitrary and unnecessary. ChrisA [1] Technically it's bisect_right but I'm using the simpler name here. Also, it recently grew a key parameter, which would only get in the way here. https://docs.python.org/3/library/bisect.html#bisect.bisect_right [View Less]

15 53

Parameter tuple unpacking in the age of positional-only arguments
by Erik Demaine Oct. 28, 2021

Oct. 28, 2021

On Tue, 26 Oct 2021, Eric V. Smith wrote: > You may or may not recall that a big reason for the removal of "tuple > parameter unpacking" in PEP 3113 was that they couldn't be supported by the > inspect module. Quoting that PEP: "Python has very powerful introspection > capabilities. These extend to function signatures. There are no hidden > details as to what a function's call signature is." > > (Aside: I loved tuple parameter unpacking, and used it all the time! I was … [View More]

3 2

Add __len__ to ipaddress._BaseNetwork
by nyuszika7h＠gmail.com Oct. 26, 2021

Oct. 26, 2021

Currently, ipaddress._BaseNetwork (and by extension, ipaddress.IPv4Network and ipaddress.IPv6Network) does not have a __len__ method, it only has num_addresses. This makes the following code not work: >>> random.choice(ipaddress.ip_network('6.0.0.0/8')) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/nyuszika7h/.pyenv/versions/3.10.0/lib/python3.10/random.py", line 378, in choice return seq[self._randbelow(len(seq))] TypeError: … [View More]

2 1

Fwd: Simple curl/wget-like download functionality in urllib (like http offers server)
by Tom P Oct. 26, 2021

Oct. 26, 2021

In the context of building Docker images, it is often required to download stuff. If curl/wget is available, great, but often slim images don't include that. The urllib could provide a very simple download functionality (like http offers a simple server): from urllib.request import urlopen data = urlopen('https://.../install-poetry.py').read() # print or save data

10 16

Re: PEP 671: Syntax for late-bound function argument defaults
by Jon Kiparsky Oct. 26, 2021

Oct. 26, 2021

While I love this idea, and I think this PEP is bound for glory, I don't love any of the proposed spellings for denoting late-binding arguments. It seems to me that a specialized symbol combination indicating that a particular argument has special behavior does not serve readability, regardless of what symbol combination is chosen. I would prefer to build on the fact that arguments already come in two flavors with somewhat different behaviors, and that the ordering of these is determined. … [View More]

3 2

Return False from __contains__ method if object not hashable for set and dict.
by Kazantcev Andrey Oct. 26, 2021

Oct. 26, 2021

Now if do something like `[] in set()` python raise an exception, but if an object isn't hashable then we know for sure that it isn't in the set. Propose return False for these cases. What do you think?

3 2

def variable = value
by blek blek Oct. 25, 2021

Oct. 25, 2021

Hello people, I was thinking of a "def" statement in Python. Its really weird that it can define only method, since it means "define". It shouldn't be hard to detect if this is a variable or a method, because method needs "()" to have arguments(even if there are zero of them), and defining variable value supposed to have "=" as the definition mark. Hope it would be added someday

5 6

os.workdir() context manager
by Marc-Andre Lemburg Oct. 22, 2021

Oct. 22, 2021

Hello all, I sometimes write Python scripts which need to work in specific work directories. When putting such code into functions, the outer function typically does not expect the current work dir (CWD) to be changed, so wrap the code which need the (possibly) modified CWD using a simply context manager along the lines of: class workdir: def __init__(self, dir): self.dir = dir def __enter__(self): self.curdir = os.getcwd() os.chdir(self.dir) def __exit__(… [View More]

17 37

Fwd: [Python-Dev] Re: Semi-proposal: Tagged None
by David Mertz, Ph.D. Oct. 21, 2021

Oct. 21, 2021

I actually thought this thread was on pyhon-ideas where it is now appropriate. Chris quotes everything I wrote, and his comments are all useful, so let me move starting with that. ---------- Forwarded message --------- From: Chris Angelico <rosuav(a)gmail.com> Date: Thu, Oct 21, 2021, 8:42 PM Subject: [Python-Dev] Re: Semi-proposal: Tagged None To: Python-Dev <python-dev(a)python.org> On Fri, Oct 22, 2021 at 3:23 AM David Mertz, Ph.D. <david.mertz(a)gmail.com> wrote: > &… [View More]gt; On Thu, Oct 21, 2021 at 2:52 AM Steven D'Aprano <steve(a)pearwood.info> wrote: >> >> On Tue, Oct 19, 2021 at 05:09:42PM -0700, Michael Selik wrote: >> > None and its ilk often conflate too many qualities. For example, is it >> > missing because it doesn't exist, it never existed, or because we never >> > received a value, despite knowing it must exist? > > >> >> 30+ years later, and we cannot easily, reliably or portably use NAN >> payloads. Most people don't care. If we offerred them a dozen or a >> thousand distinct sentinels for all the various kinds of missing data, >> how many people would use them and how many would just stick to plain >> old None? > > > In data science, I have been frustrated by the sparsity of ways of spelling "missing value." Might be worth redirecting this to -ideas. > Besides the distinction Michael points out, and that Steven did in relation to NaNs with payloads, I encounter missingness of various other sorts as well. Crucially, an important kind of missing data is data where the value I received seems unreliable and I have decided to *impute* missingness rather than accept a value I believe is unreliable. > > But there is also something akin to what Michael points out (maybe it's just an example). For example, "middle name" is something that some people simply do not have, other people choose not to provide on a survey, and others still we just don't know anything beyond "it's not there." > And some people have more than one (I have a brother with two of them). Not the best example to use, since names have WAY more complexities than different types of absence, but there are other cases where that sort of thing comes up. For instance, if someone says on a survey that s/he is in Australia, and then you ask for a postcode, then leaving it blank should be recorded as "chose not to provide"; but if the country is listed as Timor-Leste / East Timor, then "not applicable" would be appropriate, since the country doesn't use postal codes. > Of course, when I impute missingness, I can do so at various stages of data cleaning, and for various different reasons or confidences. None (or NaN) are sort of OK, but carrying metadata as to the nature of missingness would be nice. > Right. Using postcodes as an example again, for someone in Australia, a postcode of "E3B 0H8" doesn't make sense, as that isn't the format we use. So you could wipe that out and replace it with "No postal code, malformed data entered". > So my strawman suggestion is tagging None's. I suppose spellings like `None[reason]` or `None(reason)` are appealing. > > An obvious problem that I recognize is that it's not obvious this can "play nice" with the common idiom `if mydata is not None: ...`. None really is a singleton, and a "tagged singleton" or "annotated singleton" probably doesn't work well with Python's object model. > > My goal, of course, would be to have TaggedNone be a kind of subclass of None, in the same way that bool is a subclass of int, and hence True is a kind of 1. However, I'd want a large number of custom None's, with some sort of accessible string or numeric code or something to inspect which one it was. > But this is where I start to disagree. None should remain a singleton, but "no data available" could be its own thing, tied in with the way that you do your data storage and stats. As such, you wouldn't be checking it with 'is', so you wouldn't have that problem (the Python 'is' operator will only ever test for actual object identity). Keep None simple and dependable, and then "Missing Data" can be an entire class of values if you so desire. ChrisA _______________________________________________ Python-Dev mailing list -- python-dev(a)python.org To unsubscribe send an email to python-dev-leave(a)python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/6NY5NQC… Code of Conduct: http://python.org/psf/codeofconduct/ [View Less]

3 4

Making imports callable
by Abdur-Rahmaan Janhangeer Oct. 21, 2021

Oct. 21, 2021

Greetings list, Let's say i import module1 import module1 can we do module1() straight out of the box? If not i propose something like if __name__ == '__main__': __file__.callable = True Kind Regards, Abdur-Rahmaan Janhangeer about <https://compileralchemy.github.io/> | blog <https://www.pythonkitchen.com> github <https://github.com/Abdur-RahmaanJ> Mauritius

2 3

PEP able? : Request new 'fn' format specifier to specify relationship between positional arguments
by Shash Srinivasa Oct. 21, 2021

Oct. 21, 2021

Hi, Draft PEP attached: Proposal: def is_between(n1,n2,n3): fn"is number {n1} between {n2} and {n3}" fn"{n2}<{n1}<{n3}?" # the token "fn" immediately following the def statement, speficy the descriptive text # multipe "fn" specifiers describes multiple ways the funtion can be invoked # the "fn" statement is at the same indentation level as the def statement return (n1 > n2 and n1 < n3) . . . # if "fn" statements are found indented and not immediately following def statement, … [View More]

1 0

Implementing string unary operators
by Jeremiah Vivian Oct. 21, 2021

Oct. 21, 2021

I posted a previous thread about overloading the unary `+` operator in strings with `ord`, and that expanded to more than just the unary `+` operator. So I'm saying now, there should be these implementations: > +string - `int(string, 10)` (or just `int(string)`) > -string - `int(string, 8)` > ~string - `int(string, 16)` Or: > +string - `string.lstrip()` > -string - `string.rstrip()` > ~string - `string.strip()` If anyone has better ideas, they can post it here.

15 41

Re: Type-hinting dictionaries for an arbitrary number of arbitrary key/value pairs? Counterpart to PEP 589?
by Alex Waygood Oct. 19, 2021

Oct. 19, 2021

Indeed — we essentially lie to mypy about the method resolution order for list, dict, etc (mypy thinks that list directly inherits from collections.abc.MutableSequence — see the typeshed stub here: https://github.com/python/typeshed/blob/32bc2161a107db20c2ebc85aad31c29730d…), but numeric ABCs are not special-cased by typeshed in the same way as collections ABCs. Fundamentally, mypy has no knowledge about virtual subclassing. Again, I covered this very comprehensively on StackOverflow ;) https:/… [View More]

4 4

Real Positional Arguments or OO Currying
by Mathew Elman Oct. 19, 2021

Oct. 19, 2021

I don't know if this has been suggested before, or if this is outlandishly impossible (though I would be surprised if it was), so apologies in advance if so. I have on occasion come across a situation where I use/write a signature like this: def insert_x_into_y(x, y): ... or worse def insert_into(item, container): ... where, despite a driving idea of python syntax being readability in english, the function signature is distinctly not english. "I'll just go and … [View More]

9 27

Accessing target name at runtime
by Jeremiah Paige Oct. 19, 2021

Oct. 19, 2021

Hello Python community, Earlier this year I put forward a proposal for decorators on variables which I felt was well received in concept, but fell flat on the syntax. One of the responses to that thread was rather than abusing the decorator, to use a new symbol to get the target of an assignment statement as a string on the right hand side. I would call this compile-time reflection of assignment targets, or simply reflection. Although initially continuing on with my original syntax, the idea … [View More]

7 17

Except * formatting
by Rob Cliffe Oct. 18, 2021

Oct. 18, 2021

I'm sending this to python-list because my emails to python-dev keep getting bounced back (after a few days delay). I've no idea why. Instead of `except group ...`, what about `except for ...`? No new keywords. Reads naturally in English. Hints that there is more than one kind of exception coming. Best wishes Rob Cliffe

2 1

Re: Structure Pattern for annotation
by Alex Waygood Oct. 17, 2021

Oct. 17, 2021

I agree with Steven. I very much like Abdulla's proposed syntax for dicts, TypedDicts and sets. But I'm not sure that the idea for `Annotated` is workable, and the proposal for lists seems too prone to ambiguity, given how extensively square brackets are already used in typing syntax. One question about the proposals, however: how would we represent other mappings apart from dicts? Would `collections.abc.Mapping` and `types.MappingProxyType` still be spelled as `Mapping[str, int]` and `… [View More]MappingProxyType[str, int]`? If so, that would be a slightly annoying inconsistency. I do also quite like the idea of an improved syntax for tuples specifically. Tuples are already very different types to lists/strings/etc in the context of typing, essentially representing heterogeneous structs of fixed length rather than homogenous sequences of unspecified length. As such, I think it "makes sense" to special-case tuples without special-casing other sequences such as list, `collections.abc.Sequence` or `collections.deque`. def foo() -> tuple[list[list[str]], list[list[str]]] would become def foo() -> (list[list[str]], list[list[str]]) That feels quite readable and natural, to me. Best, Alex > On 14 Oct 2021, at 09:25, Steven D'Aprano <steve(a)pearwood.info> wrote: > On Thu, Oct 14, 2021 at 12:32:57AM +0400, Abdulla Al Kathiri wrote: > >> Today I found myself write a function that returns a tuple of list of >> list of strings (tuple[list[list[str]], list[list[str]]]). Wouldn’t it >> easier to read to write it like the following: >> ([[str]], [[str]])? > > Not really. Your first example is explicit and I can get the meaning by > just reading it out loud: > > tuple[list[list[str]], list[list[str]]] > > "tuple (of) list (of) list (of) str, list (of) list (of) str > > Your abbreviated version: > > ([[str]], [[str]]) > > is too terse. I have to stop and think about what it means, not just > read it out loud. Without the hint of named types (tuple and list), my > first reaction to seeing [str] is "is this an optional string?". > > And then I wonder why it's not written: > > ([[""]], [[""]]) > > Why abbreviate list and tuple but not string? > > Code is read more than it is written, and can be too terse as well as > too verbose. > > On the other hand: > >> Similarly for TypedDict, replace the following.. >> class Movie(TypedDict): >> name: str >> year: int >> with >> {‘name’: str, ‘year’: int} > > To my eye, that one does work. As far as I know, curly brackets {} > aren't otherwise used in annotations (unlike square brackets), and they > don't look like "optional" to me. They look like a dict. > > So on first glance at least, I think that: > > {'name': str, 'year': int} > > is better than the class syntax we already have. > > > Likewise: > >> dict[str, int] will be {str: int} >> set[int] will be {int}. > > work for me too. > > >> Also, Annotated[float, “seconds”] can be replaced with something like >> float #seconds indicating whatever comes after the hashtag is just >> extra information similar to a normal comment in the code. > > No, because the # indicates that the rest of the line is a comment. This > is already legal: > > def func(d: {str # this is an actual comment > : int}) -> Any: ... > > so this would be ambiguous between a real comment and an annotation. > > Even if we agreed to change the behaviour of comments, you suggested: > > func(d: {str # product label: [float] # prices from 2000 to 2015}) > > How is the interpreter to know that the first annotation is just > > "product label" > > rather than this? > > "product label: [float] # prices from 2000 to 2015" > > So I don't think this works. > > > -- > Steve > _______________________________________________ > Python-ideas mailing list -- python-ideas(a)python.org > To unsubscribe send an email to python-ideas-leave(a)python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/T7E54… > Code of Conduct: http://python.org/psf/codeofconduct/ [View Less]

7 8

Type-hinting dictionaries for an arbitrary number of arbitrary key/value pairs? Counterpart to PEP 589?
by Sebastian M. Ernst Oct. 16, 2021

Oct. 16, 2021

Hi all, disclaimer: I have no idea on potential syntax or if it is applicable to a wide audience or if there is already a good solution to this. It is more like a "gap" in the type hint spec that I ran across in a project. In function/method signatures, I can hint at dictionaries for example as follows: ```python from numbers import Number from typing import Dict from typeguard import typechecked Data = Dict[str, Number] @typechecked def foo(bar: Data): print(bar) ``` Yes, this is … [View More]

6 8

Antichecked exceptions
by Soni L. Oct. 16, 2021

Oct. 16, 2021

PEP 479 introduced this idea that basically boils down to raising your exceptions only when you mean it. Specifically: generators only raise StopIteration when they return/exit/terminate. And they prevent the user from raising it under other circumstances. Can we call this "antichecked exceptions"? Generators specifically check for StopIteration raised in undesired circumstances, and make sure they get wrapped. This is a form of runtime checking, so the "checked" part of the name makes sense. … [View More]

2 1

Structure Pattern for annotation
by Abdulla Al Kathiri Oct. 15, 2021

Oct. 15, 2021

Hello, Today I found myself write a function that returns a tuple of list of list of strings (tuple[list[list[str]], list[list[str]]]). Wouldn’t it easier to read to write it like the following: ([[str]], [[str]])? Similarly for TypedDict, replace the following.. class Movie(TypedDict): name: str year: int with {‘name’: str, ‘year’: int} dict[str, int] will be {str: int} set[int] will be {int}. This is similar to the new proposed Callable alternative (i.e., (int, float) -> … [View More]

3 3

Re: dict_items.__getitem__?
by Alex Waygood Oct. 14, 2021

Oct. 14, 2021

I think there definitely should be a more obvious way to do this (specifically the first and last keys/values/items of a dictionary — I'm ambivalent about the others, since they won't always be fast, as discussed). An anti-pattern you see quite often on Stack Overflow to get the first key of a dictionary is something like the following: first_key = list(mydict.keys())[0] Whereas obviously, a much better way (especially if it's a very large dictionary) is to do: first_key = … [View More]next(iter(mydict)) [Christopher Barker] > I'll leave it exercise for the reader to find that thead For reference, the (very long) previous thread is here: https://mail.python.org/archives/list/python-ideas@python.org/thread/S7UMTW…. [Inada Naoki] > I think we can add `itertools.first()` for this idiom, and > `itertools.last()` for `next(iter(reversed(x)))` idiom. I like this idea, a lot. Another possibility I've been wondering about was whether several methods should be added to the dict interface: dict.first_key = lambda self: next(iter(self)) dict.first_val = lambda self: next(iter(self.values())) dict.first_item = lambda self: next(iter(self.items())) dict.last_key = lambda self: next(reversed(self)) dict.last_val = lambda self: next(reversed(self.values())) dict.last_item = lambda self: next(reversed(self.items())) But I think I like a lot more the idea of adding general ways of doing these things to itertools. Best, Alex > On 5 Oct 2021, at 05:30, Christopher Barker <pythonchb(a)gmail.com> wrote: > > On Mon, Oct 4, 2021 at 5:46 PM Erik Demaine <edemaine(a)mit.edu> wrote: >> Have folks thought about allowing indexing dictionary views as in the >> following code, where d is a dict object? >> >> d.keys()[0] >> d.keys()[-1] >> d.values()[0] >> d.values()[-1] >> d.items()[0] >> d.items()[-1] # item that would be returned by d.popitem() > > since dicts were made order-preserving, indexing the keys, items, etc does make some sense. > > > > I've also often wanted to get an arbitrary item/key from a dictionary, and > > This is indeed one of the use cases identified. > >> I found some related discussion in >> https://mail.python.org/archives/list/python-ideas@python.org/thread/QVTGZD… >> but not this exact idea. > > That's a pretty different idea but this exact idea has been discussed on this list relatively recently. I still like it, but there wan't much general support. > > I'll leave it exercise for the read to find that thead, but it is there, and I suggest you look for it if you want to further pursue this idea. > > -CHB > > > -- > Christopher Barker, PhD (Chris) > > Python Language Consulting > - Teaching > - Scientific Software Development > - Desktop GUI and Web Development > - wxPython, numpy, scipy, Cython > _______________________________________________ > Python-ideas mailing list -- python-ideas(a)python.org > To unsubscribe send an email to python-ideas-leave(a)python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/RAEDZ… > Code of Conduct: http://python.org/psf/codeofconduct/ [View Less]

18 63

Overloading unary plus in strings with "ord"
by Jeremiah Vivian Oct. 13, 2021

Oct. 13, 2021

So `ord` is already a really fast function with (last check before this thread was posted) 166 nsec per loop. But I'm wondering... doing `ord(a)` produces this bytecode: > 2 4 LOAD_NAME 1 (ord) > 6 LOAD_NAME 0 (a) > 8 CALL_FUNCTION 1 > 10 POP_TOP > 12 LOAD_CONST 1 (None) > 14 RETURN_VALUE But doing `+a` only produces this: > 2 4 … [View More]

5 18

Adding a Version type
by Finn Mason Oct. 11, 2021

Oct. 11, 2021

Hello all, I was thinking about the following idioms: __version__ = "1.0" import sys if sys.version_info < (3, 6): # Yell at the user I feel like there could be a better way to define versions. There's no real standard way to specify versions in Python, other than "use semantic versioning." This can cause problems such as: * Uncertainty of the type of the version (commence the pattern matching! Or `if... elif` statements if that's your thing) * No real way to specify an alpha, … [View More]

6 8

socketserver StreamRequestHandler non-blocking read__
by Evan Greenup Oct. 11, 2021

Oct. 11, 2021

For StreamRequetHandler's rfile attribute, it will block when the read readline readlines reach the end. It would be nice to add non-blocking read because the seek() is unavailable for rfile. and there is no other way to detect if it is ended. When the stream is empty and waiting for new input, then it will return None type. This would be a nice feature.

1 0

There must be a better way to do class-attributes -- allow descriptor.__set__ on `type`, not just `object`
by Randolf Scholz Oct. 10, 2021

Oct. 10, 2021

Over the past few weeks/months I have had on an off issues with the way python handles class-attributes. Consider the default way to do it: ``` class A: attr = 2 """Docstring of the attribute""" ``` There are 2 main issues I have with this way of setting class-attributes: 1. It does not allow to set class-attributes that depend on other class-attributes. (Example: setting a path that depends on cls.__name__) 2. It does not allow to set lazily evaluated attributes. If setting the … [View More]class-attribute involves an expensive computation, it will be done at class definition time. One possible resolution is to use metaclasses, however this comes with its own issues: 1. MetaClasses do not work nice with documentation. Attributes defined my metaclasses will neither show up in help(), dir(), nor will they be in the correct place when documenting the class with tools like sphinx. 2. Who wants to write an additional metaclass whenever they want to write a class and have some minimal class-attribute processing? A new possible resolution, possible since python 3.9, is the combination of @classmethod and @property. However, this has it's own problems and pitfalls, cf. https://bugs.python.org/issue44904, https://bugs.python.org/issue45356. In fact, after lots of trial and error and reading into descriptors I figured that "true" class-properties are not possible in the current versions of python for a very simple reason: ╔══════════════════════════════════════════════════════╗ ║descriptor's __set__ method is never called when setting a class-attribute. (see MWE) ║ ╚══════════════════════════════════════════════════════╝ Therefore, I propose the following: 1. By default `type.__setattr__ ` should check whether the attribute implements `__set__` and if so, use that instead. (it already works like this for object.__setattr__), (cf. PatchedTrig below) 2. There should be a clearer distinction between class-attributes and instances-attributes. Instances should probably by default not be allowed to change class-attributes, since this can affect global state. 3. It would be very nice to have a built-in @attribute decorator, much like @property, but specifically for stetting class attributes. (again, metaclasses are possible but the issues with documentation compatibility seem very bad) (pretty much what @classmethod@property promises but doesn't fulfill.) ## MWE ```python import math class TrigConst: def __init__(self, const=math.pi): self.const = const def __get__(self, obj, objtype=None): return self.const def __set__(self, obj, value): print(f"__set__ called", f"{obj=}") self.const = value class Trig: const = TrigConst() ``` Then ```python Trig().const # calls TrigConst.__get__ Trig().const = math.tau # calls TrigConst.__set__ Trig.const # calls TrigConst.__get__ Trig.const = math.pi # overwrites TrigConst attribute with float. ``` Patched version ```python class PatchedSetattr(type): def __setattr__(cls, key, value): if hasattr(cls, key): obj = cls.__dict__[key] if hasattr(obj, "__set__"): obj.__set__(cls, value) else: super().__setattr__(key, value) class PatchedTrig(metaclass=PatchedSetattr): const = TrigConst() ``` Then ```python cls = Trig print(cls.const, type(cls.__dict__["const"])) cls.const = math.tau print(cls.const, type(cls.__dict__["const"])) ``` gives ``` 3.141592653589793 <class '__main__.TrigConst'> 6.283185307179586 <class 'float'> ``` But ```cls cls = PatchedTrig print(cls.const, type(cls.__dict__["const"])) cls.const = math.tau print(cls.const, type(cls.__dict__["const"])) ``` gives ``` 3.141592653589793 <class '__main__.TrigConst'> __set__ called obj=<class '__main__.PatchedTrig'> 6.283185307179586 <class '__main__.TrigConst'> ``` [View Less]

1 0

Invalidate list iterators after size changes?
by Tyler Hou Oct. 10, 2021

Oct. 10, 2021

Right now, the following code is valid in Python 3.9 (and infinitely loops): ``` lst = [1, 2, 3] for i in lst: lst.append(i) ``` IMO this is a source of bugs, and even though this behavior is technically documented (https://bugs.python.org/issue32767), I don't know if it should belong in the core language. For example, you cannot change the size of a set or dict during iteration; you will get a RuntimeError. Allegedly this is due to the underlying implementation of the dict/set iterators (… [View More]https://bugs.python.org/issue22084) but I think it would be beneficial this was made part of the public API for list, set, and dict iterators. Concretely, I'm proposing the following set of changes to Python: 1. If the size of a list, set, or dict changes, invalidate all existing iterators to that containers. Document this behavior as part of the public API. 2. Attempting to call next() on an invalidated iterator raises a RuntimeError. 3. Add a key_iter function to itertools (name up for debate), which replicates the existing functionality for lists. It could also make a copy of the keys of a set/dict, so you could use it to mutate those as well during iteration (to allow users to write functions that mutate an arbitrary duck-typed collection) 4. Gate (1, 2) for lists behind a __future__ flag with a long migration window (e.g. ~3.16). Benefits: 1. Reduce a source of bugs (deleting while iterating forward through a list often means that your logic skips elements; appending can lead to infinite loops or unexpected behavior). 2. Make the iterator invalidation consistent between lists, sets, and dicts. 3. More consistent behavior with collections in other languages (C++, Java). Drawbacks: 1. Migration costs. But mutating during iteration should be relatively rare, given that most people discourage mutating while iterating (https://google.github.io/styleguide/pyguide.html#284-decision, https://stackoverflow.com/a/1637875/1625484) __future__ feature flag plus deprecation warnings should go along way to help people migrate their code. We would also offer an easy in-place fix (key_iter). 2. You can't write algorithms that append to a list (e.g. BFS) with the normal list iterator. Again, you could use key_iter (or use a deque with block size == 1; see Open Questions). Open Questions Is (3) necessary? It's not hard to write an implementation yourself. What should be done about deque for block sizes > 1? If this change gets made, should we add way to create a deque with block size == 1 whose iterators are never invalidated? [View Less]

6 5

String method to check for a float
by dpalit17＠outlook.com Oct. 7, 2021

Oct. 7, 2021

It would be nice to have a string method that checks for a float. Currently there is no support for this, either built-in or in the standard library. There is a thread, dating back to Dec 2020, that proposes a trivial implementation for str.isfloat . I was thinking of a method that did more. Consider the following code. It returns True if the string is a proper float, False if it is an int and None otherwise. def isfloat(s): try: int(s) return False except ValueError: … [View More]

6 20

More efficient list copying
by Steven D'Aprano Oct. 5, 2021

Oct. 5, 2021

This half-baked idea is inspired by this thread here: https://mail.python.org/archives/list/python-ideas@python.org/message/5LGWV… which in turn was inspired by this Stackoverflow post: https://stackoverflow.com/questions/56966429/getting-pairs-of-one-item-and-… Problem: today, more and more people are using Python to work with Big Data, or at least Biggish Data. For some folks, it is not unusual to have lists with a gigabyte or more or data. So imagine you have a list with a hundred … [View More]million items, and you need a copy of the entire list. Easy: alist = [...] # 100 million items blist = alist[:] And that ought to be pretty efficient too, making the slice is basically just a copy of a bunch of pointers, plus a bit of overhead. A good implementation should have lightning fast list slicing, close to the speed of memcopy in C. Give or take. But now suppose you want a copy of all the items except the item in position (let's say) 1000. Here's a bad way: blist = alist[:] del blist[1000] That's inefficient because the list has to shift almost a hundred million items over, and I think they have to be moved one at a time. Which is slowish, even in C. Here's another way: blist = alist[:1000] + alist[1001:] That has to make a slice (copy 1000 items), a second slice (copy another many million items), then make a third copy to concatenate them, then garbage collect the two temporary slices. And while you might have enough memory for double the original list (alist and blist) you might not have enough for triple (alist, the two slices, blist). There are lots of other variants we could come up with, but ultimately we're doing lots of extra work that has to be thrown away, and that costs time or memory or both. How about if lists had an in-place method for doing a fast copy of pointers from one list to another? We could do this: blist = [None]*(len(alist)-1) # Preallocate. blist.copyfrom(alist, index=0, start=0, end=1000) blist.copyfrom(alist, index=1000, start=1001) No temporary slices are needed. Here's a pure-Python (and therefore slow) implementation: def copyfrom(dest, source, index=0, start=0, end=None): if end is None: end = len(source) if len(dest) - index < end - start: raise ValueError('destination too small') for i in range(start, end): dest[index+i-start] = source[i] In pure Python, copying each item over one by one will save memory but cost a lot of time. Doing it as a method in C should save both memory and time. The implementation could optimize the cases where the source is a list, and fall back on iteration for other sequence types. Now *personally* I don't work with a lot of Big Data, so this doesn't really scratch any of my personal itches, but I thought I'd throw it out there for others to tell me why it's a dumb idea :-) -- Steve [View Less]

6 6

Syntax Sugar for __name__ == "__main__" boilerplate?
by Jonathan Crall Oct. 5, 2021

Oct. 5, 2021

I was curious if / what sort of proposals have been considered for simplifying the pattern: ``` def main(): ... if __name__ == "__main__": main() ``` I imagine this topic must have come up before, so I'd be interested in any relevant history. But unless I'm missing something, it seems like adding some easier alternative to this cumbersome entrypoint syntax would be worth considering. My motivation for writing this suggestion is in an attempt to stop a common anti-pattern, where … [View More]

19 38

dict_items.__getitem__?
by Erik Demaine Oct. 5, 2021

Oct. 5, 2021

Have folks thought about allowing indexing dictionary views as in the following code, where d is a dict object? d.keys()[0] d.keys()[-1] d.values()[0] d.values()[-1] d.items()[0] d.items()[-1] # item that would be returned by d.popitem() I could see value to the last form in particular: you might want to inspect the last item of a dictionary before possibly popping it. I've also often wanted to get an arbitrary item/key from a dictionary, and d.items()[0] seems natural for this. Of … [View More]

3 2

Power Assertions: Is it PEP-able?
by noam＠10ne.org Oct. 5, 2021

Oct. 5, 2021

Hi all, I’d like your comments and feedback on an enhancement that introduces power assertions to the Python language. Proposal -------- This feature is inspired by a similar feature of the Groovy language[1], and is effectively a variant of the `assert` keyword. When an assertion expression evaluates to `False`, the output shows not only the failure, but also a breakdown of the evaluated expression from the inner part to the outer part. For example, a procedure like: ```python class … [View More]SomeClass: def __init__(self): self.val = {'d': 'e'} def __str__(self): return str(self.val) sc = SomeClass() assert sc.val['d'] == 'f' ``` Will result in the output: ```python Assertion failed: sc.val['d'] == f | | | | e False | {'d': 'e'} ``` See link [2] if the formatting above is screwed up. In the output above we can see the value of every part of the expression from left to right, mapped to their expression fragment with the pipe (`|`). The number of rows that are printed depend on the value of each fragment of the expression. If the value of a fragment is longer than the actual fragment (`{'d': 'e'}` is longer than `sc`), then the next value (`e`) will be printed on a new line which will appear above. Values are appended to the same line until it overflows in length to horizontal position of the next fragment. The information that’s displayed is dictated by the type. If the type is a constant value, it will be displayed as is. If the type implements `__str__`, then the return value of that will be displayed. It is important to note that expressions with side effects are affected by this feature. This is because in order to display this information, we must store references to the instances and not just the values. Rational -------- Every test boils down to the binary statement "Is this true or false?", whether you use the built-in assert keyword or a more advanced assertion method provided by a testing framework. When an assertion fails, the output is binary too — "Expected x, but got y". There are helpful libraries like Hamcrest which give you a more verbose breakdown of the difference and answer the question "What exactly is the difference between x and y?". This is extremely helpful, but it still focuses on the difference between the values. We need to keep in mind that a given state is normally an outcome of a series of states, that is, one outcome is a result of multiple conditions and causes. This is where power assertion comes in. It allows us to better understand what led to the failure. Implementation -------- I’ve already built a fully functional implementation[2] of this feature as part of my Python testing framework - Nimoy[3]. The current implementation uses AST manipulation to remap the expression to a data structure[4] at compile time, so that it can then be evaluated and printed[5] at runtime. [1] http://docs.groovy-lang.org/next/html/documentation/core-testing-guide.html… [2] https://browncoat-ninjas.github.io/nimoy/examples/#power-assertions-beta [3] https://github.com/browncoat-ninjas/nimoy/ [4] https://github.com/browncoat-ninjas/nimoy/blob/develop/nimoy/ast_tools/expr… [5] https://github.com/browncoat-ninjas/nimoy/blob/develop/nimoy/assertions/pow… [View Less]

8 17

Re: Feature request enumerate_with_rest or enumerate with skip or filter callback
by Laurent Lyaudet Oct. 4, 2021

Oct. 4, 2021

Le dim. 3 oct. 2021 à 16:21, <python-ideas-request(a)python.org> a écrit : > Date: Mon, 4 Oct 2021 01:03:34 +1100 > From: Steven D'Aprano <steve(a)pearwood.info> > Subject: [Python-ideas] Re: Feature request enumerate_with_rest or > enumerate with skip or filter callback > To: python-ideas(a)python.org > Message-ID: <20211003140333.GR16229(a)ando.pearwood.info> > Content-Type: text/plain; charset=us-ascii > > Hi Laurent, Hello Steve, > It … [View More]is not clear to me what you mean by "filter by indices". > > On Sat, Oct 02, 2021 at 10:25:05PM +0200, Laurent Lyaudet wrote: > > > The idea is to filter a list by indices : > [...] > > Since filter() returns an iterator instead of a list, it could do what > > is needed... if the callback had access to the index like the > > Javascript array filter function. > > Do you mean this? > > https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Ob… Yes exactly, > You shouldn't assume we are all experts on Javascript :-) > No problem, I'll give more links next time :) > If that is what you want, it is easy to get access to the index. We can > just do: > > filter(function, enumerate(items)) > That's exactly what I proposed. Except for the fact that I incorrectly permuted the two arguments of filter. I quote my second email : Le sam. 2 oct. 2021 à 22:35, <python-ideas-request(a)python.org> a écrit : > Date: Sat, 2 Oct 2021 22:25:05 +0200 > From: Laurent Lyaudet <laurent.lyaudet(a)gmail.com> >... > Currently, the following solution is available : > filter(enumerate(my_list), lambda x: x[0] != i) > But it is slightly ugly and unefficient to have two function calls for > such a simple task I think. I feel uncomfortable because it is often the case that I write explicitly something, and I get an answer as if the person answered without reading all my email or reading another version. > If you want something else, I'm afraid you will have to explain in more > detail what you want, sorry. Again for my second email, I would like one of these 3 options. > What would be your prefered way of doing this ? > enumerate(my_list, filter_callback=(lambda x: x != i)) > filter_by_index(my_list, lambda x: x != i) > # à la JS > filter(my_list, lambda _, x: x != i) filter_by_index(lambda x: x != i, my_list) could be defined as follow (I permuted args to have similar order to python's filter()): def filter_by_index(function, my_list): for i, item in enumerate(my_list) if function(i): yield item like filter is just (without the case function=None) filter(function, my_list): for i, item in enumerate(my_list) if function(item): yield item The problem is not that it is hard to code in Python. The problem is that it is a basic building block for an iterators tools library like itertools. enumerate(my_list, filter_callback=(lambda x: x != i)) could be defined in a similar way but there are two variants. It just needs to return one of the two indices of the item: - either the index of the item in the original sequence, - or the index of the item in the filtered sequence (new counter). For my use case, I do not need the index so I have no argument in favor of either one of the two indices. > # à la JS > filter(my_list, lambda _, x: x != i) You provided the link from MDN and an helper function for that. The lambda takes 2 or 3 parameters: item, index, array/list/sequence and returns true for item that must be output. I see nothing more to say. Best regards, Laurent Lyaudet [View Less]

1 0

Re: Python-ideas Digest, Vol 179, Issue 25
by Laurent Lyaudet Oct. 4, 2021

Oct. 4, 2021

Le dim. 3 oct. 2021 à 16:21, <python-ideas-request(a)python.org> a écrit : > Date: Mon, 4 Oct 2021 01:03:34 +1100 > From: Steven D'Aprano <steve(a)pearwood.info> > Subject: [Python-ideas] Re: Feature request enumerate_with_rest or > enumerate with skip or filter callback > To: python-ideas(a)python.org > Message-ID: <20211003140333.GR16229(a)ando.pearwood.info> > Content-Type: text/plain; charset=us-ascii > > Hi Laurent, Hello Steve, > It … [View More]is not clear to me what you mean by "filter by indices". > > On Sat, Oct 02, 2021 at 10:25:05PM +0200, Laurent Lyaudet wrote: > > > The idea is to filter a list by indices : > [...] > > Since filter() returns an iterator instead of a list, it could do what > > is needed... if the callback had access to the index like the > > Javascript array filter function. > > Do you mean this? > > https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Ob… Yes exactly, > You shouldn't assume we are all experts on Javascript :-) > No problem, I'll give more links next time :) > If that is what you want, it is easy to get access to the index. We can > just do: > > filter(function, enumerate(items)) > That's exactly what I proposed. Except for the fact that I incorrectly permuted the two arguments of filter. I quote my second email : Le sam. 2 oct. 2021 à 22:35, <python-ideas-request(a)python.org> a écrit : > Date: Sat, 2 Oct 2021 22:25:05 +0200 > From: Laurent Lyaudet <laurent.lyaudet(a)gmail.com> >... > Currently, the following solution is available : > filter(enumerate(my_list), lambda x: x[0] != i) > But it is slightly ugly and unefficient to have two function calls for > such a simple task I think. I feel uncomfortable because it is often the case that I write explicitly something, and I get an answer as if the person answered without reading all my email or reading another version. > If you want something else, I'm afraid you will have to explain in more > detail what you want, sorry. Again for my second email, I would like one of these 3 options. > What would be your prefered way of doing this ? > enumerate(my_list, filter_callback=(lambda x: x != i)) > filter_by_index(my_list, lambda x: x != i) > # à la JS > filter(my_list, lambda _, x: x != i) filter_by_index(lambda x: x != i, my_list) could be defined as follow (I permuted args to have similar order to python's filter()): def filter_by_index(function, my_list): for i, item in enumerate(my_list) if function(i): yield item like filter is just (without the case function=None) filter(function, my_list): for i, item in enumerate(my_list) if function(item): yield item The problem is not that it is hard to code in Python. The problem is that it is a basic building block for an iterators tools library like itertools. enumerate(my_list, filter_callback=(lambda x: x != i)) could be defined in a similar way but there are two variants. It just needs to return one of the two indices of the item: - either the index of the item in the original sequence, - or the index of the item in the filtered sequence (new counter). For my use case, I do not need the index so I have no argument in favor of either one of the two indices. > # à la JS > filter(my_list, lambda _, x: x != i) You provided the link from MDN and an helper function for that. The lambda takes 2 or 3 parameters: item, index, array/list/sequence and returns true for item that must be output. I see nothing more to say. Best regards, Laurent Lyaudet [View Less]

1 0

Shorthand syntax for lambda functions that have a single parameter
by Dominik Vilsmeier Oct. 4, 2021

Oct. 4, 2021

Lambda functions that have a single parameter are a common thing, e.g. for "key" functions: `sorted(items, key=lambda x: x['key'])`. For these cases however, the rather long word "lambda" together with the repetition of the parameter name, results in more overhead than actual content (the `x['key']`) and thus makes it more difficult to read. This is also a difficulty for `map` and `filter` for example where there's lots of visual overhead when using a lambda function and hence it's difficult to … [View More]read: `filter(lambda x: x > 0, items)` or `map(lambda x: f'{x:.3f}', items)`. Hence the proposal is to add a new syntax via the new token `?`. For the examples above: * `sorted(items, key=?['key'])` * `filter(? > 0, items)` * `map(f'{?:.3f}', items)` The rules are simple: whenever the token `?` is encountered as part of an expression (at a position where a name/identifier would be legal), the expression is replaced by a lambda function with a single parameter which has that expression as a return value, where any instances of `?` are replaced by the name of that single parameter. For example: * `?['key']` translates to `lambda x: x['key']` * `? > 0` translates to `lambda x: x > 0` * `f'{?:.3f}'` translates to `lambda x: f'{x:.3f}'` * `?*?` translates to `lambda x: x*x` With the difference that the replacement function would use an unnamed parameter, i.e. not shadowing any names from outer scopes. So `? * x` would use `x` from enclosing scopes, not as the lambda parameter. Regarding operator precedence, the rules would be similar as for lambda functions, i.e. it will include as much in the expression as it would for a lambda function. To find more example use cases, I used `grep -rnE 'lambda [_a-zA-Z][_a-zA-Z0-9]*:' Lib/` on the CPython standard library to find single parameter lambda functions and there are various results coming up (although not all are applicable, e.g. because they return a constant value). I'll include a few examples below, but the actual list is much longer: ``` modes[char] = max(items, key=lambda x: x[1]) modes[char] = max(items, key=?[1]) # proposed new syntax obj = unwrap(obj, stop=(lambda f: hasattr(f, "__signature__"))) obj = unwrap(obj, stop=hasattr(?, "__signature__")) s = re.sub(r"-[a-z]\b", lambda m: m.group().upper(), s) s = re.sub(r"-[a-z]\b", ?.group().upper(), s) members.sort(key=lambda t: (t[1], t[0])) members.sort(key=(?[1], ?[0])) ``` Of course, there is no requirement to use `?` as the token, it could be any other character that's currently not used by Python, i.e. `$` or `!` would be possible too. [View Less]

13 22

Re: Feature request enumerate_with_rest or enumerate with skip or filter callback
by Laurent Lyaudet Oct. 3, 2021

Oct. 3, 2021

Le sam. 2 oct. 2021 à 15:54, <python-ideas-request(a)python.org> a écrit : > Date: Sat, 2 Oct 2021 12:45:33 +0200 > From: Laurent Lyaudet <laurent.lyaudet(a)gmail.com> > Subject: [Python-ideas] Feature request enumerate_with_rest or > enumerate with skip or filter callback > To: python-ideas(a)python.org > Message-ID: > <CAB1LBmsXQJbeLuyK6tG8aP=kb36DAR==VcajO-zHuHJGnfvAJA(a)mail.gmail.com> > Content-Type: text/plain; charset="UTF-8" > … [View More]> Hello, > > This is a very simple feature request that does not break anything but > I don't know if you may find it interesting. > It would be nice to have a function or method of list objects that does this : > - First idea : > def enumerate_with_rest(my_list): > for i, item in enumerate(my_list): > yield i, item, my_list[:i] + my_list[i + 1:] > > It could be called easily with: > > for i, item, rest in enumerate_with_rest(my_list): > # do something > > or > for i, item, rest in my_list.enumerate_with_rest(): > # do something > > I am not the only one who had the same need : > https://stackoverflow.com/questions/56966429/getting-pairs-of-one-item-and-… > > It would be nice to have an optimized C function for this. > However, it may be less interesting than this : > - Second idea > enumerate_with_rest above has quadratic complexity. > It is probably true that most processes that use it will also have > quadratic complexity. > However, it would be better to return an iterator instead of a list > for the rest, > it would use less space. > For this a param skip to enumerate would do the job > def enumerate_with_rest(my_list): > for i, item in enumerate(my_list): > yield i, item, enumerate(my_list, skip=i) > There could be variants of this idea like : > - enumerate(my_list, skip=i) > - enumerate(my_list, skip=[i]) > - enumerate(my_list, filter_callback=(lambda x: x != i)) > > Please let me know what you think of it :) > > Thanks for your time, best regards, > Laurent Lyaudet Hello, Regarding the last suggestion. - enumerate(my_list, filter_callback=(lambda x: x != i)) The idea is to filter a list by indices : a quick search for that just yields : https://stackoverflow.com/questions/11847491/python-filtering-lists-by-indi… which is not really helpful Since filter() returns an iterator instead of a list, it could do what is needed... if the callback had access to the index like the Javascript array filter function. > def enumerate_with_rest(my_list): > for i, item in enumerate(my_list): > yield i, item, my_list[:i] + my_list[i + 1:] could also be : def enumerate_with_rest(my_list): for i, item in enumerate(my_list): yield i, item, filter_by_index(my_list, lambda x: x != i) I think my feature request for enumerate_with_rest is maybe too specific. However, I think there should definitely be in itertools something to filter indices like JS permits. I have not found a function to do this there: https://docs.python.org/3/library/functions.html https://docs.python.org/3/library/itertools.html What would be your prefered way of doing this ? enumerate(my_list, filter_callback=(lambda x: x != i)) filter_by_index(my_list, lambda x: x != i) # à la JS filter(my_list, lambda _, x: x != i) Currently, the following solution is available : filter(enumerate(my_list), lambda x: x[0] != i) But it is slightly ugly and unefficient to have two function calls for such a simple task I think. I would appreciate any feedback; even if none of my ideas are accepted, I may learn something :) Best regards, Laurent Lyaudet [View Less]

2 1

Feature request enumerate_with_rest or enumerate with skip or filter callback
by Laurent Lyaudet Oct. 3, 2021

Oct. 3, 2021

Hello, This is a very simple feature request that does not break anything but I don't know if you may find it interesting. It would be nice to have a function or method of list objects that does this : - First idea : def enumerate_with_rest(my_list): for i, item in enumerate(my_list): yield i, item, my_list[:i] + my_list[i + 1:] It could be called easily with: for i, item, rest in enumerate_with_rest(my_list): # do something or for i, item, rest in my_list.enumerate_with_rest(… [View More]

3 3

Better exception handling hygiene
by Soni L. Oct. 3, 2021

Oct. 3, 2021

So uh, this is a hardly at all fleshed out idea, but one thing we really don't like about python is having to do stuff like this so as to not swallow exceptions: def a_potentially_recursive_function(some, args): """ Does stuff and things. Raises ExceptionWeCareAbout under so and so conditions. """ try: some.user_code() except ExceptionWeCareAbout as exc: raise RuntimeError from exc code_we_assume_is_safe() if args.something and some_condition: &… [View More]

5 28

Allow regex group name redefinitions
by vencik＠razdva.cz Oct. 3, 2021

Oct. 3, 2021

Hello everybody, I've got a suggestion for the std. re module developers: to consider allowing match group name redefinitions, especially in alternatives. While you may not see the point at first glance, let me try to reason such a thing using a real-world example from my practice: Imagine a company, using certain codes for their products/product components (unfortunately, I'm not at liberty to disclose the true nature of them, but bare with me). Let's say that they may have the following … [View More]forms: r"(?P<type>AB|C|D)[- ](?P<prefix>[A-Z])?(?P<number>\d+)(?P<postfix>[A-Z])?" So far so good. But now, imagine that a particular type of code has a bit different syntax: r"(?P<type>E)[- ](?P<prefix>[A-Za-z])?(?P<number>\d+)[- ](?P<postfix>[A-Za-z])?" As you can see, the prefix & postfix may be lowercase in case of code type E and moreover, a space or dash is required before the postfix. If I merged the definitions, I'd have to allow that syntax even for the AB, C and D code types---but that would've been incorrect and would require post-matching checks. Ideally I'd like to have the opportunity to define the regex as an alternative: r"(?P<type>AB|C|D)[- ](?P<prefix>[A-Z])?(?P<number>\d+)(?P<postfix>[A-Z])?|(?P<type>E)[- ](?P<prefix>[A-Za-z])?(?P<number>\d+)[- ](?P<postfix>[A-Za-z])?" I can't, of course, getting the re.error: redefinition of group name error upon the regex compilation. But is that really a problem, especially in such alternatives? If you imagine the regex as a FSA, the code type branches into completely independent sub-trees of the automaton state transitions. There's no problem with efficiency; the regex might look a bit complex, but the matching is perfectly efficient---definitely more so than if I match multiple expressions. The redefinition of the match group names is IMO technically perfectly possible and note that in such alternatives, re-assignments won't really happen. And finally, even if they would happen, what's the problem with that? Might be a logical error in the regex definition of course, but that's the programmer's lookout in general... So what do you think? If the match group name redefinition was allowed, I could just match a single regex, getting match group dict and read out parsed parts of the codes by name---nice and easy. Currently, my 2 choices are: 1/ Use uniquely named groups, which requires me to do a post-match group name consolidation of sort or 2/ Match multiple reg. expressions, which is unnecessary Therefore, I ask you to reconsider issuing the error, which I deem redundant and unnecessarily limiting a justified use-case, IMO. Also note that doing that won't break any old code---anything that worked before will continue to work with unchanged semantics; so such a change would be perfectly safe. Thanks, Best Regards vasek [View Less]

3 2

Reusing more builtins for type-hinting
by Matt del Valle Oct. 2, 2021

Oct. 2, 2021

In the spirit of 3.9 allowing builtin collections like `list`, `dict`, `tuple`, and `set` to be subscripted for type-hinting to remove the need for importing their equivalents from `typing`, I'd like to propose the same thing for a few other builtin objects, namely: `type` (for use instead of`typing.Type`) `any` (for use instead of `typing.Any`) `callable` (for use instead of `typing.Callable`) (I'm aware that a better syntax for type-hinting callables using arrow-syntax is potentially … [View More]

7 14

pathlib update idea: iterable Path that returns subpaths
by Aaron Stacy Oct. 2, 2021

Oct. 2, 2021

I’d like to propose an enhancement to the wonderful pathlib module: make Path instances iterable, where the iterator returns subpaths. This would be functionally very similar to Path(some_directory).rglob(‘*’). Why not just use rglob(‘*’)? While globs are an extremely useful shorthand, in my experience they’re a bit of an artifact of Unix shells that programmers frequently aren’t aware of or don’t understand. Python does a great job of distilling the concept of what it means to iterate over … [View More]

3 2

define an idiomatic way to make a generator that throws StopIteration immediately and add it to PEP8
by Thomas Grainger Oct. 1, 2021

Oct. 1, 2021

the way to make an (aysnc)generator that immediately raises Stop(Async)Iteration immediately is to insert an unreachable `yield` statement, however there's two ways to do it: def example(): if False: yield or: def example(): return yield currently test_asyncgen.py uses both, eg: @types.coroutine def __anext__(self): if False: yield "this is a generator-based coroutine" … [View More]

3 3