Mailman 3 September 2021 - Python-ideas

zip(x, y, z, strict=True)
by Ram Rachum 01 Dec '22

01 Dec '22

Here's something that would have saved me some debugging yesterday: >>> zipped = zip(x, y, z, strict=True) I suggest that `strict=True` would ensure that all the iterables have been exhausted, raising an exception otherwise. This is useful in cases where you're assuming that the iterables all have the same lengths. When your assumption is wrong, you currently just get a shorter result, and it could take you a while to figure out why it's happening. What do you think?

27 179

disallow assignment to unknown ssl.SSLContext attributes
by Thomas Grainger 10 Jun '22

10 Jun '22

I was debugging some code that was using TLSv1.2 when I expected it to only support TLSv1.3, I tracked it down to a call to: context.miunimum_version = ssl.TLSVersion.TLSv1_3 it should have been: context.minimum_version = ssl.TLSVersion.TLSv1_3 I'd like invalid attribute assignment to be prevented at runtime

13 32

Bringing the print statement back
by Guido van Rossum 02 Mar '22

02 Mar '22

In Python 3.10 we will no longer be burdened by the old parser (though 3rd party tooling needs to catch up). One thing that the PEG parser makes possible in about 20 lines of code is something not entirely different from the old print statement. I have a prototype: Python 3.10.0a0 (heads/print-statement-dirty:5ed19fcc1a, Jun 9 2020, 16:31:17) [Clang 11.0.0 (clang-1100.0.33.8)] on darwin Type "help", "copyright", "credits" or "license" for more information. Cannot read termcap database; using dumb terminal settings. >>> print 2+2 4 >>> print "hello world" hello world >>> print "hello", input("Name:") Name:Guido hello Guido >>> print 1, 2, 3, sep=", " 1, 2, 3 >>> But wait, there's more! The same syntax will make it possible to call *any* function: >>> len "abc" 3 >>> Or any method: >>> import sys >>> sys.getrefcount "abc" 24 >>> Really, *any* method: >>> class C: ... def foo(self, arg): print arg ... >>> C().foo 2+2 4 >>> There are downsides too, though. For example, you can't call a method without arguments: >>> print <built-in function print> >>> Worse, the first argument cannot start with a parenthesis or bracket: >>> print (1, 2, 3) 1 2 3 >>> C().foo (1, 2, 3) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: C.foo() takes 2 positional arguments but 4 were given >>> print (2+2), 42 4 (None, 42) >>> C().foo [0] Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'method' object is not subscriptable >>> No, it's not April 1st. I am seriously proposing this (but I'll withdraw it if the response is a resounding "boo, hiss"). After all, we currently have a bunch of complexity in the parser just to give a helpful error message to people used to Python 2's print statement: >>> print 1, 2, 3 File "<stdin>", line 1 print 1, 2, 3 ^ SyntaxError: Missing parentheses in call to 'print'. Did you mean print(1, 2, 3)? >>> And IIRC there have been a number of aborted attempts at syntactic hacks to allow people to call functions (like print) without parentheses, although (I think) none of them made it into a PEP. The PEG parser makes this much simpler, because it can simply backtrack -- by placing the grammar rule for this syntax (tentatively called "call statement") last in the list of alternatives for "small statement" we ensure that everything that's a valid expression statement (including print() calls) is still an expression statement with exactly the same meaning, while still allowing parameter-less function calls, without lexical hacks. (There is no code in my prototype that checks for a space after 'print' -- it just checks that there's a name, number or string following a name, which is never legal syntax.) One possible extension I didn't pursue (yet -- dare me!) is to allow parameter-less calls inside other expressions. For example, my prototype does not support things like this: >>> a = (len "abc") File "<stdin>", line 1 a = (len "abc") ^ SyntaxError: invalid syntax >>> I think that strikes a reasonable balance between usability and reduced detection of common errors. I could also dial it back a bit, e.g. maybe it's too much to allow 'C().foo x' and we should only allow dotted names (sufficient to access functions in imported modules and method calls on variables). Or maybe we should only allow simple names (allowing 'len x' but disallowing 'sys.getrefcount x'. Or maybe we should really only bring back print statements. I believe there are some other languages that support a similar grammar (Ruby? R? Raku?) but I haven't investigated. Thoughts? -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-…>

42 66

Adding a thin wrapper class around the functions in stdlib.heapq
by bunslow 18 Feb '22

18 Feb '22

Nothing so bombastic this time. The heapq functions are basically all named "heapsomething", and basically all take a "heap" for their first argument, with supplementary args coming after. It's a textbook example of the (hypothetical) Object Oriented Manifesto™ where defining a class increases type safety and programmers' conceptual clarity. There're practically no drawbacks, and the code to be added would be very simple. Updating the tests and docs would probably be harder. In pure Python, such a class would look like this: class Heap(list): def __init__(self, iterable=None): if iterable: super().__init__(iterable) else: super().__init__() self.heapify() push = heapq.heappush pop = heapq.heappop pushpop = heapq.heappushpop replace = heapq.heapreplace heapify = heapq.heapify # This could be a simple wrapper as well, but I had the following thoughts anyways, so here they are def nsmallest(self, n, key=None): # heapq.nsmallest makes a *max* heap of the first n elements, # while we know that self is already a min heap, so we can # make the max heap construction faster self[:n] = reversed(self[:n]) return heapq.nsmallest(n, self, key) # do we define nlargest on a minheap?? Wrapping around the C builtin functions (which aren't descriptors) would be a bit harder, but not much so: from functools import partial class Heap(list): def __init__(self, iterable=None): if iterable: super().__init__(iterable) else: super().__init__() self.heapify = partial(heapq.heapify, self) self.push = partial(heapq.heappush, self) ... self.heapify() Thoughts?

17 32

Allow syntax "func(arg=x if condition)"
by Peter O'Connor 19 Jan '22

19 Jan '22

I often find that python lacks a nice way to say "only pass an argument under this condition". (See previous python-list email in "Idea: Deferred Default Arguments?") Example 1: Defining a list with conditional elements include_bd = True current_way = ['a'] + (['b'] if include_bd else [])+['c']+(['d'] if include_bd else []) new_way = ['a', 'b' if include_bd, 'c', 'd' if include_bd] also_new_way = list('a', 'b' if include_bd, 'c', 'd' if include_bd) Example 2: Deferring to defaults of called functions def is_close(a, b, precicion=1e-9): return abs(a-b) < precision def approach(pose, target, step=0.1, precision=None): # Defers to default precision if not otherwise specified: velocity = step*(target-pose) \ if not is_close(pose, target, precision if precision is not None) \ else 0 return velocity Not sure if this has been discussed, but I cannot see any clear downside to adding this, and it has some clear benefits (duplicated default arguments and **kwargs are the scourge of many real world code-bases)

6 9

os.workdir() context manager
by Marc-Andre Lemburg 22 Oct '21

22 Oct '21

Hello all, I sometimes write Python scripts which need to work in specific work directories. When putting such code into functions, the outer function typically does not expect the current work dir (CWD) to be changed, so wrap the code which need the (possibly) modified CWD using a simply context manager along the lines of: class workdir: def __init__(self, dir): self.dir = dir def __enter__(self): self.curdir = os.getcwd() os.chdir(self.dir) def __exit__(self, *exc): os.chdir(self.curdir) return False Would there be interest in adding something like this to the os module as os.workdir() ? Example: def backup_home_dir(account): with os.workdir(os.path.join('/home', account)): # Create a backup of the account dir, rooted at the account's # home dir restic.backup(repo, '.') Notes: - The context manager is not thread safe. There's no thread safe model for the current work dir. OTOH, scripts usually don't use threads, so not a big deal. - The context manager could be made more elaborate, e.g. adding optional logging for debugging purposes. - The same could be added to pathlib's objects as .workdir() method returning a context manager. Thoughts ? Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Sep 14 2021) >>> Python Projects, Coaching and Support ... https://www.egenix.com/ >>> Python Product Development ... https://consulting.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/

17 37

Power Assertions: Is it PEP-able?
by noam＠10ne.org 04 Oct '21

04 Oct '21

Hi all, I’d like your comments and feedback on an enhancement that introduces power assertions to the Python language. Proposal -------- This feature is inspired by a similar feature of the Groovy language[1], and is effectively a variant of the `assert` keyword. When an assertion expression evaluates to `False`, the output shows not only the failure, but also a breakdown of the evaluated expression from the inner part to the outer part. For example, a procedure like: ```python class SomeClass: def __init__(self): self.val = {'d': 'e'} def __str__(self): return str(self.val) sc = SomeClass() assert sc.val['d'] == 'f' ``` Will result in the output: ```python Assertion failed: sc.val['d'] == f | | | | e False | {'d': 'e'} ``` See link [2] if the formatting above is screwed up. In the output above we can see the value of every part of the expression from left to right, mapped to their expression fragment with the pipe (`|`). The number of rows that are printed depend on the value of each fragment of the expression. If the value of a fragment is longer than the actual fragment (`{'d': 'e'}` is longer than `sc`), then the next value (`e`) will be printed on a new line which will appear above. Values are appended to the same line until it overflows in length to horizontal position of the next fragment. The information that’s displayed is dictated by the type. If the type is a constant value, it will be displayed as is. If the type implements `__str__`, then the return value of that will be displayed. It is important to note that expressions with side effects are affected by this feature. This is because in order to display this information, we must store references to the instances and not just the values. Rational -------- Every test boils down to the binary statement "Is this true or false?", whether you use the built-in assert keyword or a more advanced assertion method provided by a testing framework. When an assertion fails, the output is binary too — "Expected x, but got y". There are helpful libraries like Hamcrest which give you a more verbose breakdown of the difference and answer the question "What exactly is the difference between x and y?". This is extremely helpful, but it still focuses on the difference between the values. We need to keep in mind that a given state is normally an outcome of a series of states, that is, one outcome is a result of multiple conditions and causes. This is where power assertion comes in. It allows us to better understand what led to the failure. Implementation -------- I’ve already built a fully functional implementation[2] of this feature as part of my Python testing framework - Nimoy[3]. The current implementation uses AST manipulation to remap the expression to a data structure[4] at compile time, so that it can then be evaluated and printed[5] at runtime. [1] http://docs.groovy-lang.org/next/html/documentation/core-testing-guide.html… [2] https://browncoat-ninjas.github.io/nimoy/examples/#power-assertions-beta [3] https://github.com/browncoat-ninjas/nimoy/ [4] https://github.com/browncoat-ninjas/nimoy/blob/develop/nimoy/ast_tools/expr… [5] https://github.com/browncoat-ninjas/nimoy/blob/develop/nimoy/assertions/pow…

8 17

Shorthand syntax for lambda functions that have a single parameter
by Dominik Vilsmeier 04 Oct '21

04 Oct '21

Lambda functions that have a single parameter are a common thing, e.g. for "key" functions: `sorted(items, key=lambda x: x['key'])`. For these cases however, the rather long word "lambda" together with the repetition of the parameter name, results in more overhead than actual content (the `x['key']`) and thus makes it more difficult to read. This is also a difficulty for `map` and `filter` for example where there's lots of visual overhead when using a lambda function and hence it's difficult to read: `filter(lambda x: x > 0, items)` or `map(lambda x: f'{x:.3f}', items)`. Hence the proposal is to add a new syntax via the new token `?`. For the examples above: * `sorted(items, key=?['key'])` * `filter(? > 0, items)` * `map(f'{?:.3f}', items)` The rules are simple: whenever the token `?` is encountered as part of an expression (at a position where a name/identifier would be legal), the expression is replaced by a lambda function with a single parameter which has that expression as a return value, where any instances of `?` are replaced by the name of that single parameter. For example: * `?['key']` translates to `lambda x: x['key']` * `? > 0` translates to `lambda x: x > 0` * `f'{?:.3f}'` translates to `lambda x: f'{x:.3f}'` * `?*?` translates to `lambda x: x*x` With the difference that the replacement function would use an unnamed parameter, i.e. not shadowing any names from outer scopes. So `? * x` would use `x` from enclosing scopes, not as the lambda parameter. Regarding operator precedence, the rules would be similar as for lambda functions, i.e. it will include as much in the expression as it would for a lambda function. To find more example use cases, I used `grep -rnE 'lambda [_a-zA-Z][_a-zA-Z0-9]*:' Lib/` on the CPython standard library to find single parameter lambda functions and there are various results coming up (although not all are applicable, e.g. because they return a constant value). I'll include a few examples below, but the actual list is much longer: ``` modes[char] = max(items, key=lambda x: x[1]) modes[char] = max(items, key=?[1]) # proposed new syntax obj = unwrap(obj, stop=(lambda f: hasattr(f, "__signature__"))) obj = unwrap(obj, stop=hasattr(?, "__signature__")) s = re.sub(r"-[a-z]\b", lambda m: m.group().upper(), s) s = re.sub(r"-[a-z]\b", ?.group().upper(), s) members.sort(key=lambda t: (t[1], t[0])) members.sort(key=(?[1], ?[0])) ``` Of course, there is no requirement to use `?` as the token, it could be any other character that's currently not used by Python, i.e. `$` or `!` would be possible too.

13 22

Better exception handling hygiene
by Soni L. 03 Oct '21

03 Oct '21

So uh, this is a hardly at all fleshed out idea, but one thing we really don't like about python is having to do stuff like this so as to not swallow exceptions: def a_potentially_recursive_function(some, args): """ Does stuff and things. Raises ExceptionWeCareAbout under so and so conditions. """ try: some.user_code() except ExceptionWeCareAbout as exc: raise RuntimeError from exc code_we_assume_is_safe() if args.something and some_condition: raise ExceptionWeCareAbout It'd be nice if there was a way to... make this easier to deal with. Perhaps something where, something like this: def a_potentially_recursive_function(some, args) with ExceptionWeCareAbout: """ Does stuff and things. Raises ExceptionWeCareAbout under so and so conditions. """ some.user_code() code_we_assume_is_safe() if args.something and some_condition: raise ExceptionWeCareAbout becomes: def a_potentially_recursive_function(some, args): """ Does stuff and things. Raises ExceptionWeCareAbout under so and so conditions. """ try: some.user_code() code_we_assume_is_safe() if args.something and some_condition: let_exception_through = True raise ExceptionWeCareAbout except ExceptionWeCareAbout as exc: if let_exception_through: raise else: raise RuntimeError from exc (or equivalent) and something like: def foo() with Bar: try: baz() except Bar: raise becomes: def foo(): try: try: baz() except Bar: allow_exception = True raise except Bar as exc: if allow_exception: raise else: raise RuntimeError from exc (thus allowing explicit exception propagation) Additionally, note that it would only apply to the raise itself - something like `raise may_raise_the_same_error()` would desugar as if: exception = may_raise_the_same_error() allow_exception = True raise exception Obviously this doesn't solve exception handling, doesn't require the caller to catch the exceptions, etc etc. It does, however, encourage better exception hygiene. Chances are something like this would significantly reduce the amount of swallowed exceptions, if it gets widely adopted. We know something like this would've saved us a lot of trouble, so we're sharing the idea in the hopes it can, in the future, help others.

5 28

define an idiomatic way to make a generator that throws StopIteration immediately and add it to PEP8
by Thomas Grainger 01 Oct '21

01 Oct '21

the way to make an (aysnc)generator that immediately raises Stop(Async)Iteration immediately is to insert an unreachable `yield` statement, however there's two ways to do it: def example(): if False: yield or: def example(): return yield currently test_asyncgen.py uses both, eg: @types.coroutine def __anext__(self): if False: yield "this is a generator-based coroutine" if self.yielded >= 2: raise StopAsyncIteration() else: self.yielded += 1 return self.yielded or: async def agenfn(): try: await _async_yield(1) except MyError: await _async_yield(2) return yield the `if False: yield` version has a disadvantage in that it can be anywhere in the function, if this one is chosen then an additional caveat is that it should be the first two statements in a function if `return; yield` version has an disadvantage in that the unreachable yield could be inserted after any `return`, if this one is chosen then an additional caveat is that it should be the last two statements in a function and de-dented as much as syntactically possible. Which alternative do you prefer? I prefer `return; yield`

3 3