Mailman 3 March 2023 - Python-ideas

PEP 671 (late-bound arg defaults), next round of discussion!
by Chris Angelico 23 Jul '23

23 Jul '23

I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/ with some additional information about the reference implementation, and some clarifications elsewhere. *PEP 671: Syntax for late-bound function argument defaults* Questions, for you all: 1) If this feature existed in Python 3.11 exactly as described, would you use it? 2) Independently: Is the syntactic distinction between "=" and "=>" a cognitive burden? (It's absolutely valid to say "yes" and "yes", and feel free to say which of those pulls is the stronger one.) 3) If "yes" to question 1, would you use it for any/all of (a) mutable defaults, (b) referencing things that might have changed, (c) referencing other arguments, (d) something else? 4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.) 5) Do you know how to compile CPython from source, and would you be willing to try this out? Please? :) I'd love to hear, also, from anyone's friends/family who know a bit of Python but haven't been involved in this discussion. If late-bound defaults "just make sense" to people, that would be highly informative. Any and all comments welcomed. I mean, this is python-ideas after all... bikeshedding is what we do best! The reference implementation currently has some test failures, which I'm looking into. I'm probably going to make this my personal default Python interpreter for a while, to see how things go. ChrisA

43 364

Make decorator-based context managers reentrant
by Tobias Bengfort 30 Mar '23

30 Mar '23

Hi, I was wondering why decorator-based context managers are not reentrant or even reusable. They could be made reusable by initializing the generator in __enter__() instead of __init__(). They could be made reentrant by storing generatos in a stack. A similar thing has been implemented for redirect_stdout in 8e113b418df7d0c8480e1e2de29a385e1f31b15b. I understand that not all generator-based context managers can be reentrant by their nature. But currently it is impossible to make them reentrant, which IMHO is an unnecessary restriction. I could find very little discussion on this. Are there any strong reasons why it was implemented this way or should I create a pull request to change it? (My specific use case is a TUI application where I want to make sure that the terminal is restored to a usable state on exit, but also need to step in and out of the TUI on SIGTSTP/SIGCONT.) thanks, tobias https://docs.python.org/3/library/contextlib.html#single-use-reusable-and-r…

2 2

An interface for `mro` methods similar to the interface for `iter`, `getattr`, etc...
by Samuel Muldoon 24 Mar '23

24 Mar '23

*Greetings Fellow Earthlings,* *In python 3.11, python has a number of built-in functions which act as an Interface for magic dunder methods. Two examples of what I am talking about are shown below: * *Begin Example 1 of 2* *rocky = object()x = getattr(rocky, "length") x = rocky.__getattr__("length") * *Begin Example 2 of 2* *box = [1, 2, 3]it = iter(box) it = box.__iter__()* *My suggestion is that in future versions of python, we will have `mro()` behave in a similar fashion. * *obj = object()x = mro(obj) x = obj.__mro__() * *Additionally, the class named `object` should have a method named `__mro__` and any class which inherits from `object` may override `__mro__` or inherit the default `* *object* *.__mro__` . * *Samuel Muldoon* *(720) 653 -2408* *muldoonsamuel(a)gmail.com <muldoonsamuel(a)gmail.com>*

3 2

C#-style "as" casting syntactic sugar for type hints
by Will Bradley 23 Mar '23

23 Mar '23

This is some real code I wrote the other day: payload = { "birthDate": str(date.strftime('%m/%d/%Y')), ... } Due to the fact that date.strftime returns Any type, for my type hinting and autocomplete to work I either need to pass it to the str function or typing.cast(str, date). It would be nice if I could just write "birthDate": date.strftime(r'%m/%d/%Y') as str, like in C#. It's shorter, (in my opinion) more readable, and avoids having to unnecessarily call a constructor or import typing.cast. It's important to note that this would be syntactic sugar for typing.cast— it wouldn't call any constructor or affect program state, just increase the power of the type hinting system. Will Bradley

3 2

Bind/normalize params for @functools.cache
by Steve Jorgensen 19 Mar '23

19 Mar '23

I was surprised to find, when I pass arguments to a function decorated with `(a)functools.cache` in different, equivalent ways, the cache does not recognize them as the same. counter = itertools.count(1) @functools.cache def example(a, b, c=0): return (next(counter), a, b, c) example(1, 2) # => (1, 1, 2, 0) example(1, b=2) # => (2, 1, 2, 0) example(1, 2, 0) # => (3, 1, 2, 0) When I wrote my own implementation as a coding exercise, I noticed the same weakness while testing it and solved that by having the decorator function get the signature of the decorated function, then use the bind method of the signature to bind the parameter values, then call the apply_defaults method on the bound arguments, and then finally, use the args and kwargs properties of the bound arguments to make the cache key. It seems like functools.cache should do the same thing. If it is undesirable for that to be the default behavior, then it could be optional (e.g. @functools.cache(normalize=True) ). I have not tested to see if functools.lru_cache has the same issue. I presume that it does, so my suggestion would apply to that as well.

1 1

join() could add separators at start or end
by Rob Cliffe 14 Mar '23

14 Mar '23

Having had my last proposal shot down in flames, up I bob with another. 😁 It seems to me that it would be useful to be able to make the str.join() function put separators, not only between the items of its operand, but also optionally at the beginning or end. E.g. '|'.join(('Spam', 'Ham', 'Eggs')) returns 'Spam|Ham|Eggs' but it might be useful to make it return one of '|Spam|Ham|Eggs' 'Spam|Ham|Eggs|' '|Spam|Ham|Eggs|' Again, I suggest that this apply to byte strings as well as strings. Going through the 3.8.3 stdlib I have found 24 examples where the separator needs to be added at the beginning 52 where the separator needs to be added at the end 4 where the separator needs to be added at the both ends I list these examples below. Apologies if there are any mistakes. While guessing is no substitute for measurement, it seems plausible that using this feature where appropriate would increase runtime performance by avoiding 1 (or 2) calls of str.__add__. This is perhaps more relevant when the separator is not a short constant string, as in this example: Lib\email\_header_value_parser.py:2854: return policy.linesep.join(lines) + policy.linesep Note also this example: Lib\site-packages\setuptools\command\build_ext.py:221: pkg = '.'.join(ext._full_name.split('.')[:-1] + ['']) where the author has used the unintuitive device of appending an empty string to a list to force join() to add an extra final dot, thereby avoiding 1 call of str.__add__ at the cost of 1 call of list.append. What I cannot decide is what the best API would be. str.join() currently takes only 1 parameter, so it would be possible to add an extra parameter or two. One scheme would be to have an atEnds parameter which could take values such as 0=default behaviour 1=add sep at start 2=add sep at end 3=add sep at both ends or 's'=add sep at start 'e'=add sep at end 'b'=add sep at both ends (some) other=default behaviour Another would be to have 2 parameters, atStart and atEnd, which would both default to False or 0. E.g. '|'.join(('Spam', 'Ham', 'Eggs'), 1) == '|Spam|Ham|Eggs' '|'.join(('Spam', 'Ham', 'Eggs'), 0, 1) == 'Spam|Ham|Eggs|' Neither scheme results in particularly transparent usage, though no worse than s.splitlines(True) # What on earth is this parameter??? Corner case: What if join() is passed an empty sequence? This is debatable, but I think it should return the separator if requested to add it at the beginning or end, and double it up if both are requested. This would preserve identities such as sep.join(seq, <PleaseAddSeparatorAtBeginning>) == sep + sep.join(seq) Best wishes Rob Cliffe EXAMPLES WHERE SEPARATOR ADDED AT START: Lib\http\server.py:933: splitpath = ('/' + '/'.join(head_parts), tail_part) Lib\site-packages\numpy\ctypeslib.py:333: name += "_"+"_".join(flags) Lib\site-packages\numpy\testing\_private\utils.py:842: err_msg += '\n' + '\n'.join(remarks) Lib\site-packages\pip\_vendor\pyparsing\core.py:2092-2095: out = [ "\n" + "\n".join(comments) if comments else "", pyparsing_test.with_line_numbers(t) if with_line_numbers else t, ] Lib\site-packages\pip\_vendor\requests\status_codes.py:121-125: __doc__ = ( __doc__ + "\n" + "\n".join(doc(code) for code in sorted(_codes)) if __doc__ is not None else None ) Lib\site-packages\reportlab\lib\utils.py:1093: self._writeln(' '+' '.join(A.__self__)) Lib\site-packages\reportlab\platypus\flowables.py:708: L = "\n"+"\n".join(L) Lib\site-packages\twisted\mail\smtp.py:1647: r.append(c + b' ' + b' '.join(v)) Lib\site-packages\twisted\protocols\ftp.py:1203: return (PWD_REPLY, '/' + '/'.join(self.workingDirectory)) Lib\site-packages\twisted\runner\procmon.py:424-426: return ('<' + self.__class__.__name__ + ' ' + ' '.join(l) + '>') Lib\site-packages\twisted\web\rewrite.py:34: request.path = '/'+'/'.join(request.prepath+request.postpath) Lib\site-packages\twisted\web\rewrite.py:51: request.path = '/'+'/'.join(request.prepath+request.postpath) Lib\site-packages\twisted\web\twcgi.py:78: scriptName = b"/" + b"/".join(request.prepath) Lib\site-packages\twisted\web\twcgi.py:95: env["PATH_INFO"] = "/" + "/".join(pp) Lib\site-packages\twisted\web\vhost.py:115: request.path = b'/' + b'/'.join(request.postpath) Lib\site-packages\twisted\web\wsgi.py:283: scriptName = b'/' + b'/'.join(request.prepath) Lib\site-packages\twisted\web\wsgi.py:288: pathInfo = b'/' + b'/'.join(request.postpath) Lib\site-packages\twisted\web\test\test_wsgi.py:272: uri = '/' + '/'.join([urlquote(seg, safe) for seg in requestSegments]) Lib\site-packages\wx\py\magic.py:55: command = 'sx("'+aliasDict[c[0]]+' '+' '.join(c[1:])+'")' Lib\site-packages\zope\interface\exceptions.py:257-260: return '\n ' + '\n '.join( x._str_details.strip() if isinstance(x, _TargetInvalid) else str(x) for x in self.exceptions ) Lib\smtplib.py:537 and 545: optionlist = ' ' + ' '.join(options) Lib\unittest\case.py:1094-1096: diffMsg = '\n' + '\n'.join( difflib.ndiff(pprint.pformat(seq1).splitlines(), pprint.pformat(seq2).splitlines())) Lib\unittest\case.py:1207-1209: diff = ('\n' + '\n'.join(difflib.ndiff( pprint.pformat(d1).splitlines(), pprint.pformat(d2).splitlines()))) SEPARATOR ADDED AT END: Lib\distutils\command\config.py:303: body = "\n".join(body) + "\n" Lib\email\contentmanager.py:145: def embedded_body(lines): return linesep.join(lines) + linesep Lib\email\contentmanager.py:146: def normal_body(lines): return b'\n'.join(lines) + b'\n' Lib\email\policy.py:215: return name + ': ' + self.linesep.join(lines) + self.linesep Lib\email\_header_value_parser.py:2854: return policy.linesep.join(lines) + policy.linesep Lib\site-packages\numpy\distutils\command\config.py:346: body = '\n'.join(body) + "\n" Lib\site-packages\numpy\distutils\command\config.py:407: body = '\n'.join(body) + "\n" Lib\site-packages\oauthlib\oauth2\rfc6749\tokens.py:158: base_string = '\n'.join(base) + '\n' Lib\site-packages\PIL\ImageCms.py:770: return "\r\n\r\n".join(arr) + "\r\n\r\n" Lib\site-packages\pip\_internal\operations\freeze.py:254: return "\n".join(list(self.comments) + [str(req)]) + "\n" Lib\site-packages\pip\_internal\operations\install\legacy.py:54: f.write("\n".join(new_lines) + "\n") Lib\site-packages\pip\_vendor\pyparsing\testing.py:323-331: return ( header1 + header2 + "\n".join( "{:{}d}:{}{}".format(i, lineno_width, line, eol_mark) for i, line in enumerate(s_lines, start=start_line) ) + "\n" ) Lib\site-packages\pycparser\c_generator.py:117: if n.storage: s += ' '.join(n.storage) + ' ' Lib\site-packages\pycparser\c_generator.py:366: if n.funcspec: s = ' '.join(n.funcspec) + ' ' Lib\site-packages\pycparser\c_generator.py:367: if n.storage: s += ' '.join(n.storage) + ' ' Lib\site-packages\pycparser\c_generator.py:382: if n.quals: s += ' '.join(n.quals) + ' ' Lib\site-packages\pycparser\c_generator.py:397: nstr += ' '.join(modifier.dim_quals) + ' ' Lib\site-packages\pycparser\c_generator.py:417: return ' '.join(n.names) + ' ' Lib\site-packages\pythonwin\pywin\framework\scriptutils.py:109: return ".".join(modBits) + "." + fname, newPathReturn Lib\site-packages\reportlab\pdfbase\pdfdoc.py:1118: code = '\n'.join(code)+'\n' Lib\site-packages\reportlab\pdfbase\pdfutils.py:102: f.write('\r\n'.join(code)+'\r\n') Lib\site-packages\reportlab\pdfbase\_can_cmap_data.py:54: src = '\n'.join(buf) + '\n' Lib\site-packages\reportlab\pdfgen\pdfimages.py:203: content = '\n'.join(self.imageData[3:-1]) + '\n' Lib\site-packages\setuptools\command\build_ext.py:221: pkg = '.'.join(ext._full_name.split('.')[:-1] + ['']) Lib\site-packages\setuptools\command\easy_install.py:1056: f.write('\n'.join(locals()[name]) + '\n') Lib\site-packages\setuptools\command\easy_install.py:1606: data = '\n'.join(lines) + '\n' Lib\site-packages\setuptools\command\egg_info.py:672: cmd.write_file("top-level names", filename, '\n'.join(sorted(pkgs)) + '\n') Lib\site-packages\setuptools\command\egg_info.py:683: value = '\n'.join(value) + '\n' Lib\site-packages\setuptools\_distutils\command\config.py:303: body = "\n".join(body) + "\n" Lib\site-packages\twisted\conch\manhole.py:360-362: return (b'\n'.join(self.interpreter.buffer) + b'\n' + b''.join(self.lineBuffer)) Lib\site-packages\twisted\conch\client\knownhosts.py:547-549: hostsFileObj.write( b"\n".join([entry.toString() for entry in self._added]) + b"\n") Lib\site-packages\twisted\conch\ssh\keys.py:1340: return b'\n'.join(lines) + b'\n' Lib\site-packages\twisted\conch\test\test_conch.py:556: expectedResult = '\n'.join(['line #%02d' % (i,) for i in range(60)]) + '\n' Lib\site-packages\twisted\conch\test\test_helper.py:360: self.term.write(b'\n'.join((s1, s2, s3)) + b'\n') Lib\site-packages\twisted\internet\test\test_process.py:769: scriptFile.write(os.linesep.join(sourceLines) + os.linesep) Lib\site-packages\twisted\mail\imap4.py:5713: hdrs = '\r\n'.join(hdrs) + '\r\n' Lib\site-packages\twisted\mail\imap4.py:5952: base = b'.'.join([(x + 1).__bytes__() for x in self.part]) + b'.' + base Lib\site-packages\twisted\mail\test\test_pop3.py:312: self.message = b'\n'.join(self.lines) + b'\n' Lib\site-packages\twisted\mail\test\test_pop3.py:376: output = b'\r\n'.join(client.response) + b'\r\n' Lib\site-packages\twisted\mail\test\test_smtp.py:100: message = b'\n'.join(self.buffer) + b'\n' Lib\site-packages\twisted\mail\test\test_smtp.py:344: message = b'\n'.join(self.buffer) + b'\n' Lib\site-packages\twisted\python\text.py:146: return '\n'.join(lines)+'\n' Lib\site-packages\twisted\test\test_iutils.py:40: scriptFile.write(os.linesep.join(sourceLines) + os.linesep) Lib\site-packages\win32comext\adsi\demos\scp.py:350: description = __doc__ + "\ncommands:\n" + "\n".join(arg_descs) + "\n" Lib\site-packages\wx\py\crust.py:259: self.SetValue('\n'.join(hist) + '\n') Lib\site-packages\wx\py\introspect.py:342: command = terminator.join(pieces[:-1]) + terminator Lib\site-packages\zope\interface\document.py:78: return "\n\n".join(r) + "\n\n" Lib\test\test_nntplib.py:495: lit = "\r\n".join(lit.splitlines()) + "\r\n" Lib\test\test_univnewlines.py:24:DATA_LF = "\n".join(DATA_TEMPLATE) + "\n" Lib\test\test_univnewlines.py:25:DATA_CR = "\r".join(DATA_TEMPLATE) + "\r" Lib\test\test_univnewlines.py:26:DATA_CRLF = "\r\n".join(DATA_TEMPLATE) + "\r\n" Lib\test\test_tools\test_pindent.py:33: return '\n'.join(line.lstrip() for line in data.splitlines()) + '\n' SEPARATOR ADDED AT BOTH ENDS: Lib\pydoc.py:1582: sys.stdout.write('\n' + '\n'.join(lines[r:r+inc]) + '\n') Lib\site-packages\office365\runtime\odata\odata_batch_request.py:129: buffer = eol + eol.join(lines) + eol Lib\test\test_generators.py:1424: print("|" + "|".join(squares) + "|") Lib\test\test_generators.py:1620: print("|" + "|".join(row) + "|")

6 7

Native support of YAML in Python STDLIB
by scott.fields＠kyndryl.com 14 Mar '23

14 Mar '23

Though this has been discussed before, the last detailed discussion appears to have been roughly 9 years ago. YAML is a technical standard widely used in various projects and at least two major ones (Ansible and containers). Though PyYAML has become ubiquitous as the primary Python solution to supporting YAML, I would think it time for Python to formally adopt this technology in the core foundation. Is there still a reason why this isn't being pursued as native infrastructure in Python?

4 3

Ampersand operator for strings
by Rob Cliffe 12 Mar '23

12 Mar '23

Tl;dr: Join strings together with exactly one space between non-blank text where they join. I propose a meaning for s1 & s2 where s1 and s2 are strings. Namely, that it should be equivalent to s1.rstrip() + (' ' if (s1.strip() and s2.strip()) else '') + s2.lstrip() Informally, this will join the two strings together with exactly one space between the last non-blank text in s1 and the first non-blank text in s2. Example: " bar " & " foo " == " bar foo " This operator is associative, so there is no ambiguity in expressions such as s1 & s2 & s3 There *is* a possible ambiguity in expressions such as s1 & s2 + s3 where the relative precedence of `&` and `+` matters when s2 consists solely of white space. E.g. " A " & " " + " C" would evaluate to " A C" not to " A C" because `+` has a higher precedence than '&'. Utility: In decades of experience with another language which had such an operator (spelt differently) I have frequently found it useful for constructing human-readable output (e.g. log output, debug/error messages, on-screen labels). Cognitive burden: This would of course be one more thing to learn. But I suggest that it is fairly intuitive that s1 + s2 s1 & s2 both suggest that two strings are being combined in some way. Bars to overcome: This change would require no fundamental change to Python; just adding an `__and__ function to the str class. Backward compatibility: Given that `str1 & str2` currently raises TypeError, this change would be close to 100% backward-compatible. Alternative meanings: As far as I know nobody has ever suggested an alternative meaning for `&` on strings. Bikeshedding: (1) I don't think it is important for the utility of this change whether `&` strips off all whitespace, or just spaces. I think it is better if it strips off all whitespace, so that it can be understood as working similarly to strip(). (2) The definition could be simplified to s1.rstrip() + ' ' + s2.lstrip() (leave an initial/final space when one string is whitespace only). Again the operator would be associative. Again I don't think this is important. Rob Cliffe

10 19

Restricting access to sensitive APIs with a permission model like Deno
by python＠ruebezahl.mozmail.com 01 Mar '23

01 Mar '23

Hello all, Supply chain attacks are becoming a pressing concern in software development due to the large number of dependencies and multiple attack vectors. Using third party modules (libraries, packages etc) is always a risk but the true potential of these attacks is now being weaponized. One way to deal with the risk is by limiting access to sensitive APIs like filesystem, shell, network and ffi so that packages which aren't explicitly granted permissions cannot use them, reducing their ability to do damage. For example, a yaml parser should not need to use ffi, network nor shell. A command line argument parser library should not use network, ffi nor filesystem. Deno, a runtime for Typescript contains an interesting implementation of a permissions model for APIs. I strongly think Python could benefit from such functionality and hacked together a quick experiment here: https://github.com/R9295/cpython Currently, it only prevents module imports in a very elementary manner but perhaps it can be of use to spark a discussion for an implementation. Looking forward to your thoughts, Aarnav

10 27