Mailman 3 - Python-ideas - {{ block.super }

Bind/normalize params for @functools.cache
by Steve Jorgensen 20 Mar '23

20 Mar '23

I was surprised to find, when I pass arguments to a function decorated with `(a)functools.cache` in different, equivalent ways, the cache does not recognize them as the same. counter = itertools.count(1) @functools.cache def example(a, b, c=0): return (next(counter), a, b, c) example(1, 2) # => (1, 1, 2, 0) example(1, b=2) # => (2, 1, 2, 0) example(1, 2, 0) # => (3, 1, 2, 0) When I wrote my own implementation as a coding exercise, I noticed the same weakness while testing it and solved that by having the decorator function get the signature of the decorated function, then use the bind method of the signature to bind the parameter values, then call the apply_defaults method on the bound arguments, and then finally, use the args and kwargs properties of the bound arguments to make the cache key. It seems like functools.cache should do the same thing. If it is undesirable for that to be the default behavior, then it could be optional (e.g. @functools.cache(normalize=True) ). I have not tested to see if functools.lru_cache has the same issue. I presume that it does, so my suggestion would apply to that as well.

1 1

join() could add separators at start or end
by Rob Cliffe 14 Mar '23

14 Mar '23

Having had my last proposal shot down in flames, up I bob with another. 😁 It seems to me that it would be useful to be able to make the str.join() function put separators, not only between the items of its operand, but also optionally at the beginning or end. E.g. '|'.join(('Spam', 'Ham', 'Eggs')) returns 'Spam|Ham|Eggs' but it might be useful to make it return one of '|Spam|Ham|Eggs' 'Spam|Ham|Eggs|' '|Spam|Ham|Eggs|' Again, I suggest that this apply to byte strings as well as strings. Going through the 3.8.3 stdlib I have found 24 examples where the separator needs to be added at the beginning 52 where the separator needs to be added at the end 4 where the separator needs to be added at the both ends I list these examples below. Apologies if there are any mistakes. While guessing is no substitute for measurement, it seems plausible that using this feature where appropriate would increase runtime performance by avoiding 1 (or 2) calls of str.__add__. This is perhaps more relevant when the separator is not a short constant string, as in this example: Lib\email\_header_value_parser.py:2854: return policy.linesep.join(lines) + policy.linesep Note also this example: Lib\site-packages\setuptools\command\build_ext.py:221: pkg = '.'.join(ext._full_name.split('.')[:-1] + ['']) where the author has used the unintuitive device of appending an empty string to a list to force join() to add an extra final dot, thereby avoiding 1 call of str.__add__ at the cost of 1 call of list.append. What I cannot decide is what the best API would be. str.join() currently takes only 1 parameter, so it would be possible to add an extra parameter or two. One scheme would be to have an atEnds parameter which could take values such as 0=default behaviour 1=add sep at start 2=add sep at end 3=add sep at both ends or 's'=add sep at start 'e'=add sep at end 'b'=add sep at both ends (some) other=default behaviour Another would be to have 2 parameters, atStart and atEnd, which would both default to False or 0. E.g. '|'.join(('Spam', 'Ham', 'Eggs'), 1) == '|Spam|Ham|Eggs' '|'.join(('Spam', 'Ham', 'Eggs'), 0, 1) == 'Spam|Ham|Eggs|' Neither scheme results in particularly transparent usage, though no worse than s.splitlines(True) # What on earth is this parameter??? Corner case: What if join() is passed an empty sequence? This is debatable, but I think it should return the separator if requested to add it at the beginning or end, and double it up if both are requested. This would preserve identities such as sep.join(seq, <PleaseAddSeparatorAtBeginning>) == sep + sep.join(seq) Best wishes Rob Cliffe EXAMPLES WHERE SEPARATOR ADDED AT START: Lib\http\server.py:933: splitpath = ('/' + '/'.join(head_parts), tail_part) Lib\site-packages\numpy\ctypeslib.py:333: name += "_"+"_".join(flags) Lib\site-packages\numpy\testing\_private\utils.py:842: err_msg += '\n' + '\n'.join(remarks) Lib\site-packages\pip\_vendor\pyparsing\core.py:2092-2095: out = [ "\n" + "\n".join(comments) if comments else "", pyparsing_test.with_line_numbers(t) if with_line_numbers else t, ] Lib\site-packages\pip\_vendor\requests\status_codes.py:121-125: __doc__ = ( __doc__ + "\n" + "\n".join(doc(code) for code in sorted(_codes)) if __doc__ is not None else None ) Lib\site-packages\reportlab\lib\utils.py:1093: self._writeln(' '+' '.join(A.__self__)) Lib\site-packages\reportlab\platypus\flowables.py:708: L = "\n"+"\n".join(L) Lib\site-packages\twisted\mail\smtp.py:1647: r.append(c + b' ' + b' '.join(v)) Lib\site-packages\twisted\protocols\ftp.py:1203: return (PWD_REPLY, '/' + '/'.join(self.workingDirectory)) Lib\site-packages\twisted\runner\procmon.py:424-426: return ('<' + self.__class__.__name__ + ' ' + ' '.join(l) + '>') Lib\site-packages\twisted\web\rewrite.py:34: request.path = '/'+'/'.join(request.prepath+request.postpath) Lib\site-packages\twisted\web\rewrite.py:51: request.path = '/'+'/'.join(request.prepath+request.postpath) Lib\site-packages\twisted\web\twcgi.py:78: scriptName = b"/" + b"/".join(request.prepath) Lib\site-packages\twisted\web\twcgi.py:95: env["PATH_INFO"] = "/" + "/".join(pp) Lib\site-packages\twisted\web\vhost.py:115: request.path = b'/' + b'/'.join(request.postpath) Lib\site-packages\twisted\web\wsgi.py:283: scriptName = b'/' + b'/'.join(request.prepath) Lib\site-packages\twisted\web\wsgi.py:288: pathInfo = b'/' + b'/'.join(request.postpath) Lib\site-packages\twisted\web\test\test_wsgi.py:272: uri = '/' + '/'.join([urlquote(seg, safe) for seg in requestSegments]) Lib\site-packages\wx\py\magic.py:55: command = 'sx("'+aliasDict[c[0]]+' '+' '.join(c[1:])+'")' Lib\site-packages\zope\interface\exceptions.py:257-260: return '\n ' + '\n '.join( x._str_details.strip() if isinstance(x, _TargetInvalid) else str(x) for x in self.exceptions ) Lib\smtplib.py:537 and 545: optionlist = ' ' + ' '.join(options) Lib\unittest\case.py:1094-1096: diffMsg = '\n' + '\n'.join( difflib.ndiff(pprint.pformat(seq1).splitlines(), pprint.pformat(seq2).splitlines())) Lib\unittest\case.py:1207-1209: diff = ('\n' + '\n'.join(difflib.ndiff( pprint.pformat(d1).splitlines(), pprint.pformat(d2).splitlines()))) SEPARATOR ADDED AT END: Lib\distutils\command\config.py:303: body = "\n".join(body) + "\n" Lib\email\contentmanager.py:145: def embedded_body(lines): return linesep.join(lines) + linesep Lib\email\contentmanager.py:146: def normal_body(lines): return b'\n'.join(lines) + b'\n' Lib\email\policy.py:215: return name + ': ' + self.linesep.join(lines) + self.linesep Lib\email\_header_value_parser.py:2854: return policy.linesep.join(lines) + policy.linesep Lib\site-packages\numpy\distutils\command\config.py:346: body = '\n'.join(body) + "\n" Lib\site-packages\numpy\distutils\command\config.py:407: body = '\n'.join(body) + "\n" Lib\site-packages\oauthlib\oauth2\rfc6749\tokens.py:158: base_string = '\n'.join(base) + '\n' Lib\site-packages\PIL\ImageCms.py:770: return "\r\n\r\n".join(arr) + "\r\n\r\n" Lib\site-packages\pip\_internal\operations\freeze.py:254: return "\n".join(list(self.comments) + [str(req)]) + "\n" Lib\site-packages\pip\_internal\operations\install\legacy.py:54: f.write("\n".join(new_lines) + "\n") Lib\site-packages\pip\_vendor\pyparsing\testing.py:323-331: return ( header1 + header2 + "\n".join( "{:{}d}:{}{}".format(i, lineno_width, line, eol_mark) for i, line in enumerate(s_lines, start=start_line) ) + "\n" ) Lib\site-packages\pycparser\c_generator.py:117: if n.storage: s += ' '.join(n.storage) + ' ' Lib\site-packages\pycparser\c_generator.py:366: if n.funcspec: s = ' '.join(n.funcspec) + ' ' Lib\site-packages\pycparser\c_generator.py:367: if n.storage: s += ' '.join(n.storage) + ' ' Lib\site-packages\pycparser\c_generator.py:382: if n.quals: s += ' '.join(n.quals) + ' ' Lib\site-packages\pycparser\c_generator.py:397: nstr += ' '.join(modifier.dim_quals) + ' ' Lib\site-packages\pycparser\c_generator.py:417: return ' '.join(n.names) + ' ' Lib\site-packages\pythonwin\pywin\framework\scriptutils.py:109: return ".".join(modBits) + "." + fname, newPathReturn Lib\site-packages\reportlab\pdfbase\pdfdoc.py:1118: code = '\n'.join(code)+'\n' Lib\site-packages\reportlab\pdfbase\pdfutils.py:102: f.write('\r\n'.join(code)+'\r\n') Lib\site-packages\reportlab\pdfbase\_can_cmap_data.py:54: src = '\n'.join(buf) + '\n' Lib\site-packages\reportlab\pdfgen\pdfimages.py:203: content = '\n'.join(self.imageData[3:-1]) + '\n' Lib\site-packages\setuptools\command\build_ext.py:221: pkg = '.'.join(ext._full_name.split('.')[:-1] + ['']) Lib\site-packages\setuptools\command\easy_install.py:1056: f.write('\n'.join(locals()[name]) + '\n') Lib\site-packages\setuptools\command\easy_install.py:1606: data = '\n'.join(lines) + '\n' Lib\site-packages\setuptools\command\egg_info.py:672: cmd.write_file("top-level names", filename, '\n'.join(sorted(pkgs)) + '\n') Lib\site-packages\setuptools\command\egg_info.py:683: value = '\n'.join(value) + '\n' Lib\site-packages\setuptools\_distutils\command\config.py:303: body = "\n".join(body) + "\n" Lib\site-packages\twisted\conch\manhole.py:360-362: return (b'\n'.join(self.interpreter.buffer) + b'\n' + b''.join(self.lineBuffer)) Lib\site-packages\twisted\conch\client\knownhosts.py:547-549: hostsFileObj.write( b"\n".join([entry.toString() for entry in self._added]) + b"\n") Lib\site-packages\twisted\conch\ssh\keys.py:1340: return b'\n'.join(lines) + b'\n' Lib\site-packages\twisted\conch\test\test_conch.py:556: expectedResult = '\n'.join(['line #%02d' % (i,) for i in range(60)]) + '\n' Lib\site-packages\twisted\conch\test\test_helper.py:360: self.term.write(b'\n'.join((s1, s2, s3)) + b'\n') Lib\site-packages\twisted\internet\test\test_process.py:769: scriptFile.write(os.linesep.join(sourceLines) + os.linesep) Lib\site-packages\twisted\mail\imap4.py:5713: hdrs = '\r\n'.join(hdrs) + '\r\n' Lib\site-packages\twisted\mail\imap4.py:5952: base = b'.'.join([(x + 1).__bytes__() for x in self.part]) + b'.' + base Lib\site-packages\twisted\mail\test\test_pop3.py:312: self.message = b'\n'.join(self.lines) + b'\n' Lib\site-packages\twisted\mail\test\test_pop3.py:376: output = b'\r\n'.join(client.response) + b'\r\n' Lib\site-packages\twisted\mail\test\test_smtp.py:100: message = b'\n'.join(self.buffer) + b'\n' Lib\site-packages\twisted\mail\test\test_smtp.py:344: message = b'\n'.join(self.buffer) + b'\n' Lib\site-packages\twisted\python\text.py:146: return '\n'.join(lines)+'\n' Lib\site-packages\twisted\test\test_iutils.py:40: scriptFile.write(os.linesep.join(sourceLines) + os.linesep) Lib\site-packages\win32comext\adsi\demos\scp.py:350: description = __doc__ + "\ncommands:\n" + "\n".join(arg_descs) + "\n" Lib\site-packages\wx\py\crust.py:259: self.SetValue('\n'.join(hist) + '\n') Lib\site-packages\wx\py\introspect.py:342: command = terminator.join(pieces[:-1]) + terminator Lib\site-packages\zope\interface\document.py:78: return "\n\n".join(r) + "\n\n" Lib\test\test_nntplib.py:495: lit = "\r\n".join(lit.splitlines()) + "\r\n" Lib\test\test_univnewlines.py:24:DATA_LF = "\n".join(DATA_TEMPLATE) + "\n" Lib\test\test_univnewlines.py:25:DATA_CR = "\r".join(DATA_TEMPLATE) + "\r" Lib\test\test_univnewlines.py:26:DATA_CRLF = "\r\n".join(DATA_TEMPLATE) + "\r\n" Lib\test\test_tools\test_pindent.py:33: return '\n'.join(line.lstrip() for line in data.splitlines()) + '\n' SEPARATOR ADDED AT BOTH ENDS: Lib\pydoc.py:1582: sys.stdout.write('\n' + '\n'.join(lines[r:r+inc]) + '\n') Lib\site-packages\office365\runtime\odata\odata_batch_request.py:129: buffer = eol + eol.join(lines) + eol Lib\test\test_generators.py:1424: print("|" + "|".join(squares) + "|") Lib\test\test_generators.py:1620: print("|" + "|".join(row) + "|")

6 7

Native support of YAML in Python STDLIB
by scott.fields＠kyndryl.com 14 Mar '23

14 Mar '23

Though this has been discussed before, the last detailed discussion appears to have been roughly 9 years ago. YAML is a technical standard widely used in various projects and at least two major ones (Ansible and containers). Though PyYAML has become ubiquitous as the primary Python solution to supporting YAML, I would think it time for Python to formally adopt this technology in the core foundation. Is there still a reason why this isn't being pursued as native infrastructure in Python?

4 3

Ampersand operator for strings
by Rob Cliffe 13 Mar '23

13 Mar '23

Tl;dr: Join strings together with exactly one space between non-blank text where they join. I propose a meaning for s1 & s2 where s1 and s2 are strings. Namely, that it should be equivalent to s1.rstrip() + (' ' if (s1.strip() and s2.strip()) else '') + s2.lstrip() Informally, this will join the two strings together with exactly one space between the last non-blank text in s1 and the first non-blank text in s2. Example: " bar " & " foo " == " bar foo " This operator is associative, so there is no ambiguity in expressions such as s1 & s2 & s3 There *is* a possible ambiguity in expressions such as s1 & s2 + s3 where the relative precedence of `&` and `+` matters when s2 consists solely of white space. E.g. " A " & " " + " C" would evaluate to " A C" not to " A C" because `+` has a higher precedence than '&'. Utility: In decades of experience with another language which had such an operator (spelt differently) I have frequently found it useful for constructing human-readable output (e.g. log output, debug/error messages, on-screen labels). Cognitive burden: This would of course be one more thing to learn. But I suggest that it is fairly intuitive that s1 + s2 s1 & s2 both suggest that two strings are being combined in some way. Bars to overcome: This change would require no fundamental change to Python; just adding an `__and__ function to the str class. Backward compatibility: Given that `str1 & str2` currently raises TypeError, this change would be close to 100% backward-compatible. Alternative meanings: As far as I know nobody has ever suggested an alternative meaning for `&` on strings. Bikeshedding: (1) I don't think it is important for the utility of this change whether `&` strips off all whitespace, or just spaces. I think it is better if it strips off all whitespace, so that it can be understood as working similarly to strip(). (2) The definition could be simplified to s1.rstrip() + ' ' + s2.lstrip() (leave an initial/final space when one string is whitespace only). Again the operator would be associative. Again I don't think this is important. Rob Cliffe

10 19

Restricting access to sensitive APIs with a permission model like Deno
by python＠ruebezahl.mozmail.com 01 Mar '23

01 Mar '23

Hello all, Supply chain attacks are becoming a pressing concern in software development due to the large number of dependencies and multiple attack vectors. Using third party modules (libraries, packages etc) is always a risk but the true potential of these attacks is now being weaponized. One way to deal with the risk is by limiting access to sensitive APIs like filesystem, shell, network and ffi so that packages which aren't explicitly granted permissions cannot use them, reducing their ability to do damage. For example, a yaml parser should not need to use ffi, network nor shell. A command line argument parser library should not use network, ffi nor filesystem. Deno, a runtime for Typescript contains an interesting implementation of a permissions model for APIs. I strongly think Python could benefit from such functionality and hacked together a quick experiment here: https://github.com/R9295/cpython Currently, it only prevents module imports in a very elementary manner but perhaps it can be of use to spark a discussion for an implementation. Looking forward to your thoughts, Aarnav

10 27

Proposal: -X importcache to supplement -X importtime for loaded modules
by Noah Kim 27 Feb '23

27 Feb '23

All, I'm writing to propose an adjacent interpreter flag to `-X importtime`: `-X importcache` (open to alternative naming). While `-X importtime` is incredibly useful for analyzing module import times, by design, it doesn't log anything if an imported module has already been loaded. `-X importcache` would provide additional output for every module that's already been loaded: ``` >>> import uuid import time: cached | cached | _io import time: cached | cached | _io import time: cached | cached | os import time: cached | cached | sys import time: cached | cached | enum import time: cached | cached | _io import time: cached | cached | _io import time: cached | cached | collections import time: cached | cached | os import time: cached | cached | re import time: cached | cached | sys import time: cached | cached | functools import time: cached | cached | itertools import time: 151 | 151 | _wmi import time: 18290 | 18440 | platform import time: 372 | 372 | _uuid import time: 10955 | 29766 | uuid ``` In codebases with convoluted/poorly managed import graphs (and consequently, workloads that suffer from long import times), the ability to record all paths to an expensive dependency--not just the first-imported--can help expedite refactoring (and help scale identification of this type of issue). More generally, this flag would provide a more efficient path to tracking runtime dependencies. As a proof of concept, I was able to hack this functionality into `-X importtime` by adding a couple lines to `import_ensure_initialized` in `Python/import.c` (hence the output above). A separate flag is probably desirable to preserve backwards compatibility. Looking forward to your feedback, Noah

5 6

Combinations of keywords
by Otomatyk dupont 21 Feb '23

21 Feb '23

Hello, here is my ideas of new keywords combinations : (the code is writed like this #code#. Sorry if I do some english mistakes) 1 - and pass if except This one is just a shortcut of # try : code here except : do something # Its syntaxe is #CODE and DO_SOMETHING if except#, it's more readable i think. It is usefull for code like this : #import A_WINDOW_MODULE and import A_UNIX_MODULE if except If DO_SOMETHING is replaced by #pass# it just skips the ligne (and the error) 2 - CODE if CONDITION else CODE I would like the #VALUE if CONDITION else OTHER_VALUE" can also executes code. Like : #Sign_in(user) if button_i_dont have_a_acount.press() else Log_in(user)# or #break if quit==True# If the #else# isnt present and the condition result is False, the interpreter don't executes that ligne What do you think about ?

5 4

Deprecate misleading escapes in strings
by Arusekk 17 Feb '23

17 Feb '23

Hi all! I was writing a tutorial on the distinction between bytes and strings and why it is important, when I saw the root cause. People coming from C, Perl, Python 2 and similar languages tend to misinterpret "\x90" for b"\x90" often. My idea is that Python could deprecate string literals containing any non-ASCII codepoints specified in any way different from unicode or unicode escapes (\u, \U, \N). (Actually I found that I started having the idea already back in 2021 on StackOverflow[1]. The question is an excellent example of what I mean.) I would not go so far to follow JSON (disallowing \x11 and \222 escapes completely), but while writing "\x00" or "\0" is useful and widely used, "\x99" (and especially "\777"!) is probably marginal and definitely less explicit than "\u0099" (in the Zen of explicit better than implicit). Byte strings do not treat b"\u00ff" as b"\xff". In the first part of implementing it, Python could raise a SyntaxWarning (or should it be DeprecationWarning? BytesWarning?), suggesting "\x99" to either become b"\u0099" or b"\x99", eventually promoting it to some equally helpful SyntaxError. All of it could be hidden behind a feature like from __future__ import backslashes (one nice name I can think of). The new regular expression for octals would be \\[01]?[0-7]{1,2} and \\x[0-7][0-9A-Fa-f] for hexadecimals, hopefully not confusing anyone, and not much more complex than the old ones. In the meantime, probably between introducing a warning and changing it to become an error (the most reasonable timeline I can think of now), the default ascii() representation should eventually use the \u0099 form for all such codepoints, to keep the invariant of eval(ascii(x)) == x without syntax warnings. repr() is also affected, but it is fortunately limited to the [\x80-\xa0\xad] range. I mean [\u0080-\u00a0\u00ad] :-) Another timeline would be to change the repr first, initially hidden under an interpreter flag or environment variable, then officially deprecate it in the documentation, then introduce the error guarded by from __future__ import backslashes or another flag, then make the repr use \u by default, then add the warning and finally make it always raise an error. As a precedent, breaking repr() was not a dealbreaker when introducing randomized seeds (even repr({"a", "b"}) is now unpredictable). This would be of course a breaking change for a lot of unit tests, and stuff like pickle should probably support old syntax, delaying any such change until a new protocol comes (if it applies to the newest one --- quite sure it does not). Such a breaking change must be used wisely. Other changes to octal escapes could be sneaked in, based on conclusions from the 2018 'Python octal escape character encoding "wats"' thread[2] (I like writing "\0" and "\4" though, just to make my opinion clear). If going the whole hog, the 2015 'Make non-meaningful backslashes illegal in string literals' thread[3] could be revived as well, maybe even with "\f\v" deprecated, "\e" = "\33" introduced and such. Please let me know what you think, what else could break, and is it useful anywhere else apart from my use case, and what similar problems you have. Cheers, Arusekk [1]: https://stackoverflow.com/q/64832281/3869724 [2]: https://mail.python.org/archives/list/python-ideas@python.org/thread/ARBCIP… [3]: https://mail.python.org/archives/list/python-ideas@python.org/message/PJXKD…

6 7

timedelta.strptime() and timedelta.strftime()
by Thomas Mc Kay 03 Feb '23

03 Feb '23

I was looking for a simple intuitive way to parse a timedelta from a string. The best I could find feels clunky at best: https://stackoverflow.com/questions/4628122/ Solution 1 looks like this: ``` from datetime import datetime, timedelta t = datetime.strptime('05:20:25', '%H:%M:%S') delta = timedelta(hours=t.hour, minutes=t.minute, seconds=t.second) ``` Solution 2 looks like this: ``` from datetime import datetime, timedelta t = datetime.strptime('05:20:25', '%H:%M:%S') delta = t - datetime.combine(t.date(), time.min) ``` Solution 3 looks like this: ``` from datetime import timedelta import re match = re.match(r'(?P<hours>\d{2}):(?P<minutes>\d{2}):(?P<seconds>\d{2})', '05:20:25') delta = timedelta(**match.groupdict()) ``` Formatting back to strings is as verbose: ``` string = f'{delta.hours:02d}:{delta.minutes:02d}:{delta.seconds:02d}' I think it would be nicer and more intuitive if timedelta had strptime and strftime like datetime has. ``` delta = timedelta.strptime('05:20:25', '%H:%M:%S') # timedelta(hours=5, minutes=20, seconds=25) string = delta.strftime('%H:%M:%S') # '05:20:25' ``` The reason I would like this in the standard lib (vs maintaining my own helpers) is that : a) it feels very natural, especially after years of using these methods on datetime. b) there's currently three ways to do it that I've seen in the wild (documented above). Having this in the std lib would help with the ZoP "There should be one-- and preferably only one --obvious way to do it." (the "obvious" part of this statement is covered in (a)). I'm sure there's corner cases I'm not seeing that make this more complicated to implement than it seems... Let me know. Cheers, Thomas

2 2

Re: Multiple arguments to str.partition and bytes.partition
by Peter Ludemann 10 Jan '23

10 Jan '23

You can get almost the same result using pattern matching. For example, your "foo:bar;baz".partition(":", ";") can be done by a well-known matching idiom: re.match(r'([^:]*):([^;]*);(.*)', 'foo:bar;baz').groups()

4 6