Mailman 3 - Python-ideas

join() could add separators at start or end
by Rob Cliffe March 14, 2023

March 14, 2023

Having had my last proposal shot down in flames, up I bob with another. 😁 It seems to me that it would be useful to be able to make the str.join() function put separators, not only between the items of its operand, but also optionally at the beginning or end. E.g. '|'.join(('Spam', 'Ham', 'Eggs')) returns 'Spam|Ham|Eggs' but it might be useful to make it return one of '|Spam|Ham|Eggs' 'Spam|Ham|Eggs|' '|Spam|Ham|… [View More]Eggs|' Again, I suggest that this apply to byte strings as well as strings. Going through the 3.8.3 stdlib I have found 24 examples where the separator needs to be added at the beginning 52 where the separator needs to be added at the end 4 where the separator needs to be added at the both ends I list these examples below. Apologies if there are any mistakes. While guessing is no substitute for measurement, it seems plausible that using this feature where appropriate would increase runtime performance by avoiding 1 (or 2) calls of str.__add__. This is perhaps more relevant when the separator is not a short constant string, as in this example: Lib\email\_header_value_parser.py:2854: return policy.linesep.join(lines) + policy.linesep Note also this example: Lib\site-packages\setuptools\command\build_ext.py:221: pkg = '.'.join(ext._full_name.split('.')[:-1] + ['']) where the author has used the unintuitive device of appending an empty string to a list to force join() to add an extra final dot, thereby avoiding 1 call of str.__add__ at the cost of 1 call of list.append. What I cannot decide is what the best API would be. str.join() currently takes only 1 parameter, so it would be possible to add an extra parameter or two. One scheme would be to have an atEnds parameter which could take values such as 0=default behaviour 1=add sep at start 2=add sep at end 3=add sep at both ends or 's'=add sep at start 'e'=add sep at end 'b'=add sep at both ends (some) other=default behaviour Another would be to have 2 parameters, atStart and atEnd, which would both default to False or 0. E.g. '|'.join(('Spam', 'Ham', 'Eggs'), 1) == '|Spam|Ham|Eggs' '|'.join(('Spam', 'Ham', 'Eggs'), 0, 1) == 'Spam|Ham|Eggs|' Neither scheme results in particularly transparent usage, though no worse than s.splitlines(True) # What on earth is this parameter??? Corner case: What if join() is passed an empty sequence? This is debatable, but I think it should return the separator if requested to add it at the beginning or end, and double it up if both are requested. This would preserve identities such as sep.join(seq, <PleaseAddSeparatorAtBeginning>) == sep + sep.join(seq) Best wishes Rob Cliffe EXAMPLES WHERE SEPARATOR ADDED AT START: Lib\http\server.py:933: splitpath = ('/' + '/'.join(head_parts), tail_part) Lib\site-packages\numpy\ctypeslib.py:333: name += "_"+"_".join(flags) Lib\site-packages\numpy\testing\_private\utils.py:842: err_msg += '\n' + '\n'.join(remarks) Lib\site-packages\pip\_vendor\pyparsing\core.py:2092-2095: out = [ "\n" + "\n".join(comments) if comments else "", pyparsing_test.with_line_numbers(t) if with_line_numbers else t, ] Lib\site-packages\pip\_vendor\requests\status_codes.py:121-125: __doc__ = ( __doc__ + "\n" + "\n".join(doc(code) for code in sorted(_codes)) if __doc__ is not None else None ) Lib\site-packages\reportlab\lib\utils.py:1093: self._writeln(' '+' '.join(A.__self__)) Lib\site-packages\reportlab\platypus\flowables.py:708: L = "\n"+"\n".join(L) Lib\site-packages\twisted\mail\smtp.py:1647: r.append(c + b' ' + b' '.join(v)) Lib\site-packages\twisted\protocols\ftp.py:1203: return (PWD_REPLY, '/' + '/'.join(self.workingDirectory)) Lib\site-packages\twisted\runner\procmon.py:424-426: return ('<' + self.__class__.__name__ + ' ' + ' '.join(l) + '>') Lib\site-packages\twisted\web\rewrite.py:34: request.path = '/'+'/'.join(request.prepath+request.postpath) Lib\site-packages\twisted\web\rewrite.py:51: request.path = '/'+'/'.join(request.prepath+request.postpath) Lib\site-packages\twisted\web\twcgi.py:78: scriptName = b"/" + b"/".join(request.prepath) Lib\site-packages\twisted\web\twcgi.py:95: env["PATH_INFO"] = "/" + "/".join(pp) Lib\site-packages\twisted\web\vhost.py:115: request.path = b'/' + b'/'.join(request.postpath) Lib\site-packages\twisted\web\wsgi.py:283: scriptName = b'/' + b'/'.join(request.prepath) Lib\site-packages\twisted\web\wsgi.py:288: pathInfo = b'/' + b'/'.join(request.postpath) Lib\site-packages\twisted\web\test\test_wsgi.py:272: uri = '/' + '/'.join([urlquote(seg, safe) for seg in requestSegments]) Lib\site-packages\wx\py\magic.py:55: command = 'sx("'+aliasDict[c[0]]+' '+' '.join(c[1:])+'")' Lib\site-packages\zope\interface\exceptions.py:257-260: return '\n ' + '\n '.join( x._str_details.strip() if isinstance(x, _TargetInvalid) else str(x) for x in self.exceptions ) Lib\smtplib.py:537 and 545: optionlist = ' ' + ' '.join(options) Lib\unittest\case.py:1094-1096: diffMsg = '\n' + '\n'.join( difflib.ndiff(pprint.pformat(seq1).splitlines(), pprint.pformat(seq2).splitlines())) Lib\unittest\case.py:1207-1209: diff = ('\n' + '\n'.join(difflib.ndiff( pprint.pformat(d1).splitlines(), pprint.pformat(d2).splitlines()))) SEPARATOR ADDED AT END: Lib\distutils\command\config.py:303: body = "\n".join(body) + "\n" Lib\email\contentmanager.py:145: def embedded_body(lines): return linesep.join(lines) + linesep Lib\email\contentmanager.py:146: def normal_body(lines): return b'\n'.join(lines) + b'\n' Lib\email\policy.py:215: return name + ': ' + self.linesep.join(lines) + self.linesep Lib\email\_header_value_parser.py:2854: return policy.linesep.join(lines) + policy.linesep Lib\site-packages\numpy\distutils\command\config.py:346: body = '\n'.join(body) + "\n" Lib\site-packages\numpy\distutils\command\config.py:407: body = '\n'.join(body) + "\n" Lib\site-packages\oauthlib\oauth2\rfc6749\tokens.py:158: base_string = '\n'.join(base) + '\n' Lib\site-packages\PIL\ImageCms.py:770: return "\r\n\r\n".join(arr) + "\r\n\r\n" Lib\site-packages\pip\_internal\operations\freeze.py:254: return "\n".join(list(self.comments) + [str(req)]) + "\n" Lib\site-packages\pip\_internal\operations\install\legacy.py:54: f.write("\n".join(new_lines) + "\n") Lib\site-packages\pip\_vendor\pyparsing\testing.py:323-331: return ( header1 + header2 + "\n".join( "{:{}d}:{}{}".format(i, lineno_width, line, eol_mark) for i, line in enumerate(s_lines, start=start_line) ) + "\n" ) Lib\site-packages\pycparser\c_generator.py:117: if n.storage: s += ' '.join(n.storage) + ' ' Lib\site-packages\pycparser\c_generator.py:366: if n.funcspec: s = ' '.join(n.funcspec) + ' ' Lib\site-packages\pycparser\c_generator.py:367: if n.storage: s += ' '.join(n.storage) + ' ' Lib\site-packages\pycparser\c_generator.py:382: if n.quals: s += ' '.join(n.quals) + ' ' Lib\site-packages\pycparser\c_generator.py:397: nstr += ' '.join(modifier.dim_quals) + ' ' Lib\site-packages\pycparser\c_generator.py:417: return ' '.join(n.names) + ' ' Lib\site-packages\pythonwin\pywin\framework\scriptutils.py:109: return ".".join(modBits) + "." + fname, newPathReturn Lib\site-packages\reportlab\pdfbase\pdfdoc.py:1118: code = '\n'.join(code)+'\n' Lib\site-packages\reportlab\pdfbase\pdfutils.py:102: f.write('\r\n'.join(code)+'\r\n') Lib\site-packages\reportlab\pdfbase\_can_cmap_data.py:54: src = '\n'.join(buf) + '\n' Lib\site-packages\reportlab\pdfgen\pdfimages.py:203: content = '\n'.join(self.imageData[3:-1]) + '\n' Lib\site-packages\setuptools\command\build_ext.py:221: pkg = '.'.join(ext._full_name.split('.')[:-1] + ['']) Lib\site-packages\setuptools\command\easy_install.py:1056: f.write('\n'.join(locals()[name]) + '\n') Lib\site-packages\setuptools\command\easy_install.py:1606: data = '\n'.join(lines) + '\n' Lib\site-packages\setuptools\command\egg_info.py:672: cmd.write_file("top-level names", filename, '\n'.join(sorted(pkgs)) + '\n') Lib\site-packages\setuptools\command\egg_info.py:683: value = '\n'.join(value) + '\n' Lib\site-packages\setuptools\_distutils\command\config.py:303: body = "\n".join(body) + "\n" Lib\site-packages\twisted\conch\manhole.py:360-362: return (b'\n'.join(self.interpreter.buffer) + b'\n' + b''.join(self.lineBuffer)) Lib\site-packages\twisted\conch\client\knownhosts.py:547-549: hostsFileObj.write( b"\n".join([entry.toString() for entry in self._added]) + b"\n") Lib\site-packages\twisted\conch\ssh\keys.py:1340: return b'\n'.join(lines) + b'\n' Lib\site-packages\twisted\conch\test\test_conch.py:556: expectedResult = '\n'.join(['line #%02d' % (i,) for i in range(60)]) + '\n' Lib\site-packages\twisted\conch\test\test_helper.py:360: self.term.write(b'\n'.join((s1, s2, s3)) + b'\n') Lib\site-packages\twisted\internet\test\test_process.py:769: scriptFile.write(os.linesep.join(sourceLines) + os.linesep) Lib\site-packages\twisted\mail\imap4.py:5713: hdrs = '\r\n'.join(hdrs) + '\r\n' Lib\site-packages\twisted\mail\imap4.py:5952: base = b'.'.join([(x + 1).__bytes__() for x in self.part]) + b'.' + base Lib\site-packages\twisted\mail\test\test_pop3.py:312: self.message = b'\n'.join(self.lines) + b'\n' Lib\site-packages\twisted\mail\test\test_pop3.py:376: output = b'\r\n'.join(client.response) + b'\r\n' Lib\site-packages\twisted\mail\test\test_smtp.py:100: message = b'\n'.join(self.buffer) + b'\n' Lib\site-packages\twisted\mail\test\test_smtp.py:344: message = b'\n'.join(self.buffer) + b'\n' Lib\site-packages\twisted\python\text.py:146: return '\n'.join(lines)+'\n' Lib\site-packages\twisted\test\test_iutils.py:40: scriptFile.write(os.linesep.join(sourceLines) + os.linesep) Lib\site-packages\win32comext\adsi\demos\scp.py:350: description = __doc__ + "\ncommands:\n" + "\n".join(arg_descs) + "\n" Lib\site-packages\wx\py\crust.py:259: self.SetValue('\n'.join(hist) + '\n') Lib\site-packages\wx\py\introspect.py:342: command = terminator.join(pieces[:-1]) + terminator Lib\site-packages\zope\interface\document.py:78: return "\n\n".join(r) + "\n\n" Lib\test\test_nntplib.py:495: lit = "\r\n".join(lit.splitlines()) + "\r\n" Lib\test\test_univnewlines.py:24:DATA_LF = "\n".join(DATA_TEMPLATE) + "\n" Lib\test\test_univnewlines.py:25:DATA_CR = "\r".join(DATA_TEMPLATE) + "\r" Lib\test\test_univnewlines.py:26:DATA_CRLF = "\r\n".join(DATA_TEMPLATE) + "\r\n" Lib\test\test_tools\test_pindent.py:33: return '\n'.join(line.lstrip() for line in data.splitlines()) + '\n' SEPARATOR ADDED AT BOTH ENDS: Lib\pydoc.py:1582: sys.stdout.write('\n' + '\n'.join(lines[r:r+inc]) + '\n') Lib\site-packages\office365\runtime\odata\odata_batch_request.py:129: buffer = eol + eol.join(lines) + eol Lib\test\test_generators.py:1424: print("|" + "|".join(squares) + "|") Lib\test\test_generators.py:1620: print("|" + "|".join(row) + "|") [View Less]

6 7

Native support of YAML in Python STDLIB
by scott.fields＠kyndryl.com March 13, 2023

March 13, 2023

Though this has been discussed before, the last detailed discussion appears to have been roughly 9 years ago. YAML is a technical standard widely used in various projects and at least two major ones (Ansible and containers). Though PyYAML has become ubiquitous as the primary Python solution to supporting YAML, I would think it time for Python to formally adopt this technology in the core foundation. Is there still a reason why this isn't being pursued as native infrastructure in Python?

4 3

Ampersand operator for strings
by Rob Cliffe March 12, 2023

March 12, 2023

Tl;dr: Join strings together with exactly one space between non-blank text where they join. I propose a meaning for s1 & s2 where s1 and s2 are strings. Namely, that it should be equivalent to s1.rstrip() + (' ' if (s1.strip() and s2.strip()) else '') + s2.lstrip() Informally, this will join the two strings together with exactly one space between the last non-blank text in s1 and the first non-blank text in s2. Example: " bar … [View More] " & " foo " == " bar foo " This operator is associative, so there is no ambiguity in expressions such as s1 & s2 & s3 There *is* a possible ambiguity in expressions such as s1 & s2 + s3 where the relative precedence of `&` and `+` matters when s2 consists solely of white space. E.g. " A " & " " + " C" would evaluate to " A C" not to " A C" because `+` has a higher precedence than '&'. Utility: In decades of experience with another language which had such an operator (spelt differently) I have frequently found it useful for constructing human-readable output (e.g. log output, debug/error messages, on-screen labels). Cognitive burden: This would of course be one more thing to learn. But I suggest that it is fairly intuitive that s1 + s2 s1 & s2 both suggest that two strings are being combined in some way. Bars to overcome: This change would require no fundamental change to Python; just adding an `__and__ function to the str class. Backward compatibility: Given that `str1 & str2` currently raises TypeError, this change would be close to 100% backward-compatible. Alternative meanings: As far as I know nobody has ever suggested an alternative meaning for `&` on strings. Bikeshedding: (1) I don't think it is important for the utility of this change whether `&` strips off all whitespace, or just spaces. I think it is better if it strips off all whitespace, so that it can be understood as working similarly to strip(). (2) The definition could be simplified to s1.rstrip() + ' ' + s2.lstrip() (leave an initial/final space when one string is whitespace only). Again the operator would be associative. Again I don't think this is important. Rob Cliffe [View Less]

10 19

Restricting access to sensitive APIs with a permission model like Deno
by python＠ruebezahl.mozmail.com March 1, 2023

March 1, 2023

Hello all, Supply chain attacks are becoming a pressing concern in software development due to the large number of dependencies and multiple attack vectors. Using third party modules (libraries, packages etc) is always a risk but the true potential of these attacks is now being weaponized. One way to deal with the risk is by limiting access to sensitive APIs like filesystem, shell, network and ffi so that packages which aren't explicitly granted permissions cannot use them, reducing their … [View More]

10 27

Proposal: -X importcache to supplement -X importtime for loaded modules
by Noah Kim Feb. 27, 2023

Feb. 27, 2023

All, I'm writing to propose an adjacent interpreter flag to `-X importtime`: `-X importcache` (open to alternative naming). While `-X importtime` is incredibly useful for analyzing module import times, by design, it doesn't log anything if an imported module has already been loaded. `-X importcache` would provide additional output for every module that's already been loaded: ``` >>> import uuid import time: cached | cached | _io import time: cached | cached | _io … [View More]

5 6

Combinations of keywords
by Otomatyk dupont Feb. 21, 2023

Feb. 21, 2023

Hello, here is my ideas of new keywords combinations : (the code is writed like this #code#. Sorry if I do some english mistakes) 1 - and pass if except This one is just a shortcut of # try : code here except : do something # Its syntaxe is #CODE and DO_SOMETHING if except#, it's more readable i think. It is usefull for code like this : #import A_WINDOW_MODULE and import A_UNIX_MODULE if except If DO_SOMETHING is replaced by #pass# it just skips the ligne (and the error) 2 - CODE … [View More]

5 4

Deprecate misleading escapes in strings
by Arusekk Feb. 17, 2023

Feb. 17, 2023

Hi all! I was writing a tutorial on the distinction between bytes and strings and why it is important, when I saw the root cause. People coming from C, Perl, Python 2 and similar languages tend to misinterpret "\x90" for b"\x90" often. My idea is that Python could deprecate string literals containing any non-ASCII codepoints specified in any way different from unicode or unicode escapes (\u, \U, \N). (Actually I found that I started having the idea already back in 2021 on … [View More]StackOverflow[1]. The question is an excellent example of what I mean.) I would not go so far to follow JSON (disallowing \x11 and \222 escapes completely), but while writing "\x00" or "\0" is useful and widely used, "\x99" (and especially "\777"!) is probably marginal and definitely less explicit than "\u0099" (in the Zen of explicit better than implicit). Byte strings do not treat b"\u00ff" as b"\xff". In the first part of implementing it, Python could raise a SyntaxWarning (or should it be DeprecationWarning? BytesWarning?), suggesting "\x99" to either become b"\u0099" or b"\x99", eventually promoting it to some equally helpful SyntaxError. All of it could be hidden behind a feature like from __future__ import backslashes (one nice name I can think of). The new regular expression for octals would be \\[01]?[0-7]{1,2} and \\x[0-7][0-9A-Fa-f] for hexadecimals, hopefully not confusing anyone, and not much more complex than the old ones. In the meantime, probably between introducing a warning and changing it to become an error (the most reasonable timeline I can think of now), the default ascii() representation should eventually use the \u0099 form for all such codepoints, to keep the invariant of eval(ascii(x)) == x without syntax warnings. repr() is also affected, but it is fortunately limited to the [\x80-\xa0\xad] range. I mean [\u0080-\u00a0\u00ad] :-) Another timeline would be to change the repr first, initially hidden under an interpreter flag or environment variable, then officially deprecate it in the documentation, then introduce the error guarded by from __future__ import backslashes or another flag, then make the repr use \u by default, then add the warning and finally make it always raise an error. As a precedent, breaking repr() was not a dealbreaker when introducing randomized seeds (even repr({"a", "b"}) is now unpredictable). This would be of course a breaking change for a lot of unit tests, and stuff like pickle should probably support old syntax, delaying any such change until a new protocol comes (if it applies to the newest one --- quite sure it does not). Such a breaking change must be used wisely. Other changes to octal escapes could be sneaked in, based on conclusions from the 2018 'Python octal escape character encoding "wats"' thread[2] (I like writing "\0" and "\4" though, just to make my opinion clear). If going the whole hog, the 2015 'Make non-meaningful backslashes illegal in string literals' thread[3] could be revived as well, maybe even with "\f\v" deprecated, "\e" = "\33" introduced and such. Please let me know what you think, what else could break, and is it useful anywhere else apart from my use case, and what similar problems you have. Cheers, Arusekk [1]: https://stackoverflow.com/q/64832281/3869724 [2]: https://mail.python.org/archives/list/python-ideas@python.org/thread/ARBCIP… [3]: https://mail.python.org/archives/list/python-ideas@python.org/message/PJXKD… [View Less]

6 7

timedelta.strptime() and timedelta.strftime()
by Thomas Mc Kay Feb. 2, 2023

Feb. 2, 2023

I was looking for a simple intuitive way to parse a timedelta from a string. The best I could find feels clunky at best: https://stackoverflow.com/questions/4628122/ Solution 1 looks like this: ``` from datetime import datetime, timedelta t = datetime.strptime('05:20:25', '%H:%M:%S') delta = timedelta(hours=t.hour, minutes=t.minute, seconds=t.second) ``` Solution 2 looks like this: ``` from datetime import datetime, timedelta t = datetime.strptime('05:20:25', '%H:%M:%S') delta = t - datetime.… [View More]

2 2

Re: Multiple arguments to str.partition and bytes.partition
by Peter Ludemann Jan. 10, 2023

Jan. 10, 2023

You can get almost the same result using pattern matching. For example, your "foo:bar;baz".partition(":", ";") can be done by a well-known matching idiom: re.match(r'([^:]*):([^;]*);(.*)', 'foo:bar;baz').groups()

4 6

Multiple arguments to str.partition and bytes.partition
by James Addison Jan. 9, 2023

Jan. 9, 2023

Hi folks, I'd like to gather some feedback on a feature suggestion to extend str.partition and bytes.partition to support multiple arguments. Please find the feature described below. # Feature or enhancement The str.partition[1] method -- and similarly the bytes.partition[2] method -- is useful when dividing an input into subcomponents while returning a tuple of deterministic length (currently 3) and potentially retaining the delimiters. The feature described here adds support for multiple … [View More]

4 9