Mailman 3 June 2021 - Python-ideas

zip(x, y, z, strict=True)
by Ram Rachum 01 Dec '22

01 Dec '22

Here's something that would have saved me some debugging yesterday: >>> zipped = zip(x, y, z, strict=True) I suggest that `strict=True` would ensure that all the iterables have been exhausted, raising an exception otherwise. This is useful in cases where you're assuming that the iterables all have the same lengths. When your assumption is wrong, you currently just get a shorter result, and it could take you a while to figure out why it's happening. What do you think?

27 179

disallow assignment to unknown ssl.SSLContext attributes
by Thomas Grainger 10 Jun '22

10 Jun '22

I was debugging some code that was using TLSv1.2 when I expected it to only support TLSv1.3, I tracked it down to a call to: context.miunimum_version = ssl.TLSVersion.TLSv1_3 it should have been: context.minimum_version = ssl.TLSVersion.TLSv1_3 I'd like invalid attribute assignment to be prevented at runtime

13 32

Bringing the print statement back
by Guido van Rossum 02 Mar '22

02 Mar '22

In Python 3.10 we will no longer be burdened by the old parser (though 3rd party tooling needs to catch up). One thing that the PEG parser makes possible in about 20 lines of code is something not entirely different from the old print statement. I have a prototype: Python 3.10.0a0 (heads/print-statement-dirty:5ed19fcc1a, Jun 9 2020, 16:31:17) [Clang 11.0.0 (clang-1100.0.33.8)] on darwin Type "help", "copyright", "credits" or "license" for more information. Cannot read termcap database; using dumb terminal settings. >>> print 2+2 4 >>> print "hello world" hello world >>> print "hello", input("Name:") Name:Guido hello Guido >>> print 1, 2, 3, sep=", " 1, 2, 3 >>> But wait, there's more! The same syntax will make it possible to call *any* function: >>> len "abc" 3 >>> Or any method: >>> import sys >>> sys.getrefcount "abc" 24 >>> Really, *any* method: >>> class C: ... def foo(self, arg): print arg ... >>> C().foo 2+2 4 >>> There are downsides too, though. For example, you can't call a method without arguments: >>> print <built-in function print> >>> Worse, the first argument cannot start with a parenthesis or bracket: >>> print (1, 2, 3) 1 2 3 >>> C().foo (1, 2, 3) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: C.foo() takes 2 positional arguments but 4 were given >>> print (2+2), 42 4 (None, 42) >>> C().foo [0] Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'method' object is not subscriptable >>> No, it's not April 1st. I am seriously proposing this (but I'll withdraw it if the response is a resounding "boo, hiss"). After all, we currently have a bunch of complexity in the parser just to give a helpful error message to people used to Python 2's print statement: >>> print 1, 2, 3 File "<stdin>", line 1 print 1, 2, 3 ^ SyntaxError: Missing parentheses in call to 'print'. Did you mean print(1, 2, 3)? >>> And IIRC there have been a number of aborted attempts at syntactic hacks to allow people to call functions (like print) without parentheses, although (I think) none of them made it into a PEP. The PEG parser makes this much simpler, because it can simply backtrack -- by placing the grammar rule for this syntax (tentatively called "call statement") last in the list of alternatives for "small statement" we ensure that everything that's a valid expression statement (including print() calls) is still an expression statement with exactly the same meaning, while still allowing parameter-less function calls, without lexical hacks. (There is no code in my prototype that checks for a space after 'print' -- it just checks that there's a name, number or string following a name, which is never legal syntax.) One possible extension I didn't pursue (yet -- dare me!) is to allow parameter-less calls inside other expressions. For example, my prototype does not support things like this: >>> a = (len "abc") File "<stdin>", line 1 a = (len "abc") ^ SyntaxError: invalid syntax >>> I think that strikes a reasonable balance between usability and reduced detection of common errors. I could also dial it back a bit, e.g. maybe it's too much to allow 'C().foo x' and we should only allow dotted names (sufficient to access functions in imported modules and method calls on variables). Or maybe we should only allow simple names (allowing 'len x' but disallowing 'sys.getrefcount x'. Or maybe we should really only bring back print statements. I believe there are some other languages that support a similar grammar (Ruby? R? Raku?) but I haven't investigated. Thoughts? -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-…>

42 66

Adding a thin wrapper class around the functions in stdlib.heapq
by bunslow 17 Feb '22

17 Feb '22

Nothing so bombastic this time. The heapq functions are basically all named "heapsomething", and basically all take a "heap" for their first argument, with supplementary args coming after. It's a textbook example of the (hypothetical) Object Oriented Manifesto™ where defining a class increases type safety and programmers' conceptual clarity. There're practically no drawbacks, and the code to be added would be very simple. Updating the tests and docs would probably be harder. In pure Python, such a class would look like this: class Heap(list): def __init__(self, iterable=None): if iterable: super().__init__(iterable) else: super().__init__() self.heapify() push = heapq.heappush pop = heapq.heappop pushpop = heapq.heappushpop replace = heapq.heapreplace heapify = heapq.heapify # This could be a simple wrapper as well, but I had the following thoughts anyways, so here they are def nsmallest(self, n, key=None): # heapq.nsmallest makes a *max* heap of the first n elements, # while we know that self is already a min heap, so we can # make the max heap construction faster self[:n] = reversed(self[:n]) return heapq.nsmallest(n, self, key) # do we define nlargest on a minheap?? Wrapping around the C builtin functions (which aren't descriptors) would be a bit harder, but not much so: from functools import partial class Heap(list): def __init__(self, iterable=None): if iterable: super().__init__(iterable) else: super().__init__() self.heapify = partial(heapq.heapify, self) self.push = partial(heapq.heappush, self) ... self.heapify() Thoughts?

17 32

Allow syntax "func(arg=x if condition)"
by Peter O'Connor 19 Jan '22

19 Jan '22

I often find that python lacks a nice way to say "only pass an argument under this condition". (See previous python-list email in "Idea: Deferred Default Arguments?") Example 1: Defining a list with conditional elements include_bd = True current_way = ['a'] + (['b'] if include_bd else [])+['c']+(['d'] if include_bd else []) new_way = ['a', 'b' if include_bd, 'c', 'd' if include_bd] also_new_way = list('a', 'b' if include_bd, 'c', 'd' if include_bd) Example 2: Deferring to defaults of called functions def is_close(a, b, precicion=1e-9): return abs(a-b) < precision def approach(pose, target, step=0.1, precision=None): # Defers to default precision if not otherwise specified: velocity = step*(target-pose) \ if not is_close(pose, target, precision if precision is not None) \ else 0 return velocity Not sure if this has been discussed, but I cannot see any clear downside to adding this, and it has some clear benefits (duplicated default arguments and **kwargs are the scourge of many real world code-bases)

6 9

copy-paste code snippets to the python console
by Raymond Bisdorff 15 Aug '21

15 Aug '21

Dear Python developers, It would be helpful, if the following issue with copy-pasting python code-snippets into the standard shell console, could be investigated and corrected. https://stackoverflow.com/questions/2501208/copying-and-pasting-code-into-t… In particular, copying and pasting from sphinx python and pycon code-blocks with copy button (only >>> and ... lines), is at present not generally working due to the "the shell-s de-indent cmd" need for empty lines. See https://digraph3.readthedocs.io/en/latest/index.html Thank you in advance for you attention, Best Regards, Raymond Bisdorff -- Raymond Bisdorff Emeritus Professor of Computer Science and Applied Mathematics University of Luxembourg http://rbisdorff.github.io/

2 1

Pre PEP: Python Literals (was custom strings before)
by Thomas Güttler 11 Jul '21

11 Jul '21

Thank you Guido, Chris, Matt and Richard for your feedback to my last email. Here is an updated version called "Template Literals". I am looking for a core developer who can sponsor this PEP. Please speak up if you want to help me. Regards, Thomas Güttler Source and updates: Pre-PEP 9999 <https://github.com/guettli/peps/blob/master/pep-9999.rst> PEP: 9999 Title: Template Literals Author: Thomas Güttler <info at thomas-guettler.de> Sponsor: TODO Status: Draft Type: Standards Track Content-Type: text/x-rst </dev/peps/pep-0012> Created: 08-Jun-2021 Python-Version: TODO Post-History: 08-Jun-2021 ------------------------------ Contents - Abstract <#abstract> - Motivation <#motivation> - Rationale <#rationale> - Specification <#specification> - Security Implications <#security-implications> - Reference Implementation <#reference-implementation> - Alternative Ideas <#alternative-ideas> - Rejected Ideas <#rejected-ideas> - Open Issues <#open-issues> - References <#references> - Copyright <#copyright> Abstract <#id6> This PEP adds Template Literals to Python. To avoid code injection like XSS or SQL-injection Template Literals can help you to write save Python code. Template Literals provide an easy way to access the local and global variables (like f-strings), so that passing a dictionary to the Template is not necessary. Motivation <#id7> In the context of web development Python can do more than providing REST APIs via http. With the trend to Server-Side-Rendering, we face a fundamental question: How to create HTML with Python? If you use the FrOW pattern (HTML fragments over the wire) [1] <#frow>, then you will be writing small methods returning small HTML fragments. As a developer I want to pass escaped data into template literals to be as simple as possible. Rationale <#id8> Imagine you want to create a small HTML fragment in Python, and return it as HTTP-Response: HttpResponse(f''' <h1>Hi {name}</h1> Your messages: {messages}''') The problem in above example is, that no escaping gets done. In above example "name" and "messages" should be treated differently. The variable "name" should get escaped. For example if the name is "Mary & Bob", the result should be "Mary & Bob". The variable "messages" contains HTML which is already escaped. It should not be escaped again. Most frameworks have a way to do this conditional escaping. For example Django uses conditional_escape() <https://docs.djangoproject.com/en/3.2/ref/utils/#django.utils.html.conditio…> [2] <#id2> With the help of conditional_escape() the above problem could be solved like this: HttpResponse(f''' <h1>Hi {conditional_escape(name)}</h1> Your messages: {conditional_escape(messages)}''') This solution has two drawbacks: 1. It is too verbose. Typing "conditional_escape(...)" again and again is cumbersome. 2. If a conditional_escape() gets forgotten Cross-site scripting attacks could be possible, since malicious users could inject HTML. Specification <#id9> Template Literals use backticks (like JavaScript Template Literals <https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_…> [3] <#id4>) Example: name = 'Mary & Bob' messages = `<ul><li>message1</li><li>message2</li></ul>` return HttpResponse(` <h1>Hi {name}</h1> Your messages: {messages} Today: {datetime.date.today()}`) Expressions within curly braces get handled liked in f-strings (PEP-498). The Template Literal creates an instance of the new class types.TemplateLiteral. types.TemplateLiteral has two attributes: - template: The raw string inside the backticks. - tokens: A list of tuples: (value, is_literal). For above example this would mean: template = ''' <h1>Hi {name}</h1> Your messages: {messages} Today: {datetime.date.today()}''' tokens = [ ('\n <h1>Hi ', True), ('Mary & Bob', False), ('</h1>\n Your messages: ', True), (<TemplateLiteral "<ul><li>message1</li><li>message2</li></ul>">, False), ('\n\n Today: ', True), (<datetime.date(2021, 6, 9)>, False) ] It is outside this PEP how a consumer of TemplateLiteral handles this data structure. For example the Django web framework could transform a TemplateLiteral to a SafeString like this: def template_literal_to_safestring(template_literal): return mark_safe( ''.join( [ conditional_escape(value) if not is_literal else value for (value, is_literal) in template_literal.tokens ] ) ) This PEP is not related or constraint to the Django framework. It is even not related to HTML. It can be used for any kind of templating. Security Implications <#id10> Template Literals can execute arbitrary code (like f-strings). Template Literals get created by Python developers, not by users. If you want to make templates available for users (for example if you develop a CMS), then please use a different solution. Reference Implementation <#id11> TODO Alternative Ideas <#id12> Instead of backticks for example t'...' could be used. Rejected Ideas <#id13> TODO Open Issues <#id14> TODO References <#id15> [1] <#id1> FrOW, "HTML Fragments Over the Wire". Frameworks like Unpoly, Hotwire or htmx. [2] <#id3> https://docs.djangoproject.com/en/3.2/ref/utils/#django.utils.html.conditio… [3] <#id5> https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_… Copyright <#id16> This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.

19 56

Implementing a 'but' statement in for-iterations
by Max Shouman 29 Jun '21

29 Jun '21

This is more of a syntactic sugar than an actual new feature, but... Exactly, 'but' is the idea: a special keyword to be used in for statements to exclude values from the iterable. E.g., when iterating over a generator: >>> for i in range(0, 10) but (2, 8): would implicitly create a new generator comprehensively, as in: >>> for i in (j for j in range(0, 10) if j not in [2, 8]): It might not add such a feature to justify the definition of a but_stmt in python.gram, but it's fully compliant with Python's philosophy of concise, clear and elegant code. #road to a programming natural language (jk)

8 12

joining paths without worrying about a leading slash
by Zbigniew Jędrzejewski-Szmek 28 Jun '21

28 Jun '21

[this is a continuation of https://bugs.python.org/issue44452] pathlib.Path() has a concatenation operator "/" that allows the right-hand-side argument to be an absolute path, which causes the left-hand-side argument to be ignored: >>> pathlib.Path('/foo') / '/bar' PosixPath('/bar') >>> pathlib.Path('/var/tmp/instroot') / '/some/path' / '/suffix' PosixPath('/suffix') This follows the precedent set by os.path.join(), and probably makes sense in the scenario of simulating a user typing 'cd' commands in a shell. But it doesn't work nicely in the case of combining paths from two different "namespaces", where we never want to go "up". For example: a web server takes an URL, strips the host, and wants to look up a file: https://example.com/some/path → "/some/path" → /src/www/root + /some/path → /src/www/root/some/path or we are constructing a container image and need to refer to a file in the container: <container foo> + /etc/shadow → /var/lib/machines/foo + /etc/shadow → /var/lib/machines/foo/etc/shadow To do this kind of operation correctly with pathlib.Path, the user needs to do two operations: verify that the rhs argument contains no '..' [*], and strip leading slashes: >>> lhs = pathlib.Path('/some/namespace/') >>> rhs = '/some/path/to/add' >>> if '..' in pathlib.Path(rhs).parts: raise ValueError >>> path = lhs / rhs.lstrip('/') Those last two lines are rather verbose, non-obvious. Also the .lstrip() operation attaches on the right side, but operates on the left side, earlier than the "/", which is overall not very nice. Proposal: add "//"-operator to pathlib.PosixPath() that means "concatenate a rhs path that is underneath the lhs". It would disallow paths with '..', and concatenate paths as relative to the specified lhs: >>> lhs = pathlib.Path('/some/namespace/') >>> lhs // "a/b/c" PosixPath('/some/namespace/a/b/c') >>> lhs // "/a/b/c" PosixPath('/some/namespace/a/b/c') >>> lhs // "a/../b/c" ValueError: cannot use // with a path with '..' on the right This would be useful for operations on containers, combining paths from namespaces like fs paths and URL components, looking up files underneath an unpacked archive, etc. [*] Why completely disallow '..' ? Components with '..' cannot be correctly resolved without access to the filesystem, because a component may be a symlink, and then "a/b/../." may not be "a/.", but something completely different. Thus, since the goal is to have a path underneath lhs, I think it's best to forbid '..'. In principle '..' at the beginning can be resolved reliably, by simply ignoring it, '/../../../whatever' is the same as '/whatever/'. But it's a tiny corner case, and I think it's better to disallow that too. Zbyszek

3 6

An alternate idea for escaping in string interpolation
by Bruce Leban 27 Jun '21

27 Jun '21

This is a response in part to Thomas Güttler's "Pre PEP: Python Literals" but I am starting a separate thread because I think it deserves separate discussion, and it's not a direct response to that proposal. Here's the problem: we want to allow escaping in strings to prevent injection in HTML, SQL, etc. Proposed solutions generally concentrate on a new kind of escaped format string. I won't go into detail on why I don't like these ideas because others have already covered that ground. Instead, here's what I think is a better solution: a mechanism to allow formatting with html and sql escaping. Specifically, I suggest these changes: (1) Modify string formatting (str.format and f-strings) to add new conversions !html !sql !xml, which escape characters for embedding in html, sql, and xml respectively.** (2) Modify string formatting to allow these new conversions to be added to the current conversions, e.g. !s!html. (3) Modify string formatting to add a new syntax that specifies a conversion to use for all subsequent interpolations. The proposed syntax is a replacement_field that starts with two exclamation points. The replacement field itself expands to nothing and only affects the conversion of subsequent fields. Thus f"{!!html}<a href='{url}'>{text!r}</a>" is equivalent to f"<a href='{url!html}'>{text!r!html}</a>" A replacement field of {!!} resets the default conversion. Yes, this is more typing than t-strings or backticks but EIBTI. And I believe this expands on existing format functions in a way that will be much more clear to someone encountering this new mechanism. And it's more flexible as it allows more granular control of escaping as in this example: f""" <h1>{title!html}</h1> <p>{pre_rendered_html_body}</p> """ **It's unclear if there's any functional benefit of having both html and xml encoding other than clarity of usage. Also, it would be nice to have a mechanism for adding additional conversions but I don't want to complicate the discussion at this point. Are there other standard escape mechanisms that would be worth including? --- Bruce

3 3