Python-ideas August 2021

python-ideas@python.org

73 participants
45 discussions

disallow assignment to unknown ssl.SSLContext attributes
by Thomas Grainger 10 Jun '22

10 Jun '22

I was debugging some code that was using TLSv1.2 when I expected it to only support TLSv1.3, I tracked it down to a call to: context.miunimum_version = ssl.TLSVersion.TLSv1_3 it should have been: context.minimum_version = ssl.TLSVersion.TLSv1_3 I'd like invalid attribute assignment to be prevented at runtime

13 32

Bringing the print statement back
by Guido van Rossum 02 Mar '22

02 Mar '22

In Python 3.10 we will no longer be burdened by the old parser (though 3rd party tooling needs to catch up). One thing that the PEG parser makes possible in about 20 lines of code is something not entirely different from the old print statement. I have a prototype: Python 3.10.0a0 (heads/print-statement-dirty:5ed19fcc1a, Jun 9 2020, 16:31:17) [Clang 11.0.0 (clang-1100.0.33.8)] on darwin Type "help", "copyright", "credits" or "license" for more information. Cannot read termcap database; using dumb terminal settings. >>> print 2+2 4 >>> print "hello world" hello world >>> print "hello", input("Name:") Name:Guido hello Guido >>> print 1, 2, 3, sep=", " 1, 2, 3 >>> But wait, there's more! The same syntax will make it possible to call *any* function: >>> len "abc" 3 >>> Or any method: >>> import sys >>> sys.getrefcount "abc" 24 >>> Really, *any* method: >>> class C: ... def foo(self, arg): print arg ... >>> C().foo 2+2 4 >>> There are downsides too, though. For example, you can't call a method without arguments: >>> print <built-in function print> >>> Worse, the first argument cannot start with a parenthesis or bracket: >>> print (1, 2, 3) 1 2 3 >>> C().foo (1, 2, 3) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: C.foo() takes 2 positional arguments but 4 were given >>> print (2+2), 42 4 (None, 42) >>> C().foo [0] Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'method' object is not subscriptable >>> No, it's not April 1st. I am seriously proposing this (but I'll withdraw it if the response is a resounding "boo, hiss"). After all, we currently have a bunch of complexity in the parser just to give a helpful error message to people used to Python 2's print statement: >>> print 1, 2, 3 File "<stdin>", line 1 print 1, 2, 3 ^ SyntaxError: Missing parentheses in call to 'print'. Did you mean print(1, 2, 3)? >>> And IIRC there have been a number of aborted attempts at syntactic hacks to allow people to call functions (like print) without parentheses, although (I think) none of them made it into a PEP. The PEG parser makes this much simpler, because it can simply backtrack -- by placing the grammar rule for this syntax (tentatively called "call statement") last in the list of alternatives for "small statement" we ensure that everything that's a valid expression statement (including print() calls) is still an expression statement with exactly the same meaning, while still allowing parameter-less function calls, without lexical hacks. (There is no code in my prototype that checks for a space after 'print' -- it just checks that there's a name, number or string following a name, which is never legal syntax.) One possible extension I didn't pursue (yet -- dare me!) is to allow parameter-less calls inside other expressions. For example, my prototype does not support things like this: >>> a = (len "abc") File "<stdin>", line 1 a = (len "abc") ^ SyntaxError: invalid syntax >>> I think that strikes a reasonable balance between usability and reduced detection of common errors. I could also dial it back a bit, e.g. maybe it's too much to allow 'C().foo x' and we should only allow dotted names (sufficient to access functions in imported modules and method calls on variables). Or maybe we should only allow simple names (allowing 'len x' but disallowing 'sys.getrefcount x'. Or maybe we should really only bring back print statements. I believe there are some other languages that support a similar grammar (Ruby? R? Raku?) but I haven't investigated. Thoughts? -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-…>

42 66

Adding a thin wrapper class around the functions in stdlib.heapq
by bunslow 17 Feb '22

17 Feb '22

Nothing so bombastic this time. The heapq functions are basically all named "heapsomething", and basically all take a "heap" for their first argument, with supplementary args coming after. It's a textbook example of the (hypothetical) Object Oriented Manifesto™ where defining a class increases type safety and programmers' conceptual clarity. There're practically no drawbacks, and the code to be added would be very simple. Updating the tests and docs would probably be harder. In pure Python, such a class would look like this: class Heap(list): def __init__(self, iterable=None): if iterable: super().__init__(iterable) else: super().__init__() self.heapify() push = heapq.heappush pop = heapq.heappop pushpop = heapq.heappushpop replace = heapq.heapreplace heapify = heapq.heapify # This could be a simple wrapper as well, but I had the following thoughts anyways, so here they are def nsmallest(self, n, key=None): # heapq.nsmallest makes a *max* heap of the first n elements, # while we know that self is already a min heap, so we can # make the max heap construction faster self[:n] = reversed(self[:n]) return heapq.nsmallest(n, self, key) # do we define nlargest on a minheap?? Wrapping around the C builtin functions (which aren't descriptors) would be a bit harder, but not much so: from functools import partial class Heap(list): def __init__(self, iterable=None): if iterable: super().__init__(iterable) else: super().__init__() self.heapify = partial(heapq.heapify, self) self.push = partial(heapq.heappush, self) ... self.heapify() Thoughts?

17 32

Allow syntax "func(arg=x if condition)"
by Peter O'Connor 19 Jan '22

19 Jan '22

I often find that python lacks a nice way to say "only pass an argument under this condition". (See previous python-list email in "Idea: Deferred Default Arguments?") Example 1: Defining a list with conditional elements include_bd = True current_way = ['a'] + (['b'] if include_bd else [])+['c']+(['d'] if include_bd else []) new_way = ['a', 'b' if include_bd, 'c', 'd' if include_bd] also_new_way = list('a', 'b' if include_bd, 'c', 'd' if include_bd) Example 2: Deferring to defaults of called functions def is_close(a, b, precicion=1e-9): return abs(a-b) < precision def approach(pose, target, step=0.1, precision=None): # Defers to default precision if not otherwise specified: velocity = step*(target-pose) \ if not is_close(pose, target, precision if precision is not None) \ else 0 return velocity Not sure if this has been discussed, but I cannot see any clear downside to adding this, and it has some clear benefits (duplicated default arguments and **kwargs are the scourge of many real world code-bases)

6 9

Python Documentation in Bengali (translation)
by sharifmehedi24＠outlook.com 16 Sep '21

16 Sep '21

Hi there all, I am preparing a team to start translating python documentation to Bengali. I'd appreciate some reference materials, comments and ideas about this. Thank you

4 4

Add check_methods function to standard library
by Finn Mason 09 Sep '21

09 Sep '21

Hello, In _collections_abc.py is a private function titled `_check_methods`. It takes a class and a number of method names (as strings), checks if the class has all of the methods, and returns NotImplemented if any are missing. The code is below: ``` def _check_methods(C, *methods): mro = C.__mro__ for method in methods: for B in mro: if method in B.__dict__: if B.__dict__[method] is None: return NotImplemented break else: return NotImplemented return True ``` This is an incredibly convenient function (referred to as check_methods here on out) for creating abstract base classes, and is much simpler than using `hasattr` for each method you want to check. For example: ``` >>> from abc import ABCMeta >>> # Without check_methods >>> class A(metaclass=ABCMeta): ... @classmethod ... def __subclasshook__(cls, subclass): ... return (hasattr(subclass, 'foo') and ... callable(subclass.foo) and ... hasattr(subclass, 'bar') and ... callable(subclass.bar) or ... NotImplemented) ... >>> # With check_methods >>> class B(metaclass=ABCMeta): ... @classmethod ... def __subclasshook(cls, subclass): ... return check_methods(subclass, 'foo', 'bar') ... >>> ``` This would be a great function to add to the standard lib, perhaps in the `abc` module. One problem with `check_methods` as defined in _collections_abc.py is that it doesn't check if the name is callable. Also, type hints and more readable variables may be desirable. The final code, if implemented, may look something like this: ``` # In imports section: from typing import Literal def check_methods(Class: type, *methods: str) -> Literal[True, NotImplemented]: """Check if class `Class` has methods `methods`.""" mro = Class.__mro__ for method in methods: for Base in mro: if (attr := getattr(Base, method, None)) is not None: if not callable(attr): return NotImplemented break else: return NotImplemented return True ``` Again, this would be a great function to add to the `abc` module or a similar one. I've proposed this in issue 44941. Another possible implementation may be similar to a dataclass that implements __subclasshook__ for you with the desired method checks. --Finn

1 2

Additional Unpacking Generalizations for assignment
by 笹原康央 09 Sep '21

09 Sep '21

Hello developers. I like star expression. It would be even better if we could extend the idea of PEP448 to provide the following syntax at the time of assignment. ``` python a, b, d, **_ = **{"a":0, " b":1, "c":2}, d=3 ``` Ideally, it would be nice to be able to provide the same syntax when calling the function. I think this applies to the zen of python called "There should be one-- and preferably only one --obvious way to do it.". ``` python a, *args, b, **kwargs = 1, 2, *[3,4], **{"b", 4}, c=4 ( a: str, *args, b: list = [], **kwargs ) = 1, 2, *[3,4], **{"b", 4}, c=4 func = lambda( a: str, *args, b: list = [], **kwargs ): print(a, b) ``` But it would be difficult because it would conflict with various syntaxes(etc. PEP 3132). We may have to declare it explicitly to avoid confusion. For example, prepare a keyword for unpack the positional arguments and keyword arguments at the same time. ``` a, *args, b, **kwargs = ***(1, 2, *[3,4], **{"b", 4}, c=4) ``` You can now explicitly recognize that you are using the behavior of the new assignment. I hope this is an idea to develop python. Thanks.

2 3

PEP8 mandatory-is rule
by Nick Parlante 04 Sep '21

04 Sep '21

Hi there python-ideas - I've been teaching Python as a first programming language for a few years, and from that experience I want to propose a change to PEP8. I'm sure the default position for PEP8 is to avoid changing it. However, for this one rule I think a good case can be made to make it optional, so let me know what you think. Let me start with what I've learned from teaching students in Java and now in Python. In Java, you use == for ints, but you need to use equals() for strings. Of course students screw this up constantly, using == in a context that calls for equals() and their code does not work right. Then for Java arrays a different comparison function is required, and so it goes. To teach comparisons in Python, I simply say "just use ==" - it works for ints, for strings, even for lists. Students are blown away by how nice and simple this is. This is how things should work. Python really gets this right. So what is the problem? The problem for Python is what I will call the "mandatory-is" rule in PEP8, which reads: Comparisons to singletons like None should always be done with is or is not, never the equality operators. For the students, this comes up in the first week of the course with lines like "if x == None:" which work perfectly with == but should use is/is-not for PEP8 conformance. My guess is that this rule is in PEP8 because, within a Python implementation, it is within the programmer's mental model that, say, False is a singleton. The mandatory-is rule is in PEP8 to reinforce that mental model by requiring the is operator. Plus it probably runs a tiny bit faster. However, for "regular" Python code, not implementing Python, forcing the use of is instead of the simpler == is unneeded and unhelpful (and analogously forcing "is not" when != works correctly). What is the benefit of forcing the is operator there? I would say it spreads an awareness of the details of how certain values are allocated within Python. That's not much of a benefit, and it's kind of circular. Like if programmers were permitted to use ==, they wouldn't need to know the details of how Python allocates those values. Being shielded from implementation details is a Python strength - think of the Java vs. Python story above. Is Java better because it builds an awareness in the programmer of the different comparison functions for different types? Of course not! Python is better in that case because it lets the programmer simply use == and not think about those details. Understanding the singleton strategy is important in some corners of coding, but forcing the is operator on all Python code is way out of proportion to the benefit. As a practical matter, the way this comes up for my students is that IDEs by default will put warning marks around PEP8 violations in their code. Mostly this IDE-coaching is very helpful for students learning Python. For example, It's great that beginning Python programmers learn to put one space around operators right from the first day. Having taught thousands of introductory Python students, the one PEP8 rule that causes problems is this mandatory-is rule. As a teacher, this is especially jarring since the "just use ==" rule is so effortless to use correctly. In contrast, the mandatory-is rule adds a little pause where the programmer should think about which comparison operator is the correct one to use. It's not hard, but it feels unnecessary. As a contrasting example, in the language C, programmers need to understand == vs. is right from the first day. You can't get anything done in C without understanding that distinction. However that is just not true for regular (not-Python-implementation) Python code, where == works correctly for the great majority of cases. Here is my proposal: Add the following parenthetical to the mandatory-is rule: (this rule is optional for code that is not part of an implementation of Python). So in effect, programmers outside of a Python implementation can choose to use == or is for the "if x == None:" case. In this way, PEP8 conforming code before the change is still conforming. Moving forward, I would expect that regular code will trend towards using == in such a case, reserving is for the rare cases where it is needed for correctness. PEP8 was originally just for Python implementations, so why is this change needed? Because as a practical matter, the vast majority of code that is using PEP8 is not part of a Python implementation. This may not have been the original mission of PEP8, but it is how things have worked out. Now we are in a situation where the rules in PEP8 are sent out to this ocean of Python programmers of many different ability levels writing regular code that is not a Python implementation. One could imagine a separate PEP800 style guide for regular code, but we don't need to do that, because in almost all cases PEP8 works great for regular code. I have taught thousands of new Python programmers, and the only place where PEP8 serves them poorly is this mandatory-is rule. Therefore instead of a separate style guide for regular code, I propose an exception for this one problem rule. Ultimately this comes down to the question - should PEP8 push regular, not-Python-implementation code to use is for singletons in cases where == works perfectly? Seeing how effortless it is for programmers to use == as their first choice, I think PEP8 should allow that practice. Best, Nick

20 84

NAN handling in statistics functions
by Steven D'Aprano 31 Aug '21

31 Aug '21

At the moment, the handling of NANs in the statistics module is implementation dependent. In practice, that *usually* means that if your data has a NAN in it, the result you get will probably be a NAN. >>> statistics.mean([1, 2, float('nan'), 4]) nan But there are unfortunate exceptions to this: >>> statistics.median([1, 2, float('nan'), 4]) nan >>> statistics.median([float('nan'), 1, 2, 4]) 1.5 I've spoken to users of other statistics packages and languages, such as R, and I cannot find any consensus on what the "right" behaviour should be for NANs except "not that!". So I propose that statistics functions gain a keyword only parameter to specify the desired behaviour when a NAN is found: - raise an exception - return NAN - ignore it (filter out NANs) which seem to be the three most common preference. (It seems to be split roughly equally between the three.) Thoughts? Objections? Does anyone have any strong feelings about what should be the default? -- Steve

18 50

Re: NAN handling in statistics functions
by Christopher Barker 31 Aug '21

31 Aug '21

On Tue, Aug 31, 2021 at 12:09 AM Stephen J. Turnbull < stephenjturnbull(a)gmail.com> wrote: > *sigh* __members__ is just the mechanism. As a consequence, Enums are > iterable, and they automatically DTRT with dir() and help() even if > there are no docstrings. but you didn't say dir(), you said __members__ ;-) But my bad, yes, dir() is easy and obvious, and works. As for help() -- for some reason, I never use it -- not sure why. I do use iPython's ?, which gives me: In [15]: NaNflag? Init signature: NaNflag( value, names=None, *, module=None, qualname=None, type=None, start=1, ) Docstring: An enumeration. File: /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/enum.py Type: EnumMeta Subclasses: which is not that useful, because the docstring is pretty worthless. help() on the other hand, is useful: class NaNflag(enum.Enum) | NaNflag(value, names=None, *, module=None, qualname=None, type=None, start=1 ) | | An enumeration. | | Method resolution order: | NaNflag | enum.Enum | builtins.object | | Data and other attributes defined here: | | Propogate = <NaNflag.Propogate: 3> | | Raise = <NaNflag.Raise: 1> | | Skip = <NaNflag.Skip: 2> | | ---------------------------------------------------------------------- | Data descriptors inherited from enum.Enum: | name | The name of the Enum member. | | value | The value of the Enum member. | | ---------------------------------------------------------------------- | Readonly properties inherited from enum.EnumMeta: | | __members__ | Returns a mapping of member name->value. | | This mapping lists all enum members, including aliases. Note that this | is a read-only view of the internal mapping. Hmm - maybe that's why I don't use help() much -- If an object has a decent docstring, I'd rather not see all that other cruft :-) So if you want a well documented Enum, you need to write a docstring, which is, or course, the case with everything in Python, so no problem there. (though a nice auto-generated docstring would be good -- maybe a feature request for the future) I've been convinced, Enums do provide a nice way to document flag options. But unfortunately, it doesn't put that documentation where I'd want it -- in the docstring of the functions that use it. DRY is hard with documentation -- this reminds me of when there's a big class hierarchy, and you have go read the docs way up the tree to know how to use a class. Now show me how to do any of those with str > codes. > In that case, the documentation needs to go in the docstring of the function where they are used, which is where I want it anyway. That's a documentation style matter, though. Since just about every > function in the statistics package will take the 'nans' argument, not > only the writer, but most readers of the documentation will favor > DRYing it once, IMHO YMMV. > I don't think readers will have a preference for having something documented in one place that is not the place you actually use it. They WILL appreciate that all functions in the statistics package work the same way though, that's for sure. Anyway, I was expressing a preference, and I still have that preference, but Enums are a fine option as well. And the proposed idea of using the string value for the Enum, so that users can use either has its appeal. For me, it's like Lays potato chips; I can't call just one. Almost > all my use cases call for mean, standard deviation, and median. > hmm, another topic, but computing mean and standard deviation at the same time would make sense. Bringing back to the decision Steven needs to make: I still prefer string flags, but there are certainly good arguments to be made for an Enum. If you do go with an Enum, I suggest: - The flag names are put in the statistics module namespace. - The Enum gets a good docstring. - The functions that use the Enum also document the flags and what they mean. - to keep that dry, you could automatically insert the Enum docstring into each function's docstring, so that it would be there where users need it. That's my $ 0.2 -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

1 0

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

Python-ideas August 2021