Mailman 3 March 2021 - Python-Dev

SC feedback: PEP 648 -- Extensible customizations of the interpreter at startup
by Barry Warsaw 30 Apr '21

30 Apr '21

Hello Mario, Thank you for your submission of PEP 648 (Extensible customizations of the interpreter at startup). The Python Steering Council has reviewed the PEP and before we can pronounce on it, we have some additional questions and comments we’d like you to address. Once these questions are settled, we are requesting that you post the PEP to python-dev for another round of comments. In general, the SC is in favor of deprecating the executable hack capabilities of pth files, and this PEP is currently the most concrete proposal to get there. We would like to eventually go farther, including deprecation of pth files entirely, but that is outside the scope of this PEP. General PEP feedback The introduction of the section titled “Benefits of __sitecustomize__” seems out of place. It forward references the solution without explanation. It also says “The use of a __sitecustomize__ will allow…”; the question is, a __sitecustomize__ what? Directory? File? Perhaps it should be moved to later in the PEP after the semantics of __sitecustomize__ is defined? Some of the terminology used in the PEP could be clarified. For example, I suggest using the term “directory” instead of “folder” throughout the PEP, and use the term “module” or “file” instead of “script” to describe the things inside __sitecustomize__ directories that Python imports, since “scripts” typically describe standalone files that implement applications. For example, must the files be named with .py extension (or presumably .pyc, .pyo) under the normal Python module import rules? (Later in the PEP you do allude to the requirement that the file name must end in .py, but it would be much better to explain this right up front, in a formal specification.) Another ambiguity is what the PEP means by “executing” said “scripts”. Does that mean importing them? Reading them then exec()’ing them? Something else? This section should be clear that Python will import the modules it found, if that’s indeed what the PEP is proposing. The “Backward compatibility” section has this incomplete sentence: “Ignoring those lines in pth files.” There’s this odd grammar in the “Do nothing” section: “After analysing the impact of this change, we believe it is worth given the enhanced experience it brings.” Perhaps it should read “...it is worth it, given the…”? Definition of “site path” The PEP should precisely define what it means when it says “...a folder named __sitecustomize__ located in any site path”. What exactly is a “site path”? Is this any directory on sys.path? Is it a directory named site-packages? This phrase is also ambiguous: “As the folder will be within sys.path, given that it is located in site paths...”. “sys.path” isn’t a thing that can hold a folder, since it’s just a list of strings, and it also can get extended by any number of means, e.g. by setting the PYTHONPATH environment variable at interpreter startup time. So for example, how does PYTHONPATH affect the algorithm? What is the rationale for adding multiple __sitecustomize__ directories, rather than a single directory (or possibly two, one in Python’s system location and one in the user location)? This would simplify the discovery process, and tools like pip can warn the user if a name collision were to occur. I think the PEP needs to be much more crisp and precise in defining the when and how __sitecustomize__ directories are discovered at start up time. Order of Execution I think this section should outline the entire startup execution order. Exactly when do pth files still get evaluated in relationship to __sitecustomize__ discovery? How do pth file sys.path extensions affect the search for __sitecustomize__ directories? This section also says files in __sitecustomize__ will be executed in “file name sorted order”, but the PEP is unclear whether all such files found in every __sitecustomize__ directory will be sorted together, or only within a single __sitecustomize__ directory. Presumably file system encodings are used to determine file name sort order, but I think the PEP should be explicit about this. I’d like to see the execution order for the entire proposed new Python startup sequence as a recipe, or perhaps even some Python pseudo-code. Impact on startup time We still think that startup performance can be a concern. This section needs more data. For example, while the executable part of pth files is being deprecated, both pth and __sitecustomize__ modules will exist. How does this affect startup time in a world where both have to be discovered and processed? The PEP says: “If the user has custom scripts, we think that the impact on the performance of walking each of the folders is acceptable, as the user wants to use this feature.” Does the user really want to use this feature? Won’t it more likely be the system administrator or Python distribution and packaging ecosystem that chooses to use the __sitecustomize__ feature? Maybe the user won’t even know that it’s being used by the packages being installed? The PEP says: ““Running "./python -c pass" with perf on 50 iterations, repeating 50 times the command on each and getting the geometric mean on a commodity laptop did not reveal any substantial raise on CPU time.” We request more information on the experiments performed, and specific numbers so that the startup impact can be reproduced and confirmed by others. E.g. did you run this experiment on a branch of CPython 3.10 with the PEP 648 reference implementation? Did you convert existing uses of executable pth files to __sitecustomize__ modules? How many such modules did you try it with? How many __sitecustomize__ directories were in your tests? Other questions and issues How does this feature interact with virtual environments? What do packaging tools have to do to support this PEP? The PEP mentions setuptools, but what about other packaging tools in common use? The PEP says “This impact will be reduced in the future as we will remove two other imports: "sitecustomize.py" and "usercustomize.py”.” Why not explicitly deprecate these and executable pth files in PEP 648 and begin the countdown at the same time __sitecustomize__ is added to Python? The PEP says: “To facilitate debugging of the Python startup, a new option will be added to the main of the site module to list all scripts that will be executed as part of the __sitecustomize__ initialization.” The PEP should be explicit about what command line switch is being proposed. We think this should probably be an -X option. Do you have any concerns that this feature will be overused? Does separating the execution feature from the path extension feature make it more or less likely to be used by third party library authors? Will the runtime have any access to the list of __sitecustomize__ modules being executed? I.e. will you collect their paths in a new sys module attribute? Will there be any functions to import or list __sitecustomize__ directories or modules after start has completed? We think io.open_code() should be used so that reading the files can be audited. Conclusion The Python Steering Council again wants to thank you for your PEP contribution! We hope you find this feedback constructive and helpful. We look forward to your responses, and an updated PEP. Cheers, -Barry (on behalf of the Python Steering Council)

5 8

Let's Fix Class Annotations -- And Maybe Annotations Generally
by Larry Hastings 24 Apr '21

24 Apr '21

Howdy howdy. While working on my PEP I stumbled over a lot of behavior by annotations that I found inconsistent and inconvenient. I think there are several problems here that needs fixing. This discussion will probably evolve into a PEP, and I'll be happy to steer that process. But I'm less certain about what the right thing to do is. (Although I do know what I'd prefer!) So let's talk about it! Annotations are represented in Python as a dictionary. They can be present on functions, classes, and modules as an attribute called "__annotations__". We start with: how do you get the annotations from one of these objects? Surely it's as easy as this line from Lib/inspect.py shows us: return func.__annotations__ And yes, that's best practice for getting an annotation from a function object. But consider this line from Lib/functools.py: ann = getattr(cls, '__annotations__', {}) Huh. Why doesn't it simply look at cls.__annotations__? It's because the language declares that __annotations__ on a class or module is optional. Since cls.__annotations__ may not be defined, evaluating that might throw an exception. Three-argument getattr() is much safer, and I assert it's best practice for getting the annotations from a module. But consider this line from Lib/dataclasses.py: cls_annotations = cls.__dict__.get('__annotations__', {}) And a very similar line from Lib/typing.py: ann = base.__dict__.get('__annotations__', {}) Huh! Why is this code skipping the attribute entirely, and examining cls.__dict__? It's because the getattr() approach has a subtle bug when dealing with classes. Consider this example: class A: ax:int=3 class B(A): pass print(getattr(B, '__annotations__', {})) That's right, B *inherits* A.__annotations__! So this prints {'ax': int}. This *can't* the intended behavior of __annotations__ on classes. It's only supposed to contain annotations for the fields of B itself, not those of one of its randomly-selected base classes. But that's how it behaves today--and people have had to work around this behavior for years. Examining the class dict is, sadly, best practice for getting __annotations__ from a class. So, already: three different objects can have __annotations__, and there are three different best practices for getting their __annotations__. Let's zoom out for a moment. Here's the list of predefined data fields you can find on classes: __annotations__ __bases__ __class__ __dict__ __doc__ __module__ __mro__ __name__ __qualname__ All of these describe metadata about the class. In every case *except one*, the field is mandatory, which also means it's never inherited. And in every case *except one*, you cannot delete the field. (Though you *are* allowed to overwrite some of them.) You guessed it: __annotations__ is the exception. It's optional, and you're allowed to delete it. And these exceptions are causing problems. It seems to me that, if the only way to correctly use a language-defined attribute of classes is by rooting around in its __dict__, the design is a misfire. (Much of the above also applies to modules, too. The big difference: since modules lack inheritance, you don't need to look in their __dict__.) Now consider what happens if my "delayed annotations of annotations using descriptors" PEP is accepted. If that happens, pulling __annotations__ out of the class dict won't work if they haven't been generated yet. So today's "best practice" becomes tomorrow's "this code doesn't work". To correctly examine class annotations, code would have to do something like this, which should work correctly in any Python 3.x version: if (getattr(cls, '__co_annotations__', None) or ('__annotations__' in cls.__dict__)): ann = cls.__annotations__ else: ann = {} This is getting ridiculous. Let's move on to a related topic. For each of the objects that can have annotations, what happens if o.__annotations__ is set, and you "del o.__annotations__", then you access o.__annotations__? It depends on what the object is, because each of them behaves differently. You already know what happens with classes: if any of the base classes has __annotations__ set, you'll get the first one you find in the MRO. If none of the bases have __annotations__ set you'll get an AttributeError. For a module, if you delete it then try to access it, you'll always get an AttributeError. For a function, if you delete it then try to get it, the function will create a new empty dict, store it as its new annotations dict, and return that. Why does it do that? I'm not sure. The relevent PEP (3107) doesn't specify this behavior. So, annotations can be set on three different object types, and each of those three have a different behavior when you delete the annotations then try to get them again. As a final topic: what are the permitted types for __annotations__? If you say "o.__annotations__ = <x>", what types are and aren't allowed for <x>? For functions, __annotations__ may be assigned to either None or a dict (an object that passes PyDict_Check). Anything else throws a TypeError. For classes and modules, no checking is done whatsoever, and you can set __annotations__ on those two to any Python object. While "a foolish consistency is the hobgoblin of little minds", I don't see the benefit of setting a module's __annotations__ to 2+3j. I think it's long past time that we cleaned up the behavior of annotations. They should be simple and consistent across all objects that support them. At the very least, I think we should make cls.__annotations__ required rather than optional, so that it's never inherited. What should its default value be? An empty dict would be more compatible, but None would be cheaper. Note that creating the empty dict on the fly, the way function objects do, wouldn't really help--because current best practice means looking in cls.__dict__. I also think you shouldn't be able to delete __annotations__ on any of the three objects (function, class, module). It should always be set, so that the best practice for accessing annotations on an object is always o.__annotations__. If I could wave my magic wand and do whatever I wanted, I'd change the semantics for __annotations__ to the following: * Functions, classes, and modules always have an __annotations__ member set. * "del o.__annotations__" always throws a TypeError. * The language will set __annotations__ to a dict if the object has annotations, or None if it has no annotations. * You may set __annotations__, but you can only set it to either None or a dict (passes PyDict_Check). * You may only access __annotations__ as an attribute, and because it's always set, best practice is to use "o.__annotations__" (though getattr will always work too). * How __annotations__ is stored is implementation-specific behavior; looking in the relevant __dict__ is unsupported. This would grant sanity and consistency to __annotations__ in a way it's never so far enjoyed. The problem is, it's a breaking change. But the existing semantics are kind of terrible, so at this point my goal is to break them. I think the best practice needs to stop requiring examining cls.__dict__; in fact I'd prefer people stop doing it altogether. If we change the behavior as part of a new release of Python, code that examines annotations on classes can do a version check: if (sys.version_info.major >=3 and sys.version_info.minor >= 10): def get_annotations(o): return o.__annotations__ or {} else: def get_annotations(o): # eight or ten lines of complex code goes here ... Or code can just use inspect.get_type_hints(), which is tied to the Python version anyway and should always do the right thing. //arry/

10 44

PEP 644 Accepted -- Require OpenSSL 1.1.1 or newer
by Pablo Galindo Salgado 17 Apr '21

17 Apr '21

Hi Christian, Thank you for submitting PEP 644 (Require OpenSSL 1.1.1). After evaluating the situation and discussing the PEP, the Steering Council is happy with the PEP, and hereby accepts it. The SC is of the opinion that this change will make it considerable easier to maintain the extension modules that depend on OpenSSL while allowing us to leverage the new APIs and improvements in the future. As indicated in the PEP, OpenSSL 1.1.1 is the default variant and version of OpenSSL on almost all supported platforms and distributions almost all known Major CI providers provide images with OpenSSL 1.1.1, so we believe this has the right balance to improve the situation without causing major impact or breakage. Nevertheless, the Steering Council would like to see this change reflected properly in the documentation, What's New (linking against the new instructions here: https://docs.python.org/3.10/using/unix.html#custom-openssl) and release manager notices so this is properly communicated to our users. The Steering Council also thanks Christian for his patience explaining different aspects of the PEP, including in the PEP some extra requested information and for considering the concerts raised. Congratulations, Christian! With thanks from the whole Python Steering Council, Pablo Galindo Salgado

2 1

Boundaries between numbers and identifiers
by Serhiy Storchaka 15 Apr '21

15 Apr '21

In Python 2.5 `0or[]` was accepted by the Python parser. It became an error in 2.6 because "0o" became recognizing as an incomplete octal number. `1or[]` still is accepted. On other hand, `1if 2else 3` is accepted despites the fact that "2e" can be recognized as an incomplete floating point number. In this case the tokenizer pushes "e" back and returns "2". Shouldn't it do the same with "0o"? It is possible to make `0or[]` be parseable again. Python implementation is able to tokenize this example: $ echo '0or[]' | ./python -m tokenize 1,0-1,1: NUMBER '0' 1,1-1,3: NAME 'or' 1,3-1,4: OP '[' 1,4-1,5: OP ']' 1,5-1,6: NEWLINE '\n' 2,0-2,0: ENDMARKER '' On other hand, all these examples look weird. There is an assymmetry: `1or 2` is a valid syntax, but `1 or2` is not. It is hard to recognize visually the boundary between a number and the following identifier or keyword, especially if numbers can contain letters ("b", "e", "j", "o", "x") and underscores, and identifiers can contain digits. On both sides of the boundary can be letters, digits, and underscores. I propose to change the Python syntax by adding a requirement that there should be a whitespace or delimiter between a numeric literal and the following keyword.

8 10

Request for comments on final version of PEP 653 (Precise Semantics for Pattern Matching)
by Mark Shannon 07 Apr '21

07 Apr '21

Hi everyone, As the 3.10 beta is not so far away, I've cut down PEP 653 down to the minimum needed for 3.10. The extensions will have to wait for 3.11. The essence of the PEP is now that: 1. The semantics of pattern matching, although basically unchanged, are more precisely defined. 2. The __match_kind__ special attribute will be used to determine which patterns to match, rather than relying on the collections.abc module. Everything else has been removed or deferred. The PEP now has only the slightest changes to semantics, which should be undetectable in normal use. For those corner cases where there is a difference, it is to make pattern matching more robust. E.g. With PEP 653, pattern matching will work in the collections.abc module. With PEP 634 it does not. As always, all thoughts and comments are welcome. Cheers, Mark.

14 54

Weird io.OpenWrapper hack to use a function as method
by Victor Stinner 01 Apr '21

01 Apr '21

Hi, The io module provides an open() function. It also provides an OpenWrapper which only exists to be able to store open as a method (class or instance method). In the _pyio module, pure Python implementation of the io module, OpenWrapper is implemented as: class OpenWrapper: """Wrapper for builtins.open Trick so that open won't become a bound method when stored as a class variable (as dbm.dumb does). See initstdio() in Python/pylifecycle.c. """ def __new__(cls, *args, **kwargs): return open(*args, **kwargs) I would like to remove this class which is causing troubles in the PEP 597 implementation, but I don't know how. Simplified problem: --- def func(): print("my func") class MyClass: method = func func() # A MyClass.method() # B obj = MyClass() obj.method() # C --- With this syntax, A and B work, but C fails with TypeError: func() takes 0 positional arguments but 1 was given. If I decorate func() with @staticmethod, B and C work, but A fails with TypeError: 'staticmethod' object is not callable. Is OpenWrapper the only way to have a callable object which works in the 3 variants A, B and C? A, B and C work if MyClass is modified to use staticmethod: class MyClass: method = staticmethod(func) Victor -- Night gathers, and now my watch begins. It shall not end until my death.

6 9

Re: PEP-376 and PEP-427 interpretation
by Theallredman 31 Mar '21

31 Mar '21

No need to be condescending. Trust me when I say I know the bit length relates to the collision resistance. Also trust me when I say there are other dimensions upon which to consider one hash algo over another other then just collision resistance such as, power consumption, execution time, whether or not the algorithm suffers from length extension attacks. I'm assuming the reason MD5 and SHA1 were both disallowed were because they have been proven to have a collision resistance less then 1/2 their bit length. But this is not the case for SHA224. It is just a truncated version of SHA256 and thus the underlying algorithm is just as strong as SHA256 except that you can expect to find a collision in about 16 bits of work less. So going back to my actual question SHA224 is disallowed in record files because it's bit length is less then 256?

4 4

Status of PEP 649 -- Deferred Evaluation Of Annotations Using Descriptors
by Guido van Rossum 29 Mar '21

29 Mar '21

Hi Larry, Can you give us a status update for your PEP 649? I don't recall reading anything about it in the past few weeks. I am super excited about this solution to the problem (even if there are a few issues to work through) and I think it will provide better backwards compatibility than the current plan for Python 3.10 (PEP 563, from __future__ import annotations, causing all annotations to be stringified). If we don't get this into 3.10, we'd have a much more complicated transition. There are only two more alphas before 3.10b1 gets released! And we'd have to get this approved by the Steering Council, which can take a few weeks (cut them some slack, they have a big backlog). Fortunately you already have an implementation that can be landed quickly once the PEP is accepted. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-…>

3 6

Where and how can I contribute?
by Marco Sulla 27 Mar '21

27 Mar '21

I would contribute to the project in my spare time. Can someone point me to some easy task? I know C and the Python C API a little.

5 5

PEP-376 and PEP-427 interpretation
by Theallredman 26 Mar '21

26 Mar '21

Forgive me if this isn't the correct venue for this question and I ask your help directing me to the correct place if it is not. In PEP-376 it states with respect to the valid hashes in a Wheel RECORD file: "The hash is either the empty string or the hash algorithm as named in hashlib.algorithms_guaranteed, followed by the equals character =, followed by the urlsafe-base64-nopad encoding of the digest (base64.urlsafe_b64encode(digest) with trailing = removed)." In PEP-427 it further restricts the valid hashes to omit md5 and sha1 and says: "The hash algorithm must be sha256 or better; specifically, md5 and sha1 are not permitted." No where does it define what on what dimension of a hash "better" should consider. From the context talking about the security of the algorithm I'd infer that "better" is with respect to collision resistance. If so does that mean sha224 should also be excluded from wheel RECORD file hashes? Eldon

2 1