Python-Dev March 2021

python-dev@python.org

96 participants
59 discussions

Boundaries between numbers and identifiers

by Serhiy Storchaka

In Python 2.5 `0or[]` was accepted by the Python parser. It became an error in 2.6 because "0o" became recognizing as an incomplete octal number. `1or[]` still is accepted. On other hand, `1if 2else 3` is accepted despites the fact that "2e" can be recognized as an incomplete floating point number. In this case the tokenizer pushes "e" back and returns "2". Shouldn't it do the same with "0o"? It is possible to make `0or[]` be parseable again. Python implementation is able to tokenize this example: $ echo '0or[]' | ./python -m tokenize 1,0-1,1: NUMBER '0' 1,1-1,3: NAME 'or' 1,3-1,4: OP '[' 1,4-1,5: OP ']' 1,5-1,6: NEWLINE '\n' 2,0-2,0: ENDMARKER '' On other hand, all these examples look weird. There is an assymmetry: `1or 2` is a valid syntax, but `1 or2` is not. It is hard to recognize visually the boundary between a number and the following identifier or keyword, especially if numbers can contain letters ("b", "e", "j", "o", "x") and underscores, and identifiers can contain digits. On both sides of the boundary can be letters, digits, and underscores. I propose to change the Python syntax by adding a requirement that there should be a whitespace or delimiter between a numeric literal and the following keyword.

1 hour, 35 minutes

PEP 654: Exception Groups and except* [REPOST]

by Irit Katriel

We would like to present for feedback a new version of PEP 654, which incorporates the feedback we received in the discussions so far: https://www.python.org/dev/peps/pep-0654/ The reference implementation has also been updated along with the PEP. The changes we made since the first post are: 1. Instead of ExceptionGroup(BaseException), we will have two new builtin types: BaseExceptionGroup(BaseException) and ExceptionGroup(BaseExceptionGroup, Exception). This is so that "except Exception" catches ExceptionGroups (but not BaseExceptionGroups). BaseExceptionGroup.__new__ inspects the wrapped exceptions, and if they are all Exception subclasses, it creates an ExceptionGroup instead of a BaseExceptionGroup. 2. The exception group classes are not final - they can be subclassed and split()/subgroup() work correctly if the subclass overrides the derive() instance method as described here: https://www.python.org/dev/peps/pep-0654/#subclassing-exception-groups 3. We had some good suggestions on formatting exception groups, which we have implemented as you can see in the output shown for the examples in the PEP. 4. We expanded the section on handling Exception Groups, to show how subgroup can be used (with side effects) to do something for each leaf exception, and how to iterate correctly when the tracebacks of leaf exceptions are needed: https://www.python.org/dev/peps/pep-0654/#handling-exception-groups 5. We expanded the sections on rationale and backwards compatibility to explain our premise and expectations regarding how exception groups will be used and how the transition to using them will be managed. 6. We added several items to the rejected ideas section. We did not receive any comments (or make any changes) to the proposed semantics of except*, hopefully this is because everyone thought they are sensible. Irit, Yury and Guido

6 days, 9 hours

Request for comments on final version of PEP 653 (Precise Semantics for Pattern Matching)

by Mark Shannon

Hi everyone, As the 3.10 beta is not so far away, I've cut down PEP 653 down to the minimum needed for 3.10. The extensions will have to wait for 3.11. The essence of the PEP is now that: 1. The semantics of pattern matching, although basically unchanged, are more precisely defined. 2. The __match_kind__ special attribute will be used to determine which patterns to match, rather than relying on the collections.abc module. Everything else has been removed or deferred. The PEP now has only the slightest changes to semantics, which should be undetectable in normal use. For those corner cases where there is a difference, it is to make pattern matching more robust. E.g. With PEP 653, pattern matching will work in the collections.abc module. With PEP 634 it does not. As always, all thoughts and comments are welcome. Cheers, Mark.

6 days, 14 hours

PEP 646 (Variadic Generics): final call for comments

by Matthew Rahtz

Hi everyone, We've got to the stage now with PEP 646 that we're feeling pretty happy with it. So far though we've mainly been workshopping it in typing-sig, so as PEP 1 requires we're asking for some feedback here too before submitting it to the steering council. If you have time over the next couple of weeks, please take a look at the current draft and let us know your thoughts: https://www.python.org/dev/peps/pep-0646/ (Note that the final couple of sections are out of date; https://github.com/python/peps/pull/1880 clarifies which grammar changes would be required, now that PEP 637 has been rejected. We also have a second PR in progress at https://github.com/python/peps/pull/1881 clarifying some of the motivation.) Thanks! Matthew and Pradeep

1 week

SC feedback: PEP 648 -- Extensible customizations of the interpreter at startup

by Barry Warsaw

Hello Mario, Thank you for your submission of PEP 648 (Extensible customizations of the interpreter at startup). The Python Steering Council has reviewed the PEP and before we can pronounce on it, we have some additional questions and comments we’d like you to address. Once these questions are settled, we are requesting that you post the PEP to python-dev for another round of comments. In general, the SC is in favor of deprecating the executable hack capabilities of pth files, and this PEP is currently the most concrete proposal to get there. We would like to eventually go farther, including deprecation of pth files entirely, but that is outside the scope of this PEP. General PEP feedback The introduction of the section titled “Benefits of __sitecustomize__” seems out of place. It forward references the solution without explanation. It also says “The use of a __sitecustomize__ will allow…”; the question is, a __sitecustomize__ what? Directory? File? Perhaps it should be moved to later in the PEP after the semantics of __sitecustomize__ is defined? Some of the terminology used in the PEP could be clarified. For example, I suggest using the term “directory” instead of “folder” throughout the PEP, and use the term “module” or “file” instead of “script” to describe the things inside __sitecustomize__ directories that Python imports, since “scripts” typically describe standalone files that implement applications. For example, must the files be named with .py extension (or presumably .pyc, .pyo) under the normal Python module import rules? (Later in the PEP you do allude to the requirement that the file name must end in .py, but it would be much better to explain this right up front, in a formal specification.) Another ambiguity is what the PEP means by “executing” said “scripts”. Does that mean importing them? Reading them then exec()’ing them? Something else? This section should be clear that Python will import the modules it found, if that’s indeed what the PEP is proposing. The “Backward compatibility” section has this incomplete sentence: “Ignoring those lines in pth files.” There’s this odd grammar in the “Do nothing” section: “After analysing the impact of this change, we believe it is worth given the enhanced experience it brings.” Perhaps it should read “...it is worth it, given the…”? Definition of “site path” The PEP should precisely define what it means when it says “...a folder named __sitecustomize__ located in any site path”. What exactly is a “site path”? Is this any directory on sys.path? Is it a directory named site-packages? This phrase is also ambiguous: “As the folder will be within sys.path, given that it is located in site paths...”. “sys.path” isn’t a thing that can hold a folder, since it’s just a list of strings, and it also can get extended by any number of means, e.g. by setting the PYTHONPATH environment variable at interpreter startup time. So for example, how does PYTHONPATH affect the algorithm? What is the rationale for adding multiple __sitecustomize__ directories, rather than a single directory (or possibly two, one in Python’s system location and one in the user location)? This would simplify the discovery process, and tools like pip can warn the user if a name collision were to occur. I think the PEP needs to be much more crisp and precise in defining the when and how __sitecustomize__ directories are discovered at start up time. Order of Execution I think this section should outline the entire startup execution order. Exactly when do pth files still get evaluated in relationship to __sitecustomize__ discovery? How do pth file sys.path extensions affect the search for __sitecustomize__ directories? This section also says files in __sitecustomize__ will be executed in “file name sorted order”, but the PEP is unclear whether all such files found in every __sitecustomize__ directory will be sorted together, or only within a single __sitecustomize__ directory. Presumably file system encodings are used to determine file name sort order, but I think the PEP should be explicit about this. I’d like to see the execution order for the entire proposed new Python startup sequence as a recipe, or perhaps even some Python pseudo-code. Impact on startup time We still think that startup performance can be a concern. This section needs more data. For example, while the executable part of pth files is being deprecated, both pth and __sitecustomize__ modules will exist. How does this affect startup time in a world where both have to be discovered and processed? The PEP says: “If the user has custom scripts, we think that the impact on the performance of walking each of the folders is acceptable, as the user wants to use this feature.” Does the user really want to use this feature? Won’t it more likely be the system administrator or Python distribution and packaging ecosystem that chooses to use the __sitecustomize__ feature? Maybe the user won’t even know that it’s being used by the packages being installed? The PEP says: ““Running "./python -c pass" with perf on 50 iterations, repeating 50 times the command on each and getting the geometric mean on a commodity laptop did not reveal any substantial raise on CPU time.” We request more information on the experiments performed, and specific numbers so that the startup impact can be reproduced and confirmed by others. E.g. did you run this experiment on a branch of CPython 3.10 with the PEP 648 reference implementation? Did you convert existing uses of executable pth files to __sitecustomize__ modules? How many such modules did you try it with? How many __sitecustomize__ directories were in your tests? Other questions and issues How does this feature interact with virtual environments? What do packaging tools have to do to support this PEP? The PEP mentions setuptools, but what about other packaging tools in common use? The PEP says “This impact will be reduced in the future as we will remove two other imports: "sitecustomize.py" and "usercustomize.py”.” Why not explicitly deprecate these and executable pth files in PEP 648 and begin the countdown at the same time __sitecustomize__ is added to Python? The PEP says: “To facilitate debugging of the Python startup, a new option will be added to the main of the site module to list all scripts that will be executed as part of the __sitecustomize__ initialization.” The PEP should be explicit about what command line switch is being proposed. We think this should probably be an -X option. Do you have any concerns that this feature will be overused? Does separating the execution feature from the path extension feature make it more or less likely to be used by third party library authors? Will the runtime have any access to the list of __sitecustomize__ modules being executed? I.e. will you collect their paths in a new sys module attribute? Will there be any functions to import or list __sitecustomize__ directories or modules after start has completed? We think io.open_code() should be used so that reading the files can be audited. Conclusion The Python Steering Council again wants to thank you for your PEP contribution! We hope you find this feedback constructive and helpful. We look forward to your responses, and an updated PEP. Cheers, -Barry (on behalf of the Python Steering Council)

1 week, 4 days

Weird io.OpenWrapper hack to use a function as method

by Victor Stinner

Hi, The io module provides an open() function. It also provides an OpenWrapper which only exists to be able to store open as a method (class or instance method). In the _pyio module, pure Python implementation of the io module, OpenWrapper is implemented as: class OpenWrapper: """Wrapper for builtins.open Trick so that open won't become a bound method when stored as a class variable (as dbm.dumb does). See initstdio() in Python/pylifecycle.c. """ def __new__(cls, *args, **kwargs): return open(*args, **kwargs) I would like to remove this class which is causing troubles in the PEP 597 implementation, but I don't know how. Simplified problem: --- def func(): print("my func") class MyClass: method = func func() # A MyClass.method() # B obj = MyClass() obj.method() # C --- With this syntax, A and B work, but C fails with TypeError: func() takes 0 positional arguments but 1 was given. If I decorate func() with @staticmethod, B and C work, but A fails with TypeError: 'staticmethod' object is not callable. Is OpenWrapper the only way to have a callable object which works in the 3 variants A, B and C? A, B and C work if MyClass is modified to use staticmethod: class MyClass: method = staticmethod(func) Victor -- Night gathers, and now my watch begins. It shall not end until my death.

1 week, 5 days

Re: PEP-376 and PEP-427 interpretation

by Theallredman

No need to be condescending. Trust me when I say I know the bit length relates to the collision resistance. Also trust me when I say there are other dimensions upon which to consider one hash algo over another other then just collision resistance such as, power consumption, execution time, whether or not the algorithm suffers from length extension attacks. I'm assuming the reason MD5 and SHA1 were both disallowed were because they have been proven to have a collision resistance less then 1/2 their bit length. But this is not the case for SHA224. It is just a truncated version of SHA256 and thus the underlying algorithm is just as strong as SHA256 except that you can expect to find a collision in about 16 bits of work less. So going back to my actual question SHA224 is disallowed in record files because it's bit length is less then 256?

1 week, 6 days

PEP 644 Accepted -- Require OpenSSL 1.1.1 or newer

by Pablo Galindo Salgado

Hi Christian, Thank you for submitting PEP 644 (Require OpenSSL 1.1.1). After evaluating the situation and discussing the PEP, the Steering Council is happy with the PEP, and hereby accepts it. The SC is of the opinion that this change will make it considerable easier to maintain the extension modules that depend on OpenSSL while allowing us to leverage the new APIs and improvements in the future. As indicated in the PEP, OpenSSL 1.1.1 is the default variant and version of OpenSSL on almost all supported platforms and distributions almost all known Major CI providers provide images with OpenSSL 1.1.1, so we believe this has the right balance to improve the situation without causing major impact or breakage. Nevertheless, the Steering Council would like to see this change reflected properly in the documentation, What's New (linking against the new instructions here: https://docs.python.org/3.10/using/unix.html#custom-openssl) and release manager notices so this is properly communicated to our users. The Steering Council also thanks Christian for his patience explaining different aspects of the PEP, including in the PEP some extra requested information and for considering the concerts raised. Congratulations, Christian! With thanks from the whole Python Steering Council, Pablo Galindo Salgado

2 weeks

Status of PEP 649 -- Deferred Evaluation Of Annotations Using Descriptors

by Guido van Rossum

Hi Larry, Can you give us a status update for your PEP 649? I don't recall reading anything about it in the past few weeks. I am super excited about this solution to the problem (even if there are a few issues to work through) and I think it will provide better backwards compatibility than the current plan for Python 3.10 (PEP 563, from __future__ import annotations, causing all annotations to be stringified). If we don't get this into 3.10, we'd have a much more complicated transition. There are only two more alphas before 3.10b1 gets released! And we'd have to get this approved by the Steering Council, which can take a few weeks (cut them some slack, they have a big backlog). Fortunately you already have an implementation that can be landed quickly once the PEP is accepted. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-…>

2 weeks, 1 day

Where and how can I contribute?

by Marco Sulla

I would contribute to the project in my spare time. Can someone point me to some easy task? I know C and the Python C API a little.

2 weeks, 3 days

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

Python-Dev March 2021