Mailman 3 March 2022 - Python-Dev

Proposal: Allow non-default after default arguments
by Sebastian Rittau Oct. 16, 2023

Oct. 16, 2023

Currently, Python doesn't allow non-default arguments after default arguments: >>> def foo(x=None, y): pass File "<stdin>", line 1 def foo(x=None, y): pass ^ SyntaxError: non-default argument follows default argument I believe that at the time this was introduced, no use cases for this were known and this is is supposed to prevent a source of bugs. I have two use cases for this, one fringe, but valid, the other more important: The fringe use case: Suppose you have a function that takes a 2D coordinate value as separate "x" and "y" arguments. The "x" argument is optional, the "y" argument isn't. Currently there are two ways to do this, none of them particularly great: def foo(y, x): # reverse x and y arguments, confusing ... def foo(x, y=None): # Treat the x argument as y if only one argument is provided if y is None: x, y = y, x ... To me, the "natural" solution looks like this: def foo(x=None, y): ... # Called like this: foo(1, 2) foo(y=2) This could also be useful when evolving APIs. For example there is a function "bar" that takes two required arguments. In a later version, the first argument gains a useful default, the second doesn't. There is no sensible way to evolve the API at the moment. The more important use case involves @overloads. A condensed example of a function where the return type depends on an "encoding" parameter, followed by further parameters that could be called like this: foo(123, "utf-8") # would return bytes foo(encoding="utf-8") foo(123, None) # would return str foo(encoding=None) foo(x=123) # would return str foo() This could ideally be written as: @overload def foo(x: int = ..., encoding: None = ...) -> str: ... @overload def foo(x: int = ..., encoding: str) -> bytes: ... # plus the actual implementation But due to the syntax constraint, this needs to be hacked around with a third overload: @overload def foo(x: int = ... encoding: None = ...) -> str: ... @overload def foo(x: int, encoding: str) -> bytes: ... # for foo(123, "utf-8") @overload def foo(*, encoding: str) -> bytes: ... # for foo(encoding="utf-8") Not only is this hard to read, real examples in typeshed are usually more complex, with many arguments before or after the affected argument or even multiple affected arguments. This often becomes too complex to write or maintain. Here is one example from the wild: https://github.com/python/typeshed/blob/b95b729b9e07ab21d252701af0f5b740467… Allowing non-default arguments after default arguments would solve both use cases above and eliminates a special case. I'm also not sure what exactly the current SyntaxError really protects us from. Adding a non-default after a default argument can't really lead bugs. - Sebastian

11 14

Python multithreading without the GIL
by Sam Gross July 31, 2023

July 31, 2023

Hi, I've been working on changes to CPython to allow it to run without the global interpreter lock. I'd like to share a working proof-of-concept that can run without the GIL. The proof-of-concept involves substantial changes to CPython internals, but relatively few changes to the C-API. It is compatible with many C extensions: extensions must be rebuilt, but usually require small or no modifications to source code. I've built compatible versions of packages from the scientific Python ecosystem, and they are installable through the bundled "pip". Source code: https://github.com/colesbury/nogil Design overview: https://docs.google.com/document/d/18CXhDb1ygxg-YXNBJNzfzZsDFosB5e6BfnXLlej… My goal with the proof-of-concept is to demonstrate that removing the GIL is feasible and worthwhile, and that the technical ideas of the project could serve as a basis of such an effort. I'd like to start a discussion about these ideas and gauge the community's interest in this approach to removing the GIL. Regards, Sam Gross colesbury(a)gmail.com / sgross(a)fb.com

24 48

Mixed Python/C debugging
by Skip Montanaro April 26, 2023

April 26, 2023

Having tried comp.lang.python with no response, I turn here... After at least ten years away from Python's run-time interpreter & byte code compiler, I'm getting set to familiarize myself with that again. This will, I think, entail debugging a mixed Python/C environment. I'm an Emacs user and am aware that GDB since 7.0 has support for debugging at the Python code level. Is Emacs+GDB my best bet? Are there any Python IDEs which support C-level breakpoints and debugging? Thanks, Skip

5 6

PEP 448 review
by Guido van Rossum March 29, 2023

March 29, 2023

I'm back, I've re-read the PEP, and I've re-read the long thread with "(no subject)". I think Georg Brandl nailed it: """ *I like the "sequence and dict flattening" part of the PEP, mostly because itis consistent and should be easy to understand, but the comprehension syntaxenhancements seem to be bad for readability and "comprehending" what the codedoes.The call syntax part is a mixed bag on the one hand it is nice to be consistent with the extended possibilities in literals (flattening), but on the other hand there would be small but annoying inconsistencies anyways (e.g. the duplicate kwarg case above).* """ Greg Ewing followed up explaining that the inconsistency between dict flattening and call syntax is inherent in the pre-existing different rules for dicts vs. keyword args: {'a':1, 'a':2} results in {'a':2}, while f(a=1, a=2) is an error. (This form is a SyntaxError; the dynamic case f(a=1, **{'a': 1}) is a TypeError.) For me, allowing f(*a, *b) and f(**d, **e) and all the other combinations for function calls proposed by the PEP is an easy +1 -- it's a straightforward extension of the existing pattern, and anybody who knows what f(x, *a) does will understand f(x, *a, y, *b). Guessing what f(**d, **e) means shouldn't be hard either. Understanding the edge case for duplicate keys with f(**d, **e) is a little harder, but the error messages are pretty clear, and it is not a new edge case. The sequence and dict flattening syntax proposals are also clean and logical -- we already have *-unpacking on the receiving side, so allowing *x in tuple expressions reads pretty naturally (and the similarity with *a in argument lists certainly helps). From here, having [a, *x, b, *y] is also natural, and then the extension to other displays is natural: {a, *x, b, *y} and {a:1, **d, b:2, **e}. This, too, gets a +1 from me. So that leaves comprehensions. IIRC, during the development of the patch we realized that f(*x for x in xs) is sufficiently ambiguous that we decided to disallow it -- note that f(x for x in xs) is already somewhat of a special case because an argument can only be a "bare" generator expression if it is the only argument. The same reasoning doesn't apply (in that form) to list, set and dict comprehensions -- while f(x for x in xs) is identical in meaning to f((x for x in xs)), [x for x in xs] is NOT the same as [(x for x in xs)] (that's a list of one element, and the element is a generator expression). The basic premise of this part of the proposal is that if you have a few iterables, the new proposal (without comprehensions) lets you create a list or generator expression that iterates over all of them, essentially flattening them: >>> xs = [1, 2, 3] >>> ys = ['abc', 'def'] >>> zs = [99] >>> [*xs, *ys, *zs] [1, 2, 3, 'abc', 'def', 99] >>> But now suppose you have a list of iterables: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xss[0], *xss[1], *xss[2]] [1, 2, 3, 'abc', 'def', 99] >>> Wouldn't it be nice if you could write the latter using a comprehension? >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs for xs in xss] [1, 2, 3, 'abc', 'def', 99] >>> This is somewhat seductive, and the following is even nicer: the *xs position may be an expression, e.g.: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs[:2] for xs in xss] [1, 2, 'abc', 'def', 99] >>> On the other hand, I had to explore the possibilities here by experimenting in the interpreter, and I discovered some odd edge cases (e.g. you can parenthesize the starred expression, but that seems a syntactic accident). All in all I am personally +0 on the comprehension part of the PEP, and I like that it provides a way to "flatten" a sequence of sequences, but I think very few people in the thread have supported this part. Therefore I would like to ask Neil to update the PEP and the patch to take out the comprehension part, so that the two "easy wins" can make it into Python 3.5 (basically, I am accepting two-thirds of the PEP :-). There is some time yet until alpha 2. I would also like code reviewers (Benjamin?) to start reviewing the patch <http://bugs.python.org/issue2292>, taking into account that the comprehension part needs to be removed. -- --Guido van Rossum (python.org/~guido)

7 17

PEP 638: Syntactic macros
by Mark Shannon Feb. 5, 2023

Feb. 5, 2023

Hi everyone, I've submitted my PEP on syntactic macros as PEP 638. https://www.python.org/dev/peps/pep-0638/ All comments and suggestions are welcome. Cheers, Mark

16 25

How about using modern C++ in development of CPython ?
by redradist＠gmail.com Oct. 19, 2022

Oct. 19, 2022

Hi all, What about of using modern C++ in developing CPython ? With C++ we can get the following benefits: 1) Readability - less code, most code is hidden by abstraction without losing performance 2) Maintainability - no code duplication in favor of using reusable classes 3) RAII - Resource Acquisition Is Initialization, predictable allocation and free resources ...

23 51

PEP 505 (None-aware operators) for Python 3.11
by Doug Swarin Sept. 21, 2022

Sept. 21, 2022

Hello, I've been following PEP 505 (https://www.python.org/dev/peps/pep-0505/) since it was first proposed for Python 3.8. It's been deferred for some time and I'm interested in seeing it in Python 3.11, but I know there were also a number of objections which resulted in it being deferred (including by one of the original authors, Mr. Dower). I did email both Mr. Dower and Mr. Haase and they graciously gave me their permission to bring it up on this list for discussion and hopefully final pronouncement one way or the other. I personally believe that the PEP will result in a significant reduction in boilerplate code, and it is substantially similar to the same operators now found in a number of other languages, especially C# and JavaScript (https://wikipedia.org/wiki/Null_coalescing_operator and https://wikipedia.org/wiki/Safe_navigation_operator). I believe strong and valid arguments can be made about the use of None being a fundamental flaw in some types of coding (and that adding additional support for it to the language will increase the use of None in this way), but I also believe there are many use cases in programming where it is by far the simplest way to express various semantics, and the fact exists that None is already used extensively in large quantities of code, and further that there is already a great deal of code written to constantly test against None and break out of a statement without throwing an error. I also understand the argument that especially the maybe-dot (?.) and maybe-subscript (?[) operators can decrease readability of code and also believe these are valid arguments against it. While I believe the existence and use of these operators in other languages definitely helps the case that these can be used and understood successfully, I think it is entirely valid to either consider other syntax (though I prefer the chosen syntax of PEP 505), or even to reduce PEP 505 to having only the coalesce operator (??) and the maybe-assign operator (??=). Separately, I have implemented a pure-Python solution for PEP505 (which is definitely rather beta) which might help test the waters for a final implementation in CPython (though the CPython implementation would of course be much more efficient). It can be found at https://pypi.org/project/pep505/ Thanks, Doug

22 53

PEP 563 and Python 3.10.
by Thomas Wouters July 22, 2022

July 22, 2022

(Starting a new thread so as not to derail any of the ongoing discussions.) Thanks, everyone, for your thoughts on Python 3.10 and the impact of PEP 563 (postponed evaluation of annotations) becoming the default. The Steering Council has considered the issue carefully, along with many of the proposed alternatives and solutions, and we’ve decided that at this point, we simply can’t risk the compatibility breakage of PEP 563. We need to roll back the change that made stringified annotations the default, at least for 3.10. (Pablo is already working on this.) To be clear, we are not reverting PEP 563 itself. The future import will keep working like it did since Python 3.7. We’re delaying making PEP 563 string-based annotations the default until Python 3.11. This will give us time to find a solution that works for everyone (or to find a feasible upgrade path for users who currently rely on evaluated annotations). Some considerations that led us to this decision: - PEP 563’s default change is clearly too disruptive to downstream users and third-party libraries to happen right now. We can’t risk breaking even a small subset of the FastAPI/pydantic users, not to mention other uses of evaluated type annotations that we’re not aware of yet. - PEP 563 provides no warning to users of the feature it’s disabling. Without that, we can’t expect users to be aware of the upcoming breakage. The lack of a warning was by design, and made sense in a world where type annotations were only consumed by static type checkers --- but that’s not actually the situation we’re in. There are clearly existing real-world, run-time uses of type annotations that would be adversely affected by this change. - Originally, PEP 563 was scheduled to take effect in Python 4, and this changed recently (after the discussion in the Language Summit of 2020). It's possible that third-party libraries and users didn’t plan to react in the current time frame as they were not aware of this change in timing. - There isn’t enough time to properly discuss PEP 649 or any of the alternatives before the beta 1 deadline, and we really need to make sure we don’t compound errors here. We need to look for a long term solution, which isn’t possible while still maintaining the release deadlines of Python 3.10. That means we’re also deferring PEP 649 to Python 3.11. In the Steering Council’s unanimous opinion, rolling back the default flip for stringified annotations in Python 3.10 is the least disruptive of all the options. We need to continue discussing the issue and potential solutions, since this merely postpones the problem until 3.11. (For the record, postponing the change further is not off the table, either, for example if the final decision is to treat evaluated annotations as a deprecated feature, with warnings on use.) For what it’s worth, the SC is also considering what we can do to reduce the odds of something like this happening again, but that’s a separate consideration, and a multi-faceted one at that. For the Steering Council, Thomas. -- Thomas Wouters <thomas(a)python.org> Hi! I'm an email virus! Think twice before sending your email to help me spread!

6 5

compiled python3.10 is unable to find _ssl
by Sandeep Gupta May 29, 2022

May 29, 2022

I am having compilation issues again with python3.10 with ssl . The ./configure was invoked with ssl options and ssl modules seems to be build successfully. """ The following modules found by detect_modules() in setup.py, have been built by the Makefile instead, as configured by the Setup files: _abc _hashlib _ssl pwd time """ However, when I do import ssl from python, I get the error: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/shared/Builds/Python-3.10.0/lib/python3.10/ssl.py", line 98, in <module> import _ssl # if we can't import it, let the error propagate ModuleNotFoundError: No module named '_ssl' I can't find _ssl.so in build or install directory. Thanks -S

6 12

C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API
by Victor Stinner April 25, 2022

April 25, 2022

Hi, I proposed two PRs to move the private C API (Include/cpython/) of PEP 523 "Adding a frame evaluation API to CPython" to the internal C API (Include/internals/): * https://github.com/python/cpython/pull/32052 * https://github.com/python/cpython/pull/32054 API: * _PyFrameEvalFunction type * _PyInterpreterState_GetEvalFrameFunc() * _PyInterpreterState_SetEvalFrameFunc() * (undocumented) _PyEval_EvalFrameDefault() The private API to get/set the eval function *is* documented at: https://docs.python.org/dev/c-api/init.html#c._PyInterpreterState_GetEvalFr… I added the Get/Set functions so debuggers don't have to access directly to the PyInterpreterState structure which has been moved to the internal C API in Python 3.8. This API causes me multiple issues: * It's a private API and I'm trying to remove the private API from the public C API header files. * The _PyFrameEvalFunction type is not stable: it got a new "tstate" parameter in Python 3.9 and the type of the second parameter changed from PyFrameObject* to _PyInterpreterFrame* in Python 3.11. * These functions use the _PyInterpreterFrame type which is part of the internal C API. While Pyston didn't bring a JIT compiler to Python with PEP 523, debuggers were made faster by using this API. Debuggers like pydevd, debugpy and ptvsd use it. I propose to move theses API to the internal header files (Include/internals/) to clarify that it's not part of the public C API and that there is no backward compatibility warranty. The change is being discussed at: https://bugs.python.org/issue46850 -- PEP 523 API added more private functions for code objects: * _PyEval_RequestCodeExtraIndex() * _PyCode_GetExtra() * _PyCode_SetExtra() The _PyEval_RequestCodeExtraIndex() function seems to be used by the pydevd debugger. The two others seem to be unused in the wild. I'm not sure if these ones should be moved to the internal C API. They can be left unchanged, since they don't use a type only defined by the internal C API. Victor -- Night gathers, and now my watch begins. It shall not end until my death.

7 20

Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API
by Jason Ansel April 20, 2022

April 20, 2022

The PyTorch team plans to use PEP 523 as a part of PyTorch 2.0, so this proposal may break the next major release of PyTorch. The related project is TorchDynamo, which can be found here: https://github.com/facebookresearch/torchdynamo We will likely move this into the core of PyTorch closer to release. If the changed happens, would PyTorch still be able to use the eval frame API? Or would it prevent from being used entirely?

11 25

Slowly bend the C API towards the limited API to get a stable ABI for everyone
by Victor Stinner April 20, 2022

April 20, 2022

Hi, There is a reason why I'm bothering C extensions maintainers and Python core developers with my incompatible C API changes since Python 3.8. Let me share my plan with you :-) In 2009 (Python 3.2), Martin v. Löwis did an amazing job with the PEP 384 "Defining a Stable ABI" to provide a "limited C API" and a "stable ABI" for C extensions: build an extension once, use it on multiple Python versions. Some projects like PyQt5 and cryptograpy use it, but it is just a drop in the PyPI ocean (353,084 projects). I'm trying to bend the "default" C API towards this "limited C API" to make it possible tomorrow to build *more* C extensions for the stable ABI. My goal is that the stable ABI would be the default, and only a minority of C extensions would opt-out because they need to access to more functions for best performance. The basic problem is that at the ABI level, C extensions must only call functions, rather than getting and setting directly to structure members. Structures changes frequently in Python (look at changes between Python 3.2 and Python 3.11), and any minor structure change breaks the ABI. The limited C API hides structures and only use function calls to solve this problem. Since 2020, I'm modifying the C API, one function by one, to slowly hide implementations (prepare the API to make strutures opaque). I focused on the following structures: * PyObject and PyVarObject (bpo-39573) * PyTypeObject (bpo-40170) * PyFrameObject (bpo-40421) * PyThreadState (bpo-39947) The majority of C extensions use functions and macros, they don't access directly structure members. There are a few members which are sometimes accessed directly which prevents making these structures opaque. For example, some old C extensions use obj->ob_type rather than Py_TYPE(obj). Fixing the minority of C extensions should benefit to the majority which may become compatible with the stable ABI. I am also converting macros to static inline functions to fix their API: define parameter types, result type and avoid surprising macros side effects ("macro pitfalls"). I wrote the PEP 670 "Convert macros to functions in the Python C API" for these changes. I wrote the upgrade_pythoncapi.py tool in my pythoncapi_project (*) which modify C code to use Py_TYPE(), Py_SIZE() and Py_REFCNT() rather than accessing directly PyObject and PyVarObject members. (*) https://github.com/pythoncapi/pythoncapi_compat In this tool, I also added "Borrow" variant of functions like PyFrame_GetCode() which returns a strong reference, to replace frame->f_code with _PyFrame_GetCodeBorrow(). In Python 3.11, you cannot use the frame->f_code member anymore, since it has been removed! You must call PyFrame_GetCode() (or pythoncapi_compat _PyFrame_GetCodeBorrow() variant). There are also a few macros which can be used as l-values like Py_TYPE(): "Py_TYPE(type1) = type2" must now be written "Py_SET_TYPE(type1, type2)" to avoid setting directly the tp_type type at the ABI level. I proposed the PEP 674 "Disallow using Py_TYPE() and Py_SIZE() macros as l-values" to solve these issues. Currently, many "functions" are still implemented as macros or static inline functions, so C extensions still access structure members at the ABI level for best Python performance. Converting these to regular functions has an impact on performance and I would prefer to first write a PEP giving the rationale for that. Today, it is not possible yet to build numpy for the stable ABI. The gap is just too large for this big C extension. But step by step, the C API becomes closer to the limited API, and more and more code is ready to be built for the stable ABI. Well, these C API changes have other advantages, like preparing Python for further optimizations, ease Python maintenance, clarify the seperation between the limited C API and the default C API, etc. ;-) Victor -- Night gathers, and now my watch begins. It shall not end until my death.

12 38

PEP 678: Enriching Exceptions with Notes
by Zac Hatfield Dodds April 12, 2022

April 12, 2022

Hi all, I've written PEP 678, proposing the addition of a __note__ attribute which can be used to enrich displayed exceptions. This is particularly useful with ExceptionGroup, or where exception chaining is unsuitable, and ensures that 'orphan' output is not produced if the annotated exception is caught. Link to the PEP: https://www.python.org/dev/peps/pep-0678/ *Please, redirect all discussions to:* https://discuss.python.org/t/pep-678-enriching-exceptions-with-notes/13374 Thanks for your time, and to those who have already made (very helpful!) comments! Best wishes from just-moved-to San Francisco, Zac Hatfield-Dodds

2 1

It's now time to deprecate the stdlib urllib module
by Victor Stinner March 31, 2022

March 31, 2022

Hi, I propose to deprecate the urllib module in Python 3.11. It would emit a DeprecationWarning which warn users, so users should consider better alternatives like urllib3 or httpx: well known modules, better maintained, more secure, support HTTP/2 (httpx), etc. I don't propose to schedule its removal. Let's discuss the removal in 1 or 2 years. -- urllib has many abstraction to support a wide range of protocols with "handlers": HTTP, HTTPS, FTP, "local file", proxy, HTTP authentication, HTTP Cookie, etc. A simple HTTP request using Basic Authentication requires 10-20 lines of code, whereas it should be a single line. Users (me included) don't like urllib API which was too complicated for common tasks. -- Unhappy users created multiple better alternatives to the stdlib urllib module. In 2008, the "urllib3" module was created to provide an API designed to be as simple as possible for the most common HTTP and HTTPS requests. Example: req = http.request('GET', 'http://httpbin.org/robots.txt'). In 2011, the "requests" module based on urllib3 was created. In 2013, the "aiohttp" module based on asyncio was created. In 2015, new "httpx" module was created: req = httpx.get('https://www.example.org/') Not only httpx has a regular "synchronous" API (blocking function calls), but it also has an asynchronous API! Sadly, while HTTP/3 is being developed, it seems like in this list, httpx is the only HTTP client library supporting HTTP/2 currently :-( For HTTP/2, I also found the "httplib2" module. For HTTP/3, I found the "http3" and "aioquic" modules. -- Let's come back to urllib: * It's API is too complicated * It doesn't support HTTP/2 nor HTTP/3 * It's barely maintained: there are 121 open issues including 3 security issues! The 3 open security issues: * bpo-33661 open 2018; * bpo-36338 open in 2019; * bpo-45795 open in 2021. Usually, it's bad when you refer to an open security issue by its creation year :-( The urllib module has long history of security vulnerabilities. List of *fixed* vulnerabilities: * 2011 (bpo-11662): https://python-security.readthedocs.io/vuln/urllib-redirect.html * 2017 (bpo-30119): https://python-security.readthedocs.io/vuln/urllib-ftp-stream-injection.html * 2017 (bpo-30500): https://python-security.readthedocs.io/vuln/urllib-connects-wrong-host.html * 2019 (bpo-35907): https://python-security.readthedocs.io/vuln/urllib-local-file-scheme.html * 2019 (bpo-38826): https://python-security.readthedocs.io/vuln/urllib-basic-auth-regex.html * 2021 (bpo-42967): https://python-security.readthedocs.io/vuln/urllib-query-string-semicolon-s… * 2021 (bpo-43075): https://python-security.readthedocs.io/vuln/urllib-basic-auth-regex2.html * 2021 (bpo-44022): https://python-security.readthedocs.io/vuln/urllib-100-continue-loop.html urllib is a package made of 4 parts: * urllib.request for opening and reading URLs * urllib.error containing the exceptions raised by urllib.request * urllib.parse for parsing URLs * urllib.robotparser for parsing robots.txt files I propose to deprecate all of them. Maybe the deprecation can be different for each sub-module? Victor -- Night gathers, and now my watch begins. It shall not end until my death.

39 75

Enhancing generic type documentation in the standard library
by Luciano Ramalho March 30, 2022

March 30, 2022

Hello, everyone! I believe our documentation about types needs another overhaul. We now have lots of generic types in the standard library, but their formal type parameters are poorly documented—or not documented at all—in the standard library documentation. More importantly: the documentation we have about specific covariant/contravariant types is now in entries in the `typing` module that are all deprecated since PEP 585 was implemented in Python 3.9. Below I present two of many examples where the documentation of a generic type is not great. However, if people agree this is a problem, we need to discuss where and how to put the documentation in a way that is not too disruptive to users of Python who don't know or don't care about type hints, for many reasons that we should not judge. For example, where do we document the fact that `dict` accepts two invariant formal type parameters, and that `frozenset` accepts one contravariant type parameter? What do you think? Cheers, Luciano _________________________________________ EXAMPLE 1: `Callable` variance is not documented For example, in the `Callable` type, the `ReturnType` parameter is covariant, and the `ParameterType` parameters are all contravariant. But there is no mention of variance in this entry: https://docs.python.org/3/library/typing.html?highlight=typing#typing.Calla… Also, no mention of the fact that `collections.abc.Callable` is generic here: https://docs.python.org/3/library/collections.abc.html#collections.abc.Call… PEP 483—The Theory of Type Hints—is the only official Python doc where I found the information about the variance of the formal type parameters of `Callable`. The explanation there is brilliant [0]. [0] https://peps.python.org/pep-0483/#covariance-and-contravariance Regardless, the intended audience of PEPs is "core developers"—which is neither a superset nor a subset of "Python devs now using type hints". We should not rely on PEPs to document features for Python users in general. _________________________________________ EXAMPLE 2: `Generator` variance could be better documented The entry for `typing.Generator` [1] has this heading: class typing.Generator(Iterator[T_co], Generic[T_co, T_contra, V_co]) Answer quickly: how many formal type parameters does `Generator` require? Which are covariant? Which are contravariant? [1] https://docs.python.org/3/library/typing.html?highlight=typing#typing.Gener… Nowhere in that page [1] there's an explanation of the `*_co` and `*_contra` naming convention, much less their semantics. Fortunately, the text of the `typing.Generator` entry says: "A generator can be annotated by the generic type `Generator[YieldType, SendType, ReturnType]". Unfortunately, `typing.Generator` is deprecated and will be gone in 5 years or so... The same issue applies to all the other generic types: builtins (`dict`, `frozenset`), ABCs, etc. Their formal type parameters they accept as generics are either undocumented, or documented in parts of the `typing` module that are already deprecated. Thoughts? -- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Technical Principal at ThoughtWorks | Twitter: @ramalhoorg

6 7

Fwd: Re: Are "Batteries Included" still a Good Thing? [was: It's now time to deprecate the stdlib urllib module]
by Sasha Kacanski March 30, 2022

March 30, 2022

---------- Forwarded message --------- From: Sasha Kacanski <skacanski(a)gmail.com> Date: Wed, Mar 30, 2022, 2:55 PM Subject: Re: [Python-Dev] Re: Are "Batteries Included" still a Good Thing? [was: It's now time to deprecate the stdlib urllib module] To: Paul Moore <p.f.moore(a)gmail.com> Cc: Christopher Barker <pythonchb(a)gmail.com>, Ethan Furman < ethan(a)stoneleaf.us>, Python Dev <python-dev(a)python.org> Agree with Paul, Best of python always was, you can do almost everything with standard libs, but then novice becomes better developer and starts exploring libs and tools that wrap stuff or perform better or has functionality that stdlib does not support right of the bet they start using variety of libs that suits their needs. Regards, Sasha On Sun, Mar 27, 2022, 8:06 PM Paul Moore <p.f.moore(a)gmail.com> wrote: > On Sun, 27 Mar 2022 at 17:11, Christopher Barker <pythonchb(a)gmail.com> > wrote: > > > > With the json package included, all they need to do is `import json`. If > that wasn't there, they's look in PyPi for a JSON implementation, and find > an absolutely astonishing number of options. I just did a search for "JSON" > > on PYPI, and it's HUGE -- most of them are for specialized JSON-using > protocols of some sort. I was actually really surprised that couple I know > about of the top of my head (ujson, orjson) are actually hard to find. > > > > "You can just pip install it" is really not a good solution. > > > > In fact, this is an example, I think, of where we should put some effort > into making the included batteries better -- it's great to have a JSON lib > built in, but it's too bad that it's not best-of-bread by pretty much any > definition (speed, memory performance, flexibility) -- there are quite a > few feature requests open for it -- it would be nice to actually implement > some of those. (but yes, that's a lot of work that someone(s) would have to > do) > > > > Back to the topic at hand, rather than remove urllib, maybe it could be > made better -- an as-easy-to-use-as-requests package in the stdlib would be > really great. > > I think that's where the mistake happens, though. Someone who needs > "best of breed" is motivated (and likely knowledgeable enough) to make > informed decisions about what's on PyPI. But someone who just wants to > get the job done probably doesn't - and that's the audience for the > stdlib. A stdlib module needs to be a good, reliable set of basic > functionality that non-experts can use successfully. There can be > better libraries on PyPI, but that doesn't mean the stdlib module is > unnecessary, nor does it mean that the stdlib has to match the PyPI > library feature for feature. > > So here, specifically, I'd rather see urlllib be the best urlllib it > can be, and not demand that it turn into requests. Requests is there > if people need/want it (as is httpx, and urllib3, and aiohttp). But > urllib is for people who want to get a file from the web, and *not* > have to deal with dependencies, 3rd party libraries, etc. > > The "batteries included" standard library and PyPI complement each > other. Neither is redundant, and neither implies the other is > unnecessary. > > Paul > _______________________________________________ > Python-Dev mailing list -- python-dev(a)python.org > To unsubscribe send an email to python-dev-leave(a)python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/E2FGXTY… > Code of Conduct: http://python.org/psf/codeofconduct/ >

1 0

Changing unittest verbose output.
by Itay Yeshaya March 29, 2022

March 29, 2022

When running unittest with the -v flag, if a test has errors, and has a docstring, the test name is shown on one line, and the docstring is shown on the next line with the `ERROR` word. For example: ./python.exe -m unittest -v Lib.test_verbose_error.randomTest test_one (Lib.test_verbose_error.randomTest) this is a doc string with helpful info ... ERROR ====================================================================== ERROR: test_one (Lib.test_verbose_error.randomTest) this is a doc string with helpful info ---------------------------------------------------------------------- Traceback (most recent call last): File "/workspaces/cpython/Lib/test_verbose_error.py", line 11, in test_one assert l[1] ~^^^ IndexError: list index out of range ---------------------------------------------------------------------- Ran 1 test in 0.004s This cause for some confusion and we would like to fix it, the possible options are: 1. print the name and docstring, and ERROR, all on one line. 2. provide a command-line switch to select that behavior. The first option has a minor backwards compatibility breakage, since the default output would change and those lines would get longer. Any thoughts on the mentioned solutions? https://bugs.python.org/issue47133

7 9

walrus operator and expression
by Ethan Furman March 28, 2022

March 28, 2022

In the following bit of code: while s := input.read(MAXBINSIZE): while len(s) < MAXBINSIZE and ns := input.read(MAXBINSIZE-len(s)): s += ns line = binascii.b2a_base64(s) output.write(line) I'm getting this error on the second line: cannot use assignment expressions with expression Can somebody explain why that isn't working? Many thanks. -- ~Ethan~

4 3

import * and __future__ imports
by Irit Katriel March 28, 2022

March 28, 2022

If you have a __future__ import in a script, and you import * from it in another script, the object for this future appears in the dir() of the other script, even though the __future__ import has no effect there. % cat x.py from __future__ import annotations % cat y.py from x import * print(dir()) class D: def f(self, a: D): return 42 % ./python.exe y.py ['__annotations__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 'annotations'] Traceback (most recent call last): File "/Users/iritkatriel/src/cpython-654/y.py", line 5, in <module> class D: ^^^^^^^^ File "/Users/iritkatriel/src/cpython-654/y.py", line 6, in D def f(self, a: D): ^ NameError: name 'D' is not defined I think we should change import * to exclude the __future__ import objects, and perhaps also to not show them in dir(x). Any objections? This came up in the discussion about https://bugs.python.org/issue26120 . See the attached PR for a technique we can use to identify those objects.

4 4

is __self__ an implementation detail
by Robin Becker March 28, 2022

March 28, 2022

A gentoo developer reported a symlink loop problem in reportlab's setup.py where we search for specific headers. The 'fixed' function looks like this def findFile(root, wanted, followlinks=True): visited = set() for p, D, F in os.walk(root,followlinks=followlinks): #scan directories to check for prior visits #use dev/inode to make unique key SD = [].append for d in D: dk = os.stat(pjoin(p,d)) dk = dk.st_dev, dk.st_ino if dk not in visited: visited.add(dk) SD(d) D[:] = SD.__self__ #set the dirs to be scanned for fn in F: if fn==wanted: return abspath(pjoin(p,fn)) the fix is reported to have worked, but the developer is querying the lifting of SD.append using the construct SD = [].append loop involving appends to the list ..... D[:] = SD.__self__ his objection was that using __self__ might be implementation sensitive. I cannot tell from the documentation (eg https://docs.python.org/3.10/reference/datamodel.html) if these magic methods are part of python or of the implementation. On the other hand as a longtime python programmer I wonder if such simple cases are already optimized out in the produced bytecode; for years I have been avoiding s += 'string' and only recently found out that it was handled as a special case. -- Robin Becker

2 1

Summary of Python tracker Issues
by Python tracker March 25, 2022

March 25, 2022

ACTIVITY SUMMARY (2022-03-18 - 2022-03-25) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7145 (-20) closed 51702 (+82) total 58847 (+62) Open issues with patches: 2903 Issues opened (53) ================== #45618: Documentation builds fail with Sphinx 3.2.1 https://bugs.python.org/issue45618 reopened by Maciej Olko #46429: Merge all deepfrozen files into one https://bugs.python.org/issue46429 reopened by kumaraditya303 #46864: Deprecate ob_shash in BytesObject https://bugs.python.org/issue46864 reopened by christian.heimes #46975: clang: error: linker command failed with exit code 1 (use -v t https://bugs.python.org/issue46975 reopened by jaraco #47060: importlib.metadata.version can return None https://bugs.python.org/issue47060 opened by David Robertson #47061: Deprecate modules listed in PEP 594 https://bugs.python.org/issue47061 opened by brett.cannon #47063: SimpleHTTPRequestHandler has hard coded index page list. https://bugs.python.org/issue47063 opened by myronww #47064: thread QOS attribute on macOS https://bugs.python.org/issue47064 opened by ronaldoussoren #47065: test_curses fails if terminal defaults to bright white text (1 https://bugs.python.org/issue47065 opened by ncoghlan #47067: Add vectorcall for generic alias object https://bugs.python.org/issue47067 opened by penguin_wwy #47068: Improve __repr__ of TypeVar https://bugs.python.org/issue47068 opened by ktbarrett #47069: socket._GLOBAL_DEFAULT_TIMEOUT being an object() makes for ugl https://bugs.python.org/issue47069 opened by ferdnyc #47070: Improve performance of array_inplace_repeat https://bugs.python.org/issue47070 opened by pieter.eendebak #47071: asyncio proactor udp transport stops responding after send to https://bugs.python.org/issue47071 opened by esoma #47072: Database corruption with the shelve module https://bugs.python.org/issue47072 opened by HubTou #47073: Solution for recursion error when comparing dataclass objects https://bugs.python.org/issue47073 opened by dankner #47074: Convenient `simplefilter()` in `warnings.catch_warnings()` https://bugs.python.org/issue47074 opened by Zac Hatfield-Dodds #47075: test_multiprocessing_spawn leaks QueueManager dangling process https://bugs.python.org/issue47075 opened by vstinner #47077: test_asyncio ignores exception in _ProactorBasePipeTransport._ https://bugs.python.org/issue47077 opened by vstinner #47078: test_ctypes modified files on aarch64 Fedora Rawhide 3.9: ldco https://bugs.python.org/issue47078 opened by vstinner #47079: Integral.denominator shouldn't return an int https://bugs.python.org/issue47079 opened by Sergey.Kirpichev #47082: No protection: `import numpy` in two different threads can lea https://bugs.python.org/issue47082 opened by jhgoebbert #47083: The __complex__ method is missing from the complex, float, and https://bugs.python.org/issue47083 opened by maggyero #47086: Include HTML docs with Windows installer instead of CHM https://bugs.python.org/issue47086 opened by steve.dower #47087: Implement PEP 655 (Required/NotRequired) https://bugs.python.org/issue47087 opened by JelleZijlstra #47088: Implement PEP 675 (LiteralString) https://bugs.python.org/issue47088 opened by JelleZijlstra #47089: Avoid sporadic failure of test_compileall on Windows https://bugs.python.org/issue47089 opened by jkloth #47090: Make zlib required on all platforms (simplifies code) https://bugs.python.org/issue47090 opened by gregory.p.smith #47091: Improve performance of list and tuple repeat methods https://bugs.python.org/issue47091 opened by pieter.eendebak #47092: [C API] Add PyFrame_GetVar(frame, name) function https://bugs.python.org/issue47092 opened by vstinner #47093: Documentation Fix: Remove .bat when activating venv on windows https://bugs.python.org/issue47093 opened by jovinator #47095: Prefer libb2 over vendored copy of blake2 https://bugs.python.org/issue47095 opened by christian.heimes #47096: Use the PDH API in WindowsLoadTracker https://bugs.python.org/issue47096 opened by eryksun #47097: Document PEP 646 https://bugs.python.org/issue47097 opened by JelleZijlstra #47098: sha3: Replace Keccak Code Package with tiny_sha3 https://bugs.python.org/issue47098 opened by christian.heimes #47099: Replace with_traceback() with exception chaining and reraising https://bugs.python.org/issue47099 opened by arhadthedev #47100: Help text for Store Python shows "null" under usage https://bugs.python.org/issue47100 opened by steve.dower #47101: hashlib.algorithms_available lists algorithms that are not ava https://bugs.python.org/issue47101 opened by christian.heimes #47102: explore hashlib use of the Linux Kernel CryptoAPI https://bugs.python.org/issue47102 opened by gregory.p.smith #47103: Copy pgort140.dll when building for PGO https://bugs.python.org/issue47103 opened by steve.dower #47105: [Doc] grp cites pwd.h instead of grp.h https://bugs.python.org/issue47105 opened by aphedges #47106: test_asyncio: test_shutdown_cleanly() failed with timeout (10 https://bugs.python.org/issue47106 opened by vstinner #47108: asyncio-stream does not document exceptions https://bugs.python.org/issue47108 opened by christian.bodt #47109: Old ctypes.macholib tests are ignored by python -m test https://bugs.python.org/issue47109 opened by arhadthedev #47110: Refactor bytearray strip methods https://bugs.python.org/issue47110 opened by pieter.eendebak #47111: ENUM TypeError using mixing https://bugs.python.org/issue47111 opened by benyamin621 #47115: Documentation inconsistency with the stable ABI https://bugs.python.org/issue47115 opened by Crowthebird #47116: Use _PyLong_FromUnsignedChar in bytearrayobject.c https://bugs.python.org/issue47116 opened by pieter.eendebak #47117: repl segfaults on non utf-8 input https://bugs.python.org/issue47117 opened by jooon #47118: test_run_keeps_context in test_asyncio fails https://bugs.python.org/issue47118 opened by serhiy.storchaka #47119: test_freeze_simple_script in test_tools fails https://bugs.python.org/issue47119 opened by serhiy.storchaka #47120: Make all jump opcodes relative https://bugs.python.org/issue47120 opened by iritkatriel #47121: math.isfinite() can raise exception when called on a number https://bugs.python.org/issue47121 opened by tfish2 Most recent 15 issues with no replies (15) ========================================== #47120: Make all jump opcodes relative https://bugs.python.org/issue47120 #47119: test_freeze_simple_script in test_tools fails https://bugs.python.org/issue47119 #47116: Use _PyLong_FromUnsignedChar in bytearrayobject.c https://bugs.python.org/issue47116 #47115: Documentation inconsistency with the stable ABI https://bugs.python.org/issue47115 #47111: ENUM TypeError using mixing https://bugs.python.org/issue47111 #47110: Refactor bytearray strip methods https://bugs.python.org/issue47110 #47109: Old ctypes.macholib tests are ignored by python -m test https://bugs.python.org/issue47109 #47108: asyncio-stream does not document exceptions https://bugs.python.org/issue47108 #47106: test_asyncio: test_shutdown_cleanly() failed with timeout (10 https://bugs.python.org/issue47106 #47105: [Doc] grp cites pwd.h instead of grp.h https://bugs.python.org/issue47105 #47103: Copy pgort140.dll when building for PGO https://bugs.python.org/issue47103 #47100: Help text for Store Python shows "null" under usage https://bugs.python.org/issue47100 #47098: sha3: Replace Keccak Code Package with tiny_sha3 https://bugs.python.org/issue47098 #47096: Use the PDH API in WindowsLoadTracker https://bugs.python.org/issue47096 #47089: Avoid sporadic failure of test_compileall on Windows https://bugs.python.org/issue47089 Most recent 15 issues waiting for review (15) ============================================= #47120: Make all jump opcodes relative https://bugs.python.org/issue47120 #47118: test_run_keeps_context in test_asyncio fails https://bugs.python.org/issue47118 #47116: Use _PyLong_FromUnsignedChar in bytearrayobject.c https://bugs.python.org/issue47116 #47110: Refactor bytearray strip methods https://bugs.python.org/issue47110 #47109: Old ctypes.macholib tests are ignored by python -m test https://bugs.python.org/issue47109 #47105: [Doc] grp cites pwd.h instead of grp.h https://bugs.python.org/issue47105 #47103: Copy pgort140.dll when building for PGO https://bugs.python.org/issue47103 #47101: hashlib.algorithms_available lists algorithms that are not ava https://bugs.python.org/issue47101 #47099: Replace with_traceback() with exception chaining and reraising https://bugs.python.org/issue47099 #47098: sha3: Replace Keccak Code Package with tiny_sha3 https://bugs.python.org/issue47098 #47097: Document PEP 646 https://bugs.python.org/issue47097 #47095: Prefer libb2 over vendored copy of blake2 https://bugs.python.org/issue47095 #47091: Improve performance of list and tuple repeat methods https://bugs.python.org/issue47091 #47090: Make zlib required on all platforms (simplifies code) https://bugs.python.org/issue47090 #47089: Avoid sporadic failure of test_compileall on Windows https://bugs.python.org/issue47089 Top 10 most discussed issues (10) ================================= #47086: Include HTML docs with Windows installer instead of CHM https://bugs.python.org/issue47086 13 msgs #44493: Missing terminated NUL in the length of sockaddr_un https://bugs.python.org/issue44493 12 msgs #25489: sys.exit() caught in async event loop exception handler https://bugs.python.org/issue25489 9 msgs #47006: PEP 646: Decide on substitution behavior https://bugs.python.org/issue47006 9 msgs #46070: [subinterpreters] crash when importing _sre in subinterpreters https://bugs.python.org/issue46070 7 msgs #47031: math.nan should note that NANs do not compare equal to anythin https://bugs.python.org/issue47031 7 msgs #47067: Add vectorcall for generic alias object https://bugs.python.org/issue47067 7 msgs #47000: Make encoding="locale" uses locale encoding even in UTF-8 mode https://bugs.python.org/issue47000 6 msgs #47064: thread QOS attribute on macOS https://bugs.python.org/issue47064 6 msgs #47078: test_ctypes modified files on aarch64 Fedora Rawhide 3.9: ldco https://bugs.python.org/issue47078 6 msgs Issues closed (76) ================== #1554: socketmodule cleanups: allow the use of keywords in socket fun https://bugs.python.org/issue1554 closed by iritkatriel #2604: doctest.DocTestCase fails when run repeatedly https://bugs.python.org/issue2604 closed by JelleZijlstra #5888: mmap enhancement - resize with sequence notation https://bugs.python.org/issue5888 closed by iritkatriel #8891: shutil: sort files before archiving for consistency https://bugs.python.org/issue8891 closed by iritkatriel #11160: ZipFile.comment expects bytes https://bugs.python.org/issue11160 closed by iritkatriel #12029: [doc] clarify that except does not match virtual subclasses of https://bugs.python.org/issue12029 closed by iritkatriel #13197: subprocess: move shell arguments to a separate keyword param https://bugs.python.org/issue13197 closed by iritkatriel #18217: Deprecate and remove gettext.install https://bugs.python.org/issue18217 closed by iritkatriel #18241: Add unique option to heapq functions https://bugs.python.org/issue18241 closed by rhettinger #19961: MacOSX: Tkinter build failure when building without command-li https://bugs.python.org/issue19961 closed by ned.deily #22013: Add at least minimal support for thread groups https://bugs.python.org/issue22013 closed by iritkatriel #22722: inheritable pipes are unwieldy without os.pipe2 https://bugs.python.org/issue22722 closed by iritkatriel #23647: imaplib.py MAXLINE value is too low for gmail https://bugs.python.org/issue23647 closed by iritkatriel #23691: re.finditer iterator is not reentrant, but doesn't protect aga https://bugs.python.org/issue23691 closed by serhiy.storchaka #26019: collections.abc documentation incomplete https://bugs.python.org/issue26019 closed by rhettinger #26362: Approved API for creating a temporary file path https://bugs.python.org/issue26362 closed by iritkatriel #27165: Skip callables when displaying exception fields in cgitb https://bugs.python.org/issue27165 closed by iritkatriel #28948: Facing issue while running Python 3.6.0rc1 windows x86-64 web https://bugs.python.org/issue28948 closed by iritkatriel #29067: Source archive: wrong directory attributes https://bugs.python.org/issue29067 closed by iritkatriel #32927: Add typeshed documentation for unittest.TestCase._feedErrorsTo https://bugs.python.org/issue32927 closed by iritkatriel #33028: [doc] tempfile.TemporaryDirectory documentation improvements https://bugs.python.org/issue33028 closed by iritkatriel #34071: asyncio: repr(task) raises AssertionError for coros which loop https://bugs.python.org/issue34071 closed by iritkatriel #34406: Typo in documentation https://bugs.python.org/issue34406 closed by iritkatriel #35577: side_effect mocked method lose reference to instance https://bugs.python.org/issue35577 closed by iritkatriel #37665: threading.TIMEOUT_MAX integer overflow on 32-bit builds with t https://bugs.python.org/issue37665 closed by iritkatriel #37789: Update doc strings for test.bytecode_helper https://bugs.python.org/issue37789 closed by iritkatriel #38256: binascii.crc32 is not 64-bit clean when USE_ZLIB_CRC32 https://bugs.python.org/issue38256 closed by gregory.p.smith #38307: Add .end_lineno attribute to pyclbr _Objects https://bugs.python.org/issue38307 closed by iritkatriel #39298: add BLAKE3 to hashlib https://bugs.python.org/issue39298 closed by larry #39394: re: DeprecationWarning for `flag not at the start of expressio https://bugs.python.org/issue39394 closed by serhiy.storchaka #39640: fall back os.fdatasync() to fsync() on POSIX systems without f https://bugs.python.org/issue39640 closed by gregory.p.smith #40074: pickle module dump and load: add support for string file names https://bugs.python.org/issue40074 closed by serhiy.storchaka #40296: help(list[int]) fails https://bugs.python.org/issue40296 closed by serhiy.storchaka #40320: Add ability to specify instance of contextvars context to Task https://bugs.python.org/issue40320 closed by asvetlov #40666: TarFile.add does not support pathlib.Path as a value to first https://bugs.python.org/issue40666 closed by iritkatriel #41864: Compiler error in _zoneinfo.c:1227: error: #pragma GCC diagnos https://bugs.python.org/issue41864 closed by iritkatriel #42369: Reading ZipFile not thread-safe https://bugs.python.org/issue42369 closed by serhiy.storchaka #42528: Improve the docs of most Py*_Check{,Exact} API calls https://bugs.python.org/issue42528 closed by iritkatriel #42871: Regex compilation crashed if I change order of alternatives un https://bugs.python.org/issue42871 closed by serhiy.storchaka #42884: array.index() missing start and end https://bugs.python.org/issue42884 closed by iritkatriel #42885: Optimize re.search() for \A (and maybe ^) https://bugs.python.org/issue42885 closed by serhiy.storchaka #42917: Block stack size for frame objects should be dynamically sizab https://bugs.python.org/issue42917 closed by Mark.Shannon #44336: Windows buildbots hang after fatal exit https://bugs.python.org/issue44336 closed by steve.dower #44832: Compiler detection is not strict enough https://bugs.python.org/issue44832 closed by iritkatriel #45098: asyncio.CancelledError should contain more information on canc https://bugs.python.org/issue45098 closed by asvetlov #45214: implement LOAD_NONE opcode https://bugs.python.org/issue45214 closed by iritkatriel #45979: Fix Tkinter tests with old Tk https://bugs.python.org/issue45979 closed by serhiy.storchaka #46013: Confusing period in object.__hash__ doc https://bugs.python.org/issue46013 closed by iritkatriel #46073: ast.unparse produces: 'FunctionDef' object has no attribute 'l https://bugs.python.org/issue46073 closed by iritkatriel #46125: Test the preferred API instead of relying on legacy for covera https://bugs.python.org/issue46125 closed by jaraco #46382: dataclass(slots=True) does not account for slots in base class https://bugs.python.org/issue46382 closed by eric.smith #46769: Improve documentation for `typing.TypeVar` https://bugs.python.org/issue46769 closed by JelleZijlstra #46774: Importlib.metadata.version picks first distribution not latest https://bugs.python.org/issue46774 closed by jaraco #46788: regrtest fails to start on missing performance counter names https://bugs.python.org/issue46788 closed by steve.dower #46829: Confusing CancelError message if multiple cancellations are sc https://bugs.python.org/issue46829 closed by gvanrossum #47012: Speed up iteration of bytes and bytearray https://bugs.python.org/issue47012 closed by kumaraditya303 #47014: ProactorEventLoop ignores Ctrl+C after closing unrelated loop https://bugs.python.org/issue47014 closed by asvetlov #47015: Update tests from asyncore to asyncio https://bugs.python.org/issue47015 closed by asvetlov #47037: Build problems on Windows https://bugs.python.org/issue47037 closed by steve.dower #47040: Fix confusing versionchanged note in crc32 and adler32 https://bugs.python.org/issue47040 closed by gregory.p.smith #47048: Python 3.10.3 + Osx Lion : fatal error (make) signalmodule or https://bugs.python.org/issue47048 closed by ned.deily #47055: `issubclass` on two different subclasses of abstract base clas https://bugs.python.org/issue47055 closed by mobiusklein #47057: Use FASTCALL convention for FutureIter.throw() https://bugs.python.org/issue47057 closed by asvetlov #47062: Implement asyncio.Runner context manager https://bugs.python.org/issue47062 closed by asvetlov #47066: Convert a warning about flags not at the start of the regular https://bugs.python.org/issue47066 closed by serhiy.storchaka #47076: test_asyncio: test_get_cancelled() fails randomly on x86-64 ma https://bugs.python.org/issue47076 closed by asvetlov #47080: Use atomic groups to simplify fnmatch https://bugs.python.org/issue47080 closed by tim.peters #47081: Replace "qualifiers" with "quantifiers" in the re module docum https://bugs.python.org/issue47081 closed by serhiy.storchaka #47084: Statically allocated Unicode objects leak cached representatio https://bugs.python.org/issue47084 closed by vstinner #47085: missing frame.f_lineno on JUMP_ABSOLUTE https://bugs.python.org/issue47085 closed by Mark.Shannon #47094: index doesn't change while looping through same elements in a https://bugs.python.org/issue47094 closed by Dennis Sweeney #47104: Rewrite asyncio.to_thread tests to use IsolatedAsyncioTestCase https://bugs.python.org/issue47104 closed by asvetlov #47107: Navigation bar in python3103.chm is broken https://bugs.python.org/issue47107 closed by steve.dower #47112: RuntimeError: expected scalar type Long but found Int https://bugs.python.org/issue47112 closed by zach.ware #47113: PermissionError on pip uninstall https://bugs.python.org/issue47113 closed by christian.heimes #47114: random.choice and random.choices have different distributions https://bugs.python.org/issue47114 closed by rhettinger

1 0

Python Language Summit at PyCon 2022 in Salt Lake City
by Mariatta March 24, 2022

March 24, 2022

We're excited to announce that the signups for Python Language Summit at PyCon 2022 are now open. Full details at: https://us.pycon.org/2022/events/language-summit/ After two years of virtual/online summit, we will be returning to in-person format. We will be following the health and safety guidelines at PyCon US: https://us.pycon.org/2022/attend/health-safety-guidelines/ *TL;DR* When: Wednesday, April 27, 2022 Where: Salt Palace Convention Center, room TBD Sign up to attend: https://forms.gle/CS8B6wJdcaN3rtWV8 <https://forms.gle/CS8B6wJdcaN3rtWV8> (closes March 25 th, 2022 AoE) Sign up to discuss a topic: https://forms.gle/LAFE6TTYi15jL5RaA (closes March 25th, 2022 AoE) *Who can attend* We welcome Python core developers and triage team members, active core contributors to Python and alternative Python implementations, and anyone else who has a topic to discuss with core developers. *Who can propose a discussion topic* If you have discussion items; seeking consensus; awaiting decision on a PEP; needing help with your core dev work; or have specific questions that need answers from core developers, please submit a proposal. According to feedback, our audience prefers more discussions and shorter talks. In your proposal, please include: - why is this topic relevant to the core developers - what is needed from core developers out of this topic This year's event will be covered by Alex Waygood. Detailed summary of the event will be published at The PSF's Blog. *Is this event recorded? Can I watch the livestream?* No, there will be no recording and no livestream available. If you'd like to participate in discussions, please sign up to attend. If you'd like to listen in, please wait for Alex's blog posts after the summit. Thank you, Mariatta, Łukasz, & Senthil

1 2

[RELEASE] Python 3.10.4 and 3.9.12 are now available out of schedule
by Łukasz Langa March 24, 2022

March 24, 2022

Did anybody say cursed releases <https://discuss.python.org/t/python-3-10-3-3-9-11-3-8-13-and-3-7-13-are-now…>? Well, it turns out that 3.10.3 and 3.9.11 both shipped a regression which caused those versions not to build on Red Hat Enterprise Linux 6. While this 11-year-old version is now out of maintenance support <https://access.redhat.com/support/policy/updates/errata>, it’s still used in production workloads. Some of those rely on Python 3.9 and/or 3.10. In particular, our own manylinux2010 <https://github.com/pypa/manylinux/tree/manylinux2010_x86_64_centos6_no_vsys…> image used to build widely compatible Linux wheels is based on CentOS 6. (Don’t worry, we do have newer manylinux* variants, see PEP 599 <https://peps.python.org/pep-0599/> and PEP 600 <https://peps.python.org/pep-0600/> for details.) Due to the out-of-schedule release, the respective versions released today contain a very limited set of changes. Python 3.9.12 only contains 12 other bug fixes on top of 3.9.11. Python 3.10.4 only contains 10 other bug fixes on top of 3.10.3. Get 3.10.4 here: Python Release 3.10.4 | Python.org <https://www.python.org/downloads/release/python-3104> Get 3.9.12 here: Python Release 3.9.12 | Python.org <https://www.python.org/downloads/release/python-3912> Hopefully, the third time’s a charm and we’ll return no sooner than May with the regularly scheduled bug fix releases of 3.9 and 3.10. <https://discuss.python.org/t/python-3-10-4-and-3-9-12-are-now-available-out…>We hope you enjoy the new releases Your friendly release team, Łukasz Langa @ambv <https://discuss.python.org/u/ambv> Pablo Galindo Salgado @pablogsal <https://discuss.python.org/u/pablogsal> Ned Deily @nad <https://discuss.python.org/u/nad> Steve Dower @steve.dower <https://discuss.python.org/u/steve.dower>

3 2

Accepting PEP 675 - Arbitrary Literal String Type
by Gregory P. Smith March 21, 2022

March 21, 2022

On behalf of the Python Steering Council, we are accepting PEP 675 - Arbitrary Literal String Type <https://peps.python.org/pep-0675/>. TL;DR - PEP 675 allows type checkers to help prevent bugs allowing attacker-controlled data to be passed to APIs that declare themselves as requiring literal, in-code strings. This is a very thorough PEP with a compelling and highly relevant set of use cases. If I tried to call out all the things we like about it, it’d turn into a table of contents. It is long, but everything has a reason to be there. :) Once implemented, we expect it to be a challenge to tighten widely used existing APIs that accept str today to be LiteralString for practical reasons of what existing code calling unrestricted APIs naturally does. The community would benefit from anyone who attempts to move a widely used existing str API to LiteralString sharing their experiences, successful or not. -gps for the steering council

2 1

Accepting PEP 655 - Marking individual TypedDict items as required or potentially-missing
by Gregory P. Smith March 21, 2022

March 21, 2022

On behalf of the Python Steering Council, we are pleased to accept PEP 655 - Marking individual TypedDict items as required or potentially-missing <https://peps.python.org/pep-0655/>. Thanks for considering the potential for confusion with Optional during the design and recommending best practices in the “how to teach” section. A couple SC members have tried using TypedDict and found it painful, this PEP helps. -gps for the steering council

1 0

A memory map based data persistence and startup speedup approach
by Yichen Yan March 20, 2022

March 20, 2022

Hi folks, as illustrated in faster-cpython#150 [1], we have implemented a mechanism that supports data persistence of a subset of python date types with mmap, therefore can reduce package import time by caching code object. This could be seen as a more eager pyc format, as they are for the same purpose, but our approach try to avoid [de]serialization. Therefore, we get a speedup in overall python startup by ~15%. Currently, we’ve made it a third-party library and have been working on open-sourcing. Our implementation (whose non-official name is “pycds”) mainly contains two parts: importlib hooks, this implements the mechanism to dump code objects to an archive and a `Finder` that supports loading code object from mapped memory. Dumping and loading (subset of) python types with mmap. In this part, we deal with 1) ASLR by patching `ob_type` fields; 2) hash seed randomization by supporting only basic types who don’t have hash-based layout (i.e. dict is not supported); 3) interned string by re-interning strings while loading mmap archive and so on. After pycds has been installed, complete workflow of our approach includes three parts: Record name of imported packages to heap.lst, `PYCDSMODE=TRACE PYCDSLIST=heap.lst python run.py` Dump memory archive of code objects of imported packages, this step does not involve the python script, `PYCDSMODE=DUMP PYCDSLIST=heap.lst PYCDSARCHIVE=heap.img python` Run other python processes with created archive, `PYCDSMODE=SHARE PYCDSARCHIVE=heap.img python run.py` We could even make use of immortal objects if PEP 683 [2] was accepted, which could give CDS more performance improvements. Currently, any archived object is virtually immortal, we add rc by 1 to who has been copied to the archive to avoid being deallocated. However, without changes to CPython, rc fields of archived objects will still be updated, therefore have extra footprint due to CoW. More background and detailed implementation could be found at [1]. We think it could be an effective way to improve python’s startup performance, and could even do more like sharing large data between python instances. As suggested in python-ideas [3], we posted this here, looking for questions/suggestions to the overall design and workflow, we also welcome code reviews after we get our lawyers happy and can publish the code. Best, Yichen Yan Alibaba Compiler Group [1] “Faster startup -- Share code objects from memory-mapped file”, https://github.com/faster-cpython/ideas/discussions/150 [2] PEP 683: "Immortal Objects, Using a Fixed Refcount" (draft), https://mail.python.org/archives/list/python-dev@python.org/message/TPLEYDC… [3] [Python-ideas] "A memory map based data persistence and startup speedup approach", https://mail.python.org/archives/list/python-ideas@python.org/thread/UKEBNH…

2 1

Restrict the type of __slots__
by Serhiy Storchaka March 19, 2022

March 19, 2022

Currently __slots__ can be either string or an iterable of strings. 1. If it is a string, it is a name of a single slot. Third-party code which iterates __slots__ will be confused. 2. If it is an iterable, it should emit names of slots. Note that non-reiterable iterators are accepted too, but it causes weird bugs if __slots__ is iterated more than once. For example it breaks default pickling and copying. I propose to restrict the type of __slots__. Require it always been a tuple of strings. Most __slots__ in real code are tuples. It is rarely we need only single slot and set __slots__ as a string. It will break some code (there are 2 occurrences in the stdlib an 1 in scripts), but that code can be easily fixed.

10 15

New PEP website is horrible to read on mobile device
by Nathan Cook March 18, 2022

March 18, 2022

Please make https://peps.python.org/ more responsive to various form factors See attached screenshot from Chrome version 99.0.4844.58 on my Pixel 3aXL running Android 12 Thank you, Nathan Cook

11 11

Summary of Python tracker Issues
by Python tracker March 18, 2022

March 18, 2022

ACTIVITY SUMMARY (2022-03-11 - 2022-03-18) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7165 (-68) closed 51620 (+139) total 58785 (+71) Open issues with patches: 2903 Issues opened (50) ================== #22628: Idle: Tree lines are spaced too close together. https://bugs.python.org/issue22628 reopened by exarkun #43253: asyncio open_connection fails when a socket is explicitly clos https://bugs.python.org/issue43253 reopened by asvetlov #45405: configure fails on macOS with non-Apple clang version 13 which https://bugs.python.org/issue45405 reopened by ned.deily #46948: [CVE-2022-26488] Escalation of privilege via Windows Installer https://bugs.python.org/issue46948 reopened by steve.dower #46989: signal.signal, et al, doesn't support [SIGRTMIN,SIGRTMAX] on F https://bugs.python.org/issue46989 opened by ngie #46990: Surprising list overallocation from .split() https://bugs.python.org/issue46990 opened by tim.peters #46992: If use textwrap.dedent with string formatting, may get uninten https://bugs.python.org/issue46992 opened by xncbf12 #46997: Invalid memory write in bytearray https://bugs.python.org/issue46997 opened by JelleZijlstra #46998: Allow subclassing Any at runtime https://bugs.python.org/issue46998 opened by hauntsaninja #46999: test_multiprocessing_fork running without any timeout https://bugs.python.org/issue46999 opened by doko #47000: Make encoding="locale" uses locale encoding even in UTF-8 mode https://bugs.python.org/issue47000 opened by methane #47002: argparse - "expected one argument" when used -: in argument https://bugs.python.org/issue47002 opened by Pythass #47006: PEP 646: Decide on substitution behavior https://bugs.python.org/issue47006 opened by JelleZijlstra #47007: [doc] str docs are inconsistent with special method lookup https://bugs.python.org/issue47007 opened by itsvs #47009: Streamline list.append for the common case https://bugs.python.org/issue47009 opened by Dennis Sweeney #47010: Implement zero copy writes in SelectorSocketTransport in async https://bugs.python.org/issue47010 opened by kumaraditya303 #47011: Cloned turtle pen is not cleared completely https://bugs.python.org/issue47011 opened by learncoding #47012: Speed up iteration of bytes and bytearray https://bugs.python.org/issue47012 opened by kumaraditya303 #47013: test_bdb and test_distutils fail on installed Python 3.9, 3.10 https://bugs.python.org/issue47013 opened by vstinner #47014: ProactorEventLoop ignores Ctrl+C after closing unrelated loop https://bugs.python.org/issue47014 opened by mhils #47015: Update tests from asyncore to asyncio https://bugs.python.org/issue47015 opened by arhadthedev #47016: Create a test verifying bundled pip and setuptools wheels https://bugs.python.org/issue47016 opened by illia-v #47017: frozen modules are on by default in dev build https://bugs.python.org/issue47017 opened by gvanrossum #47019: Fatal Python Error in sqlite3 Python 3.10 https://bugs.python.org/issue47019 opened by hydroflask #47021: Add separate match and case doc to pydoc https://bugs.python.org/issue47021 opened by duckboycool #47022: PEP 594: Document removal of asynchat, asyncore and smtpd https://bugs.python.org/issue47022 opened by hugovk #47025: bytes do not work on sys.path https://bugs.python.org/issue47025 opened by graingert #47026: BytesWarning in zipimport paths on sys.path https://bugs.python.org/issue47026 opened by graingert #47027: subprocess.run(), subprocess.Popen() should accept file descri https://bugs.python.org/issue47027 opened by ydroneaud #47029: Fix a BrokenPipeError when a multiprocessing.Queue is garbage https://bugs.python.org/issue47029 opened by maggyero #47030: singledispatch does not work with positional arguments with de https://bugs.python.org/issue47030 opened by randolf.scholz #47031: math.nan should note that NANs do not compare equal to anythin https://bugs.python.org/issue47031 opened by steven.daprano #47037: Build problems on Windows https://bugs.python.org/issue47037 opened by terry.reedy #47040: Remove invalid versionchanged in doc https://bugs.python.org/issue47040 opened by malin #47043: Argparse can't parse subparsers with parse_known_args https://bugs.python.org/issue47043 opened by rive-n #47045: Remove the RESUME instruction https://bugs.python.org/issue47045 opened by Mark.Shannon #47046: Add `f_state` attribute to FrameObjects. https://bugs.python.org/issue47046 opened by Mark.Shannon #47047: smtplib: allow custom policy or use msg.policy in send_message https://bugs.python.org/issue47047 opened by Miksus #47048: Python 3.10.3 + Osx Lion : fatal error (make) signalmodule or https://bugs.python.org/issue47048 opened by laurentang001 #47049: Incorrect shutil.copytree() behaviour with symlinks https://bugs.python.org/issue47049 opened by vajdaz #47050: Cannot install Python 3.10.3 on Windows https://bugs.python.org/issue47050 opened by AlexWaygood #47051: Windows v3.10.3 .chm file (in python310\doc) page headings are https://bugs.python.org/issue47051 opened by DonnieODonnell #47052: allow string as sep in _Py_strhex_impl ( bytearray.hex() ) https://bugs.python.org/issue47052 opened by arne123 #47053: Reduce de-optimization in BINARY_OP_INPLACE_ADD_UNICODE https://bugs.python.org/issue47053 opened by Dennis Sweeney #47054: "SyntaxError: non-default argument follows default argument" s https://bugs.python.org/issue47054 opened by Bluenix #47055: `issubclass` on two different subclasses of abstract base clas https://bugs.python.org/issue47055 opened by mobiusklein #47056: turtle.write() causes flickering when the tracer is turned off https://bugs.python.org/issue47056 opened by relent95 #47057: Use FASTCALL convention for FutureIter.throw() https://bugs.python.org/issue47057 opened by asvetlov #47058: Skip tests failing on Solaris https://bugs.python.org/issue47058 opened by kulikjak #47059: Mechanism to enable __weakref__ slot on dataclass(slots=True) https://bugs.python.org/issue47059 opened by ariebovenberg Most recent 15 issues with no replies (15) ========================================== #47059: Mechanism to enable __weakref__ slot on dataclass(slots=True) https://bugs.python.org/issue47059 #47058: Skip tests failing on Solaris https://bugs.python.org/issue47058 #47057: Use FASTCALL convention for FutureIter.throw() https://bugs.python.org/issue47057 #47056: turtle.write() causes flickering when the tracer is turned off https://bugs.python.org/issue47056 #47053: Reduce de-optimization in BINARY_OP_INPLACE_ADD_UNICODE https://bugs.python.org/issue47053 #47051: Windows v3.10.3 .chm file (in python310\doc) page headings are https://bugs.python.org/issue47051 #47049: Incorrect shutil.copytree() behaviour with symlinks https://bugs.python.org/issue47049 #47045: Remove the RESUME instruction https://bugs.python.org/issue47045 #47031: math.nan should note that NANs do not compare equal to anythin https://bugs.python.org/issue47031 #47030: singledispatch does not work with positional arguments with de https://bugs.python.org/issue47030 #47027: subprocess.run(), subprocess.Popen() should accept file descri https://bugs.python.org/issue47027 #47026: BytesWarning in zipimport paths on sys.path https://bugs.python.org/issue47026 #47022: PEP 594: Document removal of asynchat, asyncore and smtpd https://bugs.python.org/issue47022 #47021: Add separate match and case doc to pydoc https://bugs.python.org/issue47021 #47017: frozen modules are on by default in dev build https://bugs.python.org/issue47017 Most recent 15 issues waiting for review (15) ============================================= #47058: Skip tests failing on Solaris https://bugs.python.org/issue47058 #47057: Use FASTCALL convention for FutureIter.throw() https://bugs.python.org/issue47057 #47053: Reduce de-optimization in BINARY_OP_INPLACE_ADD_UNICODE https://bugs.python.org/issue47053 #47049: Incorrect shutil.copytree() behaviour with symlinks https://bugs.python.org/issue47049 #47045: Remove the RESUME instruction https://bugs.python.org/issue47045 #47040: Remove invalid versionchanged in doc https://bugs.python.org/issue47040 #47037: Build problems on Windows https://bugs.python.org/issue47037 #47029: Fix a BrokenPipeError when a multiprocessing.Queue is garbage https://bugs.python.org/issue47029 #47025: bytes do not work on sys.path https://bugs.python.org/issue47025 #47022: PEP 594: Document removal of asynchat, asyncore and smtpd https://bugs.python.org/issue47022 #47016: Create a test verifying bundled pip and setuptools wheels https://bugs.python.org/issue47016 #47015: Update tests from asyncore to asyncio https://bugs.python.org/issue47015 #47010: Implement zero copy writes in SelectorSocketTransport in async https://bugs.python.org/issue47010 #47009: Streamline list.append for the common case https://bugs.python.org/issue47009 #47007: [doc] str docs are inconsistent with special method lookup https://bugs.python.org/issue47007 Top 10 most discussed issues (10) ================================= #47019: Fatal Python Error in sqlite3 Python 3.10 https://bugs.python.org/issue47019 15 msgs #47037: Build problems on Windows https://bugs.python.org/issue47037 14 msgs #46992: If use textwrap.dedent with string formatting, may get uninten https://bugs.python.org/issue46992 10 msgs #45150: Add a file_digest() function in hashlib https://bugs.python.org/issue45150 9 msgs #47025: bytes do not work on sys.path https://bugs.python.org/issue47025 9 msgs #46382: dataclass(slots=True) does not account for slots in base class https://bugs.python.org/issue46382 8 msgs #46566: Support -3.11-arm64 in py.exe launcher https://bugs.python.org/issue46566 8 msgs #46785: On Windows, os.stat() can fail if called while another process https://bugs.python.org/issue46785 7 msgs #47013: test_bdb and test_distutils fail on installed Python 3.9, 3.10 https://bugs.python.org/issue47013 7 msgs #46890: getpath problems with framework build https://bugs.python.org/issue46890 6 msgs Issues closed (129) =================== #2148: nis module not supporting group aliases https://bugs.python.org/issue2148 closed by iritkatriel #6497: Support for digital Cinema/film DPX and Kodak Cineon image fil https://bugs.python.org/issue6497 closed by iritkatriel #8526: msilib doesn't support multiple CAB instances in same installe https://bugs.python.org/issue8526 closed by iritkatriel #8934: aifc should use str instead of bytes (wave, sunau compatibilit https://bugs.python.org/issue8934 closed by iritkatriel #9544: [doc] xdrlib.Packer().pack_fstring throws a TypeError when cal https://bugs.python.org/issue9544 closed by iritkatriel #9736: doctest.DocTestSuite doesn't handle test globs correctly https://bugs.python.org/issue9736 closed by JelleZijlstra #12026: Support more of MSI api https://bugs.python.org/issue12026 closed by iritkatriel #12201: Returning FILETIME is unsupported in msilib.SummaryInformation https://bugs.python.org/issue12201 closed by iritkatriel #12506: NIS module cant handle multiple NIS map entries for the same G https://bugs.python.org/issue12506 closed by iritkatriel #12516: imghdr.what should take one argument https://bugs.python.org/issue12516 closed by iritkatriel #13681: Aifc read compressed frames fix https://bugs.python.org/issue13681 closed by iritkatriel #14156: argparse.FileType for '-' doesn't work for a mode of 'rb' https://bugs.python.org/issue14156 closed by serhiy.storchaka #15749: cgitb prints html for text when display disabled. https://bugs.python.org/issue15749 closed by iritkatriel #18979: Use argparse in the uu module https://bugs.python.org/issue18979 closed by iritkatriel #20392: Inconsistency with uppercase file extensions in MimeTypes.gues https://bugs.python.org/issue20392 closed by asvetlov #21574: Port image types detections from PIL to the imghdr module https://bugs.python.org/issue21574 closed by iritkatriel #22094: oss_audio_device.write(data) produces short writes https://bugs.python.org/issue22094 closed by iritkatriel #22476: asyncio task chapter confusion about 'task', 'future', and 'sc https://bugs.python.org/issue22476 closed by asvetlov #22859: unittest.TestProgram.usageExit no longer invoked https://bugs.python.org/issue22859 closed by JelleZijlstra #24080: asyncio.Event.wait() Task was destroyed but it is pending https://bugs.python.org/issue24080 closed by asvetlov #24224: test_msilib is inadequate https://bugs.python.org/issue24224 closed by iritkatriel #25230: asyncio/Windows: Unix datagram sockets not supported https://bugs.python.org/issue25230 closed by asvetlov #25291: better Exception message for certain task termination scenario https://bugs.python.org/issue25291 closed by asvetlov #25292: [asyncio] ssl socket gets into broken state when client exits https://bugs.python.org/issue25292 closed by asvetlov #26337: Bypass imghdr module determines the type of image https://bugs.python.org/issue26337 closed by iritkatriel #26545: [doc] os.walk is limited by python's recursion limit https://bugs.python.org/issue26545 closed by iritkatriel #27121: imghdr does not support jpg files with Lavc bytes https://bugs.python.org/issue27121 closed by iritkatriel #28464: BaseEventLoop.close should shutdown executor before marking it https://bugs.python.org/issue28464 closed by asvetlov #28534: Replace asynchat https://bugs.python.org/issue28534 closed by asvetlov #28591: imghdr doesn't recognize some jpeg formats https://bugs.python.org/issue28591 closed by iritkatriel #29406: asyncio SSL contexts leak sockets after calling close with cer https://bugs.python.org/issue29406 closed by asvetlov #30080: Add the --duplicate option for timeit https://bugs.python.org/issue30080 closed by serhiy.storchaka #30491: Add a lightweight mechanism for detecting un-awaited coroutine https://bugs.python.org/issue30491 closed by asvetlov #30677: [doc] mention that os.mkdir() raises FileNotFound if path does https://bugs.python.org/issue30677 closed by iritkatriel #31327: [doc] Add example of platform-specific support for negative ti https://bugs.python.org/issue31327 closed by iritkatriel #32007: deprecate the nis module https://bugs.python.org/issue32007 closed by iritkatriel #32181: runaway Tasks with Task.cancel() ignored. https://bugs.python.org/issue32181 closed by asvetlov #32396: Implement method to write/read to serials without blocking on https://bugs.python.org/issue32396 closed by asvetlov #32669: cgitb file to print OSError exceptions https://bugs.python.org/issue32669 closed by iritkatriel #33507: Improving the html rendered by cgitb.html https://bugs.python.org/issue33507 closed by iritkatriel #34129: CGITB does not mangle variables names https://bugs.python.org/issue34129 closed by iritkatriel #34165: uu.decode() raises binascii.Error instead of uu.Error on inval https://bugs.python.org/issue34165 closed by iritkatriel #35212: Expressions with format specifiers in f-strings give wrong cod https://bugs.python.org/issue35212 closed by eric.smith #35392: Create asyncio/sockutils.py https://bugs.python.org/issue35392 closed by asvetlov #37529: Mimetype module duplicates https://bugs.python.org/issue37529 closed by asvetlov #38955: Non indemnpotent behavior of asyncio.get_event_loop and asynci https://bugs.python.org/issue38955 closed by asvetlov #40007: An attempt to make asyncio.transport.writelines (selector) use https://bugs.python.org/issue40007 closed by asvetlov #40227: SSLError is not passed to the client during handshake https://bugs.python.org/issue40227 closed by asvetlov #40454: DEBUG kw to asyncio.run overrides DEBUG mode set elsewhere https://bugs.python.org/issue40454 closed by asvetlov #40487: Unexpected exception handler behavior in Jupyter when returnin https://bugs.python.org/issue40487 closed by asvetlov #40735: test_nntplib depends on unreliable external servers https://bugs.python.org/issue40735 closed by iritkatriel #40811: Allow to create new Event Loops on Threads https://bugs.python.org/issue40811 closed by asvetlov #40967: asyncio.Task.all_tasks() and asyncio.Task.current_task() must https://bugs.python.org/issue40967 closed by asvetlov #40977: asyncio.trsock.TransportSocket says some APIs will be prohibit https://bugs.python.org/issue40977 closed by asvetlov #41696: asyncio.run interacts surprisingly with debug mode https://bugs.python.org/issue41696 closed by asvetlov #42682: awaiting a wrapped asyncio.Task multiple times gives long, rep https://bugs.python.org/issue42682 closed by iritkatriel #42683: asyncio should handle keyboard interrupt while the event loop https://bugs.python.org/issue42683 closed by asvetlov #42698: Deadlock in pysqlite_connection_dealloc() https://bugs.python.org/issue42698 closed by hydroflask #42760: inspect.iscoroutine returns False for asynchronous generator m https://bugs.python.org/issue42760 closed by asvetlov #42838: Wait for cleanup coroutines before event loop is closed. https://bugs.python.org/issue42838 closed by asvetlov #42916: Support for DICOM image file format in imghdr module https://bugs.python.org/issue42916 closed by iritkatriel #42941: Infinite loop in asyncio sslproto https://bugs.python.org/issue42941 closed by asvetlov #43194: Add JFXX as jpeg marker in imghdr module https://bugs.python.org/issue43194 closed by iritkatriel #43215: Document Happy Eyeballs arguments of asyncio.open_connection https://bugs.python.org/issue43215 closed by asvetlov #43721: Documentation of property.{getter,setter,deleter} fails to men https://bugs.python.org/issue43721 closed by AlexWaygood #43863: [Windows] test_distutils logs: DeprecationWarning: bdist_msi https://bugs.python.org/issue43863 closed by iritkatriel #44221: ImportError: sys.meta_path is None, Python is likely shutting https://bugs.python.org/issue44221 closed by asvetlov #44318: Asyncio classes missing __slots__ https://bugs.python.org/issue44318 closed by asvetlov #44373: make Event an Awaitable https://bugs.python.org/issue44373 closed by asvetlov #44604: [Enhancement] Asyncio task decorator to provide interface to d https://bugs.python.org/issue44604 closed by asvetlov #44795: asyncio.run does not allow for graceful shutdown of main task https://bugs.python.org/issue44795 closed by asvetlov #44799: typing.get_type_hints() raises TypeError for a variable annota https://bugs.python.org/issue44799 closed by JelleZijlstra #45279: avoid redundant _commit_removals pending_removals guard https://bugs.python.org/issue45279 closed by asvetlov #45413: Add install scheme for virtual environments https://bugs.python.org/issue45413 closed by FFY00 #45744: Fix Flawfinder C Errors https://bugs.python.org/issue45744 closed by iritkatriel #46030: socket module add couple of FreeBSD constants https://bugs.python.org/issue46030 closed by asvetlov #46223: asyncio cause infinite loop during debug https://bugs.python.org/issue46223 closed by asvetlov #46246: importlib.metadata.DeprecatedList appears to be missing __slot https://bugs.python.org/issue46246 closed by jaraco #46329: Split up the CALL_NO_KW and CALL_KW instructions. https://bugs.python.org/issue46329 closed by Mark.Shannon #46421: unittest ValueError when invoking as module https://bugs.python.org/issue46421 closed by JelleZijlstra #46457: test_unittest: TestAsyncCase.test_debug_cleanup_same_loop() ha https://bugs.python.org/issue46457 closed by asvetlov #46480: Implement typing.assert_type https://bugs.python.org/issue46480 closed by JelleZijlstra #46553: typing: get_type_hints on stringified lone ClassVar raises Typ https://bugs.python.org/issue46553 closed by JelleZijlstra #46644: typing: remove callable() check from typing._type_check https://bugs.python.org/issue46644 closed by JelleZijlstra #46677: TypedDict docs are incomplete https://bugs.python.org/issue46677 closed by JelleZijlstra #46805: Add low level UDP socket functions to asyncio https://bugs.python.org/issue46805 closed by asvetlov #46843: PersistentTaskGroup API https://bugs.python.org/issue46843 closed by gvanrossum #46884: [doc] msilib.rst uses data directive to document modules https://bugs.python.org/issue46884 closed by iritkatriel #46907: Update Windows and MacOS installer to SQLite 3.38.1 https://bugs.python.org/issue46907 closed by erlendaasland #46918: The vulnerability is included in /lib/python3.9/ensurepip afte https://bugs.python.org/issue46918 closed by ned.deily #46920: Remove code made dead long ago with #if 0 https://bugs.python.org/issue46920 closed by arhadthedev #46944: Use FASTCALL calling convention in generator.throw https://bugs.python.org/issue46944 closed by kumaraditya303 #46953: use FASTCALL for __import__ builtin https://bugs.python.org/issue46953 closed by kumaraditya303 #46967: Type union for except https://bugs.python.org/issue46967 closed by iritkatriel #46968: Insufficient sigaltstack size used by CPython prevents extensi https://bugs.python.org/issue46968 closed by vstinner #46975: clang: error: linker command failed with exit code 1 (use -v t https://bugs.python.org/issue46975 closed by ned.deily #46981: Empty typing.Tuple https://bugs.python.org/issue46981 closed by serhiy.storchaka #46983: test_sqlite3: test_trace_too_much_expanded_sql() failed on AMD https://bugs.python.org/issue46983 closed by erlendaasland #46985: Upgrade ensurepip bundled pip to 22.0.4 https://bugs.python.org/issue46985 closed by ned.deily #46986: Upgrade ensurepip bundled setuptools to 60.9.3 https://bugs.python.org/issue46986 closed by ned.deily #46987: Remove _PySys_GetObjectId / _PySys_GetObjectId https://bugs.python.org/issue46987 closed by corona10 #46991: Specialize list[slice] https://bugs.python.org/issue46991 closed by kj #46993: Speed up bytearray creation from list and tuple https://bugs.python.org/issue46993 closed by asvetlov #46994: Accept explicit contextvars.Context in asyncio create_task() A https://bugs.python.org/issue46994 closed by asvetlov #46995: Make Task.set_name() mandatory for third-parties https://bugs.python.org/issue46995 closed by asvetlov #46996: Drop support of Tcl/Tk older than 8.5.12 https://bugs.python.org/issue46996 closed by serhiy.storchaka #47001: deadlock in ctypes? https://bugs.python.org/issue47001 closed by rocco.matano #47003: Cleanup _overlapped module https://bugs.python.org/issue47003 closed by asvetlov #47004: Apply bugfixes from importlib_metadata 4.11.3. https://bugs.python.org/issue47004 closed by jaraco #47005: Improve performance of bytes_repeat https://bugs.python.org/issue47005 closed by Dennis Sweeney #47008: Add Lib/site-packages to .gitignore https://bugs.python.org/issue47008 closed by Dennis Sweeney #47018: ImportError: cannot import name '_simple_enum' from 'enum' https://bugs.python.org/issue47018 closed by steven.daprano #47020: float('nan')==math.nan does NOT evaluate to True (as suggeste https://bugs.python.org/issue47020 closed by eric.smith #47023: re.sub shows key error on regex escape chars provided in repl https://bugs.python.org/issue47023 closed by eric.smith #47024: Update to OpenSSL 1.1.1n https://bugs.python.org/issue47024 closed by christian.heimes #47028: Incorrect behaviour when zipping a bunch of maps https://bugs.python.org/issue47028 closed by Dennis Sweeney #47032: CI does not detect launcher installer build failure https://bugs.python.org/issue47032 closed by steve.dower #47033: Build failure on macOS Big Sur https://bugs.python.org/issue47033 closed by Ghyslain Leclerc #47034: pickle not working correctly when custom field is directly ini https://bugs.python.org/issue47034 closed by eric.smith #47035: Rewrite asyncio queue tests with IsolatedAsyncioTestCase https://bugs.python.org/issue47035 closed by asvetlov #47036: Nested list in Dict, multiprocessing manager https://bugs.python.org/issue47036 closed by munky99999 #47038: Rewrite asyncio.wait_for test to use IsolatedAsyncioTestCase https://bugs.python.org/issue47038 closed by asvetlov #47039: Normalize asyncio future and task repr() https://bugs.python.org/issue47039 closed by asvetlov #47041: requests module parse url failed https://bugs.python.org/issue47041 closed by christian.heimes #47042: Fix test_html_doc in test_pydoc https://bugs.python.org/issue47042 closed by serhiy.storchaka #47044: Python 2.7.18 + tck/tk 8.6.1 + Osx Lion : tkinter built failes https://bugs.python.org/issue47044 closed by christian.heimes #1573931: WSGI, cgi.FieldStorage incompatibility https://bugs.python.org/issue1573931 closed by iritkatriel #1186900: nntplib shouldn't raise generic EOFError https://bugs.python.org/issue1186900 closed by iritkatriel #1047397: cgitb failures https://bugs.python.org/issue1047397 closed by iritkatriel

1 0

Request for feedback on new features for 3.11
by Gabriele March 17, 2022

March 17, 2022

Hi there I hope you will indulge me while I request some feedback on a couple of open PRs that I would like to be considered for 3.11. https://github.com/python/cpython/pull/31150 https://github.com/python/cpython/pull/31142 The first one should be pretty easy to review as it simply makes use of the new co_qualname attribute of code objects to enrich the traceback output with qualified names. Provided it gets accepted, similar changes could also be introduced in other areas, like profiling, debugging, IDLE etc... The second one might not be as easy to review. As pointed out in the attached bpo, a similar issue was already considered back in 2012, but the proposed solution there has been "opposed" because of backwards-compatibility concerns. The change I am proposing would expose thread names via the ABI (albeit still in the form of a Python object) in a backwards-compatible way. However, I am not very satisfied with the way I have implemented this for now, nor can I think of a better solution. Cheers, Gabriele

1 0

[RELEASE] Python 3.10.3, 3.9.11, 3.8.13, and 3.7.13 are now available with security content
by Łukasz Langa March 16, 2022

March 16, 2022

Welcome again to the exciting world of releasing new Python versions! Last time around I was complaining about cursed releases <https://discuss.python.org/t/python-3-10-2-3-9-10-and-3-11-0a4-are-now-avai…>. This time around I could complain about security content galore and how one of them <https://mta.openssl.org/pipermail/openssl-announce/2022-March/000216.html> ruined my ingenious idea to release Python on Pi Day and call it Py Day <https://discuss.python.org/t/py-day-is-coming-a-joint-security-release-spre…>. Well, you can’t have everything in life. Or at least not everything at once. And yet it seems this time around we’ve got a lot of security fixes all at once. Just look at this list: 15 (sic!) CVEs: libexpat upgraded from 2.4.1 to 2.4.7 (BPO-46794 <https://bugs.python.org/issue46794>, BPO-46932 <https://bugs.python.org/issue46932>, BPO-46811 <https://bugs.python.org/issue46811>, BPO-46784 <https://bugs.python.org/issue46784>, BPO-46400 <https://bugs.python.org/issue46400>) CVE-2022-0778: OpenSSL upgraded from 1.1.1l to 1.1.1n in macOS and Windows installers (BPO-47024 <https://bugs.python.org/issue47024>) CVE-2016-3189, CVE-2019-12900: bzip2 upgraded from 1.0.6 to 1.0.8 in Windows installers (BPO-44549 <https://bugs.python.org/issue44549>) CVE-2022-26488: Windows installer now ensures the correct path is being repaired when “Add to PATH” is used (BPO-46948 <https://bugs.python.org/issue46948>) CVE-2021-28363: bundled pip upgraded from 21.2.4 to 22.0.4 (BPO-46985 <https://bugs.python.org/issue46985>) authorization bypass fixed in urllib.request (BPO-46756 <https://bugs.python.org/issue46756>) REDoS avoided in importlib.metadata (BPO-46474 <https://bugs.python.org/issue46474>) SQLite upgraded from 3.36.0 to 3.37.2 in macOS and Windows installers (BPO-45925 <https://bugs.python.org/issue45925>) <https://discuss.python.org/t/python-3-10-3-3-9-11-3-8-13-and-3-7-13-are-now…>Python 3.10.3 Get it here: https://www.python.org/downloads/release/python-3103/ <https://www.python.org/downloads/release/python-3103/> Python 3.10.3 is the third maintenance release of the newest version of the Python programming language, which contains many new features and optimizations. We recommend it over the other releases listed below. This is a large bugfix release with 220 commits since 3.10.2. Just look at the change log <https://docs.python.org/release/3.10.3/whatsnew/changelog.html>! The next maintenance release of Python 3.10 is planned for early June. <https://discuss.python.org/t/python-3-10-3-3-9-11-3-8-13-and-3-7-13-are-now…>Python 3.9.11 Get it here: https://www.python.org/downloads/release/python-3911/ <https://www.python.org/downloads/release/python-3911/> This is the penultimate planned full bugfix release of Python 3.9. In mid-May, we’ll be releasing the last one, after which the 3.9 series will enter its security-only fixes period. More details in PEP 596 <https://www.python.org/dev/peps/pep-0596/>. Python 3.9 is the first Python version since 2.7 to have a regular bugfix release larger than “.10”. It’s also still a significant release at 163 commits since 3.9.10. That’s in fact 30+ commits more than between 3.9.9 and 3.9.10. The change log <https://docs.python.org/release/3.9.11/whatsnew/changelog.html> isn’t as long as the 3.10.3 one but it’s still pretty extensive! As a reminder, on macOS, the default installer is now the new universal2 variant. It’s compatible with Mac OS X 10.9 and newer, including macOS 11 Big Sur and macOS 12 Monterey. Python installed with this variant will work natively on Apple Silicon processors. <https://discuss.python.org/t/python-3-10-3-3-9-11-3-8-13-and-3-7-13-are-now…>Python 3.8.13 Get it here: https://www.python.org/downloads/release/python-3813/ <https://www.python.org/downloads/release/python-3813/> Changes <https://docs.python.org/release/3.8.13/whatsnew/changelog.html> here are almost exclusively security-only as the life cycle of Python versions prescribes. You might have noticed there is a small number of regular bug fixes nonetheless. This is because without those we wouldn’t be able to continue running the full test suite for the 3.8 branch. This in turn could hide regressions in future security fixes. <https://discuss.python.org/t/python-3-10-3-3-9-11-3-8-13-and-3-7-13-are-now…>Python 3.7.13 Get it here: https://www.python.org/downloads/release/python-3713/ <https://www.python.org/downloads/release/python-3713/> Just like 3.8, Python 3.7 is in its security-only fixes period. In turn, the changes in 3.7.13 <https://docs.python.org/release/3.7.13/whatsnew/changelog.html> look almost identical to the ones in 3.8.13. Python 3.7 will continue to receive source-only releases until June 2023. <https://discuss.python.org/t/python-3-10-3-3-9-11-3-8-13-and-3-7-13-are-now…>We hope you enjoy the new releases Your friendly release team, Łukasz Langa @ambv <https://discuss.python.org/u/ambv> Pablo Galindo Salgado @pablogsal <https://discuss.python.org/u/pablogsal> Ned Deily @nad <https://discuss.python.org/u/nad> Steve Dower @steve.dower <https://discuss.python.org/u/steve.dower>

3 3

Python 3.9.11
by Prasad, PCRaghavendra March 16, 2022

March 16, 2022

Hi Team, Can someone please let us know the release date of Python 3.9.11 ( with libexpat 2.4.8 security issues fixed ) In the python.org releases it was mentioned as 14-march-2022, but still, I couldn't see the bin/source code. Can someone help with this Thanks, Raghavendra Internal Use - Confidential

5 4

Gathering edge cases around type annotation PEPs
by Brett Cannon March 15, 2022

March 15, 2022

I created https://discuss.python.org/t/finding-edge-cases-for-peps-484-463-and-649-ty… on behalf of the SC to help gather edge cases where the various approaches that PEPs have proposed will fail. Our hope is to get an overall picture of the trade-offs the various PEPs ask us to make. PLEASE DO NOT REPLY HERE! I'm reaching out to the typing SIG and Pydantic (at least) and it's much easier for others to log into discuss.python.org with a GitHub account than it is to sign up for this mailing list just to participate in this discussion.

1 0

Defining tiered platform support
by Brett Cannon March 15, 2022

March 15, 2022

Do we officially support NetBSD? Do you know how to find out if we do? You might think to look at https://www.python.org/dev/peps/pep-0011/#supporting-platforms , but that just loosely defines the criteria and it still doesn't list the actual platforms we support. (BTW I don't know if we do officially support NetBSD, hence this email.) I think we should clarify this sort of thing and be a bit more upfront with the level of support we expect/provide for a platform. As such, I want to restructure PEP 11 to list the platforms we support, not just list the platforms we stopped supporting. To do this I want define 3 different tiers that outline what our support requirements and promises are for platforms. Tier 1 is the stuff we run CI against: latest Windows, latest macOS, Linux w/ the latest glibc (I don't know of a better way to define Linux support as I don't know if a per-distro list is the right abstraction). These are platforms we won't even let code be committed for if they would break; they block releases if they don't work. These platforms we all implicitly promise to support. Tier 2 is the platforms we would revert a change within 24 hours if they broke: latest FeeBSD, older Windows, older macOS, Linux w/ older glibc.This is historically the "stable buildbot plus a core dev" group of platforms. The change I would like to see is two core devs (in case one is on vacation), and a policy as to how a platform ends up here (e.g. SC must okay it based on consensus of everyone). The stable buildbot would still be needed to know if a release is blocked as we would hold a release up if they were red. The platform and the core devs supporting these platforms would be listed in PEP 11. Tier 3 are platforms we accept PRs for to keep working, but they won't block a release. At least one core dev must be listed to be in charge of PRs that affect the platform. A buildbot would be nice but not required. I'm thinking either historical platforms we support or something like Emscripten that's working towards being a tier 2 platform. I'm not sure if we want to have explicit approval to be in this tier beyond the outlined requirements, but obviously if a core dev isn't keeping up with PRs then the platform will be dropped. All other platforms we do not directly support in the repository and it is up to the community to keep functioning. We can obviously keep accepting general patches to up compatibility, but platform-specific patches for anything outside of these tiers wouldn't. No issue if someone provides a buildbot for their own benefit, but these buildbots would never hold anything up. When a Python version hits b1, then that platform is considered supported for that version as long as the requirements outlined above continue to be met. Once the Python version hits EOL then like anything, support ends no matter what. We could capture the supported platforms for a version in the release schedule PEP if people want. I would expect appropriate labels being added to the builders listed at https://buildbot.python.org/all/#/builders to signify which platforms we block releases for (e.g. `tier-1,`, `tier-2`, or `release-blocker` as a generic label). I would expect PEP 11 to list the appropriate C symbol that's set for that platform, e.g. __linux__. I don't know if we want to bother listing CPU architectures since we are a pure C project which makes CPU architecture less of a thing, but I'm personally open to the idea of CPU architectures being a part of the platform definition. I don't think we should cover C compilers here as that's covered by PEP 7. Otherwise PEP 7 should only list C versions/features and PEP 11 lists compilers and their versions. FYI the tier support idea and the rough definitions above come from Rust: https://doc.rust-lang.org/nightly/rustc/platform-support.html .

9 19

PEP 684: A Per-Interpreter GIL
by Eric Snow March 14, 2022

March 14, 2022

I'd really appreciate feedback on this new PEP about making the GIL per-interpreter. The PEP targets 3.11, but we'll see if that is too close. I don't mind waiting one more release, though I'd prefer 3.11 (obviously). Regardless, I have no intention of rushing this through at the expense of cutting corners. Hence, we'll see how it goes. The PEP text is included inline below. Thanks! -eric =================================================== PEP: 684 Title: A Per-Interpreter GIL Author: Eric Snow <ericsnowcurrently(a)gmail.com> Discussions-To: python-dev(a)python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 08-Mar-2022 Python-Version: 3.11 Post-History: 08-Mar-2022 Resolution: Abstract ======== Since Python 1.5 (1997), CPython users can run multiple interpreters in the same process. However, interpreters in the same process have always shared a significant amount of global state. This is a source of bugs, with a growing impact as more and more people use the feature. Furthermore, sufficient isolation would facilitate true multi-core parallelism, where interpreters no longer share the GIL. The changes outlined in this proposal will result in that level of interpreter isolation. High-Level Summary ================== At a high level, this proposal changes CPython in the following ways: * stops sharing the GIL between interpreters, given sufficient isolation * adds several new interpreter config options for isolation settings * adds some public C-API for fine-grained control when creating interpreters * keeps incompatible extensions from causing problems The GIL ------- The GIL protects concurrent access to most of CPython's runtime state. So all that GIL-protected global state must move to each interpreter before the GIL can. (In a handful of cases, other mechanisms can be used to ensure thread-safe sharing instead, such as locks or "immortal" objects.) CPython Runtime State --------------------- Properly isolating interpreters requires that most of CPython's runtime state be stored in the ``PyInterpreterState`` struct. Currently, only a portion of it is; the rest is found either in global variables or in ``_PyRuntimeState``. Most of that will have to be moved. This directly coincides with an ongoing effort (of many years) to greatly reduce internal use of C global variables and consolidate the runtime state into ``_PyRuntimeState`` and ``PyInterpreterState``. (See `Consolidating Runtime Global State`_ below.) That project has `significant merit on its own <Benefits to Consolidation_>`_ and has faced little controversy. So, while a per-interpreter GIL relies on the completion of that effort, that project should not be considered a part of this proposal--only a dependency. Other Isolation Considerations ------------------------------ CPython's interpreters must be strictly isolated from each other, with few exceptions. To a large extent they already are. Each interpreter has its own copy of all modules, classes, functions, and variables. The CPython C-API docs `explain further <caveats_>`_. .. _caveats: https://docs.python.org/3/c-api/init.html#bugs-and-caveats However, aside from what has already been mentioned (e.g. the GIL), there are a couple of ways in which interpreters still share some state. First of all, some process-global resources (e.g. memory, file descriptors, environment variables) are shared. There are no plans to change this. Second, some isolation is faulty due to bugs or implementations that did not take multiple interpreters into account. This includes CPython's runtime and the stdlib, as well as extension modules that rely on global variables. Bugs should be opened in these cases, as some already have been. Depending on Immortal Objects ----------------------------- :pep:`683` introduces immortal objects as a CPython-internal feature. With immortal objects, we can share any otherwise immutable global objects between all interpreters. Consequently, this PEP does not need to address how to deal with the various objects `exposed in the public C-API <capi objects_>`_. It also simplifies the question of what to do about the builtin static types. (See `Global Objects`_ below.) Both issues have alternate solutions, but everything is simpler with immortal objects. If PEP 683 is not accepted then this one will be updated with the alternatives. This lets us reduce noise in this proposal. Motivation ========== The fundamental problem we're solving here is a lack of true multi-core parallelism (for Python code) in the CPython runtime. The GIL is the cause. While it usually isn't a problem in practice, at the very least it makes Python's multi-core story murky, which makes the GIL a consistent distraction. Isolated interpreters are also an effective mechanism to support certain concurrency models. :pep:`554` discusses this in more detail. Indirect Benefits ----------------- Most of the effort needed for a per-interpreter GIL has benefits that make those tasks worth doing anyway: * makes multiple-interpreter behavior more reliable * has led to fixes for long-standing runtime bugs that otherwise hadn't been prioritized * has been exposing (and inspiring fixes for) previously unknown runtime bugs * has driven cleaner runtime initialization (:pep:`432`, :pep:`587`) * has driven cleaner and more complete runtime finalization * led to structural layering of the C-API (e.g. ``Include/internal``) * also see `Benefits to Consolidation`_ below Furthermore, much of that work benefits other CPython-related projects: * performance improvements ("faster-cpython") * pre-fork application deployment (e.g. Instagram) * extension module isolation (see :pep:`630`, etc.) * embedding CPython Existing Use of Multiple Interpreters ------------------------------------- The C-API for multiple interpreters has been used for many years. However, until relatively recently the feature wasn't widely known, nor extensively used (with the exception of mod_wsgi). In the last few years use of multiple interpreters has been increasing. Here are some of the public projects using the feature currently: * `mod_wsgi <https://github.com/GrahamDumpleton/mod_wsgi>`_ * `OpenStack Ceph <https://github.com/ceph/ceph/pull/14971>`_ * `JEP <https://github.com/ninia/jep>`_ * `Kodi <https://github.com/xbmc/xbmc>`_ Note that, with :pep:`554`, multiple interpreter usage would likely grow significantly (via Python code rather than the C-API). PEP 554 ------- :pep:`554` is strictly about providing a minimal stdlib module to give users access to multiple interpreters from Python code. In fact, it specifically avoids proposing any changes related to the GIL. Consider, however, that users of that module would benefit from a per-interpreter GIL, which makes PEP 554 more appealing. Rationale ========= During initial investigations in 2014, a variety of possible solutions for multi-core Python were explored, but each had its drawbacks without simple solutions: * the existing practice of releasing the GIL in extension modules * doesn't help with Python code * other Python implementations (e.g. Jython, IronPython) * CPython dominates the community * remove the GIL (e.g. gilectomy, "no-gil") * too much technical risk (at the time) * Trent Nelson's "PyParallel" project * incomplete; Windows-only at the time * ``multiprocessing`` * too much work to make it effective enough; high penalties in some situations (at large scale, Windows) * other parallelism tools (e.g. dask, ray, MPI) * not a fit for the stdlib * give up on multi-core (e.g. async, do nothing) * this can only end in tears Even in 2014, it was fairly clear that a solution using isolated interpreters did not have a high level of technical risk and that most of the work was worth doing anyway. (The downside was the volume of work to be done.) Specification ============= As `summarized above <High-Level Summary_>`__, this proposal involves the following changes, in the order they must happen: 1. `consolidate global runtime state <Consolidating Runtime Global State_>`_ (including objects) into ``_PyRuntimeState`` 2. move nearly all of the state down into ``PyInterpreterState`` 3. finally, move the GIL down into ``PyInterpreterState`` 4. everything else * add to the public C-API * implement restrictions in ``ExtensionFileLoader`` * work with popular extension maintainers to help with multi-interpreter support Per-Interpreter State --------------------- The following runtime state will be moved to ``PyInterpreterState``: * all global objects that are not safely shareable (fully immutable) * the GIL * mutable, currently protected by the GIL * mutable, currently protected by some other per-interpreter lock * mutable, may be used independently in different interpreters * all other mutable (or effectively mutable) state not otherwise excluded below Furthermore, a number of parts of the global state have already been moved to the interpreter, such as GC, warnings, and atexit hooks. The following state will not be moved: * global objects that are safely shareable, if any * immutable, often ``const`` * treated as immutable * related to CPython's ``main()`` execution * related to the REPL * set during runtime init, then treated as immutable * mutable, protected by some global lock * mutable, atomic Note that currently the allocators (see ``Objects/obmalloc.c``) are shared between all interpreters, protected by the GIL. They will need to move to each interpreter (or a global lock will be needed). This is the highest risk part of the work to isolate interpreters and may require more than just moving fields down from ``_PyRuntimeState``. Some of the complexity is reduced if CPython switches to a thread-safe allocator like mimalloc. .. _proposed capi: C-API ----- The following private API will be made public: * ``_PyInterpreterConfig`` * ``_Py_NewInterpreter()`` (as ``Py_NewInterpreterEx()``) The following fields will be added to ``PyInterpreterConfig``: * ``own_gil`` - (bool) create a new interpreter lock (instead of sharing with the main interpreter) * ``strict_extensions`` - fail import in this interpreter for incompatible extensions (see `Restricting Extension Modules`_) Restricting Extension Modules ----------------------------- Extension modules have many of the same problems as the runtime when state is stored in global variables. :pep:`630` covers all the details of what extensions must do to support isolation, and thus safely run in multiple interpreters at once. This includes dealing with their globals. Extension modules that do not implement isolation will only run in the main interpreter. In all other interpreters, the import will raise ``ImportError``. This will be done through ``importlib._bootstrap_external.ExtensionFileLoader``. We will work with popular extensions to help them support use in multiple interpreters. This may involve adding to CPython's public C-API, which we will address on a case-by-case basis. Extension Module Compatibility '''''''''''''''''''''''''''''' As noted in `Extension Modules`_, many extensions work fine in multiple interpreters without needing any changes. The import system will still fail if such a module doesn't explicitly indicate support. At first, not many extension modules will, so this is a potential source of frustration. We will address this by adding a context manager to temporarily disable the check on multiple interpreter support: ``importlib.util.allow_all_extensions()``. Documentation ------------- The "Sub-interpreter support" section of ``Doc/c-api/init.rst`` will be updated with the added API. Impact ====== Backwards Compatibility ----------------------- No behavior or APIs are intended to change due to this proposal, with one exception noted in `the next section <Extension Modules_>`_. The existing C-API for managing interpreters will preserve its current behavior, with new behavior exposed through new API. No other API or runtime behavior is meant to change, including compatibility with the stable ABI. See `Objects Exposed in the C-API`_ below for related discussion. Extension Modules ''''''''''''''''' Currently the most common usage of Python, by far, is with the main interpreter running by itself. This proposal has zero impact on extension modules in that scenario. Likewise, for better or worse, there is no change in behavior under multiple interpreters created using the existing ``Py_NewInterpreter()``. Keep in mind that some extensions already break when used in multiple interpreters, due to keeping module state in global variables. They may crash or, worse, experience inconsistent behavior. That was part of the motivation for :pep:`630` and friends, so this is not a new situation nor a consequence of this proposal. In contrast, when the `proposed API <proposed capi_>`_ is used to create multiple interpreters, the default behavior will change for some extensions. In that case, importing an extension will fail (outside the main interpreter) if it doesn't indicate support for multiple interpreters. For extensions that already break in multiple interpreters, this will be an improvement. Now we get to the break in compatibility mentioned above. Some extensions are safe under multiple interpreters, even though they haven't indicated that. Unfortunately, there is no reliable way for the import system to infer that such an extension is safe, so importing them will still fail. This case is addressed in `Extension Module Compatibility`_ below. Extension Module Maintainers ---------------------------- One related consideration is that a per-interpreter GIL will likely drive increased use of multiple interpreters, particularly if :pep:`554` is accepted. Some maintainers of large extension modules have expressed concern about the increased burden they anticipate due to increased use of multiple interpreters. Specifically, enabling support for multiple interpreters will require substantial work for some extension modules. To add that support, the maintainer(s) of such a module (often volunteers) would have to set aside their normal priorities and interests to focus on compatibility (see :pep:`630`). Of course, extension maintainers are free to not add support for use in multiple interpreters. However, users will increasingly demand such support, especially if the feature grows in popularity. Either way, the situation can be stressful for maintainers of such extensions, particularly when they are doing the work in their spare time. The concerns they have expressed are understandable, and we address the partial solution in `Restricting Extension Modules`_ below. Alternate Python Implementations -------------------------------- Other Python implementation are not required to provide support for multiple interpreters in the same process (though some do already). Security Implications --------------------- There is no known impact to security with this proposal. Maintainability --------------- On the one hand, this proposal has already motivated a number of improvements that make CPython *more* maintainable. That is expected to continue. On the other hand, the underlying work has already exposed various pre-existing defects in the runtime that have had to be fixed. That is also expected to continue as multiple interpreters receive more use. Otherwise, there shouldn't be a significant impact on maintainability, so the net effect should be positive. Performance ----------- The work to consolidate globals has already provided a number of improvements to CPython's performance, both speeding it up and using less memory, and this should continue. Performance benefits to a per-interpreter GIL have not been explored. At the very least, it is not expected to make CPython slower (as long as interpreters are sufficiently isolated). How to Teach This ================= This is an advanced feature for users of the C-API. There is no expectation that this will be taught. That said, if it were taught then it would boil down to the following: In addition to Py_NewInterpreter(), you can use Py_NewInterpreterEx() to create an interpreter. The config you pass it indicates how you want that interpreter to behave. Reference Implementation ======================== <TBD> Open Issues =========== * What are the risks/hurdles involved with moving the allocators? * Is ``allow_all_extensions`` the best name for the context manager? Deferred Functionality ====================== * ``PyInterpreterConfig`` option to always run the interpreter in a new thread * ``PyInterpreterConfig`` option to assign a "main" thread to the interpreter and only run in that thread Rejected Ideas ============== <TBD> Extra Context ============= Sharing Global Objects ---------------------- We are sharing some global objects between interpreters. This is an implementation detail and relates more to `globals consolidation <Consolidating Runtime Global State>`_ than to this proposal, but it is a significant enough detail to explain here. The alternative is to share no objects between interpreters, ever. To accomplish that, we'd have to sort out the fate of all our static types, as well as deal with compatibility issues for the many objects `exposed in the public C-API <capi objects_>`_. That approach introduces a meaningful amount of extra complexity and higher risk, though prototyping has demonstrated valid solutions. Also, it would likely result in a performance penalty. `Immortal objects <Depending on Immortal Objects_>`_ allow us to share the otherwise immutable global objects. That way we avoid the extra costs. .. _capi objects: Objects Exposed in the C-API '''''''''''''''''''''''''''' The C-API (including the limited API) exposes all the builtin types, including the builtin exceptions, as well as the builtin singletons. The exceptions are exposed as ``PyObject *`` but the rest are exposed as the static values rather than pointers. This was one of the few non-trivial problems we had to solve for per-interpreter GIL. With immortal objects this is a non-issue. Consolidating Runtime Global State ---------------------------------- As noted in `CPython Runtime State`_ above, there is an active effort (separate from this PEP) to consolidate CPython's global state into the ``_PyRuntimeState`` struct. Nearly all the work involves moving that state from global variables. The project is particularly relevant to this proposal, so below is some extra detail. Benefits to Consolidation ''''''''''''''''''''''''' Consolidating the globals has a variety of benefits: * greatly reduces the number of C globals (best practice for C code) * the move draws attention to runtime state that is unstable or broken * encourages more consistency in how runtime state is used * makes multiple-interpreter behavior more reliable * leads to fixes for long-standing runtime bugs that otherwise haven't been prioritized * exposes (and inspires fixes for) previously unknown runtime bugs * facilitates cleaner runtime initialization and finalization * makes it easier to discover/identify CPython's runtime state * makes it easier to statically allocate runtime state in a consistent way * better memory locality for runtime state * structural layering of the C-API (e.g. ``Include/internal``) Furthermore, much of that work benefits other CPython-related projects: * performance improvements ("faster-cpython") * pre-fork application deployment (e.g. Instagram) * extension module isolation (see :pep:`630`, etc.) * embedding CPython Scale of Work ''''''''''''' The number of global variables to be moved is large enough to matter, but most are Python objects that can be dealt with in large groups (like ``Py_IDENTIFIER``). In nearly all cases, moving these globals to the interpreter is highly mechanical. That doesn't require cleverness but instead requires someone to put in the time. State To Be Moved ''''''''''''''''' The remaining global variables can be categorized as follows: * global objects * static types (incl. exception types) * non-static types (incl. heap types, structseq types) * singletons (static) * singletons (initialized once) * cached objects * non-objects * will not (or unlikely to) change after init * only used in the main thread * initialized lazily * pre-allocated buffers * state Those globals are spread between the core runtime, the builtin modules, and the stdlib extension modules. For a breakdown of the remaining globals, run: .. code-block:: bash ./python Tools/c-analyzer/table-file.py Tools/c-analyzer/cpython/globals-to-fix.tsv Already Completed Work '''''''''''''''''''''' As mentioned, this work has been going on for many years. Here are some of the things that have already been done: * cleanup of runtime initialization (see :pep:`432` / :pep:`587`) * extension module isolation machinery (see :pep:`384` / :pep:`3121` / :pep:`489`) * isolation for many builtin modules * isolation for many stdlib extension modules * addition of ``_PyRuntimeState`` * no more ``_Py_IDENTIFIER()`` * statically allocated: * empty string * string literals * identifiers * latin-1 strings * length-1 bytes * empty tuple Tooling ''''''' As already indicated, there are several tools to help identify the globals and reason about them. * ``Tools/c-analyzer/cpython/globals-to-fix.tsv`` - the list of remaining globals * ``Tools/c-analyzer/c-analyzer.py`` * ``analyze`` - identify all the globals * ``check`` - fail if there are any unsupported globals that aren't ignored * ``Tools/c-analyzer/table-file.py`` - summarize the known globals Also, the check for unsupported globals is incorporated into CI so that no new globals are accidentally added. Global Objects '''''''''''''' Global objects that are safe to be shared (without a GIL) between interpreters can stay on ``_PyRuntimeState``. Not only must the object be effectively immutable (e.g. singletons, strings), but not even the refcount can change for it to be safe. Immortality (:pep:`683`) provides that. (The alternative is that no objects are shared, which adds significant complexity to the solution, particularly for the objects `exposed in the public C-API <capi objects_>`_.) Builtin static types are a special case of global objects that will be shared. They are effectively immutable except for one part: ``__subclasses__`` (AKA ``tp_subclasses``). We expect that nothing else on a builtin type will change, even the content of ``__dict__`` (AKA ``tp_dict``). ``__subclasses__`` for the builtin types will be dealt with by making it a getter that does a lookup on the current ``PyInterpreterState`` for that type. References ========== Related: * :pep:`384` * :pep:`432` * :pep:`489` * :pep:`554` * :pep:`573` * :pep:`587` * :pep:`630` * :pep:`683` * :pep:`3121` Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.

6 10

PEP 683: "Immortal Objects, Using a Fixed Refcount" (round 3)
by Eric Snow March 14, 2022

March 14, 2022

I've updated PEP 683 for the feedback I've gotten. Thanks again for that! The updated PEP text is included below. The largest changes involve either the focus of the PEP (internal mechanism to mark objects immortal) or the possible ways that things can break on older 32-bit stable ABI extensions. All other changes are smaller. Given the last round of discussion, I'm hoping this will be the last round before we go to the steering council. -eric ---------------- PEP: 683 Title: Immortal Objects, Using a Fixed Refcount Author: Eric Snow <ericsnowcurrently(a)gmail.com>, Eddie Elizondo <eduardo.elizondorueda(a)gmail.com> Discussions-To: https://mail.python.org/archives/list/python-dev@python.org/thread/TPLEYDCX… Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 10-Feb-2022 Python-Version: 3.11 Post-History: 15-Feb-2022, 19-Feb-2022, 28-Feb-2022 Resolution: Abstract ======== Currently the CPython runtime maintains a `small amount of mutable state <Runtime Object State_>`_ in the allocated memory of each object. Because of this, otherwise immutable objects are actually mutable. This can have a large negative impact on CPU and memory performance, especially for approaches to increasing Python's scalability. This proposal mandates that, internally, CPython will support marking an object as one for which that runtime state will no longer change. Consequently, such an object's refcount will never reach 0, and so the object will never be cleaned up. We call these objects "immortal". (Normally, only a relatively small number of internal objects will ever be immortal.) The fundamental improvement here is that now an object can be truly immutable. Scope ----- Object immortality is meant to be an internal-only feature. So this proposal does not include any changes to public API or behavior (with one exception). As usual, we may still add some private (yet publicly accessible) API to do things like immortalize an object or tell if one is immortal. Any effort to expose this feature to users would need to be proposed separately. There is one exception to "no change in behavior": refcounting semantics for immortal objects will differ in some cases from user expectations. This exception, and the solution, are discussed below. Most of this PEP focuses on an internal implementation that satisfies the above mandate. However, those implementation details are not meant to be strictly proscriptive. Instead, at the least they are included to help illustrate the technical considerations required by the mandate. The actual implementation may deviate somewhat as long as it satisfies the constraints outlined below. Furthermore, the acceptability of any specific implementation detail described below does not depend on the status of this PEP, unless explicitly specified. For example, the particular details of: * how to mark something as immortal * how to recognize something as immortal * which subset of functionally immortal objects are marked as immortal * which memory-management activities are skipped or modified for immortal objects are not only CPython-specific but are also private implementation details that are expected to change in subsequent versions. Implementation Summary ---------------------- Here's a high-level look at the implementation: If an object's refcount matches a very specific value (defined below) then that object is treated as immortal. The CPython C-API and runtime will not modify the refcount (or other runtime state) of an immortal object. Aside from the change to refcounting semantics, there is one other possible negative impact to consider. A naive implementation of the approach described below makes CPython roughly 4% slower. However, the implementation is performance-neutral once known mitigations are applied. Motivation ========== As noted above, currently all objects are effectively mutable. That includes "immutable" objects like ``str`` instances. This is because every object's refcount is frequently modified as the object is used during execution. This is especially significant for a number of commonly used global (builtin) objects, e.g. ``None``. Such objects are used a lot, both in Python code and internally. That adds up to a consistent high volume of refcount changes. The effective mutability of all Python objects has a concrete impact on parts of the Python community, e.g. projects that aim for scalability like Instragram or the effort to make the GIL per-interpreter. Below we describe several ways in which refcount modification has a real negative effect on such projects. None of that would happen for objects that are truly immutable. Reducing CPU Cache Invalidation ------------------------------- Every modification of a refcount causes the corresponding CPU cache line to be invalidated. This has a number of effects. For one, the write must be propagated to other cache levels and to main memory. This has small effect on all Python programs. Immortal objects would provide a slight relief in that regard. On top of that, multi-core applications pay a price. If two threads (running simultaneously on distinct cores) are interacting with the same object (e.g. ``None``) then they will end up invalidating each other's caches with each incref and decref. This is true even for otherwise immutable objects like ``True``, ``0``, and ``str`` instances. CPython's GIL helps reduce this effect, since only one thread runs at a time, but it doesn't completely eliminate the penalty. Avoiding Data Races ------------------- Speaking of multi-core, we are considering making the GIL a per-interpreter lock, which would enable true multi-core parallelism. Among other things, the GIL currently protects against races between multiple concurrent threads that may incref or decref the same object. Without a shared GIL, two running interpreters could not safely share any objects, even otherwise immutable ones like ``None``. This means that, to have a per-interpreter GIL, each interpreter must have its own copy of *every* object. That includes the singletons and static types. We have a viable strategy for that but it will require a meaningful amount of extra effort and extra complexity. The alternative is to ensure that all shared objects are truly immutable. There would be no races because there would be no modification. This is something that the immortality proposed here would enable for otherwise immutable objects. With immortal objects, support for a per-interpreter GIL becomes much simpler. Avoiding Copy-on-Write ---------------------- For some applications it makes sense to get the application into a desired initial state and then fork the process for each worker. This can result in a large performance improvement, especially memory usage. Several enterprise Python users (e.g. Instagram, YouTube) have taken advantage of this. However, the above refcount semantics drastically reduce the benefits and have led to some sub-optimal workarounds. Also note that "fork" isn't the only operating system mechanism that uses copy-on-write semantics. Anything that uses ``mmap`` relies on copy-on-write, including sharing data from shared object files between processes. Rationale ========= The proposed solution is obvious enough that both of this proposal's authors came to the same conclusion (and implementation, more or less) independently. The Pyston project `uses a similar approach <Pyston_>`_. Other designs were also considered. Several possibilities have also been discussed on python-dev in past years. Alternatives include: * use a high bit to mark "immortal" but do not change ``Py_INCREF()`` * add an explicit flag to objects * implement via the type (``tp_dealloc()`` is a no-op) * track via the object's type object * track with a separate table Each of the above makes objects immortal, but none of them address the performance penalties from refcount modification described above. In the case of per-interpreter GIL, the only realistic alternative is to move all global objects into ``PyInterpreterState`` and add one or more lookup functions to access them. Then we'd have to add some hacks to the C-API to preserve compatibility for the may objects exposed there. The story is much, much simpler with immortal objects Impact ====== Benefits -------- Most notably, the cases described in the above examples stand to benefit greatly from immortal objects. Projects using pre-fork can drop their workarounds. For the per-interpreter GIL project, immortal objects greatly simplifies the solution for existing static types, as well as objects exposed by the public C-API. In general, a strong immutability guarantee for objects enables Python applications to scale like never before. This is because they can then leverage multi-core parallelism without a tradeoff in memory usage. This is reflected in most of the above cases. Performance ----------- A naive implementation shows `a 4% slowdown`_. We have demonstrated a return to performance-neutral with a handful of basic mitigations applied. See the `mitigation`_ section below. On the positive side, immortal objects save a significant amount of memory when used with a pre-fork model. Also, immortal objects provide opportunities for specialization in the eval loop that would improve performance. .. _a 4% slowdown: https://github.com/python/cpython/pull/19474#issuecomment-1032944709 Backward Compatibility ---------------------- Ideally this internal-only feature would be completely compatible. However, it does involve a change to refcount semantics in some cases. Only immortal objects are affected, but this includes high-use objects like ``None``, ``True``, and ``False``. Specifically, when an immortal object is involved: * code that inspects the refcount will see a really, really large value * the new noop behavior may break code that: * depends specifically on the refcount to always increment or decrement (or have a specific value from ``Py_SET_REFCNT()``) * relies on any specific refcount value, other than 0 or 1 * directly manipulates the refcount to store extra information there * in 32-bit pre-3.11 `Stable ABI`_ extensions, objects may leak due to `Accidental Immortality`_ * such extensions may crash due to `Accidental De-Immortalizing`_ Again, those changes in behavior only apply to immortal objects, not most of the objects a user will access. Furthermore, users cannot mark an object as immortal so no user-created objects will ever have that changed behavior. Users that rely on any of the changing behavior for global (builtin) objects are already in trouble. So the overall impact should be small. Also note that code which checks for refleaks should keep working fine, unless it checks for hard-coded small values relative to some immortal object. The problems noticed by `Pyston`_ shouldn't apply here since we do not modify the refcount. See `Public Refcount Details`_ below for further discussion. Accidental Immortality '''''''''''''''''''''' Hypothetically, a non-immortal object could be incref'ed so much that it reaches the magic value needed to be considered immortal. That means it would accidentally never be cleaned up (by going back to 0). On 64-bit builds, this accidental scenario is so unlikely that we need not worry. Even if done deliberately by using ``Py_INCREF()`` in a tight loop and each iteration only took 1 CPU cycle, it would take 2^60 cycles (if the immortal bit were 2^60). At a fast 5 GHz that would still take nearly 250,000,000 seconds (over 2,500 days)! Also note that it is doubly unlikely to be a problem because it wouldn't matter until the refcount got back to 0 and the object was cleaned up. So any object that hit that magic "immortal" refcount value would have to be decref'ed that many times again before the change in behavior would be noticed. Again, the only realistic way that the magic refcount would be reached (and then reversed) is if it were done deliberately. (Of course, the same thing could be done efficiently using ``Py_SET_REFCNT()`` though that would be even less of an accident.) At that point we don't consider it a concern of this proposal. On 32-bit builds it isn't so obvious. Let's say the magic refcount were 2^30. Using the same specs as above, it would take roughly 4 seconds to accidentally immortalize an object. Under reasonable conditions, it is still highly unlikely that an object be accidentally immortalized. It would have to meet these criteria: * targeting a non-immortal object (so not one of the high-use builtins) * the extension increfs without a corresponding decref (e.g. returns from a function or method) * no other code decrefs the object in the meantime Even at a much less frequent rate incref it would not take long to reach accidental immortality (on 32-bit). However, then it would have to run through the same number of (now noop-ing) decrefs before that one object would be effectively leaking. This is highly unlikely, especially because the calculations assume no decrefs. Furthermore, this isn't all that different from how such 32-bit extensions can already incref an object past 2^31 and turn the refcount negative. If that were an actual problem then we would have heard about it. Between all of the above cases, the proposal doesn't consider accidental immortality a problem. Stable ABI '''''''''' The implementation approach described in this PEP is compatible with extensions compiled to the stable ABI (with the exception of `Accidental Immortality`_ and `Accidental De-Immortalizing`_). Due to the nature of the stable ABI, unfortunately, such extensions use versions of ``Py_INCREF()``, etc. that directly modify the object's ``ob_refcnt`` field. This will invalidate all the performance benefits of immortal objects. However, we do ensure that immortal objects (mostly) stay immortal in that situation. We set the initial refcount of immortal objects to a value high above the magic refcount value, but one that still matches the high bit. Thus we can still identify such objects as immortal. (See `_Py_IMMORTAL_REFCNT`_.) At worst, objects in that situation would feel the effects described in the `Motivation`_ section. Even then the overall impact is unlikely to be significant. Accidental De-Immortalizing ''''''''''''''''''''''''''' 32-bit builds of older stable ABI extensions can take `Accidental Immortality`_ to the next level. Hypothetically, such an extension could incref an object to a value on the next highest bit above the magic refcount value. For example, if the magic value were 2^30 and the initial immortal refcount were thus 2^30 + 2^29 then it would take 2^29 increfs by the extension to reach a value of 2^31, making the object non-immortal. (Of course, a refcount that high would probably already cause a crash, regardless of immortal objects.) The more problematic case is where such a 32-bit stable ABI extension goes crazy decref'ing an already immortal object. Continuing with the above example, it would take 2^29 asymmetric decrefs to drop below the magic immortal refcount value. So an object like ``None`` could be made mortal and subject to decref. That still wouldn't be a problem until somehow the decrefs continue on that object until it reaches 0. For many immortal objects, like ``None``, the extension will crash the process if it tries to dealloc the object. For the other immortal objects, the dealloc might be okay. However, there will be runtime code expecting the formerly-immortal object to be around forever. That code will probably crash. Again, the likelihood of this happening is extremely small, even on 32-bit builds. It would require roughly a billion decrefs on that one object without a corresponding incref. The most likely scenario is the following: A "new" reference to ``None`` is returned by many functions and methods. Unlike with non-immortal objects, the 3.11 runtime will almost never incref ``None`` before giving it to the extension. However, the extension *will* decref it when done with it (unless it returns it). Each time that exchange happens with the one object, we get one step closer to a crash. How realistic is it that some form of that exchange (with a single object) will happen a billion times in the lifetime of a Python process on 32-bit? If it is a problem, how could it be addressed? As to how realistic, the answer isn't clear currently. However, the mitigation is simple enough that we can safely proceed under the assumption that it would be a problem. Here are some possible solutions (only needed on 32-bit): * periodically reset the refcount for immortal objects (only enable this if a stable ABI extension is imported?) * special-case immortal objects in tp_dealloc() for the relevant types (but not int, due to frequency?) * provide a runtime flag for disabling immortality Alternate Python Implementations -------------------------------- This proposal is CPython-specific. However, it does relate to the behavior of the C-API, which may affect other Python implementations. Consequently, the effect of changed behavior described in `Backward Compatibility`_ above also applies here (e.g. if another implementation is tightly coupled to specific refcount values, other than 0, or on exactly how refcounts change, then they may impacted). Security Implications --------------------- This feature has no known impact on security. Maintainability --------------- This is not a complex feature so it should not cause much mental overhead for maintainers. The basic implementation doesn't touch much code so it should have much impact on maintainability. There may be some extra complexity due to performance penalty mitigation. However, that should be limited to where we immortalize all objects post-init and that code will be in one place. Specification ============= The approach involves these fundamental changes: * add `_Py_IMMORTAL_REFCNT`_ (the magic value) to the internal C-API * update ``Py_INCREF()`` and ``Py_DECREF()`` to no-op for objects with the magic refcount (or its most significant bit) * do the same for any other API that modifies the refcount * stop modifying ``PyGC_Head`` for immortal GC objects ("containers") * ensure that all immortal objects are cleaned up during runtime finalization Then setting any object's refcount to ``_Py_IMMORTAL_REFCNT`` makes it immortal. (There are other minor, internal changes which are not described here.) In the following sub-sections we dive into the details. First we will cover some conceptual topics, followed by more concrete aspects like specific affected APIs. Public Refcount Details ----------------------- In `Backward Compatibility`_ we introduced possible ways that user code might be broken by the change in this proposal. Any contributing misunderstanding by users is likely due in large part to the names of the refcount-related API and to how the documentation explains those API (and refcounting in general). Between the names and the docs, we can clearly see answers to the following questions: * what behavior do users expect? * what guarantees do we make? * do we indicate how to interpret the refcount value they receive? * what are the use cases under which a user would set an object's refcount to a specific value? * are users setting the refcount of objects they did not create? As part of this proposal, we must make sure that users can clearly understand on which parts of the refcount behavior they can rely and which are considered implementation details. Specifically, they should use the existing public refcount-related API and the only refcount values with any meaning are 0 and 1. (Some code relies on 1 as an indicator that the object can be safely modified.) All other values are considered "not 0 or 1". This information will be clarified in the `documentation <Documentation_>`_. Arguably, the existing refcount-related API should be modified to reflect what we want users to expect. Something like the following: * ``Py_INCREF()`` -> ``Py_ACQUIRE_REF()`` (or only support ``Py_NewRef()``) * ``Py_DECREF()`` -> ``Py_RELEASE_REF()`` * ``Py_REFCNT()`` -> ``Py_HAS_REFS()`` * ``Py_SET_REFCNT()`` -> ``Py_RESET_REFS()`` and ``Py_SET_NO_REFS()`` However, such a change is not a part of this proposal. It is included here to demonstrate the tighter focus for user expectations that would benefit this change. Constraints ----------- * ensure that otherwise immutable objects can be truly immutable * minimize performance penalty for normal Python use cases * be careful when immortalizing objects that we don't actually expect to persist until runtime finalization. * be careful when immortalizing objects that are not otherwise immutable * ``__del__`` and weakrefs must continue working properly Regarding "truly" immutable objects, this PEP doesn't impact the effective immutability of any objects, other than the per-object runtime state (e.g. refcount). So whether or not some immortal object is truly (or even effectively) immutable can only be settled separately from this proposal. For example, str objects are generally considered immutable, but ``PyUnicodeObject`` holds some lazily cached data. This PEP has no influence on how that state affects str immutability. Immortal Mutable Objects ------------------------ Any object can be marked as immortal. We do not propose any restrictions or checks. However, in practice the value of making an object immortal relates to its mutability and depends on the likelihood it would be used for a sufficient portion of the application's lifetime. Marking a mutable object as immortal can make sense in some situations. Many of the use cases for immortal objects center on immutability, so that threads can safely and efficiently share such objects without locking. For this reason a mutable object, like a dict or list, would never be shared (and thus no immortality). However, immortality may be appropriate if there is sufficient guarantee that the normally mutable object won't actually be modified. On the other hand, some mutable objects will never be shared between threads (at least not without a lock like the GIL). In some cases it may be practical to make some of those immortal too. For example, ``sys.modules`` is a per-interpreter dict that we do not expect to ever get freed until the corresponding interpreter is finalized. By making it immortal, we no longer incur the extra overhead during incref/decref. We explore this idea further in the `mitigation`_ section below. Implicitly Immortal Objects --------------------------- If an immortal object holds a reference to a normal (mortal) object then that held object is effectively immortal. This is because that object's refcount can never reach 0 until the immortal object releases it. Examples: * containers like ``dict`` and ``list`` * objects that hold references internally like ``PyTypeObject.tp_subclasses`` * an object's type (held in ``ob_type``) Such held objects are thus implicitly immortal for as long as they are held. In practice, this should have no real consequences since it really isn't a change in behavior. The only difference is that the immortal object (holding the reference) doesn't ever get cleaned up. We do not propose that such implicitly immortal objects be changed in any way. They should not be explicitly marked as immortal just because they are held by an immortal object. That would provide no advantage over doing nothing. Un-Immortalizing Objects ------------------------ This proposal does not include any mechanism for taking an immortal object and returning it to a "normal" condition. Currently there is no need for such an ability. On top of that, the obvious approach is to simply set the refcount to a small value. However, at that point there is no way in knowing which value would be safe. Ideally we'd set it to the value that it would have been if it hadn't been made immortal. However, that value has long been lost. Hence the complexities involved make it less likely that an object could safely be un-immortalized, even if we had a good reason to do so. _Py_IMMORTAL_REFCNT ------------------- We will add two internal constants:: _Py_IMMORTAL_BIT - has the top-most available bit set (e.g. 2^62) _Py_IMMORTAL_REFCNT - has the two top-most available bits set The actual top-most bit depends on existing uses for refcount bits, e.g. the sign bit or some GC uses. We will use the highest bit possible after consideration of existing uses. The refcount for immortal objects will be set to ``_Py_IMMORTAL_REFCNT`` (meaning the value will be halfway between ``_Py_IMMORTAL_BIT`` and the value at the next highest bit). However, to check if an object is immortal we will compare (bitwise-and) its refcount against just ``_Py_IMMORTAL_BIT``. The difference means that an immortal object will still be considered immortal, even if somehow its refcount were modified (e.g. by an older stable ABI extension). Note that top two bits of the refcount are already reserved for other uses. That's why we are using the third top-most bit. Affected API ------------ API that will now ignore immortal objects: * (public) ``Py_INCREF()`` * (public) ``Py_DECREF()`` * (public) ``Py_SET_REFCNT()`` * (private) ``_Py_NewReference()`` API that exposes refcounts (unchanged but may now return large values): * (public) ``Py_REFCNT()`` * (public) ``sys.getrefcount()`` (Note that ``_Py_RefTotal`` and ``sys.gettotalrefcount()`` will not be affected.) Also, immortal objects will not participate in GC. Immortal Global Objects ----------------------- All runtime-global (builtin) objects will be made immortal. That includes the following: * singletons (``None``, ``True``, ``False``, ``Ellipsis``, ``NotImplemented``) * all static types (e.g. ``PyLong_Type``, ``PyExc_Exception``) * all static objects in ``_PyRuntimeState.global_objects`` (e.g. identifiers, small ints) The question of making them actually immutable (e.g. for per-interpreter GIL) is not in the scope of this PEP. Object Cleanup -------------- In order to clean up all immortal objects during runtime finalization, we must keep track of them. For GC objects ("containers") we'll leverage the GC's permanent generation by pushing all immortalized containers there. During runtime shutdown, the strategy will be to first let the runtime try to do its best effort of deallocating these instances normally. Most of the module deallocation will now be handled by ``pylifecycle.c:finalize_modules()`` which cleans up the remaining modules as best as we can. It will change which modules are available during ``__del__``, but that's already explicitly undefined behavior in the docs. Optionally, we could do some topological ordering to guarantee that user modules will be deallocated first before the stdlib modules. Finally, anything left over (if any) can be found through the permanent generation GC list which we can clear after ``finalize_modules()``. For non-container objects, the tracking approach will vary on a case-by-case basis. In nearly every case, each such object is directly accessible on the runtime state, e.g. in a ``_PyRuntimeState`` or ``PyInterpreterState`` field. We may need to add a tracking mechanism to the runtime state for a small number of objects. None of the cleanup will have a significant effect on performance. .. _mitigation: Performance Regression Mitigation --------------------------------- In the interest of clarity, here are some of the ways we are going to try to recover some of the lost `performance <Performance_>`_: * at the end of runtime init, mark all objects as immortal * drop refcount operations in code where we know the object is immortal (e.g. ``Py_RETURN_NONE``) * specialize for immortal objects in the eval loop (see `Pyston`_) Regarding that first point, we can apply the concept from `Immortal Mutable Objects`_ in the pursuit of getting back some of that 4% performance we lose with the naive implementation of immortal objects. At the end of runtime init we can mark *all* objects as immortal and avoid the extra cost in incref/decref. We only need to worry about immutability with objects that we plan on sharing between threads without a GIL. Note that none of this section is part of the proposal. The above is included here for clarity. Possible Changes ---------------- * mark every interned string as immortal * mark the "interned" dict as immortal if shared else share all interned strings * (Larry,MvL) mark all constants unmarshalled for a module as immortal * (Larry,MvL) allocate (immutable) immortal objects in their own memory page(s) Documentation ------------- The immortal objects behavior and API are internal, implementation details and will not be added to the documentation. However, we will update the documentation to make public guarantees about refcount behavior more clear. That includes, specifically: * ``Py_INCREF()`` - change "Increment the reference count for object o." to "Indicate taking a new reference to object o." * ``Py_DECREF()`` - change "Decrement the reference count for object o." to "Indicate no longer using a previously taken reference to object o." * similar for ``Py_XINCREF()``, ``Py_XDECREF()``, ``Py_NewRef()``, ``Py_XNewRef()``, ``Py_Clear()`` * ``Py_REFCNT()`` - add "The refcounts 0 and 1 have specific meanings and all others only mean code somewhere is using the object, regardless of the value. 0 means the object is not used and will be cleaned up. 1 means code holds exactly a single reference." * ``Py_SET_REFCNT()`` - refer to ``Py_REFCNT()`` about how values over 1 may be substituted with some over value We *may* also add a note about immortal objects to the following, to help reduce any surprise users may have with the change: * ``Py_SET_REFCNT()`` (a no-op for immortal objects) * ``Py_REFCNT()`` (value may be surprisingly large) * ``sys.getrefcount()`` (value may be surprisingly large) Other API that might benefit from such notes are currently undocumented. We wouldn't add such a note anywhere else (including for ``Py_INCREF()`` and ``Py_DECREF()``) since the feature is otherwise transparent to users. Reference Implementation ======================== The implementation is proposed on GitHub: https://github.com/python/cpython/pull/19474 Open Issues =========== * how realistic is the `Accidental De-Immortalizing`_ concern? References ========== .. _Pyston: https://mail.python.org/archives/list/python-dev@python.org/message/JLHRTBJ… Prior Art --------- * `Pyston`_ Discussions ----------- This was discussed in December 2021 on python-dev: * https://mail.python.org/archives/list/python-dev@python.org/thread/7O3FUA52… * https://mail.python.org/archives/list/python-dev@python.org/thread/PNLBJBNI… Runtime Object State -------------------- Here is the internal state that the CPython runtime keeps for each Python object: * `PyObject.ob_refcnt`_: the object's `refcount <refcounting_>`_ * `_PyGC_Head <PyGC_Head>`_: (optional) the object's node in a list of `"GC" objects <refcounting_>`_ * `_PyObject_HEAD_EXTRA <PyObject_HEAD_EXTRA>`_: (optional) the object's node in the list of heap objects ``ob_refcnt`` is part of the memory allocated for every object. However, ``_PyObject_HEAD_EXTRA`` is allocated only if CPython was built with ``Py_TRACE_REFS`` defined. ``PyGC_Head`` is allocated only if the object's type has ``Py_TPFLAGS_HAVE_GC`` set. Typically this is only container types (e.g. ``list``). Also note that ``PyObject.ob_refcnt`` and ``_PyObject_HEAD_EXTRA`` are part of ``PyObject_HEAD``. .. _PyObject.ob_refcnt: https://github.com/python/cpython/blob/80a9ba537f1f1666a9e6c5eceef4683f8696… .. _PyGC_Head: https://github.com/python/cpython/blob/80a9ba537f1f1666a9e6c5eceef4683f8696… .. _PyObject_HEAD_EXTRA: https://github.com/python/cpython/blob/80a9ba537f1f1666a9e6c5eceef4683f8696… .. _refcounting: Reference Counting, with Cyclic Garbage Collection -------------------------------------------------- Garbage collection is a memory management feature of some programming languages. It means objects are cleaned up (e.g. memory freed) once they are no longer used. Refcounting is one approach to garbage collection. The language runtime tracks how many references are held to an object. When code takes ownership of a reference to an object or releases it, the runtime is notified and it increments or decrements the refcount accordingly. When the refcount reaches 0, the runtime cleans up the object. With CPython, code must explicitly take or release references using the C-API's ``Py_INCREF()`` and ``Py_DECREF()``. These macros happen to directly modify the object's refcount (unfortunately, since that causes ABI compatibility issues if we want to change our garbage collection scheme). Also, when an object is cleaned up in CPython, it also releases any references (and resources) it owns (before it's memory is freed). Sometimes objects may be involved in reference cycles, e.g. where object A holds a reference to object B and object B holds a reference to object A. Consequently, neither object would ever be cleaned up even if no other references were held (i.e. a memory leak). The most common objects involved in cycles are containers. CPython has dedicated machinery to deal with reference cycles, which we call the "cyclic garbage collector", or often just "garbage collector" or "GC". Don't let the name confuse you. It only deals with breaking reference cycles. See the docs for a more detailed explanation of refcounting and cyclic garbage collection: * https://docs.python.org/3.11/c-api/intro.html#reference-counts * https://docs.python.org/3.11/c-api/refcounting.html * https://docs.python.org/3.11/c-api/typeobj.html#c.PyObject.ob_refcnt * https://docs.python.org/3.11/c-api/gcsupport.html Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.

3 6

SC accepted PEP 594: Removing dead batteries from the standard library
by Victor Stinner March 14, 2022

March 14, 2022

Hi, Oh, the Steering Council accepted PEP 594 "Removing dead batteries from the standard library"! I just saw the announcement on Discourse. Congratulations Christian and Brett! This PEP, first proposed in 2019, wasn't an easy one. https://peps.python.org/pep-0594/ Gregory P. Smith's message on Discourse: """ On behalf of the Python Steering Council, We are accepting PEP-594 32 Removing dead batteries from the standard library. It removes a non-controversial set of very old unmaintained or obsolete libraries from the Python standard library. We expect this PEP to be a one time event, and for future deprecations to be handled differently. One thing we’d like to see happen while implementing it: Document the status of the modules being deprecated and removed and backport those deprecation updates to older CPython branch documentation (at least back to 3.9). That gets the notices in front of more people who may use the docs for their specific Python version. Particular care should also be taken during the pre-release cycles that remove deprecated modules. If it turns out the removal of a module proves to be a problem in practice despite the clear deprecation, deferring the removal of that module should be considered to avoid disruption. Doing a “mass cleanup” of long obsolete modules is a sign that we as a project have been ignoring rather than maintaining parts of the standard library, or not doing so with the diligence being in the standard library implies they deserve. Resolving ongoing discussions around how we define the stdlib for the long term does not block this PEP. It seems worthwhile for us to conduct regular reviews of the contents of the stdlib every few releases so we can avoid accumulating such a large pile of dead batteries, but this is outside the scope of this particular PEP. – Greg for the PSC """ https://discuss.python.org/t/pep-594-take-2-removing-dead-batteries-from-th… Victor -- Night gathers, and now my watch begins. It shall not end until my death.

1 0

An unambiguous way of initializing an empty dictionary and set
by joao.p.f.batista.97＠gmail.com March 14, 2022

March 14, 2022

Currently: l = [] # new empty list t = () # new empty tuple s = set() # new empty set (no clean and consistent way of initializing regarding the others) <<< d = {} # new empty dictionary Possible solution: s = {} # new empty set d = {:} # new empty dictionary (the ":" is a reference to key-value pairs) Current workaround at least for consistency: l = list() # new empty list t = tuple() # new empty tuple s = set() # new empty set d = dict() # new empty dictionary However, it doesn't feel right to not be able to initialize an empty set as cleanly and consistently as lists, tuples and dictionaries in both forms.

5 6

Re: cpython: simplify and rewrite the zipimport part of 702009f3c0b1 a bit
by Tommie Cadieux March 13, 2022

March 13, 2022

1 0

Supported make implementations
by Lincoln Auster March 12, 2022

March 12, 2022

Hi, When attempting to build and run doctests for cpython on FreeBSD, I got a vague error from the default BSD make. gmake, of course, ran fine. Is GNU make the only supported make implementation? If so, did I miss that in the devguide, or should that be in there somewhere? -- Lincoln Auster They/Them

2 1

Summary of Python tracker Issues
by Python tracker March 11, 2022

March 11, 2022

ACTIVITY SUMMARY (2022-03-04 - 2022-03-11) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7233 ( +9) closed 51481 (+56) total 58714 (+65) Open issues with patches: 2940 Issues opened (43) ================== #45138: [sqlite3] expand bound values in traced statements when possib https://bugs.python.org/issue45138 reopened by erlendaasland #46924: make install hangs when installing zoneinfo/_zoneinfo.py https://bugs.python.org/issue46924 opened by AmericanEnglish #46925: Document dict behavior when setting equal but not identical ke https://bugs.python.org/issue46925 opened by malthe #46926: runpy.run_path didn't set __package__ to None as describe in d https://bugs.python.org/issue46926 opened by yanhao.charles #46930: Iterating over cls.__dict__ in classmethod causes RuntimeError https://bugs.python.org/issue46930 opened by PythonF #46931: zipfile library will raise uncaught oserror when reading lengt https://bugs.python.org/issue46931 opened by ultimalium #46934: Started multiprocessing.Process instances are unserialisable https://bugs.python.org/issue46934 opened by maggyero #46938: dataclass __post_init__ recursion https://bugs.python.org/issue46938 opened by bar.harel #46939: Specialize calls for Python classes https://bugs.python.org/issue46939 opened by kj #46942: Convert Object/classobject.c to Argument Clinic https://bugs.python.org/issue46942 opened by arhadthedev #46943: fix[imaplib]: call Exception with string instance https://bugs.python.org/issue46943 opened by spaceone #46944: Use FASTCALL calling convention in generator.throw https://bugs.python.org/issue46944 opened by kumaraditya303 #46945: Quantifier and Expanded Regex Expression Gives Different Resul https://bugs.python.org/issue46945 opened by vmd3.14 #46946: Port core types to Argument Clinic https://bugs.python.org/issue46946 opened by arhadthedev #46949: Print an indication if traceback exceeds sys.tracebacklimit https://bugs.python.org/issue46949 opened by JelleZijlstra #46950: Windows 11 venv https://bugs.python.org/issue46950 opened by darrel.opry #46951: Zipapp contents are in filesystem-dependent order https://bugs.python.org/issue46951 opened by h.finucane #46953: use FASTCALL for __import__ builtin https://bugs.python.org/issue46953 opened by kumaraditya303 #46956: TextIOWrapper.seek silently wraps around on very large offsets https://bugs.python.org/issue46956 opened by vlaci #46957: Logger with a custom class breaks on copy https://bugs.python.org/issue46957 opened by govinda18 #46958: json dump/dumps prints each array element on a new line (bad f https://bugs.python.org/issue46958 opened by Entirity #46959: ctypes.util.find_library can delete /dev/null https://bugs.python.org/issue46959 opened by barry.c.davis #46960: Docs: Link from settrace to frame https://bugs.python.org/issue46960 opened by guettli #46961: Caching/interning of small ints sometimes fails https://bugs.python.org/issue46961 opened by steven.daprano #46962: Fix docstrings that do not honor --without-doc-strings https://bugs.python.org/issue46962 opened by arhadthedev #46963: Deep Lazy Imports - Interpreter-level deferred module loading https://bugs.python.org/issue46963 opened by Kronuz #46964: The global config should not be stored on each interpreter https://bugs.python.org/issue46964 opened by eric.snow #46965: Enable informing callee it's awaited via vector call flag https://bugs.python.org/issue46965 opened by dino.viehland #46966: c_void_p array is a footgun on I32LP64 systems https://bugs.python.org/issue46966 opened by taralx #46967: Type union for except https://bugs.python.org/issue46967 opened by Henry Schreiner #46968: Insufficient sigaltstack size used by CPython prevents extensi https://bugs.python.org/issue46968 opened by oleksandr-pavlyk #46970: dataclass(slots=True) incompatible with __init_subclass__ https://bugs.python.org/issue46970 opened by crusaderky #46973: Provide make target to regenerate autoconf files with containe https://bugs.python.org/issue46973 opened by christian.heimes #46975: clang: error: linker command failed with exit code 1 (use -v t https://bugs.python.org/issue46975 opened by Battant #46976: Update macOS installer builds to use ncurses 6.3 https://bugs.python.org/issue46976 opened by ned.deily #46977: tempfile.TemporaryDirectory fails removing dir in some edge ca https://bugs.python.org/issue46977 opened by afeblot #46978: Doc strings for built-in, in-place operators are misleading https://bugs.python.org/issue46978 opened by nickovs #46981: Empty typing.Tuple https://bugs.python.org/issue46981 opened by serhiy.storchaka #46983: test_sqlite3: test_trace_too_much_expanded_sql() failed on AMD https://bugs.python.org/issue46983 opened by vstinner #46984: test_concurrent_futures logs many UndefinedBehaviorSanitizer " https://bugs.python.org/issue46984 opened by vstinner #46985: Upgrade ensurepip bundled pip to 22.0.4 https://bugs.python.org/issue46985 opened by pradyunsg #46986: Upgrade ensurepip bundled setuptools to 60.9.3 https://bugs.python.org/issue46986 opened by pradyunsg #46987: Remove _PySys_GetObjectId / _PySys_GetObjectId https://bugs.python.org/issue46987 opened by corona10 Most recent 15 issues with no replies (15) ========================================== #46986: Upgrade ensurepip bundled setuptools to 60.9.3 https://bugs.python.org/issue46986 #46985: Upgrade ensurepip bundled pip to 22.0.4 https://bugs.python.org/issue46985 #46984: test_concurrent_futures logs many UndefinedBehaviorSanitizer " https://bugs.python.org/issue46984 #46983: test_sqlite3: test_trace_too_much_expanded_sql() failed on AMD https://bugs.python.org/issue46983 #46981: Empty typing.Tuple https://bugs.python.org/issue46981 #46978: Doc strings for built-in, in-place operators are misleading https://bugs.python.org/issue46978 #46977: tempfile.TemporaryDirectory fails removing dir in some edge ca https://bugs.python.org/issue46977 #46968: Insufficient sigaltstack size used by CPython prevents extensi https://bugs.python.org/issue46968 #46965: Enable informing callee it's awaited via vector call flag https://bugs.python.org/issue46965 #46963: Deep Lazy Imports - Interpreter-level deferred module loading https://bugs.python.org/issue46963 #46962: Fix docstrings that do not honor --without-doc-strings https://bugs.python.org/issue46962 #46960: Docs: Link from settrace to frame https://bugs.python.org/issue46960 #46959: ctypes.util.find_library can delete /dev/null https://bugs.python.org/issue46959 #46958: json dump/dumps prints each array element on a new line (bad f https://bugs.python.org/issue46958 #46956: TextIOWrapper.seek silently wraps around on very large offsets https://bugs.python.org/issue46956 Most recent 15 issues waiting for review (15) ============================================= #46986: Upgrade ensurepip bundled setuptools to 60.9.3 https://bugs.python.org/issue46986 #46985: Upgrade ensurepip bundled pip to 22.0.4 https://bugs.python.org/issue46985 #46978: Doc strings for built-in, in-place operators are misleading https://bugs.python.org/issue46978 #46973: Provide make target to regenerate autoconf files with containe https://bugs.python.org/issue46973 #46968: Insufficient sigaltstack size used by CPython prevents extensi https://bugs.python.org/issue46968 #46964: The global config should not be stored on each interpreter https://bugs.python.org/issue46964 #46962: Fix docstrings that do not honor --without-doc-strings https://bugs.python.org/issue46962 #46958: json dump/dumps prints each array element on a new line (bad f https://bugs.python.org/issue46958 #46946: Port core types to Argument Clinic https://bugs.python.org/issue46946 #46943: fix[imaplib]: call Exception with string instance https://bugs.python.org/issue46943 #46942: Convert Object/classobject.c to Argument Clinic https://bugs.python.org/issue46942 #46939: Specialize calls for Python classes https://bugs.python.org/issue46939 #46934: Started multiprocessing.Process instances are unserialisable https://bugs.python.org/issue46934 #46920: Remove code made dead long ago with #if 0 https://bugs.python.org/issue46920 #46917: Require IEEE 754 floating point to build Python 3.11 https://bugs.python.org/issue46917 Top 10 most discussed issues (10) ================================= #43923: Can't create generic NamedTuple as of py3.9 https://bugs.python.org/issue43923 21 msgs #46896: add support for watching writes to selected dictionaries https://bugs.python.org/issue46896 10 msgs #46925: Document dict behavior when setting equal but not identical ke https://bugs.python.org/issue46925 8 msgs #46890: venv does not create "python" link in python 3.11 https://bugs.python.org/issue46890 7 msgs #46917: Require IEEE 754 floating point to build Python 3.11 https://bugs.python.org/issue46917 6 msgs #46943: fix[imaplib]: call Exception with string instance https://bugs.python.org/issue46943 6 msgs #46953: use FASTCALL for __import__ builtin https://bugs.python.org/issue46953 6 msgs #46987: Remove _PySys_GetObjectId / _PySys_GetObjectId https://bugs.python.org/issue46987 6 msgs #39298: add BLAKE3 to hashlib https://bugs.python.org/issue39298 5 msgs #45138: [sqlite3] expand bound values in traced statements when possib https://bugs.python.org/issue45138 5 msgs Issues closed (52) ================== #24959: unittest swallows part of stack trace when raising AssertionEr https://bugs.python.org/issue24959 closed by iritkatriel #25415: [io doc] Reword "there is no public constructor" https://bugs.python.org/issue25415 closed by terry.reedy #29251: [doc] Class __dict__ is only a mapping proxy https://bugs.python.org/issue29251 closed by iritkatriel #30740: SSLError when cancelling an SSL connection https://bugs.python.org/issue30740 closed by asvetlov #33886: SSL on aiomysql hangs on reconnection https://bugs.python.org/issue33886 closed by asvetlov #34810: Maximum and minimum value of C types integers from Python https://bugs.python.org/issue34810 closed by iritkatriel #35677: Do not automount in stat() by default https://bugs.python.org/issue35677 closed by serhiy.storchaka #36098: asyncio: ssl client-server with "slow" read https://bugs.python.org/issue36098 closed by asvetlov #36709: Asyncio SSL keep-alive connections raise errors after loop clo https://bugs.python.org/issue36709 closed by asvetlov #37179: asyncio loop.start_tls() provide support for TLS in TLS https://bugs.python.org/issue37179 closed by asvetlov #43253: asyncio open_connection fails when a socket is explicitly clos https://bugs.python.org/issue43253 closed by asvetlov #43292: xml.ElementTree iterparse filehandle left open https://bugs.python.org/issue43292 closed by serhiy.storchaka #44549: Update Windows installer to use bzip2 1.0.8 https://bugs.python.org/issue44549 closed by steve.dower #45033: Calls to PyErr_PrintEx in destructors cause calling async func https://bugs.python.org/issue45033 closed by gregory.p.smith #45319: Possible regression in __annotations__ descr for heap type sub https://bugs.python.org/issue45319 closed by larry #45828: [sqlite3] use unraisable exceptions in callbacks https://bugs.python.org/issue45828 closed by erlendaasland #46170: Improving the error message when subclassing NewType https://bugs.python.org/issue46170 closed by JelleZijlstra #46198: Duplicate and unused code in tests https://bugs.python.org/issue46198 closed by AlexWaygood #46245: Add support for dir_fd in shutil.rmtree() https://bugs.python.org/issue46245 closed by serhiy.storchaka #46341: duplicate paragraphs - asyncio Coroutines and Tasks file https://bugs.python.org/issue46341 closed by serhiy.storchaka #46414: Add typing.reveal_type https://bugs.python.org/issue46414 closed by JelleZijlstra #46494: Mention typing_extensions in the typing documentation https://bugs.python.org/issue46494 closed by JelleZijlstra #46581: _typevar_types and _paramspec_tvars are missing from _GenericA https://bugs.python.org/issue46581 closed by serhiy.storchaka #46860: `--with-suffix` not respected on case-insensitive file systems https://bugs.python.org/issue46860 closed by brett.cannon #46864: Deprecate ob_shash in BytesObject https://bugs.python.org/issue46864 closed by methane #46881: Statically allocate and initialize the latin1 characters. https://bugs.python.org/issue46881 closed by kumaraditya303 #46903: Crash when setting attribute with string subclass as the name https://bugs.python.org/issue46903 closed by ronaldoussoren #46912: Full gc collection blocked from collecting after some amount o https://bugs.python.org/issue46912 closed by al3x3i.k #46921: Vectorcall support for super() https://bugs.python.org/issue46921 closed by kj #46927: Improve error message for subscripting non-generic types https://bugs.python.org/issue46927 closed by serhiy.storchaka #46928: type.__qualname__ ignores module name in tp_name for static ty https://bugs.python.org/issue46928 closed by ronaldoussoren #46929: __rrshift__ for same class obj will raise TypeError https://bugs.python.org/issue46929 closed by mark.dickinson #46932: Please update bundled libexpat to 2.4.7 with an important fix https://bugs.python.org/issue46932 closed by steve.dower #46933: Make pwd module optional for wasm32-emscripten and wasi https://bugs.python.org/issue46933 closed by christian.heimes #46935: import of submodule polutes global namespace https://bugs.python.org/issue46935 closed by maxbachmann #46936: Fix grammar_grapher with the new forced directive https://bugs.python.org/issue46936 closed by pablogsal #46937: convert remaining functions to AC in _weakref https://bugs.python.org/issue46937 closed by corona10 #46940: Suggestion messages don't properly work in nested getattr call https://bugs.python.org/issue46940 closed by pablogsal #46941: Bug or plug not removed (The operator "is") https://bugs.python.org/issue46941 closed by eric.smith #46947: unicodedata.name gives ValueError for control characters https://bugs.python.org/issue46947 closed by serhiy.storchaka #46948: [CVE-2022-26488] Escalation of privilege via Windows Installer https://bugs.python.org/issue46948 closed by steve.dower #46952: pip progress bar display bug https://bugs.python.org/issue46952 closed by Dennis Sweeney #46954: Awaiting multiple times on same task increases memory usage un https://bugs.python.org/issue46954 closed by iritkatriel #46955: The documentation mentions asyncio.Server. It does not exist. https://bugs.python.org/issue46955 closed by asvetlov #46969: `pip install wrapt` fails on ast.py in Python 3.11.0a6 https://bugs.python.org/issue46969 closed by Dutcho #46971: python takes long time when return big data https://bugs.python.org/issue46971 closed by mark.dickinson #46972: Documentation: Reference says AssertionError is raised by `ass https://bugs.python.org/issue46972 closed by serhiy.storchaka #46974: set function for lists on numbers, sometimes sorts elements fr https://bugs.python.org/issue46974 closed by eric.smith #46979: Spam https://bugs.python.org/issue46979 closed by Dennis Sweeney #46980: ast.FunctionDef cannot find functions under if statement https://bugs.python.org/issue46980 closed by JelleZijlstra #46982: Error in the experts list https://bugs.python.org/issue46982 closed by JelleZijlstra #46988: if a python program is execute by subprocess, the python progr https://bugs.python.org/issue46988 closed by markhuang3310

1 0

RFC on PEP 655: Required[] and NotRequired[] for TypedDict
by David Foster March 10, 2022

March 10, 2022

This PEP [1] introduces syntax to mark individual keys of a TypedDict as either required or potentially-missing. Previously the only way to have a TypedDict with mixed required and non-required keys was to define two TypedDicts - one with total=True and one with total=False - and have one of those TypedDicts inherit from the other. This PEP introduces special forms Required[] and NotRequired[] to the "typing" module that can be used to directly mark individual items of a TypedDict as required or not-required. We have implementations for mypy, pyright, and pyanalyze. Consensus on the PEP has been reached in typing-sig. My understanding of the PEP process as described in PEP 1 [2] is that content review should be requested of core developers here in python-dev. So consider this my official request for comments! ^_^ I believe that after review & feedback from python-dev the next step would be to submit this PEP to the Steering Council. However it's not clear to me from [2] where I should actually do that when the time comes. -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy [1]: https://www.python.org/dev/peps/pep-0655/ [2]: https://www.python.org/dev/peps/pep-0001/#pep-review-resolution

4 6

PSA: Linux vulnerability
by Ethan Furman March 8, 2022

March 8, 2022

https://arstechnica.com/information-technology/2022/03/linux-has-been-bitte… -- ~Ethan~

1 0

[RELEASE] Python 3.11.0a6 is available
by Pablo Galindo Salgado March 7, 2022

March 7, 2022

There are no easy releases these days! :sweat: After a week of delay due to several release blockers, buildbot problems and pandemic-related difficulties here is 3.11.0a6 for you to test. https://www.python.org/downloads/release/python-3110a6/ **This is an early developer preview of Python 3.11** # Major new features of the 3.11 series, compared to 3.10 Python 3.11 is still in development. This release, 3.11.0a6 is the sixth of seven planned alpha releases. Alpha releases are intended to make it easier to test the current state of new features and bug fixes and to test the release process. During the alpha phase, features may be added up until the start of the beta phase (2022-05-06) and, if necessary, may be modified or deleted up until the release candidate phase (2022-08-01). Please keep in mind that this is a preview release and its use is **not** recommended for production environments. Many new features for Python 3.11 are still being planned and written. Among the new major new features and changes so far: * [PEP 657](https://www.python.org/dev/peps/pep-0657/) -- Include Fine-Grained Error Locations in Tracebacks * [PEP 654](https://www.python.org/dev/peps/pep-0654/) -- Exception Groups and except* * [PEP 673](https://www.python.org/dev/peps/pep-0673/) -- Self Type * [PEP 646](https://www.python.org/dev/peps/pep-0646/)-- Variadic Generics * The [Faster Cpython Project](https://github.com/faster-cpython) is already yielding some exciting results: this version of CPython 3.11 is ~12% faster on the geometric mean of the [PyPerformance benchmarks]( speed.python.org), compared to 3.10.0. * Hey, **fellow core developer,** if a feature you find important is missing from this list, let me know. The next pre-release of Python 3.11 will be 3.11.0a7, currently scheduled for Tuesday, 2022-04-05. # More resources * [Online Documentation](https://docs.python.org/3.11/) * [PEP 664](https://www.python.org/dev/peps/pep-0664/), 3.11 Release Schedule * Report bugs at [https://bugs.python.org](https://bugs.python.org). * [Help fund Python and its community](/psf/donations/). # And now for something completely different In astrophysics and nuclear physics, nuclear pasta is a theoretical type of degenerate matter that is postulated to exist within the crusts of neutron stars. If it does in fact exist, nuclear pasta is the strongest material in the universe. Between the surface of a neutron star and the quark-gluon plasma at the core, at matter densities of 1014 g/cm3, nuclear attraction and Coulomb repulsion forces are of similar magnitude. The competition between the forces leads to the formation of a variety of complex structures assembled from neutrons and protons. Astrophysicists call these types of structures nuclear pasta because the geometry of the structures resembles various types of pasta. There are several phases of evolution (I swear these names are real), including the gnocchi phase, the spaghetti phase, the lasagna phase, the bucatini phase and the Swiss cheese phase. # We hope you enjoy those new releases! Thanks to all of the many volunteers who help make Python Development and these releases possible! Please consider supporting our efforts by volunteering yourself or through organization contributions to the Python Software Foundation. Your friendly release team, Pablo Galindo @pablogsal Ned Deily @nad Steve Dower @steve.dower

1 0

Improvements to the sys.path initialization documentation
by Russel Webber March 7, 2022

March 7, 2022

Hi All, Steve Dower, Eryk Sun and I have been working on improving the documentation around how sys.path is initialised and therefore how Python modules are found. We're moving the details from https://docs.python.org/3.11/using/windows.html#finding-modules to a platform-agnostic new section in https://docs.python.org/3.11/using/cmdline.html We're aiming to have one place in the documentation where all the ways to customize sys.path (PYTHONPATH, ._pth, site, etc) are at least mentioned. We'd appreciate further community feedback on the changes before they are merged. https://bugs.python.org/issue31582 https://github.com/python/cpython/pull/31082 Thank you, Russel

4 6

Accepted: PEP 676 - PEP Infrastructure Process
by Barry Warsaw March 6, 2022

March 6, 2022

As PEP-Delegate, I’m very happy to accept PEP 676 (PEP Infrastructure Process). https://www.python.org/dev/peps/pep-0676/ https://discuss.python.org/t/request-for-comment-making-pep-rendering-self-… Congratulations to Adam Turner and everyone who participated in the discussions. This will make reading and authoring PEPs a much nicer experience. Cheers, -Barry

1 0

PEP 682: Format Specifier for Signed Zero
by John Belmonte March 6, 2022

March 6, 2022

Just a note that this PEP is being discussed at https://discuss.python.org/t/pep-682-format-specifier-for-signed-zero TLDR; When formatting a number (especially with rounding options), often you don't want negative-signed zero as an output. A format spec extension enables normalizing such values to positive zero. Regards, --John Belmonte

2 1

Make __mro_entries__ mandatory for non-types
by Serhiy Storchaka March 5, 2022

March 5, 2022

Currently the class can inherit from arbitrary objects, not only types. If the object was not designed for this you can get a mysterious mistake, e g.: >>> class A(1): ... ... Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: int() takes at most 2 arguments (3 given) If the object's constructor is compatible with type constructor by accident you can an unexpected result. To prevent this we usually add an explicit __mro_entries__() method which raises an exception. I propose to make __mro_entries__() mandatory if a base is not a type instance (with a deprecation period).

4 6

Summary of Python tracker Issues
by Python tracker March 4, 2022

March 4, 2022

ACTIVITY SUMMARY (2022-02-25 - 2022-03-04) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7224 ( +4) closed 51425 (+62) total 58649 (+66) Open issues with patches: 2938 Issues opened (48) ================== #44807: typing.Protocol silently overrides __init__ method of delivere https://bugs.python.org/issue44807 reopened by gvanrossum #46623: test_zlib: test_pair() and test_speech128() fail with s390x ha https://bugs.python.org/issue46623 reopened by petr.viktorin #46823: Add LOAD_FAST__LOAD_ATTR_INSTANCE_VALUE combined opcode https://bugs.python.org/issue46823 reopened by brandtbucher #46858: mmap constructor resets the file pointer on Windows https://bugs.python.org/issue46858 opened by benrg #46860: `--with-suffix` not respected on case-insensitive file systems https://bugs.python.org/issue46860 opened by brett.cannon #46862: subprocess makes environment blocks with duplicate keys on Win https://bugs.python.org/issue46862 opened by benrg #46863: Python 3.10 OpenSSL Configuration Issues https://bugs.python.org/issue46863 opened by adam #46864: Deprecate ob_shash in BytesObject https://bugs.python.org/issue46864 opened by methane #46868: Improve performance of math.prod with bignums (and functools.r https://bugs.python.org/issue46868 opened by benrg #46870: Improper Input Validation in urlparse https://bugs.python.org/issue46870 opened by P0cas #46872: Odd handling of signal raised if an illegal syscall is attempt https://bugs.python.org/issue46872 opened by bup #46878: [sqlite3] remove "non-standard" from docstrings https://bugs.python.org/issue46878 opened by erlendaasland #46879: [doc] incorrect sphinx object names https://bugs.python.org/issue46879 opened by push-f #46880: zipfile library doesn't extract windows zip files properly on https://bugs.python.org/issue46880 opened by nimrodf #46881: Statically allocate and initialize the latin1 characters. https://bugs.python.org/issue46881 opened by kumaraditya303 #46882: Clarify argument type of platform.platform(aliased, terse) to https://bugs.python.org/issue46882 opened by Rotzbua #46883: Add glossary entries to clarify the true/True and false/False https://bugs.python.org/issue46883 opened by steven.daprano #46884: [doc] msilib.rst uses data directive to document modules https://bugs.python.org/issue46884 opened by push-f #46885: Ensure PEP 663 changes are reverted from 3.11 https://bugs.python.org/issue46885 opened by pablogsal #46887: ./Programs/_freeze_module fails with MSAN: Uninitialized value https://bugs.python.org/issue46887 opened by vstinner #46888: SharedMemory.close() destroys memory https://bugs.python.org/issue46888 opened by ronny-rentner #46890: venv does not create "python" link in python 3.11 https://bugs.python.org/issue46890 opened by ronaldoussoren #46892: Async Call-Stack Reconstruction https://bugs.python.org/issue46892 opened by mpage #46893: Allow extensions to set the vectorcall field on instances of P https://bugs.python.org/issue46893 opened by itamaro #46895: Allow extensions to set a callback to be invoked when a type i https://bugs.python.org/issue46895 opened by mpage #46896: add support for watching writes to selected dictionaries https://bugs.python.org/issue46896 opened by carljm #46897: Add API to allow extensions to set callback function on creati https://bugs.python.org/issue46897 opened by mpage #46898: Add API to allow extensions to set callback function on creati https://bugs.python.org/issue46898 opened by mpage #46899: use of uninitialized value with msan from subprocess_fork_exec https://bugs.python.org/issue46899 opened by yilei #46901: Deprecate and eventually remove macOS-only PYTHONEXECUTABLE en https://bugs.python.org/issue46901 opened by ned.deily #46902: Typo hint message for from-imports? https://bugs.python.org/issue46902 opened by Jean_Abou_Samra #46903: Crash when setting attribute with string subclass as the name https://bugs.python.org/issue46903 opened by ronaldoussoren #46904: Python Decimal supports '#' format, C Decimal does not. https://bugs.python.org/issue46904 opened by const #46905: winsound.PlaySound should accept pathlib.Path instances https://bugs.python.org/issue46905 opened by Julian-O #46906: Add PyFloat_(Pack|Unpack)(2|4|8) to the public C API https://bugs.python.org/issue46906 opened by methane #46907: Update Windows and MacOS installer to SQLite 3.38.0. https://bugs.python.org/issue46907 opened by felixxm #46908: Debugger jumps to a wrong instruction in for loop https://bugs.python.org/issue46908 opened by franarenasafan #46911: Early tracing has lineno=None for modules https://bugs.python.org/issue46911 opened by nedbat #46912: Full gc collection blocked from collecting after some amount o https://bugs.python.org/issue46912 opened by al3x3i.k #46913: UBSAN: test_ctypes, test_faulthandler and test_hashlib are fai https://bugs.python.org/issue46913 opened by vstinner #46914: On Osx Monterey(12.x), Logging.handlers.SysLogHandler does not https://bugs.python.org/issue46914 opened by pbloom #46915: Build with Py_LIMITED_API fails unknown type name ‘PyModuleD https://bugs.python.org/issue46915 opened by christian.heimes #46917: Require IEEE 754 floating point to build Python 3.11 https://bugs.python.org/issue46917 opened by vstinner #46918: The vulnerability is included in /lib/python3.9/ensurepip afte https://bugs.python.org/issue46918 opened by zjmxq #46920: Remove code made dead long ago with #if 0 https://bugs.python.org/issue46920 opened by arhadthedev #46921: Vectorcall support for super() https://bugs.python.org/issue46921 opened by kj #46922: tarfile.TarFile.next() crashes on empty tar files https://bugs.python.org/issue46922 opened by progval #46923: Implement stack overflow protection for supported platforms https://bugs.python.org/issue46923 opened by Mark.Shannon Most recent 15 issues with no replies (15) ========================================== #46923: Implement stack overflow protection for supported platforms https://bugs.python.org/issue46923 #46922: tarfile.TarFile.next() crashes on empty tar files https://bugs.python.org/issue46922 #46921: Vectorcall support for super() https://bugs.python.org/issue46921 #46917: Require IEEE 754 floating point to build Python 3.11 https://bugs.python.org/issue46917 #46912: Full gc collection blocked from collecting after some amount o https://bugs.python.org/issue46912 #46904: Python Decimal supports '#' format, C Decimal does not. https://bugs.python.org/issue46904 #46901: Deprecate and eventually remove macOS-only PYTHONEXECUTABLE en https://bugs.python.org/issue46901 #46899: use of uninitialized value with msan from subprocess_fork_exec https://bugs.python.org/issue46899 #46898: Add API to allow extensions to set callback function on creati https://bugs.python.org/issue46898 #46897: Add API to allow extensions to set callback function on creati https://bugs.python.org/issue46897 #46895: Allow extensions to set a callback to be invoked when a type i https://bugs.python.org/issue46895 #46893: Allow extensions to set the vectorcall field on instances of P https://bugs.python.org/issue46893 #46892: Async Call-Stack Reconstruction https://bugs.python.org/issue46892 #46881: Statically allocate and initialize the latin1 characters. https://bugs.python.org/issue46881 #46879: [doc] incorrect sphinx object names https://bugs.python.org/issue46879 Most recent 15 issues waiting for review (15) ============================================= #46921: Vectorcall support for super() https://bugs.python.org/issue46921 #46920: Remove code made dead long ago with #if 0 https://bugs.python.org/issue46920 #46913: UBSAN: test_ctypes, test_faulthandler and test_hashlib are fai https://bugs.python.org/issue46913 #46907: Update Windows and MacOS installer to SQLite 3.38.0. https://bugs.python.org/issue46907 #46906: Add PyFloat_(Pack|Unpack)(2|4|8) to the public C API https://bugs.python.org/issue46906 #46903: Crash when setting attribute with string subclass as the name https://bugs.python.org/issue46903 #46887: ./Programs/_freeze_module fails with MSAN: Uninitialized value https://bugs.python.org/issue46887 #46884: [doc] msilib.rst uses data directive to document modules https://bugs.python.org/issue46884 #46882: Clarify argument type of platform.platform(aliased, terse) to https://bugs.python.org/issue46882 #46881: Statically allocate and initialize the latin1 characters. https://bugs.python.org/issue46881 #46879: [doc] incorrect sphinx object names https://bugs.python.org/issue46879 #46878: [sqlite3] remove "non-standard" from docstrings https://bugs.python.org/issue46878 #46864: Deprecate ob_shash in BytesObject https://bugs.python.org/issue46864 #46860: `--with-suffix` not respected on case-insensitive file systems https://bugs.python.org/issue46860 #46850: [C API] Move _PyEval_EvalFrameDefault() to the internal C API https://bugs.python.org/issue46850 Top 10 most discussed issues (10) ================================= #46841: Inline bytecode caches https://bugs.python.org/issue46841 16 msgs #46913: UBSAN: test_ctypes, test_faulthandler and test_hashlib are fai https://bugs.python.org/issue46913 15 msgs #46890: venv does not create "python" link in python 3.11 https://bugs.python.org/issue46890 14 msgs #44807: typing.Protocol silently overrides __init__ method of delivere https://bugs.python.org/issue44807 10 msgs #46863: Python 3.10 OpenSSL Configuration Issues https://bugs.python.org/issue46863 10 msgs #46868: Improve performance of math.prod with bignums (and functools.r https://bugs.python.org/issue46868 8 msgs #46896: add support for watching writes to selected dictionaries https://bugs.python.org/issue46896 8 msgs #46836: [C API] Move PyFrameObject to the internal C API https://bugs.python.org/issue46836 7 msgs #46888: SharedMemory.close() destroys memory https://bugs.python.org/issue46888 7 msgs #17505: [doc] email.header.Header.__unicode__ does not decode header https://bugs.python.org/issue17505 6 msgs Issues closed (58) ================== #6634: [doc] sys.exit() called from threads other than the main one: https://bugs.python.org/issue6634 closed by iritkatriel #10515: csv sniffer does not recognize quotes at the end of line https://bugs.python.org/issue10515 closed by iritkatriel #15002: urllib2 does not download 4 MB file completely using ftp https://bugs.python.org/issue15002 closed by iritkatriel #21910: [doc] File protocol should document if writelines must handle https://bugs.python.org/issue21910 closed by Jelle Zijlstra #30698: asyncio sslproto do not shutdown ssl layer cleanly https://bugs.python.org/issue30698 closed by asvetlov #42337: Skip building web installers on Windows https://bugs.python.org/issue42337 closed by steve.dower #45062: test_asyncio: test_huge_content_recvinto() failed on PPC64LE R https://bugs.python.org/issue45062 closed by vstinner #45065: test_asyncio failed (env changed) on s390x RHEL8 Refleaks 3.10 https://bugs.python.org/issue45065 closed by vstinner #45159: data_received called on protocol after call to pause_reading o https://bugs.python.org/issue45159 closed by asvetlov #45431: [C API] Rename CFrame or hide it to only export names starting https://bugs.python.org/issue45431 closed by vstinner #45492: stdlib inspect documentation on code.co_names is incorrect https://bugs.python.org/issue45492 closed by iritkatriel #46195: Annotated and Optional get_type_hints buggy interaction https://bugs.python.org/issue46195 closed by Jelle Zijlstra #46311: Clean up PyLong_FromLong and PyLong_FromLongLong https://bugs.python.org/issue46311 closed by Mark.Shannon #46318: asyncio and ssl: ResourceWarning: unclosed transport https://bugs.python.org/issue46318 closed by asvetlov #46389: 3.11: unused generator comprehensions cause f_lineno==None https://bugs.python.org/issue46389 closed by Mark.Shannon #46567: Add Tcl/Tk builds for ARM64 https://bugs.python.org/issue46567 closed by steve.dower #46633: AddressSanitizer: Skip tests directly in Python, not with exte https://bugs.python.org/issue46633 closed by vstinner #46643: typing.Annotated cannot wrap typing.ParamSpec args/kwargs https://bugs.python.org/issue46643 closed by Jelle Zijlstra #46661: Duplicate deprecation warnings in docs for asyncio https://bugs.python.org/issue46661 closed by gvanrossum #46675: Allow more than 16 items in split-keys dicts and "virtual" obj https://bugs.python.org/issue46675 closed by Mark.Shannon #46712: Share global string identifiers in deepfreeze https://bugs.python.org/issue46712 closed by gvanrossum #46748: Python.h includes stdbool.h https://bugs.python.org/issue46748 closed by erlendaasland #46753: Statically allocate and initialize the empty tuple. https://bugs.python.org/issue46753 closed by eric.snow #46756: Incorrect authorization check in urllib.request https://bugs.python.org/issue46756 closed by pablogsal #46786: embed, source, track, wbr HTML elements not considered empty https://bugs.python.org/issue46786 closed by scoder #46794: Please update bundled libexpat to 2.4.6 with security fixes (5 https://bugs.python.org/issue46794 closed by corona10 #46808: remove NEXT_BLOCK() from compile.c https://bugs.python.org/issue46808 closed by iritkatriel #46816: Remove declarations for non-__STDC__ compilers https://bugs.python.org/issue46816 closed by vstinner #46831: Outdated comment for __build_class__ in compile.c https://bugs.python.org/issue46831 closed by Jelle Zijlstra #46845: dict: Use smaller entry for Unicode-key only dict. https://bugs.python.org/issue46845 closed by methane #46848: Use optimized string search function in mmap.find() https://bugs.python.org/issue46848 closed by Dennis Sweeney #46849: Memory problems detected using Valgrind https://bugs.python.org/issue46849 closed by sxt1001 #46852: Remove the float.__set_format__() method https://bugs.python.org/issue46852 closed by mark.dickinson #46853: Python interpreter can get code from memory, it is not secure. https://bugs.python.org/issue46853 closed by brett.cannon #46854: Failed to compile static python3.7.12 https://bugs.python.org/issue46854 closed by ned.deily #46855: printing a string with strange characters loops forever https://bugs.python.org/issue46855 closed by svild #46857: Python leaks one reference at exit on Windows https://bugs.python.org/issue46857 closed by vstinner #46859: NameError: free variable 'outer' referenced before assignment https://bugs.python.org/issue46859 closed by Jelle Zijlstra #46861: os.environ forces variable names to upper case on Windows https://bugs.python.org/issue46861 closed by eryksun #46865: *() Invalid Syntax: iterable unpacking of empty tuple https://bugs.python.org/issue46865 closed by serhiy.storchaka #46866: bytes class extension with slices https://bugs.python.org/issue46866 closed by serhiy.storchaka #46867: difference of work https://bugs.python.org/issue46867 closed by grifonice99 #46869: platform.release() and sys returns wrong version on Windows 11 https://bugs.python.org/issue46869 closed by eryksun #46871: Lambda can't be pickled with "spawn" and only "fork" https://bugs.python.org/issue46871 closed by ronaldoussoren #46873: inspect.getsource with some lambdas in decorators does not get https://bugs.python.org/issue46873 closed by ahopkins #46874: [sqlite3] optimise user-defined functions https://bugs.python.org/issue46874 closed by erlendaasland #46875: Missing name in TaskGroup.__repr__() https://bugs.python.org/issue46875 closed by achimnol #46876: Walrus operator not in help https://bugs.python.org/issue46876 closed by AlexWaygood #46877: [doc] unittest.doModuleCleanups() does not exist https://bugs.python.org/issue46877 closed by gvanrossum #46886: ARM64 Windows Non-Debug 3.x: random build error: failure durin https://bugs.python.org/issue46886 closed by vstinner #46889: enable sqlite math functions on macos https://bugs.python.org/issue46889 closed by erlendaasland #46891: Crash in ModuleType subclass with __slots__ https://bugs.python.org/issue46891 closed by Mark.Shannon #46894: make install DESTDIR= uses /lib/python3.10/lib-dynload out of https://bugs.python.org/issue46894 closed by ned.deily #46900: marshal.dumps represents the same list object differently https://bugs.python.org/issue46900 closed by Dreeseaw #46909: Update getpath.py to look for os.pyc in __pycache__ folders https://bugs.python.org/issue46909 closed by steve.dower #46910: Expect IndentationError, get SyntaxError: 'break' outside loop https://bugs.python.org/issue46910 closed by pablogsal #46916: There is a problem with escape characters in the path passed i https://bugs.python.org/issue46916 closed by vincent.fung #46919: After Python 3.9.2 is installed, the CVE-2019-12900 and CVE-20 https://bugs.python.org/issue46919 closed by ned.deily

1 0

build cpython on M1 Pro mac
by Yves Duprat March 4, 2022

March 4, 2022

Hi, I built cpython on a mac OSX 12.01. Python runs well. When starts, Python shows that version info which looks good: 'Python 3.11.0a5+ (heads/fix-issue-43352:f899da7fe5, Feb 25 2022, 10:04:53) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin' But when I want to used it in VSCode to debug a script, I have the following message: ------- This version of python seems to be incorrectly compiled (internal generated filenames are not absolute). This may make the debugger miss breakpoints. Related bug: http://bugs.python.org/issue1666807 ------- What do I miss ? I looked for a specific option in ./configure but seen nothing ? Can someone help me please ? Thank Yves

2 4

Test error
by sachin1122621003＠gmail.com March 3, 2022

March 3, 2022

I am new to open source and think to contribute to python . I was going through the developer guide and when i run this code "./python -m test -j3" on my terminal i got this result which say it fail .Now can someone tell me this is normal or happning to me only . If this happen to everyone how can i fix this == Tests result: FAILURE == 380 tests OK. 2 tests failed: test_compileall test_distutils 43 tests skipped: test_asdl_parser test_check_c_globals test_clinic test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_epoll test_fcntl test_fork1 test_gdb test_grp test_ioctl test_kqueue test_multiprocessing_fork test_multiprocessing_forkserver test_nis test_openpty test_ossaudiodev test_pipes test_poll test_posix test_pty test_pwd test_readline test_resource test_smtpnet test_socketserver test_spwd test_syslog test_threadsignals test_timeout test_tix test_tk test_ttk_guionly test_urllib2net test_urllibnet test_wait3 test_wait4 test_winsound test_xmlrpc_net test_xxtestfuzz test_zipfile64 Total duration: 11 min 14 sec Tests result: FAILURE

3 2

Python 3.11.0a6 is blocked
by Pablo Galindo Salgado March 3, 2022

March 3, 2022

Hi everyone, Unfortunately, we have some issues marked as release blockers that are holding the 3.11.0a6 release. Some of these issues have been solved but we still have a couple of them. Hopefully, we will solve these soon and I will proceed with the release. Thanks for your understanding. Regards from sunny Salamanca, Pablo Galindo Salgado

2 3

Making Cinder more broadly available
by Dino March 1, 2022

March 1, 2022

Hi, Cinder <https://github.com/facebookincubator/cinder> is Meta’s performance-oriented version of CPython 3.8. It has been in use as the production Python behind Instagram server for years, as well as powering various other Python applications across Meta. We are interested in making the improvements in Cinder more broadly available to the Python community. As a collection of features and components built on top of CPython, we’d like to: 1. Merge the feature into CPython for core features that are highly coupled with CPython internals (after we port it over to the main branch) 2. For features that *can* work as extension modules (like Cinder JIT) - extract them as a new open source standalone pip-installable extension module. 3. To enable the pip-installable extension module, we will need to add a few extension hooks to CPython, which we intend to carve out and port over to the main branch in order to merge upstream. Please read more about our upstreaming strategy and a few Cinder features in this document: https://docs.google.com/document/d/1l8I-FDE1xrIShm9eSNJqsGmY_VanMDX5-aK_guj… Complete source code of our CPython fork: https://github.com/facebookincubator/cinder We started filing bpo issues for a few of the pieces we want to merge upstream (links are in the google doc), and are looking forward to feedback from the community on the issues, overall approach, or any feedback on the features themselves. Cheers, Dino

1 0

October/November/December Steering Council update
by Pablo Galindo Salgado March 1, 2022

March 1, 2022

We’ve just published the October, November and December steering council update for 2021, also included below: https://github.com/python/steering-council/blob/main/updates/2021-10-steeri… https://github.com/python/steering-council/blob/main/updates/2021-11-steeri… https://github.com/python/steering-council/blob/main/updates/2021-12-steeri… Just as a reminder, if you have any questions or concerns, feel free to contact us or open an issue in the SC repo: https://github.com/python/steering-council *October 11* - SC met with the Developer-in-Residence and provided feedback on the Developer-in-Residence activity blog. - SC decided to keep 2020 dates for 2021 election. - SC discussed the draft about the SC's position on the PEP 649 / PEP 563 situation (stringified annotations) and provided feedback. - SC discussed whether or not they should extend their weekly meetings and/or do more async communications. They also decided that the SC should continue having meetings and working during the elections. *October 18* - Steering Council met with the GitHub Issues PM for an update on the migration. Priority right now is getting a testing repo out to the core dev sprint and triagers by Wednesday. - SC reviewed PEP 663 (Standardizing Enum str(), repr(), and format() behaviors) and potential PEP Delegates. - SC reviewed and discussed issues open on https://github.com/python/steering-council/issues *October 19* - SC met with the Search Firm the PSF is using for the Executive Director search, to provide input on the search parameters, and suggest potential candidates. *October 25* - Steering Council did a one hour Q&A recorded session with Visionary sponsor, Bloomberg. - Steering Council discussed PEP 649 and PEP 563 (stringified annotations). The group may want to delegate this to Łukasz but will discuss this more next week. - The candidate PEP Delegate for PEP 663 (Standardizing Enum str(), repr(), and format() behaviors) declined. The Steering Council will think of who else it can be delegated to, or take it on themselves. November 1 - The SC discussed PEP 649 -- Deferred Evaluation Of Annotations Using Descriptors <https://www.python.org/dev/peps/pep-0649/> & PEP 563 -- Postponed Evaluation of Annotations <https://www.python.org/dev/peps/pep-0563/>. The SC decided it would tell the community that more information is needed. Barry will draft an email explaining the current situation and calling for help from static and dynamic typing users. - The SC reviewed the open issues in the public Steering Council repo: https://github.com/python/steering-council/issues. - The SC discussed SC promotion during the call for nomination period, which started on Nov 1. - The SC discussed PEP 663 - Standardizing Enum str(), repr(), and format() behaviors <https://www.python.org/dev/peps/pep-0663/> and that Barry would write the rejection draft. - Brett informed the SC of an issue involving the return type of iterators and the SC decided it would continue discussing it at the next meeting (https://bugs.python.org/issue45250). <https://github.com/python/steering-council/blob/main/updates/2021-11-steeri…>November 8 - The SC had their bi-weekly meeting with the Developer-in-Residence. Received an update from Łukasz on what he is working on. - The SC discussed PEP 663 - Standardizing Enum str(), repr(), and format() behaviors <https://www.python.org/dev/peps/pep-0663/> and after some initial discussion, it was decided that it needs more discussion on python-dev@. - The SC discussed PEP 467 -- Minor API improvements for binary sequences <https://www.python.org/dev/peps/pep-0467/>. bchr() was taken out like the SC had requested and then an ascii() method was added in without any mention. Pablo will draft a response. - PEP 554 -- Multiple Interpreters in the Stdlib <https://www.python.org/dev/peps/pep-0554/> was discussed. A proof of concept is needed and the SC is not yet ready to review the PEP without it. <https://github.com/python/steering-council/blob/main/updates/2021-11-steeri…>November 15 - The SC met with Ezio and got an update on what Ezio was working on. SC told Ezio not to be overly concerned with perfection and not to strive for it. The SC gave Ezio a deadline of Dec 7th to get everything migrated over. - The SC discussed a PR to the CPython repo regarding __iter__ and iterators being inconsistently required by CPython ( https://github.com/python/cpython/pull/29170). The SC's decision was to update the definitions to say it's a CPython implementation detail that it is inconsistent in its checking whether what an iterable returns is a proper iterator or not (i.e. keep the __iter__ requirement but admit it isn't rigorously enforced by CPython as a historical accident). Brett committed updates to the PR. - The SC discussed a pull request in the “python/voters” repository that allows core devs to trigger automerge behavior from labels and decided it is fine to merge. - Carol gave an update on the documentation WG: The WG is going to a series of 1 hour monthly meetings and then do the rollout next year. - Barry updated the text for the PEP 663 rejection and sent it out to python-dev@. - Group reviewed text for "The current state of typing PEPs" (i.e. request for PEP 563 and 649) and Barry sent it to python-dev@ <https://mail.python.org/archives/list/python-dev@python.org/message/VIZEBX5…> . - The SC reviewed PEP 646 (Variadic Generics) and Barry sent the approval to the PEP team. - The SC discussed how notes will be taken after Ewa leaves. - The SC discussed PEP 518 (Specifying Minimum Build System Requirements for Python Projects) and how some folks are split on toml. <https://github.com/python/steering-council/blob/main/updates/2021-11-steeri…>November 22 - The SC met with Łukasz and received an update from Łukasz on his recent work. Additionally, the SC informed Łukasz he should work on upgrading the CLA process. Łukasz will reach out to Yury and Ezio. - The SC discussed Victor's request to remove asynchat and friends prematurely. The SC decided that the removal of asynchat needs to wait another 6 months. - The SC discussed Ethan's email to the SC asking about the re module and its RegexFlag. - The SC discussed the exception for Py_SET_TYPE request. Pablo will write a response to Petr request stating that the change needs to be rolled back in case there are any issues and they need to document any breakage in 3.11. - The SC discussed the various questions pertaining to the current state of typing PEPs. Thomas will draft a response to Chris and Paul's questions and the group discussed Barry's private conversation with a core dev about this as well. - The SC decided that it should be the 2022 SC that decides what happens to python-dev@. <https://github.com/python/steering-council/blob/main/updates/2021-11-steeri…>November 29 - The SC met with Ezio. The date to do the full migration has been pushed to mid-January. By Dec 7th, Ezio will put all issues into a test repo for python-dev to test and then the migration will happen when Steffen from GitHub returns from vacation in January. - The SC discussed PEP 659 - Specializing Adaptive Interpreter <https://www.python.org/dev/peps/pep-0659/> and whether or not it should be a PEP. It was determined that the informational aspect of the writeup is lacking such as how trade-offs are decided. The SC also discussed how different companies working on Python should work especially in terms of transparency. For the time being there will be no communications about this to the PEP authors or the community as the SC wants to discuss this further. This discussion will continue with the new SC as well. December 6 - The Steering Council met with Łukasz to discuss his progress and also introduced Ee to the meeting since Ee will be taking PSF lead on supporting Łukasz going forward. Łukasz updated the team on CLA work, testing Irit's PR for Exception Groups, adding a new triager, and giving a keynote at the PyCode Conference. - The Steering Council discussed the issue Jason posted <https://github.com/python/steering-council/issues/89> about deprecating importlib.resources legacy API. It was decided that Brett would respond to the comment stating that Jason is following policy and that the SC is satisfied, but that the larger discussion on changing the policy should happen on python-dev. The group also discussed how Python's deprecation policy impacts users and library authors. - The group said their goodbyes on Ewa's last day. <https://github.com/python/steering-council/blob/main/updates/2021-12-steeri…>December 13 - The Steering Council met with Ezio. There is no full import from BPO yet, but he will create one this week. The final migration is still tentatively scheduled for mid-January. Ezio will loop in Thomas on meetings with GitHub, which Ewa previously attended, or other SC members if Thomas can't make it. - The SC prepared for a Talk Python podcast recording after the meeting. <https://github.com/python/steering-council/blob/main/updates/2021-12-steeri…>December 13 (handoff meeting) - The old Steering Council and the New Steering Council met for the handoff meeting. - The SC discussed several aspects on how it has operated so far. - The SC received an updated from Thomas on the status on the search for a new PSF director. - The SC briefly reflected and discussed the diversity in the Steering Council and how to improve the situation in the future. This topic will be discussed further in the future. - The SC approved a request for a wording change in PEP 654 - Exception groups <https://www.python.org/dev/peps/pep-0654/>. - The SC had an overview of unresolved issues: definition/rules for the standard library, split in communication channels and the situation regarding type annotations with PEP 563 and 649. - The new SC decided to use discord as a new communication channel. - The SC discussed the status on the GitHub issues migration and what to do next to push the work forward. Regards from sunny Salamanca, Pablo Galindo Salgado

3 2