Mailman 3 February 2022 - Python-Dev

Proposal: Allow non-default after default arguments
by Sebastian Rittau 16 Oct '23

16 Oct '23

Currently, Python doesn't allow non-default arguments after default arguments: >>> def foo(x=None, y): pass File "<stdin>", line 1 def foo(x=None, y): pass ^ SyntaxError: non-default argument follows default argument I believe that at the time this was introduced, no use cases for this were known and this is is supposed to prevent a source of bugs. I have two use cases for this, one fringe, but valid, the other more important: The fringe use case: Suppose you have a function that takes a 2D coordinate value as separate "x" and "y" arguments. The "x" argument is optional, the "y" argument isn't. Currently there are two ways to do this, none of them particularly great: def foo(y, x): # reverse x and y arguments, confusing ... def foo(x, y=None): # Treat the x argument as y if only one argument is provided if y is None: x, y = y, x ... To me, the "natural" solution looks like this: def foo(x=None, y): ... # Called like this: foo(1, 2) foo(y=2) This could also be useful when evolving APIs. For example there is a function "bar" that takes two required arguments. In a later version, the first argument gains a useful default, the second doesn't. There is no sensible way to evolve the API at the moment. The more important use case involves @overloads. A condensed example of a function where the return type depends on an "encoding" parameter, followed by further parameters that could be called like this: foo(123, "utf-8") # would return bytes foo(encoding="utf-8") foo(123, None) # would return str foo(encoding=None) foo(x=123) # would return str foo() This could ideally be written as: @overload def foo(x: int = ..., encoding: None = ...) -> str: ... @overload def foo(x: int = ..., encoding: str) -> bytes: ... # plus the actual implementation But due to the syntax constraint, this needs to be hacked around with a third overload: @overload def foo(x: int = ... encoding: None = ...) -> str: ... @overload def foo(x: int, encoding: str) -> bytes: ... # for foo(123, "utf-8") @overload def foo(*, encoding: str) -> bytes: ... # for foo(encoding="utf-8") Not only is this hard to read, real examples in typeshed are usually more complex, with many arguments before or after the affected argument or even multiple affected arguments. This often becomes too complex to write or maintain. Here is one example from the wild: https://github.com/python/typeshed/blob/b95b729b9e07ab21d252701af0f5b740467… Allowing non-default arguments after default arguments would solve both use cases above and eliminates a special case. I'm also not sure what exactly the current SyntaxError really protects us from. Adding a non-default after a default argument can't really lead bugs. - Sebastian

11 14

Python multithreading without the GIL
by Sam Gross 31 Jul '23

31 Jul '23

Hi, I've been working on changes to CPython to allow it to run without the global interpreter lock. I'd like to share a working proof-of-concept that can run without the GIL. The proof-of-concept involves substantial changes to CPython internals, but relatively few changes to the C-API. It is compatible with many C extensions: extensions must be rebuilt, but usually require small or no modifications to source code. I've built compatible versions of packages from the scientific Python ecosystem, and they are installable through the bundled "pip". Source code: https://github.com/colesbury/nogil Design overview: https://docs.google.com/document/d/18CXhDb1ygxg-YXNBJNzfzZsDFosB5e6BfnXLlej… My goal with the proof-of-concept is to demonstrate that removing the GIL is feasible and worthwhile, and that the technical ideas of the project could serve as a basis of such an effort. I'd like to start a discussion about these ideas and gauge the community's interest in this approach to removing the GIL. Regards, Sam Gross colesbury(a)gmail.com / sgross(a)fb.com

24 48

Mixed Python/C debugging
by Skip Montanaro 26 Apr '23

26 Apr '23

Having tried comp.lang.python with no response, I turn here... After at least ten years away from Python's run-time interpreter & byte code compiler, I'm getting set to familiarize myself with that again. This will, I think, entail debugging a mixed Python/C environment. I'm an Emacs user and am aware that GDB since 7.0 has support for debugging at the Python code level. Is Emacs+GDB my best bet? Are there any Python IDEs which support C-level breakpoints and debugging? Thanks, Skip

5 6

PEP 448 review
by Guido van Rossum 29 Mar '23

29 Mar '23

I'm back, I've re-read the PEP, and I've re-read the long thread with "(no subject)". I think Georg Brandl nailed it: """ *I like the "sequence and dict flattening" part of the PEP, mostly because itis consistent and should be easy to understand, but the comprehension syntaxenhancements seem to be bad for readability and "comprehending" what the codedoes.The call syntax part is a mixed bag on the one hand it is nice to be consistent with the extended possibilities in literals (flattening), but on the other hand there would be small but annoying inconsistencies anyways (e.g. the duplicate kwarg case above).* """ Greg Ewing followed up explaining that the inconsistency between dict flattening and call syntax is inherent in the pre-existing different rules for dicts vs. keyword args: {'a':1, 'a':2} results in {'a':2}, while f(a=1, a=2) is an error. (This form is a SyntaxError; the dynamic case f(a=1, **{'a': 1}) is a TypeError.) For me, allowing f(*a, *b) and f(**d, **e) and all the other combinations for function calls proposed by the PEP is an easy +1 -- it's a straightforward extension of the existing pattern, and anybody who knows what f(x, *a) does will understand f(x, *a, y, *b). Guessing what f(**d, **e) means shouldn't be hard either. Understanding the edge case for duplicate keys with f(**d, **e) is a little harder, but the error messages are pretty clear, and it is not a new edge case. The sequence and dict flattening syntax proposals are also clean and logical -- we already have *-unpacking on the receiving side, so allowing *x in tuple expressions reads pretty naturally (and the similarity with *a in argument lists certainly helps). From here, having [a, *x, b, *y] is also natural, and then the extension to other displays is natural: {a, *x, b, *y} and {a:1, **d, b:2, **e}. This, too, gets a +1 from me. So that leaves comprehensions. IIRC, during the development of the patch we realized that f(*x for x in xs) is sufficiently ambiguous that we decided to disallow it -- note that f(x for x in xs) is already somewhat of a special case because an argument can only be a "bare" generator expression if it is the only argument. The same reasoning doesn't apply (in that form) to list, set and dict comprehensions -- while f(x for x in xs) is identical in meaning to f((x for x in xs)), [x for x in xs] is NOT the same as [(x for x in xs)] (that's a list of one element, and the element is a generator expression). The basic premise of this part of the proposal is that if you have a few iterables, the new proposal (without comprehensions) lets you create a list or generator expression that iterates over all of them, essentially flattening them: >>> xs = [1, 2, 3] >>> ys = ['abc', 'def'] >>> zs = [99] >>> [*xs, *ys, *zs] [1, 2, 3, 'abc', 'def', 99] >>> But now suppose you have a list of iterables: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xss[0], *xss[1], *xss[2]] [1, 2, 3, 'abc', 'def', 99] >>> Wouldn't it be nice if you could write the latter using a comprehension? >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs for xs in xss] [1, 2, 3, 'abc', 'def', 99] >>> This is somewhat seductive, and the following is even nicer: the *xs position may be an expression, e.g.: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs[:2] for xs in xss] [1, 2, 'abc', 'def', 99] >>> On the other hand, I had to explore the possibilities here by experimenting in the interpreter, and I discovered some odd edge cases (e.g. you can parenthesize the starred expression, but that seems a syntactic accident). All in all I am personally +0 on the comprehension part of the PEP, and I like that it provides a way to "flatten" a sequence of sequences, but I think very few people in the thread have supported this part. Therefore I would like to ask Neil to update the PEP and the patch to take out the comprehension part, so that the two "easy wins" can make it into Python 3.5 (basically, I am accepting two-thirds of the PEP :-). There is some time yet until alpha 2. I would also like code reviewers (Benjamin?) to start reviewing the patch <http://bugs.python.org/issue2292>, taking into account that the comprehension part needs to be removed. -- --Guido van Rossum (python.org/~guido)

7 17

PEP 638: Syntactic macros
by Mark Shannon 05 Feb '23

05 Feb '23

Hi everyone, I've submitted my PEP on syntactic macros as PEP 638. https://www.python.org/dev/peps/pep-0638/ All comments and suggestions are welcome. Cheers, Mark

16 25

How about using modern C++ in development of CPython ?
by redradist＠gmail.com 19 Oct '22

19 Oct '22

Hi all, What about of using modern C++ in developing CPython ? With C++ we can get the following benefits: 1) Readability - less code, most code is hidden by abstraction without losing performance 2) Maintainability - no code duplication in favor of using reusable classes 3) RAII - Resource Acquisition Is Initialization, predictable allocation and free resources ...

23 51

PEP 505 (None-aware operators) for Python 3.11
by Doug Swarin 21 Sep '22

21 Sep '22

Hello, I've been following PEP 505 (https://www.python.org/dev/peps/pep-0505/) since it was first proposed for Python 3.8. It's been deferred for some time and I'm interested in seeing it in Python 3.11, but I know there were also a number of objections which resulted in it being deferred (including by one of the original authors, Mr. Dower). I did email both Mr. Dower and Mr. Haase and they graciously gave me their permission to bring it up on this list for discussion and hopefully final pronouncement one way or the other. I personally believe that the PEP will result in a significant reduction in boilerplate code, and it is substantially similar to the same operators now found in a number of other languages, especially C# and JavaScript (https://wikipedia.org/wiki/Null_coalescing_operator and https://wikipedia.org/wiki/Safe_navigation_operator). I believe strong and valid arguments can be made about the use of None being a fundamental flaw in some types of coding (and that adding additional support for it to the language will increase the use of None in this way), but I also believe there are many use cases in programming where it is by far the simplest way to express various semantics, and the fact exists that None is already used extensively in large quantities of code, and further that there is already a great deal of code written to constantly test against None and break out of a statement without throwing an error. I also understand the argument that especially the maybe-dot (?.) and maybe-subscript (?[) operators can decrease readability of code and also believe these are valid arguments against it. While I believe the existence and use of these operators in other languages definitely helps the case that these can be used and understood successfully, I think it is entirely valid to either consider other syntax (though I prefer the chosen syntax of PEP 505), or even to reduce PEP 505 to having only the coalesce operator (??) and the maybe-assign operator (??=). Separately, I have implemented a pure-Python solution for PEP505 (which is definitely rather beta) which might help test the waters for a final implementation in CPython (though the CPython implementation would of course be much more efficient). It can be found at https://pypi.org/project/pep505/ Thanks, Doug

22 53

PEP 563 and Python 3.10.
by Thomas Wouters 22 Jul '22

22 Jul '22

(Starting a new thread so as not to derail any of the ongoing discussions.) Thanks, everyone, for your thoughts on Python 3.10 and the impact of PEP 563 (postponed evaluation of annotations) becoming the default. The Steering Council has considered the issue carefully, along with many of the proposed alternatives and solutions, and we’ve decided that at this point, we simply can’t risk the compatibility breakage of PEP 563. We need to roll back the change that made stringified annotations the default, at least for 3.10. (Pablo is already working on this.) To be clear, we are not reverting PEP 563 itself. The future import will keep working like it did since Python 3.7. We’re delaying making PEP 563 string-based annotations the default until Python 3.11. This will give us time to find a solution that works for everyone (or to find a feasible upgrade path for users who currently rely on evaluated annotations). Some considerations that led us to this decision: - PEP 563’s default change is clearly too disruptive to downstream users and third-party libraries to happen right now. We can’t risk breaking even a small subset of the FastAPI/pydantic users, not to mention other uses of evaluated type annotations that we’re not aware of yet. - PEP 563 provides no warning to users of the feature it’s disabling. Without that, we can’t expect users to be aware of the upcoming breakage. The lack of a warning was by design, and made sense in a world where type annotations were only consumed by static type checkers --- but that’s not actually the situation we’re in. There are clearly existing real-world, run-time uses of type annotations that would be adversely affected by this change. - Originally, PEP 563 was scheduled to take effect in Python 4, and this changed recently (after the discussion in the Language Summit of 2020). It's possible that third-party libraries and users didn’t plan to react in the current time frame as they were not aware of this change in timing. - There isn’t enough time to properly discuss PEP 649 or any of the alternatives before the beta 1 deadline, and we really need to make sure we don’t compound errors here. We need to look for a long term solution, which isn’t possible while still maintaining the release deadlines of Python 3.10. That means we’re also deferring PEP 649 to Python 3.11. In the Steering Council’s unanimous opinion, rolling back the default flip for stringified annotations in Python 3.10 is the least disruptive of all the options. We need to continue discussing the issue and potential solutions, since this merely postpones the problem until 3.11. (For the record, postponing the change further is not off the table, either, for example if the final decision is to treat evaluated annotations as a deprecated feature, with warnings on use.) For what it’s worth, the SC is also considering what we can do to reduce the odds of something like this happening again, but that’s a separate consideration, and a multi-faceted one at that. For the Steering Council, Thomas. -- Thomas Wouters <thomas(a)python.org> Hi! I'm an email virus! Think twice before sending your email to help me spread!

6 5

compiled python3.10 is unable to find _ssl
by Sandeep Gupta 29 May '22

29 May '22

I am having compilation issues again with python3.10 with ssl . The ./configure was invoked with ssl options and ssl modules seems to be build successfully. """ The following modules found by detect_modules() in setup.py, have been built by the Makefile instead, as configured by the Setup files: _abc _hashlib _ssl pwd time """ However, when I do import ssl from python, I get the error: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/shared/Builds/Python-3.10.0/lib/python3.10/ssl.py", line 98, in <module> import _ssl # if we can't import it, let the error propagate ModuleNotFoundError: No module named '_ssl' I can't find _ssl.so in build or install directory. Thanks -S

6 12

Slowly bend the C API towards the limited API to get a stable ABI for everyone
by Victor Stinner 20 Apr '22

20 Apr '22

Hi, There is a reason why I'm bothering C extensions maintainers and Python core developers with my incompatible C API changes since Python 3.8. Let me share my plan with you :-) In 2009 (Python 3.2), Martin v. Löwis did an amazing job with the PEP 384 "Defining a Stable ABI" to provide a "limited C API" and a "stable ABI" for C extensions: build an extension once, use it on multiple Python versions. Some projects like PyQt5 and cryptograpy use it, but it is just a drop in the PyPI ocean (353,084 projects). I'm trying to bend the "default" C API towards this "limited C API" to make it possible tomorrow to build *more* C extensions for the stable ABI. My goal is that the stable ABI would be the default, and only a minority of C extensions would opt-out because they need to access to more functions for best performance. The basic problem is that at the ABI level, C extensions must only call functions, rather than getting and setting directly to structure members. Structures changes frequently in Python (look at changes between Python 3.2 and Python 3.11), and any minor structure change breaks the ABI. The limited C API hides structures and only use function calls to solve this problem. Since 2020, I'm modifying the C API, one function by one, to slowly hide implementations (prepare the API to make strutures opaque). I focused on the following structures: * PyObject and PyVarObject (bpo-39573) * PyTypeObject (bpo-40170) * PyFrameObject (bpo-40421) * PyThreadState (bpo-39947) The majority of C extensions use functions and macros, they don't access directly structure members. There are a few members which are sometimes accessed directly which prevents making these structures opaque. For example, some old C extensions use obj->ob_type rather than Py_TYPE(obj). Fixing the minority of C extensions should benefit to the majority which may become compatible with the stable ABI. I am also converting macros to static inline functions to fix their API: define parameter types, result type and avoid surprising macros side effects ("macro pitfalls"). I wrote the PEP 670 "Convert macros to functions in the Python C API" for these changes. I wrote the upgrade_pythoncapi.py tool in my pythoncapi_project (*) which modify C code to use Py_TYPE(), Py_SIZE() and Py_REFCNT() rather than accessing directly PyObject and PyVarObject members. (*) https://github.com/pythoncapi/pythoncapi_compat In this tool, I also added "Borrow" variant of functions like PyFrame_GetCode() which returns a strong reference, to replace frame->f_code with _PyFrame_GetCodeBorrow(). In Python 3.11, you cannot use the frame->f_code member anymore, since it has been removed! You must call PyFrame_GetCode() (or pythoncapi_compat _PyFrame_GetCodeBorrow() variant). There are also a few macros which can be used as l-values like Py_TYPE(): "Py_TYPE(type1) = type2" must now be written "Py_SET_TYPE(type1, type2)" to avoid setting directly the tp_type type at the ABI level. I proposed the PEP 674 "Disallow using Py_TYPE() and Py_SIZE() macros as l-values" to solve these issues. Currently, many "functions" are still implemented as macros or static inline functions, so C extensions still access structure members at the ABI level for best Python performance. Converting these to regular functions has an impact on performance and I would prefer to first write a PEP giving the rationale for that. Today, it is not possible yet to build numpy for the stable ABI. The gap is just too large for this big C extension. But step by step, the C API becomes closer to the limited API, and more and more code is ready to be built for the stable ABI. Well, these C API changes have other advantages, like preparing Python for further optimizations, ease Python maintenance, clarify the seperation between the limited C API and the default C API, etc. ;-) Victor -- Night gathers, and now my watch begins. It shall not end until my death.

12 38

PEP 678: Enriching Exceptions with Notes
by Zac Hatfield Dodds 12 Apr '22

12 Apr '22

Hi all, I've written PEP 678, proposing the addition of a __note__ attribute which can be used to enrich displayed exceptions. This is particularly useful with ExceptionGroup, or where exception chaining is unsuitable, and ensures that 'orphan' output is not produced if the annotated exception is caught. Link to the PEP: https://www.python.org/dev/peps/pep-0678/ *Please, redirect all discussions to:* https://discuss.python.org/t/pep-678-enriching-exceptions-with-notes/13374 Thanks for your time, and to those who have already made (very helpful!) comments! Best wishes from just-moved-to San Francisco, Zac Hatfield-Dodds

2 1

It's now time to deprecate the stdlib urllib module
by Victor Stinner 31 Mar '22

31 Mar '22

Hi, I propose to deprecate the urllib module in Python 3.11. It would emit a DeprecationWarning which warn users, so users should consider better alternatives like urllib3 or httpx: well known modules, better maintained, more secure, support HTTP/2 (httpx), etc. I don't propose to schedule its removal. Let's discuss the removal in 1 or 2 years. -- urllib has many abstraction to support a wide range of protocols with "handlers": HTTP, HTTPS, FTP, "local file", proxy, HTTP authentication, HTTP Cookie, etc. A simple HTTP request using Basic Authentication requires 10-20 lines of code, whereas it should be a single line. Users (me included) don't like urllib API which was too complicated for common tasks. -- Unhappy users created multiple better alternatives to the stdlib urllib module. In 2008, the "urllib3" module was created to provide an API designed to be as simple as possible for the most common HTTP and HTTPS requests. Example: req = http.request('GET', 'http://httpbin.org/robots.txt'). In 2011, the "requests" module based on urllib3 was created. In 2013, the "aiohttp" module based on asyncio was created. In 2015, new "httpx" module was created: req = httpx.get('https://www.example.org/') Not only httpx has a regular "synchronous" API (blocking function calls), but it also has an asynchronous API! Sadly, while HTTP/3 is being developed, it seems like in this list, httpx is the only HTTP client library supporting HTTP/2 currently :-( For HTTP/2, I also found the "httplib2" module. For HTTP/3, I found the "http3" and "aioquic" modules. -- Let's come back to urllib: * It's API is too complicated * It doesn't support HTTP/2 nor HTTP/3 * It's barely maintained: there are 121 open issues including 3 security issues! The 3 open security issues: * bpo-33661 open 2018; * bpo-36338 open in 2019; * bpo-45795 open in 2021. Usually, it's bad when you refer to an open security issue by its creation year :-( The urllib module has long history of security vulnerabilities. List of *fixed* vulnerabilities: * 2011 (bpo-11662): https://python-security.readthedocs.io/vuln/urllib-redirect.html * 2017 (bpo-30119): https://python-security.readthedocs.io/vuln/urllib-ftp-stream-injection.html * 2017 (bpo-30500): https://python-security.readthedocs.io/vuln/urllib-connects-wrong-host.html * 2019 (bpo-35907): https://python-security.readthedocs.io/vuln/urllib-local-file-scheme.html * 2019 (bpo-38826): https://python-security.readthedocs.io/vuln/urllib-basic-auth-regex.html * 2021 (bpo-42967): https://python-security.readthedocs.io/vuln/urllib-query-string-semicolon-s… * 2021 (bpo-43075): https://python-security.readthedocs.io/vuln/urllib-basic-auth-regex2.html * 2021 (bpo-44022): https://python-security.readthedocs.io/vuln/urllib-100-continue-loop.html urllib is a package made of 4 parts: * urllib.request for opening and reading URLs * urllib.error containing the exceptions raised by urllib.request * urllib.parse for parsing URLs * urllib.robotparser for parsing robots.txt files I propose to deprecate all of them. Maybe the deprecation can be different for each sub-module? Victor -- Night gathers, and now my watch begins. It shall not end until my death.

39 75

A memory map based data persistence and startup speedup approach
by Yichen Yan 20 Mar '22

20 Mar '22

Hi folks, as illustrated in faster-cpython#150 [1], we have implemented a mechanism that supports data persistence of a subset of python date types with mmap, therefore can reduce package import time by caching code object. This could be seen as a more eager pyc format, as they are for the same purpose, but our approach try to avoid [de]serialization. Therefore, we get a speedup in overall python startup by ~15%. Currently, we’ve made it a third-party library and have been working on open-sourcing. Our implementation (whose non-official name is “pycds”) mainly contains two parts: importlib hooks, this implements the mechanism to dump code objects to an archive and a `Finder` that supports loading code object from mapped memory. Dumping and loading (subset of) python types with mmap. In this part, we deal with 1) ASLR by patching `ob_type` fields; 2) hash seed randomization by supporting only basic types who don’t have hash-based layout (i.e. dict is not supported); 3) interned string by re-interning strings while loading mmap archive and so on. After pycds has been installed, complete workflow of our approach includes three parts: Record name of imported packages to heap.lst, `PYCDSMODE=TRACE PYCDSLIST=heap.lst python run.py` Dump memory archive of code objects of imported packages, this step does not involve the python script, `PYCDSMODE=DUMP PYCDSLIST=heap.lst PYCDSARCHIVE=heap.img python` Run other python processes with created archive, `PYCDSMODE=SHARE PYCDSARCHIVE=heap.img python run.py` We could even make use of immortal objects if PEP 683 [2] was accepted, which could give CDS more performance improvements. Currently, any archived object is virtually immortal, we add rc by 1 to who has been copied to the archive to avoid being deallocated. However, without changes to CPython, rc fields of archived objects will still be updated, therefore have extra footprint due to CoW. More background and detailed implementation could be found at [1]. We think it could be an effective way to improve python’s startup performance, and could even do more like sharing large data between python instances. As suggested in python-ideas [3], we posted this here, looking for questions/suggestions to the overall design and workflow, we also welcome code reviews after we get our lawyers happy and can publish the code. Best, Yichen Yan Alibaba Compiler Group [1] “Faster startup -- Share code objects from memory-mapped file”, https://github.com/faster-cpython/ideas/discussions/150 [2] PEP 683: "Immortal Objects, Using a Fixed Refcount" (draft), https://mail.python.org/archives/list/python-dev@python.org/message/TPLEYDC… [3] [Python-ideas] "A memory map based data persistence and startup speedup approach", https://mail.python.org/archives/list/python-ideas@python.org/thread/UKEBNH…

2 1

RFC on PEP 655: Required[] and NotRequired[] for TypedDict
by David Foster 10 Mar '22

10 Mar '22

This PEP [1] introduces syntax to mark individual keys of a TypedDict as either required or potentially-missing. Previously the only way to have a TypedDict with mixed required and non-required keys was to define two TypedDicts - one with total=True and one with total=False - and have one of those TypedDicts inherit from the other. This PEP introduces special forms Required[] and NotRequired[] to the "typing" module that can be used to directly mark individual items of a TypedDict as required or not-required. We have implementations for mypy, pyright, and pyanalyze. Consensus on the PEP has been reached in typing-sig. My understanding of the PEP process as described in PEP 1 [2] is that content review should be requested of core developers here in python-dev. So consider this my official request for comments! ^_^ I believe that after review & feedback from python-dev the next step would be to submit this PEP to the Steering Council. However it's not clear to me from [2] where I should actually do that when the time comes. -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy [1]: https://www.python.org/dev/peps/pep-0655/ [2]: https://www.python.org/dev/peps/pep-0001/#pep-review-resolution

4 6

PEP 682: Format Specifier for Signed Zero
by John Belmonte 06 Mar '22

06 Mar '22

Just a note that this PEP is being discussed at https://discuss.python.org/t/pep-682-format-specifier-for-signed-zero TLDR; When formatting a number (especially with rounding options), often you don't want negative-signed zero as an output. A format spec extension enables normalizing such values to positive zero. Regards, --John Belmonte

2 1

PEP 683: "Immortal Objects, Using a Fixed Refcount" (round 2)
by Eric Snow 28 Feb '22

28 Feb '22

Thanks to all those that provided feedback. I've worked to substantially update the PEP in response. The text is included below. Further feedback is appreciated. -eric ------------------------ PEP: 683 Title: Immortal Objects, Using a Fixed Refcount Author: Eric Snow <ericsnowcurrently(a)gmail.com>, Eddie Elizondo <eduardo.elizondorueda(a)gmail.com> Discussions-To: https://mail.python.org/archives/list/python-dev@python.org/thread/TPLEYDCX… Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 10-Feb-2022 Python-Version: 3.11 Post-History: 15-Feb-2022 Resolution: Abstract ======== Currently the CPython runtime maintains a `small amount of mutable state <Runtime Object State_>`_ in the allocated memory of each object. Because of this, otherwise immutable objects are actually mutable. This can have a large negative impact on CPU and memory performance, especially for approaches to increasing Python's scalability. The solution proposed here provides a way to mark an object as one for which that per-object runtime state should not change. Specifically, if an object's refcount matches a very specific value (defined below) then that object is treated as "immortal". If an object is immortal then its refcount will never be modified by ``Py_INCREF()``, etc. Consequently, the refcount will never reach 0, so that object will never be cleaned up (unless explicitly done, e.g. during runtime finalization). Additionally, all other per-object runtime state for an immortal object will be considered immutable. This approach has some possible negative impact, which is explained below, along with mitigations. A critical requirement for this change is that the performance regression be no more than 2-3%. Anything worse the performance-neutral requires that the other benefits are proportionally large. Aside from specific applications, the fundamental improvement here is that now an object can be truly immutable. (This proposal is meant to be CPython-specific and to affect only internal implementation details. There are some slight exceptions to that which are explained below. See `Backward Compatibility`_, `Public Refcount Details`_, and `scope`_.) Motivation ========== As noted above, currently all objects are effectively mutable. That includes "immutable" objects like ``str`` instances. This is because every object's refcount is frequently modified as the object is used during execution. This is especially significant for a number of commonly used global (builtin) objects, e.g. ``None``. Such objects are used a lot, both in Python code and internally. That adds up to a consistent high volume of refcount changes. The effective mutability of all Python objects has a concrete impact on parts of the Python community, e.g. projects that aim for scalability like Instragram or the effort to make the GIL per-interpreter. Below we describe several ways in which refcount modification has a real negative effect on such projects. None of that would happen for objects that are truly immutable. Reducing CPU Cache Invalidation ------------------------------- Every modification of a refcount causes the corresponding CPU cache line to be invalidated. This has a number of effects. For one, the write must be propagated to other cache levels and to main memory. This has small effect on all Python programs. Immortal objects would provide a slight relief in that regard. On top of that, multi-core applications pay a price. If two threads (running simultaneously on distinct cores) are interacting with the same object (e.g. ``None``) then they will end up invalidating each other's caches with each incref and decref. This is true even for otherwise immutable objects like ``True``, ``0``, and ``str`` instances. CPython's GIL helps reduce this effect, since only one thread runs at a time, but it doesn't completely eliminate the penalty. Avoiding Data Races ------------------- Speaking of multi-core, we are considering making the GIL a per-interpreter lock, which would enable true multi-core parallelism. Among other things, the GIL currently protects against races between multiple concurrent threads that may incref or decref the same object. Without a shared GIL, two running interpreters could not safely share any objects, even otherwise immutable ones like ``None``. This means that, to have a per-interpreter GIL, each interpreter must have its own copy of *every* object. That includes the singletons and static types. We have a viable strategy for that but it will require a meaningful amount of extra effort and extra complexity. The alternative is to ensure that all shared objects are truly immutable. There would be no races because there would be no modification. This is something that the immortality proposed here would enable for otherwise immutable objects. With immortal objects, support for a per-interpreter GIL becomes much simpler. Avoiding Copy-on-Write ---------------------- For some applications it makes sense to get the application into a desired initial state and then fork the process for each worker. This can result in a large performance improvement, especially memory usage. Several enterprise Python users (e.g. Instagram, YouTube) have taken advantage of this. However, the above refcount semantics drastically reduce the benefits and has led to some sub-optimal workarounds. Also note that "fork" isn't the only operating system mechanism that uses copy-on-write semantics. Anything that uses ``mmap`` relies on copy-on-write, including sharing data from shared objects files between processes. Rationale ========= The proposed solution is obvious enough that both of this proposal's authors came to the same conclusion (and implementation, more or less) independently. The Pyston project `uses a similar approach <Pyston_>`_. Other designs were also considered. Several possibilities have also been discussed on python-dev in past years. Alternatives include: * use a high bit to mark "immortal" but do not change ``Py_INCREF()`` * add an explicit flag to objects * implement via the type (``tp_dealloc()`` is a no-op) * track via the object's type object * track with a separate table Each of the above makes objects immortal, but none of them address the performance penalties from refcount modification described above. In the case of per-interpreter GIL, the only realistic alternative is to move all global objects into ``PyInterpreterState`` and add one or more lookup functions to access them. Then we'd have to add some hacks to the C-API to preserve compatibility for the may objects exposed there. The story is much, much simpler with immortal objects Impact ====== Benefits -------- Most notably, the cases described in the two examples above stand to benefit greatly from immortal objects. Projects using pre-fork can drop their workarounds. For the per-interpreter GIL project, immortal objects greatly simplifies the solution for existing static types, as well as objects exposed by the public C-API. In general, a strong immutability guarantee for objects enables Python applications to scale like never before. This is because they can then leverage multi-core parallelism without a tradeoff in memory usage. This is reflected in most of the above cases. Performance ----------- A naive implementation shows `a 4% slowdown`_. Several promising mitigation strategies will be pursued in the effort to bring it closer to performance-neutral. See the `mitigation`_ section below. On the positive side, immortal objects save a significant amount of memory when used with a pre-fork model. Also, immortal objects provide opportunities for specialization in the eval loop that would improve performance. .. _a 4% slowdown: https://github.com/python/cpython/pull/19474#issuecomment-1032944709 Backward Compatibility ---------------------- This proposal is meant to be completely compatible. It focuses strictly on internal implementation details. It does not involve changes to any public API, other a few minor changes in behavior related to refcounts (but only for immortal objects): * code that inspects the refcount will see a really, really large value * the new noop behavior may break code that: * depends specifically on the refcount to always increment or decrement (or have a specific value from ``Py_SET_REFCNT()``) * relies on any specific refcount value, other than 0 * directly manipulates the refcount to store extra information there Again, those changes in behavior only apply to immortal objects, not most of the objects a user will access. Furthermore, users cannot mark an object as immortal so no user-created objects will ever have that changed behavior. Users that rely on any of the changing behavior for global (builtin) objects are already in trouble. Also note that code which checks for refleaks should keep working fine, unless it checks for hard-coded small values relative to some immortal object. The problems noticed by `Pyston`_ shouldn't apply here since we do not modify the refcount. See `Public Refcount Details`_ and `scope`_ below for further discussion. Stable ABI ---------- The approach is also compatible with extensions compiled to the stable ABI. Unfortunately, they will modify the refcount and invalidate all the performance benefits of immortal objects. However, the high bit of the refcount `will still match _Py_IMMORTAL_REFCNT <_Py_IMMORTAL_REFCNT_>`_ so we can still identify such objects as immortal. At worst, objects in that situation would feel the effects described in the `Motivation`_ section. Even then the overall impact is unlikely to be significant. Also see `_Py_IMMORTAL_REFCNT`_ below. Accidental Immortality ---------------------- Hypothetically, a regular object could be incref'ed so much that it reaches the magic value needed to be considered immortal. That means it would accidentally never be cleaned up (by going back to 0). While it isn't impossible, this accidental scenario is so unlikely that we need not worry. Even if done deliberately by using ``Py_INCREF()`` in a tight loop and each iteration only took 1 CPU cycle, it would take 2^61 cycles (on a 64-bit processor). At a fast 5 GHz that would still take nearly 500,000,000 seconds (over 5,000 days)! If that CPU were 32-bit then it is (technically) more possible though still highly unlikely. Also note that it is doubly unlikely to be a problem because it wouldn't matter until the refcount got back to 0 and the object was cleaned up. So any object that hit that magic "immortal" refcount value would have to be decref'ed that many times again before the change in behavior would be noticed. Again, the only realistic way that the magic refcount would be reached (and then reversed) is if it were done deliberately. (Of course, the same thing could be done efficiently using ``Py_SET_REFCNT()`` though that would be even less of an accident.) At that point we don't consider it a concern of this proposal. Alternate Python Implementations -------------------------------- This proposal is CPython-specific. However, it does relate to the behavior of the C-API, which may affect other Python implementations. Consequently, the effect of changed behavior described in `Backward Compatibility`_ above also applies here (e.g. if another implementation is tightly coupled to specific refcount values, other than 0, or on exactly how refcounts change, then they may impacted). Security Implications --------------------- This feature has no known impact on security. Maintainability --------------- This is not a complex feature so it should not cause much mental overhead for maintainers. The basic implementation doesn't touch much code so it should have much impact on maintainability. There may be some extra complexity due to performance penalty mitigation. However, that should be limited to where we immortalize all objects post-init and that code will be in one place. Specification ============= The approach involves these fundamental changes: * add `_Py_IMMORTAL_REFCNT`_ (the magic value) to the internal C-API * update ``Py_INCREF()`` and ``Py_DECREF()`` to no-op for objects with the magic refcount (or its most significant bit) * do the same for any other API that modifies the refcount * stop modifying ``PyGC_Head`` for immortal GC objects ("containers") * ensure that all immortal objects are cleaned up during runtime finalization Then setting any object's refcount to ``_Py_IMMORTAL_REFCNT`` makes it immortal. (There are other minor, internal changes which are not described here.) In the following sub-sections we dive into the details. First we will cover some conceptual topics, followed by more concrete aspects like specific affected APIs. Public Refcount Details ----------------------- In `Backward Compatibility`_ we introduced possible ways that user code might be broken by the change in this proposal. Any contributing misunderstanding by users is likely due in large part to the names of the refcount-related API and to how the documentation explains those API (and refcounting in general). Between the names and the docs, we can clearly see answers to the following questions: * what behavior do users expect? * what guarantees do we make? * do we indicate how to interpret the refcount value they receive? * what are the use cases under which a user would set an object's refcount to a specific value? * are users setting the refcount of objects they did not create? As part of this proposal, we must make sure that users can clearly understand on which parts of the refcount behavior they can rely and which are considered implementation details. Specifically, they should use the existing public refcount-related API and the only refcount value with any meaning is 0. All other values are considered "not 0". This information will be clarified in the `documentation <Documentation_>`_. Arguably, the existing refcount-related API should be modified to reflect what we want users to expect. Something like the following: * ``Py_INCREF()`` -> ``Py_ACQUIRE_REF()`` (or only support ``Py_NewRef()``) * ``Py_DECREF()`` -> ``Py_RELEASE_REF()`` * ``Py_REFCNT()`` -> ``Py_HAS_REFS()`` * ``Py_SET_REFCNT()`` -> ``Py_RESET_REFS()`` and ``Py_SET_NO_REFS()`` However, such a change is not a part of this proposal. It is included here to demonstrate the tighter focus for user expectations that would benefit this change. Constraints ----------- * ensure that otherwise immutable objects can be truly immutable * minimize performance penalty for normal Python use cases * be careful when immortalizing objects that we don't actually expect to persist until runtime finalization. * be careful when immortalizing objects that are not otherwise immutable .. _scope: Scope of Changes ---------------- Object immortality is not meant to be a public feature but rather an internal one. So the proposal does *not* including adding any new public C-API, nor any Python API. However, this does not prevent us from adding (publicly accessible) private API to do things like immortalize an object or tell if one is immortal. The particular details of: * how to mark something as immortal * how to recognize something as immortal * which subset of functionally immortal objects are marked as immortal * which memory-management activities are skipped or modified for immortal objects are not only Cpython-specific but are also private implementation details that are expected to change in subsequent versions. Immortal Mutable Objects ------------------------ Any object can be marked as immortal. We do not propose any restrictions or checks. However, in practice the value of making an object immortal relates to its mutability and depends on the likelihood it would be used for a sufficient portion of the application's lifetime. Marking a mutable object as immortal can make sense in some situations. Many of the use cases for immortal objects center on immutability, so that threads can safely and efficiently share such objects without locking. For this reason a mutable object, like a dict or list, would never be shared (and thus no immortality). However, immortality may be appropriate if there is sufficient guarantee that the normally mutable object won't actually be modified. On the other hand, some mutable objects will never be shared between threads (at least not without a lock like the GIL). In some cases it may be practical to make some of those immortal too. For example, ``sys.modules`` is a per-interpreter dict that we do not expect to ever get freed until the corresponding interpreter is finalized. By making it immortal, we no longer incur the extra overhead during incref/decref. We explore this idea further in the `mitigation`_ section below. (Note that we are still investigating the impact on GC of immortalizing containers.) Implicitly Immortal Objects --------------------------- If an immortal object holds a reference to a normal (mortal) object then that held object is effectively immortal. This is because that object's refcount can never reach 0 until the immortal object releases it. Examples: * containers like ``dict`` and ``list`` * objects that hold references internally like ``PyTypeObject.tp_subclasses`` * an object's type (held in ``ob_type``) Such held objects are thus implicitly immortal for as long as they are held. In practice, this should have no real consequences since it really isn't a change in behavior. The only difference is that the immortal object (holding the reference) doesn't ever get cleaned up. We do not propose that such implicitly immortal objects be changed in any way. They should not be explicitly marked as immortal just because they are held by an immortal object. That would provide no advantage over doing nothing. Un-Immortalizing Objects ------------------------ This proposal does not include any mechanism for taking an immortal object and returning it to a "normal" condition. Currently there is no need for such an ability. On top of that, the obvious approach is to simply set the refcount to a small value. However, at that point there is no way in knowing which value would be safe. Ideally we'd set it to the value that it would have been if it hadn't been made immortal. However, that value has long been lost. Hence the complexities involved make it less likely that an object could safely be un-immortalized, even if we had a good reason to do so. _Py_IMMORTAL_REFCNT ------------------- We will add two internal constants:: #define _Py_IMMORTAL_BIT (1LL << (8 * sizeof(Py_ssize_t) - 4)) #define _Py_IMMORTAL_REFCNT (_Py_IMMORTAL_BIT + (_Py_IMMORTAL_BIT / 2)) The refcount for immortal objects will be set to ``_Py_IMMORTAL_REFCNT``. However, to check if an object is immortal we will compare its refcount against just the bit:: (op->ob_refcnt & _Py_IMMORTAL_BIT) != 0 The difference means that an immortal object will still be considered immortal, even if somehow its refcount were modified (e.g. by an older stable ABI extension). Note that top two bits of the refcount are already reserved for other uses. That's why we are using the third top-most bit. Affected API ------------ API that will now ignore immortal objects: * (public) ``Py_INCREF()`` * (public) ``Py_DECREF()`` * (public) ``Py_SET_REFCNT()`` * (private) ``_Py_NewReference()`` API that exposes refcounts (unchanged but may now return large values): * (public) ``Py_REFCNT()`` * (public) ``sys.getrefcount()`` (Note that ``_Py_RefTotal`` and ``sys.gettotalrefcount()`` will not be affected.) Immortal Global Objects ----------------------- All objects that we expect to be shared globally (between interpreters) will be made immortal. That includes the following: * singletons (``None``, ``True``, ``False``, ``Ellipsis``, ``NotImplemented``) * all static types (e.g. ``PyLong_Type``, ``PyExc_Exception``) * all static objects in ``_PyRuntimeState.global_objects`` (e.g. identifiers, small ints) All such objects will be immutable. In the case of the static types, they will be effectively immutable. ``PyTypeObject`` has some mutable start (``tp_dict`` and ``tp_subclasses``), but we can work around this by storing that state on ``PyInterpreterState`` instead of on the respective static type object. Then the ``__dict__``, etc. getter will do a lookup on the current interpreter, if appropriate, instead of using ``tp_dict``. Object Cleanup -------------- In order to clean up all immortal objects during runtime finalization, we must keep track of them. For GC objects ("containers") we'll leverage the GC's permanent generation by pushing all immortalized containers there. During runtime shutdown, the strategy will be to first let the runtime try to do its best effort of deallocating these instances normally. Most of the module deallocation will now be handled by ``pylifecycle.c:finalize_modules()`` which cleans up the remaining modules as best as we can. It will change which modules are available during __del__ but that's already defined as undefined behavior by the docs. Optionally, we could do some topological disorder to guarantee that user modules will be deallocated first before the stdlib modules. Finally, anything leftover (if any) can be found through the permanent generation gc list which we can clear after finalize_modules(). For non-container objects, the tracking approach will vary on a case-by-case basis. In nearly every case, each such object is directly accessible on the runtime state, e.g. in a ``_PyRuntimeState`` or ``PyInterpreterState`` field. We may need to add a tracking mechanism to the runtime state for a small number of objects. .. _mitigation: Performance Regression Mitigation --------------------------------- In the interest of clarify, here are some of the ways we are going to try to recover some of the lost `performance <Performance_>`_: * at the end of runtime init, mark all objects as immortal * drop refcount operations in code where we know the object is immortal (e.g. ``Py_RETURN_NONE``) * specialize for immortal objects in the eval loop (see `Pyston`_) Regarding that first point, we can apply the concept from `Immortal Mutable Objects`_ in the pursuit of getting back some of that 4% performance we lose with the naive implementation of immortal objects. At the end of runtime init we can mark *all* objects as immortal and avoid the extra cost in incref/decref. We only need to worry about immutability with objects that we plan on sharing between threads without a GIL. Note that none of this section is part of the proposal. The above is included here for clarity. Possible Changes ---------------- * mark every interned string as immortal * mark the "interned" dict as immortal if shared else share all interned strings * (Larry,MvL) mark all constants unmarshalled for a module as immortal * (Larry,MvL) allocate (immutable) immortal objects in their own memory page(s) Documentation ------------- The immortal objects behavior and API are internal, implementation details and will not be added to the documentation. However, we will update the documentation to make public guarantees about refcount behavior more clear. That includes, specifically: * ``Py_INCREF()`` - change "Increment the reference count for object o." to "Acquire a new reference to object o." * ``Py_DECREF()`` - change "Decrement the reference count for object o." to "Release a reference to object o." * similar for ``Py_XINCREF()``, ``Py_XDECREF()``, ``Py_NewRef()``, ``Py_XNewRef()``, ``Py_Clear()``, ``Py_REFCNT()``, and ``Py_SET_REFCNT()`` We *may* also add a note about immortal objects to the following, to help reduce any surprise users may have with the change: * ``Py_SET_REFCNT()`` (a no-op for immortal objects) * ``Py_REFCNT()`` (value may be surprisingly large) * ``sys.getrefcount()`` (value may be surprisingly large) Other API that might benefit from such notes are currently undocumented. We wouldn't add such a note anywhere else (including for ``Py_INCREF()`` and ``Py_DECREF()``) since the feature is otherwise transparent to users. Reference Implementation ======================== The implementation is proposed on GitHub: https://github.com/python/cpython/pull/19474 Open Issues =========== * is there any other impact on GC? * `are the copy-on-write benefits real? <https://mail.python.org/archives/list/python-dev@python.org/message/J53GY7X…>`__ * must the fate of this PEP be tied to acceptance of a per-interpreter GIL PEP? References ========== .. _Pyston: https://mail.python.org/archives/list/python-dev@python.org/message/TPLEYDC… Prior Art --------- * `Pyston`_ Discussions ----------- This was discussed in December 2021 on python-dev: * https://mail.python.org/archives/list/python-dev@python.org/thread/7O3FUA52… * https://mail.python.org/archives/list/python-dev@python.org/thread/PNLBJBNI… Runtime Object State -------------------- Here is the internal state that the CPython runtime keeps for each Python object: * `PyObject.ob_refcnt`_: the object's `refcount <refcounting_>`_ * `_PyGC_Head <PyGC_Head>`_: (optional) the object's node in a list of `"GC" objects <refcounting_>`_ * `_PyObject_HEAD_EXTRA <PyObject_HEAD_EXTRA>`_: (optional) the object's node in the list of heap objects ``ob_refcnt`` is part of the memory allocated for every object. However, ``_PyObject_HEAD_EXTRA`` is allocated only if CPython was built with ``Py_TRACE_REFS`` defined. ``PyGC_Head`` is allocated only if the object's type has ``Py_TPFLAGS_HAVE_GC`` set. Typically this is only container types (e.g. ``list``). Also note that ``PyObject.ob_refcnt`` and ``_PyObject_HEAD_EXTRA`` are part of ``PyObject_HEAD``. .. _PyObject.ob_refcnt: https://github.com/python/cpython/blob/80a9ba537f1f1666a9e6c5eceef4683f8696… .. _PyGC_Head: https://github.com/python/cpython/blob/80a9ba537f1f1666a9e6c5eceef4683f8696… .. _PyObject_HEAD_EXTRA: https://github.com/python/cpython/blob/80a9ba537f1f1666a9e6c5eceef4683f8696… .. _refcounting: Reference Counting, with Cyclic Garbage Collection -------------------------------------------------- Garbage collection is a memory management feature of some programming languages. It means objects are cleaned up (e.g. memory freed) once they are no longer used. Refcounting is one approach to garbage collection. The language runtime tracks how many references are held to an object. When code takes ownership of a reference to an object or releases it, the runtime is notified and it increments or decrements the refcount accordingly. When the refcount reaches 0, the runtime cleans up the object. With CPython, code must explicitly take or release references using the C-API's ``Py_INCREF()`` and ``Py_DECREF()``. These macros happen to directly modify the object's refcount (unfortunately, since that causes ABI compatibility issues if we want to change our garbage collection scheme). Also, when an object is cleaned up in CPython, it also releases any references (and resources) it owns (before it's memory is freed). Sometimes objects may be involved in reference cycles, e.g. where object A holds a reference to object B and object B holds a reference to object A. Consequently, neither object would ever be cleaned up even if no other references were held (i.e. a memory leak). The most common objects involved in cycles are containers. CPython has dedicated machinery to deal with reference cycles, which we call the "cyclic garbage collector", or often just "garbage collector" or "GC". Don't let the name confuse you. It only deals with breaking reference cycles. See the docs for a more detailed explanation of refcounting and cyclic garbage collection: * https://docs.python.org/3.11/c-api/intro.html#reference-counts * https://docs.python.org/3.11/c-api/refcounting.html * https://docs.python.org/3.11/c-api/typeobj.html#c.PyObject.ob_refcnt * https://docs.python.org/3.11/c-api/gcsupport.html Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.

9 19

Require a C compiler supporting C99 to build Python 3.11
by Victor Stinner 28 Feb '22

28 Feb '22

Hi, I made a change to require C99 <math.h> "NAN" constant and I'm was asked to update the PEP 7 to clarify the C subset is needed to build Python 3.11. Python 3.6 requires a subset of the C99 standard to build defined by the PEP 7: https://www.python.org/dev/peps/pep-0007/ I modified Python 3.11 to require more C99 features of the <math.h> header: * bpo-45440: copysign(), hypot(), isfinite(), isinf(), isnan(), round() * bpo-46640: NAN After my NAN change (bpo-46640), Petr Viktorin asked me to update the PEP 7. I proposed a change to simply say that "Python 3.11 and newer versions use C99": https://github.com/python/peps/pull/2309 I would prefer to not have to give an exhaustive list of C99 features used by CPython, since it's unclear to me what belongs to C99 or to ISO C89. As I wrote before, Python already uses C99 features since Python 3.6. On my PEP PR, Guido van Rossum asked me to escalate the discussion to python-dev, so here I am :-) In "C99", the number "99" refers to the year 1999, the standard is now 23 years old: https://en.wikipedia.org/wiki/C99 In 2022, C99 is now well supported by C compilers supported by Python: GCC, clang, MSVC. I don't know if AIX XLC supports C99. AIX provides a "c99" compiler compatible with C99. It also seems like GCC is usable on AIX. I don't know if ICC supports C99. Python doesn't officially the ICC compiler, the ICC buildbots are gone a few years ago. But sometimes I make some changes to enhance the ICC support, when the change is small enough. Note: Python also uses C11 <stdatomic.h>, but it's not required: there are fallbacks for compilers which don't support it. Victor -- Night gathers, and now my watch begins. It shall not end until my death.

16 45

Need Help
by Prasad, PCRaghavendra 25 Feb '22

25 Feb '22

Hi All, we are using the python 3.9.5 version in our application. In 3.9.5 it is using libexpat 2.2.8 version, as part of the Black duck scan, it is showing critical vulnerabilities in libexpat 2.2.8. (CVE-2022-22824 CVE-2022-23990 CVE-2022-23852 CVE-2022-25236 CVE-2022-22823) when there are any issues ( security issues ) in external modules like OpenSSL, bzip2, and zlib we were able to get the latest code and build as it is straightforward, but libexpat is an internal module to the python and we don't see how we can upgrade libexpat alone in python 3.9.5 So is there a way we can build python (ex 3.9.5) which is already carrying libexpat 2.2.8 so that it will link to the latest libexpat version (2.4.6 - fixed security issues). Another solution when we searched over the net and from the mails what we came to know is we need to wait for Python 3.9.11 where this will be linked to libexpat 2.4.6. Any inputs on this will be helpful. Thanks, Raghu Internal Use - Confidential

4 4

Summary of Python tracker Issues
by Python tracker 25 Feb '22

25 Feb '22

ACTIVITY SUMMARY (2022-02-18 - 2022-02-25) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7220 ( +2) closed 51363 (+64) total 58583 (+66) Open issues with patches: 2939 Issues opened (46) ================== #46793: expose expat XML billion laughs attack mitigation APIs https://bugs.python.org/issue46793 opened by gregory.p.smith #46794: Please update bundled libexpat to 2.4.6 with security fixes (5 https://bugs.python.org/issue46794 opened by sping #46797: ast.Constant.n deprecated without warning https://bugs.python.org/issue46797 opened by jwilk #46798: xml.etree.ElementTree: get() doesn't return default value, alw https://bugs.python.org/issue46798 opened by padremayi #46799: ShareableList memory bloat and performance improvement https://bugs.python.org/issue46799 opened by tcl326 #46803: Item not shown when using mouse wheel to scroll for Listbox/Co https://bugs.python.org/issue46803 opened by Jason990420 #46805: Add low level UDP socket functions to asyncio https://bugs.python.org/issue46805 opened by alex.gronholm #46806: Overlapping PYTHONPATH may cause import already imported modul https://bugs.python.org/issue46806 opened by aklajnert #46808: remove NEXT_BLOCK() from compile.c https://bugs.python.org/issue46808 opened by iritkatriel #46809: copy.deepcopy can fail with unhelpful diagnostics https://bugs.python.org/issue46809 opened by remdragon #46810: multiprocessing.connection.Client doesn't support ipv6 https://bugs.python.org/issue46810 opened by mhupfer #46811: Test suite needs adjustments for Expat >=2.4.5 https://bugs.python.org/issue46811 opened by sping #46812: Thread starvation with threading.Condition https://bugs.python.org/issue46812 opened by msg555 #46814: Documentation for constructing abstract base classes is mislea https://bugs.python.org/issue46814 opened by Yoshanuikabundi #46815: Extra `DeprecationWarning` when running `lib2to3` tests https://bugs.python.org/issue46815 opened by sobolevn #46816: Remove declarations for non-__STDC__ compilers https://bugs.python.org/issue46816 opened by arhadthedev #46817: Add a line-start table to the code object. https://bugs.python.org/issue46817 opened by Mark.Shannon #46823: Add LOAD_FAST__LOAD_ATTR_INSTACE_VALUE combined opcode https://bugs.python.org/issue46823 opened by Dennis Sweeney #46824: use AI_NUMERICHOST | AI_NUMERICSERV to skip getaddrinfo thread https://bugs.python.org/issue46824 opened by graingert #46826: prefixes argument to site.getsitepackages() missing documentat https://bugs.python.org/issue46826 opened by asnell #46828: math.prod can return integers (contradicts doc) https://bugs.python.org/issue46828 opened by neilwebber #46829: Confusing CancelError message if multiple cancellations are sc https://bugs.python.org/issue46829 opened by asvetlov #46831: Outdated comment for __build_class__ in compile.c https://bugs.python.org/issue46831 opened by hauntsaninja #46832: unicodeobject.c doesn't compile when defined EXPERIMENTAL_ISOL https://bugs.python.org/issue46832 opened by moytrage #46833: Installer Wizard is unclear and has redundant settings https://bugs.python.org/issue46833 opened by buhtz #46834: test_gdb started to fail on buildbot/s390x RHEL7 https://bugs.python.org/issue46834 opened by sobolevn #46835: ImportError: bad magic number in ... does not indicate where i https://bugs.python.org/issue46835 opened by hroncok #46836: [C API] Move PyFrameObject to the internal C API https://bugs.python.org/issue46836 opened by vstinner #46838: Parameters and arguments parser syntax error improvments https://bugs.python.org/issue46838 opened by Andy_kl #46840: xmlrpc.client.ServerProxy shows password in __repr__ when usin https://bugs.python.org/issue46840 opened by perrinjerome #46841: Inline bytecode caches https://bugs.python.org/issue46841 opened by brandtbucher #46842: py to pyc location mapping with sys.pycache_prefix isn't 1-to- https://bugs.python.org/issue46842 opened by benrg #46843: PersistentTaskGroup API https://bugs.python.org/issue46843 opened by achimnol #46845: dict: Use smaller entry for Unicode-key only dict. https://bugs.python.org/issue46845 opened by methane #46846: functools.partial objects should set __signature__ and _annota https://bugs.python.org/issue46846 opened by larry #46847: functools.update_wrapper doesn't understand partial objects an https://bugs.python.org/issue46847 opened by larry #46848: Use optimized string search function in mmap.find() https://bugs.python.org/issue46848 opened by rumpelsepp #46849: Memory problems detected using Valgrind https://bugs.python.org/issue46849 opened by sxt1001 #46850: [C API] Move _PyEval_EvalFrameDefault() to the internal C API https://bugs.python.org/issue46850 opened by vstinner #46851: Document multiprocessing.set_forkserver_preload https://bugs.python.org/issue46851 opened by maggyero #46852: Remove the float.__setformat__() method https://bugs.python.org/issue46852 opened by vstinner #46853: Python interpreter can get code from memory, it is not secure. https://bugs.python.org/issue46853 opened by i5-7200u #46854: Failed to compile static python3.7.12 https://bugs.python.org/issue46854 opened by aprpp #46855: printing a string with strange characters loops forever https://bugs.python.org/issue46855 opened by svild #46856: datetime.max conversion https://bugs.python.org/issue46856 opened by joris.geysens #46857: Python leaks one reference at exit on Windows https://bugs.python.org/issue46857 opened by vstinner Most recent 15 issues with no replies (15) ========================================== #46855: printing a string with strange characters loops forever https://bugs.python.org/issue46855 #46854: Failed to compile static python3.7.12 https://bugs.python.org/issue46854 #46853: Python interpreter can get code from memory, it is not secure. https://bugs.python.org/issue46853 #46849: Memory problems detected using Valgrind https://bugs.python.org/issue46849 #46847: functools.update_wrapper doesn't understand partial objects an https://bugs.python.org/issue46847 #46846: functools.partial objects should set __signature__ and _annota https://bugs.python.org/issue46846 #46845: dict: Use smaller entry for Unicode-key only dict. https://bugs.python.org/issue46845 #46842: py to pyc location mapping with sys.pycache_prefix isn't 1-to- https://bugs.python.org/issue46842 #46840: xmlrpc.client.ServerProxy shows password in __repr__ when usin https://bugs.python.org/issue46840 #46832: unicodeobject.c doesn't compile when defined EXPERIMENTAL_ISOL https://bugs.python.org/issue46832 #46831: Outdated comment for __build_class__ in compile.c https://bugs.python.org/issue46831 #46826: prefixes argument to site.getsitepackages() missing documentat https://bugs.python.org/issue46826 #46817: Add a line-start table to the code object. https://bugs.python.org/issue46817 #46816: Remove declarations for non-__STDC__ compilers https://bugs.python.org/issue46816 #46815: Extra `DeprecationWarning` when running `lib2to3` tests https://bugs.python.org/issue46815 Most recent 15 issues waiting for review (15) ============================================= #46857: Python leaks one reference at exit on Windows https://bugs.python.org/issue46857 #46852: Remove the float.__setformat__() method https://bugs.python.org/issue46852 #46850: [C API] Move _PyEval_EvalFrameDefault() to the internal C API https://bugs.python.org/issue46850 #46848: Use optimized string search function in mmap.find() https://bugs.python.org/issue46848 #46845: dict: Use smaller entry for Unicode-key only dict. https://bugs.python.org/issue46845 #46841: Inline bytecode caches https://bugs.python.org/issue46841 #46836: [C API] Move PyFrameObject to the internal C API https://bugs.python.org/issue46836 #46835: ImportError: bad magic number in ... does not indicate where i https://bugs.python.org/issue46835 #46831: Outdated comment for __build_class__ in compile.c https://bugs.python.org/issue46831 #46826: prefixes argument to site.getsitepackages() missing documentat https://bugs.python.org/issue46826 #46824: use AI_NUMERICHOST | AI_NUMERICSERV to skip getaddrinfo thread https://bugs.python.org/issue46824 #46823: Add LOAD_FAST__LOAD_ATTR_INSTACE_VALUE combined opcode https://bugs.python.org/issue46823 #46816: Remove declarations for non-__STDC__ compilers https://bugs.python.org/issue46816 #46815: Extra `DeprecationWarning` when running `lib2to3` tests https://bugs.python.org/issue46815 #46814: Documentation for constructing abstract base classes is mislea https://bugs.python.org/issue46814 Top 10 most discussed issues (10) ================================= #46771: Add some form of cancel scopes https://bugs.python.org/issue46771 30 msgs #46430: intern strings in deepfrozen modules https://bugs.python.org/issue46430 15 msgs #46836: [C API] Move PyFrameObject to the internal C API https://bugs.python.org/issue46836 15 msgs #46843: PersistentTaskGroup API https://bugs.python.org/issue46843 14 msgs #40421: [C API] Add getter functions for PyFrameObject and maybe move https://bugs.python.org/issue40421 11 msgs #46852: Remove the float.__setformat__() method https://bugs.python.org/issue46852 10 msgs #46794: Please update bundled libexpat to 2.4.6 with security fixes (5 https://bugs.python.org/issue46794 9 msgs #46761: functools.update_wrapper breaks the signature of functools.par https://bugs.python.org/issue46761 7 msgs #45100: Improve help() by making typing.overload() information accessi https://bugs.python.org/issue45100 6 msgs #45316: [C API] Functions not exported with PyAPI_FUNC() https://bugs.python.org/issue45316 6 msgs Issues closed (59) ================== #26897: [doc] Clarify Popen stdin, stdout, stderr https://bugs.python.org/issue26897 closed by Jelle Zijlstra #34429: [doc] On Windows tempfile.TemporaryFile behaves like NamedTemp https://bugs.python.org/issue34429 closed by iritkatriel #36557: [doc] Clarify the meaning of /uninstall in windows cli https://bugs.python.org/issue36557 closed by iritkatriel #39327: shutil.rmtree using vagrant synched folder fails https://bugs.python.org/issue39327 closed by serhiy.storchaka #42766: urllib.request.HTTPPasswordMgr uses commonprefix instead of co https://bugs.python.org/issue42766 closed by serhiy.storchaka #43119: asyncio.Queue.put never yields if the queue is unbounded https://bugs.python.org/issue43119 closed by asvetlov #43311: bpo-43311: PyInterpreterState_New use thread-specific data tst https://bugs.python.org/issue43311 closed by vstinner #43364: Windows: Make UTF-8 mode more accessible https://bugs.python.org/issue43364 closed by methane #44011: Borrow asyncio ssl implementation from uvloop https://bugs.python.org/issue44011 closed by asvetlov #44886: asyncio: create_datagram_endpoint() does not return a Datagram https://bugs.python.org/issue44886 closed by asvetlov #45146: Add a possibility for asyncio.Condition to determine the count https://bugs.python.org/issue45146 closed by asvetlov #45390: asyncio.Task doesn't propagate CancelledError() exception corr https://bugs.python.org/issue45390 closed by asvetlov #45883: reuse_address mistakenly removed from loop.create_server https://bugs.python.org/issue45883 closed by asvetlov #45898: ctypes cfield.c defines duplicate ffi_type_* symbols https://bugs.python.org/issue45898 closed by christian.heimes #46252: Forbid passing SSLSocket into asyncio methods https://bugs.python.org/issue46252 closed by asvetlov #46356: [C API] Enforce usage of PyFrame_GetBack() https://bugs.python.org/issue46356 closed by vstinner #46400: Please update bundled libexpat to 2.4.4 with security fixes (C https://bugs.python.org/issue46400 closed by ned.deily #46432: AMD64 FreeBSD Shared 3.x buildbot fails to build: error: error https://bugs.python.org/issue46432 closed by vstinner #46522: concurrent.futures.__getattr__ raises the wrong AttributeError https://bugs.python.org/issue46522 closed by asvetlov #46571: Strange `(a)typing.no_type_check` behavior for class variables https://bugs.python.org/issue46571 closed by Jelle Zijlstra #46603: `typing._strip_annotations` is not fully covered https://bugs.python.org/issue46603 closed by Jelle Zijlstra #46606: Large C stack usage of os.getgroups() and os.setgroups() https://bugs.python.org/issue46606 closed by methane #46623: test_zlib: test_pair() and test_speech128() fail with s390x ha https://bugs.python.org/issue46623 closed by vstinner #46656: Remove the Py_NO_NAN macro: require NAN to build Python 3.11 https://bugs.python.org/issue46656 closed by vstinner #46659: Deprecate locale.getdefaultlocale() function https://bugs.python.org/issue46659 closed by vstinner #46663: test_math test_cmath test_complex fails on Fedora Rawhide buil https://bugs.python.org/issue46663 closed by vstinner #46668: encodings: the "mbcs" alias doesn't work https://bugs.python.org/issue46668 closed by vstinner #46672: NameError in asyncio.gather when passing a invalid type as an https://bugs.python.org/issue46672 closed by asvetlov #46685: Add additional tests for new features in `typing.py` https://bugs.python.org/issue46685 closed by Jelle Zijlstra #46702: Specialize UNPACK_SEQUENCE https://bugs.python.org/issue46702 closed by brandtbucher #46717: Raising exception multiple times leaks memory https://bugs.python.org/issue46717 closed by iritkatriel #46725: Unpacking without parentheses is allowed since 3.9 https://bugs.python.org/issue46725 closed by pablogsal #46729: Better str() for BaseExceptionGroup https://bugs.python.org/issue46729 closed by iritkatriel #46732: Builtin __bool__ docstrings are wrong https://bugs.python.org/issue46732 closed by Jelle Zijlstra #46736: Generate HTML 5 with SimpleHTTPRequestHandler.list_directory https://bugs.python.org/issue46736 closed by benjamin.peterson #46757: dataclasses should define an empty __post_init__ https://bugs.python.org/issue46757 closed by eric.smith #46765: Replace Locally Cached Strings with Statically Initialized Obj https://bugs.python.org/issue46765 closed by eric.snow #46777: Fix incorrect use of directives in asyncio documentation https://bugs.python.org/issue46777 closed by serhiy.storchaka #46784: Duplicated symbols when linking embedded Python with libexpat https://bugs.python.org/issue46784 closed by gregory.p.smith #46792: Indentation not preserved with ruamel.yaml.round_trip_dump https://bugs.python.org/issue46792 closed by zach.ware #46795: Why Does 3rd/Python39/Lib/site-packages/psycopg2/_psycopg.cp39 https://bugs.python.org/issue46795 closed by zach.ware #46796: Simplify handling of removed parameter "loop" in asyncio https://bugs.python.org/issue46796 closed by asvetlov #46800: Support for pause(2) https://bugs.python.org/issue46800 closed by zach.ware #46801: test_typing emits deprecation warnings https://bugs.python.org/issue46801 closed by iritkatriel #46802: Wrong result unpacking binary data with ctypes bitfield. https://bugs.python.org/issue46802 closed by helo9 #46804: spam https://bugs.python.org/issue46804 closed by serhiy.storchaka #46807: Wrong class __annotations__ when field name and type are equal https://bugs.python.org/issue46807 closed by serhiy.storchaka #46813: Allow developer to resize the dictionary https://bugs.python.org/issue46813 closed by rhettinger #46818: Proper way to inherit from collections.abc.Coroutine https://bugs.python.org/issue46818 closed by asvetlov #46819: Add an Error / Exception / Warning when contextlib.suppress() https://bugs.python.org/issue46819 closed by serhiy.storchaka #46820: SyntaxError on `1not in...` https://bugs.python.org/issue46820 closed by serhiy.storchaka #46821: Introspection support for typing.overload https://bugs.python.org/issue46821 closed by Jelle Zijlstra #46822: test_create_server_ssl_over_ssl times out on some systems https://bugs.python.org/issue46822 closed by steve.dower #46825: slow matching on regular expression https://bugs.python.org/issue46825 closed by serhiy.storchaka #46827: asyncio SelectorEventLoop.sock_connect fails with a UDP socket https://bugs.python.org/issue46827 closed by asvetlov #46830: Add Find functionality to Squeezed Text viewer https://bugs.python.org/issue46830 closed by terry.reedy #46837: lstrip and strip not working as expected https://bugs.python.org/issue46837 closed by jigar030 #46839: Process finished with exit code -1073741819 (0xC0000005) https://bugs.python.org/issue46839 closed by zach.ware #46844: Implicit binding of PersistentTaskGroup (or virtual event loop https://bugs.python.org/issue46844 closed by gvanrossum

1 0

Embedding multiple Python runtimes in the same process on Windows
by Tony Roberts 24 Feb '22

24 Feb '22

Hi all, This is specifically about embedding Python on Windows, and I'm hoping some of the Windows Python devs might have some ideas or be interested in this. I have implemented a partial solution (linked below) and I'm interested to hear what other people think of this. Currently when embedding Python both python3x.dll and python3.dll get loaded, with python3.dll redirecting stable API calls to python3x.dll. If a process embeds two different versions of Python 3, e.g., python39.dll and python310.dll they both load their respective python3.dlls. At this point I imagine you are thinking "don't do that", but bear with me... The problem comes when the Python loaded second imports an extension module linked with python3.dll. The python3.dll that it gets linked to will be the first one that was loaded, not the one that relates to the second Python that is actually doing the import. This results in a call into the wrong Python dll and 'bad things' happen. None of this is unexpected and I'm sure that the sensible thing to do is to simply not do this... but I've been working on a way to make this work anyway. In my case I have a plugin to another application that embeds Python into that application. It's perfectly possible (and reasonable) for other plugins to also want to embed Python. This can be dealt with by having both plugins use the same Python environment easily enough in most cases. With Python 2 it used to be possible to have two different Python interpreters embedded at the same time and not have them interfere with each other (although it is possible there would still be issues with DLL versions used by extension modules). With Python 3 if we want to have two different versions of Python 3 embedded at the same time then it will fail because of the reason outlined above. My idea is to redirect all loaded python3.dll dlls to the one we want to be used before loading any extension modules (i.e., just before any call to LoadLibraryEx) and then restore them afterwards. This can be done by manipulating the loader modules list in Windows. I have implemented this as a proof of concept in my own plugin and confirmed that this works and does allow two different versions of Python 3 to be embedded at the same time. This works with an unmodified version of Python by applying the redirect in an import hook. It uses several undocumented Windows structures and APIs in order to safely manipulate the loader table. Here is my proof of concept code that performs the redirect https://gist.github.com/tonyroberts/74888762f0063238d4f7fd7c7d36f0f0 While this works for different versions of Python 3, there is still a problem when trying to embed two different instances of the same version of Python 3. The problem is basically the same but with the added complication that the pyd files are named the same, and the ones loaded first get found by the second Python runtime and you end up again calling across versions. I managed to solve this using a similar method to the code above, but rather than redirecting just python3.dll I look for any other loaded python3x.dll and then remove *all* modules loaded under that Python distribution from the loader table. This ensures that both Python runtimes are effectively isolated from each other as neither see any of the same modules. This gets more complicated once you start thinking about user site-packages folders and venvs, but for simple distributions where everything is under the same root folder this technique works. Anyway, just keen to hear what people think or whether this has been tackled before in another way. Best regards, Tony

1 0

Version 2 of PEP 670 – Convert macros to functions in the Python C API
by Victor Stinner 24 Feb '22

24 Feb '22

Hi, Since Erlend and me posted PEP 670 on python-dev last October, we took all feedback (python-dev and Steering Council) in account to clarify the intent of the PEP and to reduce its scope (remove *any* risk of backward compatibility). PEP 670: https://python.github.io/peps/pep-0670/ Changes since the first version: * Stricter policy on not changing argument types and return type. * Better explain why pointer arguments require a cast to not emit new compiler warnings. * Macros which can be used as l-values are no longer modified by the PEP (I wrote PEP 674 just for that). * Macros having multiple return types are no longer modified by the PEP. * Limited C API version 3.11 no longer casts pointer arguments (move towards a cleaner API, but only if you opt in for that!). * No longer remove return values of macros "which should not have a return value" (this rule was complicated and in fact, all macros have already been fixed). * Add "Macros converted to functions since Python 3.8" section to show that macros already converted didn't introduce performance issue nor new compiler warnings. * Add "Benchmark comparing macros and static inline functions" section to prove that there is no significant impact on performance. Note: we already submitted PEP 670 to the Steering Council at the end of November, and the SC 2022 came with feedback, so we modified the PEP: https://mail.python.org/archives/list/python-dev@python.org/message/IJ3IBVY… Following the SC decision, we already modified PEP 7 to add: "static inline functions should be preferred over macros in new code." https://www.python.org/dev/peps/pep-0007/#c-dialect Victor -- Night gathers, and now my watch begins. It shall not end until my death.

2 4

SC reply to PEP 674 -- Disallow using macros as l-values
by Petr Viktorin 23 Feb '22

23 Feb '22

Hello, Thank you for submitting PEP 674 -- Disallow using macros as l-values! For Py_TYPE, we respect the previous steering council's decision: https://github.com/python/steering-council/issues/79 The Py_TYPE macro can be changed as proposed in 3.11. However, the old SC explicitly noted that the exception only applies to Py_SET_TYPE, and that in general, there should be a deprecation notice in some way or form. After deliberating the proposed changes, the SC agrees that they will make CPython better. However, we are not convinced that they should be made *right now*. We see no reason to rush. We would like to ask you to rework the PEP to reflect these suggestions. It is not feasible to provide visible deprecation warnings for the proposed changes. We feel that this is not a good reason to skip the deprecation process entirely. Quite the opposite: if we need to skip a step, we should move even more carefully than usual. We suggest the following process: First, add notices to any documentation that using the macro as a l-value is a deprecated, CPython-specific detail. It should be clear that the only reason this usage is allowed is for backwards compatibility, and that alternate implementations of the C API are free to not allow this. Then, wait until all supported versions of CPython include any relevant replacement macros such as Py_SET_SIZE. (This doesn't mean adding "set" variants to all the changed macros -- just to those few where setting can be useful and safe.) With the current schedule, this means waiting until Python 3.14, which is planned for October 2024, when Python 3.8 will reach end-of-life. The SC can give an exception to shorten the wait for specific macros, if there is a real benefit to it. (For example: the nogil branch should include any necessary changes, and they should be approved and merged with nogil itself). Finally, ask the new Steering Council to make the change and finally disallow using the macros as l-values. If there's no negative feedback or unexpected downsides, we hope that the future SC will summarily approve. Hopefully, this strategy will also work for PyDescr_NAME and other macros that were left out of the PEP for now. Additionally, consider making the change in the 3.11 version of the limited API. This version is opt-in, so users are free to do the switch when it is convenient for them. Also, it would allow early testing and review of the changes. Further discussion about this should happen on python-dev, like usual. - Petr, on behalf of the Steering Council

2 3

PEP 683: "Immortal Objects, Using a Fixed Refcount"
by Eric Snow 23 Feb '22

23 Feb '22

Eddie and I would appreciate your feedback on this proposal to support treating some objects as "immortal". The fundamental characteristic of the approach is that we would provide stronger guarantees about immutability for some objects. A few things to note: * this is essentially an internal-only change: there are no user-facing changes (aside from affecting any 3rd party code that directly relies on specific refcounts) * the naive implementation shows a 4% slowdown * we have a number of strategies that should reduce that penalty * without immortal objects, the implementation for per-interpreter GIL will require a number of non-trivial workarounds That last one is particularly meaningful to me since it means we would definitely miss the 3.11 feature freeze. With immortal objects, 3.11 would still be in reach. -eric ----------------------- PEP: 683 Title: Immortal Objects, Using a Fixed Refcount Author: Eric Snow <ericsnowcurrently(a)gmail.com>, Eddie Elizondo <eduardo.elizondorueda(a)gmail.com> Discussions-To: python-dev(a)python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 10-Feb-2022 Python-Version: 3.11 Post-History: Resolution: Abstract ======== Under this proposal, any object may be marked as immortal. "Immortal" means the object will never be cleaned up (at least until runtime finalization). Specifically, the `refcount`_ for an immortal object is set to a sentinel value, and that refcount is never changed by ``Py_INCREF()``, ``Py_DECREF()``, or ``Py_SET_REFCNT()``. For immortal containers, the ``PyGC_Head`` is never changed by the garbage collector. Avoiding changes to the refcount is an essential part of this proposal. For what we call "immutable" objects, it makes them truly immutable. As described further below, this allows us to avoid performance penalties in scenarios that would otherwise be prohibitive. This proposal is CPython-specific and, effectively, describes internal implementation details. .. _refcount: https://docs.python.org/3.11/c-api/intro.html#reference-counts Motivation ========== Without immortal objects, all objects are effectively mutable. That includes "immutable" objects like ``None`` and ``str`` instances. This is because every object's refcount is frequently modified as it is used during execution. In addition, for containers the runtime may modify the object's ``PyGC_Head``. These runtime-internal state currently prevent full immutability. This has a concrete impact on active projects in the Python community. Below we describe several ways in which refcount modification has a real negative effect on those projects. None of that would happen for objects that are truly immutable. Reducing Cache Invalidation --------------------------- Every modification of a refcount causes the corresponding cache line to be invalidated. This has a number of effects. For one, the write must be propagated to other cache levels and to main memory. This has small effect on all Python programs. Immortal objects would provide a slight relief in that regard. On top of that, multi-core applications pay a price. If two threads are interacting with the same object (e.g. ``None``) then they will end up invalidating each other's caches with each incref and decref. This is true even for otherwise immutable objects like ``True``, ``0``, and ``str`` instances. This is also true even with the GIL, though the impact is smaller. Avoiding Data Races ------------------- Speaking of multi-core, we are considering making the GIL a per-interpreter lock, which would enable true multi-core parallelism. Among other things, the GIL currently protects against races between multiple threads that concurrently incref or decref. Without a shared GIL, two running interpreters could not safely share any objects, even otherwise immutable ones like ``None``. This means that, to have a per-interpreter GIL, each interpreter must have its own copy of *every* object, including the singletons and static types. We have a viable strategy for that but it will require a meaningful amount of extra effort and extra complexity. The alternative is to ensure that all shared objects are truly immutable. There would be no races because there would be no modification. This is something that the immortality proposed here would enable for otherwise immutable objects. With immortal objects, support for a per-interpreter GIL becomes much simpler. Avoiding Copy-on-Write ---------------------- For some applications it makes sense to get the application into a desired initial state and then fork the process for each worker. This can result in a large performance improvement, especially memory usage. Several enterprise Python users (e.g. Instagram, YouTube) have taken advantage of this. However, the above refcount semantics drastically reduce the benefits and has led to some sub-optimal workarounds. Also note that "fork" isn't the only operating system mechanism that uses copy-on-write semantics. Rationale ========= The proposed solution is obvious enough that two people came to the same conclusion (and implementation, more or less) independently. Other designs were also considered. Several possibilities have also been discussed on python-dev in past years. Alternatives include: * use a high bit to mark "immortal" but do not change ``Py_INCREF()`` * add an explicit flag to objects * implement via the type (``tp_dealloc()`` is a no-op) * track via the object's type object * track with a separate table Each of the above makes objects immortal, but none of them address the performance penalties from refcount modification described above. In the case of per-interpreter GIL, the only realistic alternative is to move all global objects into ``PyInterpreterState`` and add one or more lookup functions to access them. Then we'd have to add some hacks to the C-API to preserve compatibility for the may objects exposed there. The story is much, much simpler with immortal objects Impact ====== Benefits -------- Most notably, the cases described in the two examples above stand to benefit greatly from immortal objects. Projects using pre-fork can drop their workarounds. For the per-interpreter GIL project, immortal objects greatly simplifies the solution for existing static types, as well as objects exposed by the public C-API. In general, a strong immutability guarantee for objects enables Python applications to scale like never before. This is because they can then leverage multi-core parallelism without a tradeoff in memory usage. This is reflected in most of the above cases. Performance ----------- A naive implementation shows `a 4% slowdown`_. Several promising mitigation strategies will be pursued in the effort to bring it closer to performance-neutral. On the positive side, immortal objects save a significant amount of memory when used with a pre-fork model. Also, immortal objects provide opportunities for specialization in the eval loop that would improve performance. .. _a 4% slowdown: https://github.com/python/cpython/pull/19474#issuecomment-1032944709 Backward Compatibility ----------------------- This proposal is completely compatible. It is internal-only so no API is changing. The approach is also compatible with extensions compiled to the stable ABI. Unfortunately, they will modify the refcount and invalidate all the performance benefits of immortal objects. However, the high bit of the refcount will still match ``_Py_IMMORTAL_REFCNT`` so we can still identify such objects as immortal. No user-facing behavior changes, with the following exceptions: * code that inspects the refcount (e.g. ``sys.getrefcount()`` or directly via ``ob_refcnt``) will see a really, really large value * ``Py_SET_REFCNT()`` will be a no-op for immortal objects Neither should cause a problem. Alternate Python Implementations -------------------------------- This proposal is CPython-specific. Security Implications --------------------- This feature has no known impact on security. Maintainability --------------- This is not a complex feature so it should not cause much mental overhead for maintainers. The basic implementation doesn't touch much code so it should have much impact on maintainability. There may be some extra complexity due to performance penalty mitigation. However, that should be limited to where we immortalize all objects post-init and that code will be in one place. Non-Obvious Consequences ------------------------ * immortal containers effectively immortalize each contained item * the same is true for objects held internally by other objects (e.g. ``PyTypeObject.tp_subclasses``) * an immortal object's type is effectively immortal * though extremely unlikely (and technically hard), any object could be incref'ed enough to reach ``_Py_IMMORTAL_REFCNT`` and then be treated as immortal Specification ============= The approach involves these fundamental changes: * add ``_Py_IMMORTAL_REFCNT`` (the magic value) to the internal C-API * update ``Py_INCREF()`` and ``Py_DECREF()`` to no-op for objects with the magic refcount (or its most significant bit) * do the same for any other API that modifies the refcount * stop modifying ``PyGC_Head`` for immortal containers * ensure that all immortal objects are cleaned up during runtime finalization Then setting any object's refcount to ``_Py_IMMORTAL_REFCNT`` makes it immortal. To be clear, we will likely use the most-significant bit of ``_Py_IMMORTAL_REFCNT`` to tell if an object is immortal, rather than comparing with ``_Py_IMMORTAL_REFCNT`` directly. (There are other minor, internal changes which are not described here.) This is not meant to be a public feature but rather an internal one. So the proposal does *not* including adding any new public C-API, nor any Python API. However, this does not prevent us from adding (publicly accessible) private API to do things like immortalize an object or tell if one is immortal. Affected API ------------ API that will now ignore immortal objects: * (public) ``Py_INCREF()`` * (public) ``Py_DECREF()`` * (public) ``Py_SET_REFCNT()`` * (private) ``_Py_NewReference()`` API that exposes refcounts (unchanged but may now return large values): * (public) ``Py_REFCNT()`` * (public) ``sys.getrefcount()`` (Note that ``_Py_RefTotal`` and ``sys.gettotalrefcount()`` will not be affected.) Immortal Global Objects ----------------------- The following objects will be made immortal: * singletons (``None``, ``True``, ``False``, ``Ellipsis``, ``NotImplemented``) * all static types (e.g. ``PyLong_Type``, ``PyExc_Exception``) * all static objects in ``_PyRuntimeState.global_objects`` (e.g. identifiers, small ints) There will likely be others we have not enumerated here. Object Cleanup -------------- In order to clean up all immortal objects during runtime finalization, we must keep track of them. For container objects we'll leverage the GC's permanent generation by pushing all immortalized containers there. During runtime shutdown, the strategy will be to first let the runtime try to do its best effort of deallocating these instances normally. Most of the module deallocation will now be handled by pylifecycle.c:finalize_modules which cleans up the remaining modules as best as we can. It will change which modules are available during __del__ but that's already defined as undefined behavior by the docs. Optionally, we could do some topological disorder to guarantee that user modules will be deallocated first before the stdlib modules. Finally, anything leftover (if any) can be found through the permanent generation gc list which we can clear after finalize_modules. For non-container objects, the tracking approach will vary on a case-by-case basis. In nearly every case, each such object is directly accessible on the runtime state, e.g. in a ``_PyRuntimeState`` or ``PyInterpreterState`` field. We may need to add a tracking mechanism to the runtime state for a small number of objects. Documentation ------------- The feature itself is internal and will not be added to the documentation. We *may* add a note about immortal objects to the following, to help reduce any surprise users may have with the change: * ``Py_SET_REFCNT()`` (a no-op for immortal objects) * ``Py_REFCNT()`` (value may be surprisingly large) * ``sys.getrefcount()`` (value may be surprisingly large) Other API that might benefit from such notes are currently undocumented. We wouldn't add a note anywhere else (including for ``Py_INCREF()`` and ``Py_DECREF()``) since the feature is otherwise transparent to users. Rejected Ideas ============== Equate Immortal with Immutable ------------------------------ Making a mutable object immortal isn't particularly helpful. The exception is if you can ensure the object isn't actually modified again. Since we aren't enforcing any immutability for immortal objects it didn't make sense to emphasis that relationship. Reference Implementation ======================== The implementation is proposed on GitHub: https://github.com/python/cpython/pull/19474 Open Issues =========== * is there any other impact on GC? References ========== This was discussed in December 2021 on python-dev: * https://mail.python.org/archives/list/python-dev@python.org/thread/7O3FUA52… * https://mail.python.org/archives/list/python-dev@python.org/thread/PNLBJBNI… Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.

11 27

PyCon 2022: Call For Papers!
by Grete Kungla 22 Feb '22

22 Feb '22

Hi, I am working with the PyCon Estonia team, helping them organise a conference this year. The last three PyCon events were a great success and after a year of break due to corona, we want to come back to keep on bringing the Python community together in Tallinn. As we are preparing for PyCon Estonia 2020, we have also announced our call for papers. We are looking for talks on various topics. The theme this year is "Connect. Collab. Contribute" and we want the focus of the conference to be around the Python ecosystem. However, we are also welcoming talks on other relevant topics - Python challenges, outstanding packages, security, automation, AI/ML, web development, IoT, mental health for developers, conducive work environment for coding, etc. I would like to invite you to participate in the call for papers and be a part of PyCon Estonia 2022 as a speaker. We are waiting for the proposals before the 1st of April here: https://pyconestonia.typeform.com/to/HSWhN8YW We would have a team from our end review the proposals anonymously and reach out to you if the talk fits with the overall theme. As a speaker, you do not need to purchase the tickets for the event and if you already have, we'll be happy to make a refund or transfer. If you have any questions please do not hesitate to reach out to me. Looking forward to having you with us at PyCon on 25th of August, 2022. Best Wishes, -- Grete Kungla +372 5628 0032 Project Manager PyCon Conference 2022 https://pycon.ee/

1 0

SC Acceptance: PEP 680 -- tomllib
by Petr Viktorin 22 Feb '22

22 Feb '22

Hello, The Steering Council discussed PEP 680 -- tomllib: Support for Parsing TOML in the Standard Library, and unanimously decided to accept it. Congratulations! I'll change the PEP status to Accepted. Please open a PR for Python 3.11 at your convenience. - Petr (on behalf of the Python Steering Council)

1 0

Steering Council reply to PEP 670 -- Convert macros to functions in the Python C API
by Petr Viktorin 21 Feb '22

21 Feb '22

Thank you for submitting PEP 670 (Convert macros to functions in the Python C API)! The steering council is still discussing it. We're not ready to accept the PEP as a whole, but there are parts on which we agree unanimously. To unblock the work, we're making the following pronouncements: *All other things being equal, static inline functions are better than macros.* Specifically, inline static functions should be preferred over function-like macros in new code. Please add a note about this to PEP 7. *The performance impact of changing macros to static inline functions is negligible.* The performance changes are indistinguishable from noise caused by other factors, such as layout changes and specific compiler optimizations. It's possible that this isn't true in all cases. If any are found, they should be kept as macros and added to the benchmark suite to avoid regressions. More generally, if there is a non-obvious reason for a conscious design decision, it should be explained (e.g. in a comment) to help future developers. *When there is no backwards compatibility issue, macros can be changed to static inline functions.* In effect, the SC accepts the first part of the PEP, except cases where: a macro is converted to macro and function ("Cast to PyObject*"), and where the return value is removed. Please do the changes in several smaller pull requests, don't skip reviews, and be prepared to revert if any issues are found. The straightforward macros should be converted first. When they are, we hope that it will become easier to evaluate the remaining ones. - Petr, on behalf of the Steering Council

7 11

Move the pythoncapi_compat project under the GitHub Python or PSF organization?
by Victor Stinner 21 Feb '22

21 Feb '22

Hi, Would it make sense to move the pythoncapi_compat project under the GitHub Python or PSF organization to make it more "official" and a little bit more sustainable? "The pythoncapi_compat project can be used to write a C extension supporting a wide range of Python versions with a single code base. It is made of the pythoncapi_compat.h header file and the upgrade_pythoncapi.py script." Documentation: https://pythoncapi-compat.readthedocs.io/en/latest/ GitHub: https://github.com/pythoncapi/pythoncapi_compat In the past, I managed to move my personal pyperf project under the PSF organization. Now other core developers are contributing and using it. It's better than having it as a personal project. pythoncapi_compat respects the PSF requirements to move a project in the GitHub PSF organization: contributors are required to respect the PSF Code of Conduct and the project has 3 maintainers (Dong-hee Na, Erlend E. AAsland and me). --- Some context. Incompatible C API changes in Python 3.10 and 3.11 require adding compatibility code to write C code compatible with old and new Python versions. For example, What's New in Python 3.10 suggests adding the following code to your project to get the Py_SET_REFCNT() function on Python 3.9 and older: #if PY_VERSION_HEX < 0x030900A4 # define Py_SET_REFCNT(obj, refcnt) ((Py_REFCNT(obj) = (refcnt)), (void)0) #endif Python 3.11 requires even more compatibility functions (7) and the implementation of these functions take more lines of code: * Py_SET_TYPE() * Py_SET_SIZE() * PyFrame_GetCode() * PyFrame_GetBack() * PyThreadState_GetFrame() * PyThreadState_EnterTracing() * PyThreadState_LeaveTracing() I added a reference to the pythoncapi_compat project to What's New in Python 3.11: "Or use the pythoncapi_compat project to get these functions on old Python functions." https://docs.python.org/dev/whatsnew/3.11.html#c-api-changes The project provides an upgrade_pythoncapi.py script to automatically add Python 3.10 and 3.11 support without losing support with Python 2.7. Just run the script, maybe copy pythoncapi_compat.h, and you're done! :-) It also provides a "pythoncapi_compat.h" header file which contains all the required compatibility code. Just copy the header file and use >#include "pythoncapi_compat.h"< in your project. You don't have to maintain this compatibilty code yourself anymore (pythoncapi_compat.h is made of 400 lines of C code). pythoncapi_compat supports Python (2.7 to 3.11), PyPy (2.7, 3.6, 3.7) and C++. It is distributed under the MIT license. My concern is that currently the project lives in my https://github.com/pythoncapi organization which is not really a sustainable option for the future. --- The pythoncapi_compat project is used by more and more Python projects, like Mercurial or mypy: https://pythoncapi-compat.readthedocs.io/en/latest/users.html I already made a similar request in June 2021 (move the project under the PSF organization): https://mail.python.org/archives/list/python-dev@python.org/thread/KHDZGCNO… What changed since June 2021? Python 3.11 now requires way more compatibility code than Python 3.10 does: 7 functions instead of 1. On Discord, I was asked to ask the Steering Council. I asked the SC who asked me to ask on python-dev, so here I am :-) Victor -- Night gathers, and now my watch begins. It shall not end until my death.

7 15

Github Issues Migration is coming soon
by Łukasz Langa 18 Feb '22

18 Feb '22

I just published the migration plan and a call to action on Discourse: https://discuss.python.org/t/github-issues-migration-is-coming-soon/13791 <https://discuss.python.org/t/github-issues-migration-is-coming-soon/13791> -- See you there, Łukasz Langa CPython Developer in Residence Python Software Foundation

1 0

Summary of Python tracker Issues
by Python tracker 18 Feb '22

18 Feb '22

ACTIVITY SUMMARY (2022-02-11 - 2022-02-18) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7218 (+40) closed 51299 (+27) total 58517 (+67) Open issues with patches: 2932 Issues opened (52) ================== #46725: Unpacking without parentheses is allowed since 3.9 https://bugs.python.org/issue46725 opened by pablogsal #46726: Thread spuriously marked dead after interrupting a join call https://bugs.python.org/issue46726 opened by Kevin Shweh #46727: Should shutil functions support bytes paths? https://bugs.python.org/issue46727 opened by Jelle Zijlstra #46729: Better str() for BaseExceptionGroup https://bugs.python.org/issue46729 opened by iritkatriel #46731: posix._fcopyfile flags addition https://bugs.python.org/issue46731 opened by devnexen #46732: object.__bool__ docstring is wrong https://bugs.python.org/issue46732 opened by Jelle Zijlstra #46733: pathlib.Path methods can raise NotImplementedError https://bugs.python.org/issue46733 opened by barneygale #46734: Add Maildir.get_flags() to access message flags without openin https://bugs.python.org/issue46734 opened by gildea #46735: gettext.translations crashes when locale is unset https://bugs.python.org/issue46735 opened by amazingminecrafter2015 #46736: Generate HTML 5 with SimpleHTTPRequestHandler.list_directory https://bugs.python.org/issue46736 opened by dom1310df #46740: Improve Telnetlib's throughput https://bugs.python.org/issue46740 opened by martin_kirch #46742: Add '-d $fd' option to trace module, akin to bash -x feature https://bugs.python.org/issue46742 opened by PenelopeFudd #46743: Enable usage of object.__orig_class__ in __init__ https://bugs.python.org/issue46743 opened by Gobot1234 #46744: installers on ARM64 suggest wrong folders https://bugs.python.org/issue46744 opened by conio #46746: IDLE: Consistently handle non .py source files https://bugs.python.org/issue46746 opened by terry.reedy #46748: Python.h includes stdbool.h https://bugs.python.org/issue46748 opened by petr.viktorin #46749: Support cross compilation on macOS https://bugs.python.org/issue46749 opened by autoantwort #46750: some code paths in ssl and _socket still import idna unconditi https://bugs.python.org/issue46750 opened by slingamn #46751: Windows-style path is not recognized under cygwin https://bugs.python.org/issue46751 opened by mikekaganski #46752: Introduce task groups to asyncio and change task cancellation https://bugs.python.org/issue46752 opened by gvanrossum #46753: Statically allocate and initialize the empty tuple. https://bugs.python.org/issue46753 opened by eric.snow #46754: Improve Python Language Reference based on [K��hl 2020] https://bugs.python.org/issue46754 opened by gvanrossum #46755: QueueHandler logs stack_info twice https://bugs.python.org/issue46755 opened by erik.montnemery #46756: Incorrect authorization check in urllib.request https://bugs.python.org/issue46756 opened by serhiy.storchaka #46757: dataclasses should define an empty __post_init__ https://bugs.python.org/issue46757 opened by NeilGirdhar #46758: Incorrect behaviour creating a Structure with ctypes.c_bool bi https://bugs.python.org/issue46758 opened by dudenwatschn #46759: sys.excepthook documentation doesn't mention that it isn't cal https://bugs.python.org/issue46759 opened by cjwatson #46760: test_dis should test the dis module, not everything else https://bugs.python.org/issue46760 opened by Mark.Shannon #46761: functools.update_wrapper breaks the signature of functools.par https://bugs.python.org/issue46761 opened by larry #46763: os.path.samefile incorrect results for shadow copies https://bugs.python.org/issue46763 opened by nijave #46764: Wrapping a bound method with a @classmethod no longer works https://bugs.python.org/issue46764 opened by msullivan #46765: Replace Locally Cached Strings with Statically Initialized Obj https://bugs.python.org/issue46765 opened by eric.snow #46767: [Doc] sqlite3 Cursor.execute() return value is unspecified https://bugs.python.org/issue46767 opened by kephas #46769: Improve documentation for `typing.TypeVar` https://bugs.python.org/issue46769 opened by AlexWaygood #46770: ConfigParser(dict_type=) not behaving as expected https://bugs.python.org/issue46770 opened by malonn #46771: Add some form of cancel scopes https://bugs.python.org/issue46771 opened by gvanrossum #46772: Statically Initialize PyArg_Parser in clinic.py https://bugs.python.org/issue46772 opened by eric.snow #46773: Add a Private API for Looking Up Global Objects https://bugs.python.org/issue46773 opened by eric.snow #46774: Importlib.metadata.version picks first distribution not latest https://bugs.python.org/issue46774 opened by kkirsche-github #46775: [Windows] OSError should unconditionally call winerror_to_errn https://bugs.python.org/issue46775 opened by eryksun #46777: Fix incorrect use of directives in asyncio documentation https://bugs.python.org/issue46777 opened by serhiy.storchaka #46779: Add ssl.CERT_REQUIRED_NO_VERIFY as possible value for ssl.SSLC https://bugs.python.org/issue46779 opened by freundTech #46781: Tracing: c_return doesn't report the result https://bugs.python.org/issue46781 opened by smurfix #46782: Docs error for 3.10 https://bugs.python.org/issue46782 opened by usetohandletrush #46784: Duplicated symbols when linking embedded Python with libexpat https://bugs.python.org/issue46784 opened by yilei #46785: On Windows, os.stat() can fail if called while another process https://bugs.python.org/issue46785 opened by Antony.Lee #46786: embed, source, track, wbr HTML elements not considered empty https://bugs.python.org/issue46786 opened by jnns #46787: ProcessPoolExecutor exception memory leak https://bugs.python.org/issue46787 opened by themylogin #46788: regrtest fails to start on missing performance counter names https://bugs.python.org/issue46788 opened by jkloth #46789: Restore caching of externals on Windows buildbots https://bugs.python.org/issue46789 opened by jkloth #46790: Normalize handling of negative timeouts in subprocess.py https://bugs.python.org/issue46790 opened by jkloth #46791: Allow os.remove to defer to rmdir https://bugs.python.org/issue46791 opened by bup Most recent 15 issues with no replies (15) ========================================== #46791: Allow os.remove to defer to rmdir https://bugs.python.org/issue46791 #46788: regrtest fails to start on missing performance counter names https://bugs.python.org/issue46788 #46787: ProcessPoolExecutor exception memory leak https://bugs.python.org/issue46787 #46786: embed, source, track, wbr HTML elements not considered empty https://bugs.python.org/issue46786 #46784: Duplicated symbols when linking embedded Python with libexpat https://bugs.python.org/issue46784 #46782: Docs error for 3.10 https://bugs.python.org/issue46782 #46779: Add ssl.CERT_REQUIRED_NO_VERIFY as possible value for ssl.SSLC https://bugs.python.org/issue46779 #46775: [Windows] OSError should unconditionally call winerror_to_errn https://bugs.python.org/issue46775 #46774: Importlib.metadata.version picks first distribution not latest https://bugs.python.org/issue46774 #46773: Add a Private API for Looking Up Global Objects https://bugs.python.org/issue46773 #46772: Statically Initialize PyArg_Parser in clinic.py https://bugs.python.org/issue46772 #46765: Replace Locally Cached Strings with Statically Initialized Obj https://bugs.python.org/issue46765 #46761: functools.update_wrapper breaks the signature of functools.par https://bugs.python.org/issue46761 #46760: test_dis should test the dis module, not everything else https://bugs.python.org/issue46760 #46759: sys.excepthook documentation doesn't mention that it isn't cal https://bugs.python.org/issue46759 Most recent 15 issues waiting for review (15) ============================================= #46787: ProcessPoolExecutor exception memory leak https://bugs.python.org/issue46787 #46786: embed, source, track, wbr HTML elements not considered empty https://bugs.python.org/issue46786 #46784: Duplicated symbols when linking embedded Python with libexpat https://bugs.python.org/issue46784 #46781: Tracing: c_return doesn't report the result https://bugs.python.org/issue46781 #46777: Fix incorrect use of directives in asyncio documentation https://bugs.python.org/issue46777 #46771: Add some form of cancel scopes https://bugs.python.org/issue46771 #46765: Replace Locally Cached Strings with Statically Initialized Obj https://bugs.python.org/issue46765 #46764: Wrapping a bound method with a @classmethod no longer works https://bugs.python.org/issue46764 #46760: test_dis should test the dis module, not everything else https://bugs.python.org/issue46760 #46759: sys.excepthook documentation doesn't mention that it isn't cal https://bugs.python.org/issue46759 #46756: Incorrect authorization check in urllib.request https://bugs.python.org/issue46756 #46755: QueueHandler logs stack_info twice https://bugs.python.org/issue46755 #46753: Statically allocate and initialize the empty tuple. https://bugs.python.org/issue46753 #46752: Introduce task groups to asyncio and change task cancellation https://bugs.python.org/issue46752 #46750: some code paths in ssl and _socket still import idna unconditi https://bugs.python.org/issue46750 Top 10 most discussed issues (10) ================================= #46771: Add some form of cancel scopes https://bugs.python.org/issue46771 38 msgs #46726: Thread spuriously marked dead after interrupting a join call https://bugs.python.org/issue46726 16 msgs #46541: Replace _Py_IDENTIFIER() with statically initialized objects. https://bugs.python.org/issue46541 11 msgs #45447: Make IDLE recognize .pyi stub files as source for open, save, https://bugs.python.org/issue45447 10 msgs #46743: Enable usage of object.__orig_class__ in __init__ https://bugs.python.org/issue46743 10 msgs #46752: Introduce task groups to asyncio and change task cancellation https://bugs.python.org/issue46752 8 msgs #46751: Windows-style path is not recognized under cygwin https://bugs.python.org/issue46751 7 msgs #46716: regrtest didn't respect the timeout when running test_subproce https://bugs.python.org/issue46716 6 msgs #46733: pathlib.Path methods can raise NotImplementedError https://bugs.python.org/issue46733 6 msgs #46750: some code paths in ssl and _socket still import idna unconditi https://bugs.python.org/issue46750 6 msgs Issues closed (27) ================== #29418: inspect.isroutine does not return True for some bound builtin https://bugs.python.org/issue29418 closed by BTaskaya #38619: [Doc] UUID.hex is lowercase https://bugs.python.org/issue38619 closed by Mariatta #41086: Exception for uninstantiated interpolation (configparser) https://bugs.python.org/issue41086 closed by lukasz.langa #45948: Unexpected instantiation behavior for xml.etree.ElementTree.XM https://bugs.python.org/issue45948 closed by serhiy.storchaka #46066: Deprecate keyword args syntax for TypedDict definition https://bugs.python.org/issue46066 closed by Jelle Zijlstra #46333: ForwardRef.__eq__ does not respect module parameter https://bugs.python.org/issue46333 closed by Jelle Zijlstra #46436: Pass the -d/--directory command-line option to http.server.CGI https://bugs.python.org/issue46436 closed by eric.araujo #46634: [sqlite3]��speed up cursor.execute*() https://bugs.python.org/issue46634 closed by erlendaasland #46666: IDLE Add indent guide https://bugs.python.org/issue46666 closed by terry.reedy #46700: wrong nomenclature (options vs. arguments) in argparse https://bugs.python.org/issue46700 closed by rhettinger #46704: Parser API not checking for null-terminator https://bugs.python.org/issue46704 closed by pablogsal #46718: Feature: itertools: add batches https://bugs.python.org/issue46718 closed by rhettinger #46728: Docstring of combinations_with_replacement for consistency https://bugs.python.org/issue46728 closed by corona10 #46730: Please consider mentioning property without setter when an att https://bugs.python.org/issue46730 closed by Dennis Sweeney #46737: Default to the standard normal distribution https://bugs.python.org/issue46737 closed by rhettinger #46738: Allow http.server to emit HTML 5 https://bugs.python.org/issue46738 closed by Jelle Zijlstra #46739: dataclasses __eq__ isn't logical https://bugs.python.org/issue46739 closed by Dennis Sweeney #46741: Docstring for asyncio.protocols.BufferedProtocol appears out o https://bugs.python.org/issue46741 closed by asvetlov #46745: Typo in new PositionsIterator https://bugs.python.org/issue46745 closed by sobolevn #46747: bisect.bisect/insort don't document key parameter https://bugs.python.org/issue46747 closed by Dennis Sweeney #46762: assertion failure in f-string parsing Parser/string_parser.c https://bugs.python.org/issue46762 closed by pablogsal #46766: Add a class for file operations so a syntax such as open("file https://bugs.python.org/issue46766 closed by serhiy.storchaka #46768: CVE-201-4160 Vulnerability Is Found in Lib/site-packages/crypt https://bugs.python.org/issue46768 closed by christian.heimes #46776: RecursionError when using property() inside classes https://bugs.python.org/issue46776 closed by steven.daprano #46778: Enable parallel compilation on Windows builds https://bugs.python.org/issue46778 closed by steve.dower #46780: Allow Fractions to return 1/6 for "0.17", "0.167", "0.1667", e https://bugs.python.org/issue46780 closed by mark.dickinson #46783: Add a new feature to enumerate(iterable, start=0) built-in fun https://bugs.python.org/issue46783 closed by rhettinger

1 0

Compiling of ast.Module in Python 3.10 and co_firstlineno behavior
by Fabio Zadrozny 18 Feb '22

18 Feb '22

Hi all, I'm stumbling with an issue where the co_firstlineno behavior changed from Python 3.9 to Python 3.10 and I was wondering if this was intentional or not. i.e.: Whenever a code is compiled in Python 3.10, the `code.co_firstlineno` is now always 1, whereas previously it was equal to the first statement. Also, does anyone know if there is any way to restore the old behavior in Python 3.10? I tried setting the `module.lineno` but it didn't really make any difference... As an example, given the code below: import dis source = ''' print(1) print(2) ''' initial_module = compile(source, '<nofilename>', 'exec', PyCF_ONLY_AST, 1) import sys print(sys.version) for i in range(2): module = Module([initial_module.body[i]], []) module_code = compile(module, '<no filename>', 'exec') print(' --> First lineno:', module_code.co_firstlineno) print(' --> Line starts :', list(lineno for offset, lineno in dis.findlinestarts(module_code))) print('---- dis ---') dis.dis(module_code) I have the following outputs for Pyhon 3.9/Python 3.10: 3.9.6 (default, Jul 30 2021, 11:42:22) [MSC v.1916 64 bit (AMD64)] --> First lineno: 2 --> Line starts : [2] ---- dis --- 2 0 LOAD_NAME 0 (print) 2 LOAD_CONST 0 (1) 4 CALL_FUNCTION 1 6 POP_TOP 8 LOAD_CONST 1 (None) 10 RETURN_VALUE --> First lineno: 4 --> Line starts : [4] ---- dis --- 4 0 LOAD_NAME 0 (print) 2 LOAD_CONST 0 (2) 4 CALL_FUNCTION 1 6 POP_TOP 8 LOAD_CONST 1 (None) 10 RETURN_VALUE 3.10.0 (tags/v3.10.0:b494f59, Oct 4 2021, 19:00:18) [MSC v.1929 64 bit (AMD64)] --> First lineno: 1 --> Line starts : [2] ---- dis --- 2 0 LOAD_NAME 0 (print) 2 LOAD_CONST 0 (1) 4 CALL_FUNCTION 1 6 POP_TOP 8 LOAD_CONST 1 (None) 10 RETURN_VALUE --> First lineno: 1 --> Line starts : [4] ---- dis --- 4 0 LOAD_NAME 0 (print) 2 LOAD_CONST 0 (2) 4 CALL_FUNCTION 1 6 POP_TOP 8 LOAD_CONST 1 (None) 10 RETURN_VALUE Thanks, Fabio

3 5

Custom memory profiler and PyTraceMalloc_Track
by Sümer Cip 15 Feb '22

15 Feb '22

Hi everyone, I would like to ask a question about an issue that we faced regarding profiling memory usage: We have implemented a custom memory profiler using `PyMem_GetAllocator/PyMem_SetAllocator` APIs like `tracemalloc`. Right now, we are facing an issue with numpy: numpy seems to have its own memory allocator and they use `PyTraceMalloc_Track` APIs to notify tracemalloc about the allocation. A simple search on GitHub reveals there are more projects using this approach: https://github.com/search?q=PyTraceMalloc_Track&type=code which is fine. I believe it is common to use custom memory allocators for scientific computation which makes perfect sense. However, I would have expected to have at least some form of `PyMem_NotifyAllocator` type of API instead of a one that is specific to `tracemalloc`? I might be missing some context here. WDYT? Best,

2 3

Request for feedback: pathlib.AbstractPath prototype
by Barney Gale 11 Feb '22

11 Feb '22

Over the last couple of years I've been tidying up the pathlib internals, with a view towards adding an AbstractPath class to the hierarchy. Users would be able to subclass AbstractPath to implement other kinds of filesystems: s3, google cloud storage, pandas, ftp, git, zip and tar files, etc. By implementing some abstract methods (stat(), iterdir(), open(), etc) they'd benefit from a number of derived methods (is_dir(), glob(), read_text(), etc). There's already a healthy ecosystem of PyPI packages attempting this, but there's presently no officially-supported route. I've now submitted a PR that adds an initial underscore-prefixed implementation of this class: https://github.com/python/cpython/pull/31085. The implementation is simple: wherever Path calls functions in os, io, pwd, etc, AbstractPath instead raises NotImplementedError. The Path class becomes an implementation of AbstractPath. These methods directly raise NotImplementedError: cwd(), stat(), iterdir(), readlink(), resolve(), expanduser(), owner(), group(), open(), touch(), mkdir(), symlink_to(), hardlink_to(), rename(), replace(), chmod(), unlink(), rmdir() These methods call through to the above methods: absolute(), lstat(), exists(), is_dir(), is_file(), is_mount(), is_symlink(), is_block_device(), is_char_device(), is_fifo(), is_socket(), home(), samefile(), scandir(), glob(), rglob(), read_bytes(), read_text(), write_bytes(), write_text(), lchmod(), link_to() Some methods aren't applicable to some kinds of filesystems, e.g. zip files don't support symlinks, working directories or home directories. In these cases I think it's reasonable to raise NotImplementedError. Indeed, pathlib.Path methods already raise NotImplementedError when certain local filesystem features aren't available (readlink(), group(), etc). If any readers of this list have previously tried to extend pathlib to other domains, or are otherwise interested in pathlib development, please could you let me know what you think of the proposed API? Many thanks

3 8

Summary of Python tracker Issues
by Python tracker 11 Feb '22

11 Feb '22

ACTIVITY SUMMARY (2022-02-04 - 2022-02-11) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7178 (+34) closed 51272 (+50) total 58450 (+84) Open issues with patches: 2912 Issues opened (63) ================== #42548: debugger stops at breakpoint of `pass` that is not actually re https://bugs.python.org/issue42548 reopened by iritkatriel #44006: symbol documentation still exists https://bugs.python.org/issue44006 reopened by vstinner #45459: Limited API support for Py_buffer https://bugs.python.org/issue45459 reopened by vstinner #46430: intern strings in deepfrozen modules https://bugs.python.org/issue46430 reopened by christian.heimes #46639: Ceil division with math.ceildiv https://bugs.python.org/issue46639 opened by Vladimir Feinberg #46640: Python can now use the C99 NAN constant or __builtin_nan() https://bugs.python.org/issue46640 opened by vstinner #46642: typing: tested TypeVar instance subclass TypeError is incident https://bugs.python.org/issue46642 opened by GBeauregard #46643: typing.Annotated cannot wrap typing.ParamSpec args/kwargs https://bugs.python.org/issue46643 opened by GBeauregard #46644: typing: remove callable() check from typing._type_check https://bugs.python.org/issue46644 opened by GBeauregard #46645: Portable python3 shebang for Windows, macOS, and Linux https://bugs.python.org/issue46645 opened by joshtriplett #46646: `address` arg can be `bytes` for `ip_*` functions in `ipaddres https://bugs.python.org/issue46646 opened by sobolevn #46649: Propagate Python thread name to thread state structure https://bugs.python.org/issue46649 opened by Gabriele Tornetta #46650: `priority` in `sched.scheduler` is not sufficiently tested https://bugs.python.org/issue46650 opened by sobolevn #46652: Use code.co_qualname to provide richer information https://bugs.python.org/issue46652 opened by Gabriele Tornetta #46653: sys.path entries normalization in site.py doesn't follow POSIX https://bugs.python.org/issue46653 opened by jpoiret #46654: urllib.request.urlopen doesn't handle UNC paths produced by pa https://bugs.python.org/issue46654 opened by ikelos #46655: typing.TypeAlias is not in the list of allowed plain _SpecialF https://bugs.python.org/issue46655 opened by GBeauregard #46656: Compile fails if Py_NO_NAN is defined https://bugs.python.org/issue46656 opened by mark.dickinson #46657: Add mimalloc memory allocator https://bugs.python.org/issue46657 opened by christian.heimes #46658: shutil Lib enables sendfile on solaris for regular files https://bugs.python.org/issue46658 opened by devnexen #46659: Deprecate locale.getdefaultlocale() function https://bugs.python.org/issue46659 opened by vstinner #46661: Duplicat deprecation warnings in docs for asyncio https://bugs.python.org/issue46661 opened by gvanrossum #46662: Lib/sqlite3/dbapi2.py: convert_timestamp function failed to co https://bugs.python.org/issue46662 opened by Rayologist #46663: test_math test_cmath test_complex fails on Fedora Rawhide buil https://bugs.python.org/issue46663 opened by vstinner #46664: PY_SSIZE_T_MAX is not an integer constant expression https://bugs.python.org/issue46664 opened by ov2k #46665: IDLE Windows shortcuts by default https://bugs.python.org/issue46665 opened by primexx #46666: IDLE Add indent guide https://bugs.python.org/issue46666 opened by primexx #46667: SequenceMatcher & autojunk - false negative https://bugs.python.org/issue46667 opened by jonathan-lp #46668: encodings: the "mbcs" alias doesn't work https://bugs.python.org/issue46668 opened by vstinner #46670: Build Python with -Wundef: don't use undefined macros https://bugs.python.org/issue46670 opened by vstinner #46671: "ValueError: min() arg is an empty sequence" is wrong (builtin https://bugs.python.org/issue46671 opened by Nnarol #46672: NameError in asyncio.gather when passing a invalid type as an https://bugs.python.org/issue46672 opened by onerandomusername #46675: Allow more than 16 items in split-keys dicts and "virtual" obj https://bugs.python.org/issue46675 opened by Mark.Shannon #46677: TypedDict docs are incomplete https://bugs.python.org/issue46677 opened by Jelle Zijlstra #46679: test.support.wait_process ignores timeout argument https://bugs.python.org/issue46679 opened by notarealdeveloper #46681: gzip.compress does not forward compresslevel to zlib.compress https://bugs.python.org/issue46681 opened by iii-i #46682: python 3.10 Py_Initialize/Py_Main std path no longer includes https://bugs.python.org/issue46682 opened by pjaggi1 #46685: Add additional tests for new features in `typing.py` https://bugs.python.org/issue46685 opened by sobolevn #46686: [venv / PC/launcher] issue with a space in the installed pytho https://bugs.python.org/issue46686 opened by hokiedsp #46689: `list(FunctionType(a.gi_code, {})(0))` crashes Python https://bugs.python.org/issue46689 opened by sobolevn #46690: create_autospec() doesn't respect configure_mock style kwargs https://bugs.python.org/issue46690 opened by marchant.jm #46691: sysconfig.get_platform() raises ValueError on macOS if '-arch' https://bugs.python.org/issue46691 opened by duncanmmacleod #46692: match case does not support regex https://bugs.python.org/issue46692 opened by AliRn #46695: _io_TextIOWrapper_reconfigure_impl errors out too early https://bugs.python.org/issue46695 opened by mirabilos #46696: socketmodule add Linux SO_INCOMING_CPU constant https://bugs.python.org/issue46696 opened by devnexen #46697: _ctypes_simple_instance returns inverted logic https://bugs.python.org/issue46697 opened by hydroflask #46700: wrong nomenclature (options vs. arguments) in argparse https://bugs.python.org/issue46700 opened by mirabilos #46702: Specialize UNPACK_SEQUENCE https://bugs.python.org/issue46702 opened by brandtbucher #46704: Parser API not checking for null-terminator https://bugs.python.org/issue46704 opened by anthonypjshaw #46705: Memory optimization for set.issubset https://bugs.python.org/issue46705 opened by panda1200 #46711: test_logging: test_post_fork_child_no_deadlock() failed with t https://bugs.python.org/issue46711 opened by vstinner #46712: Share global string identifiers in deepfreeze https://bugs.python.org/issue46712 opened by kumaraditya303 #46713: Provide a C implementation of collections.abc.KeysView and fri https://bugs.python.org/issue46713 opened by jab #46715: asyncio.create_unix_server has an off-by-one error concerning https://bugs.python.org/issue46715 opened by jnsnow #46716: regrtest didn't respect the timeout when running test_subproce https://bugs.python.org/issue46716 opened by vstinner #46717: Raising exception multiple times leaks memory https://bugs.python.org/issue46717 opened by ggensure #46718: Feature: itertools: add batches https://bugs.python.org/issue46718 opened by socketpair #46719: Call not visited in ast.NodeTransformer https://bugs.python.org/issue46719 opened by davidcastells #46720: Add support of path-like objects to multiprocessing.set_execut https://bugs.python.org/issue46720 opened by maggyero #46721: Optimize set.issuperset() for non-set argument https://bugs.python.org/issue46721 opened by serhiy.storchaka #46722: Different behavior for functiools.partial between inspect.isfu https://bugs.python.org/issue46722 opened by IliyaZinoviev #46723: SimpleQueue.put_nowait() documentation error https://bugs.python.org/issue46723 opened by antonycardazzi #46724: Odd Bytecode Generation in 3.10 https://bugs.python.org/issue46724 opened by saulshanabrook Most recent 15 issues with no replies (15) ========================================== #46724: Odd Bytecode Generation in 3.10 https://bugs.python.org/issue46724 #46723: SimpleQueue.put_nowait() documentation error https://bugs.python.org/issue46723 #46722: Different behavior for functiools.partial between inspect.isfu https://bugs.python.org/issue46722 #46720: Add support of path-like objects to multiprocessing.set_execut https://bugs.python.org/issue46720 #46715: asyncio.create_unix_server has an off-by-one error concerning https://bugs.python.org/issue46715 #46713: Provide a C implementation of collections.abc.KeysView and fri https://bugs.python.org/issue46713 #46711: test_logging: test_post_fork_child_no_deadlock() failed with t https://bugs.python.org/issue46711 #46700: wrong nomenclature (options vs. arguments) in argparse https://bugs.python.org/issue46700 #46696: socketmodule add Linux SO_INCOMING_CPU constant https://bugs.python.org/issue46696 #46695: _io_TextIOWrapper_reconfigure_impl errors out too early https://bugs.python.org/issue46695 #46682: python 3.10 Py_Initialize/Py_Main std path no longer includes https://bugs.python.org/issue46682 #46681: gzip.compress does not forward compresslevel to zlib.compress https://bugs.python.org/issue46681 #46679: test.support.wait_process ignores timeout argument https://bugs.python.org/issue46679 #46672: NameError in asyncio.gather when passing a invalid type as an https://bugs.python.org/issue46672 #46671: "ValueError: min() arg is an empty sequence" is wrong (builtin https://bugs.python.org/issue46671 Most recent 15 issues waiting for review (15) ============================================= #46721: Optimize set.issuperset() for non-set argument https://bugs.python.org/issue46721 #46720: Add support of path-like objects to multiprocessing.set_execut https://bugs.python.org/issue46720 #46715: asyncio.create_unix_server has an off-by-one error concerning https://bugs.python.org/issue46715 #46712: Share global string identifiers in deepfreeze https://bugs.python.org/issue46712 #46711: test_logging: test_post_fork_child_no_deadlock() failed with t https://bugs.python.org/issue46711 #46705: Memory optimization for set.issubset https://bugs.python.org/issue46705 #46702: Specialize UNPACK_SEQUENCE https://bugs.python.org/issue46702 #46697: _ctypes_simple_instance returns inverted logic https://bugs.python.org/issue46697 #46696: socketmodule add Linux SO_INCOMING_CPU constant https://bugs.python.org/issue46696 #46685: Add additional tests for new features in `typing.py` https://bugs.python.org/issue46685 #46681: gzip.compress does not forward compresslevel to zlib.compress https://bugs.python.org/issue46681 #46679: test.support.wait_process ignores timeout argument https://bugs.python.org/issue46679 #46675: Allow more than 16 items in split-keys dicts and "virtual" obj https://bugs.python.org/issue46675 #46672: NameError in asyncio.gather when passing a invalid type as an https://bugs.python.org/issue46672 #46671: "ValueError: min() arg is an empty sequence" is wrong (builtin https://bugs.python.org/issue46671 Top 10 most discussed issues (10) ================================= #46586: In documentation contents enum.property erroneously links to b https://bugs.python.org/issue46586 15 msgs #46640: Python can now use the C99 NAN constant or __builtin_nan() https://bugs.python.org/issue46640 15 msgs #46659: Deprecate locale.getdefaultlocale() function https://bugs.python.org/issue46659 13 msgs #46644: typing: remove callable() check from typing._type_check https://bugs.python.org/issue46644 12 msgs #46639: Ceil division with math.ceildiv https://bugs.python.org/issue46639 11 msgs #46657: Add mimalloc memory allocator https://bugs.python.org/issue46657 11 msgs #46654: urllib.request.urlopen doesn't handle UNC paths produced by pa https://bugs.python.org/issue46654 10 msgs #46656: Compile fails if Py_NO_NAN is defined https://bugs.python.org/issue46656 9 msgs #46670: Build Python with -Wundef: don't use undefined macros https://bugs.python.org/issue46670 9 msgs #46166: [C API] Get "self" args or non-null co_varnames from frame obj https://bugs.python.org/issue46166 7 msgs Issues closed (48) ================== #23952: cgi: Document the 'maxlen' member of the cgi module https://bugs.python.org/issue23952 closed by ethan.furman #31369: re.RegexFlag is not included in __all__, makes type inference https://bugs.python.org/issue31369 closed by ethan.furman #34191: argparse: Missing subparser error message should be more clear https://bugs.python.org/issue34191 closed by ned.deily #41259: Find adverbs is not correct on the documentation https://bugs.python.org/issue41259 closed by serhiy.storchaka #43216: Removal of @asyncio.coroutine in Python 3.11 https://bugs.python.org/issue43216 closed by asvetlov #43366: Unclosed bracket bug in code.interact prevents identifying syn https://bugs.python.org/issue43366 closed by pablogsal #43532: Add keyword-only fields to dataclasses https://bugs.python.org/issue43532 closed by eric.smith #44953: Add vectorcall on operator.itemgetter and attrgetter objects https://bugs.python.org/issue44953 closed by Dennis Sweeney #45436: test_tk.test_configure_type() fails with Tcl/Tk 8.6.11 https://bugs.python.org/issue45436 closed by zach.ware #45863: tarfile zeroes ustar header fields unnecessarily https://bugs.python.org/issue45863 closed by vstinner #46119: Update bundled pip to 21.3.1 and setuptools to 59.7.0 https://bugs.python.org/issue46119 closed by kumaraditya303 #46161: `class A(1, 2, 3, **d): pass` gives bad bytecode https://bugs.python.org/issue46161 closed by Mark.Shannon #46323: Use _PyObject_Vectorcall in Modules/_ctypes/callbacks.c https://bugs.python.org/issue46323 closed by corona10 #46475: typing.Never and typing.assert_never https://bugs.python.org/issue46475 closed by Jelle Zijlstra #46479: Implement typing.reveal_locals https://bugs.python.org/issue46479 closed by Jelle Zijlstra #46521: codeop._maybe_compile passes code with error + triple quotes https://bugs.python.org/issue46521 closed by pablogsal #46528: Simplify the VM's stack manipulations https://bugs.python.org/issue46528 closed by brandtbucher #46554: Add append keyword argument to Path.write_text() and Path.writ https://bugs.python.org/issue46554 closed by keelung-yang #46556: pathlib.Path.__enter__() should emit DeprecationWarning https://bugs.python.org/issue46556 closed by brett.cannon #46608: Exclude marshalled-frozen data if deep-freezing to save 300 KB https://bugs.python.org/issue46608 closed by kumaraditya303 #46609: Generator-based coroutines in Python 3.10, 3.9 docs https://bugs.python.org/issue46609 closed by terry.reedy #46611: Improve coverage of `__instancecheck__` and `__subclasscheck__ https://bugs.python.org/issue46611 closed by serhiy.storchaka #46638: Inconsistent registry virtualization in Windows Store package https://bugs.python.org/issue46638 closed by steve.dower #46641: multiplication error 2.2 and 2.1 https://bugs.python.org/issue46641 closed by zach.ware #46647: `test_functools` unexpected failures when C `_functoolsmodule` https://bugs.python.org/issue46647 closed by sobolevn #46648: `test.test_urllib2.MiscTests.test_issue16464` flaky due to ext https://bugs.python.org/issue46648 closed by vstinner #46651: test_urllib2.test_issue16464() fails randomly https://bugs.python.org/issue46651 closed by vstinner #46660: datetime.timestamp() fails for naive-datetime values prior to https://bugs.python.org/issue46660 closed by smrpy #46669: Add types.Self https://bugs.python.org/issue46669 closed by rhettinger #46673: Py_BuildValue tuple creation segfaults in python3.9..3.11 https://bugs.python.org/issue46673 closed by olupton #46674: Two if in a row in generators https://bugs.python.org/issue46674 closed by BTaskaya #46676: ParamSpec args and kwargs are not equal to themselves. https://bugs.python.org/issue46676 closed by serhiy.storchaka #46678: Invalid cross device link in Lib/test/support/import_helper.py https://bugs.python.org/issue46678 closed by brett.cannon #46680: file calls itself https://bugs.python.org/issue46680 closed by Dennis Sweeney #46683: Python 3.6.15 source tarball installs 3.6.8? https://bugs.python.org/issue46683 closed by zach.ware #46684: Expose frozenset._hash classmethod https://bugs.python.org/issue46684 closed by rhettinger #46687: Update pyexpat for CVE-2021-45960 https://bugs.python.org/issue46687 closed by steve.dower #46688: Add sys.is_interned https://bugs.python.org/issue46688 closed by methane #46693: dataclass generated __str__ does not use overridden member __s https://bugs.python.org/issue46693 closed by eric.smith #46694: isdigit/isnumeric vs int() https://bugs.python.org/issue46694 closed by mark.dickinson #46701: cannot use typographical quotation marks in bug description https://bugs.python.org/issue46701 closed by eric.smith #46703: boolean operation issue (True == False == False) https://bugs.python.org/issue46703 closed by pablogsal #46706: AxelRacer https://bugs.python.org/issue46706 closed by Dennis Sweeney #46707: Parser hanging on stacked { tokens https://bugs.python.org/issue46707 closed by pablogsal #46708: test_asyncio: test_sock_client_fail() changes asyncio.events._ https://bugs.python.org/issue46708 closed by asvetlov #46709: test_urllib: testInterruptCaught() has a race condition and fa https://bugs.python.org/issue46709 closed by vstinner #46710: Install launcher for all users on the domain https://bugs.python.org/issue46710 closed by marcus.siqueira #46714: Python 3.10 - Users (except from the one who installed) not ab https://bugs.python.org/issue46714 closed by steve.dower

1 0

urllib: addressing inflexibility in scheme-based joining
by Lincoln Auster 11 Feb '22

11 Feb '22

Hi, This is a follow-up RFC on PR #30520 (BPO 46337) with regard to urllib's potentially inflexible scheme-based URL manipulation. In my ticket, I proposed supplementing the hard-coded uses_* scheme lists that control URL parsing and joining behavior with an optional enum set that may override behavior when calling a function that would require it. (More detail is in that BPO and code is obviously in that PR.) It's been about a month since I wrote that PR, and it was marked stale a day or two ago. Would anyone be willing to give it a look for feedback and a potential merge? As a side note, I was recently made aware of the bit of an existential crisis wrt urllib. Without addressing it directly, I would like to say that my PR is small, backwards compatible, and focused exclusively on URL parsing, which (at least in my reading of the relevant thread) is still considered to be useful if a bit fragile. I wouldn't consider my PR to be in conflict with any ongoing scope reconsideration (though deprecation is, of course, another matter entirely). Thanks for any feedback and review! -- Lincoln Auster They/Them

2 1

How to build cpython for x86_64 target on M1 Mac?
by jeff.pek＠autodesk.com 10 Feb '22

10 Feb '22

Hi - Newbie post here... We need to be able to produce both arm64 and x86_64 builds of cpython on a single machine. I can see that, by default, it chooses the target architecture based on the current running architecture. The docs say that we can target one platform from the other, but it's not specific about how to do that? I figure this is some sort of 'configure' option -- maybe "--build"? -- but so far nothing I've tried has worked. Can someone help point us in the right direction? What's the magic to specify one target or the other? Thank you! Jeff

3 2

PEP 677 (Callable Type Syntax): Rejection notice.
by Gregory P. Smith 10 Feb '22

10 Feb '22

Thank you PEP authors for producing a very well written and laid out PEP. That made it easy to understand the proposal, rationale, and alternatives considered. We agree that the existing typing.Callable syntax, while capable, is less than ideal for humans. That said, we deliberated last week and ultimately decided to reject PEP-677 Callable Type Syntax. Why? Several considerations led us here: 1. We feel we need to be cautious when introducing new syntax. Our new parser presents understandably exciting opportunities but we don’t want its existence to mean we add new syntax easily. A feature for use only in a fraction of type annotations, not something every Python user uses, did not feel like a strong enough reason to justify the complexity needed to parse this new syntax and provide meaningful error messages. Not only code complexity, humans are also parsers that must look forwards and backwards. 2. While the current Callable[x, y] syntax is not loved, it does work. This PEP isn’t enabling authors to express anything they cannot already. The PEP explicitly chose be conservative and not allow more syntax to express features going beyond what Callable supports. We applaud that decision, starting simple is good. But we can imagine a future where the syntax would desire to be expanded upon. 3. In line with past SC guidance, we acknowledge challenges when syntax desires do not align between typing and Python itself. Each time we add syntax solely for typing it shifts us further in the direction of typing being its own mini-language so we aim to tread lightly in what gets added here. Adopting PEP 677 would lock us into its syntax forever, potentially preventing other future syntax directions. 4. We did not like the visual and cognitive consequence of multiple `->` tokens in a def. Especially when code is not formatted nicely. Though we admit the correlation between Python typing and formatter users is high. ## Recommendation for future syntax enhancements: When proposing a syntax change, low complexity is better. While not always possible, it’s ideal if it could still be described using our old <=3.8 parser. It is important to have in mind that adding syntax that only our modern PEG parser can handle could lead to greater cognitive load and external tooling implementation costs. This should not be read as a ban on PEG-only syntax, we just think it should be used for broadly applicable features or else be relatively unintrusive. Thanks, The 3.11 Python Steering Council

2 2

Please update Cython *before* introcuding C API incompatible changes in Python
by Victor Stinner 10 Feb '22

10 Feb '22

Hi, It became more and more common that a C API incompatible change introduced in Python breaks Cython and all Python projects using Cython (ex: numpy). Hopefully, usually only some projects using Cython are broken, not all of them. Some of you may remind the PEP 590 (vectorcall) implementation which removed the tp_print member of PyTypeObject in Python 3.8. This change broke a large number of projects using Cython. Cython was updated, the member removal was reverted, etc. It took a few weeks to solve the issue. It was unpleasant to have to revert the change (add again tp_print). The tp_print member was removed again in Python 3.9. IMO the PEP 570 (positional-only arguments) implementation went even worse when it added a parameter to PyCode_New(). Cython was modified and released with a change, and *then* the PyCode_New() change was reverted in Python (if I recall correctly)! It was very confusing. There were multiple communication issues between Python and Cython. The first Cython fix was incorrect, etc. Last December, another Python change related to exceptions (bpo-45711) broke Cython on purpose. A commit message says "Add to what's new, because this change breaks things like Cython". It's kind of annoying that the each time we have a long period of time (several weeks) when Cython is unusable and there is pressure on Cython get a fix and then release a new version. At Red Hat, we are rebuilding Python frequently with an up to date Python 3.11: right now, numpy fails to build because of that. -- I would prefer to introduce C API incompatible changes differently: first fix Cython, and *then* introduce the change. - (1) Propose a Cython PR and get it merged - (2) Wait until a new Cython version is released - (3) If possible, wait until numpy is released with regenerated Cython code - (4) Introduce the incompatible change in Python Note: Fedora doesn't need (3) since we always regenerated Cython code in numpy. None of these C API incompatible changes are wrong. For each change, there are good reasons to introduce them. I'm only proposing to change *how* we introduce them to avoid a long period of time when Cython and/or numpy are not usable on the development version of Python. -- If you want to test if a C API change is going to break Cython or numpy, you can try my tool: https://github.com/vstinner/pythonci/ Usage: build a modified Python with your change, and then run pythonci with it. pythonci uses pinned version of all dependencies to try to be more reproducible. I update them manually infrequently (see pythonci/requirements.txt). -- Right now, I propose to revert the incompatible change related to exceptions until Cython is prepared for it: https://bugs.python.org/issue45711#msg412264 Victor -- Night gathers, and now my watch begins. It shall not end until my death.

14 49

Suggestion: a little language for type definitions
by jack.jansen＠cwi.nl 10 Feb '22

10 Feb '22

I posted this suggestion earlier in the callable type syntax discussion, at which point it was completely ignored. Possibly because it’s a really stupid idea, but let me post it again on the off chance that it isn’t a stupid idea but was overlooked. > If I can make a wild suggestion: why not create a little language for type specifications? > > If you look at other programming languages you’ll see that the “type definition sub-language” is often completely different from the “execution sub-language”, with only some symbols in common and used in vaguely related ways. `bool (*myfuncptr)(int, float*)` uses a completely different set of syntactic rules than `rv = (*myfunptr)(*myintptr, &myfloat)`. So with some grains of salt you could say that C is comprised of a declarative typing sublanguage and an imperative execution sublanguage. And an even better example is Pascal, which uses a set of syntactic constructs for typing that are completely different from the execution statement syntax: `var a : array[1..10] of real` looks very different from `a[1]`, where C `float a[10]` looks pretty similar to `a[10]`. The next bit of my original email is another wild idea, the previous bit doesn’t depend on it really. I can imagine completely different ways of doing a typing sublanguage: > Python typing uses basically a subset of the execution expression syntax as its declarative typing language. > > What if we created a little language that is clearly flagged, for example as t”….” or t’….’? Then we could simply define the typestring language to be readable, so you could indeed say t”(int, str) -> bool”. And we could even allow escapes (similar to f-strings) so that the previous expression could also be specified, if you really wanted to, as t”{typing.Callable[[int, str], bool}”. -- Jack Jansen, <Jack.Jansen(a)cwi.nl>, http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman

15 35

Replace debug runtime checks in release mode with assertions in debug mode
by Victor Stinner 09 Feb '22

09 Feb '22

Hi, I would like to replace runtime checks on C function arguments with assertions. For example, replace: PyObject * PyDict_GetItemWithError(PyObject *op, PyObject *key) { if (!PyDict_Check(op)) { PyErr_BadInternalCall(); return NULL; } ... } with: PyObject * PyDict_GetItemWithError(PyObject *op, PyObject *key) { assert(PyDict_Check(op)); ... } I'm not asking to remove all of them. For example, PyList_GetItem(list, index) will continuing raising an IndexError if the index is out of range. I'm asking to replace runtime checks with assertions when the C API is "obviously" misused: replace PyErr_BadInternalCall(), _Py_CheckFunctionResult() and _Py_CheckSlotResult() with assertions. The exact scope should be defined. -- Python tries to be nice to C extensions authors and calls PyErr_BadInternalCall() to emit a nice SystemError exception when the C API is misused. There are many sanity checks run at runtime. We pay the cost of these runtime checks even if a C extension is well written (has no bug). In Python 3.6, I added the _Py_CheckFunctionResult() function to check the result of any function call. A function must return a non-NULL object and not raises an exception, or return NULL and raises an exception. Otherwise, SystemError is raised. For example, PyObject_Call() uses it to check the PyTypeObject.tp_call result. In Python 3.10, I added a similar _Py_CheckSlotResult() function to check the result of slot functions. For example, PyObject_Size() checks the result of the PyTypeObject.tp_as_sequence.sq_length slot function. -- I modified Python 3.8 to be able to use a debug Python build as a drop-in replacement of a Python release build: debug and release builds are now ABI compatible. It is no longer needed to rebuild C extensions in debug mode. My first goal was to be able to use gdb on a Python binary built with -O0: https://developers.redhat.com/articles/2021/09/08/debugging-python-c-extens… A debug Python build adds many other debug features useful to develop C extensions: https://docs.python.org/dev/using/configure.html#python-debug-build IMO with Python 3.11, it's now easy enough to get a Python debug build to develop or debug your C extensions. End users should not have to pay the price of these runtime checks. On most Linux distribution, a Python debug build is available as "python3-debug" executable. To check if you are using a debug build, check if the sys.gettotalrefcount() function is available. By the way, it is also possible to build Python in release mode with assertions using "./configure --with-assertions": https://docs.python.org/dev/using/configure.html#cmdoption-with-assertions So, what do you think? -- In 2019, I already proposed this idea on the bug tracker, but I abandoned my idea: https://bugs.python.org/issue37406 Victor -- Night gathers, and now my watch begins. It shall not end until my death.

8 11

RFC on PEP 675: Arbitrary Literal Strings
by Pradeep Kumar Srinivasan 08 Feb '22

08 Feb '22

PEP 675 [1] introduces a supertype for precise literal string types, such as `Literal["foo"]`, called `LiteralString`. The PEP allows libraries to distinguish the type of command strings from data. Powerful, command-executing libraries try to prevent security vulnerabilities by accepting arguments separately from the SQL query or shell command. However, these libraries have no way to prevent programmers from, say, using f-strings to embed arguments within the command string, which can allow malicious users to execute arbitrary commands (called "SQL/shell injection"). With this PEP, libraries can rely on type checkers to prevent such common, undesired uses. Scala has a very similar concept that is used to prevent SQL injection at compile time. [5] We also discuss the drawbacks of alternative approaches, such as security linters, full taint analysis, and NewTypes [6]. Since we have reached consensus on the PEP in typing-sig [2], we wanted to get your comments and suggestions before submitting to the Steering Council. This PEP is mainly useful for type checking. There are no changes to Python syntax or runtime behavior. Discussions: + typing-sig: [2] + BPO issue suggesting using this PEP to prevent vulnerabilities in loggers: [4] + Reddit thread on r/Python discussing this PEP: [3] Thanks, S Pradeep Kumar Graham Bleaney [1]: https://www.python.org/dev/peps/pep-0675/ [2]: https://mail.python.org/archives/list/typing-sig@python.org/thread/VB74EHNM… [3]: https://www.reddit.com/r/Python/comments/r71lzi/pep_675_arbitrary_literal_s… [4]: https://bugs.python.org/issue46200 [5]: https://www.python.org/dev/peps/pep-0675/#literal-string-types-in-scala [6]: https://www.python.org/dev/peps/pep-0675/#rejected-alternatives

3 3

"immortal" objects and how they would help per-interpreter GIL
by Eric Snow 08 Feb '22

08 Feb '22

Most of the work toward interpreter isolation and a per-interpreter GIL involves moving static global variables to _PyRuntimeState or PyInterpreterState (or module state). Through the effort of quite a few people, we've made good progress. However, many globals still remain, with the majority being objects and most of those being static strings (e.g. _Py_Identifier), static types (incl. exceptions), and singletons. On top of that, a number of those objects are exposed in the public C-API and even in the limited API. :( Dealing with this specifically is probably the trickiest thing I've had to work through in this project. There is one solution that would help both of the above in a nice way: "immortal" objects. The idea of objects that never get deallocated isn't new and has been explored here several times. Not that long ago I tried it out by setting the refcount really high. That worked. Around the same time Eddie Elizondo at Facebook did something similar but modified Py_INCREF() and Py_DECREF() to keep the refcount from changing. Our solutions were similar but with different goals in mind. (Facebook wants to avoid copy-on-write in their pre-fork model.) A while back I concluded that neither approach would work for us. The approach I had taken would have significant cache performance penalties in a per-interpreter GIL world. The approach that modifies Py_INCREF() has a significant performance penalty due to the extra branch on such a frequent operation. Recently I've come back to the idea of immortal objects because it's much simpler than the alternate (working) solution I found. So how do we get around that performance penalty? Let's say it makes CPython 5% slower. We have some options: * live with the full penalty * make other changes to reduce the penalty to a more acceptable threshold than 5% * eliminate the penalty (e.g. claw back 5% elsewhere) * abandon all hope Mark Shannon suggested to me some things we can do. Also, from a recent conversation with Dino Viehland it sounds like Eddie was able to reach performance-neutral with a few techniques. So here are some things we can do to reduce or eliminate that penalty: * reduce refcount operations on high-activity objects (e.g. None, True, False) * reduce refcount operations in general * walk the heap at the end of runtime initialization and mark all objects as immortal * mark all global objects as immortal (statics or in _PyRuntimeState; for PyInterpreterState not needed) What do you think? Does this sound realistic? Are there additional things we can do to counter that penalty? -eric

16 35

Moving away from _Py_IDENTIFIER().
by Eric Snow 08 Feb '22

08 Feb '22

I'm planning on moving us to a simpler, more efficient alternative to _Py_IDENTIFIER(), but want to see if there are any objections first before moving ahead. Also see https://bugs.python.org/issue46541. _Py_IDENTIFIER() was added in 2011 to replace several internal string object caches and to support cleaning up the cached objects during finalization. A number of "private" functions (each with a _Py_Identifier param) were added at that time, mostly corresponding to existing functions that take PyObject* or char*. Note that at present there are several hundred uses of _Py_IDENTIFIER(), including a number of duplicates. My plan is to replace our use of _Py_IDENTIFIER() with statically initialized string objects (as fields under _PyRuntimeState). That involves the following: * add a PyUnicodeObject field (not a pointer) to _PyRuntimeState for each string that currently uses _Py_IDENTIFIER() (or _Py_static_string()) * statically initialize each object as part of the initializer for _PyRuntimeState * add a macro to look up a given global string * update each location that currently uses _Py_IDENTIFIER() to use the new macro instead Pros: * reduces indirection (and extra calls) for C-API functions that need the strings (making the code a little easier to understand and speeding it up) * the objects are referenced from a fixed address in the static data section instead of the heap (speeding things up and allowing the C compiler to optimize better) * there is no lazy allocation (or lookup, etc.) so there are fewer possible failures when the objects get used (thus less error return checking) * saves memory (at little, at least) * if needed, the approach for per-interpreter is simpler * helps us get rid of several hundred static variables throughout the code base * allows us to get rid of _Py_IDENTIFIER() and a bunch of related C-API functions * "deep frozen" modules can use the global strings * commonly-used strings could be pre-allocated by adding _PyRuntimeState fields for them Cons: * a little less convenient: adding a global string requires modifying a separate file from the one where you actually want to use the string * strings can get "orphaned" (I'm planning on checking in CI) * some strings may never get used for any given ./python invocation (not that big a difference though) I have a PR up (https://github.com/python/cpython/pull/30928) that adds the global strings and replaces use of _Py_IDENTIFIER() in our code base, except for in non-builtin stdlib extension modules. (Those will be handled separately if we proceed.) The PR also adds a CI check for "orphaned" strings. It leaves _Py_IDENTIFIER() for now, but disallows any Py_BUILD_CORE code from using it. With that change I'm seeing a 1% improvement in performance (see https://github.com/faster-cpython/ideas/issues/230). I'd also like to actually get rid of _Py_IDENTIFIER(), along with other related API including ~14 (private) C-API functions. Dropping all that helps reduce maintenance costs. However, at least one PyPI project (blender) is using _Py_IDENTIFIER(). So, before we could get rid of it, we'd first have to deal with that project (and any others). To sum up, I wanted to see if there are any objections before I start merging anything. Thanks! -eric

12 36

Python-Dev] Replace debug runtime checks in release mode, with assertions in debug mode
by Matti Picus 07 Feb '22

07 Feb '22

On 7/2/22 17:55, Victor Stinner wrote: > End users should not have to > pay the price of these runtime checks. Is the cost of the checks measurable and significant? Surely sanitizing arguments to prevent segfaults is worth a few nano-seconds? Matti

1 0

PEP 674 – Disallow using macros as l-value (version 2)
by Victor Stinner 07 Feb '22

07 Feb '22

Hi, I made the following changes to the PEP since the version 1 (November 30): * Elaborate the HPy section and add a new "Relationship with the HPy project" section * Elaborate which projects are impacted and which changes are needed * Remove the PyPy section The PEP can be read online: https://python.github.io/peps/ep-0674/ And you can find the plain text below. Victor PEP: 674 Title: Disallow using macros as l-value Author: Victor Stinner <vstinner(a)python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 30-Nov-2021 Python-Version: 3.11 Abstract ======== Incompatible C API change disallowing using macros as l-value to: * Allow evolving CPython internals; * Ease the C API implementation on other Python implementation; * Help migrating existing C extensions to the HPy API. On the PyPI top 5000 projects, only 14 projects (0.3%) are affected by 4 macro changes. Moreover, 24 projects just have to regenerate their Cython code to use ``Py_SET_TYPE()``. In practice, the majority of affected projects only have to make two changes: * Replace ``Py_TYPE(obj) = new_type;`` with ``Py_SET_TYPE(obj, new_type);``. * Replace ``Py_SIZE(obj) = new_size;`` with ``Py_SET_SIZE(obj, new_size);``. Rationale ========= Using a macro as a l-value -------------------------- In the Python C API, some functions are implemented as macro because writing a macro is simpler than writing a regular function. If a macro exposes directly a structure member, it is technically possible to use this macro to not only get the structure member but also set it. Example with the Python 3.10 ``Py_TYPE()`` macro:: #define Py_TYPE(ob) (((PyObject *)(ob))->ob_type) This macro can be used as a **r-value** to **get** an object type:: type = Py_TYPE(object); It can also be used as **l-value** to **set** an object type:: Py_TYPE(object) = new_type; It is also possible to set an object reference count and an object size using ``Py_REFCNT()`` and ``Py_SIZE()`` macros. Setting directly an object attribute relies on the current exact CPython implementation. Implementing this feature in other Python implementations can make their C API implementation less efficient. CPython nogil fork ------------------ Sam Gross forked Python 3.9 to remove the GIL: the `nogil branch <https://github.com/colesbury/nogil/>`_. This fork has no ``PyObject.ob_refcnt`` member, but a more elaborated implementation for reference counting, and so the ``Py_REFCNT(obj) = new_refcnt;`` code fails with a compiler error. Merging the nogil fork into the upstream CPython main branch requires first to fix this C API compatibility issue. It is a concrete example of a Python optimization blocked indirectly by the C API. This issue was already fixed in Python 3.10: the ``Py_REFCNT()`` macro has been already modified to disallow using it as a l-value. These statements are endorsed by Sam Gross (nogil developer). HPy project ----------- The `HPy project <https://hpyproject.org/>`_ is a brand new C API for Python using only handles and function calls: handles are opaque, structure members cannot be accessed directly, and pointers cannot be dereferenced. Searching and replacing ``Py_SET_SIZE()`` is easier and safer than searching and replacing some strange macro uses of ``Py_SIZE()``. ``Py_SIZE()`` can be semi-mechanically replaced by ``HPy_Length()``, whereas seeing ``Py_SET_SIZE()`` would immediately make clear that the code needs bigger changes in order to be ported to HPy (for example by using ``HPyTupleBuilder`` or ``HPyListBuilder``). The fewer internal details exposed via macros, the easier it will be for HPy to provide direct equivalents. Any macro that references "non-public" interfaces effectively exposes those interfaces publicly. These statements are endorsed by Antonio Cuni (HPy developer). GraalVM Python -------------- In GraalVM, when a Python object is accessed by the Python C API, the C API emulation layer has to wrap the GraalVM objects into wrappers that expose the internal structure of the CPython structures (PyObject, PyLongObject, PyTypeObject, etc). This is because when the C code accesses it directly or via macros, all GraalVM can intercept is a read at the struct offset, which has to be mapped back to the representation in GraalVM. The smaller the "effective" number of exposed struct members (by replacing macros with functions), the simpler GraalVM wrappers can be. This PEP alone is not enough to get rid of the wrappers in GraalVM, but it is a step towards this long term goal. GraalVM already supports HPy which is a better solution in the long term. These statements are endorsed by Tim Felgentreff (GraalVM Python developer). Specification ============= Disallow using macros as l-value -------------------------------- PyObject and PyVarObject macros ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * ``Py_TYPE()``: ``Py_SET_TYPE()`` must be used instead * ``Py_SIZE()``: ``Py_SET_SIZE()`` must be used instead "GET" macros ^^^^^^^^^^^^ * ``PyByteArray_GET_SIZE()`` * ``PyBytes_GET_SIZE()`` * ``PyCFunction_GET_CLASS()`` * ``PyCFunction_GET_FLAGS()`` * ``PyCFunction_GET_FUNCTION()`` * ``PyCFunction_GET_SELF()`` * ``PyCell_GET()`` * ``PyCode_GetNumFree()`` * ``PyDict_GET_SIZE()`` * ``PyFunction_GET_ANNOTATIONS()`` * ``PyFunction_GET_CLOSURE()`` * ``PyFunction_GET_CODE()`` * ``PyFunction_GET_DEFAULTS()`` * ``PyFunction_GET_GLOBALS()`` * ``PyFunction_GET_KW_DEFAULTS()`` * ``PyFunction_GET_MODULE()`` * ``PyHeapType_GET_MEMBERS()`` * ``PyInstanceMethod_GET_FUNCTION()`` * ``PyList_GET_SIZE()`` * ``PyMemoryView_GET_BASE()`` * ``PyMemoryView_GET_BUFFER()`` * ``PyMethod_GET_FUNCTION()`` * ``PyMethod_GET_SELF()`` * ``PySet_GET_SIZE()`` * ``PyTuple_GET_SIZE()`` * ``PyUnicode_GET_DATA_SIZE()`` * ``PyUnicode_GET_LENGTH()`` * ``PyUnicode_GET_LENGTH()`` * ``PyUnicode_GET_SIZE()`` * ``PyWeakref_GET_OBJECT()`` "AS" macros ^^^^^^^^^^^ * ``PyByteArray_AS_STRING()`` * ``PyBytes_AS_STRING()`` * ``PyFloat_AS_DOUBLE()`` * ``PyUnicode_AS_DATA()`` * ``PyUnicode_AS_UNICODE()`` PyUnicode macros ^^^^^^^^^^^^^^^^ * ``PyUnicode_1BYTE_DATA()`` * ``PyUnicode_2BYTE_DATA()`` * ``PyUnicode_4BYTE_DATA()`` * ``PyUnicode_DATA()`` * ``PyUnicode_IS_ASCII()`` * ``PyUnicode_IS_COMPACT()`` * ``PyUnicode_IS_READY()`` * ``PyUnicode_KIND()`` * ``PyUnicode_READ()`` * ``PyUnicode_READ_CHAR()`` PyDateTime "GET" macros ^^^^^^^^^^^^^^^^^^^^^^^ * ``PyDateTime_DATE_GET_FOLD()`` * ``PyDateTime_DATE_GET_HOUR()`` * ``PyDateTime_DATE_GET_MICROSECOND()`` * ``PyDateTime_DATE_GET_MINUTE()`` * ``PyDateTime_DATE_GET_SECOND()`` * ``PyDateTime_DATE_GET_TZINFO()`` * ``PyDateTime_DELTA_GET_DAYS()`` * ``PyDateTime_DELTA_GET_MICROSECONDS()`` * ``PyDateTime_DELTA_GET_SECONDS()`` * ``PyDateTime_GET_DAY()`` * ``PyDateTime_GET_MONTH()`` * ``PyDateTime_GET_YEAR()`` * ``PyDateTime_TIME_GET_FOLD()`` * ``PyDateTime_TIME_GET_HOUR()`` * ``PyDateTime_TIME_GET_MICROSECOND()`` * ``PyDateTime_TIME_GET_MINUTE()`` * ``PyDateTime_TIME_GET_SECOND()`` * ``PyDateTime_TIME_GET_TZINFO()`` PyDescr macros ^^^^^^^^^^^^^^ * ``PyDescr_NAME()`` * ``PyDescr_TYPE()`` Port C extensions to Python 3.11 -------------------------------- In practice, the majority of projects affected by these PEP only have to make two changes: * Replace ``Py_TYPE(obj) = new_type;`` with ``Py_SET_TYPE(obj, new_type);``. * Replace ``Py_SIZE(obj) = new_size;`` with ``Py_SET_SIZE(obj, new_size);``. The `pythoncapi_compat project <https://github.com/pythoncapi/pythoncapi_compat>`_ can be used to update automatically C extensions: add Python 3.11 support without losing support with older Python versions. The project provides a header file which provides ``Py_SET_REFCNT()``, ``Py_SET_TYPE()`` and ``Py_SET_SIZE()`` functions to Python 3.8 and older. PyTuple_GET_ITEM() and PyList_GET_ITEM() ---------------------------------------- The ``PyTuple_GET_ITEM()`` and ``PyList_GET_ITEM()`` macros are left unchanged. The code pattern ``&PyTuple_GET_ITEM(tuple, 0)`` and ``&PyList_GET_ITEM(list, 0)`` is still commonly used to get access to the inner ``PyObject**`` array. Changing these macros is out of the scope of this PEP. Backwards Compatibility ======================= The proposed C API changes are backward incompatible on purpose. At December 1, 2021, a code search on the PyPI top 5000 projects (4760 projects in practice, others don't have a source archive) found that `only 14 projects are affected <https://bugs.python.org/issue45476#msg407456>`_ (0.3%): * datatable (1.0.0) * frozendict (2.1.1) * guppy3 (3.1.2) * M2Crypto (0.38.0) * mecab-python3 (1.0.4) * mypy (0.910) * Naked (0.1.31) * pickle5 (0.0.12) * pysha3 (1.0.2) * python-snappy (0.6.0) * recordclass (0.16.3) * scipy (1.7.3) * zodbpickle (2.2.0) * zstd (1.5.0.2) These 14 projects only use 4 macros as l-value: * ``PyDescr_NAME()`` and ``PyDescr_TYPE()`` (2 projects) * ``Py_SIZE()`` (8 projects) * ``Py_TYPE()`` (4 projects) Moreover, `24 projects just have to regenerate their Cython code <https://bugs.python.org/issue45476#msg407416>`_ to use ``Py_SET_TYPE()``. This change does not follow the PEP 387 deprecation process. There is no known way to emit a deprecation warning only when a macro is used as a l-value, but not when it's used differently (ex: as a r-value). Relationship with the HPy project ================================= The HPy project --------------- The hope with the HPy project is indeed to provide a C API that is close to the original API to make porting easy and have it perform as close to the existing API as possible. At the same time, HPy is sufficently removed to be a good "C extension API" (as opposed to a stable subset of the CPython implementation API) that does not leak implementation details. To ensure this latter property is why the HPy project tries to develop everything in parallel for CPython, PyPy, and GraalVM Python. HPy is still evolving very fast. Issues are still being solved while migrating NumPy, and work has begun on adding support for HPy to Cython. Work on pybind11 is starting soon. Tim Felgentreff believes by the time HPy has these users of the existing C API working, HPy should be in a state where it is generally useful and can be deemed stable enough that further development can follow a more stable process. In the long run the HPy project would like to become a promoted API to write Python C extensions. The HPy project is a good solution for the long term. It has the advantage of being developed outside Python and it doesn't require any C API change. The C API is here is stay for a few more years ---------------------------------------------- The first concern about HPy is that right now, HPy is not mature nor widely used, and CPython still has to continue supporting a large amount of C extensions which are not likely to be ported to HPy soon. The second concern is the inability to evolve CPython internals to implement new optimizations, and the inefficient implementation of the current C API in PyPy, GraalPython, etc. Sadly, HPy will only solve these problems when most C extensions will be fully ported to HPy: when it will become reasonable to consider dropping the "legacy" Python C API. While porting a C extension to HPy can be done incrementally on CPython, it requires to modify a lot of code and takes time. Porting most C extensions to HPy is expected to take a few years. This PEP proposes to make the C API "less bad" by fixing one problem which is clearily identified as causing practical issues: macros used as l-values. This PEP only requires to update a minority of C extensions, and usually only a few lines need to be changed in impacted extensions. For example, numpy 1.22 is made of 307,300 lines of C code, and adapting numpy to the this PEP only modified 11 lines (use Py_SET_TYPE and Py_SET_SIZE) and adding 4 lines (to define Py_SET_TYPE and Py_SET_SIZE for Python 3.8 and older). Porting numpy to HPy is expected to require modifying more lines than that. Right now, it's hard to bet which approach is the best: fixing the current C API, or focusing on HPy. It would be risky to only focus on HPy. Rejected Idea: Leave the macros as they are =========================================== The documentation of each function can discourage developers to use macros to modify Python objects. If these is a need to make an assignment, a setter function can be added and the macro documentation can require to use the setter function. For example, a ``Py_SET_TYPE()`` function has been added to Python 3.9 and the ``Py_TYPE()`` documentation now requires to use the ``Py_SET_TYPE()`` function to set an object type. If developers use macros as l-value, it's their responsibility when their code breaks, not the Python responsibility. We are operating under the consenting adults principle: we expect users of the Python C API to use it as documented and expect them to take care of the fallout, if things break when they don't. This idea was rejected because only few developers read the documentation, and only a minority is tracking changes of the Python C API documentation. The majority of developers are only using CPython and so are not aware of compatibility issues with other Python implementations. Moreover, continuing to allow using macros as l-value does not help the HPy project and leaves the burden of emulating them on GraalVM's Python implementation. Macros already modified ======================= The following C API macros have already been modified to disallow using them as l-value: * ``PyCell_SET()`` * ``PyList_SET_ITEM()`` * ``PyTuple_SET_ITEM()`` * ``Py_REFCNT()`` (Python 3.10): ``Py_SET_REFCNT()`` must be used * ``_PyGCHead_SET_FINALIZED()`` * ``_PyGCHead_SET_NEXT()`` * ``asdl_seq_GET()`` * ``asdl_seq_GET_UNTYPED()`` * ``asdl_seq_LEN()`` * ``asdl_seq_SET()`` * ``asdl_seq_SET_UNTYPED()`` For example, ``PyList_SET_ITEM(list, 0, item) < 0`` now fails with a compiler error as expected. Discussion ========== * `PEP 674: Disallow using macros as l-value <https://mail.python.org/archives/list/python-dev@python.org/thread/KPIJPPJ6…>`_ (Nov 30, 2021) References ========== * `Python C API: Add functions to access PyObject <https://vstinner.github.io/c-api-abstract-pyobject.html>`_ (October 2021) article by Victor Stinner * `[C API] Disallow using PyFloat_AS_DOUBLE() as l-value <https://bugs.python.org/issue45476>`_ (October 2021) * `[capi-sig] Py_TYPE() and Py_SIZE() become static inline functions <https://mail.python.org/archives/list/capi-sig@python.org/thread/WGRLTHTHC3…>`_ (September 2021) * `[C API] Avoid accessing PyObject and PyVarObject members directly: add Py_SET_TYPE() and Py_IS_TYPE(), disallow Py_TYPE(obj)=type <https://bugs.python.org/issue39573>`__ (February 2020) * `bpo-30459: PyList_SET_ITEM could be safer <https://bugs.python.org/issue30459>`_ (May 2017) Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.

3 7

I want to contribute to Python.
by Ezekiel Adetoro 06 Feb '22

06 Feb '22

Hello, My name is Ezekiel, and it is my desire to start contributing to Python, be part of the core development of Python. I have forked the CPython and cloned it. What is the next step I need to do?

3 2

PyPy on PySide6 is there: PyPy with a Gui
by Christian Tismer-Sperling 05 Feb '22

05 Feb '22

Hi Guido et. al., since May 2021 I have been working at running PyPy on PySide6, which was a difficult undertaking, since PyPy internally is quite a bit different. I declared the project to be ready-to-use when the Mandelbrot example of the PySide examples (examples/corelib/threads/mandelbrot.py) is working. This was finally solved this week on 2022-02-01, so we have the first advanced Gui working with PyPy and with the amazing result of speed: PyPy 3.8 works 10 times faster than the identical code on Python 3.10 and 5.5 times slower than the same example in C++ Qt. I think to send an official announce when this is available on pip. This effort marks the completion of my PyPy support, which began in 2003 and ended involuntarily in 2006 due to a stroke. All the best -- Chris -- Christian Tismer-Sperling :^) tismer(a)stackless.com Software Consulting : http://www.stackless.com/ Strandstraße 37 : https://github.com/PySide 24217 Schönberg : GPG key -> 0xFB7BEE0E phone +49 173 24 18 776 fax +49 (30) 700143-0023

3 3

Summary of Python tracker Issues
by Python tracker 04 Feb '22

04 Feb '22

ACTIVITY SUMMARY (2022-01-28 - 2022-02-04) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7144 (-13) closed 51222 (+84) total 58366 (+71) Open issues with patches: 2890 Issues opened (44) ================== #45711: Simplify the interpreter's (type, val, tb) exception represent https://bugs.python.org/issue45711 reopened by vstinner #45773: Compiler hangs during jump elimination https://bugs.python.org/issue45773 reopened by brandtbucher #46570: Windows support for OpenSSL 3.0 https://bugs.python.org/issue46570 opened by jay0lee #46571: Strange `(a)typing.no_type_check` behavior for class variables https://bugs.python.org/issue46571 opened by sobolevn #46575: One-off errors in hashlib.scrypt error messages https://bugs.python.org/issue46575 opened by ron_kaminsky #46577: Hostname spoofing via backslashes in URL https://bugs.python.org/issue46577 opened by meetdash #46578: cant DEBUG os.spawnv() https://bugs.python.org/issue46578 opened by michaellongge #46580: email.utils.unquote strips too many slashes https://bugs.python.org/issue46580 opened by hbandi #46581: _typevar_types and _paramspec_tvars are missing from _GenericA https://bugs.python.org/issue46581 opened by posita #46585: Should we re-export `PyObj_FromPtr` in `ctypes`? https://bugs.python.org/issue46585 opened by sobolevn #46586: In documentation contents enum.property erroneously links to b https://bugs.python.org/issue46586 opened by Dutcho #46587: datetime and time tests use non-portal "%4Y" format https://bugs.python.org/issue46587 opened by christian.heimes #46589: Improve documentation for typing._GenericAlias https://bugs.python.org/issue46589 opened by matthew.rahtz #46592: Undocumented behavior in strptime for ISO week dates https://bugs.python.org/issue46592 opened by Jonatan Skogsfors #46593: memoryview lacks support for half floats https://bugs.python.org/issue46593 opened by pitrou #46594: Windows "Edit with IDLE >" only has one selection https://bugs.python.org/issue46594 opened by terry.reedy #46596: PyLineTable_InitAddressRange isn't exported - causing C Extens https://bugs.python.org/issue46596 opened by nathan3 #46598: ElementTree: wrong XML prolog for the utf-8-sig encoding https://bugs.python.org/issue46598 opened by prikryl #46600: Python built with clang -O0 allocates 10x more stack memory th https://bugs.python.org/issue46600 opened by vstinner #46601: macOS installer "Install Certificates.command" fails if pip is https://bugs.python.org/issue46601 opened by cryptophoto #46603: `typing._strip_annotations` is not fully covered https://bugs.python.org/issue46603 opened by sobolevn #46604: Documentation fix in ssl module https://bugs.python.org/issue46604 opened by glk0 #46605: Py_XDECREF() module on fail in Py_mod_exec https://bugs.python.org/issue46605 opened by ov2k #46606: Large C stack usage of os.getgroups() and os.setgroups() https://bugs.python.org/issue46606 opened by methane #46607: Add DeprecationWarning to configparser's LegacyInterpolation https://bugs.python.org/issue46607 opened by hugovk #46608: Exclude marshalled-frozen data if deep-freezing to save 300 KB https://bugs.python.org/issue46608 opened by kumaraditya303 #46609: Generator-based coroutines in Python 3.10 docs https://bugs.python.org/issue46609 opened by srittau #46611: Improve coverage of `__instancecheck__` and `__subclasscheck__ https://bugs.python.org/issue46611 opened by sobolevn #46613: Add PyType_GetModuleByDef to the public & limited API https://bugs.python.org/issue46613 opened by petr.viktorin #46614: Add option to output UTC datetimes as "Z" in `.isoformat()` https://bugs.python.org/issue46614 opened by p-ganssle #46615: Use-after-free by mutating set during set operations https://bugs.python.org/issue46615 opened by Dennis Sweeney #46619: lazy module property not recognized by doctests https://bugs.python.org/issue46619 opened by jaraco #46620: Documentation of ipaddress behavior for prefix length with lea https://bugs.python.org/issue46620 opened by lay20114 #46621: Should map(function, iterable, ...) replace StopIteration with https://bugs.python.org/issue46621 opened by xavieryao #46622: Support decorating a coroutine with functools.cached_property https://bugs.python.org/issue46622 opened by uranusjr #46623: test_zlib: test_pair() and test_speech128() fail with s390x ha https://bugs.python.org/issue46623 opened by vstinner #46625: timeout option of socket.create_connection is not respected https://bugs.python.org/issue46625 opened by Nicolas SURRIBAS #46631: Implement a "strict" mode for getpass.getuser() https://bugs.python.org/issue46631 opened by eryksun #46632: test_ssl: 2 tests fail on cstratak-CentOS9-fips-x86_64 https://bugs.python.org/issue46632 opened by vstinner #46633: AddressSanitizer: Skip tests directly in Python, not with exte https://bugs.python.org/issue46633 opened by vstinner #46634: [sqlite3]��speed up cursor.execute*() https://bugs.python.org/issue46634 opened by erlendaasland #46635: unittest.defaultTestLoader.discover fails for namespace packag https://bugs.python.org/issue46635 opened by tpapastylianou #46637: Incorrect error message: "missing 1 required positional argume https://bugs.python.org/issue46637 opened by Anders.Hovm��ller #46638: Inconsistent registry virtualization in Windows Store package https://bugs.python.org/issue46638 opened by steve.dower Most recent 15 issues with no replies (15) ========================================== #46635: unittest.defaultTestLoader.discover fails for namespace packag https://bugs.python.org/issue46635 #46634: [sqlite3]��speed up cursor.execute*() https://bugs.python.org/issue46634 #46632: test_ssl: 2 tests fail on cstratak-CentOS9-fips-x86_64 https://bugs.python.org/issue46632 #46631: Implement a "strict" mode for getpass.getuser() https://bugs.python.org/issue46631 #46625: timeout option of socket.create_connection is not respected https://bugs.python.org/issue46625 #46623: test_zlib: test_pair() and test_speech128() fail with s390x ha https://bugs.python.org/issue46623 #46622: Support decorating a coroutine with functools.cached_property https://bugs.python.org/issue46622 #46620: Documentation of ipaddress behavior for prefix length with lea https://bugs.python.org/issue46620 #46619: lazy module property not recognized by doctests https://bugs.python.org/issue46619 #46614: Add option to output UTC datetimes as "Z" in `.isoformat()` https://bugs.python.org/issue46614 #46613: Add PyType_GetModuleByDef to the public & limited API https://bugs.python.org/issue46613 #46611: Improve coverage of `__instancecheck__` and `__subclasscheck__ https://bugs.python.org/issue46611 #46609: Generator-based coroutines in Python 3.10 docs https://bugs.python.org/issue46609 #46606: Large C stack usage of os.getgroups() and os.setgroups() https://bugs.python.org/issue46606 #46605: Py_XDECREF() module on fail in Py_mod_exec https://bugs.python.org/issue46605 Most recent 15 issues waiting for review (15) ============================================= #46638: Inconsistent registry virtualization in Windows Store package https://bugs.python.org/issue46638 #46634: [sqlite3]��speed up cursor.execute*() https://bugs.python.org/issue46634 #46623: test_zlib: test_pair() and test_speech128() fail with s390x ha https://bugs.python.org/issue46623 #46615: Use-after-free by mutating set during set operations https://bugs.python.org/issue46615 #46613: Add PyType_GetModuleByDef to the public & limited API https://bugs.python.org/issue46613 #46611: Improve coverage of `__instancecheck__` and `__subclasscheck__ https://bugs.python.org/issue46611 #46608: Exclude marshalled-frozen data if deep-freezing to save 300 KB https://bugs.python.org/issue46608 #46607: Add DeprecationWarning to configparser's LegacyInterpolation https://bugs.python.org/issue46607 #46606: Large C stack usage of os.getgroups() and os.setgroups() https://bugs.python.org/issue46606 #46604: Documentation fix in ssl module https://bugs.python.org/issue46604 #46603: `typing._strip_annotations` is not fully covered https://bugs.python.org/issue46603 #46600: Python built with clang -O0 allocates 10x more stack memory th https://bugs.python.org/issue46600 #46589: Improve documentation for typing._GenericAlias https://bugs.python.org/issue46589 #46581: _typevar_types and _paramspec_tvars are missing from _GenericA https://bugs.python.org/issue46581 #46571: Strange `(a)typing.no_type_check` behavior for class variables https://bugs.python.org/issue46571 Top 10 most discussed issues (10) ================================= #45711: Simplify the interpreter's (type, val, tb) exception represent https://bugs.python.org/issue45711 13 msgs #46600: Python built with clang -O0 allocates 10x more stack memory th https://bugs.python.org/issue46600 13 msgs #46571: Strange `(a)typing.no_type_check` behavior for class variables https://bugs.python.org/issue46571 11 msgs #29688: Add support for Path.absolute() https://bugs.python.org/issue29688 7 msgs #46521: codeop._maybe_compile passes code with error + triple quotes https://bugs.python.org/issue46521 7 msgs #46555: Unicode-mangled names refer inconsistently to constants https://bugs.python.org/issue46555 7 msgs #46615: Use-after-free by mutating set during set operations https://bugs.python.org/issue46615 7 msgs #42238: Deprecate suspicious.py? https://bugs.python.org/issue42238 6 msgs #45975: Simplify some while-loops with walrus operator https://bugs.python.org/issue45975 6 msgs #33222: Various test failures if PYTHONUSERBASE is not canonicalized https://bugs.python.org/issue33222 5 msgs Issues closed (73) ================== #6682: Default traceback does not handle PEP302 loaded modules https://bugs.python.org/issue6682 closed by iritkatriel #12010: Compile fails when sizeof(wchar_t) == 1 https://bugs.python.org/issue12010 closed by iritkatriel #26276: Inconsistent behaviour of PEP 3101 formatting between versions https://bugs.python.org/issue26276 closed by eric.smith #26552: Failing ensure_future still creates a Task https://bugs.python.org/issue26552 closed by gvanrossum #27101: Compilation of python (modules) for foreign target platform pr https://bugs.python.org/issue27101 closed by iritkatriel #29155: test.test_spwd.TestSpwdNonRoot failure with FileNotFoundError https://bugs.python.org/issue29155 closed by iritkatriel #29172: blake2: Use lowest-common denominator signature of #pragma pac https://bugs.python.org/issue29172 closed by iritkatriel #30210: No Documentation on tkinter dnd module https://bugs.python.org/issue30210 closed by iritkatriel #30393: test_readline hangs https://bugs.python.org/issue30393 closed by iritkatriel #36346: Prepare for removing the legacy Unicode C API https://bugs.python.org/issue36346 closed by methane #37588: Py_DEPRECATED and unavoidable warnings https://bugs.python.org/issue37588 closed by iritkatriel #38682: struct timeval is not declared https://bugs.python.org/issue38682 closed by iritkatriel #38902: image/webp support in mimetypes https://bugs.python.org/issue38902 closed by eric.araujo #39339: ProcessPoolExecutor() Exception in thread QueueManagerThread https://bugs.python.org/issue39339 closed by iritkatriel #39711: SIGBUS and core dumped during tests of 3.8.1 https://bugs.python.org/issue39711 closed by iritkatriel #40746: test_gdb failing on 32-bit armv7l when built with GCC -Og: <cl https://bugs.python.org/issue40746 closed by gregory.p.smith #42784: issues with object.h includes https://bugs.python.org/issue42784 closed by iritkatriel #43012: Remove pathlib accessors https://bugs.python.org/issue43012 closed by eric.araujo #43320: test test_webbrowser "can't locate runnable browser" with enab https://bugs.python.org/issue43320 closed by iritkatriel #43359: Dead assignment in Py_UniversalNewlineFgets https://bugs.python.org/issue43359 closed by iritkatriel #43540: importlib: Document how to replace load_module() in What's New https://bugs.python.org/issue43540 closed by vstinner #43583: make test failures, 2 tests failed: test_embed test_tabnan https://bugs.python.org/issue43583 closed by iritkatriel #44031: test_embed and test_tabnanny fails if the current directory is https://bugs.python.org/issue44031 closed by iritkatriel #44359: test_ftplib.test_makeport() fails as "env changes" if a socket https://bugs.python.org/issue44359 closed by vstinner #44977: Deprecate delegation of int to __trunc__ https://bugs.python.org/issue44977 closed by serhiy.storchaka #45459: Limited API support for Py_buffer https://bugs.python.org/issue45459 closed by christian.heimes #45555: Object stays alive for weak reference if an exception happens https://bugs.python.org/issue45555 closed by iritkatriel #45703: importlib.invalidate_caches() does not invalidate _NamespacePa https://bugs.python.org/issue45703 closed by petr.viktorin #45925: Upgrade macOS and Windows installers to use SQLite 3.37.2 https://bugs.python.org/issue45925 closed by ned.deily #45990: Exception notes need more documentation https://bugs.python.org/issue45990 closed by iritkatriel #46101: argparse: using parents & subcommands, options can be ignored https://bugs.python.org/issue46101 closed by paul.j3 #46440: ArgumentParser.parse_args exits on missing required argument w https://bugs.python.org/issue46440 closed by paul.j3 #46451: Tracing causes module globals to be mutated when calling funct https://bugs.python.org/issue46451 closed by seberg #46458: Optimise try-except code generation for the happy path https://bugs.python.org/issue46458 closed by iritkatriel #46482: `typing.Annotation.__new__` is not covered https://bugs.python.org/issue46482 closed by sobolevn #46483: `pathlib.PurePath.__class_getitem__` does not return `GenericA https://bugs.python.org/issue46483 closed by serhiy.storchaka #46487: `_SSLProtocolTransport` doesn't have the `get_write_buffer_lim https://bugs.python.org/issue46487 closed by asvetlov #46489: webbrowser crashes Ubuntu kernel https://bugs.python.org/issue46489 closed by dizcza #46524: test_peg_generator takes 8 minutes on Windows https://bugs.python.org/issue46524 closed by gregory.p.smith #46540: dylibs not loading properly from NFS mounts https://bugs.python.org/issue46540 closed by ned.deily #46542: test_json and test_lib2to3 crash on s390x Fedora Clang 3.x bui https://bugs.python.org/issue46542 closed by vstinner #46550: __slots__ updates despite being read-only https://bugs.python.org/issue46550 closed by ronaldoussoren #46560: ParamSpec: Typo in doc string https://bugs.python.org/issue46560 closed by kj #46563: Add tests regarding context of exception in a finally block https://bugs.python.org/issue46563 closed by iritkatriel #46568: non awaited coroutines on a IsolatedAsyncioTestCase results on https://bugs.python.org/issue46568 closed by asvetlov #46569: final note on StrEnum documentation incorrectly refers to int. https://bugs.python.org/issue46569 closed by ethan.furman #46572: Unicode identifiers not necessarily unique https://bugs.python.org/issue46572 closed by eryksun #46573: Python modules such as pyglet or pygame crash Python when tkin https://bugs.python.org/issue46573 closed by ronaldoussoren #46574: itertools.count should work with non-number types https://bugs.python.org/issue46574 closed by rhettinger #46576: test_peg_generator is extremely slow https://bugs.python.org/issue46576 closed by gregory.p.smith #46579: Compiling from source prints ��ModuleNotFoundError: No module https://bugs.python.org/issue46579 closed by christian.heimes #46582: Windows builds fail: fatal error RC1116: RC terminating after https://bugs.python.org/issue46582 closed by vstinner #46583: Modernize `selectors.py` by removing unused `sys.version_info` https://bugs.python.org/issue46583 closed by asvetlov #46584: Modernize `ctypes/test_python_api` by removing old version che https://bugs.python.org/issue46584 closed by zach.ware #46588: IDLE fix typo in test_calltip.py https://bugs.python.org/issue46588 closed by caioagiani #46590: I suggests to modify Include/object.h in 98 line https://bugs.python.org/issue46590 closed by 4d656f77 #46591: Make About IDLE links clickable https://bugs.python.org/issue46591 closed by terry.reedy #46595: [doc] Undocumented behavior of importlib.metadata.version https://bugs.python.org/issue46595 closed by cowlinator #46597: Remove Python 3.3 compatibility code from overlapped.c https://bugs.python.org/issue46597 closed by kumaraditya303 #46599: Objects/object.c:767:24: runtime error: member access within n https://bugs.python.org/issue46599 closed by christian.heimes #46602: Subtle trouble with heredoc append in configure. https://bugs.python.org/issue46602 closed by ned.deily #46610: assertCountEqual doesn't work as expected for dictionary eleme https://bugs.python.org/issue46610 closed by serhiy.storchaka #46612: Unclear behavior of += operator https://bugs.python.org/issue46612 closed by eric.smith #46616: test_importlib leaves stray registry entries on Windows https://bugs.python.org/issue46616 closed by steve.dower #46617: CSV Creation occasional off by one error https://bugs.python.org/issue46617 closed by matthewstidham #46618: Exponent operator(**) interpreter issue https://bugs.python.org/issue46618 closed by tim.peters #46624: random.randrange removed support for non-integer types after j https://bugs.python.org/issue46624 closed by rhettinger #46626: expose IP_BIND_ADDRESS_NO_PORT linux socket option https://bugs.python.org/issue46626 closed by benjamin.peterson #46627: Regex hangs indefinitely https://bugs.python.org/issue46627 closed by tim.peters #46628: Can't install YARL https://bugs.python.org/issue46628 closed by eric.smith #46629: Cannot sideload MSIX package on Windows https://bugs.python.org/issue46629 closed by steve.dower #46630: IDLE: Set query focus to entry box on Windows https://bugs.python.org/issue46630 closed by terry.reedy #46636: Bugs of 2to3 on built-in function or types(classes) https://bugs.python.org/issue46636 closed by gregory.p.smith

1 0

[RELEASE] Python 3.11.0a5 is available
by Pablo Galindo Salgado 03 Feb '22

03 Feb '22

We needed to tame some angry buildbots, but after a small fight, we won with just some scratches! Here you have a shiny new alpha release: Python 3.11.0a5. https://www.python.org/downloads/release/python-3110a5/ **This is an early developer preview of Python 3.11** # Major new features of the 3.11 series, compared to 3.10 Python 3.11 is still in development. This release, 3.11.0a5 is the fifth of seven planned alpha releases. Alpha releases are intended to make it easier to test the current state of new features and bug fixes and to test the release process. During the alpha phase, features may be added up until the start of the beta phase (2022-05-06) and, if necessary, may be modified or deleted up until the release candidate phase (2022-08-01). Please keep in mind that this is a preview release and its use is **not** recommended for production environments. Many new features for Python 3.11 are still being planned and written. Among the new major new features and changes so far: * [PEP 657](https://www.python.org/dev/peps/pep-0657/) -- Include Fine-Grained Error Locations in Tracebacks * [PEP 654](https://www.python.org/dev/peps/pep-0654/) -- Exception Groups and except* * [PEP 673](https://www.python.org/dev/peps/pep-0673/) -- Self Type * [PEP 646](https://www.python.org/dev/peps/pep-0646/)-- Variadic Generics * The [Faster Cpython Project](https://github.com/faster-cpython) is already yielding some exciting results: this version of CPython 3.11 is ~12% faster on the geometric mean of the [PyPerformance benchmarks]( speed.python.org), compared to 3.10.0. * Hey, **fellow core developer,** if a feature you find important is missing from this list, let me know. The next pre-release of Python 3.11 will be 3.11.0a6, currently scheduled for Monday, 2022-02-28. # More resources * [Online Documentation](https://docs.python.org/3.11/) * [PEP 664](https://www.python.org/dev/peps/pep-0664/), 3.11 Release Schedule * Report bugs at [https://bugs.python.org](https://bugs.python.org). * [Help fund Python and its community](/psf/donations/). # And now for something completely different In physics, the Poynting vector (Umov-Poynting vector) represents the directional energy flux (the energy transfer per unit area per unit time) or power flow of an electromagnetic field. It is named after its discoverer John Henry Poynting who first derived it in 1884. Oliver Heaviside also discovered it independently in the more general form that recognises the freedom of adding the curl of an arbitrary vector field to the definition. The Poynting vector is used throughout electromagnetics in conjunction with Poynting’s theorem, the continuity equation expressing conservation of electromagnetic energy, to calculate the power flow in electromagnetic fields. # We hope you enjoy those new releases! Thanks to all of the many volunteers who help make Python Development and these releases possible! Please consider supporting our efforts by volunteering yourself or through organization contributions to the Python Software Foundation. Your friendly release team, Pablo Galindo @pablogsal Ned Deily @nad Steve Dower @steve.dower

1 0

Re: Please update Cython *before* introcuding C API incompatible changes in Python
by Thomas Caswell 02 Feb '22

02 Feb '22

I disagree with point (3). I think it would be better to discourage projects from including the output of cython in their sdists. They should either have cython as a build-time requirement or provide built wheels (which are specific a platform and CPython version). The middle ground of not expecting the user to have cython while expecting them to have a working c-complier is a very narrow case and I think asking those users to install cython is worth the forward compatibility for Python versions you get by requiring people installing from source to re-cythonize. Tom -- Thomas Caswell tcaswell(a)gmail.com

5 7

Increase of Spammy PRs and PR reviews
by Lrupert 01 Feb '22

01 Feb '22

As someone who is watching the python/cpython repository, I'm very used to see lots of traffic. But lately there have been a surge of spammy PRs which are about the same, generally very trivial subject but individually fixing each occurrence one by one: - Add coverage to X (tens of them, as separate PRs) - Make `test_xxx` executable with direct invocation (tens of them, as separate PRs) - Lint source with flake8, fix linting errors (stylistic changes) And lots of non-committer PR reviews that only approve. Am I the only one who feels like this is only done to 'grind' commits (get a higher number of commits into the repository, and boast about it)? In the past there were one or two people who would submit typo fixes, but most of them weren't making it continuously. The situation right now feels much worse than those.

21 29