Mailman 3 March 2022 - Python-Dev

Proposal: Allow non-default after default arguments
by Sebastian Rittau Oct. 16, 2023

Oct. 16, 2023

Currently, Python doesn't allow non-default arguments after default arguments: >>> def foo(x=None, y): pass File "<stdin>", line 1 def foo(x=None, y): pass ^ SyntaxError: non-default argument follows default argument I believe that at the time this was introduced, no use cases for this were known and this is is supposed to … [View More]prevent a source of bugs. I have two use cases for this, one fringe, but valid, the other more important: The fringe use case: Suppose you have a function that takes a 2D coordinate value as separate "x" and "y" arguments. The "x" argument is optional, the "y" argument isn't. Currently there are two ways to do this, none of them particularly great: def foo(y, x): # reverse x and y arguments, confusing ... def foo(x, y=None): # Treat the x argument as y if only one argument is provided if y is None: x, y = y, x ... To me, the "natural" solution looks like this: def foo(x=None, y): ... # Called like this: foo(1, 2) foo(y=2) This could also be useful when evolving APIs. For example there is a function "bar" that takes two required arguments. In a later version, the first argument gains a useful default, the second doesn't. There is no sensible way to evolve the API at the moment. The more important use case involves @overloads. A condensed example of a function where the return type depends on an "encoding" parameter, followed by further parameters that could be called like this: foo(123, "utf-8") # would return bytes foo(encoding="utf-8") foo(123, None) # would return str foo(encoding=None) foo(x=123) # would return str foo() This could ideally be written as: @overload def foo(x: int = ..., encoding: None = ...) -> str: ... @overload def foo(x: int = ..., encoding: str) -> bytes: ... # plus the actual implementation But due to the syntax constraint, this needs to be hacked around with a third overload: @overload def foo(x: int = ... encoding: None = ...) -> str: ... @overload def foo(x: int, encoding: str) -> bytes: ... # for foo(123, "utf-8") @overload def foo(*, encoding: str) -> bytes: ... # for foo(encoding="utf-8") Not only is this hard to read, real examples in typeshed are usually more complex, with many arguments before or after the affected argument or even multiple affected arguments. This often becomes too complex to write or maintain. Here is one example from the wild: https://github.com/python/typeshed/blob/b95b729b9e07ab21d252701af0f5b740467… Allowing non-default arguments after default arguments would solve both use cases above and eliminates a special case. I'm also not sure what exactly the current SyntaxError really protects us from. Adding a non-default after a default argument can't really lead bugs. - Sebastian [View Less]

11 14

Python multithreading without the GIL
by Sam Gross July 31, 2023

July 31, 2023

Hi, I've been working on changes to CPython to allow it to run without the global interpreter lock. I'd like to share a working proof-of-concept that can run without the GIL. The proof-of-concept involves substantial changes to CPython internals, but relatively few changes to the C-API. It is compatible with many C extensions: extensions must be rebuilt, but usually require small or no modifications to source code. I've built compatible versions of packages from the scientific Python ecosystem,… [View More]

24 48

Mixed Python/C debugging
by Skip Montanaro April 26, 2023

April 26, 2023

Having tried comp.lang.python with no response, I turn here... After at least ten years away from Python's run-time interpreter & byte code compiler, I'm getting set to familiarize myself with that again. This will, I think, entail debugging a mixed Python/C environment. I'm an Emacs user and am aware that GDB since 7.0 has support for debugging at the Python code level. Is Emacs+GDB my best bet? Are there any Python IDEs which support C-level breakpoints and debugging? Thanks, Skip

5 6

PEP 448 review
by Guido van Rossum March 29, 2023

March 29, 2023

I'm back, I've re-read the PEP, and I've re-read the long thread with "(no subject)". I think Georg Brandl nailed it: """ *I like the "sequence and dict flattening" part of the PEP, mostly because itis consistent and should be easy to understand, but the comprehension syntaxenhancements seem to be bad for readability and "comprehending" what the codedoes.The call syntax part is a mixed bag on the one hand it is nice to be consistent with the extended possibilities in literals (… [View More]flattening), but on the other hand there would be small but annoying inconsistencies anyways (e.g. the duplicate kwarg case above).* """ Greg Ewing followed up explaining that the inconsistency between dict flattening and call syntax is inherent in the pre-existing different rules for dicts vs. keyword args: {'a':1, 'a':2} results in {'a':2}, while f(a=1, a=2) is an error. (This form is a SyntaxError; the dynamic case f(a=1, **{'a': 1}) is a TypeError.) For me, allowing f(*a, *b) and f(**d, **e) and all the other combinations for function calls proposed by the PEP is an easy +1 -- it's a straightforward extension of the existing pattern, and anybody who knows what f(x, *a) does will understand f(x, *a, y, *b). Guessing what f(**d, **e) means shouldn't be hard either. Understanding the edge case for duplicate keys with f(**d, **e) is a little harder, but the error messages are pretty clear, and it is not a new edge case. The sequence and dict flattening syntax proposals are also clean and logical -- we already have *-unpacking on the receiving side, so allowing *x in tuple expressions reads pretty naturally (and the similarity with *a in argument lists certainly helps). From here, having [a, *x, b, *y] is also natural, and then the extension to other displays is natural: {a, *x, b, *y} and {a:1, **d, b:2, **e}. This, too, gets a +1 from me. So that leaves comprehensions. IIRC, during the development of the patch we realized that f(*x for x in xs) is sufficiently ambiguous that we decided to disallow it -- note that f(x for x in xs) is already somewhat of a special case because an argument can only be a "bare" generator expression if it is the only argument. The same reasoning doesn't apply (in that form) to list, set and dict comprehensions -- while f(x for x in xs) is identical in meaning to f((x for x in xs)), [x for x in xs] is NOT the same as [(x for x in xs)] (that's a list of one element, and the element is a generator expression). The basic premise of this part of the proposal is that if you have a few iterables, the new proposal (without comprehensions) lets you create a list or generator expression that iterates over all of them, essentially flattening them: >>> xs = [1, 2, 3] >>> ys = ['abc', 'def'] >>> zs = [99] >>> [*xs, *ys, *zs] [1, 2, 3, 'abc', 'def', 99] >>> But now suppose you have a list of iterables: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xss[0], *xss[1], *xss[2]] [1, 2, 3, 'abc', 'def', 99] >>> Wouldn't it be nice if you could write the latter using a comprehension? >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs for xs in xss] [1, 2, 3, 'abc', 'def', 99] >>> This is somewhat seductive, and the following is even nicer: the *xs position may be an expression, e.g.: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs[:2] for xs in xss] [1, 2, 'abc', 'def', 99] >>> On the other hand, I had to explore the possibilities here by experimenting in the interpreter, and I discovered some odd edge cases (e.g. you can parenthesize the starred expression, but that seems a syntactic accident). All in all I am personally +0 on the comprehension part of the PEP, and I like that it provides a way to "flatten" a sequence of sequences, but I think very few people in the thread have supported this part. Therefore I would like to ask Neil to update the PEP and the patch to take out the comprehension part, so that the two "easy wins" can make it into Python 3.5 (basically, I am accepting two-thirds of the PEP :-). There is some time yet until alpha 2. I would also like code reviewers (Benjamin?) to start reviewing the patch <http://bugs.python.org/issue2292>, taking into account that the comprehension part needs to be removed. -- --Guido van Rossum (python.org/~guido) [View Less]

7 17

PEP 638: Syntactic macros
by Mark Shannon Feb. 5, 2023

Feb. 5, 2023

Hi everyone, I've submitted my PEP on syntactic macros as PEP 638. https://www.python.org/dev/peps/pep-0638/ All comments and suggestions are welcome. Cheers, Mark

16 25

How about using modern C++ in development of CPython ?
by redradist＠gmail.com Oct. 19, 2022

Oct. 19, 2022

Hi all, What about of using modern C++ in developing CPython ? With C++ we can get the following benefits: 1) Readability - less code, most code is hidden by abstraction without losing performance 2) Maintainability - no code duplication in favor of using reusable classes 3) RAII - Resource Acquisition Is Initialization, predictable allocation and free resources ...

23 51

PEP 505 (None-aware operators) for Python 3.11
by Doug Swarin Sept. 21, 2022

Sept. 21, 2022

Hello, I've been following PEP 505 (https://www.python.org/dev/peps/pep-0505/) since it was first proposed for Python 3.8. It's been deferred for some time and I'm interested in seeing it in Python 3.11, but I know there were also a number of objections which resulted in it being deferred (including by one of the original authors, Mr. Dower). I did email both Mr. Dower and Mr. Haase and they graciously gave me their permission to bring it up on this list for discussion and hopefully final … [View More]

22 53

PEP 563 and Python 3.10.
by Thomas Wouters July 22, 2022

July 22, 2022

(Starting a new thread so as not to derail any of the ongoing discussions.) Thanks, everyone, for your thoughts on Python 3.10 and the impact of PEP 563 (postponed evaluation of annotations) becoming the default. The Steering Council has considered the issue carefully, along with many of the proposed alternatives and solutions, and we’ve decided that at this point, we simply can’t risk the compatibility breakage of PEP 563. We need to roll back the change that made stringified annotations the … [View More]default, at least for 3.10. (Pablo is already working on this.) To be clear, we are not reverting PEP 563 itself. The future import will keep working like it did since Python 3.7. We’re delaying making PEP 563 string-based annotations the default until Python 3.11. This will give us time to find a solution that works for everyone (or to find a feasible upgrade path for users who currently rely on evaluated annotations). Some considerations that led us to this decision: - PEP 563’s default change is clearly too disruptive to downstream users and third-party libraries to happen right now. We can’t risk breaking even a small subset of the FastAPI/pydantic users, not to mention other uses of evaluated type annotations that we’re not aware of yet. - PEP 563 provides no warning to users of the feature it’s disabling. Without that, we can’t expect users to be aware of the upcoming breakage. The lack of a warning was by design, and made sense in a world where type annotations were only consumed by static type checkers --- but that’s not actually the situation we’re in. There are clearly existing real-world, run-time uses of type annotations that would be adversely affected by this change. - Originally, PEP 563 was scheduled to take effect in Python 4, and this changed recently (after the discussion in the Language Summit of 2020). It's possible that third-party libraries and users didn’t plan to react in the current time frame as they were not aware of this change in timing. - There isn’t enough time to properly discuss PEP 649 or any of the alternatives before the beta 1 deadline, and we really need to make sure we don’t compound errors here. We need to look for a long term solution, which isn’t possible while still maintaining the release deadlines of Python 3.10. That means we’re also deferring PEP 649 to Python 3.11. In the Steering Council’s unanimous opinion, rolling back the default flip for stringified annotations in Python 3.10 is the least disruptive of all the options. We need to continue discussing the issue and potential solutions, since this merely postpones the problem until 3.11. (For the record, postponing the change further is not off the table, either, for example if the final decision is to treat evaluated annotations as a deprecated feature, with warnings on use.) For what it’s worth, the SC is also considering what we can do to reduce the odds of something like this happening again, but that’s a separate consideration, and a multi-faceted one at that. For the Steering Council, Thomas. -- Thomas Wouters <thomas(a)python.org> Hi! I'm an email virus! Think twice before sending your email to help me spread! [View Less]

6 5

compiled python3.10 is unable to find _ssl
by Sandeep Gupta May 29, 2022

May 29, 2022

I am having compilation issues again with python3.10 with ssl . The ./configure was invoked with ssl options and ssl modules seems to be build successfully. """ The following modules found by detect_modules() in setup.py, have been built by the Makefile instead, as configured by the Setup files: _abc _hashlib _ssl pwd time """ However, when I do import ssl from python, I get the error: Traceback (most recent call last): File "<stdin>",… [View More]

6 12

C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API
by Victor Stinner April 25, 2022

April 25, 2022

Hi, I proposed two PRs to move the private C API (Include/cpython/) of PEP 523 "Adding a frame evaluation API to CPython" to the internal C API (Include/internals/): * https://github.com/python/cpython/pull/32052 * https://github.com/python/cpython/pull/32054 API: * _PyFrameEvalFunction type * _PyInterpreterState_GetEvalFrameFunc() * _PyInterpreterState_SetEvalFrameFunc() * (undocumented) _PyEval_EvalFrameDefault() The private API to get/set the eval function *is* documented at: https://… [View More]

7 20

Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API
by Jason Ansel April 20, 2022

April 20, 2022

The PyTorch team plans to use PEP 523 as a part of PyTorch 2.0, so this proposal may break the next major release of PyTorch. The related project is TorchDynamo, which can be found here: https://github.com/facebookresearch/torchdynamo We will likely move this into the core of PyTorch closer to release. If the changed happens, would PyTorch still be able to use the eval frame API? Or would it prevent from being used entirely?

11 25

Slowly bend the C API towards the limited API to get a stable ABI for everyone
by Victor Stinner April 20, 2022

April 20, 2022

Hi, There is a reason why I'm bothering C extensions maintainers and Python core developers with my incompatible C API changes since Python 3.8. Let me share my plan with you :-) In 2009 (Python 3.2), Martin v. Löwis did an amazing job with the PEP 384 "Defining a Stable ABI" to provide a "limited C API" and a "stable ABI" for C extensions: build an extension once, use it on multiple Python versions. Some projects like PyQt5 and cryptograpy use it, but it is just a drop in the PyPI ocean (… [View More]353,084 projects). I'm trying to bend the "default" C API towards this "limited C API" to make it possible tomorrow to build *more* C extensions for the stable ABI. My goal is that the stable ABI would be the default, and only a minority of C extensions would opt-out because they need to access to more functions for best performance. The basic problem is that at the ABI level, C extensions must only call functions, rather than getting and setting directly to structure members. Structures changes frequently in Python (look at changes between Python 3.2 and Python 3.11), and any minor structure change breaks the ABI. The limited C API hides structures and only use function calls to solve this problem. Since 2020, I'm modifying the C API, one function by one, to slowly hide implementations (prepare the API to make strutures opaque). I focused on the following structures: * PyObject and PyVarObject (bpo-39573) * PyTypeObject (bpo-40170) * PyFrameObject (bpo-40421) * PyThreadState (bpo-39947) The majority of C extensions use functions and macros, they don't access directly structure members. There are a few members which are sometimes accessed directly which prevents making these structures opaque. For example, some old C extensions use obj->ob_type rather than Py_TYPE(obj). Fixing the minority of C extensions should benefit to the majority which may become compatible with the stable ABI. I am also converting macros to static inline functions to fix their API: define parameter types, result type and avoid surprising macros side effects ("macro pitfalls"). I wrote the PEP 670 "Convert macros to functions in the Python C API" for these changes. I wrote the upgrade_pythoncapi.py tool in my pythoncapi_project (*) which modify C code to use Py_TYPE(), Py_SIZE() and Py_REFCNT() rather than accessing directly PyObject and PyVarObject members. (*) https://github.com/pythoncapi/pythoncapi_compat In this tool, I also added "Borrow" variant of functions like PyFrame_GetCode() which returns a strong reference, to replace frame->f_code with _PyFrame_GetCodeBorrow(). In Python 3.11, you cannot use the frame->f_code member anymore, since it has been removed! You must call PyFrame_GetCode() (or pythoncapi_compat _PyFrame_GetCodeBorrow() variant). There are also a few macros which can be used as l-values like Py_TYPE(): "Py_TYPE(type1) = type2" must now be written "Py_SET_TYPE(type1, type2)" to avoid setting directly the tp_type type at the ABI level. I proposed the PEP 674 "Disallow using Py_TYPE() and Py_SIZE() macros as l-values" to solve these issues. Currently, many "functions" are still implemented as macros or static inline functions, so C extensions still access structure members at the ABI level for best Python performance. Converting these to regular functions has an impact on performance and I would prefer to first write a PEP giving the rationale for that. Today, it is not possible yet to build numpy for the stable ABI. The gap is just too large for this big C extension. But step by step, the C API becomes closer to the limited API, and more and more code is ready to be built for the stable ABI. Well, these C API changes have other advantages, like preparing Python for further optimizations, ease Python maintenance, clarify the seperation between the limited C API and the default C API, etc. ;-) Victor -- Night gathers, and now my watch begins. It shall not end until my death. [View Less]

12 38

PEP 678: Enriching Exceptions with Notes
by Zac Hatfield Dodds April 12, 2022

April 12, 2022

Hi all, I've written PEP 678, proposing the addition of a __note__ attribute which can be used to enrich displayed exceptions. This is particularly useful with ExceptionGroup, or where exception chaining is unsuitable, and ensures that 'orphan' output is not produced if the annotated exception is caught. Link to the PEP: https://www.python.org/dev/peps/pep-0678/ *Please, redirect all discussions to:* https://discuss.python.org/t/pep-678-enriching-exceptions-with-notes/13374 Thanks for … [View More]

2 1

It's now time to deprecate the stdlib urllib module
by Victor Stinner March 31, 2022

March 31, 2022

Hi, I propose to deprecate the urllib module in Python 3.11. It would emit a DeprecationWarning which warn users, so users should consider better alternatives like urllib3 or httpx: well known modules, better maintained, more secure, support HTTP/2 (httpx), etc. I don't propose to schedule its removal. Let's discuss the removal in 1 or 2 years. -- urllib has many abstraction to support a wide range of protocols with "handlers": HTTP, HTTPS, FTP, "local file", proxy, HTTP authentication, … [View More]HTTP Cookie, etc. A simple HTTP request using Basic Authentication requires 10-20 lines of code, whereas it should be a single line. Users (me included) don't like urllib API which was too complicated for common tasks. -- Unhappy users created multiple better alternatives to the stdlib urllib module. In 2008, the "urllib3" module was created to provide an API designed to be as simple as possible for the most common HTTP and HTTPS requests. Example: req = http.request('GET', 'http://httpbin.org/robots.txt'). In 2011, the "requests" module based on urllib3 was created. In 2013, the "aiohttp" module based on asyncio was created. In 2015, new "httpx" module was created: req = httpx.get('https://www.example.org/') Not only httpx has a regular "synchronous" API (blocking function calls), but it also has an asynchronous API! Sadly, while HTTP/3 is being developed, it seems like in this list, httpx is the only HTTP client library supporting HTTP/2 currently :-( For HTTP/2, I also found the "httplib2" module. For HTTP/3, I found the "http3" and "aioquic" modules. -- Let's come back to urllib: * It's API is too complicated * It doesn't support HTTP/2 nor HTTP/3 * It's barely maintained: there are 121 open issues including 3 security issues! The 3 open security issues: * bpo-33661 open 2018; * bpo-36338 open in 2019; * bpo-45795 open in 2021. Usually, it's bad when you refer to an open security issue by its creation year :-( The urllib module has long history of security vulnerabilities. List of *fixed* vulnerabilities: * 2011 (bpo-11662): https://python-security.readthedocs.io/vuln/urllib-redirect.html * 2017 (bpo-30119): https://python-security.readthedocs.io/vuln/urllib-ftp-stream-injection.html * 2017 (bpo-30500): https://python-security.readthedocs.io/vuln/urllib-connects-wrong-host.html * 2019 (bpo-35907): https://python-security.readthedocs.io/vuln/urllib-local-file-scheme.html * 2019 (bpo-38826): https://python-security.readthedocs.io/vuln/urllib-basic-auth-regex.html * 2021 (bpo-42967): https://python-security.readthedocs.io/vuln/urllib-query-string-semicolon-s… * 2021 (bpo-43075): https://python-security.readthedocs.io/vuln/urllib-basic-auth-regex2.html * 2021 (bpo-44022): https://python-security.readthedocs.io/vuln/urllib-100-continue-loop.html urllib is a package made of 4 parts: * urllib.request for opening and reading URLs * urllib.error containing the exceptions raised by urllib.request * urllib.parse for parsing URLs * urllib.robotparser for parsing robots.txt files I propose to deprecate all of them. Maybe the deprecation can be different for each sub-module? Victor -- Night gathers, and now my watch begins. It shall not end until my death. [View Less]

39 75

Enhancing generic type documentation in the standard library
by Luciano Ramalho March 30, 2022

March 30, 2022

Hello, everyone! I believe our documentation about types needs another overhaul. We now have lots of generic types in the standard library, but their formal type parameters are poorly documented—or not documented at all—in the standard library documentation. More importantly: the documentation we have about specific covariant/contravariant types is now in entries in the `typing` module that are all deprecated since PEP 585 was implemented in Python 3.9. Below I present two of many examples … [View More]where the documentation of a generic type is not great. However, if people agree this is a problem, we need to discuss where and how to put the documentation in a way that is not too disruptive to users of Python who don't know or don't care about type hints, for many reasons that we should not judge. For example, where do we document the fact that `dict` accepts two invariant formal type parameters, and that `frozenset` accepts one contravariant type parameter? What do you think? Cheers, Luciano _________________________________________ EXAMPLE 1: `Callable` variance is not documented For example, in the `Callable` type, the `ReturnType` parameter is covariant, and the `ParameterType` parameters are all contravariant. But there is no mention of variance in this entry: https://docs.python.org/3/library/typing.html?highlight=typing#typing.Calla… Also, no mention of the fact that `collections.abc.Callable` is generic here: https://docs.python.org/3/library/collections.abc.html#collections.abc.Call… PEP 483—The Theory of Type Hints—is the only official Python doc where I found the information about the variance of the formal type parameters of `Callable`. The explanation there is brilliant [0]. [0] https://peps.python.org/pep-0483/#covariance-and-contravariance Regardless, the intended audience of PEPs is "core developers"—which is neither a superset nor a subset of "Python devs now using type hints". We should not rely on PEPs to document features for Python users in general. _________________________________________ EXAMPLE 2: `Generator` variance could be better documented The entry for `typing.Generator` [1] has this heading: class typing.Generator(Iterator[T_co], Generic[T_co, T_contra, V_co]) Answer quickly: how many formal type parameters does `Generator` require? Which are covariant? Which are contravariant? [1] https://docs.python.org/3/library/typing.html?highlight=typing#typing.Gener… Nowhere in that page [1] there's an explanation of the `*_co` and `*_contra` naming convention, much less their semantics. Fortunately, the text of the `typing.Generator` entry says: "A generator can be annotated by the generic type `Generator[YieldType, SendType, ReturnType]". Unfortunately, `typing.Generator` is deprecated and will be gone in 5 years or so... The same issue applies to all the other generic types: builtins (`dict`, `frozenset`), ABCs, etc. Their formal type parameters they accept as generics are either undocumented, or documented in parts of the `typing` module that are already deprecated. Thoughts? -- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Technical Principal at ThoughtWorks | Twitter: @ramalhoorg [View Less]

6 7

Fwd: Re: Are "Batteries Included" still a Good Thing? [was: It's now time to deprecate the stdlib urllib module]
by Sasha Kacanski March 30, 2022

March 30, 2022

---------- Forwarded message --------- From: Sasha Kacanski <skacanski(a)gmail.com> Date: Wed, Mar 30, 2022, 2:55 PM Subject: Re: [Python-Dev] Re: Are "Batteries Included" still a Good Thing? [was: It's now time to deprecate the stdlib urllib module] To: Paul Moore <p.f.moore(a)gmail.com> Cc: Christopher Barker <pythonchb(a)gmail.com>, Ethan Furman < ethan(a)stoneleaf.us>, Python Dev <python-dev(a)python.org> Agree with Paul, Best of python always was, you can do … [View More]almost everything with standard libs, but then novice becomes better developer and starts exploring libs and tools that wrap stuff or perform better or has functionality that stdlib does not support right of the bet they start using variety of libs that suits their needs. Regards, Sasha On Sun, Mar 27, 2022, 8:06 PM Paul Moore <p.f.moore(a)gmail.com> wrote: > On Sun, 27 Mar 2022 at 17:11, Christopher Barker <pythonchb(a)gmail.com> > wrote: > > > > With the json package included, all they need to do is `import json`. If > that wasn't there, they's look in PyPi for a JSON implementation, and find > an absolutely astonishing number of options. I just did a search for "JSON" > > on PYPI, and it's HUGE -- most of them are for specialized JSON-using > protocols of some sort. I was actually really surprised that couple I know > about of the top of my head (ujson, orjson) are actually hard to find. > > > > "You can just pip install it" is really not a good solution. > > > > In fact, this is an example, I think, of where we should put some effort > into making the included batteries better -- it's great to have a JSON lib > built in, but it's too bad that it's not best-of-bread by pretty much any > definition (speed, memory performance, flexibility) -- there are quite a > few feature requests open for it -- it would be nice to actually implement > some of those. (but yes, that's a lot of work that someone(s) would have to > do) > > > > Back to the topic at hand, rather than remove urllib, maybe it could be > made better -- an as-easy-to-use-as-requests package in the stdlib would be > really great. > > I think that's where the mistake happens, though. Someone who needs > "best of breed" is motivated (and likely knowledgeable enough) to make > informed decisions about what's on PyPI. But someone who just wants to > get the job done probably doesn't - and that's the audience for the > stdlib. A stdlib module needs to be a good, reliable set of basic > functionality that non-experts can use successfully. There can be > better libraries on PyPI, but that doesn't mean the stdlib module is > unnecessary, nor does it mean that the stdlib has to match the PyPI > library feature for feature. > > So here, specifically, I'd rather see urlllib be the best urlllib it > can be, and not demand that it turn into requests. Requests is there > if people need/want it (as is httpx, and urllib3, and aiohttp). But > urllib is for people who want to get a file from the web, and *not* > have to deal with dependencies, 3rd party libraries, etc. > > The "batteries included" standard library and PyPI complement each > other. Neither is redundant, and neither implies the other is > unnecessary. > > Paul > _______________________________________________ > Python-Dev mailing list -- python-dev(a)python.org > To unsubscribe send an email to python-dev-leave(a)python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/E2FGXTY… > Code of Conduct: http://python.org/psf/codeofconduct/ > [View Less]

1 0

Changing unittest verbose output.
by Itay Yeshaya March 29, 2022

March 29, 2022

When running unittest with the -v flag, if a test has errors, and has a docstring, the test name is shown on one line, and the docstring is shown on the next line with the `ERROR` word. For example: ./python.exe -m unittest -v Lib.test_verbose_error.randomTest test_one (Lib.test_verbose_error.randomTest) this is a doc string with helpful info ... ERROR ====================================================================== ERROR: test_one (Lib.test_verbose_error.randomTest) this is a doc … [View More]

7 9

walrus operator and expression
by Ethan Furman March 28, 2022

March 28, 2022

In the following bit of code: while s := input.read(MAXBINSIZE): while len(s) < MAXBINSIZE and ns := input.read(MAXBINSIZE-len(s)): s += ns line = binascii.b2a_base64(s) output.write(line) I'm getting this error on the second line: cannot use assignment expressions with expression Can somebody explain why that isn't working? Many thanks. -- ~Ethan~

4 3

import * and __future__ imports
by Irit Katriel March 28, 2022

March 28, 2022

If you have a __future__ import in a script, and you import * from it in another script, the object for this future appears in the dir() of the other script, even though the __future__ import has no effect there. % cat x.py from __future__ import annotations % cat y.py from x import * print(dir()) class D: def f(self, a: D): return 42 % ./python.exe y.py ['__annotations__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '… [View More]

4 4

is __self__ an implementation detail
by Robin Becker March 28, 2022

March 28, 2022

A gentoo developer reported a symlink loop problem in reportlab's setup.py where we search for specific headers. The 'fixed' function looks like this def findFile(root, wanted, followlinks=True): visited = set() for p, D, F in os.walk(root,followlinks=followlinks): #scan directories to check for prior visits #use dev/inode to make unique key SD = [].append for d in D: dk = os.stat(pjoin(p,d)) dk = dk.st_dev, dk.st_ino … [View More]

2 1

Summary of Python tracker Issues
by Python tracker March 25, 2022

March 25, 2022

ACTIVITY SUMMARY (2022-03-18 - 2022-03-25) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7145 (-20) closed 51702 (+82) total 58847 (+62) Open issues with patches: 2903 Issues opened (53) ================== #45618: Documentation builds fail with Sphinx 3.2.1 https://bugs.python.org/issue45618 reopened by Maciej Olko #46429: Merge all deepfrozen … [View More]

1 0

Python Language Summit at PyCon 2022 in Salt Lake City
by Mariatta March 24, 2022

March 24, 2022

We're excited to announce that the signups for Python Language Summit at PyCon 2022 are now open. Full details at: https://us.pycon.org/2022/events/language-summit/ After two years of virtual/online summit, we will be returning to in-person format. We will be following the health and safety guidelines at PyCon US: https://us.pycon.org/2022/attend/health-safety-guidelines/ *TL;DR* When: Wednesday, April 27, 2022 Where: Salt Palace Convention Center, room TBD Sign up to attend: https://… [View More]

1 2

[RELEASE] Python 3.10.4 and 3.9.12 are now available out of schedule
by Łukasz Langa March 24, 2022

March 24, 2022

Did anybody say cursed releases <https://discuss.python.org/t/python-3-10-3-3-9-11-3-8-13-and-3-7-13-are-now…>? Well, it turns out that 3.10.3 and 3.9.11 both shipped a regression which caused those versions not to build on Red Hat Enterprise Linux 6. While this 11-year-old version is now out of maintenance support <https://access.redhat.com/support/policy/updates/errata>, it’s still used in production workloads. Some of those rely on Python 3.9 and/or 3.10. In particular, our own … [View More]

3 2

Accepting PEP 675 - Arbitrary Literal String Type
by Gregory P. Smith March 21, 2022

March 21, 2022

On behalf of the Python Steering Council, we are accepting PEP 675 - Arbitrary Literal String Type <https://peps.python.org/pep-0675/>. TL;DR - PEP 675 allows type checkers to help prevent bugs allowing attacker-controlled data to be passed to APIs that declare themselves as requiring literal, in-code strings. This is a very thorough PEP with a compelling and highly relevant set of use cases. If I tried to call out all the things we like about it, it’d turn into a table of contents. It … [View More]

2 1

Accepting PEP 655 - Marking individual TypedDict items as required or potentially-missing
by Gregory P. Smith March 21, 2022

March 21, 2022

On behalf of the Python Steering Council, we are pleased to accept PEP 655 - Marking individual TypedDict items as required or potentially-missing <https://peps.python.org/pep-0655/>. Thanks for considering the potential for confusion with Optional during the design and recommending best practices in the “how to teach” section. A couple SC members have tried using TypedDict and found it painful, this PEP helps. -gps for the steering council

1 0

A memory map based data persistence and startup speedup approach
by Yichen Yan March 20, 2022

March 20, 2022

Hi folks, as illustrated in faster-cpython#150 [1], we have implemented a mechanism that supports data persistence of a subset of python date types with mmap, therefore can reduce package import time by caching code object. This could be seen as a more eager pyc format, as they are for the same purpose, but our approach try to avoid [de]serialization. Therefore, we get a speedup in overall python startup by ~15%. Currently, we’ve made it a third-party library and have been working on open-… [View More]sourcing. Our implementation (whose non-official name is “pycds”) mainly contains two parts: importlib hooks, this implements the mechanism to dump code objects to an archive and a `Finder` that supports loading code object from mapped memory. Dumping and loading (subset of) python types with mmap. In this part, we deal with 1) ASLR by patching `ob_type` fields; 2) hash seed randomization by supporting only basic types who don’t have hash-based layout (i.e. dict is not supported); 3) interned string by re-interning strings while loading mmap archive and so on. After pycds has been installed, complete workflow of our approach includes three parts: Record name of imported packages to heap.lst, `PYCDSMODE=TRACE PYCDSLIST=heap.lst python run.py` Dump memory archive of code objects of imported packages, this step does not involve the python script, `PYCDSMODE=DUMP PYCDSLIST=heap.lst PYCDSARCHIVE=heap.img python` Run other python processes with created archive, `PYCDSMODE=SHARE PYCDSARCHIVE=heap.img python run.py` We could even make use of immortal objects if PEP 683 [2] was accepted, which could give CDS more performance improvements. Currently, any archived object is virtually immortal, we add rc by 1 to who has been copied to the archive to avoid being deallocated. However, without changes to CPython, rc fields of archived objects will still be updated, therefore have extra footprint due to CoW. More background and detailed implementation could be found at [1]. We think it could be an effective way to improve python’s startup performance, and could even do more like sharing large data between python instances. As suggested in python-ideas [3], we posted this here, looking for questions/suggestions to the overall design and workflow, we also welcome code reviews after we get our lawyers happy and can publish the code. Best, Yichen Yan Alibaba Compiler Group [1] “Faster startup -- Share code objects from memory-mapped file”, https://github.com/faster-cpython/ideas/discussions/150 [2] PEP 683: "Immortal Objects, Using a Fixed Refcount" (draft), https://mail.python.org/archives/list/python-dev@python.org/message/TPLEYDC… [3] [Python-ideas] "A memory map based data persistence and startup speedup approach", https://mail.python.org/archives/list/python-ideas@python.org/thread/UKEBNH… [View Less]

2 1

Restrict the type of __slots__
by Serhiy Storchaka March 19, 2022

March 19, 2022

Currently __slots__ can be either string or an iterable of strings. 1. If it is a string, it is a name of a single slot. Third-party code which iterates __slots__ will be confused. 2. If it is an iterable, it should emit names of slots. Note that non-reiterable iterators are accepted too, but it causes weird bugs if __slots__ is iterated more than once. For example it breaks default pickling and copying. I propose to restrict the type of __slots__. Require it always been a tuple of … [View More]

10 15

New PEP website is horrible to read on mobile device
by Nathan Cook March 18, 2022

March 18, 2022

Please make https://peps.python.org/ more responsive to various form factors See attached screenshot from Chrome version 99.0.4844.58 on my Pixel 3aXL running Android 12 Thank you, Nathan Cook

11 11

Summary of Python tracker Issues
by Python tracker March 18, 2022

March 18, 2022

ACTIVITY SUMMARY (2022-03-11 - 2022-03-18) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7165 (-68) closed 51620 (+139) total 58785 (+71) Open issues with patches: 2903 Issues opened (50) ================== #22628: Idle: Tree lines are spaced too close together. https://bugs.python.org/issue22628 reopened by exarkun #43253: asyncio … [View More]

1 0

Request for feedback on new features for 3.11
by Gabriele March 17, 2022

March 17, 2022

Hi there I hope you will indulge me while I request some feedback on a couple of open PRs that I would like to be considered for 3.11. https://github.com/python/cpython/pull/31150 https://github.com/python/cpython/pull/31142 The first one should be pretty easy to review as it simply makes use of the new co_qualname attribute of code objects to enrich the traceback output with qualified names. Provided it gets accepted, similar changes could also be introduced in other areas, like profiling, … [View More]

1 0

[RELEASE] Python 3.10.3, 3.9.11, 3.8.13, and 3.7.13 are now available with security content
by Łukasz Langa March 16, 2022

March 16, 2022

Welcome again to the exciting world of releasing new Python versions! Last time around I was complaining about cursed releases <https://discuss.python.org/t/python-3-10-2-3-9-10-and-3-11-0a4-are-now-avai…>. This time around I could complain about security content galore and how one of them <https://mta.openssl.org/pipermail/openssl-announce/2022-March/000216.html> ruined my ingenious idea to release Python on Pi Day and call it Py Day <https://discuss.python.org/t/py-day-is-… [View More]

3 3

Python 3.9.11
by Prasad, PCRaghavendra March 16, 2022

March 16, 2022

Hi Team, Can someone please let us know the release date of Python 3.9.11 ( with libexpat 2.4.8 security issues fixed ) In the python.org releases it was mentioned as 14-march-2022, but still, I couldn't see the bin/source code. Can someone help with this Thanks, Raghavendra Internal Use - Confidential

5 4

Gathering edge cases around type annotation PEPs
by Brett Cannon March 15, 2022

March 15, 2022

I created https://discuss.python.org/t/finding-edge-cases-for-peps-484-463-and-649-ty… on behalf of the SC to help gather edge cases where the various approaches that PEPs have proposed will fail. Our hope is to get an overall picture of the trade-offs the various PEPs ask us to make. PLEASE DO NOT REPLY HERE! I'm reaching out to the typing SIG and Pydantic (at least) and it's much easier for others to log into discuss.python.org with a GitHub account than it is to sign up for this mailing … [View More]

1 0

Defining tiered platform support
by Brett Cannon March 15, 2022

March 15, 2022

Do we officially support NetBSD? Do you know how to find out if we do? You might think to look at https://www.python.org/dev/peps/pep-0011/#supporting-platforms , but that just loosely defines the criteria and it still doesn't list the actual platforms we support. (BTW I don't know if we do officially support NetBSD, hence this email.) I think we should clarify this sort of thing and be a bit more upfront with the level of support we expect/provide for a platform. As such, I want to … [View More]restructure PEP 11 to list the platforms we support, not just list the platforms we stopped supporting. To do this I want define 3 different tiers that outline what our support requirements and promises are for platforms. Tier 1 is the stuff we run CI against: latest Windows, latest macOS, Linux w/ the latest glibc (I don't know of a better way to define Linux support as I don't know if a per-distro list is the right abstraction). These are platforms we won't even let code be committed for if they would break; they block releases if they don't work. These platforms we all implicitly promise to support. Tier 2 is the platforms we would revert a change within 24 hours if they broke: latest FeeBSD, older Windows, older macOS, Linux w/ older glibc.This is historically the "stable buildbot plus a core dev" group of platforms. The change I would like to see is two core devs (in case one is on vacation), and a policy as to how a platform ends up here (e.g. SC must okay it based on consensus of everyone). The stable buildbot would still be needed to know if a release is blocked as we would hold a release up if they were red. The platform and the core devs supporting these platforms would be listed in PEP 11. Tier 3 are platforms we accept PRs for to keep working, but they won't block a release. At least one core dev must be listed to be in charge of PRs that affect the platform. A buildbot would be nice but not required. I'm thinking either historical platforms we support or something like Emscripten that's working towards being a tier 2 platform. I'm not sure if we want to have explicit approval to be in this tier beyond the outlined requirements, but obviously if a core dev isn't keeping up with PRs then the platform will be dropped. All other platforms we do not directly support in the repository and it is up to the community to keep functioning. We can obviously keep accepting general patches to up compatibility, but platform-specific patches for anything outside of these tiers wouldn't. No issue if someone provides a buildbot for their own benefit, but these buildbots would never hold anything up. When a Python version hits b1, then that platform is considered supported for that version as long as the requirements outlined above continue to be met. Once the Python version hits EOL then like anything, support ends no matter what. We could capture the supported platforms for a version in the release schedule PEP if people want. I would expect appropriate labels being added to the builders listed at https://buildbot.python.org/all/#/builders to signify which platforms we block releases for (e.g. `tier-1,`, `tier-2`, or `release-blocker` as a generic label). I would expect PEP 11 to list the appropriate C symbol that's set for that platform, e.g. __linux__. I don't know if we want to bother listing CPU architectures since we are a pure C project which makes CPU architecture less of a thing, but I'm personally open to the idea of CPU architectures being a part of the platform definition. I don't think we should cover C compilers here as that's covered by PEP 7. Otherwise PEP 7 should only list C versions/features and PEP 11 lists compilers and their versions. FYI the tier support idea and the rough definitions above come from Rust: https://doc.rust-lang.org/nightly/rustc/platform-support.html . [View Less]

9 19

PEP 684: A Per-Interpreter GIL
by Eric Snow March 14, 2022

March 14, 2022

I'd really appreciate feedback on this new PEP about making the GIL per-interpreter. The PEP targets 3.11, but we'll see if that is too close. I don't mind waiting one more release, though I'd prefer 3.11 (obviously). Regardless, I have no intention of rushing this through at the expense of cutting corners. Hence, we'll see how it goes. The PEP text is included inline below. Thanks! -eric =================================================== PEP: 684 Title: A Per-Interpreter GIL Author: … [View More]Eric Snow <ericsnowcurrently(a)gmail.com> Discussions-To: python-dev(a)python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 08-Mar-2022 Python-Version: 3.11 Post-History: 08-Mar-2022 Resolution: Abstract ======== Since Python 1.5 (1997), CPython users can run multiple interpreters in the same process. However, interpreters in the same process have always shared a significant amount of global state. This is a source of bugs, with a growing impact as more and more people use the feature. Furthermore, sufficient isolation would facilitate true multi-core parallelism, where interpreters no longer share the GIL. The changes outlined in this proposal will result in that level of interpreter isolation. High-Level Summary ================== At a high level, this proposal changes CPython in the following ways: * stops sharing the GIL between interpreters, given sufficient isolation * adds several new interpreter config options for isolation settings * adds some public C-API for fine-grained control when creating interpreters * keeps incompatible extensions from causing problems The GIL ------- The GIL protects concurrent access to most of CPython's runtime state. So all that GIL-protected global state must move to each interpreter before the GIL can. (In a handful of cases, other mechanisms can be used to ensure thread-safe sharing instead, such as locks or "immortal" objects.) CPython Runtime State --------------------- Properly isolating interpreters requires that most of CPython's runtime state be stored in the ``PyInterpreterState`` struct. Currently, only a portion of it is; the rest is found either in global variables or in ``_PyRuntimeState``. Most of that will have to be moved. This directly coincides with an ongoing effort (of many years) to greatly reduce internal use of C global variables and consolidate the runtime state into ``_PyRuntimeState`` and ``PyInterpreterState``. (See `Consolidating Runtime Global State`_ below.) That project has `significant merit on its own <Benefits to Consolidation_>`_ and has faced little controversy. So, while a per-interpreter GIL relies on the completion of that effort, that project should not be considered a part of this proposal--only a dependency. Other Isolation Considerations ------------------------------ CPython's interpreters must be strictly isolated from each other, with few exceptions. To a large extent they already are. Each interpreter has its own copy of all modules, classes, functions, and variables. The CPython C-API docs `explain further <caveats_>`_. .. _caveats: https://docs.python.org/3/c-api/init.html#bugs-and-caveats However, aside from what has already been mentioned (e.g. the GIL), there are a couple of ways in which interpreters still share some state. First of all, some process-global resources (e.g. memory, file descriptors, environment variables) are shared. There are no plans to change this. Second, some isolation is faulty due to bugs or implementations that did not take multiple interpreters into account. This includes CPython's runtime and the stdlib, as well as extension modules that rely on global variables. Bugs should be opened in these cases, as some already have been. Depending on Immortal Objects ----------------------------- :pep:`683` introduces immortal objects as a CPython-internal feature. With immortal objects, we can share any otherwise immutable global objects between all interpreters. Consequently, this PEP does not need to address how to deal with the various objects `exposed in the public C-API <capi objects_>`_. It also simplifies the question of what to do about the builtin static types. (See `Global Objects`_ below.) Both issues have alternate solutions, but everything is simpler with immortal objects. If PEP 683 is not accepted then this one will be updated with the alternatives. This lets us reduce noise in this proposal. Motivation ========== The fundamental problem we're solving here is a lack of true multi-core parallelism (for Python code) in the CPython runtime. The GIL is the cause. While it usually isn't a problem in practice, at the very least it makes Python's multi-core story murky, which makes the GIL a consistent distraction. Isolated interpreters are also an effective mechanism to support certain concurrency models. :pep:`554` discusses this in more detail. Indirect Benefits ----------------- Most of the effort needed for a per-interpreter GIL has benefits that make those tasks worth doing anyway: * makes multiple-interpreter behavior more reliable * has led to fixes for long-standing runtime bugs that otherwise hadn't been prioritized * has been exposing (and inspiring fixes for) previously unknown runtime bugs * has driven cleaner runtime initialization (:pep:`432`, :pep:`587`) * has driven cleaner and more complete runtime finalization * led to structural layering of the C-API (e.g. ``Include/internal``) * also see `Benefits to Consolidation`_ below Furthermore, much of that work benefits other CPython-related projects: * performance improvements ("faster-cpython") * pre-fork application deployment (e.g. Instagram) * extension module isolation (see :pep:`630`, etc.) * embedding CPython Existing Use of Multiple Interpreters ------------------------------------- The C-API for multiple interpreters has been used for many years. However, until relatively recently the feature wasn't widely known, nor extensively used (with the exception of mod_wsgi). In the last few years use of multiple interpreters has been increasing. Here are some of the public projects using the feature currently: * `mod_wsgi <https://github.com/GrahamDumpleton/mod_wsgi>`_ * `OpenStack Ceph <https://github.com/ceph/ceph/pull/14971>`_ * `JEP <https://github.com/ninia/jep>`_ * `Kodi <https://github.com/xbmc/xbmc>`_ Note that, with :pep:`554`, multiple interpreter usage would likely grow significantly (via Python code rather than the C-API). PEP 554 ------- :pep:`554` is strictly about providing a minimal stdlib module to give users access to multiple interpreters from Python code. In fact, it specifically avoids proposing any changes related to the GIL. Consider, however, that users of that module would benefit from a per-interpreter GIL, which makes PEP 554 more appealing. Rationale ========= During initial investigations in 2014, a variety of possible solutions for multi-core Python were explored, but each had its drawbacks without simple solutions: * the existing practice of releasing the GIL in extension modules * doesn't help with Python code * other Python implementations (e.g. Jython, IronPython) * CPython dominates the community * remove the GIL (e.g. gilectomy, "no-gil") * too much technical risk (at the time) * Trent Nelson's "PyParallel" project * incomplete; Windows-only at the time * ``multiprocessing`` * too much work to make it effective enough; high penalties in some situations (at large scale, Windows) * other parallelism tools (e.g. dask, ray, MPI) * not a fit for the stdlib * give up on multi-core (e.g. async, do nothing) * this can only end in tears Even in 2014, it was fairly clear that a solution using isolated interpreters did not have a high level of technical risk and that most of the work was worth doing anyway. (The downside was the volume of work to be done.) Specification ============= As `summarized above <High-Level Summary_>`__, this proposal involves the following changes, in the order they must happen: 1. `consolidate global runtime state <Consolidating Runtime Global State_>`_ (including objects) into ``_PyRuntimeState`` 2. move nearly all of the state down into ``PyInterpreterState`` 3. finally, move the GIL down into ``PyInterpreterState`` 4. everything else * add to the public C-API * implement restrictions in ``ExtensionFileLoader`` * work with popular extension maintainers to help with multi-interpreter support Per-Interpreter State --------------------- The following runtime state will be moved to ``PyInterpreterState``: * all global objects that are not safely shareable (fully immutable) * the GIL * mutable, currently protected by the GIL * mutable, currently protected by some other per-interpreter lock * mutable, may be used independently in different interpreters * all other mutable (or effectively mutable) state not otherwise excluded below Furthermore, a number of parts of the global state have already been moved to the interpreter, such as GC, warnings, and atexit hooks. The following state will not be moved: * global objects that are safely shareable, if any * immutable, often ``const`` * treated as immutable * related to CPython's ``main()`` execution * related to the REPL * set during runtime init, then treated as immutable * mutable, protected by some global lock * mutable, atomic Note that currently the allocators (see ``Objects/obmalloc.c``) are shared between all interpreters, protected by the GIL. They will need to move to each interpreter (or a global lock will be needed). This is the highest risk part of the work to isolate interpreters and may require more than just moving fields down from ``_PyRuntimeState``. Some of the complexity is reduced if CPython switches to a thread-safe allocator like mimalloc. .. _proposed capi: C-API ----- The following private API will be made public: * ``_PyInterpreterConfig`` * ``_Py_NewInterpreter()`` (as ``Py_NewInterpreterEx()``) The following fields will be added to ``PyInterpreterConfig``: * ``own_gil`` - (bool) create a new interpreter lock (instead of sharing with the main interpreter) * ``strict_extensions`` - fail import in this interpreter for incompatible extensions (see `Restricting Extension Modules`_) Restricting Extension Modules ----------------------------- Extension modules have many of the same problems as the runtime when state is stored in global variables. :pep:`630` covers all the details of what extensions must do to support isolation, and thus safely run in multiple interpreters at once. This includes dealing with their globals. Extension modules that do not implement isolation will only run in the main interpreter. In all other interpreters, the import will raise ``ImportError``. This will be done through ``importlib._bootstrap_external.ExtensionFileLoader``. We will work with popular extensions to help them support use in multiple interpreters. This may involve adding to CPython's public C-API, which we will address on a case-by-case basis. Extension Module Compatibility '''''''''''''''''''''''''''''' As noted in `Extension Modules`_, many extensions work fine in multiple interpreters without needing any changes. The import system will still fail if such a module doesn't explicitly indicate support. At first, not many extension modules will, so this is a potential source of frustration. We will address this by adding a context manager to temporarily disable the check on multiple interpreter support: ``importlib.util.allow_all_extensions()``. Documentation ------------- The "Sub-interpreter support" section of ``Doc/c-api/init.rst`` will be updated with the added API. Impact ====== Backwards Compatibility ----------------------- No behavior or APIs are intended to change due to this proposal, with one exception noted in `the next section <Extension Modules_>`_. The existing C-API for managing interpreters will preserve its current behavior, with new behavior exposed through new API. No other API or runtime behavior is meant to change, including compatibility with the stable ABI. See `Objects Exposed in the C-API`_ below for related discussion. Extension Modules ''''''''''''''''' Currently the most common usage of Python, by far, is with the main interpreter running by itself. This proposal has zero impact on extension modules in that scenario. Likewise, for better or worse, there is no change in behavior under multiple interpreters created using the existing ``Py_NewInterpreter()``. Keep in mind that some extensions already break when used in multiple interpreters, due to keeping module state in global variables. They may crash or, worse, experience inconsistent behavior. That was part of the motivation for :pep:`630` and friends, so this is not a new situation nor a consequence of this proposal. In contrast, when the `proposed API <proposed capi_>`_ is used to create multiple interpreters, the default behavior will change for some extensions. In that case, importing an extension will fail (outside the main interpreter) if it doesn't indicate support for multiple interpreters. For extensions that already break in multiple interpreters, this will be an improvement. Now we get to the break in compatibility mentioned above. Some extensions are safe under multiple interpreters, even though they haven't indicated that. Unfortunately, there is no reliable way for the import system to infer that such an extension is safe, so importing them will still fail. This case is addressed in `Extension Module Compatibility`_ below. Extension Module Maintainers ---------------------------- One related consideration is that a per-interpreter GIL will likely drive increased use of multiple interpreters, particularly if :pep:`554` is accepted. Some maintainers of large extension modules have expressed concern about the increased burden they anticipate due to increased use of multiple interpreters. Specifically, enabling support for multiple interpreters will require substantial work for some extension modules. To add that support, the maintainer(s) of such a module (often volunteers) would have to set aside their normal priorities and interests to focus on compatibility (see :pep:`630`). Of course, extension maintainers are free to not add support for use in multiple interpreters. However, users will increasingly demand such support, especially if the feature grows in popularity. Either way, the situation can be stressful for maintainers of such extensions, particularly when they are doing the work in their spare time. The concerns they have expressed are understandable, and we address the partial solution in `Restricting Extension Modules`_ below. Alternate Python Implementations -------------------------------- Other Python implementation are not required to provide support for multiple interpreters in the same process (though some do already). Security Implications --------------------- There is no known impact to security with this proposal. Maintainability --------------- On the one hand, this proposal has already motivated a number of improvements that make CPython *more* maintainable. That is expected to continue. On the other hand, the underlying work has already exposed various pre-existing defects in the runtime that have had to be fixed. That is also expected to continue as multiple interpreters receive more use. Otherwise, there shouldn't be a significant impact on maintainability, so the net effect should be positive. Performance ----------- The work to consolidate globals has already provided a number of improvements to CPython's performance, both speeding it up and using less memory, and this should continue. Performance benefits to a per-interpreter GIL have not been explored. At the very least, it is not expected to make CPython slower (as long as interpreters are sufficiently isolated). How to Teach This ================= This is an advanced feature for users of the C-API. There is no expectation that this will be taught. That said, if it were taught then it would boil down to the following: In addition to Py_NewInterpreter(), you can use Py_NewInterpreterEx() to create an interpreter. The config you pass it indicates how you want that interpreter to behave. Reference Implementation ======================== <TBD> Open Issues =========== * What are the risks/hurdles involved with moving the allocators? * Is ``allow_all_extensions`` the best name for the context manager? Deferred Functionality ====================== * ``PyInterpreterConfig`` option to always run the interpreter in a new thread * ``PyInterpreterConfig`` option to assign a "main" thread to the interpreter and only run in that thread Rejected Ideas ============== <TBD> Extra Context ============= Sharing Global Objects ---------------------- We are sharing some global objects between interpreters. This is an implementation detail and relates more to `globals consolidation <Consolidating Runtime Global State>`_ than to this proposal, but it is a significant enough detail to explain here. The alternative is to share no objects between interpreters, ever. To accomplish that, we'd have to sort out the fate of all our static types, as well as deal with compatibility issues for the many objects `exposed in the public C-API <capi objects_>`_. That approach introduces a meaningful amount of extra complexity and higher risk, though prototyping has demonstrated valid solutions. Also, it would likely result in a performance penalty. `Immortal objects <Depending on Immortal Objects_>`_ allow us to share the otherwise immutable global objects. That way we avoid the extra costs. .. _capi objects: Objects Exposed in the C-API '''''''''''''''''''''''''''' The C-API (including the limited API) exposes all the builtin types, including the builtin exceptions, as well as the builtin singletons. The exceptions are exposed as ``PyObject *`` but the rest are exposed as the static values rather than pointers. This was one of the few non-trivial problems we had to solve for per-interpreter GIL. With immortal objects this is a non-issue. Consolidating Runtime Global State ---------------------------------- As noted in `CPython Runtime State`_ above, there is an active effort (separate from this PEP) to consolidate CPython's global state into the ``_PyRuntimeState`` struct. Nearly all the work involves moving that state from global variables. The project is particularly relevant to this proposal, so below is some extra detail. Benefits to Consolidation ''''''''''''''''''''''''' Consolidating the globals has a variety of benefits: * greatly reduces the number of C globals (best practice for C code) * the move draws attention to runtime state that is unstable or broken * encourages more consistency in how runtime state is used * makes multiple-interpreter behavior more reliable * leads to fixes for long-standing runtime bugs that otherwise haven't been prioritized * exposes (and inspires fixes for) previously unknown runtime bugs * facilitates cleaner runtime initialization and finalization * makes it easier to discover/identify CPython's runtime state * makes it easier to statically allocate runtime state in a consistent way * better memory locality for runtime state * structural layering of the C-API (e.g. ``Include/internal``) Furthermore, much of that work benefits other CPython-related projects: * performance improvements ("faster-cpython") * pre-fork application deployment (e.g. Instagram) * extension module isolation (see :pep:`630`, etc.) * embedding CPython Scale of Work ''''''''''''' The number of global variables to be moved is large enough to matter, but most are Python objects that can be dealt with in large groups (like ``Py_IDENTIFIER``). In nearly all cases, moving these globals to the interpreter is highly mechanical. That doesn't require cleverness but instead requires someone to put in the time. State To Be Moved ''''''''''''''''' The remaining global variables can be categorized as follows: * global objects * static types (incl. exception types) * non-static types (incl. heap types, structseq types) * singletons (static) * singletons (initialized once) * cached objects * non-objects * will not (or unlikely to) change after init * only used in the main thread * initialized lazily * pre-allocated buffers * state Those globals are spread between the core runtime, the builtin modules, and the stdlib extension modules. For a breakdown of the remaining globals, run: .. code-block:: bash ./python Tools/c-analyzer/table-file.py Tools/c-analyzer/cpython/globals-to-fix.tsv Already Completed Work '''''''''''''''''''''' As mentioned, this work has been going on for many years. Here are some of the things that have already been done: * cleanup of runtime initialization (see :pep:`432` / :pep:`587`) * extension module isolation machinery (see :pep:`384` / :pep:`3121` / :pep:`489`) * isolation for many builtin modules * isolation for many stdlib extension modules * addition of ``_PyRuntimeState`` * no more ``_Py_IDENTIFIER()`` * statically allocated: * empty string * string literals * identifiers * latin-1 strings * length-1 bytes * empty tuple Tooling ''''''' As already indicated, there are several tools to help identify the globals and reason about them. * ``Tools/c-analyzer/cpython/globals-to-fix.tsv`` - the list of remaining globals * ``Tools/c-analyzer/c-analyzer.py`` * ``analyze`` - identify all the globals * ``check`` - fail if there are any unsupported globals that aren't ignored * ``Tools/c-analyzer/table-file.py`` - summarize the known globals Also, the check for unsupported globals is incorporated into CI so that no new globals are accidentally added. Global Objects '''''''''''''' Global objects that are safe to be shared (without a GIL) between interpreters can stay on ``_PyRuntimeState``. Not only must the object be effectively immutable (e.g. singletons, strings), but not even the refcount can change for it to be safe. Immortality (:pep:`683`) provides that. (The alternative is that no objects are shared, which adds significant complexity to the solution, particularly for the objects `exposed in the public C-API <capi objects_>`_.) Builtin static types are a special case of global objects that will be shared. They are effectively immutable except for one part: ``__subclasses__`` (AKA ``tp_subclasses``). We expect that nothing else on a builtin type will change, even the content of ``__dict__`` (AKA ``tp_dict``). ``__subclasses__`` for the builtin types will be dealt with by making it a getter that does a lookup on the current ``PyInterpreterState`` for that type. References ========== Related: * :pep:`384` * :pep:`432` * :pep:`489` * :pep:`554` * :pep:`573` * :pep:`587` * :pep:`630` * :pep:`683` * :pep:`3121` Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. [View Less]

6 10

PEP 683: "Immortal Objects, Using a Fixed Refcount" (round 3)
by Eric Snow March 14, 2022

March 14, 2022

I've updated PEP 683 for the feedback I've gotten. Thanks again for that! The updated PEP text is included below. The largest changes involve either the focus of the PEP (internal mechanism to mark objects immortal) or the possible ways that things can break on older 32-bit stable ABI extensions. All other changes are smaller. Given the last round of discussion, I'm hoping this will be the last round before we go to the steering council. -eric ---------------- PEP: 683 Title: Immortal … [View More]Objects, Using a Fixed Refcount Author: Eric Snow <ericsnowcurrently(a)gmail.com>, Eddie Elizondo <eduardo.elizondorueda(a)gmail.com> Discussions-To: https://mail.python.org/archives/list/python-dev@python.org/thread/TPLEYDCX… Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 10-Feb-2022 Python-Version: 3.11 Post-History: 15-Feb-2022, 19-Feb-2022, 28-Feb-2022 Resolution: Abstract ======== Currently the CPython runtime maintains a `small amount of mutable state <Runtime Object State_>`_ in the allocated memory of each object. Because of this, otherwise immutable objects are actually mutable. This can have a large negative impact on CPU and memory performance, especially for approaches to increasing Python's scalability. This proposal mandates that, internally, CPython will support marking an object as one for which that runtime state will no longer change. Consequently, such an object's refcount will never reach 0, and so the object will never be cleaned up. We call these objects "immortal". (Normally, only a relatively small number of internal objects will ever be immortal.) The fundamental improvement here is that now an object can be truly immutable. Scope ----- Object immortality is meant to be an internal-only feature. So this proposal does not include any changes to public API or behavior (with one exception). As usual, we may still add some private (yet publicly accessible) API to do things like immortalize an object or tell if one is immortal. Any effort to expose this feature to users would need to be proposed separately. There is one exception to "no change in behavior": refcounting semantics for immortal objects will differ in some cases from user expectations. This exception, and the solution, are discussed below. Most of this PEP focuses on an internal implementation that satisfies the above mandate. However, those implementation details are not meant to be strictly proscriptive. Instead, at the least they are included to help illustrate the technical considerations required by the mandate. The actual implementation may deviate somewhat as long as it satisfies the constraints outlined below. Furthermore, the acceptability of any specific implementation detail described below does not depend on the status of this PEP, unless explicitly specified. For example, the particular details of: * how to mark something as immortal * how to recognize something as immortal * which subset of functionally immortal objects are marked as immortal * which memory-management activities are skipped or modified for immortal objects are not only CPython-specific but are also private implementation details that are expected to change in subsequent versions. Implementation Summary ---------------------- Here's a high-level look at the implementation: If an object's refcount matches a very specific value (defined below) then that object is treated as immortal. The CPython C-API and runtime will not modify the refcount (or other runtime state) of an immortal object. Aside from the change to refcounting semantics, there is one other possible negative impact to consider. A naive implementation of the approach described below makes CPython roughly 4% slower. However, the implementation is performance-neutral once known mitigations are applied. Motivation ========== As noted above, currently all objects are effectively mutable. That includes "immutable" objects like ``str`` instances. This is because every object's refcount is frequently modified as the object is used during execution. This is especially significant for a number of commonly used global (builtin) objects, e.g. ``None``. Such objects are used a lot, both in Python code and internally. That adds up to a consistent high volume of refcount changes. The effective mutability of all Python objects has a concrete impact on parts of the Python community, e.g. projects that aim for scalability like Instragram or the effort to make the GIL per-interpreter. Below we describe several ways in which refcount modification has a real negative effect on such projects. None of that would happen for objects that are truly immutable. Reducing CPU Cache Invalidation ------------------------------- Every modification of a refcount causes the corresponding CPU cache line to be invalidated. This has a number of effects. For one, the write must be propagated to other cache levels and to main memory. This has small effect on all Python programs. Immortal objects would provide a slight relief in that regard. On top of that, multi-core applications pay a price. If two threads (running simultaneously on distinct cores) are interacting with the same object (e.g. ``None``) then they will end up invalidating each other's caches with each incref and decref. This is true even for otherwise immutable objects like ``True``, ``0``, and ``str`` instances. CPython's GIL helps reduce this effect, since only one thread runs at a time, but it doesn't completely eliminate the penalty. Avoiding Data Races ------------------- Speaking of multi-core, we are considering making the GIL a per-interpreter lock, which would enable true multi-core parallelism. Among other things, the GIL currently protects against races between multiple concurrent threads that may incref or decref the same object. Without a shared GIL, two running interpreters could not safely share any objects, even otherwise immutable ones like ``None``. This means that, to have a per-interpreter GIL, each interpreter must have its own copy of *every* object. That includes the singletons and static types. We have a viable strategy for that but it will require a meaningful amount of extra effort and extra complexity. The alternative is to ensure that all shared objects are truly immutable. There would be no races because there would be no modification. This is something that the immortality proposed here would enable for otherwise immutable objects. With immortal objects, support for a per-interpreter GIL becomes much simpler. Avoiding Copy-on-Write ---------------------- For some applications it makes sense to get the application into a desired initial state and then fork the process for each worker. This can result in a large performance improvement, especially memory usage. Several enterprise Python users (e.g. Instagram, YouTube) have taken advantage of this. However, the above refcount semantics drastically reduce the benefits and have led to some sub-optimal workarounds. Also note that "fork" isn't the only operating system mechanism that uses copy-on-write semantics. Anything that uses ``mmap`` relies on copy-on-write, including sharing data from shared object files between processes. Rationale ========= The proposed solution is obvious enough that both of this proposal's authors came to the same conclusion (and implementation, more or less) independently. The Pyston project `uses a similar approach <Pyston_>`_. Other designs were also considered. Several possibilities have also been discussed on python-dev in past years. Alternatives include: * use a high bit to mark "immortal" but do not change ``Py_INCREF()`` * add an explicit flag to objects * implement via the type (``tp_dealloc()`` is a no-op) * track via the object's type object * track with a separate table Each of the above makes objects immortal, but none of them address the performance penalties from refcount modification described above. In the case of per-interpreter GIL, the only realistic alternative is to move all global objects into ``PyInterpreterState`` and add one or more lookup functions to access them. Then we'd have to add some hacks to the C-API to preserve compatibility for the may objects exposed there. The story is much, much simpler with immortal objects Impact ====== Benefits -------- Most notably, the cases described in the above examples stand to benefit greatly from immortal objects. Projects using pre-fork can drop their workarounds. For the per-interpreter GIL project, immortal objects greatly simplifies the solution for existing static types, as well as objects exposed by the public C-API. In general, a strong immutability guarantee for objects enables Python applications to scale like never before. This is because they can then leverage multi-core parallelism without a tradeoff in memory usage. This is reflected in most of the above cases. Performance ----------- A naive implementation shows `a 4% slowdown`_. We have demonstrated a return to performance-neutral with a handful of basic mitigations applied. See the `mitigation`_ section below. On the positive side, immortal objects save a significant amount of memory when used with a pre-fork model. Also, immortal objects provide opportunities for specialization in the eval loop that would improve performance. .. _a 4% slowdown: https://github.com/python/cpython/pull/19474#issuecomment-1032944709 Backward Compatibility ---------------------- Ideally this internal-only feature would be completely compatible. However, it does involve a change to refcount semantics in some cases. Only immortal objects are affected, but this includes high-use objects like ``None``, ``True``, and ``False``. Specifically, when an immortal object is involved: * code that inspects the refcount will see a really, really large value * the new noop behavior may break code that: * depends specifically on the refcount to always increment or decrement (or have a specific value from ``Py_SET_REFCNT()``) * relies on any specific refcount value, other than 0 or 1 * directly manipulates the refcount to store extra information there * in 32-bit pre-3.11 `Stable ABI`_ extensions, objects may leak due to `Accidental Immortality`_ * such extensions may crash due to `Accidental De-Immortalizing`_ Again, those changes in behavior only apply to immortal objects, not most of the objects a user will access. Furthermore, users cannot mark an object as immortal so no user-created objects will ever have that changed behavior. Users that rely on any of the changing behavior for global (builtin) objects are already in trouble. So the overall impact should be small. Also note that code which checks for refleaks should keep working fine, unless it checks for hard-coded small values relative to some immortal object. The problems noticed by `Pyston`_ shouldn't apply here since we do not modify the refcount. See `Public Refcount Details`_ below for further discussion. Accidental Immortality '''''''''''''''''''''' Hypothetically, a non-immortal object could be incref'ed so much that it reaches the magic value needed to be considered immortal. That means it would accidentally never be cleaned up (by going back to 0). On 64-bit builds, this accidental scenario is so unlikely that we need not worry. Even if done deliberately by using ``Py_INCREF()`` in a tight loop and each iteration only took 1 CPU cycle, it would take 2^60 cycles (if the immortal bit were 2^60). At a fast 5 GHz that would still take nearly 250,000,000 seconds (over 2,500 days)! Also note that it is doubly unlikely to be a problem because it wouldn't matter until the refcount got back to 0 and the object was cleaned up. So any object that hit that magic "immortal" refcount value would have to be decref'ed that many times again before the change in behavior would be noticed. Again, the only realistic way that the magic refcount would be reached (and then reversed) is if it were done deliberately. (Of course, the same thing could be done efficiently using ``Py_SET_REFCNT()`` though that would be even less of an accident.) At that point we don't consider it a concern of this proposal. On 32-bit builds it isn't so obvious. Let's say the magic refcount were 2^30. Using the same specs as above, it would take roughly 4 seconds to accidentally immortalize an object. Under reasonable conditions, it is still highly unlikely that an object be accidentally immortalized. It would have to meet these criteria: * targeting a non-immortal object (so not one of the high-use builtins) * the extension increfs without a corresponding decref (e.g. returns from a function or method) * no other code decrefs the object in the meantime Even at a much less frequent rate incref it would not take long to reach accidental immortality (on 32-bit). However, then it would have to run through the same number of (now noop-ing) decrefs before that one object would be effectively leaking. This is highly unlikely, especially because the calculations assume no decrefs. Furthermore, this isn't all that different from how such 32-bit extensions can already incref an object past 2^31 and turn the refcount negative. If that were an actual problem then we would have heard about it. Between all of the above cases, the proposal doesn't consider accidental immortality a problem. Stable ABI '''''''''' The implementation approach described in this PEP is compatible with extensions compiled to the stable ABI (with the exception of `Accidental Immortality`_ and `Accidental De-Immortalizing`_). Due to the nature of the stable ABI, unfortunately, such extensions use versions of ``Py_INCREF()``, etc. that directly modify the object's ``ob_refcnt`` field. This will invalidate all the performance benefits of immortal objects. However, we do ensure that immortal objects (mostly) stay immortal in that situation. We set the initial refcount of immortal objects to a value high above the magic refcount value, but one that still matches the high bit. Thus we can still identify such objects as immortal. (See `_Py_IMMORTAL_REFCNT`_.) At worst, objects in that situation would feel the effects described in the `Motivation`_ section. Even then the overall impact is unlikely to be significant. Accidental De-Immortalizing ''''''''''''''''''''''''''' 32-bit builds of older stable ABI extensions can take `Accidental Immortality`_ to the next level. Hypothetically, such an extension could incref an object to a value on the next highest bit above the magic refcount value. For example, if the magic value were 2^30 and the initial immortal refcount were thus 2^30 + 2^29 then it would take 2^29 increfs by the extension to reach a value of 2^31, making the object non-immortal. (Of course, a refcount that high would probably already cause a crash, regardless of immortal objects.) The more problematic case is where such a 32-bit stable ABI extension goes crazy decref'ing an already immortal object. Continuing with the above example, it would take 2^29 asymmetric decrefs to drop below the magic immortal refcount value. So an object like ``None`` could be made mortal and subject to decref. That still wouldn't be a problem until somehow the decrefs continue on that object until it reaches 0. For many immortal objects, like ``None``, the extension will crash the process if it tries to dealloc the object. For the other immortal objects, the dealloc might be okay. However, there will be runtime code expecting the formerly-immortal object to be around forever. That code will probably crash. Again, the likelihood of this happening is extremely small, even on 32-bit builds. It would require roughly a billion decrefs on that one object without a corresponding incref. The most likely scenario is the following: A "new" reference to ``None`` is returned by many functions and methods. Unlike with non-immortal objects, the 3.11 runtime will almost never incref ``None`` before giving it to the extension. However, the extension *will* decref it when done with it (unless it returns it). Each time that exchange happens with the one object, we get one step closer to a crash. How realistic is it that some form of that exchange (with a single object) will happen a billion times in the lifetime of a Python process on 32-bit? If it is a problem, how could it be addressed? As to how realistic, the answer isn't clear currently. However, the mitigation is simple enough that we can safely proceed under the assumption that it would be a problem. Here are some possible solutions (only needed on 32-bit): * periodically reset the refcount for immortal objects (only enable this if a stable ABI extension is imported?) * special-case immortal objects in tp_dealloc() for the relevant types (but not int, due to frequency?) * provide a runtime flag for disabling immortality Alternate Python Implementations -------------------------------- This proposal is CPython-specific. However, it does relate to the behavior of the C-API, which may affect other Python implementations. Consequently, the effect of changed behavior described in `Backward Compatibility`_ above also applies here (e.g. if another implementation is tightly coupled to specific refcount values, other than 0, or on exactly how refcounts change, then they may impacted). Security Implications --------------------- This feature has no known impact on security. Maintainability --------------- This is not a complex feature so it should not cause much mental overhead for maintainers. The basic implementation doesn't touch much code so it should have much impact on maintainability. There may be some extra complexity due to performance penalty mitigation. However, that should be limited to where we immortalize all objects post-init and that code will be in one place. Specification ============= The approach involves these fundamental changes: * add `_Py_IMMORTAL_REFCNT`_ (the magic value) to the internal C-API * update ``Py_INCREF()`` and ``Py_DECREF()`` to no-op for objects with the magic refcount (or its most significant bit) * do the same for any other API that modifies the refcount * stop modifying ``PyGC_Head`` for immortal GC objects ("containers") * ensure that all immortal objects are cleaned up during runtime finalization Then setting any object's refcount to ``_Py_IMMORTAL_REFCNT`` makes it immortal. (There are other minor, internal changes which are not described here.) In the following sub-sections we dive into the details. First we will cover some conceptual topics, followed by more concrete aspects like specific affected APIs. Public Refcount Details ----------------------- In `Backward Compatibility`_ we introduced possible ways that user code might be broken by the change in this proposal. Any contributing misunderstanding by users is likely due in large part to the names of the refcount-related API and to how the documentation explains those API (and refcounting in general). Between the names and the docs, we can clearly see answers to the following questions: * what behavior do users expect? * what guarantees do we make? * do we indicate how to interpret the refcount value they receive? * what are the use cases under which a user would set an object's refcount to a specific value? * are users setting the refcount of objects they did not create? As part of this proposal, we must make sure that users can clearly understand on which parts of the refcount behavior they can rely and which are considered implementation details. Specifically, they should use the existing public refcount-related API and the only refcount values with any meaning are 0 and 1. (Some code relies on 1 as an indicator that the object can be safely modified.) All other values are considered "not 0 or 1". This information will be clarified in the `documentation <Documentation_>`_. Arguably, the existing refcount-related API should be modified to reflect what we want users to expect. Something like the following: * ``Py_INCREF()`` -> ``Py_ACQUIRE_REF()`` (or only support ``Py_NewRef()``) * ``Py_DECREF()`` -> ``Py_RELEASE_REF()`` * ``Py_REFCNT()`` -> ``Py_HAS_REFS()`` * ``Py_SET_REFCNT()`` -> ``Py_RESET_REFS()`` and ``Py_SET_NO_REFS()`` However, such a change is not a part of this proposal. It is included here to demonstrate the tighter focus for user expectations that would benefit this change. Constraints ----------- * ensure that otherwise immutable objects can be truly immutable * minimize performance penalty for normal Python use cases * be careful when immortalizing objects that we don't actually expect to persist until runtime finalization. * be careful when immortalizing objects that are not otherwise immutable * ``__del__`` and weakrefs must continue working properly Regarding "truly" immutable objects, this PEP doesn't impact the effective immutability of any objects, other than the per-object runtime state (e.g. refcount). So whether or not some immortal object is truly (or even effectively) immutable can only be settled separately from this proposal. For example, str objects are generally considered immutable, but ``PyUnicodeObject`` holds some lazily cached data. This PEP has no influence on how that state affects str immutability. Immortal Mutable Objects ------------------------ Any object can be marked as immortal. We do not propose any restrictions or checks. However, in practice the value of making an object immortal relates to its mutability and depends on the likelihood it would be used for a sufficient portion of the application's lifetime. Marking a mutable object as immortal can make sense in some situations. Many of the use cases for immortal objects center on immutability, so that threads can safely and efficiently share such objects without locking. For this reason a mutable object, like a dict or list, would never be shared (and thus no immortality). However, immortality may be appropriate if there is sufficient guarantee that the normally mutable object won't actually be modified. On the other hand, some mutable objects will never be shared between threads (at least not without a lock like the GIL). In some cases it may be practical to make some of those immortal too. For example, ``sys.modules`` is a per-interpreter dict that we do not expect to ever get freed until the corresponding interpreter is finalized. By making it immortal, we no longer incur the extra overhead during incref/decref. We explore this idea further in the `mitigation`_ section below. Implicitly Immortal Objects --------------------------- If an immortal object holds a reference to a normal (mortal) object then that held object is effectively immortal. This is because that object's refcount can never reach 0 until the immortal object releases it. Examples: * containers like ``dict`` and ``list`` * objects that hold references internally like ``PyTypeObject.tp_subclasses`` * an object's type (held in ``ob_type``) Such held objects are thus implicitly immortal for as long as they are held. In practice, this should have no real consequences since it really isn't a change in behavior. The only difference is that the immortal object (holding the reference) doesn't ever get cleaned up. We do not propose that such implicitly immortal objects be changed in any way. They should not be explicitly marked as immortal just because they are held by an immortal object. That would provide no advantage over doing nothing. Un-Immortalizing Objects ------------------------ This proposal does not include any mechanism for taking an immortal object and returning it to a "normal" condition. Currently there is no need for such an ability. On top of that, the obvious approach is to simply set the refcount to a small value. However, at that point there is no way in knowing which value would be safe. Ideally we'd set it to the value that it would have been if it hadn't been made immortal. However, that value has long been lost. Hence the complexities involved make it less likely that an object could safely be un-immortalized, even if we had a good reason to do so. _Py_IMMORTAL_REFCNT ------------------- We will add two internal constants:: _Py_IMMORTAL_BIT - has the top-most available bit set (e.g. 2^62) _Py_IMMORTAL_REFCNT - has the two top-most available bits set The actual top-most bit depends on existing uses for refcount bits, e.g. the sign bit or some GC uses. We will use the highest bit possible after consideration of existing uses. The refcount for immortal objects will be set to ``_Py_IMMORTAL_REFCNT`` (meaning the value will be halfway between ``_Py_IMMORTAL_BIT`` and the value at the next highest bit). However, to check if an object is immortal we will compare (bitwise-and) its refcount against just ``_Py_IMMORTAL_BIT``. The difference means that an immortal object will still be considered immortal, even if somehow its refcount were modified (e.g. by an older stable ABI extension). Note that top two bits of the refcount are already reserved for other uses. That's why we are using the third top-most bit. Affected API ------------ API that will now ignore immortal objects: * (public) ``Py_INCREF()`` * (public) ``Py_DECREF()`` * (public) ``Py_SET_REFCNT()`` * (private) ``_Py_NewReference()`` API that exposes refcounts (unchanged but may now return large values): * (public) ``Py_REFCNT()`` * (public) ``sys.getrefcount()`` (Note that ``_Py_RefTotal`` and ``sys.gettotalrefcount()`` will not be affected.) Also, immortal objects will not participate in GC. Immortal Global Objects ----------------------- All runtime-global (builtin) objects will be made immortal. That includes the following: * singletons (``None``, ``True``, ``False``, ``Ellipsis``, ``NotImplemented``) * all static types (e.g. ``PyLong_Type``, ``PyExc_Exception``) * all static objects in ``_PyRuntimeState.global_objects`` (e.g. identifiers, small ints) The question of making them actually immutable (e.g. for per-interpreter GIL) is not in the scope of this PEP. Object Cleanup -------------- In order to clean up all immortal objects during runtime finalization, we must keep track of them. For GC objects ("containers") we'll leverage the GC's permanent generation by pushing all immortalized containers there. During runtime shutdown, the strategy will be to first let the runtime try to do its best effort of deallocating these instances normally. Most of the module deallocation will now be handled by ``pylifecycle.c:finalize_modules()`` which cleans up the remaining modules as best as we can. It will change which modules are available during ``__del__``, but that's already explicitly undefined behavior in the docs. Optionally, we could do some topological ordering to guarantee that user modules will be deallocated first before the stdlib modules. Finally, anything left over (if any) can be found through the permanent generation GC list which we can clear after ``finalize_modules()``. For non-container objects, the tracking approach will vary on a case-by-case basis. In nearly every case, each such object is directly accessible on the runtime state, e.g. in a ``_PyRuntimeState`` or ``PyInterpreterState`` field. We may need to add a tracking mechanism to the runtime state for a small number of objects. None of the cleanup will have a significant effect on performance. .. _mitigation: Performance Regression Mitigation --------------------------------- In the interest of clarity, here are some of the ways we are going to try to recover some of the lost `performance <Performance_>`_: * at the end of runtime init, mark all objects as immortal * drop refcount operations in code where we know the object is immortal (e.g. ``Py_RETURN_NONE``) * specialize for immortal objects in the eval loop (see `Pyston`_) Regarding that first point, we can apply the concept from `Immortal Mutable Objects`_ in the pursuit of getting back some of that 4% performance we lose with the naive implementation of immortal objects. At the end of runtime init we can mark *all* objects as immortal and avoid the extra cost in incref/decref. We only need to worry about immutability with objects that we plan on sharing between threads without a GIL. Note that none of this section is part of the proposal. The above is included here for clarity. Possible Changes ---------------- * mark every interned string as immortal * mark the "interned" dict as immortal if shared else share all interned strings * (Larry,MvL) mark all constants unmarshalled for a module as immortal * (Larry,MvL) allocate (immutable) immortal objects in their own memory page(s) Documentation ------------- The immortal objects behavior and API are internal, implementation details and will not be added to the documentation. However, we will update the documentation to make public guarantees about refcount behavior more clear. That includes, specifically: * ``Py_INCREF()`` - change "Increment the reference count for object o." to "Indicate taking a new reference to object o." * ``Py_DECREF()`` - change "Decrement the reference count for object o." to "Indicate no longer using a previously taken reference to object o." * similar for ``Py_XINCREF()``, ``Py_XDECREF()``, ``Py_NewRef()``, ``Py_XNewRef()``, ``Py_Clear()`` * ``Py_REFCNT()`` - add "The refcounts 0 and 1 have specific meanings and all others only mean code somewhere is using the object, regardless of the value. 0 means the object is not used and will be cleaned up. 1 means code holds exactly a single reference." * ``Py_SET_REFCNT()`` - refer to ``Py_REFCNT()`` about how values over 1 may be substituted with some over value We *may* also add a note about immortal objects to the following, to help reduce any surprise users may have with the change: * ``Py_SET_REFCNT()`` (a no-op for immortal objects) * ``Py_REFCNT()`` (value may be surprisingly large) * ``sys.getrefcount()`` (value may be surprisingly large) Other API that might benefit from such notes are currently undocumented. We wouldn't add such a note anywhere else (including for ``Py_INCREF()`` and ``Py_DECREF()``) since the feature is otherwise transparent to users. Reference Implementation ======================== The implementation is proposed on GitHub: https://github.com/python/cpython/pull/19474 Open Issues =========== * how realistic is the `Accidental De-Immortalizing`_ concern? References ========== .. _Pyston: https://mail.python.org/archives/list/python-dev@python.org/message/JLHRTBJ… Prior Art --------- * `Pyston`_ Discussions ----------- This was discussed in December 2021 on python-dev: * https://mail.python.org/archives/list/python-dev@python.org/thread/7O3FUA52… * https://mail.python.org/archives/list/python-dev@python.org/thread/PNLBJBNI… Runtime Object State -------------------- Here is the internal state that the CPython runtime keeps for each Python object: * `PyObject.ob_refcnt`_: the object's `refcount <refcounting_>`_ * `_PyGC_Head <PyGC_Head>`_: (optional) the object's node in a list of `"GC" objects <refcounting_>`_ * `_PyObject_HEAD_EXTRA <PyObject_HEAD_EXTRA>`_: (optional) the object's node in the list of heap objects ``ob_refcnt`` is part of the memory allocated for every object. However, ``_PyObject_HEAD_EXTRA`` is allocated only if CPython was built with ``Py_TRACE_REFS`` defined. ``PyGC_Head`` is allocated only if the object's type has ``Py_TPFLAGS_HAVE_GC`` set. Typically this is only container types (e.g. ``list``). Also note that ``PyObject.ob_refcnt`` and ``_PyObject_HEAD_EXTRA`` are part of ``PyObject_HEAD``. .. _PyObject.ob_refcnt: https://github.com/python/cpython/blob/80a9ba537f1f1666a9e6c5eceef4683f8696… .. _PyGC_Head: https://github.com/python/cpython/blob/80a9ba537f1f1666a9e6c5eceef4683f8696… .. _PyObject_HEAD_EXTRA: https://github.com/python/cpython/blob/80a9ba537f1f1666a9e6c5eceef4683f8696… .. _refcounting: Reference Counting, with Cyclic Garbage Collection -------------------------------------------------- Garbage collection is a memory management feature of some programming languages. It means objects are cleaned up (e.g. memory freed) once they are no longer used. Refcounting is one approach to garbage collection. The language runtime tracks how many references are held to an object. When code takes ownership of a reference to an object or releases it, the runtime is notified and it increments or decrements the refcount accordingly. When the refcount reaches 0, the runtime cleans up the object. With CPython, code must explicitly take or release references using the C-API's ``Py_INCREF()`` and ``Py_DECREF()``. These macros happen to directly modify the object's refcount (unfortunately, since that causes ABI compatibility issues if we want to change our garbage collection scheme). Also, when an object is cleaned up in CPython, it also releases any references (and resources) it owns (before it's memory is freed). Sometimes objects may be involved in reference cycles, e.g. where object A holds a reference to object B and object B holds a reference to object A. Consequently, neither object would ever be cleaned up even if no other references were held (i.e. a memory leak). The most common objects involved in cycles are containers. CPython has dedicated machinery to deal with reference cycles, which we call the "cyclic garbage collector", or often just "garbage collector" or "GC". Don't let the name confuse you. It only deals with breaking reference cycles. See the docs for a more detailed explanation of refcounting and cyclic garbage collection: * https://docs.python.org/3.11/c-api/intro.html#reference-counts * https://docs.python.org/3.11/c-api/refcounting.html * https://docs.python.org/3.11/c-api/typeobj.html#c.PyObject.ob_refcnt * https://docs.python.org/3.11/c-api/gcsupport.html Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. [View Less]

3 6

SC accepted PEP 594: Removing dead batteries from the standard library
by Victor Stinner March 14, 2022

March 14, 2022

Hi, Oh, the Steering Council accepted PEP 594 "Removing dead batteries from the standard library"! I just saw the announcement on Discourse. Congratulations Christian and Brett! This PEP, first proposed in 2019, wasn't an easy one. https://peps.python.org/pep-0594/ Gregory P. Smith's message on Discourse: """ On behalf of the Python Steering Council, We are accepting PEP-594 32 Removing dead batteries from the standard library. It removes a non-controversial set of very old unmaintained … [View More]

1 0

An unambiguous way of initializing an empty dictionary and set
by joao.p.f.batista.97＠gmail.com March 14, 2022

March 14, 2022

Currently: l = [] # new empty list t = () # new empty tuple s = set() # new empty set (no clean and consistent way of initializing regarding the others) <<< d = {} # new empty dictionary Possible solution: s = {} # new empty set d = {:} # new empty dictionary (the ":" is a reference to key-value pairs) Current workaround at least for consistency: l = list() # new empty list t = tuple() # new empty tuple s = set() # new empty set d = dict() # new empty dictionary However, it doesn't … [View More]

5 6

Re: cpython: simplify and rewrite the zipimport part of 702009f3c0b1 a bit
by Tommie Cadieux March 13, 2022

March 13, 2022

1 0

Supported make implementations
by Lincoln Auster March 12, 2022

March 12, 2022

Hi, When attempting to build and run doctests for cpython on FreeBSD, I got a vague error from the default BSD make. gmake, of course, ran fine. Is GNU make the only supported make implementation? If so, did I miss that in the devguide, or should that be in there somewhere? -- Lincoln Auster They/Them

2 1

Summary of Python tracker Issues
by Python tracker March 11, 2022

March 11, 2022

ACTIVITY SUMMARY (2022-03-04 - 2022-03-11) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7233 ( +9) closed 51481 (+56) total 58714 (+65) Open issues with patches: 2940 Issues opened (43) ================== #45138: [sqlite3] expand bound values in traced statements when possib https://bugs.python.org/issue45138 reopened by erlendaasland #46924: … [View More]

1 0

RFC on PEP 655: Required[] and NotRequired[] for TypedDict
by David Foster March 10, 2022

March 10, 2022

This PEP [1] introduces syntax to mark individual keys of a TypedDict as either required or potentially-missing. Previously the only way to have a TypedDict with mixed required and non-required keys was to define two TypedDicts - one with total=True and one with total=False - and have one of those TypedDicts inherit from the other. This PEP introduces special forms Required[] and NotRequired[] to the "typing" module that can be used to directly mark individual items of a TypedDict as … [View More]

4 6

PSA: Linux vulnerability
by Ethan Furman March 8, 2022

March 8, 2022

https://arstechnica.com/information-technology/2022/03/linux-has-been-bitte… -- ~Ethan~

1 0

[RELEASE] Python 3.11.0a6 is available
by Pablo Galindo Salgado March 7, 2022

March 7, 2022

There are no easy releases these days! :sweat: After a week of delay due to several release blockers, buildbot problems and pandemic-related difficulties here is 3.11.0a6 for you to test. https://www.python.org/downloads/release/python-3110a6/ **This is an early developer preview of Python 3.11** # Major new features of the 3.11 series, compared to 3.10 Python 3.11 is still in development. This release, 3.11.0a6 is the sixth of seven planned alpha releases. Alpha releases are intended to … [View More]make it easier to test the current state of new features and bug fixes and to test the release process. During the alpha phase, features may be added up until the start of the beta phase (2022-05-06) and, if necessary, may be modified or deleted up until the release candidate phase (2022-08-01). Please keep in mind that this is a preview release and its use is **not** recommended for production environments. Many new features for Python 3.11 are still being planned and written. Among the new major new features and changes so far: * [PEP 657](https://www.python.org/dev/peps/pep-0657/) -- Include Fine-Grained Error Locations in Tracebacks * [PEP 654](https://www.python.org/dev/peps/pep-0654/) -- Exception Groups and except* * [PEP 673](https://www.python.org/dev/peps/pep-0673/) -- Self Type * [PEP 646](https://www.python.org/dev/peps/pep-0646/)-- Variadic Generics * The [Faster Cpython Project](https://github.com/faster-cpython) is already yielding some exciting results: this version of CPython 3.11 is ~12% faster on the geometric mean of the [PyPerformance benchmarks]( speed.python.org), compared to 3.10.0. * Hey, **fellow core developer,** if a feature you find important is missing from this list, let me know. The next pre-release of Python 3.11 will be 3.11.0a7, currently scheduled for Tuesday, 2022-04-05. # More resources * [Online Documentation](https://docs.python.org/3.11/) * [PEP 664](https://www.python.org/dev/peps/pep-0664/), 3.11 Release Schedule * Report bugs at [https://bugs.python.org](https://bugs.python.org). * [Help fund Python and its community](/psf/donations/). # And now for something completely different In astrophysics and nuclear physics, nuclear pasta is a theoretical type of degenerate matter that is postulated to exist within the crusts of neutron stars. If it does in fact exist, nuclear pasta is the strongest material in the universe. Between the surface of a neutron star and the quark-gluon plasma at the core, at matter densities of 1014 g/cm3, nuclear attraction and Coulomb repulsion forces are of similar magnitude. The competition between the forces leads to the formation of a variety of complex structures assembled from neutrons and protons. Astrophysicists call these types of structures nuclear pasta because the geometry of the structures resembles various types of pasta. There are several phases of evolution (I swear these names are real), including the gnocchi phase, the spaghetti phase, the lasagna phase, the bucatini phase and the Swiss cheese phase. # We hope you enjoy those new releases! Thanks to all of the many volunteers who help make Python Development and these releases possible! Please consider supporting our efforts by volunteering yourself or through organization contributions to the Python Software Foundation. Your friendly release team, Pablo Galindo @pablogsal Ned Deily @nad Steve Dower @steve.dower [View Less]

1 0

Improvements to the sys.path initialization documentation
by Russel Webber March 7, 2022

March 7, 2022

Hi All, Steve Dower, Eryk Sun and I have been working on improving the documentation around how sys.path is initialised and therefore how Python modules are found. We're moving the details from https://docs.python.org/3.11/using/windows.html#finding-modules to a platform-agnostic new section in https://docs.python.org/3.11/using/cmdline.html We're aiming to have one place in the documentation where all the ways to customize sys.path (PYTHONPATH, ._pth, site, etc) are at least mentioned. We'… [View More]

4 6

Accepted: PEP 676 - PEP Infrastructure Process
by Barry Warsaw March 6, 2022

March 6, 2022

As PEP-Delegate, I’m very happy to accept PEP 676 (PEP Infrastructure Process). https://www.python.org/dev/peps/pep-0676/ https://discuss.python.org/t/request-for-comment-making-pep-rendering-self-… Congratulations to Adam Turner and everyone who participated in the discussions. This will make reading and authoring PEPs a much nicer experience. Cheers, -Barry

1 0

PEP 682: Format Specifier for Signed Zero
by John Belmonte March 6, 2022

March 6, 2022

Just a note that this PEP is being discussed at https://discuss.python.org/t/pep-682-format-specifier-for-signed-zero TLDR; When formatting a number (especially with rounding options), often you don't want negative-signed zero as an output. A format spec extension enables normalizing such values to positive zero. Regards, --John Belmonte

2 1

Make __mro_entries__ mandatory for non-types
by Serhiy Storchaka March 5, 2022

March 5, 2022

Currently the class can inherit from arbitrary objects, not only types. If the object was not designed for this you can get a mysterious mistake, e g.: >>> class A(1): ... ... Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: int() takes at most 2 arguments (3 given) If the object's constructor is compatible with type constructor by accident you can an unexpected result. To prevent this we usually add an explicit __mro_entries__() … [View More]

4 6

Summary of Python tracker Issues
by Python tracker March 4, 2022

March 4, 2022

ACTIVITY SUMMARY (2022-02-25 - 2022-03-04) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7224 ( +4) closed 51425 (+62) total 58649 (+66) Open issues with patches: 2938 Issues opened (48) ================== #44807: typing.Protocol silently overrides __init__ method of delivere https://bugs.python.org/issue44807 reopened by gvanrossum #46623: … [View More]

1 0

build cpython on M1 Pro mac
by Yves Duprat March 4, 2022

March 4, 2022

Hi, I built cpython on a mac OSX 12.01. Python runs well. When starts, Python shows that version info which looks good: 'Python 3.11.0a5+ (heads/fix-issue-43352:f899da7fe5, Feb 25 2022, 10:04:53) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin' But when I want to used it in VSCode to debug a script, I have the following message: ------- This version of python seems to be incorrectly compiled (internal generated filenames are not absolute). This may make the debugger miss breakpoints. Related … [View More]

2 4

Test error
by sachin1122621003＠gmail.com March 3, 2022

March 3, 2022

I am new to open source and think to contribute to python . I was going through the developer guide and when i run this code "./python -m test -j3" on my terminal i got this result which say it fail .Now can someone tell me this is normal or happning to me only . If this happen to everyone how can i fix this == Tests result: FAILURE == 380 tests OK. 2 tests failed: test_compileall test_distutils 43 tests skipped: test_asdl_parser test_check_c_globals test_clinic test_curses … [View More]

3 2

Python 3.11.0a6 is blocked
by Pablo Galindo Salgado March 3, 2022

March 3, 2022

Hi everyone, Unfortunately, we have some issues marked as release blockers that are holding the 3.11.0a6 release. Some of these issues have been solved but we still have a couple of them. Hopefully, we will solve these soon and I will proceed with the release. Thanks for your understanding. Regards from sunny Salamanca, Pablo Galindo Salgado

2 3

Making Cinder more broadly available
by Dino March 1, 2022

March 1, 2022

Hi, Cinder <https://github.com/facebookincubator/cinder> is Meta’s performance-oriented version of CPython 3.8. It has been in use as the production Python behind Instagram server for years, as well as powering various other Python applications across Meta. We are interested in making the improvements in Cinder more broadly available to the Python community. As a collection of features and components built on top of CPython, we’d like to: 1. Merge the feature into CPython for core … [View More]

1 0

October/November/December Steering Council update
by Pablo Galindo Salgado March 1, 2022

March 1, 2022

We’ve just published the October, November and December steering council update for 2021, also included below: https://github.com/python/steering-council/blob/main/updates/2021-10-steeri… https://github.com/python/steering-council/blob/main/updates/2021-11-steeri… https://github.com/python/steering-council/blob/main/updates/2021-12-steeri… Just as a reminder, if you have any questions or concerns, feel free to contact us or open an issue in the SC repo: https://github.com/python/steering-… [View More]council *October 11* - SC met with the Developer-in-Residence and provided feedback on the Developer-in-Residence activity blog. - SC decided to keep 2020 dates for 2021 election. - SC discussed the draft about the SC's position on the PEP 649 / PEP 563 situation (stringified annotations) and provided feedback. - SC discussed whether or not they should extend their weekly meetings and/or do more async communications. They also decided that the SC should continue having meetings and working during the elections. *October 18* - Steering Council met with the GitHub Issues PM for an update on the migration. Priority right now is getting a testing repo out to the core dev sprint and triagers by Wednesday. - SC reviewed PEP 663 (Standardizing Enum str(), repr(), and format() behaviors) and potential PEP Delegates. - SC reviewed and discussed issues open on https://github.com/python/steering-council/issues *October 19* - SC met with the Search Firm the PSF is using for the Executive Director search, to provide input on the search parameters, and suggest potential candidates. *October 25* - Steering Council did a one hour Q&A recorded session with Visionary sponsor, Bloomberg. - Steering Council discussed PEP 649 and PEP 563 (stringified annotations). The group may want to delegate this to Łukasz but will discuss this more next week. - The candidate PEP Delegate for PEP 663 (Standardizing Enum str(), repr(), and format() behaviors) declined. The Steering Council will think of who else it can be delegated to, or take it on themselves. November 1 - The SC discussed PEP 649 -- Deferred Evaluation Of Annotations Using Descriptors <https://www.python.org/dev/peps/pep-0649/> & PEP 563 -- Postponed Evaluation of Annotations <https://www.python.org/dev/peps/pep-0563/>. The SC decided it would tell the community that more information is needed. Barry will draft an email explaining the current situation and calling for help from static and dynamic typing users. - The SC reviewed the open issues in the public Steering Council repo: https://github.com/python/steering-council/issues. - The SC discussed SC promotion during the call for nomination period, which started on Nov 1. - The SC discussed PEP 663 - Standardizing Enum str(), repr(), and format() behaviors <https://www.python.org/dev/peps/pep-0663/> and that Barry would write the rejection draft. - Brett informed the SC of an issue involving the return type of iterators and the SC decided it would continue discussing it at the next meeting (https://bugs.python.org/issue45250). <https://github.com/python/steering-council/blob/main/updates/2021-11-steeri…>November 8 - The SC had their bi-weekly meeting with the Developer-in-Residence. Received an update from Łukasz on what he is working on. - The SC discussed PEP 663 - Standardizing Enum str(), repr(), and format() behaviors <https://www.python.org/dev/peps/pep-0663/> and after some initial discussion, it was decided that it needs more discussion on python-dev@. - The SC discussed PEP 467 -- Minor API improvements for binary sequences <https://www.python.org/dev/peps/pep-0467/>. bchr() was taken out like the SC had requested and then an ascii() method was added in without any mention. Pablo will draft a response. - PEP 554 -- Multiple Interpreters in the Stdlib <https://www.python.org/dev/peps/pep-0554/> was discussed. A proof of concept is needed and the SC is not yet ready to review the PEP without it. <https://github.com/python/steering-council/blob/main/updates/2021-11-steeri…>November 15 - The SC met with Ezio and got an update on what Ezio was working on. SC told Ezio not to be overly concerned with perfection and not to strive for it. The SC gave Ezio a deadline of Dec 7th to get everything migrated over. - The SC discussed a PR to the CPython repo regarding __iter__ and iterators being inconsistently required by CPython ( https://github.com/python/cpython/pull/29170). The SC's decision was to update the definitions to say it's a CPython implementation detail that it is inconsistent in its checking whether what an iterable returns is a proper iterator or not (i.e. keep the __iter__ requirement but admit it isn't rigorously enforced by CPython as a historical accident). Brett committed updates to the PR. - The SC discussed a pull request in the “python/voters” repository that allows core devs to trigger automerge behavior from labels and decided it is fine to merge. - Carol gave an update on the documentation WG: The WG is going to a series of 1 hour monthly meetings and then do the rollout next year. - Barry updated the text for the PEP 663 rejection and sent it out to python-dev@. - Group reviewed text for "The current state of typing PEPs" (i.e. request for PEP 563 and 649) and Barry sent it to python-dev@ <https://mail.python.org/archives/list/python-dev@python.org/message/VIZEBX5…> . - The SC reviewed PEP 646 (Variadic Generics) and Barry sent the approval to the PEP team. - The SC discussed how notes will be taken after Ewa leaves. - The SC discussed PEP 518 (Specifying Minimum Build System Requirements for Python Projects) and how some folks are split on toml. <https://github.com/python/steering-council/blob/main/updates/2021-11-steeri…>November 22 - The SC met with Łukasz and received an update from Łukasz on his recent work. Additionally, the SC informed Łukasz he should work on upgrading the CLA process. Łukasz will reach out to Yury and Ezio. - The SC discussed Victor's request to remove asynchat and friends prematurely. The SC decided that the removal of asynchat needs to wait another 6 months. - The SC discussed Ethan's email to the SC asking about the re module and its RegexFlag. - The SC discussed the exception for Py_SET_TYPE request. Pablo will write a response to Petr request stating that the change needs to be rolled back in case there are any issues and they need to document any breakage in 3.11. - The SC discussed the various questions pertaining to the current state of typing PEPs. Thomas will draft a response to Chris and Paul's questions and the group discussed Barry's private conversation with a core dev about this as well. - The SC decided that it should be the 2022 SC that decides what happens to python-dev@. <https://github.com/python/steering-council/blob/main/updates/2021-11-steeri…>November 29 - The SC met with Ezio. The date to do the full migration has been pushed to mid-January. By Dec 7th, Ezio will put all issues into a test repo for python-dev to test and then the migration will happen when Steffen from GitHub returns from vacation in January. - The SC discussed PEP 659 - Specializing Adaptive Interpreter <https://www.python.org/dev/peps/pep-0659/> and whether or not it should be a PEP. It was determined that the informational aspect of the writeup is lacking such as how trade-offs are decided. The SC also discussed how different companies working on Python should work especially in terms of transparency. For the time being there will be no communications about this to the PEP authors or the community as the SC wants to discuss this further. This discussion will continue with the new SC as well. December 6 - The Steering Council met with Łukasz to discuss his progress and also introduced Ee to the meeting since Ee will be taking PSF lead on supporting Łukasz going forward. Łukasz updated the team on CLA work, testing Irit's PR for Exception Groups, adding a new triager, and giving a keynote at the PyCode Conference. - The Steering Council discussed the issue Jason posted <https://github.com/python/steering-council/issues/89> about deprecating importlib.resources legacy API. It was decided that Brett would respond to the comment stating that Jason is following policy and that the SC is satisfied, but that the larger discussion on changing the policy should happen on python-dev. The group also discussed how Python's deprecation policy impacts users and library authors. - The group said their goodbyes on Ewa's last day. <https://github.com/python/steering-council/blob/main/updates/2021-12-steeri…>December 13 - The Steering Council met with Ezio. There is no full import from BPO yet, but he will create one this week. The final migration is still tentatively scheduled for mid-January. Ezio will loop in Thomas on meetings with GitHub, which Ewa previously attended, or other SC members if Thomas can't make it. - The SC prepared for a Talk Python podcast recording after the meeting. <https://github.com/python/steering-council/blob/main/updates/2021-12-steeri…>December 13 (handoff meeting) - The old Steering Council and the New Steering Council met for the handoff meeting. - The SC discussed several aspects on how it has operated so far. - The SC received an updated from Thomas on the status on the search for a new PSF director. - The SC briefly reflected and discussed the diversity in the Steering Council and how to improve the situation in the future. This topic will be discussed further in the future. - The SC approved a request for a wording change in PEP 654 - Exception groups <https://www.python.org/dev/peps/pep-0654/>. - The SC had an overview of unresolved issues: definition/rules for the standard library, split in communication channels and the situation regarding type annotations with PEP 563 and 649. - The new SC decided to use discord as a new communication channel. - The SC discussed the status on the GitHub issues migration and what to do next to push the work forward. Regards from sunny Salamanca, Pablo Galindo Salgado [View Less]

3 2