Mailman 3 March 2022 - Python-Dev

Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API
by Jason Ansel April 20, 2022

April 20, 2022

The PyTorch team plans to use PEP 523 as a part of PyTorch 2.0, so this proposal may break the next major release of PyTorch. The related project is TorchDynamo, which can be found here: https://github.com/facebookresearch/torchdynamo We will likely move this into the core of PyTorch closer to release. If the changed happens, would PyTorch still be able to use the eval frame API? Or would it prevent from being used entirely?

11 25

Slowly bend the C API towards the limited API to get a stable ABI for everyone
by Victor Stinner April 20, 2022

April 20, 2022

Hi, There is a reason why I'm bothering C extensions maintainers and Python core developers with my incompatible C API changes since Python 3.8. Let me share my plan with you :-) In 2009 (Python 3.2), Martin v. Löwis did an amazing job with the PEP 384 "Defining a Stable ABI" to provide a "limited C API" and a "stable ABI" for C extensions: build an extension once, use it on multiple Python versions. Some projects like PyQt5 and cryptograpy use it, but it is just a drop in the PyPI ocean (353,084 projects). I'm trying to bend the "default" C API towards this "limited C API" to make it possible tomorrow to build *more* C extensions for the stable ABI. My goal is that the stable ABI would be the default, and only a minority of C extensions would opt-out because they need to access to more functions for best performance. The basic problem is that at the ABI level, C extensions must only call functions, rather than getting and setting directly to structure members. Structures changes frequently in Python (look at changes between Python 3.2 and Python 3.11), and any minor structure change breaks the ABI. The limited C API hides structures and only use function calls to solve this problem. Since 2020, I'm modifying the C API, one function by one, to slowly hide implementations (prepare the API to make strutures opaque). I focused on the following structures: * PyObject and PyVarObject (bpo-39573) * PyTypeObject (bpo-40170) * PyFrameObject (bpo-40421) * PyThreadState (bpo-39947) The majority of C extensions use functions and macros, they don't access directly structure members. There are a few members which are sometimes accessed directly which prevents making these structures opaque. For example, some old C extensions use obj->ob_type rather than Py_TYPE(obj). Fixing the minority of C extensions should benefit to the majority which may become compatible with the stable ABI. I am also converting macros to static inline functions to fix their API: define parameter types, result type and avoid surprising macros side effects ("macro pitfalls"). I wrote the PEP 670 "Convert macros to functions in the Python C API" for these changes. I wrote the upgrade_pythoncapi.py tool in my pythoncapi_project (*) which modify C code to use Py_TYPE(), Py_SIZE() and Py_REFCNT() rather than accessing directly PyObject and PyVarObject members. (*) https://github.com/pythoncapi/pythoncapi_compat In this tool, I also added "Borrow" variant of functions like PyFrame_GetCode() which returns a strong reference, to replace frame->f_code with _PyFrame_GetCodeBorrow(). In Python 3.11, you cannot use the frame->f_code member anymore, since it has been removed! You must call PyFrame_GetCode() (or pythoncapi_compat _PyFrame_GetCodeBorrow() variant). There are also a few macros which can be used as l-values like Py_TYPE(): "Py_TYPE(type1) = type2" must now be written "Py_SET_TYPE(type1, type2)" to avoid setting directly the tp_type type at the ABI level. I proposed the PEP 674 "Disallow using Py_TYPE() and Py_SIZE() macros as l-values" to solve these issues. Currently, many "functions" are still implemented as macros or static inline functions, so C extensions still access structure members at the ABI level for best Python performance. Converting these to regular functions has an impact on performance and I would prefer to first write a PEP giving the rationale for that. Today, it is not possible yet to build numpy for the stable ABI. The gap is just too large for this big C extension. But step by step, the C API becomes closer to the limited API, and more and more code is ready to be built for the stable ABI. Well, these C API changes have other advantages, like preparing Python for further optimizations, ease Python maintenance, clarify the seperation between the limited C API and the default C API, etc. ;-) Victor -- Night gathers, and now my watch begins. It shall not end until my death.

12 38

PEP 678: Enriching Exceptions with Notes
by Zac Hatfield Dodds April 12, 2022

April 12, 2022

Hi all, I've written PEP 678, proposing the addition of a __note__ attribute which can be used to enrich displayed exceptions. This is particularly useful with ExceptionGroup, or where exception chaining is unsuitable, and ensures that 'orphan' output is not produced if the annotated exception is caught. Link to the PEP: https://www.python.org/dev/peps/pep-0678/ *Please, redirect all discussions to:* https://discuss.python.org/t/pep-678-enriching-exceptions-with-notes/13374 Thanks for your time, and to those who have already made (very helpful!) comments! Best wishes from just-moved-to San Francisco, Zac Hatfield-Dodds

2 1

It's now time to deprecate the stdlib urllib module
by Victor Stinner March 31, 2022

March 31, 2022

Hi, I propose to deprecate the urllib module in Python 3.11. It would emit a DeprecationWarning which warn users, so users should consider better alternatives like urllib3 or httpx: well known modules, better maintained, more secure, support HTTP/2 (httpx), etc. I don't propose to schedule its removal. Let's discuss the removal in 1 or 2 years. -- urllib has many abstraction to support a wide range of protocols with "handlers": HTTP, HTTPS, FTP, "local file", proxy, HTTP authentication, HTTP Cookie, etc. A simple HTTP request using Basic Authentication requires 10-20 lines of code, whereas it should be a single line. Users (me included) don't like urllib API which was too complicated for common tasks. -- Unhappy users created multiple better alternatives to the stdlib urllib module. In 2008, the "urllib3" module was created to provide an API designed to be as simple as possible for the most common HTTP and HTTPS requests. Example: req = http.request('GET', 'http://httpbin.org/robots.txt'). In 2011, the "requests" module based on urllib3 was created. In 2013, the "aiohttp" module based on asyncio was created. In 2015, new "httpx" module was created: req = httpx.get('https://www.example.org/') Not only httpx has a regular "synchronous" API (blocking function calls), but it also has an asynchronous API! Sadly, while HTTP/3 is being developed, it seems like in this list, httpx is the only HTTP client library supporting HTTP/2 currently :-( For HTTP/2, I also found the "httplib2" module. For HTTP/3, I found the "http3" and "aioquic" modules. -- Let's come back to urllib: * It's API is too complicated * It doesn't support HTTP/2 nor HTTP/3 * It's barely maintained: there are 121 open issues including 3 security issues! The 3 open security issues: * bpo-33661 open 2018; * bpo-36338 open in 2019; * bpo-45795 open in 2021. Usually, it's bad when you refer to an open security issue by its creation year :-( The urllib module has long history of security vulnerabilities. List of *fixed* vulnerabilities: * 2011 (bpo-11662): https://python-security.readthedocs.io/vuln/urllib-redirect.html * 2017 (bpo-30119): https://python-security.readthedocs.io/vuln/urllib-ftp-stream-injection.html * 2017 (bpo-30500): https://python-security.readthedocs.io/vuln/urllib-connects-wrong-host.html * 2019 (bpo-35907): https://python-security.readthedocs.io/vuln/urllib-local-file-scheme.html * 2019 (bpo-38826): https://python-security.readthedocs.io/vuln/urllib-basic-auth-regex.html * 2021 (bpo-42967): https://python-security.readthedocs.io/vuln/urllib-query-string-semicolon-s… * 2021 (bpo-43075): https://python-security.readthedocs.io/vuln/urllib-basic-auth-regex2.html * 2021 (bpo-44022): https://python-security.readthedocs.io/vuln/urllib-100-continue-loop.html urllib is a package made of 4 parts: * urllib.request for opening and reading URLs * urllib.error containing the exceptions raised by urllib.request * urllib.parse for parsing URLs * urllib.robotparser for parsing robots.txt files I propose to deprecate all of them. Maybe the deprecation can be different for each sub-module? Victor -- Night gathers, and now my watch begins. It shall not end until my death.

39 75

Enhancing generic type documentation in the standard library
by Luciano Ramalho March 30, 2022

March 30, 2022

Hello, everyone! I believe our documentation about types needs another overhaul. We now have lots of generic types in the standard library, but their formal type parameters are poorly documented—or not documented at all—in the standard library documentation. More importantly: the documentation we have about specific covariant/contravariant types is now in entries in the `typing` module that are all deprecated since PEP 585 was implemented in Python 3.9. Below I present two of many examples where the documentation of a generic type is not great. However, if people agree this is a problem, we need to discuss where and how to put the documentation in a way that is not too disruptive to users of Python who don't know or don't care about type hints, for many reasons that we should not judge. For example, where do we document the fact that `dict` accepts two invariant formal type parameters, and that `frozenset` accepts one contravariant type parameter? What do you think? Cheers, Luciano _________________________________________ EXAMPLE 1: `Callable` variance is not documented For example, in the `Callable` type, the `ReturnType` parameter is covariant, and the `ParameterType` parameters are all contravariant. But there is no mention of variance in this entry: https://docs.python.org/3/library/typing.html?highlight=typing#typing.Calla… Also, no mention of the fact that `collections.abc.Callable` is generic here: https://docs.python.org/3/library/collections.abc.html#collections.abc.Call… PEP 483—The Theory of Type Hints—is the only official Python doc where I found the information about the variance of the formal type parameters of `Callable`. The explanation there is brilliant [0]. [0] https://peps.python.org/pep-0483/#covariance-and-contravariance Regardless, the intended audience of PEPs is "core developers"—which is neither a superset nor a subset of "Python devs now using type hints". We should not rely on PEPs to document features for Python users in general. _________________________________________ EXAMPLE 2: `Generator` variance could be better documented The entry for `typing.Generator` [1] has this heading: class typing.Generator(Iterator[T_co], Generic[T_co, T_contra, V_co]) Answer quickly: how many formal type parameters does `Generator` require? Which are covariant? Which are contravariant? [1] https://docs.python.org/3/library/typing.html?highlight=typing#typing.Gener… Nowhere in that page [1] there's an explanation of the `*_co` and `*_contra` naming convention, much less their semantics. Fortunately, the text of the `typing.Generator` entry says: "A generator can be annotated by the generic type `Generator[YieldType, SendType, ReturnType]". Unfortunately, `typing.Generator` is deprecated and will be gone in 5 years or so... The same issue applies to all the other generic types: builtins (`dict`, `frozenset`), ABCs, etc. Their formal type parameters they accept as generics are either undocumented, or documented in parts of the `typing` module that are already deprecated. Thoughts? -- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Technical Principal at ThoughtWorks | Twitter: @ramalhoorg

6 7

Fwd: Re: Are "Batteries Included" still a Good Thing? [was: It's now time to deprecate the stdlib urllib module]
by Sasha Kacanski March 30, 2022

March 30, 2022

---------- Forwarded message --------- From: Sasha Kacanski <skacanski(a)gmail.com> Date: Wed, Mar 30, 2022, 2:55 PM Subject: Re: [Python-Dev] Re: Are "Batteries Included" still a Good Thing? [was: It's now time to deprecate the stdlib urllib module] To: Paul Moore <p.f.moore(a)gmail.com> Cc: Christopher Barker <pythonchb(a)gmail.com>, Ethan Furman < ethan(a)stoneleaf.us>, Python Dev <python-dev(a)python.org> Agree with Paul, Best of python always was, you can do almost everything with standard libs, but then novice becomes better developer and starts exploring libs and tools that wrap stuff or perform better or has functionality that stdlib does not support right of the bet they start using variety of libs that suits their needs. Regards, Sasha On Sun, Mar 27, 2022, 8:06 PM Paul Moore <p.f.moore(a)gmail.com> wrote: > On Sun, 27 Mar 2022 at 17:11, Christopher Barker <pythonchb(a)gmail.com> > wrote: > > > > With the json package included, all they need to do is `import json`. If > that wasn't there, they's look in PyPi for a JSON implementation, and find > an absolutely astonishing number of options. I just did a search for "JSON" > > on PYPI, and it's HUGE -- most of them are for specialized JSON-using > protocols of some sort. I was actually really surprised that couple I know > about of the top of my head (ujson, orjson) are actually hard to find. > > > > "You can just pip install it" is really not a good solution. > > > > In fact, this is an example, I think, of where we should put some effort > into making the included batteries better -- it's great to have a JSON lib > built in, but it's too bad that it's not best-of-bread by pretty much any > definition (speed, memory performance, flexibility) -- there are quite a > few feature requests open for it -- it would be nice to actually implement > some of those. (but yes, that's a lot of work that someone(s) would have to > do) > > > > Back to the topic at hand, rather than remove urllib, maybe it could be > made better -- an as-easy-to-use-as-requests package in the stdlib would be > really great. > > I think that's where the mistake happens, though. Someone who needs > "best of breed" is motivated (and likely knowledgeable enough) to make > informed decisions about what's on PyPI. But someone who just wants to > get the job done probably doesn't - and that's the audience for the > stdlib. A stdlib module needs to be a good, reliable set of basic > functionality that non-experts can use successfully. There can be > better libraries on PyPI, but that doesn't mean the stdlib module is > unnecessary, nor does it mean that the stdlib has to match the PyPI > library feature for feature. > > So here, specifically, I'd rather see urlllib be the best urlllib it > can be, and not demand that it turn into requests. Requests is there > if people need/want it (as is httpx, and urllib3, and aiohttp). But > urllib is for people who want to get a file from the web, and *not* > have to deal with dependencies, 3rd party libraries, etc. > > The "batteries included" standard library and PyPI complement each > other. Neither is redundant, and neither implies the other is > unnecessary. > > Paul > _______________________________________________ > Python-Dev mailing list -- python-dev(a)python.org > To unsubscribe send an email to python-dev-leave(a)python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/E2FGXTY… > Code of Conduct: http://python.org/psf/codeofconduct/ >

1 0

Changing unittest verbose output.
by Itay Yeshaya March 29, 2022

March 29, 2022

When running unittest with the -v flag, if a test has errors, and has a docstring, the test name is shown on one line, and the docstring is shown on the next line with the `ERROR` word. For example: ./python.exe -m unittest -v Lib.test_verbose_error.randomTest test_one (Lib.test_verbose_error.randomTest) this is a doc string with helpful info ... ERROR ====================================================================== ERROR: test_one (Lib.test_verbose_error.randomTest) this is a doc string with helpful info ---------------------------------------------------------------------- Traceback (most recent call last): File "/workspaces/cpython/Lib/test_verbose_error.py", line 11, in test_one assert l[1] ~^^^ IndexError: list index out of range ---------------------------------------------------------------------- Ran 1 test in 0.004s This cause for some confusion and we would like to fix it, the possible options are: 1. print the name and docstring, and ERROR, all on one line. 2. provide a command-line switch to select that behavior. The first option has a minor backwards compatibility breakage, since the default output would change and those lines would get longer. Any thoughts on the mentioned solutions? https://bugs.python.org/issue47133

7 9

walrus operator and expression
by Ethan Furman March 28, 2022

March 28, 2022

In the following bit of code: while s := input.read(MAXBINSIZE): while len(s) < MAXBINSIZE and ns := input.read(MAXBINSIZE-len(s)): s += ns line = binascii.b2a_base64(s) output.write(line) I'm getting this error on the second line: cannot use assignment expressions with expression Can somebody explain why that isn't working? Many thanks. -- ~Ethan~

4 3

import * and __future__ imports
by Irit Katriel March 28, 2022

March 28, 2022

If you have a __future__ import in a script, and you import * from it in another script, the object for this future appears in the dir() of the other script, even though the __future__ import has no effect there. % cat x.py from __future__ import annotations % cat y.py from x import * print(dir()) class D: def f(self, a: D): return 42 % ./python.exe y.py ['__annotations__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 'annotations'] Traceback (most recent call last): File "/Users/iritkatriel/src/cpython-654/y.py", line 5, in <module> class D: ^^^^^^^^ File "/Users/iritkatriel/src/cpython-654/y.py", line 6, in D def f(self, a: D): ^ NameError: name 'D' is not defined I think we should change import * to exclude the __future__ import objects, and perhaps also to not show them in dir(x). Any objections? This came up in the discussion about https://bugs.python.org/issue26120 . See the attached PR for a technique we can use to identify those objects.

4 4

is __self__ an implementation detail
by Robin Becker March 28, 2022

March 28, 2022

A gentoo developer reported a symlink loop problem in reportlab's setup.py where we search for specific headers. The 'fixed' function looks like this def findFile(root, wanted, followlinks=True): visited = set() for p, D, F in os.walk(root,followlinks=followlinks): #scan directories to check for prior visits #use dev/inode to make unique key SD = [].append for d in D: dk = os.stat(pjoin(p,d)) dk = dk.st_dev, dk.st_ino if dk not in visited: visited.add(dk) SD(d) D[:] = SD.__self__ #set the dirs to be scanned for fn in F: if fn==wanted: return abspath(pjoin(p,fn)) the fix is reported to have worked, but the developer is querying the lifting of SD.append using the construct SD = [].append loop involving appends to the list ..... D[:] = SD.__self__ his objection was that using __self__ might be implementation sensitive. I cannot tell from the documentation (eg https://docs.python.org/3.10/reference/datamodel.html) if these magic methods are part of python or of the implementation. On the other hand as a longtime python programmer I wonder if such simple cases are already optimized out in the produced bytecode; for years I have been avoiding s += 'string' and only recently found out that it was handled as a special case. -- Robin Becker

2 1