Mailman 3 April 2022 - Python-Dev

Proposal to deprecate mailcap
by Brett Cannon April 26, 2022

April 26, 2022

A CVE has been opened against mailcap (see https://github.com/python/cpython/issues/68966 for details). I'm not aware of anyone trying to maintain the module and Victor did a search online and didn't find any use of the module in the top 5000 projects on PyPI (see the issue). The module is also under 300 lines of Python code that only ( https://github.com/python/cpython/blob/main/Lib/mailcap.py), so vendoring wouldn't be burdensome. As such, I'm proposing we deprecate mailcap in 3.11 and remove it in 3.13. Any explicit objections?

6 9

Proto-PEP part 2: Alternate implementation proposal for "forward class" using a proxy object
by Larry Hastings April 25, 2022

April 25, 2022

Here's one alternate idea for how to implement the "forward class" syntax. The entire point of the "forward class" statement is that it creates the real actual class object. But what if it wasn't actually the "real" class object? What if it was only a proxy for the real object? In this scenario, the syntax of "forward object" remains the same. You define the class's bases and metaclass. But all "forward class" does is create a simple, lightweight class proxy object. This object has a few built-in dunder values, __name__ etc. It also allows you to set attributes, so let's assume (for now) it calls metaclass.__prepare__ and uses the returned "dict-like object" as the class proxy object __dict__. "continue class" internally performs all the rest of the class-creation machinery. (Everything except __prepare__, as we already called that.) The first step is metaclass.__new__, which returns the real class object. "continue class" takes that object and calls a method on the class proxy object that says "here's your real class object". From that moment on, the proxy becomes a pass-through for the "real" class object, and nobody ever sees a reference to the "real" class object ever again. Every interaction with the class proxy object is passed through to the underlying class object. __getattribute__ calls on the proxy look up the attribute in the underlying class object. If the object returned is a bound method object, it rebinds that callable with the class proxy instead, so that the "self" passed in to methods is the proxy object. Both base_cls.__init_subclass__ and cls.__init__ see the proxy object during class creation. As far as Python user code is concerned, the class proxy *is* the class, in every way, important or not. The upside: this moves all class object creation code into "continue class" call. We don't have to replace __new__ with two new calls. The downside: a dinky overhead to every interaction with a "forward class" class object and with instances of a "forward class" class object. A huge concern: how does this interact with metaclasses implemented in C? If you make a method call on a proxy class object, and that calls a C function from the metaclass, we'd presumably have to pass in the "real class object", not the proxy class object. Which means references to the real class object could leak out somewhere, and now we have a real-class-object vs proxy-class-object identity crisis. Is this a real concern? A possible concern: what if metaclass.__new__ keeps a reference to the object it created? Now we have two objects with an identity crisis. I don't know if people ever do that. Fingers crossed that they don't. Or maybe we add a new dunder method: @special_cased_staticmethod metaclass.__bind_proxy__(metaclass, proxy, cls) This tells the metaclass "bind cls to this proxy object", so metaclasses that care can update their database or whatever. The default implementation uses the appropriate mechanism, whatever it is. One additional probably-bad idea: in the case where it's just a normal "class" statement, and we're not binding it to a proxy, should we call this? metaclass.__bind_proxy__(metaclass, None, cls) The idea there being "if you register the class objects you create, do the registration in __bind_proxy__, it's always called, and you'll always know the canonical object in there". I'm guessing probably not, in which case we tell metaclasses that track the class objects we create "go ahead and track the object you return from __new__, but be prepared to update your tracking info in case we call __bind_proxy__ on you". A small but awfully complicated wrinkle here: what do we do if the metaclass implements __del__? Obviously, we have to call __del__ with the "real" class object, so it can be destroyed properly. But __del__ might resurrect that object, which means someone took a reference to it. One final note. Given that, in this scenario, all real class creation happens in "continue class", we could move the bases and metaclass declaration down to the "continue class" statement. The resulting syntax would look like: forward class X ... continue class X(base1, base2, metaclass=AmazingMeta, rocket="booster") Is that better? worse? doesn't matter? I don't have an intuition about it right now--I can see advantages to both sides, and no obvious deciding factor. Certainly this syntax prevents us from calling __prepare__ so early, so we'd have to use a real dict in the "forward class" proxy object until we reached continue, then copy the values from that dict into the "dict-like object", etc.

3 3

C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API
by Victor Stinner April 25, 2022

April 25, 2022

Hi, I proposed two PRs to move the private C API (Include/cpython/) of PEP 523 "Adding a frame evaluation API to CPython" to the internal C API (Include/internals/): * https://github.com/python/cpython/pull/32052 * https://github.com/python/cpython/pull/32054 API: * _PyFrameEvalFunction type * _PyInterpreterState_GetEvalFrameFunc() * _PyInterpreterState_SetEvalFrameFunc() * (undocumented) _PyEval_EvalFrameDefault() The private API to get/set the eval function *is* documented at: https://docs.python.org/dev/c-api/init.html#c._PyInterpreterState_GetEvalFr… I added the Get/Set functions so debuggers don't have to access directly to the PyInterpreterState structure which has been moved to the internal C API in Python 3.8. This API causes me multiple issues: * It's a private API and I'm trying to remove the private API from the public C API header files. * The _PyFrameEvalFunction type is not stable: it got a new "tstate" parameter in Python 3.9 and the type of the second parameter changed from PyFrameObject* to _PyInterpreterFrame* in Python 3.11. * These functions use the _PyInterpreterFrame type which is part of the internal C API. While Pyston didn't bring a JIT compiler to Python with PEP 523, debuggers were made faster by using this API. Debuggers like pydevd, debugpy and ptvsd use it. I propose to move theses API to the internal header files (Include/internals/) to clarify that it's not part of the public C API and that there is no backward compatibility warranty. The change is being discussed at: https://bugs.python.org/issue46850 -- PEP 523 API added more private functions for code objects: * _PyEval_RequestCodeExtraIndex() * _PyCode_GetExtra() * _PyCode_SetExtra() The _PyEval_RequestCodeExtraIndex() function seems to be used by the pydevd debugger. The two others seem to be unused in the wild. I'm not sure if these ones should be moved to the internal C API. They can be left unchanged, since they don't use a type only defined by the internal C API. Victor -- Night gathers, and now my watch begins. It shall not end until my death.

7 20

What is Modules/rotatingtree.c for?
by Patrick Reader April 24, 2022

April 24, 2022

I've just noticed Modules/rotatingtree.{h,c}, which don't seem to be used anywhere. Are they just dead code? If so, is there a reason they haven't been removed?

2 1

Proto-PEP part 3: Closing thoughts on "forward class", etc.
by Larry Hastings April 24, 2022

April 24, 2022

Just a quick note from me on the proto-PEP and the two proposed implementations. When I started exploring this approach, I didn't suspect it'd require such sweeping changes to be feasible. Specifically, I didn't think I was going to propose changing the fundamental mechanism used to create class objects. That's an enormous change, and it makes me uncomfortable; I suspect I won't be alone in having that reaction. The alternate implementation with proxy objects was borne of my reaction, but it's worrisome too. It's a hack--though whether it's a "big" hack or a "small" hack is debatable. Anyway, I'm specifically worried about the underlying class object escaping the proxy and becoming visible inside Python somehow. If that happened, we'd have two objects representing the same "type" at runtime, a situation that could quickly become confusing. Also, as I hopefully made clear in the "alternate implementation" approach using a class proxy object, I'm not 100% certain that the proxy will work in all cases. I ran out of time to investigate it more--I wanted to post this idea with some lead time before the 2022 Language Summit, so that folks had time to read and digest it and discuss it before the Summit. I have some implementation ideas--the "class proxy" class may need its own exotic metaclass. Ultimately I'm posting this proto-PEP to foster discussion. I'm confident that "forward class" / "continue class" could solve all our forward-reference and circular-reference problems; the questions we need to collectively answer are: * how should the implementation work, and * is the cost of the implementation worth it? Best wishes, //arry/

2 1

Re: Proto-PEP part 1: Forward declaration of classes
by Tobias HT April 23, 2022

April 23, 2022

This seems like it would solve a huge problem that I've often faced myself, but by introducing something that might later on cause problems I anticipate. Like some people suggested earlier, introducing new keyword in python is not a good solution, I feel it just adds to the already existing bulk and increases complexity. Also Like some else mentioned again, this solution doesn't seem pythonic. I don't know what they meant by that, but in hindsight, it does really not feel pythonic. On the other hand, there's something I've been seeing around. I don't know if it was introduced by Mypy or something, but its the use of declaration files. I think they are saved as pyi files. They just have the declaration of a python object, be it class or variable. What if we just found a way of reusing that instead?

2 1

Summary of Python tracker Issues
by Python tracker April 22, 2022

April 22, 2022

ACTIVITY SUMMARY (2022-04-15 - 2022-04-22) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7146 ( +0) closed 51841 ( +0) total 58987 ( +0) Open issues with patches: 2890 Most recent 15 issues with no replies (15) ========================================== #47258: Python 3.10 hang at exit in drop_gil() (due to resource warnin https://bugs.python.org/issue47258 #47256: re: limit the maximum capturing group to 1,073,741,823, reduce https://bugs.python.org/issue47256 #47253: LOAD_GLOBAL instruction with wrong source position https://bugs.python.org/issue47253 #47252: socket.makefile documentation is missing data regarding the 'b https://bugs.python.org/issue47252 #47251: Merge BINARY_SUBSCR_LIST_INT with BINARY_SUBSCR_LIST_TUPLE https://bugs.python.org/issue47251 #47244: email.utils.formataddr does not respect double spaces https://bugs.python.org/issue47244 #47242: Annoying white bar in IDLE (line 457 in sidebar.py) https://bugs.python.org/issue47242 #47241: [C API] Move the PyCodeObject structure to the internal C API https://bugs.python.org/issue47241 #47238: Python threading.Event().wait() depends on the system time https://bugs.python.org/issue47238 #47236: Document types.CodeType.replace() changes about co_exceptionta https://bugs.python.org/issue47236 #47228: Document that naïve datetime objects represent local time https://bugs.python.org/issue47228 #47222: subprocess.Popen() should allow capturing output and sending i https://bugs.python.org/issue47222 #47219: asyncio with two interpreter instances https://bugs.python.org/issue47219 #47218: adding name to lzmafile https://bugs.python.org/issue47218 #47217: adding name to BZ2File https://bugs.python.org/issue47217 Most recent 15 issues waiting for review (15) ============================================= #47256: re: limit the maximum capturing group to 1,073,741,823, reduce https://bugs.python.org/issue47256 #47255: Many broken :meth: roles in the docs https://bugs.python.org/issue47255 #47254: enhanced dir? https://bugs.python.org/issue47254 #47251: Merge BINARY_SUBSCR_LIST_INT with BINARY_SUBSCR_LIST_TUPLE https://bugs.python.org/issue47251 #47243: Duplicate entry in 'Objects/unicodetype_db.h' https://bugs.python.org/issue47243 #47233: show_caches option affects code positions reported by dis.get_ https://bugs.python.org/issue47233 #47222: subprocess.Popen() should allow capturing output and sending i https://bugs.python.org/issue47222 #47218: adding name to lzmafile https://bugs.python.org/issue47218 #47217: adding name to BZ2File https://bugs.python.org/issue47217 #47216: adding mtime option to gzip open() https://bugs.python.org/issue47216 #47215: Add "unstable" frame stack api https://bugs.python.org/issue47215 #47208: Support libffi implementations that cannot support invocations https://bugs.python.org/issue47208 #47205: posix.sched_{get|set}affinity(-1) no longer returns ProcessLoo https://bugs.python.org/issue47205 #47200: Add ZipInfo.mode property https://bugs.python.org/issue47200 #47199: multiprocessing: micro-optimize Connection.send_bytes() method https://bugs.python.org/issue47199

1 0

Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API
by Jason Ansel April 20, 2022

April 20, 2022

The PyTorch team plans to use PEP 523 as a part of PyTorch 2.0, so this proposal may break the next major release of PyTorch. The related project is TorchDynamo, which can be found here: https://github.com/facebookresearch/torchdynamo We will likely move this into the core of PyTorch closer to release. If the changed happens, would PyTorch still be able to use the eval frame API? Or would it prevent from being used entirely?

11 25

Slowly bend the C API towards the limited API to get a stable ABI for everyone
by Victor Stinner April 20, 2022

April 20, 2022

Hi, There is a reason why I'm bothering C extensions maintainers and Python core developers with my incompatible C API changes since Python 3.8. Let me share my plan with you :-) In 2009 (Python 3.2), Martin v. Löwis did an amazing job with the PEP 384 "Defining a Stable ABI" to provide a "limited C API" and a "stable ABI" for C extensions: build an extension once, use it on multiple Python versions. Some projects like PyQt5 and cryptograpy use it, but it is just a drop in the PyPI ocean (353,084 projects). I'm trying to bend the "default" C API towards this "limited C API" to make it possible tomorrow to build *more* C extensions for the stable ABI. My goal is that the stable ABI would be the default, and only a minority of C extensions would opt-out because they need to access to more functions for best performance. The basic problem is that at the ABI level, C extensions must only call functions, rather than getting and setting directly to structure members. Structures changes frequently in Python (look at changes between Python 3.2 and Python 3.11), and any minor structure change breaks the ABI. The limited C API hides structures and only use function calls to solve this problem. Since 2020, I'm modifying the C API, one function by one, to slowly hide implementations (prepare the API to make strutures opaque). I focused on the following structures: * PyObject and PyVarObject (bpo-39573) * PyTypeObject (bpo-40170) * PyFrameObject (bpo-40421) * PyThreadState (bpo-39947) The majority of C extensions use functions and macros, they don't access directly structure members. There are a few members which are sometimes accessed directly which prevents making these structures opaque. For example, some old C extensions use obj->ob_type rather than Py_TYPE(obj). Fixing the minority of C extensions should benefit to the majority which may become compatible with the stable ABI. I am also converting macros to static inline functions to fix their API: define parameter types, result type and avoid surprising macros side effects ("macro pitfalls"). I wrote the PEP 670 "Convert macros to functions in the Python C API" for these changes. I wrote the upgrade_pythoncapi.py tool in my pythoncapi_project (*) which modify C code to use Py_TYPE(), Py_SIZE() and Py_REFCNT() rather than accessing directly PyObject and PyVarObject members. (*) https://github.com/pythoncapi/pythoncapi_compat In this tool, I also added "Borrow" variant of functions like PyFrame_GetCode() which returns a strong reference, to replace frame->f_code with _PyFrame_GetCodeBorrow(). In Python 3.11, you cannot use the frame->f_code member anymore, since it has been removed! You must call PyFrame_GetCode() (or pythoncapi_compat _PyFrame_GetCodeBorrow() variant). There are also a few macros which can be used as l-values like Py_TYPE(): "Py_TYPE(type1) = type2" must now be written "Py_SET_TYPE(type1, type2)" to avoid setting directly the tp_type type at the ABI level. I proposed the PEP 674 "Disallow using Py_TYPE() and Py_SIZE() macros as l-values" to solve these issues. Currently, many "functions" are still implemented as macros or static inline functions, so C extensions still access structure members at the ABI level for best Python performance. Converting these to regular functions has an impact on performance and I would prefer to first write a PEP giving the rationale for that. Today, it is not possible yet to build numpy for the stable ABI. The gap is just too large for this big C extension. But step by step, the C API becomes closer to the limited API, and more and more code is ready to be built for the stable ABI. Well, these C API changes have other advantages, like preparing Python for further optimizations, ease Python maintenance, clarify the seperation between the limited C API and the default C API, etc. ;-) Victor -- Night gathers, and now my watch begins. It shall not end until my death.

12 38

Updating inspect APIs
by Pablo Galindo Salgado April 20, 2022

April 20, 2022

Hi, We are currently debating in gh-88116 ( https://github.com/python/cpython/issues/88116) what's the best way forward to update the APIs in the inspect module to include the new position information. These APIs are inspect.getframeinfo, inspect.getouterframes, inspect.getinnerframes, inspect.stack and inspect.trace. The problem is that these APIs return a named tuple that now needs to include several new attributes (or one 4 tuple for the positions). Being named tuples, if we add a new attribute, existing unpackings of the tuple will now fail because there are more elements or the elements are in different positions. Also, it will be quite weird to add the new attributes at the end but leave the line number at the beginning. What's the best way to proceed here? The suggested way is to create a namedtuple subclass that adds the extra attributes but doesn't allow indexed access to it (there is a precedent to this in how we handled updating os.stat_result). I personally find this quite confusing but it certainly works. There may be other options. What do you think? Cheers from sunny London, Pablo Galindo Salgado

7 9