Mailman 3 April 2021 - Python-Dev

[Release manager team communication] 3.10 feature freeze is 2 weeks away
by Pablo Galindo Salgado 19 Apr '21

19 Apr '21

Hi everyone, This is a friendly reminder from the release manager team that Python 3.10 feature freeze is two weeks away (Monday, 3 May 2021). Please, be aware of the following: * No new features or improvements should be merged after feature freeze. From the devguide (https://devguide.python.org/devcycle/#beta) : > After a first beta release is published, no new features are accepted. Only bug fixes can now be committed. This is when core developers should > concentrate on the task of fixing regressions and other new issues filed by users who have downloaded the alpha and beta release * If your change involves some platform-specific behaviour or has a non-trivial amount of C code, *make sure you run the buildbotsin your Pull Request* by using the "test-with-buildbots" label ( https://discuss.python.org/t/now-you-can-test-a-pr-with-the-buildbots-befor… ). Alternatively you could check the buildbots post-merge in the buildbot server: https://buildbot.python.org/all/#/builders?tags=%2B3.x&tags=%2Bstable This is *very important* because if problems are detected at the time of the release, *the release management team may have to revertthe changes* and therefore those will not be included in Python 3.10. * The master branch will be renamed to main (check our previous communication about how this will be done and how you should proceed: https://mail.python.org/archives/list/python-dev@python.org/thread/QWW7KGBW… ) If you have any questions or concerns, please contact me as soon as possible. Regards from sunny London, Pablo Galindo Salgado

2 1

In support of PEP 649
by Samuel Colvin 19 Apr '21

19 Apr '21

I've read the recent discussions <https://mail.python.org/archives/list/python-dev@python.org/thread/QSASX6PZ…> regarding PEP 649 and PEP 563 with interest, Larry Hastings recently contacted me when prompted <https://mail.python.org/archives/list/python-dev@python.org/message/YKVYJML…> to do so in a related discussion. I maintain pydantic <https://pydantic-docs.helpmanual.io/> which uses type annotations to provide runtime data validation. I'm extremely worried that if PEP 649 gets rejected, pydantic will not be able to fully support python 3.10 and may even end up getting abandoned, at best it will be much slower and more brittle. Please, please put pragmatism over principle and accept PEP 649. As discussed in this issue <https://github.com/samuelcolvin/pydantic/issues/2678>, using `typing.get_type_hints()` is not a good replacement for type annotations that are accessible as python objects. I know this is not a popularity contest, but I think it's worth explaining how widely used pydantic is for those who don't use it or haven't heard of it: - it's downloaded ~7m times a month and is the most popular standalone python data validation library by stars - it's used (in production and open source projects) by most of the big tech companies you've heard of - it's widely used both in web development and data science - it is one of the foundations of FastAPI <https://fastapi.tiangolo.com/> which is by far the fastest growing python web framework and recently came third (behind django and flask) in web frameworks by usage in the python developer survey <https://www.jetbrains.com/lp/python-developers-survey-2020/> Pydantic is not the only one, there are numerous other libraries that use type annotations at runtime. In short, you may not like it, but runtime inspection of type annotations makes python better, and a massive and growing number of people rely on that. Rejecting PEP 649 would put that in peril, please accept it. Samuel

20 32

PEP 647 Accepted
by Barry Warsaw 18 Apr '21

18 Apr '21

The Python Steering Council reviewed PEP 647 -- User-Defined Type Guards, and is happy to accept the PEP for Python 3.10. Congratulations Eric! We have one concern about the semantics of the PEP however. In a sense, the PEP subverts the meaning of the return type defined in the signature of the type guard, to express an attribute of the type guard function. Meaning, type guard functions actually *do* return bools, but this is not reflected in the return type: "Using this new mechanism, the is_str_list function in the above example would be modified slightly. Its return type would be changed from bool to TypeGuard[List[str]]. This promises not merely that the return value is boolean, but that a true indicates the input to the function was of the specified type.” In fact, the promise that it returns a bool is de-facto knowledge you must have when you see “TypeGuard” in the return type. It is an implicit assumption. Generally this might not be a problem, however when a type guard function is used for multiple purposes (e.g. a type guard and a “regular” function), then the return type is misleading, since a TypeGuard object is *not* returned. It’s unclear what type checkers would do in this case. The SC debated alternatives, including the decorator syntax specifically mentioned in the Rejected Ideas. We also discussed making TypeGuard a “wrapping” type defining an __bool__() so that e.g. is_str_list() would be defined as such: def is_str_list(val: List[object]) -> TypeGuard[List[str]]: """Determines whether all objects in the list are strings""" return TypeGuard(all(isinstance(x, str) for x in val)) but this also isn’t quite accurate, and we were concerned that this might be highly inconvenient in practice. In a sense, the type guard-ness of the function is an attribute about the function, not about the parameters or return type, but there is no way to currently express that using Python or type checking syntax. I am not sure whether you considered and rejected this option, but if so, perhaps you could add some language to the Rejected Ideas about it. Ultimately we couldn’t come up with anything better, so we decided that the PEP as it stands solves the problem in a practical manner, and that this is for the most part a wart that users will just have to learn and internalize. Cheers, -Barry (on behalf of the Python Steering Council)

12 18

PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2
by Larry Hastings 17 Apr '21

17 Apr '21

Attached is my second draft of PEP 649. The PEP and the prototype have both seen a marked improvement since round 1 in January; PEP 649 now allows annotations to refer to any variable they could see under stock semantics: * Local variables in the current function scope or in enclosing function scopes become closures and use LOAD_DEFER. * Class variables in the current class scope are made available using a new mechanism, in which the class dict is attached to the bound annotation function, then loaded into f_locals when the annotation function is run. Thus permitting LOAD_NAME opcodes to function normally. I look forward to your comments, //arry/

14 58

PEP-0467: Minor API improvements for binary sequences
by Ethan Furman 17 Apr '21

17 Apr '21

Looking for final comments before submitting for SC approval. It would be nice to finally get this resolved. :) Full text follows. --------------------------------------------------------------------------- PEP: 467 Title: Minor API improvements for binary sequences Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan <ncoghlan(a)gmail.com>, Ethan Furman <ethan(a)stoneleaf.us> Status: Deferred Type: Standards Track Content-Type: text/x-rst Created: 30-Mar-2014 Python-Version: 3.9 Post-History: 2014-03-30 2014-08-15 2014-08-16 2016-06-07 2016-09-01 2021-04-13 Abstract ======== During the initial development of the Python 3 language specification, the core ``bytes`` type for arbitrary binary data started as the mutable type that is now referred to as ``bytearray``. Other aspects of operating in the binary domain in Python have also evolved over the course of the Python 3 series. This PEP proposes five small adjustments to the APIs of the ``bytes`` and ``bytearray`` types to make it easier to operate entirely in the binary domain: * Discourage passing single integer values to ``bytes`` and ``bytearray`` * Add ``bytes.fromsize`` and ``bytearray.fromsize`` alternative constructors * Add ``bytes.fromord`` and ``bytearray.fromord`` alternative constructors * Add ``bytes.getbyte`` and ``bytearray.getbyte`` byte retrieval methods * Add ``bytes.iterbytes`` and ``bytearray.iterbytes`` alternative iterators And one built-in:: * ``bchr`` PEP Deferral ============ This PEP has been deferred until Python 3.9 at the earliest, as the open questions aren't currently expected to be resolved in time for the Python 3.8 feature addition deadline in May 2019 (if you're keen to see these changes implemented and are willing to drive that resolution process, contact the PEP authors). Proposals ========= Discourage use of current "zero-initialised sequence" behaviour --------------------------------------------------------------- Currently, the ``bytes`` and ``bytearray`` constructors accept an integer argument and interpret it as meaning to create a zero-initialised sequence of the given size:: >>> bytes(3) b'\x00\x00\x00' >>> bytearray(3) bytearray(b'\x00\x00\x00') This PEP proposes to update the documentation to discourage making use of that input type dependent behaviour in Python 3.10, suggesting to use a new, more explicit, ``bytes.fromsize(n)`` or ``bytearray.fromsize(n)`` spelling instead (see next section). However, the current handling of numeric inputs in the default constructors would remain in place indefinitely to avoid introducing a compatibility break. No other changes are proposed to the existing constructors. Addition of explicit "count and byte initialised sequence" constructors ----------------------------------------------------------------------- To replace the now discouraged behaviour, this PEP proposes the addition of an explicit ``fromsize`` alternative constructor as a class method on both ``bytes`` and ``bytearray`` whose first argument is the count, and whose second argument is the fill byte to use (defaults to ``\x00``):: >>> bytes.fromsize(3) b'\x00\x00\x00' >>> bytearray.fromsize(3) bytearray(b'\x00\x00\x00') >>> bytes.fromsize(5, b'\x0a') b'\x0a\x0a\x0a\x0a\x0a' >>> bytearray.fromsize(5, b'\x0a') bytearray(b'\x0a\x0a\x0a\x0a\x0a') ``fromsize`` will behave just as the current constructors behave when passed a single integer, while allowing for non-zero fill values when needed. Similar to ``str.center``, ``str.ljust``, and ``str.rjust``, both parameters would be positional-only with no externally visible name. Addition of "bchr" function and explicit "single byte" constructors ------------------------------------------------------------------- As binary counterparts to the text ``chr`` function, this PEP proposes the addition of a ``bchr`` function and an explicit ``fromord`` alternative constructor as a class method on both ``bytes`` and ``bytearray``:: >>> bchr(ord("A")) b'A' >>> bchr(ord(b"A")) b'A' >>> bytes.fromord(65) b'A' >>> bytearray.fromord(65) bytearray(b'A') These methods will only accept integers in the range 0 to 255 (inclusive):: >>> bytes.fromord(512) Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: integer must be in range(0, 256) >>> bytes.fromord(1.0) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'float' object cannot be interpreted as an integer While this does create some duplication, there are valid reasons for it: * the ``bchr`` builtin is to recreate the ``ord``/``chr``/``unichr`` trio from Python 2 under a different naming scheme (however, see the Open Questions section below) * the class method is mainly for the ``bytearray.fromord`` case, with ``bytes.fromord`` added for consistency The documentation of the ``ord`` builtin will be updated to explicitly note that ``bchr`` is the primary inverse operation for binary data, while ``chr`` is the inverse operation for text data, and that ``bytes.fromord`` and ``bytearray.fromord`` also exist. Behaviourally, ``bytes.fromord(x)`` will be equivalent to the current ``bytes([x])`` (and similarly for ``bytearray``). The new spelling is expected to be easier to discover and easier to read (especially when used in conjunction with indexing operations on binary sequence types). As a separate method, the new spelling will also work better with higher order functions like ``map``. Addition of "getbyte" method to retrieve a single byte ------------------------------------------------------ This PEP proposes that ``bytes`` and ``bytearray`` gain the method ``getbyte`` which will always return ``bytes``:: >>> b'abc'.getbyte(0) b'a' If an index is asked for that doesn't exist, ``IndexError`` is raised:: >>> b'abc'.getbyte(9) Traceback (most recent call last): File "<stdin>", line 1, in <module> IndexError: index out of range Addition of optimised iterator methods that produce ``bytes`` objects --------------------------------------------------------------------- This PEP proposes that ``bytes`` and ``bytearray`` gain an optimised ``iterbytes`` method that produces length 1 ``bytes`` objects rather than integers:: for x in data.iterbytes(): # x is a length 1 ``bytes`` object, rather than an integer For example:: >>> tuple(b"ABC".iterbytes()) (b'A', b'B', b'C') Design discussion ================= Why not rely on sequence repetition to create zero-initialised sequences? ------------------------------------------------------------------------- Zero-initialised sequences can be created via sequence repetition:: >>> b'\x00' * 3 b'\x00\x00\x00' >>> bytearray(b'\x00') * 3 bytearray(b'\x00\x00\x00') However, this was also the case when the ``bytearray`` type was originally designed, and the decision was made to add explicit support for it in the type constructor. The immutable ``bytes`` type then inherited that feature when it was introduced in PEP 3137. This PEP isn't revisiting that original design decision, just changing the spelling as users sometimes find the current behaviour of the binary sequence constructors surprising. In particular, there's a reasonable case to be made that ``bytes(x)`` (where ``x`` is an integer) should behave like the ``bytes.fromord(x)`` proposal in this PEP. Providing both behaviours as separate class methods avoids that ambiguity. Why use positional-only parameters? ----------------------------------- This is for consistency with the other methods on the affected types, and to avoid having to devise sensible names for them. Open Issue: memoryview ====================== Updating ``memoryview`` with these new methods is outside the scope of this PEP. >>>>>>> 467: update deprecation and memoryview statements References ========== .. [1] Initial March 2014 discussion thread on python-ideas (https://mail.python.org/pipermail/python-ideas/2014-March/027295.html) .. [2] Guido's initial feedback in that thread (https://mail.python.org/pipermail/python-ideas/2014-March/027376.html) .. [3] Issue proposing moving zero-initialised sequences to a dedicated API (http://bugs.python.org/issue20895) .. [4] Issue proposing to use calloc() for zero-initialised binary sequences (http://bugs.python.org/issue21644) .. [5] August 2014 discussion thread on python-dev (https://mail.python.org/pipermail/python-ideas/2014-March/027295.html) .. [6] June 2016 discussion thread on python-dev (https://mail.python.org/pipermail/python-dev/2016-June/144875.html) Copyright ========= This document has been placed in the public domain.

3 3

f_lasti not correct with PREDICT (is it fixable?)
by Fabio Zadrozny 17 Apr '21

17 Apr '21

Hi all, I currently have a use-case where I need to rely on `f_lasti`, but it seems that it's not really possible to rely on it as it doesn't accurately point to the actual bytecode last executed when PREDICT is used. So, I'd like to know: is it possible to change that so that the `f_lasti` is indeed properly updated (even when PREDICT) is used (to actually match what the docs say about it: https://docs.python.org/3/library/inspect.html)? As a note, my use case is doing a step into a function in a debugger. This is needed in order to know whether a given function call maps the target being step into. For instance, given something as: def method(c): return c+1 a = method a(a(a(1))) The user can choose to step into the `a` at the middle. The only conceivable way out I found to implement it reliably was by identifying that when the user enters the `method()` function the f_lasti on the parent frame is the one which would result in that function call. So, I have this working on a POC, but to have it working on all the cases I'm having to handle all existing bytecode instructions and then going to ceval.c and then identify which ones have a PREDICT and then having to do the adjustments to match the `f_lasti` accordingly when the function call is reached (which is a bit of an herculean task to support multiple versions of CPython as that's definitely an implementation detail), so, I'd really be interested in having the f_lasti work reliably. Thanks, Fabio

1 0

Memory use of PEP 563 vs PEP 649
by Mark Shannon 17 Apr '21

17 Apr '21

Hi, There has been some discussion on this mailing list about the memory use of PEP 563 vs PEP 649. It doesn't matter. The memory use of either is small, and any measurements are merely measuring artifacts of the current implementations, and the current on-disk representation of code objects. In an ideal implementation, of either PEP, the underlying data for `__annotations__` will sit on disk at zero cost in memory and load time. There are much more important differences between the two PEPs. I support keeping the 3.9 behavior for 3.10 to give us time to resolve the discussion and maybe for us to implement some of the above "ideal implementation". Cheers, Mark.

1 0

Question regarding the value of PyThreadState.thread_id
by Gabriele 17 Apr '21

17 Apr '21

Hi all. My apologies if this is a topic that's been discussed already, but I wasn't able to locate anything in the archive on the subject. I was wondering if there's a fundamental reason for using PyThread_get_thread_ident instead of PyThread_get_thread_native_id for the value of the thread_id field of the PyThreadState object. The reason why I'm asking is that I would like to have easy access to the native TID on Linux to identify the /proc/stat file associated with the thread from an external process. At the moment I have to resort to calling process_vm_readv to copy the struct pthread over to local VM and then guess where the tid field might be. So, if there's no fundamental reason for thread_id to be PyThread_get_thread_ident, I would like to propose to change it to PyThread_get_thread_native_id instead. Thanks, Gabriele -- "Egli è scritto in lingua matematica, e i caratteri son triangoli, cerchi, ed altre figure geometriche, senza i quali mezzi è impossibile a intenderne umanamente parola; senza questi è un aggirarsi vanamente per un oscuro laberinto." -- G. Galilei, Il saggiatore.

2 2

PEP 644 Accepted -- Require OpenSSL 1.1.1 or newer
by Pablo Galindo Salgado 17 Apr '21

17 Apr '21

Hi Christian, Thank you for submitting PEP 644 (Require OpenSSL 1.1.1). After evaluating the situation and discussing the PEP, the Steering Council is happy with the PEP, and hereby accepts it. The SC is of the opinion that this change will make it considerable easier to maintain the extension modules that depend on OpenSSL while allowing us to leverage the new APIs and improvements in the future. As indicated in the PEP, OpenSSL 1.1.1 is the default variant and version of OpenSSL on almost all supported platforms and distributions almost all known Major CI providers provide images with OpenSSL 1.1.1, so we believe this has the right balance to improve the situation without causing major impact or breakage. Nevertheless, the Steering Council would like to see this change reflected properly in the documentation, What's New (linking against the new instructions here: https://docs.python.org/3.10/using/unix.html#custom-openssl) and release manager notices so this is properly communicated to our users. The Steering Council also thanks Christian for his patience explaining different aspects of the PEP, including in the PEP some extra requested information and for considering the concerts raised. Congratulations, Christian! With thanks from the whole Python Steering Council, Pablo Galindo Salgado

2 1

Re: Typing syntax and ecosystem
by Hugh Fisher 16 Apr '21

16 Apr '21

> Message: 1 > Date: Sun, 11 Apr 2021 13:31:12 -0700 > From: Barry Warsaw <barry(a)python.org> > Subject: [Python-Dev] Re: PEP 647 Accepted > > This is something the SC has been musing about, but as it’s not a fully formed idea, I’m a little hesitant to bring it up. That said, it’s somewhat relevant: We wonder if it may be time to in a sense separate the typing syntax from Python’s regular syntax. TypeGuards are a case where if typing had more flexibility to adopt syntax that wasn’t strictly legal “normal” Python, maybe something more intuitive could have been proposed. I wonder if the typing-sig has discussed this possibility (in the future, of course)? [ munch ] > > Agreed. It’s interesting that PEP 593 proposes a different approach to enriching the typing system. Typing itself is becoming a little ecosystem of its own, and given that many Python users are still not fully embracing typing, maybe continuing to tie the typing syntax to Python syntax is starting to strain. I would really like to see either "Typed Python" become a different programming language, or progress to building type checking into the CPython implementation itself. (Python 4 seems to me the obvious release.) The current halfway approach is confusing and slightly ridiculous. The first, a separate programming language, would be like RATFOR and CFront in the past and TypeScript today. Typed Python can have whatever syntax the designers want because it doesn't have to be compatible with Python, just as TypeScript is not constrained by JavaScript. A type checker translates the original Typed Python source into "dynamic" or "classic" Python for execution. (Maybe into .pyc instead of .py?) This would mean no overhead for type checking in CPython itself. No need to contort the parser into ignoring bits of code that are, in effect, syntax checked comments. And for the typing in Python enthusiasts, you won't have to listen to people like me complaining. The second approach is to assume that type checking in Python is useful and popular. Not with me, but I'm willing to accept that I'm in the minority and can be ignored - after all, I can still write my Python code without type annotations. If so running a type checker as a separate step, as we do at the moment, is like asking C programmers to run the preprocessor by hand. In today's world of continuous build and integration, it seems silly to me to have a type checker read the source, scan into lexical tokens, build an abstract syntax tree, perform semantic analysis with type checking, and then throw it away before running an interpreter which reads the same source, scans into lexical tokens, builds an abstract syntax tree, and executes. On the purely pragmatic level there is an extra chance for mismatches and things to go wrong; and from an environmental viewpoint it isn't a great use of resources. -- cheers, Hugh Fisher

13 19