Mailman 3 May 2022 - Python-Dev

[RELEASE] Python 3.9.13 is now available
by Łukasz Langa 17 May '22

17 May '22

This is the thirteenth maintenance release of Python 3.9. Get it here: Python 3.9.13 <https://www.python.org/downloads/release/python-3913/> According to the release calendar specified in PEP 596 <https://www.python.org/dev/peps/pep-0596/>, Python 3.9.13 is the final regular maintenance release. Starting now, the 3.9 branch will only accept security fixes and releases of those will be made in source-only form until October 2025. This is a milestone moment for me as it means that now both of my release series are security-only. My work as release manager enters its final stage. I’m not crying, you’re crying! 🥲 Compared to the 3.8 series, this last regular bugfix release is still pretty active at 166 commits since 3.9.12. In comparison, version 3.8.10, the final regular bugfix release of Python 3.8, included only 92 commits. However, it’s likely that it was 3.8 that was special here with the governance changes occupying core developers’ minds. For reference, version 3.7.8, the final regular bugfix release of Python 3.7, included 187 commits. In any case, 166 commits is quite a few changes, some of which being pretty important fixes. Take a look at the change log <https://docs.python.org/release/3.9.13/whatsnew/changelog.html> for details. <https://discuss.python.org/t/python-3-9-13-is-now-available/15815#major-new…>Major new features of the 3.9 series, compared to 3.8 Some of the new major new features and changes in Python 3.9 are: PEP 573 <https://www.python.org/dev/peps/pep-0573/>, Module State Access from C Extension Methods PEP 584 <https://www.python.org/dev/peps/pep-0584/>, Union Operators in dict PEP 585 <https://www.python.org/dev/peps/pep-0585/>, Type Hinting Generics In Standard Collections PEP 593 <https://www.python.org/dev/peps/pep-0593/>, Flexible function and variable annotations PEP 602 <https://www.python.org/dev/peps/pep-0602/>, Python adopts a stable annual release cadence PEP 614 <https://www.python.org/dev/peps/pep-0614/>, Relaxing Grammar Restrictions On Decorators PEP 615 <https://www.python.org/dev/peps/pep-0615/>, Support for the IANA Time Zone Database in the Standard Library PEP 616 <https://www.python.org/dev/peps/pep-0616/>, String methods to remove prefixes and suffixes PEP 617 <https://www.python.org/dev/peps/pep-0617/>, New PEG parser for CPython BPO 38379 <https://bugs.python.org/issue38379>, garbage collection does not block on resurrected objects; BPO 38692 <https://bugs.python.org/issue38692>, os.pidfd_open added that allows process management without races and signals; BPO 39926 <https://bugs.python.org/issue39926>, Unicode support updated to version 13.0.0; BPO 1635741 <https://bugs.python.org/issue1635741>, when Python is initialized multiple times in the same process, it does not leak memory anymore; A number of Python builtins (range, tuple, set, frozenset, list, dict) are now sped up using PEP 590 <https://www.python.org/dev/peps/pep-0590> vectorcall; A number of Python modules (_abc, audioop, _bz2, _codecs, _contextvars, _crypt, _functools, _json, _locale, operator, resource, time, _weakref) now use multiphase initialization as defined by PEP 489 <https://www.python.org/dev/peps/pep-0489/>; A number of standard library modules (audioop, ast, grp, _hashlib, pwd, _posixsubprocess, random, select, struct, termios, zlib) are now using the stable ABI defined by PEP 384 <https://www.python.org/dev/peps/pep-0384/>. You can find a more comprehensive list in this release’s “What’s New <https://docs.python.org/release/3.9.13/whatsnew/3.9.html>” document. <https://discuss.python.org/t/python-3-9-13-is-now-available/15815#we-hope-y…>We hope you enjoy Python 3.9! Thanks to all of the many volunteers who help make Python Development and these releases possible! Please consider supporting our efforts by volunteering yourself or through organization contributions to the Python Software Foundation. Your friendly release team, Ned Deily @nad <https://discuss.python.org/u/nad> Steve Dower @steve.dower <https://discuss.python.org/u/steve.dower> Łukasz Langa @ambv <https://discuss.python.org/u/ambv>

1 0

New public PyUnicodeBuilder C API
by Victor Stinner 17 May '22

17 May '22

Hi, I propose adding a new C API to "build an Unicode string". What do you think? Would it be efficient with any possible Unicode string storage and any Python implementation? PyPy has an UnicodeBuilder type in Python, but here I only propose C API. Later, if needed, it would be easy to add a Python API for it. PyPy has UnicodeBuilder to replace "str += str" pattern which is inefficient in PyPy: CPython has a micro-optimization (in ceval.c) to keep this pattern performance interesting. Adding a Python API was discussed in 2020, see the LWN article: https://lwn.net/Articles/816415/ Example without error handling, naive implementation which doesn't use known length of key and value strings (calling Preallocate may be more efficient): --------------------------- // Format "key=value" PyObject *format_with_builder(PyObject *key, PyObject *value) { assert(PyUnicode_Check(key)); assert(PyUnicode_Check(value)); // Allocated on the stack PyUnicodeBuilder builder; PyUnicodeBuilder_Init(&builder); // Overallocation is more efficient if the final length is unknown PyUnicodeBuilder_EnableOverallocation(&builder); PyUnicodeBuilder_WriteStr(&builder, key); PyUnicodeBuilder_WriteChar(&builder, '='); // Disable overallocation before the last write PyUnicodeBuilder_DisableOverallocation(&builder); PyUnicodeBuilder_WriteStr(&builder, value); PyUnicode *str = PyUnicodeBuilder_Finish(&builder); // ... use str ... return error: PyUnicodeBuilder_Clear(&builder); ... } --------------------------- Proposed API (11 functions, 1 type): --------------------------- typedef struct { ... } PyUnicodeBuilder; void PyUnicodeBuilder_Init(PyUnicodeBuilder *builder); int PyUnicodeBuilder_Preallocate(PyUnicodeBuilder *builder, Py_ssize_t length, uint32_t maxchar); void PyUnicodeBuilder_EnableOverallocation(PyUnicodeBuilder *builder); void PyUnicodeBuilder_DisableOverallocation(PyUnicodeBuilder *builder); int PyUnicodeBuilder_WriteChar(PyUnicodeBuilder *builder, uint32_t ch); int PyUnicodeBuilder_WriteStr(PyUnicodeBuilder *builder, PyObject *str); int PyUnicodeBuilder_WriteSubstr(PyUnicodeBuilder *builder, PyObject *str, Py_ssize_t start, Py_ssize_t end); int PyUnicodeBuilder_WriteASCIIStr(PyUnicodeBuilder *builder, const char *str, Py_ssize_t len); int PyUnicodeBuilder_WriteLatin1Str(PyUnicodeBuilder *builder, const char *str, Py_ssize_t len); PyObject* PyUnicodeBuilder_Finish(PyUnicodeBuilder *builder); void PyUnicodeBuilder_Clear(PyUnicodeBuilder *builder); --------------------------- The proposed API is based on the private _PyUnicodeWriter C API that I added in Python 3.3 to optimize PEP 393 implementation. PyUnicodeBuilder_WriteASCIIStr() is an optimization: in release mode, the function doesn't have to check if the string contains non-ASCII characters. In debug mode, it must fail. If you consider that this API is too likely to introduce bugs in release mode, it can be removed. PyUnicodeBuilder_Preallocate() maxchar can be zero, but for the current Python implementation (PEP 393 compact string), it's more efficient if maxchar matchs the expected Unicode storage: 127 for ASCII, 255 for Latin1, 0xffff for UCS2, or 0x10ffff for UCS4. The value doesn't have to the exact, for example, it can be any valiue in [128; 255] for Latin1. The problem is that computing maxchar (need to read characters, decode a byte strings from a codec, etc.) can be expensive and an PyUnicodeBuilder_Preallocate() implementation can ignore maxchar depending on the chosen Unicode string storage. PyUnicode_MAX_CHAR_VALUE(str) can be used to create maxchar, but this function is specific to the current CPython implementation. Maybe a second "preallocate" function without maxchar should be added (more convenient, but less efficient). I don't know. Rationale for adding a new public C API. Currently, the Python C API is designed to allocate an Unicode string on the heap memory with uninitialized characters, and then basically gives a direct access to these characters. Since Python 3.3, creating an Unicode string in a C extension became more complicated: the caller must know in advance what will be the optimal storage for characters: ASCII, Latin1 UCS1 [U+0000; U+00FF], BMP UCS-2 [U+0000; U+FFFF],or full Unicode character set UCS4 [U+0000; U+10FFFF]. When writing a codec decoder (like decoding UTF-8), the maximum code point is not known in advance and so the decoder may need to change the buffer format while decoding (start with UCS1, switch to UCS2, maybe switch a second time to UCS4). The current C API has multiple drawbacks: * It is designed to target the exact format "PEP 393 compact strings" ("kind + data"). * It is inefficient for PyPy which uses UTF-8 internally. So it would also be inefficient if Python is modified to also use UTF-8 internally. * It leaks too many implementation details. * It creates an uninitialized string which might be exposed by mistake to Python and so can lead to bugs or even crashes I propose adding a new API to "build a string" which would be efficient on CPython and PyPy. Later, it should help Python experimenting a different storage for Unicode strings (with different trade-offs, like UTF-8). Discussion about changing maybe the Unicode storage in Python tomorrow, especially issues caused by the C API which prevent that: * https://discuss.python.org/t/un-deprecate-pyunicode-ready-for-future-unicod… * https://github.com/python/cpython/pull/92705#issuecomment-1125869198 In 2016, I wrote an article about private _PyUnicodeWriter and _PyBytesWriter C API that I added to optimize Python: https://vstinner.github.io/pybyteswriter.html Victor -- Night gathers, and now my watch begins. It shall not end until my death.

3 8

Issue: 92359 - Python 3.10 IDLE 64-bit doesn't open any files names code (code.py, code.pyw) - found a partial fix but looking for input
by ward.davidj＠gmail.com 16 May '22

16 May '22

Hello, Link to the github issue page is here https://github.com/python/cpython/issues/92359 This bug has been lurking in waiting for roughly 7 years or possibly longer. The root issue is that the "Edit with IDLE" context menu executes idle with `python.exe -m idlelib` which puts the current working directory in sys.path but inside idlelib it has a `from code import ...` statement. A kind of ugly hack is sitting in my fork of cpython here https://github.com/devdave/cpython/blob/issue_92359/Lib/idlelib/__main__.py . All I did was put the Lib directory at the front of sys.path. Perhaps this is the best solution? I don't know. Would appreciate any ideas for an alternative fix (ex perhaps changing how "Edit with IDLE" works?) or like I said, perhaps my fix is the best option because of how simple it is? Thanks, DevDave

4 4

Problems building 3.7.13 on Windows
by Joseph L. Casale 16 May '22

16 May '22

I am trying to build a full release of 3.7.13 with the Wix bundle on Windows. I have installed Visual Studio 2015 and included its bundled Windows 10 SDK. The SdkBinPath property in .\PCbuild\pyproject.props on line 182 seems to incorrectly set the path to: C:\Program Files (x86)\Windows Kits\10\bin\10.0.14393.0\x86 This does not include any of the tools referenced later such as Makecat.exe. The right path would be set further down to (but is skipped): C:\Program Files (x86)\Windows Kits\10\bin\x86 I commented out line 182, and that allowed .\Tools\msibuild.bat to complete, but that appears to leave the bundle named as "python-3.7.13.7804-amd64.exe"\ and it did not include the msi files (the final size was 1269 Kb). I'd be grateful for any guidance. Thanks, Joseph Casale

1 1

Summary of Python tracker Issues
by Python tracker 15 May '22

15 May '22

ACTIVITY SUMMARY (2022-05-06 - 2022-05-13) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7146 ( +0) closed 51841 ( +0) total 58987 ( +0) Open issues with patches: 2890 Most recent 15 issues with no replies (15) ========================================== #47258: Python 3.10 hang at exit in drop_gil() (due to resource warnin https://bugs.python.org/issue47258 #47256: re: limit the maximum capturing group to 1,073,741,823, reduce https://bugs.python.org/issue47256 #47253: LOAD_GLOBAL instruction with wrong source position https://bugs.python.org/issue47253 #47252: socket.makefile documentation is missing data regarding the 'b https://bugs.python.org/issue47252 #47251: Merge BINARY_SUBSCR_LIST_INT with BINARY_SUBSCR_LIST_TUPLE https://bugs.python.org/issue47251 #47244: email.utils.formataddr does not respect double spaces https://bugs.python.org/issue47244 #47242: Annoying white bar in IDLE (line 457 in sidebar.py) https://bugs.python.org/issue47242 #47241: [C API] Move the PyCodeObject structure to the internal C API https://bugs.python.org/issue47241 #47238: Python threading.Event().wait() depends on the system time https://bugs.python.org/issue47238 #47236: Document types.CodeType.replace() changes about co_exceptionta https://bugs.python.org/issue47236 #47228: Document that naïve datetime objects represent local time https://bugs.python.org/issue47228 #47222: subprocess.Popen() should allow capturing output and sending i https://bugs.python.org/issue47222 #47219: asyncio with two interpreter instances https://bugs.python.org/issue47219 #47218: adding name to lzmafile https://bugs.python.org/issue47218 #47217: adding name to BZ2File https://bugs.python.org/issue47217 Most recent 15 issues waiting for review (15) ============================================= #47256: re: limit the maximum capturing group to 1,073,741,823, reduce https://bugs.python.org/issue47256 #47255: Many broken :meth: roles in the docs https://bugs.python.org/issue47255 #47254: enhanced dir? https://bugs.python.org/issue47254 #47251: Merge BINARY_SUBSCR_LIST_INT with BINARY_SUBSCR_LIST_TUPLE https://bugs.python.org/issue47251 #47243: Duplicate entry in 'Objects/unicodetype_db.h' https://bugs.python.org/issue47243 #47233: show_caches option affects code positions reported by dis.get_ https://bugs.python.org/issue47233 #47222: subprocess.Popen() should allow capturing output and sending i https://bugs.python.org/issue47222 #47218: adding name to lzmafile https://bugs.python.org/issue47218 #47217: adding name to BZ2File https://bugs.python.org/issue47217 #47216: adding mtime option to gzip open() https://bugs.python.org/issue47216 #47215: Add "unstable" frame stack api https://bugs.python.org/issue47215 #47208: Support libffi implementations that cannot support invocations https://bugs.python.org/issue47208 #47205: posix.sched_{get|set}affinity(-1) no longer returns ProcessLoo https://bugs.python.org/issue47205 #47200: Add ZipInfo.mode property https://bugs.python.org/issue47200 #47199: multiprocessing: micro-optimize Connection.send_bytes() method https://bugs.python.org/issue47199

7 6

Re: Summary of Python tracker Issues
by Harry Lees 15 May '22

15 May '22

> > My understanding was that it would be updated to get its information > from Github once the transition was complete. Is that not going to > happen now? > There is an open PR to fix the roundup summary script at: https://github.com/python/cpython/pull/91738 It will move the roundup-summary script into the Python repo and have it run as a Github action instead of a cron job. Harry

1 0

Starting a new thread
by Barney Stratford 13 May '22

13 May '22

Hello all. It occurs to me that creating threads in Python is more awkward than it needs to be. Every time I want to start a new thread, I end up writing the same thing over and over again: def target(*args, **kwds): ... t = threading.Thread(target = target, args = <something>, kwargs= <something>) t.start() This becomes especially repetitive when calling a target function that only makes sense when run in a new thread, such as a timer. In my own code, I’ve taken to decorating functions that always run in new threads. Then I can call the function using the usual function call syntax, and have the new thread returned to me. With the decorator, the code reads instead: @threaded def target(*args, **kwds): … t = target(*args, **kwds) This has a couple of advantages. I don’t have to import the threading module all over my code. I can use the neater syntax of function calls. The function’s definition makes it clear it’s returning a new thread since it’s decorated. It gets the plumbing out of the way so I can concentrate more on what my code does and less in how it does it. It feels like the right place for this decorator is the standard library, so I’ve created PR #91878 for it. @rhettinger suggests that this is a bit premature, and that we should discuss it here first. Thoughts? Cheers, Barney.

8 10

Announcing the Python Language Summit 2022 blog posts
by Mariatta 11 May '22

11 May '22

The blog posts about presentations and discussions from Python Language Summit 2022 are now up for your enjoyment. Main article: https://pyfound.blogspot.com/2022/05/the-2022-python-language-summit_016788… - Python without the GIL <https://pyfound.blogspot.com/2022/05/the-2022-python-language-summit-python…> , Sam Gross - Reaching a per-interpreter GIL <https://pyfound.blogspot.com/2022/05/the-2022-python-language-summit-per.ht…> , Eric Snow - The “Faster CPython” project: 3.12 and beyond <https://pyfound.blogspot.com/2022/05/the-2022-python-language-summit_2.html> , Mark Shannon - WebAssembly: Python in the browser and beyond <https://pyfound.blogspot.com/2022/05/the-2022-python-language-summit-python…> , Christian Heimes - F-strings in the grammar <https://pyfound.blogspot.com/2022/05/the-2022-python-language-summit-f.html>, Pablo Galindo Salgado - Cinder Async Optimisations <https://pyfound.blogspot.com/2022/05/the-2022-python-language-summit_60.html>, Itamar Ostricher - The issue and PR backlog <https://pyfound.blogspot.com/2022/05/the-2022-python-language-summit-dealin…>, Irit Katriel - The path forward for immortal objects <https://pyfound.blogspot.com/2022/05/the-2022-python-language-summit_11.html>, Eddie Elizondo and Eric Snow - Lightning talks <https://pyfound.blogspot.com/2022/05/the-2022-python-language-summit.html>, short presentations by Carl Meyer, Thomas Wouters, Kevin Modzelewski, and Larry Hastings Thanks Alex Waygood for the write ups! Also thank you to all presenters and attendees for great content and discussions. This is the fourth Language Summit that I co-chaired with Łukasz, and also my last one. I’m stepping down from this responsibility to free up my time to take on other responsibilities and challenges. I’m grateful to have been given this opportunity to serve the Python community. This has been a great privilege and valuable experience. Thank you Lukasz for working alongside me the past four years, and Senthil for helping out this year. Thank you!

2 1

Re: A memory map based data persistence and startup speedup approach
by 严懿宸(文极) 11 May '22

11 May '22

Hi Nick, Sorry for the late reply, and thanks for the feedback! We’ve been working on publishing the package, and the first version is available at https://github.com/alibaba/code-data-share-for-python/, with user guide and some statistics (TL;DR: ~15% speedup in startup). We welcome code review, comments or questions. > I assume the files wouldn't be portable across architectures That’s true, this file is basically a snapshot of part of the CPython heap that could be shared between processes. > so does the cache file naming scheme take that into account? Currently no, this file is intended to be generated on demand (rather than generating a huge archive from all the third-party packages installed). Thus the file itself and the name should be managed by user. > (The idea is interesting regardless of whether it produces arch-specific files - kind of a middle ground between portable serialisation based pycs and fully frozen modules) I think our package could be the substitution of the frozen module mechanism for third-party packages — while builtin modules can be compiled to C code, code-data-share could automatically create a similar file that requires no compilation / deserialization. Actually we do have a POC which is integrated with CPython and can speedup importing builtin modules, but after make it third-party package, there’s not much we can do to the builtins, so freeze and deep-freeze is quite exciting to us. Best, Yichen > On Mar 20, 2022, at 23:26, Nick Coghlan <ncoghlan(a)gmail.com> wrote: > > (belated follow-up as I noticed there hadn't been a reply on list yet, just the previous feedback on the faster-cpython ticket) > > On Mon, 21 Feb 2022, 6:53 pm Yichen Yan via Python-Dev, <python-dev(a)python.org> wrote: >> >> Hi folks, as illustrated in faster-cpython#150 [1], we have implemented a mechanism that supports data persistence of a subset of python date types with mmap, therefore can reduce package import time by caching code object. This could be seen as a more eager pyc format, as they are for the same purpose, but our approach try to avoid [de]serialization. Therefore, we get a speedup in overall python startup by ~15%. > > > This certainly sounds interesting! > >> >> Currently, we’ve made it a third-party library and have been working on open-sourcing. >> >> Our implementation (whose non-official name is “pycds”) mainly contains two parts: >> importlib hooks, this implements the mechanism to dump code objects to an archive and a `Finder` that supports loading code object from mapped memory. >> Dumping and loading (subset of) python types with mmap. In this part, we deal with 1) ASLR by patching `ob_type` fields; 2) hash seed randomization by supporting only basic types who don’t have hash-based layout (i.e. dict is not supported); 3) interned string by re-interning strings while loading mmap archive and so on. > > I assume the files wouldn't be portable across architectures, so does the cache file naming scheme take that into account? > > (The idea is interesting regardless of whether it produces arch-specific files - kind of a middle ground between portable serialisation based pycs and fully frozen modules) > > Cheers, > Nick. >

1 0

Migration plan for the distutils removal in Python 3.12
by Victor Stinner 10 May '22

10 May '22

Hi, Two years ago, PEP 632 "Deprecate distutils module" was accepted: the distutils package was deprecated in Python 3.10 and scheduled for removal in Python 3.12. Questions. * Is the Python ecosystem ready for the distutils removal? How many projects are expected to be broken by this removal? * Is setuptools a drop-in replacement of distutils for most cases? Are there tools and documentation explaining how to replace distutils with setuptools? * Is there a tool to migrate from setup.py (distutils) to pyproject.toml (setuptools)? The dephell project can convert a setup.py script to pyproject.toml using poetry as the build system. * Can we simply suggest installing setuptools to users who still use "import distutils"? setuptools now provides the a "distutils" package. * Should we keep distutils in Python stdlib a few more Python releases if the removal causes too many practical issues? A code search in top 5000 PyPI projects (at 2022-01-26) using the regex '(import|from) distutils' matchs 5,567 lines in 1,229 projects. Some statistics: * 851 projects (1,372 lines) import distutils in setup.py * 233 projects (700 lines) use distutils.version * 92 projects (205 lines) use distutils.util * 1,018 lines are type annotations (.pyi files) I failed to group the remaining 2,272 lines. Converting 851 projects from setup.py with distutils to pyproject.toml is going to take a few months if not years. Python 3.12 will likely be released in October 2023 (in year and a half). Since setuptools 60 (December 2021), setuptools installs a distutils-precedence.pth file to override the stdlib module with its local distutils copy by default. The SETUPTOOLS_USE_DISTUTILS=stdlib environment variable can be used to explicitly request the stdlib flavor. If I understood correct, distutils is now maintained in setuptools, rather than in the Python stdlib. Python "make install" installs an up to date setuptools and so indirectly its distutils-precedence.pth file which makes the distutils local copy of setuptools available. But many Linux distributions splits setuptools and pip from the "python" package. Right now, Python still uses distutils internally for multiple use cases. I propose to start with renaming the distutils package to _distutils in the stdlib: * https://github.com/python/cpython/issues/92584 * https://github.com/python/cpython/pull/92585 Right now, the change is blocked by pip which still imports distutils (even if it prefers sysconfig in practice on Python 3.10 and newer). I reported the issue to pip, it should be easy to fix. Once Python will no longer depend on _distutils, it will be possible to fully remove it. By the way, Fedora 35 now longer provides automatically setuptools when building a Python package. A dependency to setuptools must now be explicit, since they are projects which don't use setuptools nor distutils, but other build systems. https://fedoraproject.org/wiki/Changes/Reduce_dependencies_on_python3-setup… Victor -- Night gathers, and now my watch begins. It shall not end until my death.

9 16

Python-Dev May 2022