Mailman 3 May 2022 - Python-Dev

Re: A memory map based data persistence and startup speedup approach
by 严懿宸(文极) May 11, 2022

May 11, 2022

Hi Nick, Sorry for the late reply, and thanks for the feedback! We’ve been working on publishing the package, and the first version is available at https://github.com/alibaba/code-data-share-for-python/, with user guide and some statistics (TL;DR: ~15% speedup in startup). We welcome code review, comments or questions. > I assume the files wouldn't be portable across architectures That’s true, this file is basically a snapshot of part of the CPython heap that could be shared between processes. > so does the cache file naming scheme take that into account? Currently no, this file is intended to be generated on demand (rather than generating a huge archive from all the third-party packages installed). Thus the file itself and the name should be managed by user. > (The idea is interesting regardless of whether it produces arch-specific files - kind of a middle ground between portable serialisation based pycs and fully frozen modules) I think our package could be the substitution of the frozen module mechanism for third-party packages — while builtin modules can be compiled to C code, code-data-share could automatically create a similar file that requires no compilation / deserialization. Actually we do have a POC which is integrated with CPython and can speedup importing builtin modules, but after make it third-party package, there’s not much we can do to the builtins, so freeze and deep-freeze is quite exciting to us. Best, Yichen > On Mar 20, 2022, at 23:26, Nick Coghlan <ncoghlan(a)gmail.com> wrote: > > (belated follow-up as I noticed there hadn't been a reply on list yet, just the previous feedback on the faster-cpython ticket) > > On Mon, 21 Feb 2022, 6:53 pm Yichen Yan via Python-Dev, <python-dev(a)python.org> wrote: >> >> Hi folks, as illustrated in faster-cpython#150 [1], we have implemented a mechanism that supports data persistence of a subset of python date types with mmap, therefore can reduce package import time by caching code object. This could be seen as a more eager pyc format, as they are for the same purpose, but our approach try to avoid [de]serialization. Therefore, we get a speedup in overall python startup by ~15%. > > > This certainly sounds interesting! > >> >> Currently, we’ve made it a third-party library and have been working on open-sourcing. >> >> Our implementation (whose non-official name is “pycds”) mainly contains two parts: >> importlib hooks, this implements the mechanism to dump code objects to an archive and a `Finder` that supports loading code object from mapped memory. >> Dumping and loading (subset of) python types with mmap. In this part, we deal with 1) ASLR by patching `ob_type` fields; 2) hash seed randomization by supporting only basic types who don’t have hash-based layout (i.e. dict is not supported); 3) interned string by re-interning strings while loading mmap archive and so on. > > I assume the files wouldn't be portable across architectures, so does the cache file naming scheme take that into account? > > (The idea is interesting regardless of whether it produces arch-specific files - kind of a middle ground between portable serialisation based pycs and fully frozen modules) > > Cheers, > Nick. >

1 0

Migration plan for the distutils removal in Python 3.12
by Victor Stinner May 10, 2022

May 10, 2022

Hi, Two years ago, PEP 632 "Deprecate distutils module" was accepted: the distutils package was deprecated in Python 3.10 and scheduled for removal in Python 3.12. Questions. * Is the Python ecosystem ready for the distutils removal? How many projects are expected to be broken by this removal? * Is setuptools a drop-in replacement of distutils for most cases? Are there tools and documentation explaining how to replace distutils with setuptools? * Is there a tool to migrate from setup.py (distutils) to pyproject.toml (setuptools)? The dephell project can convert a setup.py script to pyproject.toml using poetry as the build system. * Can we simply suggest installing setuptools to users who still use "import distutils"? setuptools now provides the a "distutils" package. * Should we keep distutils in Python stdlib a few more Python releases if the removal causes too many practical issues? A code search in top 5000 PyPI projects (at 2022-01-26) using the regex '(import|from) distutils' matchs 5,567 lines in 1,229 projects. Some statistics: * 851 projects (1,372 lines) import distutils in setup.py * 233 projects (700 lines) use distutils.version * 92 projects (205 lines) use distutils.util * 1,018 lines are type annotations (.pyi files) I failed to group the remaining 2,272 lines. Converting 851 projects from setup.py with distutils to pyproject.toml is going to take a few months if not years. Python 3.12 will likely be released in October 2023 (in year and a half). Since setuptools 60 (December 2021), setuptools installs a distutils-precedence.pth file to override the stdlib module with its local distutils copy by default. The SETUPTOOLS_USE_DISTUTILS=stdlib environment variable can be used to explicitly request the stdlib flavor. If I understood correct, distutils is now maintained in setuptools, rather than in the Python stdlib. Python "make install" installs an up to date setuptools and so indirectly its distutils-precedence.pth file which makes the distutils local copy of setuptools available. But many Linux distributions splits setuptools and pip from the "python" package. Right now, Python still uses distutils internally for multiple use cases. I propose to start with renaming the distutils package to _distutils in the stdlib: * https://github.com/python/cpython/issues/92584 * https://github.com/python/cpython/pull/92585 Right now, the change is blocked by pip which still imports distutils (even if it prefers sysconfig in practice on Python 3.10 and newer). I reported the issue to pip, it should be easy to fix. Once Python will no longer depend on _distutils, it will be possible to fully remove it. By the way, Fedora 35 now longer provides automatically setuptools when building a Python package. A dependency to setuptools must now be explicit, since they are projects which don't use setuptools nor distutils, but other build systems. https://fedoraproject.org/wiki/Changes/Reduce_dependencies_on_python3-setup… Victor -- Night gathers, and now my watch begins. It shall not end until my death.

9 16

Python grammar test cases
by Venkat Ramakrishnan May 10, 2022

May 10, 2022

Greetings! I'm wondering if there's a repository of test cases that test the Python grammar. Any help would be appreciated. Thanks & Best Regards, Venkat.

4 7

Test Python 3.11 beta1 with PYTHONSAFEPATH=1
by Victor Stinner May 9, 2022

May 9, 2022

Hi, I added the -P command line option and the PYTHONSAFEPATH=1 environment variable to Python 3.11 beta 1 to not prepend an "unsafe path" to sys.path such as the script's directory or the current directory: https://docs.python.org/dev/using/cmdline.html#cmdoption-P Example: ---------- $ echo 'Nope!' >random.py # broken module $ echo 'import random; print(random.randint(1, 6))' >dice.py $ python3.11 dice.py # default behavior (...) File "/home/vstinner/random.py", line 1 (...) SyntaxError: invalid syntax $ python3.11 -P dice.py # ignore local random.py 4 ---------- Please test Python 3.11 beta1 with the PYTHONSAFEPATH=1 environment variable set, or at least run python with the -P option. I'm curious which use cases are not affected and which use cases are affected. The PYTHONSAFEPATH=1 environment variable is inherited and so affect child processes. It can break applications relying on Python 3.10 sys.path behavior. I proposed adding -p option which does the opposite of the -P option: opt-in for Python 3.10 "unsafe" sys.path behavior, ignore the PYTHONSAFEPATH environment variable: https://github.com/python/cpython/pull/92361 If PYTHONSAFEPATH=1 only breaks a minority of use cases, maybe we can consider to make it the default, as Perl did in Perl 5.26 (2017) to increase its security: https://perldoc.perl.org/perl5260delta#Removal-of-the-current-directory-(%2… Perl has an environment variable to get the old (Perl 5.24) behavior: PERL_USE_UNSAFE_INC=1. If enough people consider that it would be a good idea to change the default, I can maybe write a full PEP (I already have some notes). Even if the PEP is rejected, it might be a good thing to write down everything about this topic since it's a common issue hit by users learning Python and a common question of people auditing the Python security. I was asked a few months ago about changing the default to increase Python security. Victor -- Night gathers, and now my watch begins. It shall not end until my death.

2 1

[RELEASE] The first Python 3.11 beta (3.11.0b1) is available - Feature freeze is here
by Pablo Galindo Salgado May 8, 2022

May 8, 2022

We did it, team!! After quite a bumpy release process and a bunch of last-time fixes, we have reached **beta 1** and **feature freeze**. What a ride eh? You can get the shiny new release artefacts from here: https://www.python.org/downloads/release/python-3110b1/ ## This is a beta preview of Python 3.11 Python 3.11 is still in development. 3.11.0b1 is the first of four planned beta release previews. Beta release previews are intended to give the wider community the opportunity to test new features and bug fixes and to prepare their projects to support the new feature release. We **strongly encourage** maintainers of third-party Python projects to **test with 3.11** during the beta phase and report issues found to [the Python bug tracker](https://bugs.python.org) as soon as possible. While the release is planned to be feature complete entering the beta phase, it is possible that features may be modified or, in rare cases, deleted up until the start of the release candidate phase (Monday, 2021-08-02). Our goal is to have no ABI changes after beta 4 and as few code changes as possible after 3.11.0rc1, the first release candidate. To achieve that, it will be **extremely important** to get as much exposure for 3.11 as possible during the beta phase. Please keep in mind that this is a preview release and its use is **not** recommended for production environments. # Major new features of the 3.11 series, compared to 3.10 Python 3.11 is still in development. This release, 3.11.0b1 is the **first** of four beta releases. Beta release previews are intended to give the wider community the opportunity to test new features and bug fixes and to prepare their projects to support the new feature release. Many new features for Python 3.11 are still being planned and written. Among the new major new features and changes so far: * [PEP 657](https://www.python.org/dev/peps/pep-0657/) -- Include Fine-Grained Error Locations in Tracebacks * [PEP 654](https://www.python.org/dev/peps/pep-0654/) -- Exception Groups and except* * [PEP 673](https://www.python.org/dev/peps/pep-0673/) -- Self Type * [PEP 646](https://www.python.org/dev/peps/pep-0646/)-- Variadic Generics * [PEP 680](https://www.python.org/dev/peps/pep-0680/)-- tomllib: Support for Parsing TOML in the Standard Library * [PEP 675](https://www.python.org/dev/peps/pep-0675/)-- Arbitrary Literal String Type * [PEP 655](https://www.python.org/dev/peps/pep-0655/)-- Marking individual TypedDict items as required or potentially-missing * [bpo-46752](https://bugs.python.org/issue46752)-- Introduce task groups to asyncio * The Faster Cpython Project <https://github.com/faster-cpython> is already yielding some exciting results. Python 3.11 is up to 10-60% faster than Python 3.10. On average, we measured a 1.22x speedup on the standard benchmark suite. See Faster CPython <https://docs.python.org/3.11/whatsnew/3.11.html#faster-cpython>for details. * Hey, **fellow core developer,** if a feature you find important is missing from this list, let me know. The next pre-release of Python 3.11 will be 3.11.0b2, currently scheduled for Monday, 2022-05-30. # More resources * [Online Documentation](https://docs.python.org/3.11/) * [PEP 664](https://www.python.org/dev/peps/pep-0664/), 3.11 Release Schedule * Report bugs at [https://bugs.python.org](https://bugs.python.org). * [Help fund Python and its community](/psf/donations/). # And now for something completely different The holographic principle is a tenet of string theories and a supposed property of quantum gravity that states that the description of a volume of space can be thought of as encoded on a lower-dimensional boundary to the region—such as a light-like boundary like a gravitational horizon. First proposed by Gerard 't Hooft, it was given a precise string-theory interpretation by Leonard Susskind, who combined his ideas with previous ones of 't Hooft and Charles Thorn.[ Leonard Susskind said, “The three-dimensional world of ordinary experience––the universe filled with galaxies, stars, planets, houses, boulders, and people––is a hologram, an image of reality cited on a distant two-dimensional (2D) surface." As pointed out by Raphael Bousso, Thorn observed in 1978 that string theory admits a lower-dimensional description in which gravity emerges from it in what would now be called a holographic way. The holographic principle was inspired by black hole thermodynamics, which conjectures that the maximal entropy in any region scales with the radius squared, and not cubed as might be expected. In the case of a black hole, the insight was that the informational content of all the objects that have fallen into the hole might be entirely contained in surface fluctuations of the event horizon. The holographic principle resolves the black hole information paradox within the framework of string theory. However, there exist classical solutions to the Einstein equations that allow values of the entropy larger than those allowed by an area law, hence in principle larger than those of a black hole. These are the so-called "Wheeler's bags of gold". The existence of such solutions conflicts with the holographic interpretation, and their effects in a quantum theory of gravity including the holographic principle are not fully understood yet. # We hope you enjoy the new releases! Thanks to all of the many volunteers who help make Python Development and these releases possible! Please consider supporting our efforts by volunteering yourself or through organization contributions to the Python Software Foundation. https://www.python.org/psf/ Regards from chilly London, Your friendly release team, Pablo Galindo Salgado

3 4

Re: [RELEASE] The first Python 3.11 beta (3.11.0b1) is available - Feature freeze is here
by Pablo Galindo Salgado May 8, 2022

May 8, 2022

Done in all places that I can edit! Thanks for the catch :) On Sun, 8 May 2022 at 09:14, Oleg Iarygin <oleg(a)arhadthedev.net> wrote: > - python-announce@, python-committers@, python-list@ > > > report issues found to [the Python bug tracker](https://bugs.python.org) > as soon as possible > > The template needs to be updated. >

1 0

Summary of Python tracker Issues
by Python tracker May 6, 2022

May 6, 2022

ACTIVITY SUMMARY (2022-04-29 - 2022-05-06) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7146 ( +0) closed 51841 ( +0) total 58987 ( +0) Open issues with patches: 2890 Most recent 15 issues with no replies (15) ========================================== #47258: Python 3.10 hang at exit in drop_gil() (due to resource warnin https://bugs.python.org/issue47258 #47256: re: limit the maximum capturing group to 1,073,741,823, reduce https://bugs.python.org/issue47256 #47253: LOAD_GLOBAL instruction with wrong source position https://bugs.python.org/issue47253 #47252: socket.makefile documentation is missing data regarding the 'b https://bugs.python.org/issue47252 #47251: Merge BINARY_SUBSCR_LIST_INT with BINARY_SUBSCR_LIST_TUPLE https://bugs.python.org/issue47251 #47244: email.utils.formataddr does not respect double spaces https://bugs.python.org/issue47244 #47242: Annoying white bar in IDLE (line 457 in sidebar.py) https://bugs.python.org/issue47242 #47241: [C API] Move the PyCodeObject structure to the internal C API https://bugs.python.org/issue47241 #47238: Python threading.Event().wait() depends on the system time https://bugs.python.org/issue47238 #47236: Document types.CodeType.replace() changes about co_exceptionta https://bugs.python.org/issue47236 #47228: Document that naïve datetime objects represent local time https://bugs.python.org/issue47228 #47222: subprocess.Popen() should allow capturing output and sending i https://bugs.python.org/issue47222 #47219: asyncio with two interpreter instances https://bugs.python.org/issue47219 #47218: adding name to lzmafile https://bugs.python.org/issue47218 #47217: adding name to BZ2File https://bugs.python.org/issue47217 Most recent 15 issues waiting for review (15) ============================================= #47256: re: limit the maximum capturing group to 1,073,741,823, reduce https://bugs.python.org/issue47256 #47255: Many broken :meth: roles in the docs https://bugs.python.org/issue47255 #47254: enhanced dir? https://bugs.python.org/issue47254 #47251: Merge BINARY_SUBSCR_LIST_INT with BINARY_SUBSCR_LIST_TUPLE https://bugs.python.org/issue47251 #47243: Duplicate entry in 'Objects/unicodetype_db.h' https://bugs.python.org/issue47243 #47233: show_caches option affects code positions reported by dis.get_ https://bugs.python.org/issue47233 #47222: subprocess.Popen() should allow capturing output and sending i https://bugs.python.org/issue47222 #47218: adding name to lzmafile https://bugs.python.org/issue47218 #47217: adding name to BZ2File https://bugs.python.org/issue47217 #47216: adding mtime option to gzip open() https://bugs.python.org/issue47216 #47215: Add "unstable" frame stack api https://bugs.python.org/issue47215 #47208: Support libffi implementations that cannot support invocations https://bugs.python.org/issue47208 #47205: posix.sched_{get|set}affinity(-1) no longer returns ProcessLoo https://bugs.python.org/issue47205 #47200: Add ZipInfo.mode property https://bugs.python.org/issue47200 #47199: multiprocessing: micro-optimize Connection.send_bytes() method https://bugs.python.org/issue47199

1 0

Release of Python 3.11 beta 1 is currently blocked
by Pablo Galindo Salgado May 6, 2022

May 6, 2022

Hi everyone, Today we need to start the release of Python 3.11 beta 1. Currently, we have the following blockers: * https://github.com/python/cpython/issues/91162 (was deferred blocker - but all deferred blockers are bumped to release blockers on beta1). * https://github.com/python/cpython/issues/91350 * We have 3 buildbots failing: https://buildbot.python.org/all/#/release_status (some of the failures are due to flaky tests but others look real). Please, if you are involved in the above issues check if they are resolved or if they are not released blockers (notice any semantic change, API change, public data structure change or grammar change must be done before beta 1 is released). I will look at the buildbots later today but if you have some time to investigate, that would help a lot! *I am blocking the main branch so only the release team can merge changes* until these problems are addressed and we can continue with the release. Please, add me as a reviewer to any PR that needs to be merged to address these issues or any other change that *absolutely needs to go into beta 1*. Thanks for your help! Regards from sunny London, Pablo Galindo Salgado

2 2

Add -P command line option to not add sys.path[0]
by Victor Stinner May 5, 2022

May 5, 2022

Hi, There are 4 main ways to run Python: (1) python -m module [...] (2) python script.py [...] (3) python -c code [...] (4) python [...] (1) and (2) insert the directory of the module/script at sys.path[0]. (3) and (4) insert an empty string at sys.path[0]. This behavior is convenient and is maybe part of Python usability success: importing a module in the current directory is as easy as "import other_module" (load other_module.py). But it's also a threat to security: an attacker can override a stdlib module by creating a Python script with the same name than a stdlib module, like os.py or shutil.py. People learning Python commonly create a file with the same name than a stdlib module (ex: random.py) and then are clueless in face of an ImportError exception. Changing the default behavior was discussed multiple times. No consensus was reached, maybe because most users like the current default behavior and are not affected by corner cases (see below). I propose adding a -P option to Python command line interface to "not add sys.path[0]": https://github.com/python/cpython/pull/31542 See the documentation in the PR for the exact behavior of this option. I prefer to add an environment variable, only pass the option explicitly on the command line. Since Python 3.4, there is already the -I ("isolated mode") option: https://docs.python.org/dev/using/cmdline.html#cmdoption-I The -I option has other effects like disabling user site directories, it option doesn't fit use cases of the -P option. One annoying issue of the Python default behavior is that running a script in /usr/bin/ as root can create or override .pyc files in the /usr directory, even in the /usr/bin/ directory. Example of this surprising and annoying issue: https://github.com/benjaminp/six/issues/359#issuecomment-996159668 The -P option can be used in #!/usr/bin/python shebang to avoid this issue. -- An alternative would be to change the default behavior to not add sys.path[0], and add an option to opt-in for Python 3.10 behavior. Here are my notes about it: https://github.com/vstinner/misc/blob/main/cpython/pep_path0.rst What do you think? Victor -- Night gathers, and now my watch begins. It shall not end until my death.

9 19

Can I ask a real dumb procedural question about GitHub email?
by Skip Montanaro May 5, 2022

May 5, 2022

I subscribe to the python/cpython stuff on GitHub. I find it basically impossible to follow because of the volume. I realize there are probably plenty of extra changes going in based on the recent language summit (and maybe some sprints at PyCon?) as well as the proximity to the beta 1 freeze. Still, does anyone actually try to follow everything that comes out of that firehose? I've received updates to about 125 new GMail conversations (more total messages than that) in the last 24 hours, and that's after using filters to delete Miss-Islington-type messages altogether. As far as I can quickly ascertain, all the messages are from real people, not bots. How (if at all) do people deal with this firehose of email? Am I the only person dumb enough to have tried? I used to scan for csv-module-related messages, but don't even try to do that now. My only real reason for continuing to subscribe is that it feeds into a process that updates a dictionary of "common" words used by my XKCD-936-derived password generator. Thx, Skip

5 4