Mailman 3 September 2021 - Python-Dev

Re: PEP 467 feedback from the Steering Council
by Barry Warsaw Sept. 14, 2021

Sept. 14, 2021

There is some discussion going on in bpo-45155 about what the default value of the `byteorder` argument should be for int.to_bytes() and int.from_bytes(): https://bugs.python.org/issue45155 I’ve created a short Discourse poll if you’d like to weigh in. https://discuss.python.org/t/what-should-be-the-default-value-for-int-to-by… Cheers, -Barry

2 1

PEP 663: Improving and Standardizing Enum str(), repr(), and format() behaviors
by Ethan Furman Sept. 13, 2021

Sept. 13, 2021

PEP 663 is presented below for your viewing pleasure. Comments, questions, and concerns are welcome. -------------------------------------------------------------------------------- PEP: 663 Title: Improving and Standardizing Enum str(), repr(), and format() behaviors Version: $Revision$ Last-Modified: $Date$ Author: Ethan Furman <ethan(a)stoneleaf.us> Discussions-To: python-dev(a)python.org Status: Draft Type: Informational Content-Type: text/x-rst Created: 23-Feb-2013 Python-Version: … [View More]3.11 Post-History: 20-Jul-2021 10-Sep-2021 Resolution: Abstract ======== Now that we have a few years experience with Enum usage it is time to update the ``repr()``, ``str()``, and ``format()`` of the various enumerations by their intended purpose. Motivation ========== The addition of ``StrEnum`` with its requirement to have its ``str()`` be its ``value`` is inconsistent with other provided Enum's ``str``. Having the ``str()`` of ``IntEnum`` and ``IntFlag`` not be the value causes bugs and extra work when replacing existing constants. Having the ``str()`` and ``format()`` of an enum member be different can be confusing. The iteration of ``Flag`` members, which directly affects their ``repr()``, is inelegant at best, and buggy at worst. Rationale ========= Enums are becoming more common in the standard library; being able to recognize enum members by their ``repr()``, and having that ``repr()`` be easy to parse, is useful and can save time and effort in understanding and debugging code. However, the enums with mixed-in data types (``IntEnum``, ``IntFlag``, and the new ``StrEnum``) need to be more backwards compatible with the constants they are replacing -- specifically, ``str(replacement_enum_member) == str(original_constant)`` should be true (and the same for ``format()``). IntEnum, IntFlag, and StrEnum should be as close to a drop-in replacement of existing integer and string constants as is possible. Towards that goal, the str() output of each should be its inherent value; e.g. if ``Color`` is an ``IntEnum``:: >>> Color.RED <Color.RED: 1> >>> str(Color.RED) '1' >>> format(Color.RED) '1' Note that format() already produces the correct output in 3.10, only str() needs updating. As much as possible, the ``str()`, ``repr()``, and ``format()`` of enum members should be standardized across the stardard library. The repr() of Flag currently includes aliases, which it should not; fixing that will, of course, already change its ``repr()`` in certain cases. Specification ============= There a three broad categories of enum usage: - standard: Enum or Flag a new enum class is created, and the members are used as ``class.member_name`` - drop-in replacement: IntEnum, IntFlag, StrEnum a new enum class is created which also subclasses ``int`` or ``str`` and uses ``int.__str__`` or ``str.__str__``; the ``repr`` can be changed by using the ``global_enum`` decorator - user-mixed enums and flags the user creates their own integer-, float-, str-, whatever-enums instead of using enum.IntEnum, etc. Some sample enums:: # module: tools.py class Hue(Enum): # or IntEnum LIGHT = -1 NORMAL = 0 DARK = +1 class Color(Flag): # or IntFlag RED = 1 GREEN = 2 BLUE = 4 class Grey(int, Enum): # or (int, Flag) BLACK = 0 WHITE = 1 Using the above enumerations, the following table shows the old and new behavior, while the last shows the final result: +-------------+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | type | enum repr() | enum str() | enum format() | flag repr() | flag str() | flag format() | +-------------+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | standard | 3.10 | | | | <Color.RED|GREEN: 3> | Color.RED|GREEN | Color.RED|GREEN | | +----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | | new | | | | <Color(3): RED|GREEN> | Color.RED|Color.GREEN | Color.RED|Color.GREEN | +-------------+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ +-------------+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | user mixed | 3.10 | | | 1 | <Grey.WHITE: 1> | | 1 | | +----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | | new | | | Grey.WHITE | <Grey(1): WHITE> | | Grey.WHITE | +-------------+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ +-------------+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | int drop-in | 3.10 | | Hue.LIGHT | | <Color.RED|GREEN: 3> | Color.RED|GREEN | | | +----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | | new | | -1 | | <Color(3): RED|GREEN> | 3 | | +-------------+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ +-------------+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | global | 3.10 | <Hue.LIGHT: -1> | Hue.LIGHT | Hue.LIGHT | <Color.RED|GREEN: 3> | Color.RED|GREEN | Color.RED|GREEN | | +----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | | new | tools.LIGHT | LIGHT | LIGHT | tools.RED|tools.GREEN | RED|GREEN | RED|GREEN | +-------------+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ +-------------+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | user mixed | 3.10 | <Grey.WHITE: 1 | Grey.WHITE | Grey.WHITE | <Grey.WHITE: 1> | Grey.WHITE | 1 | | +----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | | new | tools.WHITE | WHITE | WHITE | tools.WHITE | WHITE | WHITE | +-------------+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ +-------------+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | int drop-in | 3.10 | <Hue.LIGHT: -1> | Hue.LIGHT | | <Color.RED|GREEN: 3> | Color.RED|GREEN | | | +----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | | new | tools.LIGHT | -1 | | tools.RED|tools.GREEN | 3 | | +-------------+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ Which will result in: +-------------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | type | enum repr() | enum str() | enum format() | flag repr() | flag str() | flag format() | +-------------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | standard | <Hue.LIGHT: -1> | Hue.LIGHT | Hue.LIGHT | <Color(3): RED|GREEN> | Color.RED|Color.GREEN | Color.RED|Color.GREEN | +-------------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | user mixed | <Grey.WHITE: 1> | Grey.WHITE | Grey.WHITE | <Grey(1): WHITE> | Grey.WHITE | Grey.WHITE | +-------------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | int drop-in | <Hue.LIGHT: -1> | -1 | -1 | <Color(3): RED|GREEN> | 3 | 3 | +-------------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | global | tools.LIGHT | LIGHT | LIGHT | tools.RED|tools.GREEN | RED|GREEN | RED|GREEN | +-------------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | user mixed | tools.WHITE | WHITE | WHITE | tools.WHITE | WHITE | WHITE | +-------------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | int drop-in | tools.LIGHT | -1 | -1 | tools.RED|tools.GREEN | 3 | 3 | +-------------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ As can be seen, ``repr()`` is primarily affected by whether the members are global, while ``str()`` is affected by being global or by being a drop-in replacement, with the drop-in replacement status having a higher priority. Also, the basic ``repr()`` and ``str()`` have changed for flags as the old style was very clunky. The ``repr()`` for Enum vs Flag are different, primarily because the Enum ``repr()`` does not work well for flags. I like being able to tell whether an enum member is a Flag or an Enum based on the ``repr()`` alone, but am open to arguments for changing Enum's ``repr()`` to match Flag's. Backwards Compatibility ======================= Backwards compatibility of stringified objects is not guarenteed across major Python versions, and there will be backwards compatibility breaks where software uses the ``repr()``, ``str()``, and ``format()`` output of enums in tests, documentation, data structures, and/or code generation. Normal usage of enum members will not change: ``re.ASCII`` can still be used as ``re.ASCII`` and will still compare equal to ``256``. If the previous output needs to be maintained, for example to ensure compatibily between different Python versions, software projects will need to create their own enum base class with the appropriate methods overridden. Note that by changing the ``str()`` of the drop-in category, we will actually prevent future breakage when ``IntEnum``, et al, are used to replace existing constants. Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. [View Less]

4 6

A better way to freeze modules
by Gregory Szorc Sept. 13, 2021

Sept. 13, 2021

Over in https://bugs.python.org/issue45020 there is some exciting work around expanding the use of the frozen importer to speed up Python interpreter startup. I wholeheartedly support the effort and don't want to discourage progress in this area. Simultaneously, I've been down this path before with PyOxidizer and feel like I have some insight to share. I don't think I'll be offending anyone by saying the existing CPython frozen importer is quite primitive in terms of functionality: it does … [View More]the minimum it needs to do to support importing module bytecode embedded in the interpreter binary [for purposes of bootstrapping the Python-based importlib modules]. The C struct representing frozen modules is literally just the module name and a pointer to a sized buffer containing bytecode. In issue45020 there is talk of enhancing the functionality of the frozen importer to support its potential broader use. For example, setting __file__ or exposing .__loader__.get_source(). I support the overall initiative. However, introducing enhanced functionality of the frozen importer will at the C level require either: a) backwards incompatible changes to the C API to support additional metadata on frozen modules (or at the very least a supplementary API that fragments what a "frozen" module is). b) CPython only hacks to support additional functionality for "freezing" the standard library for purposes of speeding up startup. I'm not a CPython core developer, but neither "a" nor "b" seem ideal to me. "a" is backwards incompatible. "b" seems like a stop-gap solution until a more generic version is available outside the CPython standard library. From my experience with PyOxidizer and software in general, here is what I think is going to happen: 1. CPython enhances the frozen importer to be usable in more situations. 2. Python programmers realize this solution has performance and ease-of-distribution wins and want to use it more. 3. Limitations in the frozen importer are found. Bugs are reported. Feature requests are made. 4. The frozen importer keeps getting incrementally extended or Python developers grow frustrated that its enhancements are only available to the standard library. You end up slowly reimplementing the importing mechanism in C (remember Python 2?) or disappoint users. Rather than extending the frozen importer, I would suggest considering an alternative solution that is far more useful to the long-term success of Python: I would consider building a fully-featured, generic importer that is capable of importing modules and resource data from a well-defined and portable serialization format / data structure that isn't defined by C structs and APIs. Instead of defining module bytecode (and possible additional minimal metadata) in C structs in a frozen modules array (or an equivalent C API), what if we instead defined a serialization format for representing the contents of loadable Python data (module source, module bytecode, resource files, extension module library data, etc)? We could then point the Python interpreter at instances of this data structure (in memory or in files) so it could import/load the resources within using a meta path importer. What if this serialization format were designed so that it was extremely efficient to parse and imports could be serviced with the same trivially minimal overhead that the frozen importer currently has? We could embed these data structures in produced binaries and achieve the same desirable results we'll be getting in issue45020 all while delivering a more generic solution. What if this serialization format were portable across machines? The entire Python ecosystem could leverage it as a container format for distributing Python resources. Rather than splatting dozens or hundreds of files on the filesystem, you could write a single file with all of a package's resources. Bugs around filesystem implementation details such as case (in)sensitivity and Unicode normalization go away. Package installs are quicker. Run-time performance is better due to faster imports. (OK, maybe that last point brings back bad memories of eggs and you instinctively reject the idea. Or you have concerns about development ergonomics when module source code isn't in standalone editable files. These are fair points!) What if the Python interpreter gains an "app mode" where it is capable of being paired with a single "resources file" and running the application within? Think running zip applications today, but a bit faster, more tailored to Python, and more fully featured. What if an efficient binary serialization format could be leveraged as a cache to speed up subsequent interpreter startups? These were all considerations on my mind in the early days of PyOxidizer when I realized that the frozen importer and zip importers were lacking the features I desired and I would need to find an alternative solution. One thing led to another and I have incrementally developed the "Python packed resources" data format ( https://pyoxidizer.readthedocs.io/en/stable/pyoxidizer_packed_resources.html). This is a binary format for representing Python source code, bytecode, resource files, extension modules, even shared libraries that extension modules rely on! Coupled with this format is the oxidized_importer meta path finder ( https://pypi.org/project/oxidized-importer/ and https://pyoxidizer.readthedocs.io/en/latest/oxidized_importer.html) capable of servicing imports and resource loading from these "Python packed resources" data structures. From a super high level, PyOxidizer assembles an instance of "Python packed resources" containing the CPython standard library and any additional Python packages you point it at and produces an executable with a main() that starts a Python interpreter, configures oxidized_importer.OxidizedFinder to read from the configured packed resources data structure (which may be embedded in the binary or loaded from a mmap()d file), and invokes some Python code inside to run your application. oxidized_importer has an API for reading and writing "Python packed resources" data structures. You can even use it to build your own PyOxidizer-like utilities ( https://pyoxidizer.readthedocs.io/en/latest/oxidized_importer_freezing_appl… ). I bring this work up because I believe that if you set yourself on a path to build a performant and fully featured importer/finder, you will inevitably build something with properties very similar to what I have built. To be uncompromising on performance, you'll want to roll your own data format that is in tune with Python's specific needs and avoids I/O and overhead when possible. To fully support the long-tail of features in Python's importing mechanism, you need the ability to richly - and efficiently - express metadata like whether a module is a package. It is possible to shoehorn this [meta]data into formats like tar and zip. But it won't be as efficient as rolling your own data structure. And when it comes to interpreter startup overhead, performance does matter. Am I suggesting CPython use oxidized_importer? No. It is implemented in Rust and CPython can't take a Rust dependency. Am I suggesting CPython support the "Python packed resources" data format as-is? No. The exact format today isn't suitable for CPython: I didn't design it with consideration for use beyond PyOxidizer's use case and there are still a ton of missing features. What I am suggesting is that Python developers think about the idea of standardizing a Python-centric container format for holding "Python resources" and a built-in/stdlib meta path finder for using it. Think of this as "frozen/zip importer 2.0" but with a more strongly defined and portable data format that is detached from C struct definitions. This could potentially solve a lot of problems around startup/import performance. And if you wanted to extend it to packaging/distribution, I think it could solve a lot of problems there too. (If you designed the format properly, I think it would be possible to converge with the use case of wheels.) (But I understand the skepticism about making the leap to packaging: that is an absurdly complex problem space!) If this idea sounds radical to you, I get the skepticism. I didn't want to incur this work/complexity when writing PyOxidizer either. But a long series of investigations and ruling out alternatives lead me down this path. With the benefit of hindsight I believe the type of solution is sound and it is inevitable Python gains something like this in the standard library or at least sees something like this in wide use in the wild. I say that because multiple Python app distribution tools have reinvented solutions to the general problem of "package multiple modules/resources in a single, efficient-to-load file/binary" in different ways because the solutions in the standard library (frozen and zip importers) or package distribution (wheels or eggs) just aren't sufficient because they each lack critical features. oxidized_importer _might_ be the most robust of these solutions to also be available as a standalone package on PyPI. I would encourage you to play around with oxidized_importer outside the context of PyOxidizer. I think you'll be pleasantly surprised by its performance and ability to emulate most of the common parts of the importlib APIs. The API for working with "Python packed resources" data structures isn't great. But only because I haven't spent much effort in making it so. I believe there's a path to adding a meta path importer to the stdlib that - like oxidized_importer - reads resource data from a well-defined data structure while retaining the performance of the frozen importer with the full feature set of PathFinder. I would suggest this as a better longer term solution than trying to incrementally evolve the frozen or zip importers to fit this use case. You could probably implement most of it in Python and freeze the bytecode into the interpreter like we do with PathFinder, leaving only the performance-sensitive parser to be implemented in C. All that being said, what I advocate for is obviously a lot of scope bloat versus doing some quick work to enable use of the frozen importer on a few dozen stdlib modules to speed up interpreter startup as is being discussed in issue45020. The practical engineer in me supports doing the quick and dirty solution now for the quick win. But I do encourage thinking bigger towards longer-term solutions, especially if you find yourself tempted to incrementally add features to frozen importer. I believe there is a market need for a stdlib meta path importer that reads a highly optimized and portable format similar to the solutions I've devised for PyOxidizer. Let me know how I can help incorporate one in the standard library. Gregory [View Less]

6 8

Bug fixed in the bugs.python.org OAuth-based authentication: user logged as the wrong account
by Victor Stinner Sept. 13, 2021

Sept. 13, 2021

Hi, A bug has been identified and *fixed* in the OAuth-based authentication code used on the Python bug tracker bugs.python.org (BPO) to log in with GitHub, Launchpad or Google. Under some conditions, it was possible to be logged as another person account. We are only aware of a single user affected by the issue. We are not aware of any account takeover. All bugs at bugs.python.org are public: being logged as the wrong account cannot give access to private bugs. The main risk is if an … [View More]

2 1

PEP 467 feedback from the Steering Council
by Barry Warsaw Sept. 12, 2021

Sept. 12, 2021

Hello Nick, Ethan, The Python Steering Council reviewed PEP 467 -- Minor API improvements for binary sequences at our 2021-07-26 meeting. Thank you for work on this PEP. We’re generally very favorable for adding to Python 3.11 the features and APIs described in the PEP. We have some requests for changes that we’d like you to consider. * The Python-Version in the PEP needs to target Python 3.11 of course. * We think it would be better if bytes.fromsize()’s second argument was a keyword-… [View More]

21 57

Summary of Python tracker Issues
by Python tracker Sept. 10, 2021

Sept. 10, 2021

ACTIVITY SUMMARY (2021-09-03 - 2021-09-10) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7376 (-33) closed 49524 (+105) total 56900 (+72) Open issues with patches: 2929 Issues opened (53) ================== #36550: Avoid creating AttributeError exceptions in the debugger https://bugs.python.org/issue36550 reopened by iritkatriel #45083: Need to … [View More]

1 0

Spam from Daniel Scott
by Victor Stinner Sept. 10, 2021

Sept. 10, 2021

Hi, For 1 or 2 years, I'm getting spam messages from "Daniel Scott". It started with empty replies to multiple Python mailing lists. Mailing list owners: please block his email address. I never ever saw any useless message from this name (sorry if a real Daniel Scott plans to contribute to Python and will be blocked by mistake). Today, I got an email to my @python.org email account. And "Daniel Scott" also commented a closed issue: "I left you to see conduct on phython assigned." https://… [View More]

1 0

Re: PEP 646 (Variadic Generics): final call for comments
by bas van beek Sept. 9, 2021

Sept. 9, 2021

Hi all, I very much share Stephan's opinion here and look forward to integrating the new PEP 646 variadics into numpy. In the context of numpy (and tensor typing general): the typing of array shapes is a fairly complicated subject and the introduction of variadics will likely play in big role in laying its foundation, as it allows for the expression of both dimensioability as well as basic shape manipulation . All in all, I'm very interested in where both PEP 646 and future PEPs will take us … [View More]

1 0

Discrepancy between what aiter() and `async for` requires on purpose?
by Brett Cannon Sept. 8, 2021

Sept. 8, 2021

If you look at https://github.com/python/cpython/blob/b11a951f16f0603d98de24fee5c023df83ea… you will see that `async for` requires that the iterator returned from `__aiter__` define `__anext__`. But if you look at aiter() which uses PyObject_GetAiter() from https://github.com/python/cpython/blob/f0a6fde8827d5d4f7a1c741ab1a8b206b66f… and PyAiter_Check() from https://github.com/python/cpython/blob/f0a6fde8827d5d4f7a1c741ab1a8b206b66f… you will notice that aiter() requires `__anext__` *and* `… [View More]

7 9

Re: [Python-ideas] Re: open functions in dbm submodule need to support path-like object
by David Mertz, Ph.D. Sept. 8, 2021

Sept. 8, 2021

I attempted to do this today, as my first actual contribution to CPython itself. I think the prior attempt went down a wrong path, which is why neither PR could actually pass tests. I've been looking at `posixmodule.c` for comparison, specifically. The key thing, I believe, is not to use `PyObject *filename` in the `dpmopen_impl()` calls at all, but rather `path_t *path`. I tried some absolutely crude instrumentation to see why the `PyOS_FSPath` macro and the rest wasn't doing what it seems … [View More]

2 2