Python-ideas

python-ideas@python.org

39 participants
5150 discussions

by Jack DeVries

Hi! I'm interested in this workflow issue of cpython pull requests that get tons of review and feedback, but, for whatever reason, ultimately lose the attention of the original contributor. I opened a GitHub issue on the dev guide repo where I wrote more about the problem, and proposed some solutions/ideas. I'm hoping that the issue might get more attention if I mentioned it here, too, so I'm wondering if you can take a moment to check it out and continue the conversation on GitHub. https://github.com/python/devguide/issues/731 Thanks!! Jack

15 hours, 42 minutes

RFC for PEP 663: Improving and Standardizing Enum str(), repr(), and format() behaviors

by Ethan Furman

PEP: 663 Title: Improving and Standardizing Enum str(), repr(), and format() behaviors Version: $Revision$ Last-Modified: $Date$ Author: Ethan Furman <ethan(a)stoneleaf.us> Discussions-To: python-dev(a)python.org Status: Draft Type: Informational Content-Type: text/x-rst Created: 23-Feb-2013 Python-Version: 3.11 Post-History: 20-Jul-2021 Resolution: Abstract ======== Now that we have a few years experience with Enum usage it is time to update the ``repr()``, ``str()``, and ``format()`` of the various enumerations by their intended purpose. Motivation ========== The addition of ``StrEnum`` with its requirement to have its ``str()`` be its ``value`` is inconsistent with other provided Enum's ``str``. Having the ``str()`` of ``IntEnum`` and ``IntFlag`` not be the value causes bugs and extra work when replacing existing constants. Having the ``str()`` and ``format()`` of an enum member be different can be confusing. The iteration of ``Flag`` members, which directly affects their ``repr()``, is inelegant at best, and buggy at worst. Rationale ========= Enums are becoming more common in the standard library; being able to recognize enum members by their ``repr()``, and having that ``repr()`` be easy to parse, is useful and can save time and effort in understanding and debugging code. However, the enums with mixed-in data types (``IntEnum``, ``IntFlag``, and the new ``StrEnum``) need to be more backwards compatible with the constants they are replacing -- specifically, ``str(replacement_enum_member) == str(original_constant)`` should be true (and the same for ``format()``). IntEnum, IntFlag, and StrEnum should be as close to a drop-in replacement of existing integer and string constants as is possible. Towards that goal, the str() output of each should be its inherent value; e.g. if ``Color`` is an ``IntEnum``:: >>> Color.RED <Color.RED: 1> >>> str(Color.RED) '1' >>> format(Color.RED) '1' Note that format() already produces the correct output in 3.10, only str() needs updating. As much as possible, the ``str()`, ``repr()``, and ``format()`` of enum members should be standardized across the stardard library. The repr() of Flag currently includes aliases, which it should not; fixing that will, of course, already change its ``repr()`` in certain cases. Specification ============= There a three broad categories of enum usage: - standard: Enum or Flag a new enum class is created, and the members are used as ``class.member_name`` - drop-in replacement: IntEnum, IntFlag, StrEnum a new enum class is created which also subclasses ``int`` or ``str`` and uses ``int.__str__`` or ``str.__str__``; the ``repr`` can be changed by using the ``global_enum`` decorator - user-mixed enums and flags the user creates their own integer-, float-, str-, whatever-enums instead of using enum.IntEnum, etc. Some sample enums:: # module: tools.py class Hue(Enum): # or IntEnum LIGHT = -1 NORMAL = 0 DARK = +1 class Color(Flag): # or IntFlag RED = 1 GREEN = 2 BLUE = 4 class Grey(int, Enum): # or (int, Flag) BLACK = 0 WHITE = 1 Using the above enumerations, the following table shows the old and new behavior, while the last shows the final result: +-------------+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | type | enum repr() | enum str() | enum format() | flag repr() | flag str() | flag format() | +-------------+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | standard | 3.10 | | | | <Color.RED|GREEN: 3> | Color.RED|GREEN | Color.RED|GREEN | | +----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | | new | | | | <Color(3): RED|GREEN> | Color.RED|Color.GREEN | Color.RED|Color.GREEN | +-------------+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ +-------------+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | user mixed | 3.10 | | | 1 | <Grey.WHITE: 1> | | 1 | | +----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | | new | | | Grey.WHITE | <Grey(1): WHITE> | | Grey.WHITE | +-------------+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ +-------------+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | int drop-in | 3.10 | | Hue.LIGHT | | <Color.RED|GREEN: 3> | Color.RED|GREEN | | | +----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | | new | | -1 | | <Color(3): RED|GREEN> | 3 | | +-------------+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ +-------------+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | global | 3.10 | <Hue.LIGHT: -1> | Hue.LIGHT | Hue.LIGHT | <Color.RED|GREEN: 3> | Color.RED|GREEN | Color.RED|GREEN | | +----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | | new | tools.LIGHT | LIGHT | LIGHT | tools.RED|tools.GREEN | RED|GREEN | RED|GREEN | +-------------+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ +-------------+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | user mixed | 3.10 | <Grey.WHITE: 1 | Grey.WHITE | Grey.WHITE | <Grey.WHITE: 1> | Grey.WHITE | 1 | | +----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | | new | tools.WHITE | WHITE | WHITE | tools.WHITE | WHITE | WHITE | +-------------+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ +-------------+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | int drop-in | 3.10 | <Hue.LIGHT: -1> | Hue.LIGHT | | <Color.RED|GREEN: 3> | Color.RED|GREEN | | | +----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | | new | tools.LIGHT | -1 | | tools.RED|tools.GREEN | 3 | | +-------------+----------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ Which will result in: +-------------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | type | enum repr() | enum str() | enum format() | flag repr() | flag str() | flag format() | +-------------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | standard | <Hue.LIGHT: -1> | Hue.LIGHT | Hue.LIGHT | <Color(3): RED|GREEN> | Color.RED|Color.GREEN | Color.RED|Color.GREEN | +-------------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | user mixed | <Grey.WHITE: 1> | Grey.WHITE | Grey.WHITE | <Grey(1): WHITE> | Grey.WHITE | Grey.WHITE | +-------------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | int drop-in | <Hue.LIGHT: -1> | -1 | -1 | <Color(3): RED|GREEN> | 3 | 3 | +-------------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | global | tools.LIGHT | LIGHT | LIGHT | tools.RED|tools.GREEN | RED|GREEN | RED|GREEN | +-------------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | user mixed | tools.WHITE | WHITE | WHITE | tools.WHITE | WHITE | WHITE | +-------------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ | int drop-in | tools.LIGHT | -1 | -1 | tools.RED|tools.GREEN | 3 | 3 | +-------------+-----------------+------------+-----------------------+-----------------------+------------------------+-----------------------+ As can be seen, ``repr()`` is primarily affected by whether the members are global, while ``str()`` is affected by being global or by being a drop-in replacement, with the drop-in replacement status having a higher priority. Also, the basic ``repr()`` and ``str()`` have changed for flags as the old style was very clunky. The ``repr()`` for Enum vs Flag are different, primarily because the Enum ``repr()`` does not work well for flags. I like being able to tell whether an enum member is a Flag or an Enum based on the ``repr()`` alone, but am open to arguments for changing Enum's ``repr()`` to match Flag's. Backwards Compatibility ======================= My understanding is that ``str()`` and ``repr()`` output has much lower backwards compatibility requirements. Even so, I expect the majority of breakage to be in doc and unit tests. I'm less clear on the policy for ``format()``. Note that by changing the ``str()`` of the drop-in category, we will actually prevent future breakage when ``IntEnum``, et al, are used to replace existing constants. Mitigation ========== Normal usage of enum members will not change: ``re.ASCII`` can still be used as ``re.ASCII`` and will still compare equal to ``256``. If one wants their own enums in their own code to remain the same they will need to write their own base Enum class and then write the appropriate ``repr``, ``str()``, and ``format()`` methods (or copy them from the 3.10 enum module). Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.

4 days, 15 hours

Nested Dictionary Pointers and Logic - "flatten / nested iteration" Proposal

by Sven Voigt

Hello Python-ideas, I am working with lots of JSON objects in my work and need to obtain JSON pointers to particular components of a JSON object. These pointers are defined as part of the JSON-LD specifications as relative IRIs and as part of the JSON-Schema specifications. However, I am not aware of these pointers being part of the JSON specification. The pointers are really straightforward and follow the same syntax as other IRIs (URLs) or file paths. For example, the following flatten function illustrates the mapping of nested dictionaries and arrays to a single flat dictionary, where each nested key is mapped to a unique path. d = {"a": 1, "b": {"ba": 2, "bb": [{"bba": 3, "bbb": None}]}} flatten(d) >> {'a': 1, 'b/ba': 2, 'b/bb/0/bba': 3, 'b/bb/0/bbb': None} Not only does this conversion help in generating JSON pointers, but it helps with logic for nested data structures. Specifically, assume there is a dictionary, which includes nested dictionaries, lists, and tuples, and any of these elements are contained within another dictionary or that other dictionaries elements. Then it is not sufficient to simply compare the items of the two dictionaries to check whether the first is a subset of the other. However, the flat structures can be compared directly. 1. A nested dictionary is a subset of another nested dictionary. The flatten function ensures the nested dictionaries are not checked for equivalence but subset of. # Current {"a": 1, "c": {"d": 4}}.items() <= {"a": 1, "b": 2, "c": {"d": 4, "e": 5}}.items() >> False # Proposed flatten({"a": 1, "c": {"d": 4}}).items() <= flatten({"a": 1, "b": 2, "c": {"d": 4, "e": 5}}).items() >> True 2. A nested list or tuple is a subset of a dictionary. # Current {"a": 1, "c": [3]}.items() <= {"a": 1, "b": 2, "c": [3,3]}.items() >> False # Proposed flatten({"a": 1, "c": [3]}).items() <= flatten({"a": 1, "b": 2, "c": [3,3]}).items() >> True Note that these examples only have one level of nesting, but the flatten function must handle any level of nesting. For the current version of the flatten function, I essentially borrowed the JSON "dumps" source code. However, the dumps code doesn't utilize any other functions and I believe there may be a missing "nestedIter" function in python. This function could be used for JSON dumps or flatten or anytime someone wants to visit all nested elements of a dictionary and perform some operation. Therefore, I think this function could be included in the JSON library if python-ideas thinks that the pointers are very specific to JSON or the itertools library, where the nested iteration function could be generally used for all nested data types. Let me know what you think of this proposal and I am looking forward to your responses. Best, Sven

4 days, 17 hours

binascii generalization

by Ronald Hoogenboom

I see that the binascii module has several {a2b,b2a}_xyz functions for several flavors of ascii <-> binary conversions. But it misses a general base function set parameterized with a radix. It would be nice to add a generic set of {a2b|b2a}(s, radix=r, case={'u','l'}), where radix is a power of 2 < 36 (2, 4, 8, 16 or 32) and case is only significant for b2a, to cover all trivial ascii <-> binary conversions using an efficient native implementation. Most {a2b,b2a}_xyz can just call the generic {a2b|b2a} with the appropriate radix. 'radix' could also be called 'base' for consistency with the int() functions. I was actually surprised that it wasn't the case already...

5 days, 22 hours

weakref.link: Keep A alive while B is alive (and/or update WeakKeyDictionary)

by Mark Gordon

Proposal: Have a weakref.link (not at all attached to the naming) primitive that allows one to keep object A alive while object B is alive. This would be functionally similar to adding A as an attribute of B from the GC's perspective without making any assumptions about what attributes are available in B. Here's a really course example of how this might be used: # Associate "extra_data" with "obj" where "extra_data" will lose its reference when "obj" is no longer reachable through strong refs. my_link = weakref.link(obj, extra_data) ... obj, extra_data = my_link() if obj is not None: do_something(obj, extra_data) This would also allow us to fix or create a new version of WeakKeyDictionary where values can have strong references back to their key without preventing their key from being reclaimed. This has been an issue for tensorflow in the past (see https://github.com/tensorflow/tensorflow/issues/42022) and is likely to be a subtle issue in many more libraries. Even the suggested use case in the docs "This can be especially useful with objects that override attribute accesses." is likely to be broken because of this issue. While I'd be weary of changing the semantics of an existing data structure it seems highly unlikely anything depends on this behavior since it makes WeakKeyDictionary degrade to behaving like a normal dictionary for that key. Notes: * Not attached to this specific approach or API * Mostly interesting in updating/creating new WeakKeyDictionary as mentioned

6 days, 11 hours

builtins for running context managers

by Thomas Grainger

currently lots of code manages contexts incorrectly by doing: ``` @dataclasses.dataclass class WrapCmgr: _cmgr: ContextManager[T] def __enter__(self) -> Wrapped[T]: return wrap(_cmgr.__enter__()) def __exit__(self, \, t: Type[BaseException] | None, v: BaseException, tb: types.TracebackType) -> bool: return _cmgr.__exit__(t, v, tb) ``` since https://bugs.python.org/issue44471 using `with WrapCmgr(cmgr()):` incorrectly raises an AttributeError rather than a TypeError and https://www.python.org/dev/peps/pep-0343/ still includes this bug, raising an AttributeError ``` mgr = (EXPR) exit = type(mgr).__exit__ # Not calling it yet value = type(mgr).__enter__(mgr) ``` contextlib.ExitStack also had this bug https://bugs.python.org/issue12022 given that it's hard to implement this correctly I think there should be a builtins.enter and builtins.aenter for use like this: ``` @dataclasses.dataclass class WrapCmgr: _cmgr: ContextManager[T] def __enter__(self) -> Wrapped[T]: exit, value = enter(self._cmgr) self.__exit = exit return wrap(value) def __exit__(self, \, t: Type[BaseException] | None, v: BaseException, tb: types.TracebackType) -> bool: return self.__exit(t, v, tb) ```

1 week, 2 days

Looking for a PEP sponsor

by Nick Humrich

Hello Python Devs, In accordance to PEP 0, I am looking for a Python-core-developer to sponsor my new PEP. It is essentially a revisit of PEP 501. https:// github.com/nhumrich/peps/blob/master/pep-9999.rst <https://t.co/cCrqfWztAW?amp=1> Please let me know if you have questions, or would be interested in sponsoring this PEP. Nick Humrich

1 week, 3 days

Advanced RGB Coloring

by z3ntl3discord＠gmail.com

Hey dear community, I want to use advanced RGB Coloring in python for teminals. There are already some modules but not advancrd enough. I hope this feature will be added in the next python versions! ❤ I believe in this strong community, this will powerup our ascii designs in terminals most users made!

1 week, 4 days

writelines2?

by sandhoners123＠gmail.com

Right now, writelines is a very big misnomer to many python developers, especially beginners who would expect writelines to put new lines at the end of every element in the list My suggestion is to have either a writelines2 or a newline kwarg which does put new lines automatically at the end of every line written

1 week, 5 days

Deprecation APIs and language tooling

by Sergei Lebedev

Hi python-ideas, The subject is by no means new nor unique, and in fact has been discussed on python-ideas at least once <https://mail.python.org/archives/list/python-ideas@python.org/thread/OPY3E4…> before as well as in python/mypy#2403 <https://github.com/python/mypy/issues/2403>.. However, to date, there seems to be no standard way of deprecating things declaratively, and ~every major library has its own internal API for that. For example, * numpy.deprecate, * similar decorators in pandas <https://github.com/pandas-dev/pandas/blob/42adb9f80fb6a048995087276eda2a16e…> and TensorFlow <https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/util…>, * twisted.python.deprecate <https://twistedmatrix.com/documents/current/api/twisted.python.deprecate.ht…>, * warn_about_renamed_method <https://github.com/django/django/blob/ca9872905559026af82000e46cde6f7dedc89…> in Django, * etc etc. The proliferation of deprecation APIs itself is not a problem, but it does make it difficult (if not impossible) for language tooling to consume that deprecation information and help users of these libraries write better code. Imagine, if there was a standard way to flag something as deprecated. That would allow, for example, an IDE or a linter to statically flag a particular usage site as problematic and, assuming such information is available, to suggest an auto-generated fix (see @Deprecated in Kotlin <https://kotlinlang.org/api/latest/jvm/stdlib/kotlin/-deprecated/>, for example). What do you think? Is anyone interested in discussing this further and perhaps sketching an API which could work reasonably well for library authors as well as for tooling? Cheers, Sergei

1 week, 5 days

Jump to page:

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

Python-ideas