Mailman 3 March 2021 - Python-ideas

Allow adding custom logging levels as attributes
by harry.lees＠gmail.com 26 Mar '21

26 Mar '21

I have just been writing some code in which I define a custom log level with the logging module. I wanted to call the new log level as an attribute but currently the logging library lacks a nice way to do this. I know that in the docs https://docs.python.org/3/howto/logging.html#custom-levels it discourages creating custom log levels so perhaps this was why there is no way to do this. This https://stackoverflow.com/questions/2183233/how-to-add-a-custom-loglevel-to… StackOverflow solution allows for the syntax I was after, but it is hard to understand for someone looking at the code. import logging CUSTOM_LEVEL = 31 CUSTOM_LEVEL_NAME = 'CUSTOM_WARNING' def custom_warning(self, message, *args, **kwargs): if logger.isEnabledFor(CUSTOM_LEVEL): logger._log(CUSTOM_LEVEL, message, args, **kwargs) logging.Logger.custom_warning = custom_warning logger = logging.getLogger(__name__) logger.warning('a warning here!') logger.custom_warning('a custom warning!') I would propose something like logger.addCustomLevel(CUSTOM_LEVEL, CUSTOM_LEVEL_NAME) There are some obvious limitations to this approach, all level names would also have to be valid function names otherwise something like logger.addCustomLevel(CUSTOM_LEVEL, "1 2 3") would be an issue what does everybody think?

1 0

Make fnmatch.filter accept a tuple of patterns
by Andre Delfino 26 Mar '21

26 Mar '21

Frequently, while globbing, one needs to work with multiple extensions. I’d like to propose for fnmatch.filter to handle a tuple of patterns (while preserving the single str argument functionality, alas str.endswith), as a first step for glob.i?glob to accept multiple patterns as well. Here is the implementation I came up with: https://github.com/python/cpython/compare/master...andresdelfino:fnmatch-mu… If this is deemed reasonable, I’ll write tests and documentation updates. Any opinion?

9 15

Improve the textual representation of IPv4-mapped IPv6 addresses
by Maxime Mouchet 25 Mar '21

25 Mar '21

Hi, Python supports IPv4-mapped IPv6 addresses as defined by RFC 4038: "the IPv6 address ::FFFF:x.y.z.w represents the IPv4 address x.y.z.w.” The current behavior is as follows: from ipaddress import ip_address addr = ip_address('::ffff:8.8.4.4') # IPv6Address('::ffff:808:404') addr.ipv4_mapped # IPv4Address('8.8.4.4') Note that the textual representation of the IPv6Address is *not* in IPv4-mapped format. It prints ::ffff:808:404 instead of ::ffff:8.8.4.4. This is technically correct, but it’s somewhat frustrating as it makes it harder to read IPv4s embedded in IPv6 addresses. My proposal would be to check, in __str__, if an IPv6 is an IPv4-mapped, and to return the appropriate representation : from ipaddress import ip_address addr = ip_address('::ffff:8.8.4.4') # Current behavior str(addr) # '::ffff:808:404' repr(addr) # IPv6Address('::ffff:808:404') # Proposed behavior str(addr) # '::ffff:8.8.4.4' repr(addr) # IPv6Address('::ffff:8.8.4.4') A few data points: - Julia prints ::ffff:808:404 (current behavior) - C (glibc) and ClickHouse prints ::ffff:8.8.4.4 (proposed behavior) Any thoughts? Maxime

2 1

Looking for people interested in a Python register virtual machine project
by Skip Montanaro 24 Mar '21

24 Mar '21

Back in the late 90s (!) I worked on a reimagining of the Python virtual machine as a register-based VM based on 1.5.2. I got part of the way with that, but never completed it. In the early 2010s, Victor Stinner got much further using 3.4 as a base. The idea (and dormant code) has been laying around in my mind (and computers) these past couple decades, so I took another swing at it starting in late 2019 after retirement, mostly as a way to keep my head in the game. While I got a fair bit of the way, it stalled. I've picked it up and put it down a number of times in the past year, often needing to resolve conflicts because of churn in the current Python virtual machine. Though I kept getting things back in sync, I realize this is not a one-person project, at least not this one person. There are several huge chunks of Python I've ignored over the past 20 years, and not just the internals. (I've never used async anything, for example.) If it is ever to truly be a viable demonstration of the concept, I will need help. I forked the CPython repo and have a branch (register2) of said fork which is currently synced up with the 3.10 (currently master) branch: https://github.com/smontanaro/cpython/tree/register2 I started on what could only very generously be called a PEP which you can read here. It includes some of the history of this work as well as details about what I've managed to do so far: https://github.com/smontanaro/cpython/blob/register2/pep-9999.rst If you think any of this is remotely interesting (whether or not you think you'd like to help), please have a look at the "PEP". Because this covers a fair bit of the CPython implementation, chances to contribute in a number of areas exist, even if you have never delved into Python's internals. Questions/comments/pull requests welcome. Skip Montanaro

8 20

Suggesting feature for python - Big Set, Big Dictionary
by Vijay Patel 24 Mar '21

24 Mar '21

Hi, (This is great language, sincere thanks to all!) I took a quick glance of https://devguide.python.org/langchanges/ (20.3. Suggesting new features and language changes), while writing this post. Please ignore this, if similar proposals are discussed earlier. Suggestion: 1. Big Set, Big Dictionary - During design time, i am considering this question. With limited RAM (example 4 GB), how can I store a dictionary/set of size greater than RAM. Is it possible to introduce Big Set / Big Dictionary which stores data in hard disk (based on some partition mechanism - like big data)? (i.e.Big Set/Big Dictionary with set access mechanism identical to current set and dictionary). Background: I am working on machine learning NLP and would like to use such feature in data generator. Regards, Vijay <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> Virus-free. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

4 3

Itertools generator injection
by Alastair Stanley 22 Mar '21

22 Mar '21

This is closely related to https://bugs.python.org/issue40230 - "Itertools.product() Out of Memory Errors." `itertools.product()` completely consumes all input iterables before yielding any values, which can cause memory issues in certain (extreme) cases. I recently needed to partially iterate through the product of two very large iterators which were themselves power sets of some input values. For unlucky inputs these iterators were enormous (but not infinite) and `itertools.product()` used up all the memory on my machine, even though I sometimes only needed the first few values. My solution was to write my own version of `itertools.product()` that takes generators instead of iterables. This allowed me to recreate and loop through the required iterables multiple times without ever having to store all their values. Product tuples are produced immediately and the iteration can be completed for large inputs without runaway memory consumption. I'm proposing that equivalent functions accepting functions/generators: maybe `fproduct()`, `fcombinations()` etc, be added to `itertools`. e.g. simplified `fproduct` for 2 inputs: ``` def fproduct(f1, f2): for a in f1(): for b in f2(): yield (a, b) ``` Would others find this useful? Are there any drawbacks I'm missing? Best, Alastair

5 5

Installer/target architecture mismatch
by Dany Lisiansky 22 Mar '21

22 Mar '21

An odd suggestion/request here, hope it's the right place to discuss it. So I was trying to install python on the Xbox series S (yup..), so far I got the embedded x86_64 version for windows to work, however, I was unable to get packages to install properly. I'm not familiar enough with Windows to know how the installation is supposed to be laid out, but I figured it might be just a side affect from using the embedded version. Now I need to mention that the Xbox is an odd NT based system, it can run win32/UWP apps, but it comes with a strict requirement of x86_64 and therefore has no SysWoW64 to support i386. Naturally, I tried to use the distributed installer, and that's when I found out that the x86_64 installer itself was in fact compiled for i386. I can compile a one-off build for myself and go on with my life, but I also realized it's an oversight because the installer should never assume the target system has backwards compatibility for older architectures and stick to the target architecture throughout the installation phase. I know It's an extreme edge case, but I guess it's valid and make sense. Let me know what you think, and how (or if) I should proceed from here. DanyL.

2 1

shouldn't slices be iterable ?
by Cameron Simpson 19 Mar '21

19 Mar '21

I know that range(start,end,stride) will produce what I'd want from iter(slice(start,end,stride)), but wouldn't it be reasonable for a slice itself to be iterable? Yes, only one obvious way and all that, but inside eg __getitem__ it seems to me that: if isinstance(index, slice): for i in index: ... do stuff with i ... is the obvious thing to do. Cheers, Cameron Simpson <cs(a)cskk.id.au>

4 5

self documenting kwarg passing in functions similar to f-strings in python3.8
by alexei.ciobanu＠adelaide.edu.au 18 Mar '21

18 Mar '21

Hi all, Not sure if this feature already exists in python or if it has been proposed already but it would be extremely helpful for me and doesn't seem too difficult to implement. I often write functions with really long kwarg names that call other functions that share a kwarg name. What I currently have is something like (very simplified example) def function1(x, y, some_really_long_kwarg=True, **kwargs): other_kwarg = x>y function2(x, y, some_really_long_kwarg=some_really_long_kwarg, other_kwarg=other_kwarg) return What I would like is to be able to write something like def function1(x, y, some_really_long_kwarg=True, **kwargs): other_kwarg = x>y function2(x, y, some_really_long_kwarg=, other_kwarg=) return where the kwarg= behaves similarly to how it does for f-strings in python3.8 when you write something like f'{other_kwarg=}'

2 1

dataclasses keyword-only fields, take 2
by Eric V. Smith 18 Mar '21

18 Mar '21

[I'm sort of loose with the terms field, parameter, and argument here. Forgive me: I think it's still understandable. Also I'm not specifying types here, I'm using Any everywhere. Use your imagination and substitute real types if it helps you.] Here's version 2 of my proposal: There have been many requests to add keyword-only fields to dataclasses. These fields would result in __init__ parameters that are keyword-only. In a previous proposal, I suggested also including positional arguments for dataclasses. That proposal is at https://mail.python.org/archives/list/python-ideas@python.org/message/I3RKK… . After some discussion, I think it's clear that positional arguments aren't going to work well with dataclasses. The deal breaker for me is that the generated repr would either not work with eval(), or it would contain fields without names (since they're positional). There are additional concerns mentioned in that thread. Accordingly, I'm going to drop positional arguments from this proposal. Basically, I want to add a flag to each field, stating whether the field results in a normal parameter or a keyword-only parameter to __init__. Then when I'm generating __init__, I'll examine those flags and put the normal arguments first, followed by the keyword-only ones. The trick becomes: how do you specify what type of parameter each field represents? What attrs does --------------- First, here's what attrs does. There's a parameter to their attr.ib() function (the moral equivalent of dataclasses.field()) named kw_only, which if set, marks the field as being keyword-only. From https://www.attrs.org/en/stable/examples.html#keyword-only-attributes : >>> @attr.s ... class A: ... a = attr.ib(kw_only=True) >>> A() Traceback (most recent call last): ... TypeError: A() missing 1 required keyword-only argument: 'a' >>> A(a=1) A(a=1) There's also a parameter to attr.s (the equivalent of dataclasses.dataclass), also named kw_only, which if true marks every field as being keyword-only: >>> @attr.s(kw_only=True) ... class A: ... a = attr.ib() ... b = attr.ib() >>> A(1, 2) Traceback (most recent call last): ... TypeError: __init__() takes 1 positional argument but 3 were given >>> A(a=1, b=2) A(a=1, b=2) dataclasses proposal -------------------- I propose to adopt both of these methods (dataclass(kw_ony=True) and field(kw_only=True) in dataclasses. The above example would become: >>> @dataclasses.dataclass ... class A: ... a: Any = field(kw_only=True) >>> @dataclasses.dataclass(kw_only=True) ... class A: ... a: Any ... b: Any But, I'd also like to make this a little easier to use, especially in the case where you're defining a dataclass that has some normal fields and some keyword-only fields. Using the attrs approach, you'd need to declare the keyword-only fields using the "=field(kw_only=True)" syntax, which I think is needlessly verbose, especially when you have many keyword-only fields. The problem is that if you have 1 normal parameter and 10 keyword-only ones, you'd be forced to say: @dataclasses.dataclass class LotsOfFields: a: Any b: Any = field(kw_only=True, default=0) c: Any = field(kw_only=True, default='foo') d: Any = field(kw_only=True) e: Any = field(kw_only=True, default=0.0) f: Any = field(kw_only=True) g: Any = field(kw_only=True, default=()) h: Any = field(kw_only=True, default='bar') i: Any = field(kw_only=True, default=3+4j) j: Any = field(kw_only=True, default=10) k: Any = field(kw_only=True) That's way too verbose for me. Ideally, I'd like something like this example: @dataclasses.dataclass class A: a: Any # pragma: KW_ONLY b: Any And then b would become a keyword-only field, while a is a normal field. But we need some way of telling dataclasses.dataclass what's going on, since obviously pragmas are out. I propose the following. I'll add a singleton to the dataclasses module: KW_ONLY. When scanning the __attribute__'s that define the fields, a field with this type would be ignored, except for assigning the kw_only flag to fields declared after these singletons are used. So you'd get: @dataclasses.dataclass class B: a: Any _: dataclasses.KW_ONLY b: Any This would generate: def __init__(self, a, *, b): This example is equivalent to: @dataclasses.dataclass class B: a: Any b: Any = field(kw_only=True) The name of the KW_ONLY field doesn't matter, since it's discarded. I think _ is a fine name, and '_: dataclasses.KW_ONLY' would be the pythonic way of saying "the following fields are keyword-only". My example above would become: @dataclasses.dataclass class LotsOfFields: a: Any _: dataclasses.KW_ONLY b: Any = 0 c: Any = 'foo' d: Any e: Any = 0.0 f: Any g: Any = () h: Any = 'bar' i: Any = 3+4j j: Any = 10 k: Any Which I think is a lot clearer. The generated __init__ would look like: def __init__(self, a, *, b=0, c='foo', d, e=0.0, f, g=(), h='bar', i=3+4j, j=10, k): The idea is that all normal argument fields would appear first in the class definition, then all keyword argument fields. This is the same requirement as in a function definition. There would be no switching back and forth between the two types of fields: once you use KW_ONLY, all subsequent fields are keyword-only. A field of type KW_ONLY can appear only once in a particular dataclass (but see the discussion below about inheritance). Re-ordering args in __init__ ---------------------------- If, using field(kw_only=True), you specify keyword-only fields before non-keyword-only fields, all of the keyword-only fields will be moved to the end of the __init__ argument list. Within the list of non-keyword-only arguments, all arguments will keep the same relative order as in the class definition. Ditto for within keyword-only arguments. So: @dataclasses.dataclass class C: a: Any b: Any = field(kw_only=True) c: Any d: Any = field(kw_only=True) Then the generated __init__ will look like: def __init__(self, a, c, *, b, d): __init__ is the only place where this rearranging will take place. Everywhere else, and importantly in __repr__ and any dunder comparison methods, the order will be the same as it is now: in field declaration order. This is the same behavior that attrs uses. Inheritance ----------- There are a few additional quirks involving inheritance, but the behavior would follow naturally from how dataclasses already handles fields via inheritance and the __init__ argument re-ordering discussed above. Basically, all fields in a derived class are computed like they are today. Then any __init__ argument re-ordering will take place, as discussed above. Consider: @dataclasses.dataclass(kw_only=True) class D: a: Any @dataclasses.dataclass class E(D): b: Any @dataclasses.dataclass(kw_only=True) class F(E): c: Any This will result in the __init__ signature of: def __init__(self, b, *, a, c): However, the repr() will still produce the fields in order a, b, c. Comparisons will also use the same order. Conclusion ---------- Remember, the only point of all of these hoops is to add a flag to each field saying what type of __init__ argument it becomes: normal or keyword-only. Any of the 3 methods discussed above (kw_only flag to @dataclass(), kw_only flag to field(), or the KW_ONLY marker) all have the same result: setting the kw_only flag on one or more fields. The value of that flag, on a per-field basis, is used to re-order __init__ arguments, and is used in generating the __init__ signature. It's not used anywhere else. I expect the two most common use cases to be the kw_only flag to @dataclass() and the KW_ONLY marker. I would expect the usage of the kw_only flag on field() to be rare, but since it's the underlying mechanism and it's needed for more complex field layouts, it is included in this proposal. So, what do you think? Is this a horrible idea? Should it be a PEP, or just a 'simple' feature addition to dataclasses? I'm worried that if I have to do a full blown PEP I won't get to this for 3.10. mypy and other type checkers would need to be taught about all of this. -- Eric

8 20