Mailman 3 March 2021 - Python-ideas

Enum: determining if a value is valid
by Ethan Furman March 17, 2021

March 17, 2021

A question that comes up quite a bit on Stackoverflow is how to test to see if a value will result in an Enum member, preferably without having to go through the whole try/except machinery. A couple versions ago one could use a containment check: if 1 in Color: but than was removed as Enums are considered containers of members, not containers of the member values. It was also possible to define one's own `_missing_` method and have it return None or the value passed in, but that has also been locked down to either return a member or raise an exception. At this point I see three options: 1) add a `get(value, default=None)` to EnumMeta (similar to `dict.get()` 2) add a recipe to the docs 3) do nothing Thoughts? -- ~Ethan~

6 16

Dataclass (or get_type_hints) wishlist item: Optional[...]
by Paul Bryan March 16, 2021

March 16, 2021

With some of the dataclass threads in python-ideas ongoing, I wanted to add another to the fray. There's an area where typing.get_type_hints on dataclass and __init__ produce inconsistent results. Example: @dataclass class Foo: x: int = None >>> typing.get_type_hints(Foo) {'x': <class 'int'>} >>> typing.get_type_hints(Foo.__init__) {'x': typing.Optional[int], 'return': <class 'NoneType'>} My wish list item would be for the dataclass annotations to match __init__'s. Rationale: by explicitly specifying a default value of a different type, the resulting type annotation for the field should be a Union of the specified type and the type of the default value. I might go so far as to say this could be how the type hints of function parameters should be encoded. Example to illustrate my thoughts—it does not work this way presently: def foo(x: str = 1): ... >>> typing.get_type_hints(foo) {'x': typing.Union[str, int]} This example could be construed as an error that a static type checker should catch and flag; if this is the consensus, then I fall back on the proposal to only supporting Union[type, None]—i.e. Optional[type]—which is how get_type_hints performs currently. Thoughts? Paul

1 0

Multiple dispatch transpilation
by Marvin van Aalst March 16, 2021

March 16, 2021

Hi fellow Pythonistas! A colleague of mine recently got me involved in the Julia language and while I was really impressed by the performance, the complete lack of OOP and thus bad discoverability (not to mention all their unicode function names) for me leaves a lot to be desired. TL;DR: Julia uses a type-based multi-method system (multiple dispatch), that because of type-stability allows JIT compilation of very efficient code. Since Pythons single-dispatch classes are a subset of that system, it should in principle be possible to transform the code from |Class.method|to |method(class:type)|, *before compilation*essentially allowing the same optimizations. For the other arguments could utilize the existing type annotations. The idea would be to allow this via decorators etc for code where it makes sense (e.g. high volume number crunching), while keeping the rest of the language free from JIT compiler issues and other nuisances. Very long version: Julia is a high level dynamically typed language, that often approaches or matches C performance. This is due to type-based multiple dispatch and type-stability through specialization. However, the coolest part is that the type system is set-theoretic, which makes it unnecessary to repeat functions for related types. This page <https://ucidatascienceinitiative.github.io/IntroToJulia/Html/WhyJulia>and this blog post <http://www.stochasticlifestyle.com/like-julia-scales-productive-insights-ju…>give a good explanations of how this works. *However*, since with their multiple dispatch system everything can be written without using classes, they have opted to not include any OOP. This leads to a very high amount of functions that are accessible in the global namespace and hinders one to explore the packages in the usual, playful way that probably all of us are used to. So the usual resort is to have to read documentation. What a shame ;) Since all of Julia needs to be JIT compiled, importing packages usually takes a lot of time (it does some pre-compilation on first import, which takes a lot longer, but depending on the system its still quite a while). A quick search for "time-to-first-plot" or "latency" will be plenty to show that this is quite a nuisance. I believe that its perfectly fine for some code to be interpreted, as the python community has frequently shown, so that might not be the best approach. Only performance-critical code needs to be fast. In our world this is usually done by offloading the quick stuff into C, but in that case it gets really hard to expand those libraries (e.g. numpy has its own dispatch system, so pypy cannot help it as much as it could). Also extending numerical types for arrays is basically impossible. So how can we try to retain usability and still gain the performance for the code that needs it? In general I'd argue that multiple dispatch and JIT compilation is fine, as our performance critical code usually runs for very long times, so the added overhead is minimal. If the code doesn't take a large amount of time, you probably don't need to optimize it anyway. I'd argue that one way to get there would be to "transpile" our Python code by inserting a step before compilation that utilizes the type annotations that already exist and then pulls typed versions out of the classes. So e.g. |Class.method(x:int)|would be transformed into the typed version |method(class:type, x:int)|, from where on multiple dispatch could be run. This would have the further advantage, that in Julia it can be very hard to see, which attributes a type should have and which methods necessarily have to be defined in order to guarantee library compatibility. In OOP, we can easily achieve that by defining the type as an abstract base class. A lot of the statements about Julia here come from Jeff Bezansons (one of the Julia main devs PhD thesis) <https://dspace.mit.edu/handle/1721.1/99811>. It's a quick read and I would in general recommend reading it. On the one hand, in general I think that julia is more compiler-friendly than it is user or tooling-friendly (since methods for your type can be defined outside of the scope of the class, its very hard for your IDE to tell you if that function you just misspelled doesn't exist). On the other hand, its quite nice to be able to create a custom array and just push it through all the numerical libraries in a very efficient way, which works because you have defined all the required methods. I think this way of speeding up things would be cooler than enabling static typing which many people are asking for, as we still retain the flexibility (or might even increase it in the case of arrays), while greatly speeding up code. Since I only understand all these things at a high level, I'm sure that I've greatly simplified or misrepresented something. The real hope here is that someone smarter than me is inspired by that idea and manages to build something cool. I'd by happy to help in any way I can though. Thanks for reading this far, you are a good person :* Cheers, Marvin

4 3

dataclasses: position-only and keyword-only fields
by Eric V. Smith March 14, 2021

March 14, 2021

[I'm sort of loose with the terms field, parameter, and argument here. Forgive me: I think it's still understandable. Also I'm not specifying types here, I'm using Any everywhere. Use your imagination and substitute real types if it helps you.] There have been many requests to add keyword-only fields to dataclasses. These fields would result in __init__ parameters that are keyword-only. As long as I'm doing this, I'd like to add positional-only fields as well. Basically, I want to add a flag, to each field, stating whether the field results in a normal parameter, a positional-only parameter, or a keyword-only parameter to __init__. Then when I'm generating __init__, I'll examine those flags and put the positional-only ones first, followed by the normal ones, followed by the keyword-only ones. The trick becomes: how do you specify what type of parameter each field represents? First, here's what attrs does. There's a parameter to their attr.ib() function (the moral equivalent of dataclasses.field()) named kw_only, which if set, marks the field as being keyword-only. From https://www.attrs.org/en/stable/examples.html#keyword-only-attributes : >>> @attr.s ... class A: ... a = attr.ib(kw_only=True) >>> A() Traceback (most recent call last): ... TypeError: A() missing 1 required keyword-only argument: 'a' >>> A(a=1) A(a=1) There's also a parameter to attr.s, also named kw_only, which if true marks every field as being keyword-only: >>> @attr.s(kw_only=True) ... class A: ... a = attr.ib() ... b = attr.ib() >>> A(1, 2) Traceback (most recent call last): ... TypeError: __init__() takes 1 positional argument but 3 were given >>> A(a=1, b=2) A(a=1, b=2) In dataclasses, these example become: >>> @dataclasses.dataclass ... class A: ... a: Any = field(kw_only=True) >>> @dataclasses.dataclass(kw_only=True) ... class A: ... a: Any ... b: Any Aside from the name 'kw_only', which we can bikeshed about, I think these features are good, and I'd like to implement them as shown here. But, I'd like to do two other things: make it easier to use, and support positional-only fields. Since the second one is easier, let's tackle it first. I'd do the same thing as kw_only, but name it something like pos_only. Again, we can argue about the name. Like kw_only, you can either specify individual fields as positional-only, or declare that every field is positional-only. It would be an error to specify both kw_only and pos_only. As far as making it simpler: I dislike needing to use field(kw_only=True), although it would certainly work. The problem is that if you have 1 normal parameter, and 10 keyword-only ones, you'd be forced to say: @dataclasses.dataclass class A: a: Any b: Any = field(kw_only=True, default=0) c: Any = field(kw_only=True, default='foo') e: Any = field(kw_only=True, default=0.0) f: Any = field(kw_only=True) g: Any = field(kw_only=True, default=()) h: Any = field(kw_only=True, default='bar') i: Any = field(kw_only=True, default=3+4j) j: Any = field(kw_only=True, default=10) k: Any = field(kw_only=True) That's way too verbose for me. Ideally, I'd like something like this example: @dataclasses.dataclass class A: a: Any # pragma: KW_ONLY b: Any # pragma: POS_ONLY c: Any And then b would become a keyword-only field and c would be positional-only. But we need some way of telling dataclasses.dataclass what's going on, since obviously pragmas are out. I propose the following. I'll add 2 (or 3, keep reading) singletons to the dataclasses module: KW_ONLY and POS_ONLY. When scanning the __attribute__'s that define fields, fields with these types would be ignored, except for assigning the kw_only/pos_only/normal flag to fields declared after these singletons are used. So you'd get: @dataclasses.dataclass class A: a: Any _: dataclasses.KW_ONLY b: Any __: dataclasses.POS_ONLY c: Any This would generate: def __init__(self, c, /, a, *, b): The names of the KW_ONLY and POS_ONLY fields don't matter, since they're discarded. But as you see above, they still need to be unique. I think _ is a fine name, and since KW_ONLY will be used much more than POS_ONLY, '_: dataclasses.KW_ONLY' would be the pythonic way of saying "the following fields are keyword-only". I do think I'll add a third singleton to specify that subsequent fields are "normal" fields, neither keyword-only or positional-only. I don't know that we have a name for such a thing, let's call it NORMAL_ARG here and bikeshed it later. Then you could say: @dataclasses.dataclass class A: a: Any _: dataclasses.KW_ONLY b: Any __: dataclasses.POS_ONLY c: Any ___: dataclasses.NORMAL_ARG d: Any Then a and d are "normal" fields, while b is keyword-only and c is positional-only. This would generate: def __init__(self, c, /, a, d, *, b): I normally wouldn't propose adding NORMAL_ARG, but since the order of fields matters (for repr, comparisons, etc.) I figure it might be desirable to have their order there be different from the order they're declared. I could be talked out of NORMAL_ARG and they'd just always have to go first (although you could play games with inheritance to change that). My "complex" example above would become: @dataclasses.dataclass class A: a: Any _: dataclasses.KW_ONLY b: Any = 0 c: Any = 'foo' e: Any = 0.0 f: Any g: Any = () h: Any = 'bar' i: Any = 3+4j j: Any = 10 k: Any Which I think is a lot better. There are a few additional quirks involving inheritance, but the behavior would follow naturally from how dataclasses already does inheritance. I can address that later when I work on the docs for this. Remember, the only point of all of these hoops is to add a flag to each field saying what type of __init__ argument it becomes: positional-only, normal, or keyword-only. So, what do you think? Is this a horrible idea? Should it be a PEP, or just a 'simple' feature addition to dataclasses? I'm worried that if I have to do a full blown PEP I won't get to this for 3.10. I should mention another idea that showed up on python-ideas, at https://mail.python.org/archives/list/python-ideas@python.org/message/WBL4X… . It would allow you to specify the flag via code like: @dataclasses.dataclass class Parent: with dataclasses.positional(): a: int c: bool = False with dataclasses.keyword(): e: list I'm not crazy about it, and it looks like it would require stack inspection to get it to work, but I mention it here for completeness. One last thought: mypy and other type checkers would need to be taught about all of this. -- Eric

9 19

Implicit line continuation for method chaining
by Matt Williams March 13, 2021

March 13, 2021

Hi, It's becoming more popular in Python to have interfaces which are built around method chaining as a way of applying operations to data. For example in pandas is moving more towards a default model where methods do not change the object in-place but instead return an altered version (or at least an altered view) of it. The major downside to method chaining is that it ends up creating long lines of code which can become hard to read. The two main solutions to this are 1) break down the chain and give the steps their own variable names and 2) split it over multiple lines using `\` as a line continuation. e.g.: y = x.rstrip("\n").split(":")[0].lower() would become y = x.rstrip("\n") \ .split(":")[0] \ .lower() I find the continuation character visually distracting, easy to forget and does not allow for line-by-line commenting (causing `SyntaxError: unexpected character after line continuation character`): y = x.rstrip("\n") \ .split(":")[0] \ # grab the key name .lower() My idea is to alter the syntax of Python to allow doing: y = x.rstrip("\n") .split(":")[0] .lower() i.e., not requiring an explicit line continuation character in the case where the next line starts with a period. I've had a search through the archives and I couldn't see this discussion before. Feel free to point me to it if I've missed anything. Cheers, Matt

10 13

dataclasses to have a way to make hashable fields immutable
by Tushar Sadhwani March 12, 2021

March 12, 2021

currently in a `dataclasses.dataclass` based class, you can either have it hashable and completely immutable (using frozen=True and eq=True), or you can have it hashable but completely mutable (by using unsafe_hash=True) unsafe_hash provides the convenience of being able to mutate some fields, while computing your hash by other, non-mutable fields. But there is nothing enforcing the fact that the fields marked with `hash=True` should stay immutable, otherwise it completely breaks hashability. The suggestion is, for the dataclass to throw an error if you try to mutate a field that contributes to the hash of that dataclass. Or, to have a flag that allows you to do so. Suggestions for the flag name could be @dataclass(freeze_hashable_fields=True) class A: ... or @dataclass(frozen_hash=True) class A: ...

1 0

Dataclasses, keyword args, and inheritance
by George Leslie-Waksman March 12, 2021

March 12, 2021

The proposed implementation of dataclasses prevents defining fields with defaults before fields without defaults. This can create limitations on logical grouping of fields and on inheritance. Take, for example, the case: @dataclass class Foo: some_default: dict = field(default_factory=dict) @dataclass class Bar(Foo): other_field: int this results in the error: 5 @dataclass ----> 6 class Bar(Foo): 7 other_field: int 8 ~/.pyenv/versions/3.6.2/envs/clover_pipeline/lib/python3.6/site-packages/dataclasses.py in dataclass(_cls, init, repr, eq, order, hash, frozen) 751 752 # We're called as @dataclass, with a class. --> 753 return wrap(_cls) 754 755 ~/.pyenv/versions/3.6.2/envs/clover_pipeline/lib/python3.6/site-packages/dataclasses.py in wrap(cls) 743 744 def wrap(cls): --> 745 return _process_class(cls, repr, eq, order, hash, init, frozen) 746 747 # See if we're being called as @dataclass or @dataclass(). ~/.pyenv/versions/3.6.2/envs/clover_pipeline/lib/python3.6/site-packages/dataclasses.py in _process_class(cls, repr, eq, order, hash, init, frozen) 675 # in __init__. Use "self" if possible. 676 '__dataclass_self__' if 'self' in fields --> 677 else 'self', 678 )) 679 if repr: ~/.pyenv/versions/3.6.2/envs/clover_pipeline/lib/python3.6/site-packages/dataclasses.py in _init_fn(fields, frozen, has_post_init, self_name) 422 seen_default = True 423 elif seen_default: --> 424 raise TypeError(f'non-default argument {f.name!r} ' 425 'follows default argument') 426 TypeError: non-default argument 'other_field' follows default argument I understand that this is a limitation of positional arguments because the effective __init__ signature is: def __init__(self, some_default: dict = <something>, other_field: int): However, keyword only arguments allow an entirely reasonable solution to this problem: def __init__(self, *, some_default: dict = <something>, other_field: int): And have the added benefit of making the fields in the __init__ call entirely explicit. So, I propose the addition of a keyword_only flag to the @dataclass decorator that renders the __init__ method using keyword only arguments: @dataclass(keyword_only=True) class Bar(Foo): other_field: int --George Leslie-Waksman

12 35

Add lazy import statement to protect legitimate circular imports from failing
by Matt del Valle March 11, 2021

March 11, 2021

I have seen a lot of interest in this topic, and I think the sheer number of different lazy import implementations (generally using proxy objects for modules and then loading them when an attribute is accessed) should present pretty good evidence that this is a very desired feature, particularly for writers of library code seeking to optimize start-up times and similar. Unfortunately, all of these implementations have serious limitations in that often static code analysis tools cannot understand them, resulting in autocompletions and useful diagnostic errors being lost. Also, they are fundamentally incapable of dealing with 'from x import y' syntax, because this is *always *loaded eagerly by the Python interpreter. With the increasingly widespread adoption of type hints I think it is time to revive the proposal for lazy imports as an official language feature. It is very common to type hint the return type or argument type of a function as a type which cannot be imported at the top of the file inside the module namespace because it would trigger a circular import, and must therefore be imported inside the function. In these instances it is currently recommended to use the 'TYPE_CHECKING' value in the 'typing' module which is always 'False' at runtime, in order to be able to provide the return or argument type hint: some_module.py > from typing import TYPE_CHECKING > > if TYPE_CHECKING: > from module_that_depends_on_me import MyClass > > > def some_func() -> MyClass: > from module_that_depends_on_me import MyClass > > return MyClass() However, this is unnecessarily verbose and confusing to most users who are not familiar with this usage pattern. IDEs will also be unable to warn you that MyClass is undefined if you try to use it inside 'some_func' without importing it within 'some_func' first, because it appears to be defined even though it isn't. It also is fundamentally incapable of handling this other extremely common usage pattern: some_library/ __init__.py > __all__ = ["SomeClass", "AnotherClass"] > > from .extremely_expensive_module import SomeClass > from .another_extremely_expensive_module import AnotherClass some_module.py > from some_library import SomeClass Where a library author wants to make several classes in their library's public API available at package level in the __init__.py, but doesn't want to eagerly load all of them when many users will only ever use one in a given program. In this situation if I try to import SomeClass from some_library/__init__.py I will *always* trigger AnotherClass to be loaded as well, totally unnecessarily. My proposed syntax is pretty much the same as was suggested by Nicolas Cellier in 2017 here: https://mail.python.org/pipermail/python-ideas/2017-February/044894.html, but with the additional stipulation that 'from x import y' format imports should *also* be supported. The two new lazy import statements should therefore be: > lazy import some_module > from some_module lazy import SomeClass For the two examples given above the equivalent code should be. some_module.py > from module_that_depends_on_me lazy import MyClass # does not raise ImportError when module_that_depends_on_me tries to import me, because this import is deferred > > > def some_func() -> MyClass: > return MyClass() # MyClass is not imported until this line, in a function that isn't called until after module level imports have already been resolved some_library /__init__.py > __all__ = ["SomeClass", "AnotherClass"] > > from .extremely_expensive_module lazy import SomeClass # this import is deferred > from .another_extremely_expensive_module lazy import AnotherClass # this import is deferred some_module.py > from some_library import AnotherClass # this causes SomeClass to be imported from some_library/__init__.py which in turn causes the import statement on line 4 of some_library/__init__.py to finally happen for real. The import of AnotherClass on line 3 *still* hasn't actually happened, and may never happen if AnotherClass isn't imported (or lazy imported and then used) during the lifetime of this program. These are just some examples that I could think of off the top of my head. I am certain there are more. This would be insanely useful. I really hope this sparks some discussion :)

3 8

Arrow functions polyfill
by Paul Sokolovsky March 11, 2021

March 11, 2021

Hello, On Fri, 12 Feb 2021 18:26:53 +1100 Chris Angelico <rosuav(a)gmail.com> wrote: > On Fri, Feb 12, 2021 Paul Sokolovsky <pmiscml(a)gmail.com> wrote: > > ... And on the 2nd thought, that won't work. The reason it works in > > JS is that it doesn't have tuples. In Python, "(a, b) => (1, 2)" > > means "compare a tuple for greater-or-equal". > > Should be safe actually - "=>" is not a valid comparison operator. To punish myself for making such stupid mistakes, I volunteered to implement PoC of that. And by long honorable tradition, improvements to Python get implemented in Python first. So with my "imphook" thingy https://pypi.org/project/imphook/, and with the hook module at the end, following works as expected: =========== $ cat example_arrow_func.py f = (a, b) => a + b print(f(1, 2)) res = ((a, b) => a + b)(3, 4) print(res) print(list(map((x) => x * 2, [1, 2, 3, 4]))) # Confirm there's no crashing on bare tuple at the tail of file. (1, 2) $ python3 -m imphook -i mod_arrow_func -m example_arrow_func 3 7 [2, 4, 6, 8] =========== The implementation was written a bit cowboyishly in 15 mins, maybe, just maybe, you can still crash it. (For example, it clearly doesn't support newlines in arrow param list): ===== mod_arrow_func.py ====== import sys import tokenize import imphook class TokBuf: def __init__(self): self.tokens = [] def append(self, t): self.tokens.append(t) def clear(self): self.tokens.clear() def empty(self): return not self.tokens def spool(self): yield from self.tokens self.clear() def xform(token_stream): tokbuf = TokBuf() for t in token_stream: if t[1] == "(": # We're interested only in the deepest parens. if not tokbuf.empty(): yield from tokbuf.spool() tokbuf.append(t) elif t[1] == ")": nt1 = next(token_stream) nt2 = next(token_stream) if nt1[1] == "=" and nt2[1] == ">": yield (tokenize.NAME, "lambda") yield from tokbuf.tokens[1:] tokbuf.clear() yield (tokenize.OP, ":") else: yield from tokbuf.tokens tokbuf.clear() yield t yield nt1 yield nt2 elif not tokbuf.empty(): tokbuf.append(t) else: yield t def hook(modname, filename): with open(filename, "rb") as f: # Fairly speaking, tokenizing just to convert back to string form # isn't too efficient, but CPython doesn't offer us a way to parse # token stream so far, so we have no choice. source = tokenize.untokenize(xform(tokenize.tokenize(f.readline))) mod = type(imphook)(modname) exec(source, vars(mod)) return mod imphook.add_import_hook(hook, (".py",)) =========== -- Best regards, Paul mailto:pmiscml@gmail.com

23 62

Basic toolkit for async iterators
by Sam Frances March 11, 2021

March 11, 2021

Apologies if this has already been raised at some point, but I wanted to raise the idea of a basic standard library toolkit for operations on async iterators. This would include versions of functions that are already in the standard library for non-async iterators, such as *map*, *filter*,*zip *etc. It could also include a *merge* function for async iterators, for example (this would yield items in whatever order they are yielded from from the constituent iterators). There are, of course, open source libraries providing these functions. But in general there is far too wide a range of such libraries, and it is difficult to tell which ones are best maintained. For the most basic and universally useful of operations on async iterators, it would be far better if they were part of the standard library.

2 1