Mailman 3 July 2022 - Python-ideas

PEP 671 (late-bound arg defaults), next round of discussion!
by Chris Angelico 23 Jul '23

23 Jul '23

I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/ with some additional information about the reference implementation, and some clarifications elsewhere. *PEP 671: Syntax for late-bound function argument defaults* Questions, for you all: 1) If this feature existed in Python 3.11 exactly as described, would you use it? 2) Independently: Is the syntactic distinction between "=" and "=>" a cognitive burden? (It's absolutely valid to say "yes" and "yes", and feel free to say which of those pulls is the stronger one.) 3) If "yes" to question 1, would you use it for any/all of (a) mutable defaults, (b) referencing things that might have changed, (c) referencing other arguments, (d) something else? 4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.) 5) Do you know how to compile CPython from source, and would you be willing to try this out? Please? :) I'd love to hear, also, from anyone's friends/family who know a bit of Python but haven't been involved in this discussion. If late-bound defaults "just make sense" to people, that would be highly informative. Any and all comments welcomed. I mean, this is python-ideas after all... bikeshedding is what we do best! The reference implementation currently has some test failures, which I'm looking into. I'm probably going to make this my personal default Python interpreter for a while, to see how things go. ChrisA

43 364

zip(x, y, z, strict=True)
by Ram Rachum 01 Dec '22

01 Dec '22

Here's something that would have saved me some debugging yesterday: >>> zipped = zip(x, y, z, strict=True) I suggest that `strict=True` would ensure that all the iterables have been exhausted, raising an exception otherwise. This is useful in cases where you're assuming that the iterables all have the same lengths. When your assumption is wrong, you currently just get a shorter result, and it could take you a while to figure out why it's happening. What do you think?

27 179

Re: Void type
by Михаил Крупенков 27 Jul '22

27 Jul '22

Yes, I want that when Void is received in a function parameter it is not processed: func(Void) == func() == "default"

9 19

Void type
by Крупенков Михаил 24 Jul '22

24 Jul '22

Hello, I'll give an example: def func(param="default"): ... def func_wrapper(..., param="default"): ... func(param) We have to duplicate value of parameter "default" if we want to use this as the default. Or we can do that: def func(param="default"): ... def func_wrapper(..., *args): ... func(*args) But it won't always be possible. In addition, auto-completion in IDEs won't work. It would be cool to have an empty type like Void (not None) for explicit full control: def func(param="default"): ... def func_wrapper(..., param=Void): ... func(param)

2 1

Use 'bin' in virtual environments on Windows
by Svein Seldal 24 Jul '22

24 Jul '22

Using py in the three OS-es (unix-like, Mac and Win) has become very similar in the last years, which is great. Making portable py code has become very easy. However, there is one peculiarity in Win which I find annoying: virtual environments insists on placing executables in the `Scripts` directory, while the other platforms use `bin`. This forces portable scripts and programs on the outside of py to be OS-aware in order to interact with a venv on a Win-system vs other systems. I would like to proposing the ability to configure virtual environments to use `bin` directory for executable on Windows. Personally I think it would be smart move if it were changed to `bin` as default, making all py platform act consistently. However, I do understand that there might be existing code out there that depend on using `Scripts`. Best regards, Svein

7 14

Make dataclass aware that it might be used with Enum
by Steve Jorgensen 21 Jul '22

21 Jul '22

Perhaps, this has already been addressed in a newer release (?) but in Python 3.9, making `@dataclass` work with `Enum` is a bit awkward. Currently, it order to make it work, I have to: 1. Pass `init=False` to `@dataclass` and hand-write the `__init__` method 2. Pass `repr=False` to `@dataclass` and use `Enum`'s representation or write a custom __repr__ Example: In [72]: @dataclass(frozen=True, init=False, repr=False) ...: class Creature(Enum): ...: legs: int ...: size: str ...: Beetle = (6, 'small') ...: Dog = (4, 'medium') ...: def __init__(self, legs, size): ...: self.legs = legs ...: self.size = size ...: In [73]: Creature.Dog Out[73]: <Creature.Dog: (4, 'medium')>

6 19

Dataclasses for complex models, A proposal for datatrees,
by gianni＠mariani.ws 15 Jul '22

15 Jul '22

I'm new to this forum, any advice for a newbie is welcome. I have found datatrees particularly useful in generating hierarchical 3D models of shape classes with a large number of parameters. Copious parameters is evidently inevitable with 3D modelling. I've had a few people suggest pulling it out as a separate library so hence the proposal. Note the `anchorscad.datatrees` module is fully functional, it could use some cleanups but I'd rather discuss the merits of such a proposal to augment dataclasses and then maybe suggestions for next steps depending on what people think. # Datatrees Proposal Building complex hierarchical data objects using [`dataclasses`](https://docs.python.org/library/dataclasses.html) reduces much of the needed boilerplate code. This obviously being the point of [`dataclasses`](https://docs.python.org/library/dataclasses.html). While using it to develop [AnchorSCAD](https://github.com/owebeeone/anchorscad) I found it to be still a very verbose and repetitive when building complex 3D models. # Introducing datatrees [`datatrees`](https://github.com/owebeeone/anchorscad/blob/master/src/anchorscad/datatrees.py) extends (as a wrapper over `datatlasses.dataclass`) to include: * Field injection * Field binding * `self` factory default The [`datatrees`](https://github.com/owebeeone/anchorscad/blob/master/src/anchorscad/datatrees.py) link points to a working implementation including a dubious `override` feature that is seldom used and only in debugging situations. # Field injection and Field binding Often one desires building dataclass objects containing other dataclass objects from within the `__post_init__` function potentially with many instances of the same dataclass type. When creating [AnchorSCAD](https://github.com/owebeeone/anchorscad) models in particular it may be necessary to build many instances of a dataclass type. For example a 3D model of hole gauge, a plate with multiple holes with different sizes. The height (z) parameter may be shared with all the contained holes while the radius of each individual hole would be specific to that hole. Such a model can be found here ([`hole_guage`](https://github.com/owebeeone/anchorscad/blob/master/src/anchorscad/models/tools/hole_gauge.py)). # Datatree Wraps Dataclass ` @datatree class A: '''Demonstrates Python dataclasses default value functionality. Even though @datatree is used, this example only uses the base Python @dataclass functionality. ''' v1: int v2: int=2 v3: int=field(default=3) v4: int=field(default_factory=lambda: 7 - 3) ` Construct with v1 provided.. ` A(v1=1) -> A(v1=1, v2=2, v3=3, v4=4) ` Comparison of two different instances with the same value. A(v1=2) == A(v1=2) -> True ## Help for Class A ` class A(builtins.object) | A(v1: int, v2: int = 2, v3: int = 3, v4: int = <factory>, override: anchorscad.datatrees.Overrides = None) -> None | | Demonstrates Python dataclasses default value | functionality. Even though @datatree is used, this | example only uses the base Python @dataclass | functionality. | … ` ## Datatree Inject and Bind ` @datatree class Anode: '''Injects fields from class A and provides a default value for A.v1. The a_node field becomes a callable object that provides an instance of A with field values retrieved from self. ''' v1: int=55 a_node: Node=Node(A) # Inject field names from A ` ### Injected Fields Datatree uses the `a_node` field in `class Anode` above to inject `class A` constructor parameters as fields in the containing class (`class Anode`). Node allows specification of name mappings with mapping dictionaries, prefixes, suffixes or specific exclusion of specific parameters. In the `class Anode` example above, the field `v1` is specified so the field `v1` from `class A` will not be injected and the default parameters specified in `class Anode` will remain. However the remaining parameters (`v2`, `v3` and `v4`) will be injected with the default values specified in `class A`. Constructing a default `class Anode` will demonstrate the injected values, see parameters `v2`, `v3` and `v4` below: ``` Anode() -> Anode(v1=55, a_node=BoundNode(node=Node(clz_or_func=A, use_defaults=True, suffix='', prefix='', expose_all=True, node_doc=None)), v4=4, v2=2, v3=3) ``` Note how the `a_node` field is transformed into a `BoundNode`. This is a factory for A objects that will pull all constructor parameters from the `Anode` object that created it. The example below shows how the `a_node` constructs a ` class A` object pulling all parameters from the instance containing it. # Example of field binding, ``` Anode().a_node() -> A(v1=55, v2=2, v3=3, v4=4) ``` The `BoundNode` factory also allows all the parameters provided in the associated class (or function) from the corresponding Node. In this case, the `v3` parameter is overridden with the value `33`. # Example of overriding the field binding. ``` Anode().a_node(v3=33) -> A(v1=55, v2=2, v3=33, v4=4) ``` # Binding Parameters In `self default` Fields `dataclasses.field` provides field attributes `default` and `factory_default` parameters, the latter being evaluated at object initialization. datatree.dtfield wraps `field` additionally providing a `self_default` parameter that takes a function object with a single parameter. At class initialization, these functions are evaluated with the object instance as its value. The order of evaluation is the order in which they are declared in the class but after all the Node fields have been bound (transformed into BoundNode factories). This allows fields specified with a `self_default` attribute to use any field specified with `Node` as long as the BoundNodes do not attempt to access self_default fields that are not yet initialised. # Injecting Computed Defaults In the example below, `class Bind` defines the field `v1` as a computed sum `v2 + v3`. ``` @datatree class Bind: '''Demonstrates the use of a computed default value. Often a value used in nodes should be computed with other parameters provided to this instance.''' v1: int=dtfield(self_default=lambda s: s.v2 + s.v3) a_node: Node=field(default=Node(A), repr=False, init=False ``` The default value of `class Bind` is `Bind(v1=5, v2=2, v4=4, v3=3)`. Note the value of `v1` being the specified sum. This demonstrates the value of `v1` is performed with the values of `v2` and `v2` passed into the constructor. ``` Bind(v2=10) -> Bind(v1=13, v2=10, v4=4, v3=3) ``` # Injecting Function Parameters The example below shows that any function (or lambda) can be used as a `datatrees.Node`. In the example below, the fields `lamba_node` and `func_node` use different declaration of essentially the same function but the parameter names from `func_a` are also injected. ``` def func_a(x: int=1, y:int=3): return x + y @datatree class BindFunc: '''Injected function parameters.''' a: int=1 b: int=1 lambda_node: Node=Node(lambda a, b: a + b) func_node: Node=Node(func_a) ``` The lambda_node will bind the values `a` and `b` and pass it to the lambda in the in the class declaration. ``` BindFunc().lambda_node() -> 2 ``` This demonstrates the default value for `x` being overridden and the default value for `y` being used in the call to `func_a`. ``` BindFunc(x=5).func_node() -> 8 ``` # Inheritance with Datatrees ``` @datatree class E: '''Using the dtfield() function to create a BindingDefault entry.''' v1: int=1 v2: int=2 v_computed: Node=dtfield(self_default=lambda s: s.v1 + s.v2) @datatree class F(E, A): '''Inheritance of datatree classes is allowed, fields are merged. in the same way as dataclasses.''' ``` ``` F() -> F(v1=1, v2=2, v3=3, v4=4, v_computed=3) F(v4=44) -> F(v1=1, v2=2, v3=3, v4=44, v_computed=3) ``` # Mapping Injected Names with a Prefix or Suffix It is often important to inject the same or similar class with identical field names. `datatree.Node` provides for `prefix=` and `suffix=` specifiers that can be used to map injected parameter names. ``` @datatree class C: '''Multiple nodes of the same type with parameter name mapping.''' a_v1: int=11 a_node: Node=field(default=Node(A, prefix='a_'), repr=False) b_v1: int=12 b_node: Node=field(default=Node(A, prefix='b_'), repr=False) computed: int=BindingDefault(lambda s: s.a_v2 + s.b_v2) def __post_init__(self): pass # Called after initialization is complete. def make_stuff(self): return self.a_node(v2=22), self.b_node(), self.computed ``` Note the call below to `make_stuff()` that returns a tuple of the result of invoking `a_node()` and `b_node()` from a default `class C` instance. The fields `a_v1` and `b_v1` are injected into calls for `a_node()` and `b_node()` respectively and hence the difference in the instances of the instances of `class A` being created. ``` C().make_stuff() -> (A(v1=11, v2=22, v3=3, v4=4), A(v1=12, v2=2, v3=3, v4=4), 4) ``` # Override As mentioned earlier, the `override` parameter should only be used for debugging purposes. This uses the `datatrees.override` function to generate a `datatrees.Override` instance. The parameters to `datatrees.override` are simply parameters to override that will replace any provided value including explicit values. Since the override parameter is passed down, it also is able to hierarchically provide override values for contained bound nodes. ``` @datatree class O: '''Deep tree of injected fields.''' c_node: Node=dtfield(Node(C, 'a_v1'), init=False, repr=False ``` Note that `class O` has only one injected parameter, `a_v1`. The default constructor demonstrates the only provided parameter. The default of `a_v1` is from `class C`’s default value for `a_v1`, ``` O() -> O(a_v1=11) ``` This demonstrates the node generating an instance of `class C`. ``` O().c_node() -> C(a_v1=11, a_v2=2, a_v4=4, a_v3=3, b_v1=12, b_v2=2, b_v4=4, b_v3=3, computed=4) ``` This demonstrates using the override parameter with an `Override` specifier that overrides multiple layers within the `class O` datatree class. Node how the `b_node` override specifier for `v2=909090` is reflected in the resulting value of `A()` while `b_node` is called with a value for `v2` that is ignored. ``` O( override=override( c_node=dtargs( override=override( b_node=dtargs(v2=909090) )))).c_node().b_node(v2=55555) -> A(v1=12, v2=909090, v3=3, v4=4) ``` Overriding values in this way can cause code to be fragile as disregarding parameter values with those provided by some unrelated function will inevitably cause unexpected and silent issues to appear which are difficult to diagnose. This is why the `override` parameter should only be used for debugging purposes and not otherwise used. # Hole Gauge Example The `hole_guage.py` code below (found in [`hole_guage`](https://github.com/owebeeone/anchorscad/blob/master/src/anchorscad/models/tools/hole_gauge.py)) demonstrates how two `anchorscad.CompositeShape` 3D models can be used where one shape class instance consists of multiple instances of another 3D model and the parameters can be shared. It also demonstrates how `self_default` bindings can be used in conjunction with `BoundNode` factories. To demonstrate the doc field attribute, the following is the `pydoc` generated for `HoleGuage`. Note the fields `fn`, fa` and `fs` are used in AnchorSCAD and it’s desirable to pass these attributes to constructors of contained classes as these are used by the rendering engine to determine the complexity of the polyhedrons approximations of curved surfaces. This is a feature of the `anchorscad.ShapeNode` subclass of `datatrees.Node`. ``` class HoleGauge(anchorscad.core.CompositeShape) | HoleGauge(hole_rss: Tuple[Tuple[float]], sep: float = 5, fn: int = None, epsilon: float = 0.005, fa: float = None, h: float = 5, fs: float = None, override: anchorscad.datatrees.Overrides = None) -> None | | A plate with a matrix of holes of different radii provided in hole_rss. | Args: | hole_rss: Tuple of tuple of hole radii | sep: None: Margin of separation between holes and edges | fn: None: None: fixed number of segments. Overrides fa and fs | epsilon: None: Fudge factor to remove aliasing | fa: None: None: minimum angle (in degrees) of each segment | h: None: Depth of plate | fs: None: None: minimum length of each segment | Other args: | override … ``` ### From: [`hole_guage`](https://github.com/owebeeone/anchorscad/blob/master/src/anchorscad/models/tools/hole_gauge.py) The code below is a 3D model using AnchorSCAD that generates a plate of holes. The `SingleHoleGauge` class generates a single row of holes with radii specified by the hole_rs parameter and the thickness of the plate specified by the `h` parameter. The `HoleGauge` class is similar but generates a grid of holes specified by a tuple of tuples of hole radii via parameter `hole_rss`. `HoleGauge` generates a number of `SingleHoleGauge` shapes and stitches them together but also creates an overall plate sized to contain all the `SingleHoleGauge` shapes in a single ‘assembly’ plate. ``` import anchorscad as ad from typing import Tuple @ad.shape @ad.datatree(frozen=True) class SingleHoleGauge(ad.CompositeShape): '''A plate with a line of holes of different radii provided in hole_rs.''' hole_rs: Tuple[float]=ad.dtfield(doc='Tuple of hole radii') h: float=ad.dtfield(5, 'Depth of plate') sep: float=ad.dtfield(5, 'Margin of separation between holes and edges') x: float=ad.dtfield(doc='Width (x) of plate', self_default=lambda s: sum(s.hole_rs) * 2 + (len(s.hole_rs) + 1) * s.sep) y: float=ad.dtfield(doc='Depth (y) of plate', self_default=lambda s: max(s.hole_rs) * 2 + 2 * s.sep) plate_size: Tuple[float]=ad.dtfield(doc='The (x, y, z) size of the plate Box shape', self_default=lambda s: (s.x, s.y, s.h - 2 * s.epsilon)) plate_node: ad.Node=ad.dtfield(ad.ShapeNode(ad.Box, prefix='plate_'), init=False) hole_node: ad.Node=ad.dtfield(ad.ShapeNode(ad.Cylinder, 'h'), init=False) epsilon: float=ad.dtfield(0.005, 'Fudge factor to remove aliasing') EXAMPLE_SHAPE_ARGS=ad.args(fn=64, hole_rs=(3, 4, 5, 6, 10)) EXAMPLE_ANCHORS=() def build(self) -> ad.Maker: maker = self.plate_node().solid('plate').at('centre') offset = self.sep for i, r in enumerate(self.hole_rs): hole = self.hole_node(r=r) offset += r maker.add_at( hole.hole(('hole', i)) .at('base', post=ad.translate((0, -offset, -self.epsilon))), 'face_edge', 'base', 1) offset += r + self.sep return maker @ad.shape @ad.datatree(frozen=True) class HoleGauge(ad.CompositeShape): '''A plate with a matrix of holes of different radii provided in hole_rss.''' hole_rss: Tuple[Tuple[float]]=ad.dtfield(doc='Tuple of tuple of hole radii') single_hole_gauge: ad.Node=ad.dtfield( ad.ShapeNode(SingleHoleGauge, exclude=('hole_rs', 'x', 'y', 'plate_size')), init=False) shapes: Tuple[ad.Shape]=ad.dtfield( doc='Tuple of shapes placed in the hole gauge plate', self_default=lambda s: tuple(s.single_hole_gauge(hole_rs=rs) for rs in s.hole_rss), init=False) plate_size: Tuple[float]=ad.dtfield( doc='The (x, y, z) size of the plate Box shape', self_default=lambda s:( max(sh.x for sh in s.shapes), sum(sh.y for sh in s.shapes) - (len(s.shapes) - 1) * s.sep, s.h - 2 * s.epsilon), init=False) plate_node: ad.Node=ad.dtfield(ad.ShapeNode(ad.Box, prefix='plate_'), doc='The plate node factory', init=False) EXAMPLE_SHAPE_ARGS=ad.args( fn=64, hole_rss=((3, 4, 5, 6, 8, 10), (16, 14, 12))) def build(self) -> ad.Maker: # Create a builder plate the size of the entire assembly. maker = self.plate_node().solid('plate').at('centre') # Adds all shapes into the assembly plate. offset = 0 for i, shape in enumerate(self.shapes): maker.add_at(shape.composite(('inner_plate', i)) .at('face_edge', 'base', 0, post=ad.tranY(-offset)), 'face_edge', 'base', 0) offset += shape.y - self.sep return maker MAIN_DEFAULT=ad.ModuleDefault(True) # Default to --write if __name__ == "__main__": ad.anchorscad_main() ```

3 2

Confusing naming of Optional type should be changed
by nverich07＠gmail.com 13 Jul '22

13 Jul '22

(I apologize in advance if I've posted anything incorrectly before, I believe I might have sent this in python-dev instead but not sure as it's not appearing in the posts for my account). I am aware this is clarified in the Python documentation for the typing module but I and others have thought the naming of Optional is still quite confusing by itself. "Note that this is not the same concept as an optional argument, which is one that has a default. An optional argument with a default does not require the Optional qualifier on its type annotation just because it is optional." - typing.Optional docs Google defines optional as this, "available to be chosen but not obligatory." Pretend we have a function like this: def test_func(param: Optional[str]) -> None: ... The argument param is typed as Optional[str] meaning Union[str, None] or str | None. Optional here if we follow the definition above, basically means it can be str but not required or obligated to be that type. See the problem with the naming? This is a function where param can be None or str, not just it can be str but not obligated. Some interpretations may think optional means left to one's choice as well. The docs even have to clarify the use of Optional with typing because of this confusion. I believe this has been proposed before (not sure) but something like Nullable or Noneable (not sure about naming) would make much more sense in the context of typing. def test_func(param: Noneable[str]) -> None: ... It also would work and still make sense if there is a default value for the argument: def test_func(param: Noneable[str] = None) -> None: ... def test_func(param: Noneable[str] = "hello world") -> None: ...

5 9

dataclass field argument to allow converting value on init
by Dexter Hill 02 Jul '22

02 Jul '22

The idea is to have a `default_factory` like argument (either in the `field` function, or a new function entirely) that takes a function as an argument, and that function, with the value provided by `__init__`, is called and the return value is used as the value for the respective field. For example: ```py @dataclass class Foo: x: str = field(init_fn=chr) f = Foo(65) f.x # "A" ``` The `chr` function is called, given the value `65` and `x` is set to its return value of `"A"`. I understand that there is both `__init__` and `__post_init__` which can be used for this purpose, but sometimes it isn't ideal to override them. If you overrided `__init__`, and were using `__post_init__`, you would need to manually call it, and in my case, `__post_init__` is implemented on a base class, which all other classes inherit, and so overloading it would require re-implementing the logic from it (and that's ignoring the fact that you also need to type the field with `InitVar` to even have it passed to `__post_init__` in the first place). I've created a proof of concept, shown below: ```py def initfn(fn, default=None): class Inner: def __set_name__(_, owner_cls, owner_name): old_setattr = getattr(owner_cls, "__setattr__") def __setattr__(self, attr_name, value): if attr_name == owner_name: # Bypass `__setattr__` self.__dict__[attr_name] = fac(value) else: old_setattr(self, attr_name, value) setattr(owner_cls, "__setattr__", __setattr__) def fac(value): if isinstance(value, Inner): return default return fn(value) return field(default=Inner()) ``` It makes use of the fact that providing `default` as an argument to `field` means it checks the value for a `__set_name__` function, and calls it with the class and field name as arguments. Overriding `__setattr__` is just used to catch when a value is being assigned to a field, and if that field's name matches the name given to `__set_name__`, it calls the function on the value, at sets the field to that instead. It can be used like so: ```py @dataclass class Foo: x: str = initfn(fn=chr, default="Z") f = Foo(65) f2 = Foo() f.x # "A" f2.x # "Z" ``` It adds a little overhead, especially with having to override `__setattr__` however, I believe it would have very little overhead if directly implemented in the dataclass library. Even in the case of being able to override one of the init functions, I still think it would be nice to have as a quality of life feature as I feel calling a function is too simple to want to override the functions, if that makes sense. Thanks. Dexter

5 22

Typescript-Like Notational Shortcuts
by kevinlu1248＠gmail.com 01 Jul '22

01 Jul '22

On the theme of the addition of pip operators to indicate union, I was wondering what people think of using the following shortcuts for type hints. * type[] be a short-hand for List[type] * [type1, type2, type3] be a shorthand for Tuple[type1, type2, type3] * similar ideas for dictionaries and callables with (perhaps with javascript-like arrow notations?) I just think something like this would be easier to read. For example, a lot of functions that return tuples generally are immediately unpacked when it's used, so I think this is a more clear and concise way of stating that this is that it returns three elements: an element of type1, an element of type2 and an element of type3. In other words, the fact that it's a tuple does not matter much.

3 2