Mailman 3 May 2022 - Python-ideas

Fwd: Re: Heterogeneous numeric data in statistics library
by Christopher Barker 15 May '22

15 May '22

accidentally sent off list. -CHB ---------- Forwarded message --------- From: Christopher Barker <pythonchb(a)gmail.com> Date: Fri, May 13, 2022 at 8:14 PM Subject: Re: [Python-ideas] Re: Heterogeneous numeric data in statistics library To: Cameron Simpson <cs(a)cskk.id.au> On Thu, May 12, 2022 at 5:06 PM Cameron Simpson <cs(a)cskk.id.au > - a rule like the above requiring homogeneous data > - some convenience functions to produce homongenous data from mixed data I was going to suggest the same thing: type casting can have non-obvious effects, so explicit is better than implicit in this case. Another option (which would be a shorthand for the above) would be an optional parameter to specify the type you want used in the calculation. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

1 0

Heterogeneous numeric data in statistics library
by Steven D'Aprano 13 May '22

13 May '22

Users of the statistics module, how often do you use it with heterogeneous data (mixed numeric types)? Currently most of the functions try hard to honour homogeneous data, e.g. if your data is Decimal or Fraction, you will (usually) get Decimal or Fraction results: >>> statistics.variance([Decimal('0.5'), Decimal(2)/3, Decimal(5)/2]) Decimal('1.231481481481481481481481481') >>> statistics.variance([Fraction(1, 2), Fraction(2, 3), Fraction(5, 2)]) Fraction(133, 108) With mixed types, the functions usually try to coerce the values into a sensible common type, honouring subclasses: >>> class MyFloat(float): ... def __repr__(self): ... return "MyFloat(%s)" % super().__repr__() ... >>> statistics.mean([1.5, 2.25, MyFloat(1.0), 3.125, 1.75]) MyFloat(1.925) but that's harder than you might expect and the extra complexity causes some significant performance costs. And not all combinations are supported (Decimal is particularly difficult). If you are a user of statistics, how important to you is the ability to **mix** numeric types, in the same data set? Which combinations do you care about? Would you be satisfied with a rule that said that the statistics functions expect homogeneous data and that the result of calling the functions on mixed types is not guaranteed? -- Steve

6 5

Formatted structural patterns
by Matsuoka Takuo 13 May '22

13 May '22

I would like to propose a new type of pattern for structural pattern matching. A motivation will be described later below. The pattern may be written in a form like {<literal or variable>} (with braces), where <literal or variable> is either a literal pattern or an identifier (or "NAME" in the structural pattern matching terminology in the language reference, meaning whatever the name of a variable can be). At matching against a value, it should be evaluated to the series of characters held as the string format(<literal or variable>) returned (if computed successfully) by the built-in function ``format``. If those series of characters is a valid pattern of another type, then the pattern should be matched as the obtained pattern. Otherwise, an exception must be raised. Here are simple examples to clarify the idea. ``` subject = "x" pattern = "a" print(format(pattern)) # --> a match subject: case {pattern}: # interpreted as 'case a', so binds the subject # value to the variable "a" print(eval(format(pattern))) # --> x # For comparison, match subject: case pattern: # binds the subject value to the variable "pattern" print(pattern) # --> x # Some more examples. print(format(a)) # --> x match subject: case {a}: # interpreted as 'case x', so binds the subject value to # the variable "x" print(eval(format(a))) # --> x case {"a"}: # interpreted as 'case a' pass case "a": # matches a string equal to "a" pass case {'"a"'}: # interpreted as 'case "a"', so matches a string # equal to "a" pass ``` What if you format an object other than strings? Sometimes, a pattern of this type can be used like a value pattern. ``` from dataclasses import dataclass @dataclass class Point: x: int y: int point = Point(x=0, y=0) print(format(point)) # --> Point(x=0, y=0) match subject: case {point}: # interpreted as 'case Point(x=0, y=0)', and matches # a Point object at coordinates x=0, y=0 pass case point: # binds the subject value to the variable "point" pass ``` Motivation ---------- Consider what control you can have over a match statement using parameters. I'm considering the following context. Suppose you would like to define a function for which a match statement would be useful to make it do one of its steps. The definition of the function you write may look like ``` def f(*args): <some procedure> # Do a step with a match statement match <expression>: case <pattern 1> if <condition 1>: <procedure 1> case <pattern 2> if <condition 2>: <procedure 2> case ... ... ...... <another procedure> ``` Of the parts of the match statement, <expression>, <condition i> and <procedure i> will be expressions or series of statements, and can contain many parameters (variables) whose value may vary with arguments passed to the function. Through these parameters, what the match statement does can be controlled. I'm talking about such control. However, note the parts <pattern i> cannot contain parameters other than value patterns in them. This is rather restricted control. In order to have more control over patterns, one thing you might try may be to specify with an expression a string whose content will be the pattern you'd like to use, which can then be considered to be given parametrized by the value of variables contained in the expression, and do, e.g., ``` exec( f"""match subject: case {pattern}: pass """ ) ``` where the variable "pattern" in the braces holds the prametrized pattern as its value (or you could put the expression for the string directly between the braces). However, there is a problem with this approach that this done in the local scope doesn't do assignments, but only updates locals() directly, which means, if you want to use e.g., a captured value in some procedure later, then you would be able to get it not by simply writing the name of the variable for it but only by doing eval of that name, or by doing exec of the whole statement (or more) containing the variable, which would again only update locals(). This will be inconvenient since you can't necessarily just exec the whole code for your function since returning or yielding values can't be done with exec. Introduction of the proposed type of pattern would solve this problem since that would enable you to achieve what you want simply with ``` match subject: case {pattern}: pass ``` Best regards, Takuo Matsuoka

1 0

New Tool Proposal
by anthony.flury 11 May '22

11 May '22

A proposal for a new tool to be implemented - It is often the case that developer write Code in Python and then convert to a C extension module for performance regions. A C extension module has a lot of boiler plate code - for instance the Structures required for each class, the functions for Module initialization etc. My Idea is a simple tool that uses introspection tools to take a Python module and to generate the relevant boiler plate for the module - including blank functions for the module classes and for methods. This tool would use type annotations (if given) to make sensible choices for parameter and attribute types, including using int and float directly rather than Internal objects (depending on tool options). The idea is that the C extension module would present the same 'API' as the original Python module. The tool would not attempt to write the C code within the functions/methods. I am aware of the challenges writing this tool, but I think a tool could make a good fist of the conversion. --------------------- What do you think ? Would you use the tool if provided ? If there is interested I probably could develop a version 0.1, but it would be useful to know if there is interest and if there is anyone else interested in joining me in this journey. -- Anthony Flury *Moble*: +44 07743 282707 *Home*: +44 (0)1206 391294 *email*: anthony.flury(a)btinternet.com

5 5

Time to relax some restrictions on the walrus operator?
by Steven D'Aprano 10 May '22

10 May '22

I don't have an opinion one way or the other, but there is a discussion on Discourse about the walrus operator: https://discuss.python.org/t/walrus-fails-with/15606/1 Just a quick straw poll, how would people feel about relaxing the restriction on the walrus operator so that iterable unpacking is allowed? # Currently a syntax error. results = (1, 2, (a, b) := (3, 4), 5) which would create the following bindings: results = (1, 2, (3, 4), 5) a = 3 b = 4 A more complex example: expression = "uvwxyz" results = (1, 2, ([a, *b, c] := expression), 5) giving: results = (1, 2, "uvwxyz", 5) a = "u" b = ("v", "w", "x", "y") c = "z" Thoughts? -- Steve

9 19

Expression formed with structural pattern matching
by Matsuoka Takuo 09 May '22

09 May '22

I've been unable to find a discussion on the following subject. I think structural pattern matching could sometimes be more useful to be available to form an expression than to form a statement. To be specific, given the definition of a function of the form ``` def f(): match <expression>: case <pattern 1> if <condition 1>: return <expression 1> case <pattern 2> if <condition 2>: return <expression 2> case ... ...... ``` I imagine something like ``` (<expression 1>) if <condition 1> case <pattern 1>\ else <expression 2> if <condition 2> case <pattern 2>\ else ... ...... else <expression last> case _ match (<expression>) ``` as an expression such that it preceded by "lambda : " would specify an equivalent function. This of course doesn't fully characterize the expression, which could be evaluated not just in the scope of the function. To in terms of the two functions, they could be equivalent including how the local namespace is modified. (You might look at a few remarks on the syntax at the bottom.) What may it be good for? Such an expression is analogous to a conditional expression. ``` *** if ... else *** if ... else *** ``` An if-statement gives you a list of statements for each case, but a conditional expression gives an expression for each case. You'd more like the latter at some places (while more the former at some other places). The similar thing can be thought of about structural pattern matching. On the other hand, a match statement can be more preferable than an equivalent if-statement in some cases, so I think there can be something more preferable than a conditional expression in similar cases. Here's a picture of the idea. if-statement ======> conditional expression | | | | V V match statement ===> Do we want something here? I expect some expressions using structural pattern matching would be less repetitive to write and easier to read than equivalent conditional expressions. Comparison would be similar to that of match statements with if-statements. Remark ------ A larger theme may be a kind of expression associated to any kind of compound statement. Note the form of function like f above can be considered with any kind of compound statement. Namely, one can consider a function whose code is a compound statement of the considered kind with every suite a return statement, or perhaps better, of the form yield <expression>. One may then wish for an "expression version" of it like considred above for the case of match statement. Another example of existing such thing than conditional expression is comprehension, essentially corresponding to for-statement (where the dict comprehension is a slight extension of it). Similary, one may consider comprehension corresponding to a while-loop instead of a for-loop. for-loop comprehension can also be done with the functions map (and filter). One can consider a similar function map_while (or "apply_while"?) for while-comprehension. Another thing about comprehension is that the form of the model function f can very meaningfully be modified so the suite is to be of the form yield from <expression>. I've seen a proposal of syntax for the corresponding comprehension. An expression analogue of match statement seems to be a good place to start from (whether the larger theme may go anywhere). Remarks on the syntax --------------------- (1) "if <condition i>" can be omitted where no guard is necessary. (2) I expect the last case in the expression not being the wildcard pattern "_" without a guard would result in SyntaxError raised. (<expression last> can be None if the match statement in the original definition of the function doesn't have this case.) (3) Parentheses around <expression 1> and <expression> in the expression are in case those expressions are themselves conditional or similarly formed with structural pattern matching (let me call an expression of the form a match expression for short). Actual necessity of parentheses in these and other places (such as around a match expression within a conditional expression) depends on the order in which parts are to be placed in a match expression, as well as conventions such as whether priority be set between conditional and match expressions. Requiring some redundant parentheses any way might help with avoiding bugs and/or making parsing easier (for programs as well as for human beings). Conditional expression seems to be taking this approach. >>> True if False if False else False else False File "<stdin>", line 1 True if False if False else False else False ^ SyntaxError: invalid syntax Best regards, Takuo Matsuoka

1 0

Auto assignment of attributes
by Pablo Alcain 08 May '22

08 May '22

The problem of assigning init arguments as attributes has appeared several times in the past ( https://mail.python.org/archives/list/python-ideas@python.org/message/VLI3D… was the most recent we could find) and is already handled in dataclasses. Lately, discussing this topic with a friend, we thought that using a specific token could be a possible approach, so you could do: class MyClass: def __init__(self, @a, @b, c): pass and it would be analogous to doing: class MyClass: def __init__(self, a, b, c): self.a = a self.b = b Then, you would instantiate the class as usual, and the variables tagged with `@` would be bound to the object: >>> objekt = MyClass(2, 3, 4) >>> print(objekt.b) 3 >>> print(objekt.c) AttributeError: 'MyClass' object has no attribute 'c' We have a working implementation here if anyone wants to take a look at: https://github.com/pabloalcain/cpython/tree/feature/auto_attribute. Keep in mind that we have limited knowledge about how to modify cpython itself, and which would the best places be to do the modifications, so it's more than likely that some design decisions aren't very sound ( https://devguide.python.org/grammar/ and https://devguide.python.org/parser/ were incredibly helpful). Besides the implementation, we would like to know what the community thinks on whether this might have any value. While developing this, we realized that Crystal already has this feature (eg https://github.com/askn/crystal-by-example/blob/master/struct/struct.cr) with the same syntax; which is kind of expected, considering it's syntax is based on Ruby. Random collection of thoughts: 1. If auto-assignment made sense in general, one of the reasons we went for this rather than the decorator approach is that we wouldn't like to have a list of strings that can vary decoupled from the actual argument name. 2. The current implementation of `@` works for any function, not only init. We don't know if this would actually be a desirable feature. 3. It also works with any function in the wild. This mostly allows for monkey-patching to work out of the box: >>> class Klass: ... def __init__(self): ... pass ... >>> def add_parameter(k, @p): ... pass ... >>> Klass.add_parameter = add_parameter >>> objekt = Klass() >>> print(objekt.p) Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'Klass' object has no attribute 'p' >>> objekt.add_parameter(11) >>> print(objekt.p) 11 Again, we are not sure if this is desirable, but it's what made most sense for us at the moment. 4. Adding the `@` token to the argument doesn’t remove the variable from the function/method scope, so this would be perfectly valid: >>> def my_function(k, @parameter): ... print(parameter) >>> my_function(objekt, 4) 4 >>> k.parameter 4 5. We didn’t implement it for lambda functions. Cheers, Pablo and Quimey

12 50

Adding a .find() method to list
by python-ideas＠lucas.rs 08 May '22

08 May '22

In its current implementation, the list type does not provide a simple and straightforward way to retrieve one of its elements that fits a certain criteria. If you had to get the user where user['id'] == 2 from this list of users, for example, how would you do it? users = [ {'id': 1,'name': 'john'}, {'id': 2, 'name': 'anna'}, {'id': 3, 'name': 'bruce'}, ] # way too verbose and not pythonic ids = [user['id'] for user in users] index = ids.index(2) user_2 = users[index] # short, but it feels a bit janky user_2 = next((user for user in users if user['id'] == 2), None) # this is okay-ish, i guess users_dict = {user['id']: user for user in users} user_2 = users_dict.get(2) In my opinion, the list type could have something along these lines: class MyList(list): def find(self, func, default=None): for i in self: if func(i): return i return default my_list = MyList(users) user_2 = my_list.find(lambda user: user['id'] == 2) print(user_2) # {'id': 2, 'name': 'anna'}

6 6

Get the value of an attribute of an object hierarchy with a path expression
by Dr. Guta Gábor 06 May '22

06 May '22

This proposal is about adding a builtin function or a package to the standard library to access an element in an object hierarchy with default behavior i.e. if at any level the attribute/key/index does not exist in the path the function returns a default value. Motivation: mapping an object tree to a tuple or a flat object cannot be achieved compactly with proper error handling. Need: I have found a half dozen of packages implementing this functionality on github and there is a high number of loosely related libraries. Here is a motivating example: animal = { 'name': 'Wombat', 'avg_properties': { 'height': {'value': 66, 'unit': 'cm'}, 'length':{'value': 108, 'unit': 'cm'}, 'weight': {'value': 27, 'unit': 'kg'} } } assert getpath(animal, ('avg_properties', 'length', 'value'), default='-') == 108 assert getpath(animal, ('min_properties', 'length', 'value'), default='-') == '-' assert getpaths(animal, 'avg_properties.length.value', default='-') == 108 assert getpaths(animal, 'avg_properties/width/value', sep='/') == None I created a package to demonstrate the idea: https://github.com/axonmatics/quickpath

5 7

Re: Auto assignment of attributes
by Christopher Barker 05 May '22

05 May '22

Bringing this back on list -- I hope that was an accident. (NOTE: getting a touch off-topic here -- but I do bring it back around at the end) On Wed, May 4, 2022 at 4:38 AM Paul Moore <p.f.moore(a)gmail.com> wrote: > On Tue, 3 May 2022 at 17:46, Christopher Barker <pythonchb(a)gmail.com> > wrote: > > On Tue, May 3, 2022 at 5:39 AM Paul Moore <p.f.moore(a)gmail.com> wrote: > >> > >> > > I have classes with 20+ parameters (packaging metadata). You can > argue > >> > > that a dataclass would be better, > > > > I sure would — I’ve done exactly this, and it seems to me to be exactly > the right solution. > > Sorry, you're right. There are issues with a dataclass based approach > (I've tried it, so I know ;-)) but they aren't around repeating self.x > = x over and over. My point here was that the @-based solution isn't > useful as often as people are claiming. Agreed here -- I'm probably responsible for much of the talk about dataclasses, so sorry about that. In fact, my only point there was that dataclasses are a solution to the "I'm writing many versions of self.x = x in my __init__ method, and that seems unnecessary" problem, and that I think that very problem is the strongest case for auto-assignment. Ergo, auto-assignment isn't a critical feature. > "lots of > parameters are simply hard, and in practical cases, @ doesn't help > enough to make a difference" Agreed -- if it's a complex case, neither datclasses nor auto-assignment would help. I've used attrs, which has field converters. ... > The problem with __post_init__ (I think) is > that it confuses the type checkers (beware, untested code below!): > Does the attrs approach work any better for this? @dataclass > class Metadata: > version: str # Actually a Version, but that's not the type we > want the constructor to accept. > > def __post_init__(self): > self.version = Version(self.version) > > And the reality is worse. We want the constructor to take a Version, > or anything that can be converted to a Version > (which, if I check the > source of the library, *is* just str - but I don't want to necessarily > copy that declaration as it might change). And yet we want the > typechecker to know that metadata.version is always a Version. All of > that information is available statically (we call Version on the input > data, and self.version is always the result of that call) but I can't > see how to explain that to the type checker. Frankly, I see that as a limitation of static typing -- how does one describe "anything that can be turned into a Version? -- sure, the Version __init__ can take either a Version or a str -- but only certain strs will make any sense -- so what's the point? Honestly, I think if you want your code to be statically "safe" then you should have this metadata class require a Version object, which would require its users to do the conversion first. Which is why I like the Dynamic nature of Python, and haven't gotten excited about static typing. I suppose the way to solve this without dataclasses or auto-assignment is to type the class attribute and the __init__ parameter separately, yes? (which auto-assignment wouldn't help with, either) (untested, and probably wrong, 'cause I'm not that familiar with typing) Class Metadata: version: Version ... def __init__(self, version: Version | str, ...): ... Is that right? What this tells me is that if you want dataclasses to be more typing-friendly, then there should be a way to specify the __init__ type separately, which could be done with a parameter to the Field object. Bringing it back OT: As much as I hate to say it: For folks advocating an auto-assignment feature -- you should probably consider how it would play with static typing :-( -CHB > I'm not obsessive about > typing, but dataclasses do naturally encourage using them. And if I do > use types, I want them to be *right*, or VS Code and mypy will spend > all their time complaining about correct code :-( > > This is all irrelevant to the discussion at hand, and quite possibly > way too nitpicking to even be worth caring about, but it's what I mean > when I say "things are still hard". > > Paul > -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

3 2