Mailman 3 May 2021 - Python-ideas

zip(x, y, z, strict=True)
by Ram Rachum Dec. 1, 2022

Dec. 1, 2022

Here's something that would have saved me some debugging yesterday: >>> zipped = zip(x, y, z, strict=True) I suggest that `strict=True` would ensure that all the iterables have been exhausted, raising an exception otherwise. This is useful in cases where you're assuming that the iterables all have the same lengths. When your assumption is wrong, you currently just get a shorter result, and it could take you a while to figure out why it's happening. What do you think?

27 179

Bringing the print statement back
by Guido van Rossum March 2, 2022

March 2, 2022

In Python 3.10 we will no longer be burdened by the old parser (though 3rd party tooling needs to catch up). One thing that the PEG parser makes possible in about 20 lines of code is something not entirely different from the old print statement. I have a prototype: Python 3.10.0a0 (heads/print-statement-dirty:5ed19fcc1a, Jun 9 2020, 16:31:17) [Clang 11.0.0 (clang-1100.0.33.8)] on darwin Type "help", "copyright", "credits" or "license" for more information. Cannot read termcap database; using … [View More]dumb terminal settings. >>> print 2+2 4 >>> print "hello world" hello world >>> print "hello", input("Name:") Name:Guido hello Guido >>> print 1, 2, 3, sep=", " 1, 2, 3 >>> But wait, there's more! The same syntax will make it possible to call *any* function: >>> len "abc" 3 >>> Or any method: >>> import sys >>> sys.getrefcount "abc" 24 >>> Really, *any* method: >>> class C: ... def foo(self, arg): print arg ... >>> C().foo 2+2 4 >>> There are downsides too, though. For example, you can't call a method without arguments: >>> print <built-in function print> >>> Worse, the first argument cannot start with a parenthesis or bracket: >>> print (1, 2, 3) 1 2 3 >>> C().foo (1, 2, 3) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: C.foo() takes 2 positional arguments but 4 were given >>> print (2+2), 42 4 (None, 42) >>> C().foo [0] Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'method' object is not subscriptable >>> No, it's not April 1st. I am seriously proposing this (but I'll withdraw it if the response is a resounding "boo, hiss"). After all, we currently have a bunch of complexity in the parser just to give a helpful error message to people used to Python 2's print statement: >>> print 1, 2, 3 File "<stdin>", line 1 print 1, 2, 3 ^ SyntaxError: Missing parentheses in call to 'print'. Did you mean print(1, 2, 3)? >>> And IIRC there have been a number of aborted attempts at syntactic hacks to allow people to call functions (like print) without parentheses, although (I think) none of them made it into a PEP. The PEG parser makes this much simpler, because it can simply backtrack -- by placing the grammar rule for this syntax (tentatively called "call statement") last in the list of alternatives for "small statement" we ensure that everything that's a valid expression statement (including print() calls) is still an expression statement with exactly the same meaning, while still allowing parameter-less function calls, without lexical hacks. (There is no code in my prototype that checks for a space after 'print' -- it just checks that there's a name, number or string following a name, which is never legal syntax.) One possible extension I didn't pursue (yet -- dare me!) is to allow parameter-less calls inside other expressions. For example, my prototype does not support things like this: >>> a = (len "abc") File "<stdin>", line 1 a = (len "abc") ^ SyntaxError: invalid syntax >>> I think that strikes a reasonable balance between usability and reduced detection of common errors. I could also dial it back a bit, e.g. maybe it's too much to allow 'C().foo x' and we should only allow dotted names (sufficient to access functions in imported modules and method calls on variables). Or maybe we should only allow simple names (allowing 'len x' but disallowing 'sys.getrefcount x'. Or maybe we should really only bring back print statements. I believe there are some other languages that support a similar grammar (Ruby? R? Raku?) but I haven't investigated. Thoughts? -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-…> [View Less]

42 66

Adding a thin wrapper class around the functions in stdlib.heapq
by bunslow Feb. 18, 2022

Feb. 18, 2022

Nothing so bombastic this time. The heapq functions are basically all named "heapsomething", and basically all take a "heap" for their first argument, with supplementary args coming after. It's a textbook example of the (hypothetical) Object Oriented Manifesto™ where defining a class increases type safety and programmers' conceptual clarity. There're practically no drawbacks, and the code to be added would be very simple. Updating the tests and docs would probably be harder. In pure Python, … [View More]

17 32

Allow syntax "func(arg=x if condition)"
by Peter O'Connor Jan. 19, 2022

Jan. 19, 2022

I often find that python lacks a nice way to say "only pass an argument under this condition". (See previous python-list email in "Idea: Deferred Default Arguments?") Example 1: Defining a list with conditional elements include_bd = True current_way = ['a'] + (['b'] if include_bd else [])+['c']+(['d'] if include_bd else []) new_way = ['a', 'b' if include_bd, 'c', 'd' if include_bd] also_new_way = list('a', 'b' if include_bd, 'c', 'd' if include_bd) Example 2: Deferring to … [View More]

6 9

Introduce constant variables in Python
by Shreyan Avigyan June 17, 2021

June 17, 2021

Many times a programmer would want a variable that will be a constant. The programmer could have been careless and suddenly changed a very important variable. Proposed syntax, constant variable = 10 variable = 20 # Error Now constant means making a variable value immutable. It means now we can neither change the variable to point to another value nor the value itself anymore. For example, if we have a list we cannot change it. If we try to then "Error". So for lists, constant variable = ["… [View More]

10 16

A __decoration_call__ method for Callable objects (WAS: Decorators on variables)
by Ricky Teachey June 1, 2021

June 1, 2021

On Wed, May 26, 2021 at 7:54 AM Steven D'Aprano <steve(a)pearwood.info> wrote: > On Wed, May 26, 2021 at 01:33:07PM +0200, Stéfane Fermigier wrote: > > > Last point that I like in the decorator syntax: it's > > > > I can compose N different decorators and keep the intent obvious: > > > > @acl(READ, WRITE) > > @constraint(10 < _ < 100) > > @not_null > > @indexed > > @depends_on(whatever) > > @inject > > @... … [View More]> > first_name: str > > Hmm, that's a good point. > > On the other hand, it would be terribly confusing if the same syntax: > > @decorate > > had radically different meaning depending on whether it was followed by > a class/function or a bare name. > > > -- > Steve > ....and previously Steve also said: On Tue, May 25, 2021 at 3:28 AM Steven D'Aprano <steve(a)pearwood.info> wrote: > On Mon, May 24, 2021 at 06:36:47PM -0700, micro codery wrote: > > > Basically this would add syntax to python that would transform > > @decorator("spam this") variable > > into > > variable = decorator("variable", "spam this") > > That is confusingly different from decorator syntax in other contexts.... > > @decorator("spam this") > def func(): pass > > # transformed to: > > def func(): pass > func = decorator("spam this")(func) > > ...the critical difference is that the argument "spam this" should be > passed to the decorator *factory*, which then returns the actual > decorator that gets applied to the variable. (Or function/ class in the > case of regulator decorator syntax.) Well, in actuality even if it were implemented the way you described, it is still going to be very different--- I might even say radically different. These two ideas of a decorator syntax result are not the same: RESULT A: function decorator # func = decorator("spam")(func) RESULT B: variable decorator # name = decorator("spam")("name") ...because func is passed as an object, but "name" a string representing the name of the object. Two very different things. For this reason I think I would agree even more so that the differences in the decorator behavior would be an extremely significant point of confusion. This got me to thinking: what if access to the variable name were provided by another means, and ONLY when the decorator syntax is employed? So then I started to write this big long email and it kind of got out of hand. Hopefully it isn't a disaster. FIRST LAYER: RICK'S (LIKELY HALF-BAKED) COUNTER PROPOSAL Maybe employment of decorator syntax could OPTIONALLY trigger a new dunder method-- here I'll just call it __decoration_call__-- with the signature: def __decoration_call__(self, obj: Any, by_name: str) -> Any: ... Before I describe what I intend by this, first I will stipulated that what I am proposing here should only be implemented so that the behavior of all currently existing decorators (i.e., all callables) would remain exactly as it does today. So, for any existing callable with the name decorator, this: @decorator("spam this") def func(): ... ...continues to mean this, just as it does today: def func(): ... func = decorator("spam this")(func) My idea is to optionally allow any callable object to write a __decoration_call__ method that gets called in lieu of the __call__ method when the callable object is employed using decorator syntax. When this happens, the decorated named is supplied- not counting self- as the first argument (e.g., by_name), which contains the str value of the name the decorator was applied to. Let's explain using code examples. In actuality, unless I'm wrong (I might be; not an expert) current decorator syntax is really sugar for: def func(): ... func = decorator.__call__("spam this").__call__(func) My proposal is to make it such that: @decorator def func(): ... ...*can result* in this: def func(): ... func = decorator.__decoration_call__( func, "func") And also so that this: @decorator("spam this") def func(): ... ...*can result* in this: def func(): ... func = decorator.__call__("spam this").__decoration_call__(func, "func") I say "*can result*" because this has the following limitations: 1. occurs only when the callable object is employed as a decorator 2. occurs only when __decoration_call__ exists (and if it doesn't, just revert back to the usual __call__(func_object, obj)) Here is an example to further illustrate the suggested behavior. It is not intended to demonstrate a useful example. You could write a callable object like this: class Decorator: def __call__(self, obj): print("spam") return obj def __decoration_call__(self , obj, by_name): print("eggs") print(by_name) return obj decorator = Decorator() And the behavior would be like this: def func(): ... func = decorator(func) # Console output: # 'spam' @decorator def func(): ... # Console output: # 'eggs' # 'func' In order to preserve current behavior, the built-in <class 'function'> type would not grow its own new __decoration_call__ method. So in the case of existing normal functions, and also in the case of existing custom Callable objects, the fallback would be to just use __call__ as normal. In this way, all of the proposal is optional. But as illustrated above, the new dunder can be implemented by any customized Callable class. I think __decoration_call__ by itself could provide a lot of interesting possibilities for existing class and function decoration. But my main motivation for this is as a different way of implementing the variable decorator idea proposed by Jeremiah. SECOND LAYER: VARIABLE DECORATORS With __decoration_call__ in place, we could do a lot of interesting things if we take the additional step of broadening existing decorator syntax such that you can decorate not just functions and classes but also variables (similar the variable decorator proposal recently proposed by Jeremiah). But first let's define the variable decorator syntax behavior. Using the same decorator object I have defined above, and borrowing Jeremiah's original example, the new syntax would transform this: @decorator variable ...into this: variable = decorator.__decoration_call__(None, "variable") EXPLANATION OF None: in the example above there is no assignment on the decorated line, and so there is no object to pass to the dunder at that point . Because of this, the first argument, which is usually the object being decorated, is supplied as None in the call to __decoration_call__ . Only the name being assigned to the as-yet uncreated object exists. And the new syntax would transform this: @decorator variable = "spam" ...into this: variable = decorator.__decoration_call__(variable, "variable") With the behavior of this variable decoration defined as shown above, here are some of the interesting possibilities I was considering. One possibility is avoiding the pitfalls of name repetition in the RAD framework context previously mentioned by Stéfane Fermigier in Jeremiah's variable decorator thread (see his post for examples). It could see other frameworks, like web frameworks, making use if this too. (Here I will repeat some of the possibilities suggested by Jeremiah in his first post) The standard library and other library functions could, as needed, make use of __decoration_call__ as they see fit. I could see factories like namedtuple, make_dataclass, several typing factories, and Enum, implementing __decoration_call__ so you can say things like: @namedtuple Point = "x y z" @make_dataclass Point = [("x", int), ("y", int), ("z", int)] @typing.TypeVar T = (str, bytes) @typing.NewType UserId = int @enum.Enum Colors = "RED GREEN BLUE" Admittedly, these examples are not all that compelling on their own, but I think they read really well and they would allow you to avoid retyping a name (and potentially making a mistake in retyping it), so I think I really like it. ___ SIDEBAR I also note that the above examples use a different idiom than suggested by Jeremiah. He has suggested these: @namedtuple("x y z") Point @make_dataclass([("x", int), ("y", int), ("z", int)]) Point @typing.TypeVar( (str, bytes) ) T @typing.NewType(int) UserId @enum.Enum("RED GREEN BLUE") Colors ...but using my proposal I do not see a way to make this idiom easily work with the existing factories. I am not sure which of these two idioms people would prefer. I can see pros and cons for both idioms. If people preferred the second idiom, new decorator objects could be added for it. ___ One interesting possibility might be to modify inspect.getsource so that it can retrieve the RHS of the line on which it appears*: @inspect.getsource my_var = 1/2 print(my_var) # Result # '@inspect.getsource my_var = 1/2' * NOTE: I am admittedly hand-waving this. Maybe there are reasons getsource would not be able to do this? Using the rhs string from this new version of getsource, it would be a pretty simple matter for a symbolic math library like sympy to replace the float value resulting from 1/2 in my_var with a symbolic math value: @sympy.sympify my_var = 1/2 # Result is not 0.5, but: # 1/2 ...or create a Fraction (using a modified Fraction type): @Fraction my_var = 1/2 # Result is not 0.5, but: # Fraction(1, 2) ..or a Decimal (using a modified Decimal type): @Decimal my_var = 0.5 # Result is not 0.5, but: # Decimal('0.5') MAYBE A DEFAULT DUNDER AFTER ALL...? One final possibility: I have said above we would not supply any default __decoration_call__ method, and instead fallback on __call__. But it might be beneficial to supply a default __decoration_call__ with the following implementation: def __decoration_call__(self, func, by_name): if func is None: return self(by_name) return self(func) This would allow things like this, "out of the box": @typing.NewType Color = str @Color GREEN ...which becomes: Color = typing.NewType.__decoration_call__(str, "Color") GREEN = Color.__decoration_call__(None, "GREEN") In the above, Color is just a stand-in for str. If there is a default __decoration_call__ method as I wrote above, then: Color.__decoration_call__(None, "GREEN") ...is actually just: str.__decoration_call__(None, "GREEN") ..and the default implementation just returns: str("GREEN") ...and assigns it to the name, GREEN. Guess I'll leave it at that. Let's see how this is received. --- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler [View Less]

9 29

The name Ellipsis should be a constant
by André Roberge June 1, 2021

June 1, 2021

In Python `...` is referred to as `Ellipsis` and cannot be assigned to. Yet, one can assign any value to the name `Ellipsis`. Consider the following: ``` >>> ... Ellipsis >>> ... == Ellipsis True >>> Ellipsis Ellipsis >>> Ellipsis = 3 >>> Ellipsis 3 >>> ... = 4 File "<stdin>", line 1 ... = 4 ^ SyntaxError: cannot assign to Ellipsis >>> # But I just did assign a new value to the name Ellipsis above. >>> … [View More]

14 33

symbolic math in Python
by Martin Teichmann May 31, 2021

May 31, 2021

Hi list, as you might have noticed, I am trying to improve the syntax and semantics for symbolic math in Python. Until now, I have to say, my ideas were not that well received, but I learned from the discussion and maybe this time I come up with something people like. To frame the problem, let us try to solve the equation x ** 2 == 1/2 using sympy: >>> from sympy import Eq, solve, symbols, S >>> x = symbols("x") >>> solve(Eq(x**2, S(1)/2)) [-sqrt(2)… [View More]/2, sqrt(2)/2] that worked well, but actually we would like to write the last line simply as >>> solve(x**2 == 1/2) as you might notice, this is fully legal Python syntax. Unfortunately the semantics is such that sympy has no way to determine what is actually going on, this is why they invented all those helper functions shown above. My idea is now to start at the line above, "x = symbols('x')". I propose a new syntax, "symbolic x", which tells the parser that x is a symbolic variable, and expressions should not be executed, but handed over as such to the calling functions. To stay with the example, the code would look like this (this is fake, I did not prototype this yet): >>> from sympy import solve >>> symbolic x >>> solve(x**2 == 1/2) [-sqrt(2)/2, sqrt(2)/2] Now to the details. The scope that "symbolic" acts on should be the same as the scope of "global". Note that "symbolic x" is currently illegal syntax, so we would not even need to make "symbolic" a keyword. Expressions that contain a symbolic variable are not executed, but instead the expression should be given to the function as a tuple, so in the example above, solve would be given ('==', ('**', ('x', 2)), ('/', 1, 2)). If that looks like LISP to you, then you are not completely wrong. The boundaries of expressions are function calls, assignments, getitems and getattrs, as they can be overloaded much easier by other means. To give an example with gory details (again, this is fake): >>> symbolic x >>> d = {"a" : 5} >>> c = 7 # not symbolic! >>> print(c * x + d["a"]) ('+', ('*', 7, 'x'), 5) Maybe we would want to have a new class "expressiontuple" which simply acts as a pretty-printer, then the last lines would become >>> print(x + d["a"]) 7 * x + 5 What sympy does with this tuple would be fully at its discretion. I think that other libraries could also benefit from this proposal. As an example, in a numerics library (numpy? scipy?) one could improve numerical integration as in >>> symbolic x >>> integrate(sin(x), x, 0, pi) then the integrator could even compile the expression to machine code for faster integration. numpy could also compile expressions to machine code, making it even faster than it is now, as in >>> symbolic x >>> f = compile(5 * x + 7 * y, (x, y)) >>> f(matrix_a, matrix_b) Let's see how well this proposal fares. Cheers Martin [View Less]

15 20

dict.sort()?
by Marco Sulla May 30, 2021

May 30, 2021

Since `dict` now is ordered, how about a `sort()` method? It could have the same signature of list.sort(), with an optional parameter "by" that can be "keys" or "values" ("keys" could be the default).

6 8

Cleaner tracebacks from Python code
by André Roberge May 29, 2021

May 29, 2021

With CPython, tracebacks obtained from code written in C can be extremely clean compared with functionally equivalent code written in Python. Consider the following test file where I am using a local copy of Python's datetime.py module. ```py from local_datetime import date d = date(2021, 13, 1) ``` Executing this code results in the following, very "clean" traceback: ```pytb Traceback (most recent call last): File "test.py", line 2, in <module> d = date(2021, 13, 1) ValueError: … [View More]month must be in 1..12 ``` A single line of code is shown, which is the one written by the end-user of this library; all code details from inside the library module are hidden. As it turns out, the datetime module imports (if it is available) the _datetime module written in C. If we comment out this import, so that the pure Python code is used instead, here is the new traceback: ```pytb Traceback (most recent call last): File "test.py", line 2, in <module> d = date(2021, 13, 1) File "C:\Users\andre\local_datetime.py", line 847, in __new__ year, month, day = _check_date_fields(year, month, day) File "C:\Users\andre\local_datetime.py", line 422, in _check_date_fields raise ValueError('month must be in 1..12', month) ValueError: ('month must be in 1..12', 13) ``` In addition to the call by the end-user of that library, we see some inner code of that library that has no practical value to the end-user. We can suppress some lines of the traceback by replacing the line ```py year, month, day = _check_date_fields(year, month, day) ``` by ```py try: year, month, day = _check_date_fields(year, month, day) except Exception: _, _, tb = sys.exc_info() tb.tb_next = None raise ``` which simplifies the traceback somewhat: ```pytb Traceback (most recent call last): File "test.py", line 2, in <module> d = date(2021, 13, 1) File "C:\Users\andre\local_datetime.py", line 848, in __new__ year, month, day = _check_date_fields(year, month, day) ValueError: ('month must be in 1..12', 13) ``` but there is still one line of code from inside the library. === Idea: it would be nice if one could somehow define **callables** with a custom attribute that would indicate that tracebacks should not include frames "below" it. More explicitly, using the notation of the above example, imagine that we could write ```py date.__hide_tb__ = True ``` and that any call to `date()` that generate a traceback would not show frames below the calling frame, thus reproducing what we see when using the C code in this example. For library writers, this would be easily reverted when attempting to debug. For end users, this would result in much more readable tracebacks, unencumbered by extra lines of code that are outside of their control. André Roberge [View Less]

4 8