Mailman 3 April 2020 - Python-ideas

python interpreter docker builder
by Antonio Cavallo 16 Jul '20

16 Jul '20

Hi is there any interest (or anyone has done it before), building the python interpreter using docker? The basic idea is building the toolchain (gcc) and on top of that the python interpreter. On mac/linux it can build natively, but it can use docker to target linux from mac/windows. Thanks

5 15

Make `del x` an expression evaluating to `x`
by Eric Wieser 12 Jun '20

12 Jun '20

TL;DR: should we make `del x` an expression that returns the value of `x`. ## Motivation I noticed yesterday that `itertools.combinations` has an optimization for when the returned tuple has no remaining ref-counts, and reuses it - namely, the following code: >>> for v in itertools.combinations([1, 2, 3], 1): ... print(id(v)) ... del v # without this, the optimization can't take place 2500926199840 2500926199840 2500926199840 will print the same id three times. However, when used as a list comprehension, the optimization can't step in, and I have no way of using the `del` keyword >>> [id(v) for v in itertools.combinations([1, 2, 3], 1)] [2500926200992, 2500926199072, 2500926200992] `itertools.combinations` is not the only place to make this optimization - parts of numpy use it too, allowing a = (b * c) + d to elide the temporary `b*c`. This elision can't happen with the spelling bc = b * c a = bc + d My suggestion would be to make `del x` an expression, with semantics "unbind the name `x`, and evaluate to its value". This would allow: >>> [id(del v) for v in itertools.combinations([1, 2, 3], 1)] [2500926200992, 2500926200992, 2500926200992] and bc = b * c a = (del bc) + d # in C++, this would be `std::move(bc) + d` ## Why a keyword Unbinding a name is not something that a user would expect a function to do. Functions operate on values, not names, so `move(bc)` would be surprising. ## Why `del` `del` already has "unbind this name" semantics, and allow it to be used as an expression would break no existing code. `move x` might be more understandable, but adding new keywords is expensive ## Optional extension For consistency, `x = (del foo.attr)` and `x = (del foo[i])` could also become legal expressions, and `__delete__`, `__delattr__`, and `__delitem__` would now have return values. Existing types would be free to continue to return `None`.

15 50

PYTHONLOGGING env variable
by Bar Harel 06 Jun '20

06 Jun '20

Another idea I've had that may be of use: PYTHONLOGGING environment variable. Setting PYTHONLOGGING to any log level or level name will initialize logging.basicConfig() with that appropriate level. Another option would be that -x dev or a different -x logging will initialize basic config. Will be useful mostly for debugging purposes instead of temporarily modifying the code. Kinda surprised it doesn't exist tbh. Bar Harel

9 22

Adding a "once" function to functools
by Tom Forbes 07 May '20

07 May '20

Hello, I would like to suggest adding a simple “once” method to functools. As the name suggests, this would be a decorator that would call the decorated function, cache the result and return it with subsequent calls. My rationale for suggesting this addition is twofold: First: It’s fairly common to use `lru_cache()` to implement this behaviour. We use this inside Django (example <https://github.com/django/django/blob/77aa74cb70dd85497dbade6bc0f394aa41e88…>), internally in other projects at my workplace, inside the stdlib itself <https://github.com/python/cpython/blob/2fa67df605e4b0803e7e3aac0b85d851b4b4…> and in numerous other projects. In the first few pages of a Github code search <https://github.com/search?l=Python&q=%22functools.lru_cache%22&type=Code> it is fairly easy to find examples, any decorated method with no parameters is using `lru_cache()` like `once()`. Using lru_cache like this works but it’s not as efficient as it could be - in every case you’re adding lru_cache overhead despite not requiring it. Second: Implementing this in Python, in my opinion, crosses the line of “annoying and non-trivial enough to not want to repeatedly do it”. While a naive (untested) implementation might be: def once(func): sentinel = object() # in case the wrapped method returns None obj = sentinel @functools.wraps(func) def inner(): nonlocal obj, sentinel if obj is sentinel: obj = func() return obj return inner While to the people who are likely going to be reading this mailing this the code above is understandable and potentially even somewhat simple. However to a lot of people who might not have had experience with writing decorators or understand sentinel objects and their use the above code might be incomprehensible. A much more common, and in my opinion worse, implementation that I’ve seen is something along the lines of this: _value = None def get_value(): nonlocal _value if _value is None: _value = some_function() return _value Which is not ideal for obvious reasons. And these are not even including a potentially key feature: locking the wrapped function so that it is only called once if it is invoked from multiple threads at once. So, I’d like to propose adding a `once()` decorator to functools that: 1. Has a C implementation, keeping the speed on-par with `lru_cache()` 2. Ensures that the wrapped function is only called once when invoked by multiple threads For some related discussion about this idea and lru_cache, please see my thread on <https://discuss.python.org/t/reduce-the-overhead-of-functools-lru-cache-for…>discuss.python.org <http://discuss.python.org/>.

22 48

Re: Adding a "once" function to functools
by Tom Forbes 02 May '20

02 May '20

> Thread 2 wakes up with the lock, calls the function, fills the cache, and releases the lock. What exactly would the issue be with this: ``` import functools from threading import Lock def once(func): sentinel = object() cache = sentinel lock = Lock() @functools.wraps(func) def _wrapper(): nonlocal cache, lock, sentinel if cache is sentinel: with lock: if cache is sentinel: cache = func() return cache return _wrapper ``` It ensures it’s invoked once, it’s fast and it doesn’t require acquiring a lock needlessly. Always wrapping a lock around a static value to account for a small edge case seems redundant, more than “it’s usually not optimal”. It’s never optimal: in multi-threaded cases you’re causing needless contention, and in single threaded cases all your calls are now slower. Seems generally more correct, even in single threaded cases, to pay the overhead only in the first call if you want `call_once` semantics. Which is why you would be using `call_once` in the first place? > On 29 Apr 2020, at 18:51, Andrew Barnert <abarnert(a)yahoo.com> wrote: > > On Apr 29, 2020, at 00:34, Tom Forbes <tom(a)tomforb.es> wrote: >> >> It’s not quite that easy, either you needlessly lock all calls or you end up invoking it twice. >> >> What you want is to acquire a lock if the cache is empty, check if another thread has filled the cache while you where waiting on the lock, call the function, fill the cache and return. > > Thread 1 sees the cache is empty, so it acquires the lock, runs the function. > > Thread 2 sees the cache is empty, so it tries to acquire the cache and blocks. > > Thread 1 finishes the function, fills the cache, and releases the lock. > > Thread 2 wakes up with the lock, calls the function, fills the cache, and releases the lock. > > You’ve now run the function twice instead of once, and the first caller got a different value than the one every subsequent caller can get. > > There are actually plenty of cases where this is acceptable (the function is safe, and idempotent, and while it’s slow it’s not so slow that occasionally running it twice is worse than locking a zillion times)—but in those cases, just not using a lock is even better. In fact, if you can compare-and-swap at the end (which requires a lock to simulate in Python, but it’s a much more granular one than locking over the whole expensive function call), you can even use it in more cases (in particular, where the function isn’t idempotent but you need to value to be). > > This is exactly why users should be encouraged to just lock the whole thing. It’s always correct. It’s usually not optimal, but often good enough. The one really common and simple case where it’s not good enough is single-threaded code. Anything else where you need to optimize, you really do need to understand the pattern you’re trying to optimize for (and the environment(s) you’re running under) and write your own function. > >

4 7

deque: Allow efficient operations
by Ram Rachum 30 Apr '20

30 Apr '20

I was playing around with deque today, and there were a couple of operations I wanted to do, that can't really be done efficiently with deque because of its implementation. I was iterating through items of a deque, and in some cases I wanted to delete an item that I've found. As far as I understand, this is an operation that should be O(1) in a linked list, but Python only provides an O(N) way to do that, which is `del d[i]`. The same can be said for inserting items in the middle. What do you think about adding this to `deque`? The API will be tricky, admittedly, because you'll have to save some kind of reference to a cell in the deque. Thanks, Ram.

13 20

extended for-else, extended continue, and a rant about zip()
by Soni L. 28 Apr '20

28 Apr '20

I wanna propose making generators even weirder! so, extended continue is an oldie: https://www.python.org/dev/peps/pep-0342/#the-extended-continue-statement it'd allow one to turn: yield from foo into: for bar in foo: continue (yield bar) but what's this extended for-else? well, currently you have for-else: for x, y, z in zip(a, b, c): ... else: pass and this works. you get the stuff from the iterators, and if you break the loop, the else doesn't run. the else basically behaves like "except StopIteration:"... so I propose an extended for-else, that behaves like "except StopIteration as foo:". that is, assuming we could get a zip() that returns partial results in the StopIteration (see other threads), we could do: for x, y, z in zip(a, b, c): do_stuff_with(x, y, z) else as partial_xy: if len(partial_xy) == 0: x = dummy try: y = next(b) except StopIteration: y = dummy try: z = next(c) except StopIteration: z = dummy if (x, y, z) != (dummy, dummy dummy): do_stuff_with(x, y, z) if len(partial_xy) == 1: x, = partial_xy y = dummy try: z = next(c) except StopIteration: z = dummy do_stuff_with(x, y, z) if len(partial_xy) == 2: x, y = partial_xy z = dummy do_stuff_with(x, y, z) (this example is better served by zip_longest. however, it's nevertheless a good way to demonstrate functionality, thanks to zip_longest's (and zip's) trivial/easy to understand behaviour.) this would enable one to turn: return yield from foo into: for bar in foo: continue (yield bar) else as baz: return baz allowing one to pick apart and modify the yielded and sent parts, while still getting access to the return values. currently if you have an arbitrary generator, you can't modify the yielded values without breaking send or return. in fact you can either pass-through yield and send and collect the return, or modify yield and forget about send and the return, or use very ugly syntax that makes everything look lower-level than it should, to be able to pick apart everything: try: bar = next(foo) except StopIteration as exc: baz = exc.value else: while True: try: bar = foo.send((yield bar)) except StopIteration as exc: baz = exc.value break return baz (this is exactly equivalent (if I didn't overlook anything) to the previous endearingly simple loop with extended for-else and extended continue)

4 19

`if` condition with `as` keyword
by Николай Мостовенко 28 Apr '20

28 Apr '20

Hi to everybody. I was looking for any resolution for my pain in PEP and Stack Overflow and can't find an answer. As well the similar ideas here are 13 years old and completely missed. I feel that I need to create a garbage code doing things like: ``` if myDict.get('spam'): print(myDict.get('spam')) ``` or ``` x = myDict.get('spam') if x is not None: print(x) ``` or ``` days = seconds // 86400 if days > 1: print(days) ``` and I want to use the `as` keyword instead i.e. ``` if myDict.get('spam') as x: print(x) ``` ``` if myDict.get('spam') as x is None: print(x) ``` ``` if (seconds // 86400) as days > 1: print(days) ```

3 2

Keyword arguments self-assignment
by oliveira.rodrigo.m＠gmail.com 27 Apr '20

27 Apr '20

I'm opening this thread to discuss and collect feedback on a language change to support keyword arguments to be self-assigned using variables names. Proposal -------- Taking the syntax from [bpo-36817](https://bugs.python.org/issue36817) which just [made it to Python 3.8](https://docs.python.org/3/whatsnew/3.8.html#f-strings-support-for-self… the `<keyword name>=` syntax would be valid and have the the same effect as `<keyword name>=<keyword name>`, so these two statements would be equivalent: ```python foo(bar=bar, qux=qux) foo(bar=, qux=) ``` This syntax would only be valid inside functions calls so all of the following would remain invalid syntax: ```python x = 42 x= # SyntaxError ``` ```python x: Any [x= for x in foo] # SyntaxError ``` ```python x = 42 def foo(x=): ... # SyntaxError ``` The self-documenting variables in f-strings would also remain unchanged as it is not related to function calls: ```python x = "foo" f"{x=}" # x=foo ``` ```python x = "..." f"{foo(x=)=}" # foo(x=)=... ``` Also, this proposal does not intent to extend this syntax in any form to the dictionary literal declaration syntax. Although possible, language changes to support these are not being discussed here: ```python x = "..." foo["x"] = # just as stated previously, SyntaxError still ``` ```python x, y = 1, 2 foo = { x=, y=, } # SyntaxError ``` Worth noting that the following would now be possible with the proposed syntax: ```python x, y = 1, 2 foo = dict( x=, y=, ) ``` Rationale --------- Keyword arguments are useful for giving better semantics to parameters being passed onto a function and to lift the responsibility from the caller to know arguments order (and possibly avoid mismatching parameters due to wrong order). Said that, keyword and keyword-only arguments are extensively used across Python code and it is quite common to find code that forwards keyword parameters having to re-state keyword arguments names: ```python def foo(a, b, c): d = ... bar(a=a, b=b, c=c, d=d) ``` Of course if `bar` arguments are not keyword-only one could just omit keywords, i.e. `bar(a, b, c, d)`. Though now the caller has to be aware of arguments ordering again. And yet, if arguments are keyword-only then there is nothing you can do. Except for relying on varkwargs: ```python def foo(**kwargs): kwargs["d"] = ... bar(**kwargs) ``` Which, beyond having its own disadvantages, defeats its own purpose of avoiding repeating or reassigning the keys real quickly if one has to work with these arguments in `foo` before calling `bar`: ```python def foo(**kwargs): kwargs["d"] = kwargs["a"] * kwargs["b"] / kwargs["c"] bar(**kwargs) ``` With all that's being said, introducing self-assigned keywords would be useful for a wide range of Python users while being little intrusive on the Python syntax and backwards compatible. Best regards, Rodrigo Martins de Oliveira

25 226

Make type(None) the no-op function
by Soni L. 26 Apr '20

26 Apr '20

Currently type(None) returns None if you call it with no args: >>> print(type(None)()) None it'd be nice if it did the same regardless of args. currently it raises: >>> print(type(None)(1)) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: NoneType takes no arguments inspired by PEP 559: https://www.python.org/dev/peps/pep-0559/ >>> NoneType Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'NoneType' is not defined

4 5