Here's something that would have saved me some debugging yesterday:
>>> zipped = zip(x, y, z, strict=True)
I suggest that `strict=True` would ensure that all the iterables have been
exhausted, raising an exception otherwise.
This is useful in cases where you're assuming that the iterables all have
the same lengths. When your assumption is wrong, you currently just get a
shorter result, and it could take you a while to figure out why it's
happening.
What do you think?
Hello,
I would like to suggest adding a simple “once” method to functools. As the name suggests, this would be a decorator that would call the decorated function, cache the result and return it with subsequent calls. My rationale for suggesting this addition is twofold:
First: It’s fairly common to use `lru_cache()` to implement this behaviour. We use this inside Django (example <https://github.com/django/django/blob/77aa74cb70dd85497dbade6bc0f394aa41e88…>), internally in other projects at my workplace, inside the stdlib itself <https://github.com/python/cpython/blob/2fa67df605e4b0803e7e3aac0b85d851b4b4…> and in numerous other projects. In the first few pages of a Github code search <https://github.com/search?l=Python&q=%22functools.lru_cache%22&type=Code> it is fairly easy to find examples, any decorated method with no parameters is using `lru_cache()` like `once()`. Using lru_cache like this works but it’s not as efficient as it could be - in every case you’re adding lru_cache overhead despite not requiring it.
Second: Implementing this in Python, in my opinion, crosses the line of “annoying and non-trivial enough to not want to repeatedly do it”. While a naive (untested) implementation might be:
def once(func):
sentinel = object() # in case the wrapped method returns None
obj = sentinel
@functools.wraps(func)
def inner():
nonlocal obj, sentinel
if obj is sentinel:
obj = func()
return obj
return inner
While to the people who are likely going to be reading this mailing this the code above is understandable and potentially even somewhat simple. However to a lot of people who might not have had experience with writing decorators or understand sentinel objects and their use the above code might be incomprehensible. A much more common, and in my opinion worse, implementation that I’ve seen is something along the lines of this:
_value = None
def get_value():
nonlocal _value
if _value is None:
_value = some_function()
return _value
Which is not ideal for obvious reasons. And these are not even including a potentially key feature: locking the wrapped function so that it is only called once if it is invoked from multiple threads at once.
So, I’d like to propose adding a `once()` decorator to functools that:
1. Has a C implementation, keeping the speed on-par with `lru_cache()`
2. Ensures that the wrapped function is only called once when invoked by multiple threads
For some related discussion about this idea and lru_cache, please see my thread on <https://discuss.python.org/t/reduce-the-overhead-of-functools-lru-cache-for…>discuss.python.org <http://discuss.python.org/>.
> Thread 2 wakes up with the lock, calls the function, fills the cache, and releases the lock.
What exactly would the issue be with this:
```
import functools
from threading import Lock
def once(func):
sentinel = object()
cache = sentinel
lock = Lock()
@functools.wraps(func)
def _wrapper():
nonlocal cache, lock, sentinel
if cache is sentinel:
with lock:
if cache is sentinel:
cache = func()
return cache
return _wrapper
```
It ensures it’s invoked once, it’s fast and it doesn’t require acquiring a lock needlessly. Always wrapping a lock around a static value to account for a small edge case seems redundant, more than “it’s usually not optimal”. It’s never optimal: in multi-threaded cases you’re causing needless contention, and in single threaded cases all your calls are now slower.
Seems generally more correct, even in single threaded cases, to pay the overhead only in the first call if you want `call_once` semantics. Which is why you would be using `call_once` in the first place?
> On 29 Apr 2020, at 18:51, Andrew Barnert <abarnert(a)yahoo.com> wrote:
>
> On Apr 29, 2020, at 00:34, Tom Forbes <tom(a)tomforb.es> wrote:
>>
>> It’s not quite that easy, either you needlessly lock all calls or you end up invoking it twice.
>>
>> What you want is to acquire a lock if the cache is empty, check if another thread has filled the cache while you where waiting on the lock, call the function, fill the cache and return.
>
> Thread 1 sees the cache is empty, so it acquires the lock, runs the function.
>
> Thread 2 sees the cache is empty, so it tries to acquire the cache and blocks.
>
> Thread 1 finishes the function, fills the cache, and releases the lock.
>
> Thread 2 wakes up with the lock, calls the function, fills the cache, and releases the lock.
>
> You’ve now run the function twice instead of once, and the first caller got a different value than the one every subsequent caller can get.
>
> There are actually plenty of cases where this is acceptable (the function is safe, and idempotent, and while it’s slow it’s not so slow that occasionally running it twice is worse than locking a zillion times)—but in those cases, just not using a lock is even better. In fact, if you can compare-and-swap at the end (which requires a lock to simulate in Python, but it’s a much more granular one than locking over the whole expensive function call), you can even use it in more cases (in particular, where the function isn’t idempotent but you need to value to be).
>
> This is exactly why users should be encouraged to just lock the whole thing. It’s always correct. It’s usually not optimal, but often good enough. The one really common and simple case where it’s not good enough is single-threaded code. Anything else where you need to optimize, you really do need to understand the pattern you’re trying to optimize for (and the environment(s) you’re running under) and write your own function.
>
>
I was playing around with deque today, and there were a couple of
operations I wanted to do, that can't really be done efficiently with deque
because of its implementation.
I was iterating through items of a deque, and in some cases I wanted to
delete an item that I've found. As far as I understand, this is an
operation that should be O(1) in a linked list, but Python only provides an
O(N) way to do that, which is `del d[i]`.
The same can be said for inserting items in the middle.
What do you think about adding this to `deque`? The API will be tricky,
admittedly, because you'll have to save some kind of reference to a cell in
the deque.
Thanks,
Ram.
I wanna propose making generators even weirder!
so, extended continue is an oldie:
https://www.python.org/dev/peps/pep-0342/#the-extended-continue-statement
it'd allow one to turn:
yield from foo
into:
for bar in foo:
continue (yield bar)
but what's this extended for-else? well, currently you have for-else:
for x, y, z in zip(a, b, c):
...
else:
pass
and this works. you get the stuff from the iterators, and if you break
the loop, the else doesn't run. the else basically behaves like "except
StopIteration:"...
so I propose an extended for-else, that behaves like "except
StopIteration as foo:". that is, assuming we could get a zip() that
returns partial results in the StopIteration (see other threads), we
could do:
for x, y, z in zip(a, b, c):
do_stuff_with(x, y, z)
else as partial_xy:
if len(partial_xy) == 0:
x = dummy
try:
y = next(b)
except StopIteration: y = dummy
try:
z = next(c)
except StopIteration: z = dummy
if (x, y, z) != (dummy, dummy dummy):
do_stuff_with(x, y, z)
if len(partial_xy) == 1:
x, = partial_xy
y = dummy
try:
z = next(c)
except StopIteration: z = dummy
do_stuff_with(x, y, z)
if len(partial_xy) == 2:
x, y = partial_xy
z = dummy
do_stuff_with(x, y, z)
(this example is better served by zip_longest. however, it's
nevertheless a good way to demonstrate functionality, thanks to
zip_longest's (and zip's) trivial/easy to understand behaviour.)
this would enable one to turn:
return yield from foo
into:
for bar in foo:
continue (yield bar)
else as baz:
return baz
allowing one to pick apart and modify the yielded and sent parts, while
still getting access to the return values.
currently if you have an arbitrary generator, you can't modify the
yielded values without breaking send or return. in fact you can either
pass-through yield and send and collect the return, or modify yield and
forget about send and the return, or use very ugly syntax that makes
everything look lower-level than it should, to be able to pick apart
everything:
try:
bar = next(foo)
except StopIteration as exc:
baz = exc.value
else:
while True:
try:
bar = foo.send((yield bar))
except StopIteration as exc:
baz = exc.value
break
return baz
(this is exactly equivalent (if I didn't overlook anything) to the
previous endearingly simple loop with extended for-else and extended
continue)
Hi to everybody. I was looking for any resolution for my pain in PEP and Stack Overflow and can't find an answer. As well the similar ideas here are 13 years old and completely missed.
I feel that I need to create a garbage code doing things like:
```
if myDict.get('spam'):
print(myDict.get('spam'))
```
or
```
x = myDict.get('spam')
if x is not None:
print(x)
```
or
```
days = seconds // 86400
if days > 1:
print(days)
```
and I want to use the `as` keyword instead i.e.
```
if myDict.get('spam') as x: print(x)
```
```
if myDict.get('spam') as x is None: print(x)
```
```
if (seconds // 86400) as days > 1: print(days)
```
I'm opening this thread to discuss and collect feedback on a language change to support keyword arguments to be self-assigned using variables names.
Proposal
--------
Taking the syntax from [bpo-36817](https://bugs.python.org/issue36817) which just [made it to Python 3.8](https://docs.python.org/3/whatsnew/3.8.html#f-strings-support-for-self… the `<keyword name>=` syntax would be valid and have the the same effect as `<keyword name>=<keyword name>`, so these two statements would be equivalent:
```python
foo(bar=bar, qux=qux)
foo(bar=, qux=)
```
This syntax would only be valid inside functions calls so all of the following would remain invalid syntax:
```python
x = 42
x=
# SyntaxError
```
```python
x: Any
[x= for x in foo]
# SyntaxError
```
```python
x = 42
def foo(x=):
...
# SyntaxError
```
The self-documenting variables in f-strings would also remain unchanged as it is not related to function calls:
```python
x = "foo"
f"{x=}"
# x=foo
```
```python
x = "..."
f"{foo(x=)=}"
# foo(x=)=...
```
Also, this proposal does not intent to extend this syntax in any form to the dictionary literal declaration syntax. Although possible, language changes to support these are not being discussed here:
```python
x = "..."
foo["x"] =
# just as stated previously, SyntaxError still
```
```python
x, y = 1, 2
foo = {
x=,
y=,
}
# SyntaxError
```
Worth noting that the following would now be possible with the proposed syntax:
```python
x, y = 1, 2
foo = dict(
x=,
y=,
)
```
Rationale
---------
Keyword arguments are useful for giving better semantics to parameters being passed onto a function and to lift the responsibility from the caller to know arguments order (and possibly avoid mismatching parameters due to wrong order).
Said that, keyword and keyword-only arguments are extensively used across Python code and it is quite common to find code that forwards keyword parameters having to re-state keyword arguments names:
```python
def foo(a, b, c):
d = ...
bar(a=a, b=b, c=c, d=d)
```
Of course if `bar` arguments are not keyword-only one could just omit keywords, i.e. `bar(a, b, c, d)`. Though now the caller has to be aware of arguments ordering again. And yet, if arguments are keyword-only then there is nothing you can do. Except for relying on varkwargs:
```python
def foo(**kwargs):
kwargs["d"] = ...
bar(**kwargs)
```
Which, beyond having its own disadvantages, defeats its own purpose of avoiding repeating or reassigning the keys real quickly if one has to work with these arguments in `foo` before calling `bar`:
```python
def foo(**kwargs):
kwargs["d"] = kwargs["a"] * kwargs["b"] / kwargs["c"]
bar(**kwargs)
```
With all that's being said, introducing self-assigned keywords would be useful for a wide range of Python users while being little intrusive on the Python syntax and backwards compatible.
Best regards,
Rodrigo Martins de Oliveira
Currently type(None) returns None if you call it with no args:
>>> print(type(None)())
None
it'd be nice if it did the same regardless of args. currently it raises:
>>> print(type(None)(1))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: NoneType takes no arguments
inspired by PEP 559: https://www.python.org/dev/peps/pep-0559/
>>> NoneType
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'NoneType' is not defined
I feel like zip could return partial results:
try:
next(zip([0], []))
except StopIteration as exc:
assert StopIteration.args == (0,)
how much would this break?
Forgive me if this idea has been discussed before, I searched the mailing lists, the CPython repo, and the issue tracker and was unable to find anything.
I have found myself a few times in a position where I have a repeated argument that uses the `append` action, along with some convenience arguments that append a specific const to that same dest (eg: `--filter-x` being made equivalent to `--filter x` via `append_const`). This is particularly useful in cli apps that expose some kind of powerful-but-verbose filtering capability, while also providing shorter aliases for common invocations. I'm sure there are other use cases, but this is the one I'm most familiar with.
The natural extension to this filtering idea are convenience args that set two const values (eg: `--filter x --filter y` being equivalent to `--filter-x-y`), but there is no `extend_const` action to enable this.
While this is possible (and rather straight forward) to add via a custom action, I feel like this should be a built-in action instead. `append` has `append_const`, it seems intuitive and reasonable to expect `extend` to have `extend_const` too (my anecdotal experience the first time I came across this need was that I simply tried using `extend_const` without checking the docs, assuming it already existed).
Please see this gist for a working example that may help explain the idea and intended use case more clearly: https://gist.github.com/roganartu/7c2ec129d868ecda95acfbd655ef0ab2
