Hi,
In Python, there are multiple [compound
statements](https://docs.python.org/3/reference/compound_stmts.html)
with the `else` keyword.
For example:
```
for x in iterable:
if x == sentinel:
break
else:
print("Sentinel not found.")
```
or:
```
try:
do_something_sensitive()
except MyError:
print("Oops!")
else:
print("We're all safe.")
```
In my situation, I would like to mix the `with` statement with `else`.
In this case I would like that if no exception is raised within the
`with` to run the `else` part.
For example:
```
with my_context():
do_something_sensitive()
else:
print("We're all safe.")
```
Now imagine that in my `try .. except` block I have some heavy setup
to do before `do_something_sensitive()` and some heavy cleanup when
the exception occurs.
I'd like my context manager to do the preparation work, execute the
body, and cleanup. Or execute my else block only if there is no
exception.
Is there already a way to accomplish this in Python or can this be a
nice to have?
Regards,
Jimmy
As an aside I have a perfect example to back up what Paul is saying below. I work for a large corporation where developers are permitted to install python modules on their development machines, (subject to some licence restrictions), but the proxy settings required to get to PyPi vary from user to user, site to site, day to day (sometimes hour to hour) and connection type to connection type, e.g. If you are connected via WiFi you need to use a different proxy to that needed if you are on a wired connection and on a VPN (we have more than one in use) different again.
To help with this I have put together a script that will try pip, if it fails will check the current location for pac.pac, get and parse it for proxies, correct a common incompatibility and then try each discovered proxy and if sill none work fall back to a list of common proxies. Once it finds a working setting it advises the user what to use in the current situation.
Obviously, since the script is intended to work out how to get to PyPi, I needed to stick 100% to the standard libraries.
-----Original Message-----
From: Paul Moore <p.f.moore(a)gmail.com>
Sent: 06 April 2020 09:11
To: Stephen J. Turnbull <turnbull.stephen.fw(a)u.tsukuba.ac.jp>
Cc: Pete Wicken <petewicken(a)gmail.com>; Python-Ideas <python-ideas(a)python.org>
Subject: [Python-ideas] Re: macaddress or networkaddress module
On Mon, 6 Apr 2020 at 04:14, Stephen J. Turnbull <turnbull.stephen.fw(a)u.tsukuba.ac.jp> wrote:
> That's OK for developers on their "own" machines, but I think there
> are still a lot of folks who need approval from corporate (though I
> haven't heard from Paul Moore on that theme in quite a while, maybe
> it's gotten better?)
Howdy :-)
It's actually got a bit worse, to the point where I don't bother trying to fight the system as much any more, so things have gone quieter because of that :-) But yes, I still believe that there are important use cases for code that only uses the stdlib, and "not being able to get to the internet" is a valid use case that we shouldn't just dismiss.
On the other hand, with regard to the comment you were replying to:
> Up to around a decade ago, installing third-party libraries > was a huge mess, but nowadays, PyPI works, Python comes with pip > pre-installed, and telling developers they need internet access > when they first start a new project isn’t considered onerous.
I think all of that is true. What I think is more important, though, is around things that aren't so much "projects" as "scripts", and that
*distributing* such scripts that depend on 3rd party libraries is still problematic. Here, I'm not talking about making full-fledged packages, or py2exe style fully bundled apps, but "here's this monitoring script I wrote, let's ship it across 10 production systems and why don't you try it out on your dev boxes?" There's a
*significant* step change in complexity between doing that if the script depends on just the stdlib, versus if it depends on a 3rd party module. And we don't have a really good answer to that yet (zipapps, and tools like shiv that wrap zipapps, are pretty good in practice, but they are not well known or commonly used).
Paul
_______________________________________________
Python-ideas mailing list -- python-ideas(a)python.org To unsubscribe send an email to python-ideas-leave(a)python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/5BN5D…
Code of Conduct: http://python.org/psf/codeofconduct/
Hi
is there any interest (or anyone has done it before), building the python
interpreter using docker?
The basic idea is building the toolchain (gcc) and on top of that the
python interpreter. On mac/linux it can build natively, but it can use
docker to target linux from mac/windows.
Thanks
TL;DR: should we make `del x` an expression that returns the value of `x`.
## Motivation
I noticed yesterday that `itertools.combinations` has an optimization for when the returned tuple has no remaining ref-counts, and reuses it - namely, the following code:
>>> for v in itertools.combinations([1, 2, 3], 1):
... print(id(v))
... del v # without this, the optimization can't take place
2500926199840
2500926199840
2500926199840
will print the same id three times. However, when used as a list comprehension, the optimization can't step in, and I have no way of using the `del` keyword
>>> [id(v) for v in itertools.combinations([1, 2, 3], 1)]
[2500926200992, 2500926199072, 2500926200992]
`itertools.combinations` is not the only place to make this optimization - parts of numpy use it too, allowing
a = (b * c) + d
to elide the temporary `b*c`. This elision can't happen with the spelling
bc = b * c
a = bc + d
My suggestion would be to make `del x` an expression, with semantics "unbind the name `x`, and evaluate to its value".
This would allow:
>>> [id(del v) for v in itertools.combinations([1, 2, 3], 1)]
[2500926200992, 2500926200992, 2500926200992]
and
bc = b * c
a = (del bc) + d # in C++, this would be `std::move(bc) + d`
## Why a keyword
Unbinding a name is not something that a user would expect a function to do. Functions operate on values, not names, so `move(bc)` would be surprising.
## Why `del`
`del` already has "unbind this name" semantics, and allow it to be used as an expression would break no existing code.
`move x` might be more understandable, but adding new keywords is expensive
## Optional extension
For consistency, `x = (del foo.attr)` and `x = (del foo[i])` could also become legal expressions, and `__delete__`, `__delattr__`, and `__delitem__` would now have return values. Existing types would be free to continue to return `None`.
Another idea I've had that may be of use:
PYTHONLOGGING environment variable.
Setting PYTHONLOGGING to any log level or level name will initialize
logging.basicConfig() with that appropriate level.
Another option would be that -x dev or a different -x logging will
initialize basic config.
Will be useful mostly for debugging purposes instead of temporarily
modifying the code.
Kinda surprised it doesn't exist tbh.
Bar Harel
Here's something that would have saved me some debugging yesterday:
>>> zipped = zip(x, y, z, strict=True)
I suggest that `strict=True` would ensure that all the iterables have been
exhausted, raising an exception otherwise.
This is useful in cases where you're assuming that the iterables all have
the same lengths. When your assumption is wrong, you currently just get a
shorter result, and it could take you a while to figure out why it's
happening.
What do you think?
Hello,
I would like to suggest adding a simple “once” method to functools. As the name suggests, this would be a decorator that would call the decorated function, cache the result and return it with subsequent calls. My rationale for suggesting this addition is twofold:
First: It’s fairly common to use `lru_cache()` to implement this behaviour. We use this inside Django (example <https://github.com/django/django/blob/77aa74cb70dd85497dbade6bc0f394aa41e88…>), internally in other projects at my workplace, inside the stdlib itself <https://github.com/python/cpython/blob/2fa67df605e4b0803e7e3aac0b85d851b4b4…> and in numerous other projects. In the first few pages of a Github code search <https://github.com/search?l=Python&q=%22functools.lru_cache%22&type=Code> it is fairly easy to find examples, any decorated method with no parameters is using `lru_cache()` like `once()`. Using lru_cache like this works but it’s not as efficient as it could be - in every case you’re adding lru_cache overhead despite not requiring it.
Second: Implementing this in Python, in my opinion, crosses the line of “annoying and non-trivial enough to not want to repeatedly do it”. While a naive (untested) implementation might be:
def once(func):
sentinel = object() # in case the wrapped method returns None
obj = sentinel
@functools.wraps(func)
def inner():
nonlocal obj, sentinel
if obj is sentinel:
obj = func()
return obj
return inner
While to the people who are likely going to be reading this mailing this the code above is understandable and potentially even somewhat simple. However to a lot of people who might not have had experience with writing decorators or understand sentinel objects and their use the above code might be incomprehensible. A much more common, and in my opinion worse, implementation that I’ve seen is something along the lines of this:
_value = None
def get_value():
nonlocal _value
if _value is None:
_value = some_function()
return _value
Which is not ideal for obvious reasons. And these are not even including a potentially key feature: locking the wrapped function so that it is only called once if it is invoked from multiple threads at once.
So, I’d like to propose adding a `once()` decorator to functools that:
1. Has a C implementation, keeping the speed on-par with `lru_cache()`
2. Ensures that the wrapped function is only called once when invoked by multiple threads
For some related discussion about this idea and lru_cache, please see my thread on <https://discuss.python.org/t/reduce-the-overhead-of-functools-lru-cache-for…>discuss.python.org <http://discuss.python.org/>.
> Thread 2 wakes up with the lock, calls the function, fills the cache, and releases the lock.
What exactly would the issue be with this:
```
import functools
from threading import Lock
def once(func):
sentinel = object()
cache = sentinel
lock = Lock()
@functools.wraps(func)
def _wrapper():
nonlocal cache, lock, sentinel
if cache is sentinel:
with lock:
if cache is sentinel:
cache = func()
return cache
return _wrapper
```
It ensures it’s invoked once, it’s fast and it doesn’t require acquiring a lock needlessly. Always wrapping a lock around a static value to account for a small edge case seems redundant, more than “it’s usually not optimal”. It’s never optimal: in multi-threaded cases you’re causing needless contention, and in single threaded cases all your calls are now slower.
Seems generally more correct, even in single threaded cases, to pay the overhead only in the first call if you want `call_once` semantics. Which is why you would be using `call_once` in the first place?
> On 29 Apr 2020, at 18:51, Andrew Barnert <abarnert(a)yahoo.com> wrote:
>
> On Apr 29, 2020, at 00:34, Tom Forbes <tom(a)tomforb.es> wrote:
>>
>> It’s not quite that easy, either you needlessly lock all calls or you end up invoking it twice.
>>
>> What you want is to acquire a lock if the cache is empty, check if another thread has filled the cache while you where waiting on the lock, call the function, fill the cache and return.
>
> Thread 1 sees the cache is empty, so it acquires the lock, runs the function.
>
> Thread 2 sees the cache is empty, so it tries to acquire the cache and blocks.
>
> Thread 1 finishes the function, fills the cache, and releases the lock.
>
> Thread 2 wakes up with the lock, calls the function, fills the cache, and releases the lock.
>
> You’ve now run the function twice instead of once, and the first caller got a different value than the one every subsequent caller can get.
>
> There are actually plenty of cases where this is acceptable (the function is safe, and idempotent, and while it’s slow it’s not so slow that occasionally running it twice is worse than locking a zillion times)—but in those cases, just not using a lock is even better. In fact, if you can compare-and-swap at the end (which requires a lock to simulate in Python, but it’s a much more granular one than locking over the whole expensive function call), you can even use it in more cases (in particular, where the function isn’t idempotent but you need to value to be).
>
> This is exactly why users should be encouraged to just lock the whole thing. It’s always correct. It’s usually not optimal, but often good enough. The one really common and simple case where it’s not good enough is single-threaded code. Anything else where you need to optimize, you really do need to understand the pattern you’re trying to optimize for (and the environment(s) you’re running under) and write your own function.
>
>
I was playing around with deque today, and there were a couple of
operations I wanted to do, that can't really be done efficiently with deque
because of its implementation.
I was iterating through items of a deque, and in some cases I wanted to
delete an item that I've found. As far as I understand, this is an
operation that should be O(1) in a linked list, but Python only provides an
O(N) way to do that, which is `del d[i]`.
The same can be said for inserting items in the middle.
What do you think about adding this to `deque`? The API will be tricky,
admittedly, because you'll have to save some kind of reference to a cell in
the deque.
Thanks,
Ram.
I wanna propose making generators even weirder!
so, extended continue is an oldie:
https://www.python.org/dev/peps/pep-0342/#the-extended-continue-statement
it'd allow one to turn:
yield from foo
into:
for bar in foo:
continue (yield bar)
but what's this extended for-else? well, currently you have for-else:
for x, y, z in zip(a, b, c):
...
else:
pass
and this works. you get the stuff from the iterators, and if you break
the loop, the else doesn't run. the else basically behaves like "except
StopIteration:"...
so I propose an extended for-else, that behaves like "except
StopIteration as foo:". that is, assuming we could get a zip() that
returns partial results in the StopIteration (see other threads), we
could do:
for x, y, z in zip(a, b, c):
do_stuff_with(x, y, z)
else as partial_xy:
if len(partial_xy) == 0:
x = dummy
try:
y = next(b)
except StopIteration: y = dummy
try:
z = next(c)
except StopIteration: z = dummy
if (x, y, z) != (dummy, dummy dummy):
do_stuff_with(x, y, z)
if len(partial_xy) == 1:
x, = partial_xy
y = dummy
try:
z = next(c)
except StopIteration: z = dummy
do_stuff_with(x, y, z)
if len(partial_xy) == 2:
x, y = partial_xy
z = dummy
do_stuff_with(x, y, z)
(this example is better served by zip_longest. however, it's
nevertheless a good way to demonstrate functionality, thanks to
zip_longest's (and zip's) trivial/easy to understand behaviour.)
this would enable one to turn:
return yield from foo
into:
for bar in foo:
continue (yield bar)
else as baz:
return baz
allowing one to pick apart and modify the yielded and sent parts, while
still getting access to the return values.
currently if you have an arbitrary generator, you can't modify the
yielded values without breaking send or return. in fact you can either
pass-through yield and send and collect the return, or modify yield and
forget about send and the return, or use very ugly syntax that makes
everything look lower-level than it should, to be able to pick apart
everything:
try:
bar = next(foo)
except StopIteration as exc:
baz = exc.value
else:
while True:
try:
bar = foo.send((yield bar))
except StopIteration as exc:
baz = exc.value
break
return baz
(this is exactly equivalent (if I didn't overlook anything) to the
previous endearingly simple loop with extended for-else and extended
continue)
