Mailman 3 February 2020 - Python-ideas

Global thread executor
by Bar Harel Feb. 15, 2020

Feb. 15, 2020

Howdy fellows, I had a thought of adding a global thread executor in python. It can use something like concurrent.futures, and be created on demand. There are times when I find myself needing to run just a single function in a thread, cause there's practically no other way. The function is a fast-acting one, and needs to run infrequently throughout the code. Let's take socket.getaddrinfo for example. Even in non-blocking mode, this function will still always block. Even asyncio solves it by running it on a different thread. Each program, library or piece of code that wants to use it in an asynchronous manner, will have to initialize a thread (with all the respective overhead), for the use of a single function. This can happen for plenty of libraries, each initializing their own thread for a one-night stand. If we would have had a global ThreadPoolExecutor, we could have use it exactly for that. The threads would be shared, the overhead would only occur once. Users of the executor will know that it's a limited resource that may be full at times, and as responsible programmers will not use it for infinite loops, clogging the whole system. Even if a programmer uses it irresponsibly (shame), we still have future.result(timeout) to cover us and issue a warning if we must. Any inputs would be appreciated, Bar Harel.

3 3

Traits
by Soni L. Feb. 15, 2020

Feb. 15, 2020

I'd like to see traits some day, with a syntax similar to this one: trait Trait: def x(self): raise NotImplementedError def y(self): raise NotImplementedError trait Anoher: def x(self): raise NotImplementedError def y(self): raise NotImplementedError def foo(Trait(x)): x.x() class Bar: def y(self): print("hello") impl Trait: def x(self): self.y() # resolves to Bar.y impl Another: def x(self): raise ValueError foo(Bar()) # prints 'hello' Trait(Bar()).x() # also prints 'hello' Bar().x() # AttributeError: ambiguous reference to trait method x if the trait isn't used in the function definition you get the raw object, where name conflicts between traits (but not between traits and inherent methods) result in an error about name conflicts. otherwise you get a friendly wrapper.

16 29

Asyncio Future.set_result_threadsafe
by Brian Allen Vanderburg II Feb. 13, 2020

Feb. 13, 2020

Currently asyncio.futures.Future.set_result will result in any callbacks being scheduled using loop.call_soon instead of loop.call_soon_threadsafe. However in situations where the future's result is set from a different thread, the loop might not wake up as a result of this if it is currently sleeping. def compute_something(...): loop = asyncio.get_event_loop() future = loop.create_future() manager.run_computation(future, ...) # add the computation to a thread that is running it return future async def compute_stuff(): result = await compute_something(...) loop = asyncio.get_event_loop() loop.run_until_complete(compute stuff()) loop.close() The reasoning behind it is after yielding the Future object, the ready list of the event loop is empty so when waiting for selectors the timeout value is None and it just waits. When the other thread calls set_result, the loop ready list will get updated, but it's still waiting on selectors. The other thread could call future._loop.call_soon_threadsafe(future.set_result, ...), which writes some bytes to a socket in order to wake the even loop. I'm aware there are other ways to do things like this (queues, executors, etc). It just seems that, regardless of any method used, the code holding the future with the purpose of setting a result on it should be able to be blind to other implementation details. future.set_result should be all it needs to know to call even if from another thread, or if not, a method like future.set_result_threadsafe, which would hide the details of the loop.call_soon/call_soon_threadsafe

4 4

PEP 614: Relaxing Grammar Restrictions On Decorators
by Brandt Bucher Feb. 13, 2020

Feb. 13, 2020

I have just pushed the first draft of PEP 614: https://www.python.org/dev/peps/pep-0614/ It proposes that the current decorator syntax restrictions be relaxed to allow any valid expression. Please let me know what you think! Brandt

6 12

PEP 572: Is it a good ideas that walrus operator can be used in assertions since it causes side effects?
by jdveiga＠gmail.com Feb. 13, 2020

Feb. 13, 2020

Hi, Recently the idea has come to me that walrus operator can have some issues when use in assertions. I have been taught that assertions must not produce side effects at all. This is a problem in languages such as C. But not in Python (until 3.8) since it does not allow assignment expressions. But now, for best or worst, we have the walrus operator and a code like this is possible: `python3.8 -c "assert (always := True); print(f'{always=}')"` Which outputs: ``` always=True ``` since the assertion is always true. However, if assertions are disabled: `python3.8 -Oc "assert (always := True); print(f'{always=}')"` it raises an exception: ``` Traceback (most recent call last): File "<string>", line 1, in <module> NameError: name 'always' is not defined ``` because of the side effect caused by the walrus operator. Do not you think that disabling walrus operators in assertions would be a good idea? Would be it possible? Personally, I do not see many advantages in walrus operator (versus disadvantages, of course). However, in the case of assertions, a nice and secure feature of Python has been lost without any apparent benefit. Thank you.

9 15

Re: New syntax for decorators that are compatible with both normal and async functions
by Andrew Barnert Feb. 11, 2020

Feb. 11, 2020

> On Feb 10, 2020, at 02:34, Bar Harel <bharel(a)barharel.com> wrote: > > While I'm not this idea can totally work, I do think we need a method of closing the gap between asyc coroutines and normal functions. > > As of today, async applications need to create an entirely different codebase, sometimes just by copy and paste, while changing function definitions to async and function calls to await. > > This creates a lot of work, from porting code to maintaining both variants, to just having code duplication all around. It’s a well-known problem that async is “contagious”: once you turn one function into a coroutine, you have to turn the whole call tree async up to some point where you can either block or daemonize a coroutine. Together with the fact that async operations have not just different syntax but different names and APIs than their sync equivalents, this makes it hard to port code between the two paradigms. But C# and every other language that’s borrowed the idea has the same problem, and as far as I know, nobody’s thought of a good answer yet. Just saying “this is bad, so we should do something” doesn’t get us anywhere unless that something is doable and solves the problem and we know what it is. And I don’t know if that’s the case here. The OP proposed something that can only work by impossible magic, and you suggested it would be nice if we had something actually implementable that was just as easy to use, and it definitely would be nice, but unless someone has an idea for that something… (That definitely doesn’t mean it isn’t worth pursuing. Just because nobody else has thought of it yet doesn’t mean it’s impossible, after all.) I think breaking down decorators into smaller building blocks that can be composed might be promising. If you have a 30 different decorators that all want to do something (that doesn’t block) before calling the wrapper function or coro, you can write a single `@before` that does have to duplicate its code and then 30 decorators that use `before` and don’t need any duplication. I’m not sure how far you can get with that. Certainly not all the way to the full generality of arbitrary-code decorators (although I don’t think even the OP’s magic would get use there—you still can’t put any code into the decorator that might be blocking except for the wrapped function, especially since usually the difference isn’t just a matter of await or not, but of calling a different method on a different object…). But maybe far enough to be useful? I don’t know. Another possible avenue is to drop back a level and write wrappers that work on futures. Async futures and concurrent futures duck type as lowest-common-denominator futures, and maybe some of the helpers you want can be built that way without duplicating their code? Meanwhile, for porting large existing threaded or synchronous codebases, you may be better off using the gevent model, where awaiting is internal and invisible. You can find out pretty quickly whether all of your libraries are compatible or not. If they are, you won’t get all of the advantages (e.g., with await it’s often obvious when you can remove a mutex or other sync object; with gevent, because it looks nearly identical to preemptive threading, if that mutex was necessary in the preemptive version, it still looks necessary in the gevent version even though it isn’t, unless you think it through carefully), but only having to rewrite 2% of your code instead of 60% may be worth it anyway. Just because asyncio is usually the best model for new code doesn’t mean it’s the always best model for porting all code.

5 6

Multi-threading interface idea
by Sean McIntyre Feb. 11, 2020

Feb. 11, 2020

Hi folks, I'd like to get some feedback on a multi-threading interface I've been thinking about and using for the past year or so. I won't bury the lede, see my approach here <https://gist.github.com/boxysean/3ed325ebb75db0303002f9484821e553#file-my_e…> . *Background / problem:* A couple of years ago, I inherited my company's codebase to get data into our data warehouse using an ELT approach (extract-and-loads done in python, transforms done in dbt/SQL). The codebase has dozens of python scripts to integrate first-party and third-party data from databases, FTPs, and APIs, which are run on a scheduler (typically daily or hourly). The scripts I inherited were single-threaded procedural scripts, looking like glue code, and spending most of their time in network I/O. (See example. <https://gist.github.com/boxysean/3ed325ebb75db0303002f9484821e553#file-unth…>) This got my company pretty far! As my team and I added more and more integrations with more and more data, we wanted to have faster and faster scripts to reduce our dev cycles and reduce our multi-hour nightly jobs to minutes. Because our scripts were network-bound, multi-threading was a good way to accomplish this, and so I looked into concurrent.futures (example <https://gist.github.com/boxysean/3ed325ebb75db0303002f9484821e553#file-conc…>) and asyncio (example <https://gist.github.com/boxysean/3ed325ebb75db0303002f9484821e553#file-asyn…>), but I decided against these options because: 1. It wasn't immediately apparently how to adapt my codebase to use these libraries without either some fundamental changes to our execution platform and/or reworking of our scripts from the ground up and/or adding significant lines of multi-threading code to each script. 2. I couldn't wrap my head around the async/await and future constructs particularly quickly, and I was concerned that my team would also struggle with this change. 3. I believe the procedural style glue code we have is quite easy to comprehend, which I think has a positive impact on scale. *Solution:* And so, as mentioned at the top, I designed a different interface to concurrent.futures.ThreadPoolExecutor that we are successfully using for our extract-and-load pattern, see a basic example here <https://gist.github.com/boxysean/3ed325ebb75db0303002f9484821e553#file-my_e…>. The design considerations of this interface include: - The usage is minimally-invasive to the original unthreaded approach of the codebase. (And so, teaching the library to team members has been fairly straightforward despite the multi-threaded paradigm shift.) - The @parallel.task decorator should be used to encapsulate a homogeneous method accepting different parameters. The contents of the method should be primarily I/O to achieve the concurrency gains of python multi-threading. - If no parallel.threads context manager has been entered, the @parallel.task decorator acts as a no-op (and the code runs serially). - If an environment variable is set to disable the context manager, the @parallel.task decorator acts as a no-op (and the code runs serially). - There is also an environment variable to change the number of workers provided by parallel.threads (if not hard-coded). While it's possible to return a value from a @parallel.task method, I encourage my team to use the decorator to start-and-complete work; think of writing "embarrassingly parallel" methods that can be "mapped". A couple of other things we've implemented include a "thread barrier" in the case where we want a set tasks to complete before a set of other tasks, and a decorator for factory methods to produce cached thread-local objects (helpful for ensuring thread-safe access to network clients that are not thread-safe). *Your feedback:* - I'd love to hear your thoughts on my problem and solution. - I've done a bit of research of existing libraries in PyPI and PEPs but I don't see any similar libraries; are you aware of anything? - What do you suggest I do next? I'm considering publishing it, but could use some tips on what to here! Thanks! Sean McIntyre

4 4

`raise as` to raise with current exception as cause
by Ram Rachum Feb. 11, 2020

Feb. 11, 2020

Hi, I'd like to suggest an idea, that builds on PEPs 3134 and 409. This idea came up when discussing a problem on the django-developers mailing list: https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/dj… I'll first explain my idea, and then the original problem. The idea is to add `raise as` syntax, that raises an exception while setting the currently caught exception to be the cause. It'll look like this: try: 1/0 except ZeroDivisionError: raise as ValueError('Whatever') What it does is a shorter version of this: try: 1/0 except ZeroDivisionError as error: raise ValueError('Whatever') from error Of course, this syntax could only be used inside an except clause, similarly to a blank `raise`. The `raise as` syntax was one of the rejected syntaxes in PEP 409, for a different use-case, of suppressing context: https://www.python.org/dev/peps/pep-0409/#alternatives I think that this syntax might be a good fit for this use-case. The reason I propose this, is that I had a hard time convincing the Django maintainers to use the current `raise foo from bar` syntax in cases where it's appropriate, i.e. when they wrap an exception with another exception. This is important because then the users see the correct message between the two tracebacks, which is "The above exception was the direct cause of the following exception:" rather than "During handling of the above exception, another exception occurred" which is for other cases. This is my first PR in this topic that was merged: https://github.com/django/django/pull/12263 Even though that first PR was merged, after we discussed adding this all over Django, and recommending that `raise foo from bar` be used in the style guide, Carlton Gibson changed his mind. He gave his arguments in the Django-developers thread I linked to at the top. That's sad for me, because if Django doesn't accept the new syntax, and is okay with the inaccurate "During handling of" message between exceptions, chances are low that there will be widespread adoption of the current `raise foo from bar` syntax. It's possible that introducing the simpler `raise as` would increase adoption and make users pay attention to the message between exception tracebacks. What do you think? Thanks, Ram.

14 29

Improving Traceback for Certain Errors by Expanding Source Displayed
by Henk-Jaap Wagenaar Feb. 10, 2020

Feb. 10, 2020

Dear Ideas, TL;DR include multiple lines in traceback when appropriate (expressions broken up over multiple lines) to make it easier to spot error from looking at traceback Apologies if this has been suggested/discussed before. I am suggesting that if you do: ``` [ x for x in 0 ] ``` the traceback will be: ``` Traceback (most recent call last): File "foo.py", line 1-3, in <module> [ x for x in 0 ] TypeError: 'int' object is not iterable ``` instead of the current ``` Traceback (most recent call last): File "foo.py", line 1, in <module> [ TypeError: 'int' object is not iterable ``` Similarly for when an operator is applied over different lines: ``` ( 0 & "world" ) ``` would lead to ``` Traceback (most recent call last): File "foo.py", line 2-3, in <module> 0 & "world" TypeError: unsupported operand type(s) for &: 'int' and 'str' ``` instead of ``` Traceback (most recent call last): File "foo.py", line 2-3, in <module> "hello" & TypeError: unsupported operand type(s) for &: 'int' and 'str' ``` Or when an open has been split over multiple lines (note missing comma): ``` open( "foo" "r" ) ``` to be ``` Traceback (most recent call last): File "foo.py", line 1-4, in <module> open( "foo" "r" ) FileNotFoundError: [Errno 2] No such file or directory: 'foor' ``` instead of ``` Traceback (most recent call last): File "foo.py", line 1-3, in <module> open( FileNotFoundError: [Errno 2] No such file or directory: 'foor' ``` I don't think this change would rock the world, but I think it is a nice quality-of-life improvement that would ease debugging. In particular the last example (open function call) showcases how it can make easier to spot a bug, especially if it is to do with how the expression is broken up What do people think? Any particular reason this is not possible or done? I'm guessing it would require quite a bit of restructuring in the traceback object/module and potentially have backwards compatibility implications. All the best, Henk-Jaap

5 4

PyPI should include PEP 440-compliant version strings for including in install_requires
by Soni L. Feb. 10, 2020

Feb. 10, 2020

Figuring out versioning is hard. PyPI should encourage semver by including PEP 440 version strings that you can copy and paste into install_requires. This'll encourage module authors to do their best to follow semver. And also it means I can copy and paste the things and my code won't silently break because I didn't specify dependency versioning. More specifically, it'll just have me specify dependency versioning by default.

8 11