Python-ideas May 2015

python-ideas@python.org

83 participants
57 discussions

Re: [Python-ideas] async/await and synchronous code (and PEP492 ?)

by Koos Zevenhoven

Hi all, I noticed a typo in my first email (had written__aenter__ instead of __enter__). I fixed the typo below. -- Koos On 5.5.2015 16:55, Koos Zevenhoven wrote: > > Relation to PEP492: > > This of course still needs more thinking, but I wanted to post it here > now in case there is desire to prepare for something like this already > in PEP492. It is not completely clear if/how this would need to affect > PEP492, but some things come to mind. For example, this could > potentially remove the need for __aenter__, __aiter__, etc. or even > "async for" and "async with". If __enter__ is defined as "async def", > then a with statement would do an "await" on it, and the context > manager would have __async__.framework (or whatever it would be > called) available, for determining what behavior is appropriate. >

5 years

async/await and synchronous code (and PEP492 ?)

by Koos Zevenhoven

Hi all! I am excited about seeing what's going on with asyncio and PEP492 etc. I really like that Python is becoming more suitable for the increasing amount of async code and that the distinction between async functions and generators is increasing. In addition, however, I would also like to see the async functions and methods come even closer to regular functions and methods. This is something that is keeping me from using asyncio at the moment even if I would like to. Below I'll try to explain what and why, and a little bit of how. If it is not clear, please ask :) Motivation: One of the best things about asyncio and coroutines/async functions is that you can write asynchronous code as if it were synchronous, the difference in many places being just the use of "await" ("yield from") when calling something that may end up doing IO (somewhere down the function call chain) and that the code is run from an event loop. When writing a package that does IO, you have the option to make it either synchronous or asynchronous. Regardless of the choice, the code will look roughly the same. But what if you want to be able to do both? Should you maintain two versions, one with "async" and "await" everywhere and one without? Besides the keywords "async" and "await", async code of course differs from synchronous code by the functions/coroutines that are used for IO at the end of the function call chain. Here, I mean the end (close to) where the "yield" expressions are hidden in the async versions. At the other end of the calling chain, async code needs the event loop and associated framework (almost always asyncio?) which hides all the async scheduling fanciness etc. I'm not sure about the terminology, but I will use "L end" and "Y end" to refer to the two ends here. (L for event Loop; Y for Yield) The Y and L ends need to be compatible with each other for the code to work. While asyncio and the standard library might provide both ends in many cases, there can also be situations where a package would want to work with different combinations of L and Y end, or completely without an event loop, i.e. synchronously. In a very simple example, one might want to wrap different implementations of sleep() in a function that would pick the right one depending on the context. Perhaps something like this: async def any_sleep(seconds): if __async__.framework is None: time.sleep(1) elif __async__.framework is asyncio: await asyncio.sleep(1) else: raise RuntimeError("Was called with an unsupported async framework.") [You could of course replace sleep() with socket IO or whatever, but sleep is nice and simple. Also, a larger library would probably have a whole chain of async functions and methods before calling something like this] But if await is only allowed inside "async def", then how can any_sleep() be conveniently run in non-async code? Also, there is nothing like __async__.framework. Below, I describe what I think a potential solution might look like. Potential solution: This is simplified version; for instance, as "awaitables", I consider only async function objects here. I describe the idea in three parts: (1) next(...): Add a keyword argument "async_framework" (or whatever) to next(...) with a default value of None. When an async framework, typically asyncio, starts an async function object (coroutine) with a call to next(...), it would do something like next(coro, async_framework = asyncio). Here, asyncio could of course be replaced with any object that identifies the framework. This information would then be somehow attached to the async function object. (2) __async__.framework or something similar: Add something like __async__ that has an attribute such as .framework that allows the code inside the async function to access the information passed to next(...) by the framework (L end) using the keyword argument of next [see (1)]. (3) Generalized "await": [When the world is ready:] Allow using "await" anywhere, not just within async functions. Inside async functions, the behavior of "await" would be the same as in PEP492, with the addition that it would somehow propagate the __async__.framework value to the awaited coroutine. Outside async functions, "await" would do roughly the same as this function: def await(async_func_obj): try: next(async_func_obj) # same as next(async_func_obj, async_framework = None) except StopIteration as si: return si.value raise RuntimeError("The function does not support synchronous execution") (This function would, of course, work in Python 3.4, but it would be mostly useless because the async functions would not know that they are being called in a 'synchronous program'. IIUC, this *function* would be valid even with PEP492, but having this as a function would be ugly in the long run.) Some random thoughts: With this addition to Python, one could write libraries that work both async and non-async. When await is not inside async def, one would expect it to potentially do blocking IO, just like an await inside async def would suggest that there is a yield/suspend somewhere in there. For testing, I tried to see if there is a reasonable way to make a hack with __async__.framework that could be set by next(), but did not find an obvious way. For instance, coro.gi_frame.f_locals is read-only, I believe. An alternative to this approach could be that await would implicitly start a temporary event loop for running the coroutine, but how would it know which event loop? This might also have a huge performance overhead. Relation to PEP492: This of course still needs more thinking, but I wanted to post it here now in case there is desire to prepare for something like this already in PEP492. It is not completely clear if/how this would need to affect PEP492, but some things come to mind. For example, this could potentially remove the need for __aenter__, __aiter__, etc. or even "async for" and "async with". If __aenter__ is defined as "async def", then a with statement would do an "await" on it, and the context manager would have __async__.framework (or whatever it would be called) available, for determining what behavior is appropriate. Was this clear enough to understand which problem(s) this would be solving and how? I'd be happy to hear about any thoughts on this :). Best regards, Koos

5 years

Support 1.x notation in version specifiers

by anatoly techtonik

pip team said they won't support setting limit for major version of package being installed in the way below until it is supported by PEP 440. pip install patch==1.x The current way ==1.* conflicts with system shell expansion and the other way is not known / not intuitive. https://github.com/pypa/pip/issues/2737#issuecomment-97621684 -- anatoly t.

5 years

More general "for" loop handling

by Todd

Looking at pep 492, it seems to me the handling of "for" loops has use outside of just asyncio. The primary use-case I can think of is multiprocessing and multithreading. For example, you could create a multiprocessing pool, and let the pool handle the items in a "for" loop, like so: from multiprocessing import Pool mypool = Pool(10, maxtasksperchild=2) mypool for item in items: do_something_here do_something_else do_yet_another_thing Or something similar with third-party modules: from greenlet import greenlet greenlet for item in items: do_something_here do_something_else do_yet_another_thing Of course this sort of thing is possible with iterators and maps today, but I think a lot of the same advantages that apply to asyncio also apply to these sorts of cases. So I think that, rather than having a special keyword just for asyncio, I think it would be better to have a more flexible approach. Perhaps something like a "__for__" magic method that lets a class implement "for" loop handling, along with the corresponding changes in how the language processes the "for" loop.

5 years

awaiting iterables

by Ron Adam

(I had posted this in the "more general 'for' loop" thread, but this really is a different idea from that.) Initialising several coroutines at once still doesn't seem clear/clean to me. Here is what I would like. values = awaiting [awaitable, awaitable, ...] a, b, ... = awaiting (awaitable, awaitable, ...) This doesn't have the issues of order because a list of values is returned with the same order of the awaitables. But the awaitables are scheduled in parallel. A regular for loop could still do these in order, but would pause when it gets to a values that hasn't returned/resolved yet. That would probably be expected. for x in awaiting [awaitable, awaitable, ...]: print(x) X is printed in the order of the awaitables. Awaiting sets would be different... they are unordered. So we could use a set and get the items that become available as they become available... for x in awaiting {awaitable, awaitable, ...}: print(x) x would print in an arbitrary order, but that would be what I would expect here. The body could have await calls in it, and so it could cooperate along with the awaiting set of awaitablers. Of course if the for body is only a few statements, that probably wouldn't make much difference. This seems like it's both explicit and simple to think about. It also seems like it might not be that hard to do, I think most of the parts are already worked out. One option is to allow await to work with iterables in this way. But the awaiting keyword would make the code clearer and error messages nicer. The last piece of the puzzle is how to specify the current coroutine manager/runner. import asyncio with asyncio.coroutine_loop(): main() That seems simple enough. It prettty much abstracts out all the coroutine specific stuff to three keyword. async, await, and awaiting. Are async for and async with needed if we have awaiting? Can they be impelented in terms of awaiting? import asyncio async def factorial(name, number): f = 1 for i in range(2, number+1): print("Task %s: Compute factorial(%s)..." % (name, i)) await yielding() f *= i print("Task %s: factorial(%s) = %s" % (name, number, f)) with asyncio.coroutine_loop(): awaiting [ factorial("A", 2), factorial("B", 3), factorial("C", 4)] Compared to the example in asyncio docs... import asyncio @asyncio.coroutine def factorial(name, number): f = 1 for i in range(2, number+1): print("Task %s: Compute factorial(%s)..." % (name, i)) yield from asyncio.sleep(1) f *= i print("Task %s: factorial(%s) = %s" % (name, number, f)) loop = asyncio.get_event_loop() tasks = [ asyncio.async(factorial("A", 2)), asyncio.async(factorial("B", 3)), asyncio.async(factorial("C", 4))] loop.run_until_complete(asyncio.wait(tasks)) loop.close() Cheers, Ron

5 years

PEP 492 terminology - (native) coroutine objects

by Oscar Benjamin

Looking through PEP 492 I dislike the terminology. With generators I can do: >>> def f(): yield ... >>> g = f() >>> g <generator object f at 0x7fb81dadc798> So f is a generator function and g is generator. That's all pretty clear and "generator" is IMO a great name. The term "generator function" is also unambiguous for f. With PEP 492 it seems that I would get something like: >>> async def af(): pass >>> ag = af() >>> ag <coroutine_object object af at 0x7fb81dadc828> According to the terminology in the PEP af is a coroutine but since the word coroutine also refers to some generators in a looser sense I should use "native coroutine" to disambiguate. ag is a "coroutine object" but again I should use "native coroutine object" to explicitly name the type because without the word "native" it also refers to other things. I think that if the terminology is already ambiguous with related language constructs before it's introduced then it would be better to change it now. The word coroutine (without "native" modifier) is used by the PEP to refer to both generator based coroutines and the new async def functions. I think it's reasonable to use the word as a generic term in this sense. Python's idea of coroutines (both types) doesn't seem to match with the pre-existing general definitions but they are at least generalisations of functions so it can be reasonable to describe them that way loosely. Greg's suggestion to call an async def function (af above) an "async function" seems like a big improvement. It clearly labels the purpose: a function for use in a asynchronous execution probably with the asyncio module. It also matches directly with the syntax: a function prefixed with the word "async". There would be no ambiguity between which of af or ag is referred to by the term. It seems harder to think of a good name for ag though. ag is a wrapper around a suspended call stack with methods to resume the stack so describing what is is doesn't lead to anything helpful. OTOH the purpose of ag is often described as implementing a "minithread" so the word "minithread" makes intuitive sense to me. That way async code is done by writing async functions that return minithreads. An event loop schedules the minthreads etc. (I'm trying to imagine explaining this to someone...) I'm not sure what the best word is for a coroutine object but the current terminology clearly has room for improvement. For a start using the word "object" in the name of a type is a bit rubbish in a language where everything is an object. Worse the PEP is reusing words that have already been used with different meanings so that it's already ambiguous. A concrete builtin language type such as the coroutine object deserves a good, short, one-word name that will not be confused with other things. -- Oscar

5 years, 1 month

Re: [Python-ideas] Top 10 Python modules that need a redesign Was: Geo coordinates conversion in stdlib

by Alexander Belopolsky

On Mon, Mar 23, 2015 at 2:08 AM, anatoly techtonik <techtonik(a)gmail.com> wrote: > > That's nice to know, but IIRC datetime is from the top 10 Python > modules that need a redesign. Things contained therein doesn't pass > human usability check, and are not used as a result. Where have you been when PEP 3108 was discussed? I have not seen any other list of Python modules that needed a redesign, so I cannot tell what's on your top ten list. Speaking of the datetime module, in what sense does it not "pass human usability check"? It does have a few quirks, for example I would rather see date accept a single argument in the constructor which may be a string, another date or a tuple, but I am not even sure this desire is shared by many other humans. It would be nice if datetime classes were named in CamelCase according to PEP 8 conventions, but again this is a very minor quirk. In my view, if anyone is to blame for the "human usability" of the datetime module, it would be Pope Gregory XIII, Benjamin Franklin and scores of unnamed astronomers who made modern timekeeping such a mess.

5 years, 1 month

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

Python-ideas May 2015