Hi all!
I am excited about seeing what's going on with asyncio and PEP492 etc. I
really like that Python is becoming more suitable for the increasing
amount of async code and that the distinction between async functions
and generators is increasing.
In addition, however, I would also like to see the async functions and
methods come even closer to regular functions and methods. This is
something that is keeping me from using asyncio at the moment even if I
would like to. Below I'll try to explain what and why, and a little bit
of how. If it is not clear, please ask :)
Motivation:
One of the best things about asyncio and coroutines/async functions is
that you can write asynchronous code as if it were synchronous, the
difference in many places being just the use of "await" ("yield from")
when calling something that may end up doing IO (somewhere down the
function call chain) and that the code is run from an event loop.
When writing a package that does IO, you have the option to make it
either synchronous or asynchronous. Regardless of the choice, the code
will look roughly the same. But what if you want to be able to do both?
Should you maintain two versions, one with "async" and "await"
everywhere and one without?
Besides the keywords "async" and "await", async code of course differs
from synchronous code by the functions/coroutines that are used for IO
at the end of the function call chain. Here, I mean the end (close to)
where the "yield" expressions are hidden in the async versions. At the
other end of the calling chain, async code needs the event loop and
associated framework (almost always asyncio?) which hides all the async
scheduling fanciness etc. I'm not sure about the terminology, but I will
use "L end" and "Y end" to refer to the two ends here. (L for event
Loop; Y for Yield)
The Y and L ends need to be compatible with each other for the code to
work. While asyncio and the standard library might provide both ends in
many cases, there can also be situations where a package would want to
work with different combinations of L and Y end, or completely without
an event loop, i.e. synchronously.
In a very simple example, one might want to wrap different
implementations of sleep() in a function that would pick the right one
depending on the context. Perhaps something like this:
async def any_sleep(seconds):
if __async__.framework is None:
time.sleep(1)
elif __async__.framework is asyncio:
await asyncio.sleep(1)
else:
raise RuntimeError("Was called with an unsupported async
framework.")
[You could of course replace sleep() with socket IO or whatever, but
sleep is nice and simple. Also, a larger library would probably have a
whole chain of async functions and methods before calling something like
this]
But if await is only allowed inside "async def", then how can
any_sleep() be conveniently run in non-async code? Also, there is
nothing like __async__.framework. Below, I describe what I think a
potential solution might look like.
Potential solution:
This is simplified version; for instance, as "awaitables", I consider
only async function objects here. I describe the idea in three parts:
(1) next(...):
Add a keyword argument "async_framework" (or whatever) to next(...) with
a default value of None. When an async framework, typically asyncio,
starts an async function object (coroutine) with a call to next(...), it
would do something like next(coro, async_framework = asyncio). Here,
asyncio could of course be replaced with any object that identifies the
framework. This information would then be somehow attached to the async
function object.
(2) __async__.framework or something similar:
Add something like __async__ that has an attribute such as .framework
that allows the code inside the async function to access the information
passed to next(...) by the framework (L end) using the keyword argument
of next [see (1)].
(3) Generalized "await":
[When the world is ready:] Allow using "await" anywhere, not just within
async functions. Inside async functions, the behavior of "await" would
be the same as in PEP492, with the addition that it would somehow
propagate the __async__.framework value to the awaited coroutine.
Outside async functions, "await" would do roughly the same as this function:
def await(async_func_obj):
try:
next(async_func_obj) # same as next(async_func_obj,
async_framework = None)
except StopIteration as si:
return si.value
raise RuntimeError("The function does not support synchronous
execution")
(This function would, of course, work in Python 3.4, but it would be
mostly useless because the async functions would not know that they are
being called in a 'synchronous program'. IIUC, this *function* would be
valid even with PEP492, but having this as a function would be ugly in
the long run.)
Some random thoughts:
With this addition to Python, one could write libraries that work both
async and non-async. When await is not inside async def, one would
expect it to potentially do blocking IO, just like an await inside async
def would suggest that there is a yield/suspend somewhere in there.
For testing, I tried to see if there is a reasonable way to make a hack
with __async__.framework that could be set by next(), but did not find
an obvious way. For instance, coro.gi_frame.f_locals is read-only, I
believe.
An alternative to this approach could be that await would implicitly
start a temporary event loop for running the coroutine, but how would it
know which event loop? This might also have a huge performance overhead.
Relation to PEP492:
This of course still needs more thinking, but I wanted to post it here
now in case there is desire to prepare for something like this already
in PEP492. It is not completely clear if/how this would need to affect
PEP492, but some things come to mind. For example, this could
potentially remove the need for __aenter__, __aiter__, etc. or even
"async for" and "async with". If __aenter__ is defined as "async def",
then a with statement would do an "await" on it, and the context manager
would have __async__.framework (or whatever it would be called)
available, for determining what behavior is appropriate.
Was this clear enough to understand which problem(s) this would be
solving and how? I'd be happy to hear about any thoughts on this :).
Best regards,
Koos
pip team said they won't support setting limit for major version
of package being installed in the way below until it is supported
by PEP 440.
pip install patch==1.x
The current way ==1.* conflicts with system shell expansion
and the other way is not known / not intuitive.
https://github.com/pypa/pip/issues/2737#issuecomment-97621684
--
anatoly t.
Looking at pep 492, it seems to me the handling of "for" loops has use
outside of just asyncio. The primary use-case I can think of is
multiprocessing and multithreading.
For example, you could create a multiprocessing pool, and let the pool
handle the items in a "for" loop, like so:
from multiprocessing import Pool
mypool = Pool(10, maxtasksperchild=2)
mypool for item in items:
do_something_here
do_something_else
do_yet_another_thing
Or something similar with third-party modules:
from greenlet import greenlet
greenlet for item in items:
do_something_here
do_something_else
do_yet_another_thing
Of course this sort of thing is possible with iterators and maps today, but
I think a lot of the same advantages that apply to asyncio also apply to
these sorts of cases. So I think that, rather than having a special
keyword just for asyncio, I think it would be better to have a more
flexible approach. Perhaps something like a "__for__" magic method that
lets a class implement "for" loop handling, along with the corresponding
changes in how the language processes the "for" loop.
(I had posted this in the "more general 'for' loop" thread, but this really
is a different idea from that.)
Initialising several coroutines at once still doesn't seem clear/clean to me.
Here is what I would like.
values = awaiting [awaitable, awaitable, ...]
a, b, ... = awaiting (awaitable, awaitable, ...)
This doesn't have the issues of order because a list of values is returned
with the same order of the awaitables. But the awaitables are scheduled in
parallel.
A regular for loop could still do these in order, but would pause when it
gets to a values that hasn't returned/resolved yet. That would probably be
expected.
for x in awaiting [awaitable, awaitable, ...]:
print(x)
X is printed in the order of the awaitables.
Awaiting sets would be different... they are unordered. So we could use a
set and get the items that become available as they become available...
for x in awaiting {awaitable, awaitable, ...}:
print(x)
x would print in an arbitrary order, but that would be what I would expect
here.
The body could have await calls in it, and so it could cooperate along with
the awaiting set of awaitablers. Of course if the for body is only a few
statements, that probably wouldn't make much difference.
This seems like it's both explicit and simple to think about. It also
seems like it might not be that hard to do, I think most of the parts are
already worked out.
One option is to allow await to work with iterables in this way. But the
awaiting keyword would make the code clearer and error messages nicer.
The last piece of the puzzle is how to specify the current coroutine
manager/runner.
import asyncio
with asyncio.coroutine_loop():
main()
That seems simple enough. It prettty much abstracts out all the coroutine
specific stuff to three keyword. async, await, and awaiting.
Are async for and async with needed if we have awaiting? Can they be
impelented in terms of awaiting?
import asyncio
async def factorial(name, number):
f = 1
for i in range(2, number+1):
print("Task %s: Compute factorial(%s)..." % (name, i))
await yielding()
f *= i
print("Task %s: factorial(%s) = %s" % (name, number, f))
with asyncio.coroutine_loop():
awaiting [
factorial("A", 2),
factorial("B", 3),
factorial("C", 4)]
Compared to the example in asyncio docs...
import asyncio
@asyncio.coroutine
def factorial(name, number):
f = 1
for i in range(2, number+1):
print("Task %s: Compute factorial(%s)..." % (name, i))
yield from asyncio.sleep(1)
f *= i
print("Task %s: factorial(%s) = %s" % (name, number, f))
loop = asyncio.get_event_loop()
tasks = [
asyncio.async(factorial("A", 2)),
asyncio.async(factorial("B", 3)),
asyncio.async(factorial("C", 4))]
loop.run_until_complete(asyncio.wait(tasks))
loop.close()
Cheers,
Ron
Looking through PEP 492 I dislike the terminology.
With generators I can do:
>>> def f(): yield
...
>>> g = f()
>>> g
<generator object f at 0x7fb81dadc798>
So f is a generator function and g is generator. That's all pretty
clear and "generator" is IMO a great name. The term "generator
function" is also unambiguous for f.
With PEP 492 it seems that I would get something like:
>>> async def af(): pass
>>> ag = af()
>>> ag
<coroutine_object object af at 0x7fb81dadc828>
According to the terminology in the PEP af is a coroutine but since
the word coroutine also refers to some generators in a looser sense I
should use "native coroutine" to disambiguate. ag is a "coroutine
object" but again I should use "native coroutine object" to explicitly
name the type because without the word "native" it also refers to
other things.
I think that if the terminology is already ambiguous with related
language constructs before it's introduced then it would be better to
change it now. The word coroutine (without "native" modifier) is used
by the PEP to refer to both generator based coroutines and the new
async def functions. I think it's reasonable to use the word as a
generic term in this sense. Python's idea of coroutines (both types)
doesn't seem to match with the pre-existing general definitions but
they are at least generalisations of functions so it can be reasonable
to describe them that way loosely.
Greg's suggestion to call an async def function (af above) an "async
function" seems like a big improvement. It clearly labels the purpose:
a function for use in a asynchronous execution probably with the
asyncio module. It also matches directly with the syntax: a function
prefixed with the word "async". There would be no ambiguity between
which of af or ag is referred to by the term.
It seems harder to think of a good name for ag though. ag is a wrapper
around a suspended call stack with methods to resume the stack so
describing what is is doesn't lead to anything helpful. OTOH the
purpose of ag is often described as implementing a "minithread" so the
word "minithread" makes intuitive sense to me. That way async code is
done by writing async functions that return minithreads. An event loop
schedules the minthreads etc. (I'm trying to imagine explaining this
to someone...)
I'm not sure what the best word is for a coroutine object but the
current terminology clearly has room for improvement. For a start
using the word "object" in the name of a type is a bit rubbish in a
language where everything is an object. Worse the PEP is reusing words
that have already been used with different meanings so that it's
already ambiguous. A concrete builtin language type such as the
coroutine object deserves a good, short, one-word name that will not
be confused with other things.
--
Oscar
On Mon, Mar 23, 2015 at 2:08 AM, anatoly techtonik <techtonik(a)gmail.com>
wrote:
>
> That's nice to know, but IIRC datetime is from the top 10 Python
> modules that need a redesign. Things contained therein doesn't pass
> human usability check, and are not used as a result.
Where have you been when PEP 3108 was discussed? I have not seen any other
list of Python modules that needed a redesign, so I cannot tell what's on
your top ten list.
Speaking of the datetime module, in what sense does it not "pass human
usability check"? It does have a few quirks, for example I would rather
see date accept a single argument in the constructor which may be a string,
another date or a tuple, but I am not even sure this desire is shared by
many other humans. It would be nice if datetime classes were named in
CamelCase according to PEP 8 conventions, but again this is a very minor
quirk.
In my view, if anyone is to blame for the "human usability" of the datetime
module, it would be Pope Gregory XIII, Benjamin Franklin and scores of
unnamed astronomers who made modern timekeeping such a mess.
