[Python-Dev] PEP 492: What is the real goal?

Jim J. Jewett jimjjewett at gmail.com
Wed Apr 29 19:43:29 CEST 2015


On Tue Apr 28 23:49:56 CEST 2015, Guido van Rossum quoted PEP 492:

> Rationale and Goals
> ===================
>
> Current Python supports implementing coroutines via generators (PEP
> 342), further enhanced by the ``yield from`` syntax introduced in PEP
> 380. This approach has a number of shortcomings:
>
> * it is easy to confuse coroutines with regular generators, since they
>   share the same syntax; async libraries often attempt to alleviate
>   this by using decorators (e.g. ``@asyncio.coroutine`` [1]_);

So?  PEP 492 never says what coroutines *are* in a way that explains
why it matters that they are different from generators.

Do you really mean "coroutines that can be suspended while they wait
for something slow"?

As best I can guess, the difference seems to be that a "normal"
generator is using yield primarily to say:

    "I'm not done; I have more values when you want them",

but an asynchronous (PEP492) coroutine is primarily saying:

    "This might take a while, go ahead and do something else meanwhile."

> As shown later in this proposal, the new ``async
> with`` statement lets Python programs perform asynchronous calls when
> entering and exiting a runtime context, and the new ``async for``
> statement makes it possible to perform asynchronous calls in iterators.

Does it really permit *making* them, or does it just signal that you
will be waiting for them to finish processing anyhow, and it doesn't
need to be a busy-wait?

As nearly as I can tell, "async with" doesn't start processing the
managed block until the "asynchronous" call finishes its work -- the
only point of the async is to signal a scheduler that the task is
blocked.

Similarly, "async for" is still linearized, with each step waiting
until the previous "asynchronous" step was not merely launched, but
fully processed.  If anything, it *prevents* within-task parallelism.

> It uses the ``yield from`` implementation with an extra step of
> validating its argument.  ``await`` only accepts an *awaitable*, which
> can be one of:

What justifies this limitation?

Is there anything wrong awaiting something that eventually uses
"return" instead of "yield", if the "this might take a while" signal
is still true?  Is the problem just that the current implementation
might not take proper advantage of task-switching?

>   Objects with ``__await__`` method are called *Future-like* objects in
>   the rest of this PEP.
>
>   Also, please note that ``__aiter__`` method (see its definition
>   below) cannot be used for this purpose.  It is a different protocol,
>   and would be like using ``__iter__`` instead of ``__call__`` for
>   regular callables.
>
>   It is a ``TypeError`` if ``__await__`` returns anything but an
>   iterator.

What would be wrong if a class just did __await__ = __anext__  ?
If the problem is that the result of __await__ should be iterable,
then why isn't __await__ = __aiter__ OK?

> ``await`` keyword is defined differently from ``yield`` and ``yield
> from``.  The main difference is that *await expressions* do not require
> parentheses around them most of the times.

Does that mean

"The ``await`` keyword has slightly higher precedence than ``yield``,
so that fewer expressions require parentheses"?

>     class AsyncContextManager:
>         async def __aenter__(self):
>             await log('entering context')

Other than the arbitrary "keyword must be there" limitations imposed
by this PEP, how is that different from:

     class AsyncContextManager:
         async def __aenter__(self):
             log('entering context')

or even:

     class AsyncContextManager:
         def __aenter__(self):
             log('entering context')

Will anything different happen when calling __aenter__ or log?
Is it that log itself now has more freedom to let other tasks run
in the middle?


> It is an error to pass a regular context manager without ``__aenter__``
> and ``__aexit__`` methods to ``async with``.  It is a ``SyntaxError``
> to use ``async with`` outside of a coroutine.

Why?  Does that just mean they won't take advantage of the freedom
you offered them?  Or are you concerned that they are more likely to
cooperate badly with the scheduler in practice?

> It is a ``TypeError`` to pass a regular iterable without ``__aiter__``
> method to ``async for``.  It is a ``SyntaxError`` to use ``async for``
> outside of a coroutine.

The same questions about why -- what is the harm?

> The following code illustrates new asynchronous iteration protocol::
>
>     class Cursor:
>         def __init__(self):
>             self.buffer = collections.deque()
>
>         def _prefetch(self):
>             ...
>
>         async def __aiter__(self):
>             return self
>
>         async def __anext__(self):
>             if not self.buffer:
>                 self.buffer = await self._prefetch()
>                 if not self.buffer:
>                     raise StopAsyncIteration
>             return self.buffer.popleft()
>
> then the ``Cursor`` class can be used as follows::
>
>     async for row in Cursor():
>         print(row)

Again, I don't see what this buys you except that a scheduler has
been signaled that it is OK to pre-empt between rows.  That is worth
signaling, but I don't see why a regular iterator should be
forbidden.

> For debugging this kind of mistakes there is a special debug mode in
> asyncio, in which ``@coroutine`` decorator wraps all functions with a
> special object with a destructor logging a warning.
...
> The only problem is how to enable these debug capabilities.  Since
> debug facilities should be a no-op in production mode, ``@coroutine``
> decorator makes the decision of whether to wrap or not to wrap based on
> an OS environment variable ``PYTHONASYNCIODEBUG``.

So the decision is made at compile-time, and can't be turned on later?
Then what is wrong with just offering an alternative @coroutine that
can be used to override the builtin?

Or why not just rely on set_coroutine_wrapper entirely, and simply
set it to None (so no wasted wrappings) by default?


-jJ

--

If there are still threading problems with my replies, please
email me with details, so that I can try to resolve them.  -jJ


More information about the Python-Dev mailing list