[Python-Dev] PEP 492: What is the real goal?

[Yury Selivanov]

Jim,

On 2015-04-30 2:41 PM, Jim J. Jewett wrote:
[...]
>>> Does it really permit *making* them [asynchronous calls], or does
>>> it just signal that you will be waiting for them to finish processing
>>> anyhow, and it doesn't need to be a busy-wait?
>> I does.
> Bad phrasing on my part.  Is there anything that prevents an
> asynchronous call (or waiting for one) without the "async with"?
>
> If so, I'm missing something important.  Either way, I would
> prefer different wording in the PEP.

Yes, you can't use 'yield from' in __exit__/__enter__
in current Python.

>>>> It uses the ``yield from`` implementation with an extra step of
>>>> validating its argument.  ``await`` only accepts an *awaitable*,
>>>> which can be one of:
>>> What justifies this limitation?
>> We want to avoid people passing regular generators and random
>> objects to 'await', because it is a bug.
> Why?
>
> Is it a bug just because you defined it that way?
>
> Is it a bug because the "await" makes timing claims that an
> object not making such a promise probably won't meet?  (In
> other words, a marker interface.)
>
> Is it likely to be a symptom of something that wasn't converted
> correctly, *and* there are likely to be other bugs caused by
> that same lack of conversion?

Same as 'yield from' is expecting an iterable, await
is expecting an awaitable.  That's the protocol.

You can't pass random objects to 'with' statements,
'yield from', 'for..in', etc.

If you write

    def gen(): yield 1
    await gen()

then it's a bug.

>
>> For coroutines in PEP 492:
>> __await__ = __anext__ is the same as __call__ = __next__
>> __await__ = __aiter__ is the same as __call__ = __iter__
> That tells me that it will be OK sometimes, but will usually
> be either a mistake or an API problem -- and it explains why.
>
> Please put those 3 lines in the PEP.

There is a line like that:
https://www.python.org/dev/peps/pep-0492/#await-expression
Look for "Also, please note..." line.

>> This is OK. The point is that you can use 'await log' in
>> __aenter__.  If you don't need awaits in __aenter__ you can
>> use them in __aexit__. If you don't need them there too,
>> then just define a regular context manager.
> Is it an error to use "async with" on a regular context manager?
> If so, why?  If it is just that doing so could be misleading,
> then what about "async with mgr1, mgr2, mgr3" -- is it enough
> that one of the three might suspend itself?

'with' requires an object with __enter__ and __exit__

'async with' requires an object with __aenter__ and __aexit__

You can have an object that implements both interfaces.

>
>>>      class AsyncContextManager:
>>>          def __aenter__(self):
>>>              log('entering context')
>
>> __aenter__ must return an awaitable
> Why?  Is there a fundamental reason, or it is just to avoid the
> hassle of figuring out whether or not the returned object is a
> future that might still need awaiting?

The fundamental reason why 'async with' is proposed is because
you can't suspend execution in __enter__ and __exit__.
If you need to suspend it there, use 'async with' and
its __a*__ methods, but they have to return awaitable
(see https://www.python.org/dev/peps/pep-0492/#new-syntax
and look what 'async with' is semantically equivalent to)

>
> Is there an assumption that the scheduler will let the thing-being
> awaited run immediately, but look for other tasks when it returns,
> and a further assumption that something which finishes the whole
> task would be too slow to run right away?
>
>> It doesn't make any sense in using 'async with' outside of a
>> coroutine.  The interpeter won't know what to do with them:
>> you need an event loop for that.
> So does the PEP also provide some way of ensuring that there is
> an event loop?  Does it assume that self-suspending coroutines
> will only ever be called by an already-running event loop
> compatible with asyncio.get_event_loop()?  If so, please make
> these contextual assumptions explicit near the beginning of the PEP.

You need some kind of loop, but it doesn't have to the one
from asyncio.  There is at least one place in the PEP where
it's mentioned that the PEP introduses a generic concept
that can be used by asyncio *and* other frameworks.

>
>
>>>> It is a ``TypeError`` to pass a regular iterable without ``__aiter__``
>>>> method to ``async for``.  It is a ``SyntaxError`` to use ``async for``
>>>> outside of a coroutine.
>>> The same questions about why -- what is the harm?
> I can imagine that as an implementation detail, the async for wouldn't
> be taken advtange of unless it was running under an event loop that
> knew to look for "aync for" as suspension points.

Event loop doesn't need to know anything about 'async with'
and 'async for'. For loop it's always one thing -- something
is awaiting somewhere for some result.

>
> I'm not seeing what the actual harm is in either not happening to
> suspend (less efficient, but still correct), or in suspending between
> every step of a regular iterator (because, why not?)
>
>
>>>> For debugging this kind of mistakes there is a special debug mode in
>>>> asyncio, in which ``@coroutine``
> ...
>>>> decorator makes the decision of whether to wrap or not to wrap based on
>>>> an OS environment variable ``PYTHONASYNCIODEBUG``.
> (1)  How does this differ from the existing asynchio.coroutine?
> (2)  Why does it need to have an environment variable?  (Sadly,
>       the answer may be "backwards compatibility", if you're really
>       just specifying the existing asynchio interface better.)
> (3)  Why does it need [set]get_coroutine_wrapper, instead of just
>       setting the asynchio.coroutines.coroutine attribute?
> (4)  Why do the get/set need to be in sys?

That section describes some hassles we had in asyncio to
enable better debugging.

(3) because it allows to enable debug selectively when
we need it

(4) because it's where functions like 'set_trace' live.
set_coroutine_wrapper() also requires some modifications
in the eval loop, so sys looks like the right place.


>
> Is the intent to do anything more than preface execution with:
>
> import asynchio.coroutines
> asynchio.coroutines._DEBUG = True

This won't work, unfortunately.  You need to set the
debug flag *before* you import asyncio package (otherwise
we would have an unavoidable performance cost for debug
features).  If you enable it after you import asyncio,
then asyncio itself won't be instrumented.  Please
see the implementation of asyncio.coroutine for details.

set_coroutine_wrapper solves these problems.

Yury