[Python-Dev] PEP 492: No new syntax is required

[Yury Selivanov]

On 2015-04-26 5:48 PM, Mark Shannon wrote:
>
>
> On 26/04/15 21:40, Yury Selivanov wrote:
>> Hi Mark,
>>
>> On 2015-04-26 4:21 PM, Mark Shannon wrote:
>>> Hi,
>>>
>>> I was looking at PEP 492 and it seems to me that no new syntax is
>>> required.
>>
>> Mark, all your points are explained in the PEP in a great detail:
> I did read the PEP. I do think that clarifying the distinction between 
> coroutines and 'normal' generators is a good idea. Adding stuff to the 
> standard library to help is fine. I just don't think that any new 
> syntax is necessary.

Well, unfortunately, I can't explain why the new syntax is necessary 
better than it is already explained in the PEP, sorry.

>
>>
>>>
>>> Looking at the code, it does four things; all of which, or a
>>> functional equivalent, could be done with no new syntax.
>>
>> Yes, everything that the PEP proposes can be done without new syntax.
>> That's how people use asyncio right now, with only what we have in 3.4.
>>
>> But it's hard.  Iterating through something asynchronously? Write a
>> 'while True' loop.  Instead of 1 line you now have 5 or 6.  Want to
>> commit your database transaction?  Instead of 'async with' you will
>> write 'try..except..finally' block, with a very high probability to
>> introduce a bug, because you don't rollback or commit properly or
>> propagate exception.
> I don't see why you can't do transactions using a 'with' statement.

Because you can't use 'yield from' in __exit__. And allowing it there 
isn't a very good idea.

>>
>>> 1. Make a normal function into a generator or coroutine. This can be
>>> done with a decorator.
>>
>> https://www.python.org/dev/peps/pep-0492/#rationale-and-goals
> states that """
> it is not possible to natively define a coroutine which has no yield 
> or yield from statement
> """
> which is just not true.

It *is* true.  Please note the word "natively".

See coroutine decorator from asyncio:
https://github.com/python/cpython/blob/master/Lib/asyncio/coroutines.py#L130

To turn a regular function into a coroutine you have three options:

1. wrap the function;

2. use "if 0: yield" terrible hack;

3. flip CO_GENERATOR flag for the code object (which is CPython 
implementation detail); also terrible hack.

@coroutine decorator is great because we can use asyncio even in 3.3. 
But it's

a) easy to forget

b) hard to statically analyze

c) hard to explain why functions decorated with it must only contain 
'yield from' instead of 'yield'

d) few more reasons listed in the PEP

>
>> https://www.python.org/dev/peps/pep-0492/#debugging-features
> Requires the addition of the CO_COROUTINE flag, not any new keywords.

True.  But the importance of new keywords is covered in other sections.
>
>> https://www.python.org/dev/peps/pep-0492/#importance-of-async-keyword
> Seems to be repeating the above.
>>
>>> 2. Support a parallel set of special methods starting with 'a' or
>>> 'async'. Why not just use the current set of special methods?
>>
>> Because you can't reuse them.
>>
>> https://www.python.org/dev/peps/pep-0492/#why-not-reuse-existing-for-and-with-statements 
>>
> Which seems back to front. The argument is that existing syntax 
> constructs cannot be made to work with asynchronous objects. Why not 
> make the asynchronous objects work with the existing syntax?

Because a) it's not possible; b) the point is to make suspend points 
visible in the code. That's one of the key principles of asyncio and 
other frameworks.

>
>>
>> https://www.python.org/dev/peps/pep-0492/#why-not-reuse-existing-magic-names 
>>
> The argument here relies on the validity of the previous points.
>>
>>
>>> 3. "await". "await" is an operator that takes one argument and
>>> produces a single result, without altering flow control and can thus
>>> be replaced by an function.
>>
>> It can't be replaced by a function. Only if you use greenlets or
>> Stackless Python.
> Why not? The implementation of await is here:
> https://github.com/python/cpython/compare/master...1st1:await#diff-23c87bfada1d01335a3019b9321502a0R642 
>
> which clearly could be made into a function.

Implementation of 'await' requires YIELD_FROM opcode. As far as I know 
functions in Python can't push opcodes to the eval loop while running.  
The only way to do what you propose is to use greenlets or merge 
stackless into cpython.

>>
>>> 4. Asynchronous with statement. The PEP lists the equivalent as "with
>>> (yield from xxx)" which doesn't seem so bad.
>>
>> There is no equivalent to 'async with'. "with (yield from xxx)" only
>> allows you to suspend execution
>> in __enter__ (and it's not actually in __enter__, but in a coroutine
>> that returns a context manager).
>>
>> https://www.python.org/dev/peps/pep-0492/#asynchronous-context-managers-and-async-with 
>>
>> see "New Syntax" section to see what 'async with' is equivalent too.
> Which, by comparing with PEP 343, can be translated as:
>     with expr as e:
>         e = await(e)
>         ...
>>
>>>
>>> Please don't add unnecessary new syntax.
>>
>>
>> It is necessary.
> This isn't an argument, it's just contradiction ;)
>
>  Perhaps you haven't spent a lot of time maintaining
>> huge code-bases developed with frameworks like asyncio, so I understand
>> why it does look unnecessary to you.
> This is a good reason for clarifying the distinction between 'normal' 
> generators and coroutines. It is not, IMO, justification for burdening 
> the language (and everyone porting Python 2 code) with extra syntax.
>

Everybody loves that Python is minimalistic, extremely easy to learn, 
and joyful to use.  I agree that adding new syntax shouldn't be done 
without careful consideration.  However, when new patterns and 
approaches evolve and the language isn't ready for them, we have to make 
using them easier.  Otherwise we wouldn't have 'with' statements and 
many other useful concepts in Python at all.


Thanks for the feedback,

Yury