[Python-ideas] yield from multiple iterables (was Re: The async API of the future: yield-from)

[Christian Tismer]

s/twi/two/

Sent from my Ei4Steve

On Oct 20, 2012, at 0:31, Christian Tismer <tismer at stackless.com> wrote:

> Hi Guido, Marc, all,
> 
> this is a veery promising result, telling
> me that the big Oh can in fact be
> neglected in real applications. 20 for
> twi is good!
> 
> I will of course do an analysis and find
> the parameters of the quadratic, but my
> concern is pretty much tamed. 
> 
> For me that means there will soon be
> a library that contains real generators
> and more building blocks. 
> 
> I think using those would simplify the
> design of the async API quite a lot. 
> 
> I suggest to regard current generator
> constructs as low-level helpers for
> Implementing the real concurrency
> building blocks. 
> 
> Instead of using the existing re.compile("yield (from)?") pattern, I think we can abstract
> from this now and think in terms of
> higher level constructs. 
> 
> Let's assume generators and coroutines,
> and model concurrency from that. I
> believe this unwinds the brains and
> clarifies things a lot. 
> 
> I will provide sone classes for that at
> the pycon.de sprint, unless somebody
> implements it earlier (please don't).
> 
> This email was written in non-linear
> order, so please ignore logic inversions. 
> 
> Cheers - chris
> 
> Sent from my Ei4Steve
> 
> On Oct 19, 2012, at 23:31, Guido van Rossum <guido at python.org> wrote:
> 
>> On Fri, Oct 19, 2012 at 2:15 PM, Mark Shannon <mark at hotpy.org> wrote:
>>> On 19/10/12 13:55, Christian Tismer wrote:
>>>> 
>>>> Hi Nick,
>>>> 
>>>> On 16.10.12 03:49, Nick Coghlan wrote:
>>>>> 
>>>>> On Tue, Oct 16, 2012 at 10:44 AM, Greg Ewing
>>>>> <greg.ewing at canterbury.ac.nz> wrote:
>>>>>> 
>>>>>> My original implementation of yield-from actually *did* avoid
>>>>>> this, by keeping a C-level pointer chain of yielding-from frames.
>>>>>> But that part was ripped out at the last minute when someone
>>>>>> discovered that it had a detrimental effect on tracebacks.
>>>>>> 
>>>>>> There are probably other ways the traceback problem could be
>>>>>> fixed, so maybe we will get this optimisation back one day.
>>>>> 
>>>>> Ah, I thought I remembered something along those lines. IIRC, it was a
>>>>> bug report on one of the alphas that prompted us to change it.
>>>> 
>>>> I was curious and searched quite a lot.
>>>> It was v3.3.0a1 from 2012-03-15 as a reaction to #14230 and #14220
>>>> from Marc Shannon, patched by Benjamin.
>>>> 
>>>> Now I found the original implementation. That looks very much
>>>> as I'm thinking it should be.
>>>> 
>>>> Quite a dramatic change which works well, but really seems to remove
>>>> what I would call "now I can emulate most of Stackless" efficiently.
>>>> 
>>>> Maybe I should just try to think it would be implemented as before,
>>>> build an abstraction and just use it for now.
>>>> 
>>>> I will spend my time at PyCon de for sprinting on "yield from".
>>> 
>>> The current implementation may not be much slower than Greg's original
>>> version. One of the main costs of making a call is the creation of a new
>>> frame. But calling into a generator does not need a new frame, so the cost
>>> will be reduced.
>>> Unless anyone has evidence to the contrary :)
>>> 
>>> Rather than increasing the performance of this special case, I would suggest
>>> that improving the performance of calls & returns in general would be a more
>>> worthwhile goal.
>>> Calls and returns ought to be cheap.
>> 
>> I did a basic timing test using a simple recursive function and a
>> recursive PEP-380 coroutine computing the same value (see attachment).
>> The coroutine version is a little over twice as slow as the function
>> version. I find that acceptable. This went 20 deep, making 2 recursive
>> calls at each level (except at the deepest level).
>> 
>> Output on my MacBook Pro:
>> 
>> plain 2097151 0.5880069732666016
>> coro. 2097151 1.2958409786224365
>> 
>> This was a Python 3.3 built a few days ago from the 3.3 branch.
>> 
>> -- 
>> --Guido van Rossum (python.org/~guido)
>> <p3time.py>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> http://mail.python.org/mailman/listinfo/python-ideas
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas