[Python-ideas] yield from multiple iterables (was Re: The async API of the future: yield-from)

[Christian Tismer]

Errhm....,

On 20.10.12 00:31, Christian Tismer wrote:
> Hi Guido, Marc, all,
>
> this is a veery promising result, telling
> me that the big Oh can in fact be
> neglected in real applications. 20 for
> twi is good!
>
> I will of course do an analysis and find
> the parameters of the quadratic, but my
> concern is pretty much tamed.
>
> For me that means there will soon be
> a library that contains real generators
> and more building blocks.
>
> I think using those would simplify the
> design of the async API quite a lot.
>
> I suggest to regard current generator
> constructs as low-level helpers for
> Implementing the real concurrency
> building blocks.
>
> Instead of using the existing re.compile("yield (from)?") pattern, I think we can abstract
> from this now and think in terms of
> higher level constructs.
>
> Let's assume generators and coroutines,
> and model concurrency from that. I
> believe this unwinds the brains and
> clarifies things a lot.
>
> I will provide sone classes for that at
> the pycon.de sprint, unless somebody
> implements it earlier (please don't).
>
> This email was written in non-linear
> order, so please ignore logic inversions.
>
> Cheers - chris
>
> Sent from my Ei4Steve
>
> On Oct 19, 2012, at 23:31, Guido van Rossum <guido at python.org> wrote:
>
>> On Fri, Oct 19, 2012 at 2:15 PM, Mark Shannon <mark at hotpy.org> wrote:
>>> On 19/10/12 13:55, Christian Tismer wrote:
>>>> Hi Nick,
>>>>
>>>> On 16.10.12 03:49, Nick Coghlan wrote:
>>>>> On Tue, Oct 16, 2012 at 10:44 AM, Greg Ewing
>>>>> <greg.ewing at canterbury.ac.nz> wrote:
>>>>>> My original implementation of yield-from actually *did* avoid
>>>>>> this, by keeping a C-level pointer chain of yielding-from frames.
>>>>>> But that part was ripped out at the last minute when someone
>>>>>> discovered that it had a detrimental effect on tracebacks.
>>>>>>
>>>>>> There are probably other ways the traceback problem could be
>>>>>> fixed, so maybe we will get this optimisation back one day.
>>>>> Ah, I thought I remembered something along those lines. IIRC, it was a
>>>>> bug report on one of the alphas that prompted us to change it.
>>>> I was curious and searched quite a lot.
>>>> It was v3.3.0a1 from 2012-03-15 as a reaction to #14230 and #14220
>>>> from Marc Shannon, patched by Benjamin.
>>>>
>>>> Now I found the original implementation. That looks very much
>>>> as I'm thinking it should be.
>>>>
>>>> Quite a dramatic change which works well, but really seems to remove
>>>> what I would call "now I can emulate most of Stackless" efficiently.
>>>>
>>>> Maybe I should just try to think it would be implemented as before,
>>>> build an abstraction and just use it for now.
>>>>
>>>> I will spend my time at PyCon de for sprinting on "yield from".
>>> The current implementation may not be much slower than Greg's original
>>> version. One of the main costs of making a call is the creation of a new
>>> frame. But calling into a generator does not need a new frame, so the cost
>>> will be reduced.
>>> Unless anyone has evidence to the contrary :)
>>>
>>> Rather than increasing the performance of this special case, I would suggest
>>> that improving the performance of calls & returns in general would be a more
>>> worthwhile goal.
>>> Calls and returns ought to be cheap.
>> I did a basic timing test using a simple recursive function and a
>> recursive PEP-380 coroutine computing the same value (see attachment).
>> The coroutine version is a little over twice as slow as the function
>> version. I find that acceptable. This went 20 deep, making 2 recursive
>> calls at each level (except at the deepest level).
>>
>> Output on my MacBook Pro:
>>
>> plain 2097151 0.5880069732666016
>> coro. 2097151 1.2958409786224365
>>
>> This was a Python 3.3 built a few days ago from the 3.3 branch.
>>

What you are comparing seems to have a constant factor of about 2.5.

minimax:py3 tismer$ python3 p3time.py
plain 0 1   0.00000
coro. 0 1   0.00001
relat 0 1   8.50000
plain 1 3   0.00000
coro. 1 3   0.00001
relat 1 3   2.77778
plain 2 7   0.00000
coro. 2 7   0.00001
relat 2 7   3.62500
plain 3 15   0.00000
coro. 3 15   0.00001
relat 3 15   2.87500
plain 4 31   0.00001
coro. 4 31   0.00002
relat 4 31   2.42424
plain 5 63   0.00002
coro. 5 63   0.00004
relat 5 63   2.46032
plain 6 127   0.00003
coro. 6 127   0.00007
relat 6 127   2.52542
plain 7 255   0.00006
coro. 7 255   0.00014
relat 7 255   2.38272
plain 8 511   0.00011
coro. 8 511   0.00028
relat 8 511   2.49356
plain 9 1023   0.00022
coro. 9 1023   0.00055
relat 9 1023   2.50327
plain 10 2047   0.00042
coro. 10 2047   0.00106
relat 10 2047   2.50956
plain 11 4095   0.00083
coro. 11 4095   0.00204
relat 11 4095   2.44699
plain 12 8191   0.00167
coro. 12 8191   0.00441
relat 12 8191   2.64792
plain 13 16383   0.00340
coro. 13 16383   0.00855
relat 13 16383   2.51881
plain 14 32767   0.00876
coro. 14 32767   0.01823
relat 14 32767   2.08106
plain 15 65535   0.01419
coro. 15 65535   0.03507
relat 15 65535   2.47131
plain 16 131071   0.02669
coro. 16 131071   0.06874
relat 16 131071   2.57515
plain 17 262143   0.05448
coro. 17 262143   0.13699
relat 17 262143   2.51467
plain 18 524287   0.10843
coro. 18 524287   0.27395
relat 18 524287   2.52660
plain 19 1048575   0.21310
coro. 19 1048575   0.54573
relat 19 1048575   2.56095
plain 20 2097151   0.42802
coro. 20 2097151   1.06199
relat 20 2097151   2.48114
plain 21 4194303   0.86531
coro. 21 4194303   2.19048
relat 21 4194303   2.53143

ciao - chris

-- 
Christian Tismer             :^)   <mailto:tismer at stackless.com>
Software Consulting          :     Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121     :    *Starship* http://starship.python.net/
14482 Potsdam                :     PGP key -> http://pgp.uni-mainz.de
phone +49 173 24 18 776  fax +49 (30) 700143-0023
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
       whom do you want to sponsor today?   http://www.stackless.com/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: p3time.py
Type: text/x-python
Size: 946 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20121020/46fb7aba/attachment.py>