[Python-ideas] [Python-Dev] Python needs a standard asynchronous return object

[Guido van Rossum]

(Summary: I want to make an apology, and reopen the debate. Possibly
relevant: PEP 342, PEP 380, PEP 3148, PEP 3152.)

On Sat, Sep 11, 2010 at 9:03 AM, Jesse Noller <jnoller at gmail.com> wrote:
> On Fri, Sep 10, 2010 at 5:07 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> On Sat, Sep 11, 2010 at 9:25 AM, Guido van Rossum <guido at python.org> wrote:
>>> Moving to python-ideas.
>>>
>>> Have you seen http://www.python.org/dev/peps/pep-3148/ ? That seems
>>> exactly what you want.
>>
>> James did mention that in the post,

Whoops. I was a bit quick at the trigger there.

>> although he didn't say what
>> deferreds really added beyond what futures provide, and why the
>> "add_done_callback" method isn't adequate to provide interoperability
>> between futures and deferreds (which would be odd, since Brian made
>> changes to that part of PEP 3148 to help with that interoperability
>> after discussions with Glyph).
>>
>> Between PEP 380 and PEP 3148 I'm not really seeing a lot more scope
>> for standardisation in this space though.
>>
>> Cheers,
>> Nick.
>
> That was my initial reaction as well, but I'm more than open to
> hearing from Jean Paul/Glyph and the other twisted folks on this.

Re-reading the OP's post[0] and the blog[1] he references, I notice
that he did not mention PEP 380 (which for the blog's example doesn't
actually add much except adding a nicer way to return a value from a
generator) but he did mention the awesomeness of not needing threads
when using deferreds. He sounds as if the python-dev community had
never heard of that style of handling concurrency, which seems
backwards: the generator-based style of doing it was introduced in PEP
342 which enabled Twisted's inline callbacks. (Though he does mention
Python Enhanced Generators which could be an implicit reference to PEP
342 -- "Coroutines via Enhanced Generators".)

But thinking about this more I don't know that it will be easy to mix
PEP 3148, which is solidly thread-based, with a PEP 342 style
scheduler (whether or not the PEP 380 enhancements are applied, or
even PEP 3152). And if we take the OP's message at face value, his
point isn't so much that Twisted is great, but that in order to
benefit maximally from PEP 342 there needs to be a standard way of
using callbacks. I think that's probably true. And comparing the
blog's examples to PEP 3148, I find Twisted's terminology rather
confusing compared to the PEP's clean Futures API (where IMO you can
ignore almost everything except result()).

Maybe it's possible to write a little framework that lets you create
Futures using either threads, processes (both supported by PEP 3148)
or generators. But I haven't tried it. And maybe the need to use
'yield' for everything that may block when using generators, but not
when using threads or processes, will make this awkward. So maybe
we'll be stuck with at least two Future-like APIs: PEP 3148 and
something else, generator-based. Or maybe PEP 3152.

So, yes, there may be something here, and let's reopen the discussion.
And I apologize for shooting first and asking questions second.

[0] http://mail.python.org/pipermail/python-dev/2010-September/103576.html
[1] http://blog.mekk.waw.pl/archives/14-Twisted-inlineCallbacks-and-deferredGenerator.html
-- 
--Guido van Rossum (python.org/~guido)