[Python-ideas] Better integration of multiprocessing with asyncio

[Guido van Rossum]

It takes time to get used to it, but eventually you don't have to think
about the event loop. There really is no need for any comment of that kind
at all. The beauty is that *nothing* blocks the event loop...

On Wednesday, October 8, 2014, Dan O'Reilly <oreilldf at gmail.com> wrote:

> Ah, right. I find the terminology confusing myself, and I probably got it
> wrong. I meant "non-blocking" as in - "this won't block the event loop",
> not "this won't block the coroutine". I'll try to clarify that. Thanks!
>
> On Thu, Oct 9, 2014 at 12:03 AM, Guido van Rossum <guido at python.org
> <javascript:_e(%7B%7D,'cvml','guido at python.org');>> wrote:
>
>> This looks neat. I skimmed the README, and I noticed you marked up most
>> "yield from" epressions with "# non blocking". That feels confusing to me,
>> because when I read asyncio code, I think fo "yield from" as blocking (the
>> task, if not the world :-).  What do you think?
>>
>> --Guido
>>
>>
>> On Tuesday, October 7, 2014, Dan O'Reilly <oreilldf at gmail.com
>> <javascript:_e(%7B%7D,'cvml','oreilldf at gmail.com');>> wrote:
>>
>>> For what it's worth, I ended up writing a module that takes the "threads
>>> on top of regular multiprocessing" approach to integrate the two:
>>> https://github.com/dano/aioprocessing.
>>>
>>> Re-implementing multiprocessing to use asyncio internally, while an
>>> interesting exercise, would require a very large amount of effort both to
>>> implement and maintain alongside the current multiprocessing module. I'm
>>> not sure it's really worth it when using threads on top of multiprocessing
>>> gives you the same effect without requiring you to basically re-implement
>>> large parts of the multiprocessing module.
>>>
>>> Anyway, we'll see if it ends up getting much use...
>>>
>>> On Sat, Jul 26, 2014 at 11:48 PM, Ryan Hiebert <ryan at ryanhiebert.com>
>>> wrote:
>>>
>>>>
>>>> On Jul 26, 2014, at 10:43 PM, Guido van Rossum <guido at python.org>
>>>> wrote:
>>>>
>>>> I'm going to go out on a limb here and say that it feels too early to
>>>> me. First someone has to actually solve this problem well as a 3rd party
>>>> package before we can talk about adding it to the asyncio package. It
>>>> doesn't actually sound like Billiards has adapted to asyncio yet (not that
>>>> I have any idea what Billiards is -- it sounds like a fork of
>>>> multiprocessing actually?).
>>>>
>>>>
>>>> Yep, Billiard is a fork of multiprocessing:
>>>> https://pypi.python.org/pypi/billiard
>>>>
>>>
>>>
>>
>> --
>> --Guido van Rossum (on iPad)
>>
>
>

-- 
--Guido van Rossum (on iPad)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141008/f4b4727c/attachment.html>