[Async-sig] async/sync library reusage

Mon Jun 12 15:09:27 EDT 2017

I think we're getting way beyond the rationale Pau Freixes requested...

On Mon, Jun 12, 2017 at 12:05 PM, Andrew Svetlov <andrew.svetlov at gmail.com>
wrote:

> Agree in general but current asyncio still may shoot your leg.
> The solution (at least for my unittest example) might be in adding top
> level functions for running asyncio code (asyncio.run() and
> asyncio.run_forever() as Yury Selivanov proposed in
> https://github.com/python/asyncio/pull/465)
> After this we could raise a warning in `asyncio.get_event_loop()` if the
> loop was not set explicitly by `asyncio.set_event_loop()`.
>
> On Mon, Jun 12, 2017 at 9:50 PM Guido van Rossum <guido at python.org> wrote:
>
>> Honestly I think we're in agreement. There's never a use for one loop
>> running while another is the default. There are some rare use cases for
>> multiple loops running but before the mentioned commit it was up to the app
>> to ensure to switch the default loop when running a loop. The commit took
>> the ability to screw up there out of the user's hand.
>>
>> On Mon, Jun 12, 2017 at 9:57 AM, Andrew Svetlov <andrew.svetlov at gmail.com
>> > wrote:
>>
>>> Yes, but with one exception: default event loop created on module import
>>> stage might co-exist with a loop created for test.
>>> It leads to mystic hangs, you know.
>>> Please recall code like:
>>>     class A:
>>>          mongodb = motor.motor_asyncio.AsyncIOMotorClient()
>>>
>>> On Mon, Jun 12, 2017 at 7:37 PM Guido van Rossum <guido at python.org>
>>> wrote:
>>>
>>>> Yes, but not co-existing, I hope!
>>>>
>>>> On Mon, Jun 12, 2017 at 9:25 AM, Andrew Svetlov <
>>>> andrew.svetlov at gmail.com> wrote:
>>>>
>>>>> Unit tests at least. Running every test in own loop is crucial fro
>>>>> tests isolation.
>>>>>
>>>>> On Mon, Jun 12, 2017 at 7:04 PM Guido van Rossum <guido at python.org>
>>>>> wrote:
>>>>>
>>>>>> Multiple loops in the same thread is purely theoretical -- the API
>>>>>> allows it but there's no use case. It might be necessary if a platform has
>>>>>> a UI-only event loop that cannot be extended to do I/O -- the only solution
>>>>>> to do background I/O might be to alternate between two loops. (Though in
>>>>>> that case I would still prefer a thread for the background I/O.)
>>>>>>
>>>>>> On Mon, Jun 12, 2017 at 8:49 AM, Pau Freixes <pfreixes at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> And what about the rationale of having multiple loop instances in
>>>>>>> the same thread switching btw them. Im still trying to find out what
>>>>>>> patterns need this... Do you have an example?
>>>>>>>
>>>>>>> Btw thanks for the first explanation
>>>>>>>
>>>>>>> El 12/06/2017 17:36, "Guido van Rossum" <guido at python.org> escribió:
>>>>>>>
>>>>>>>> In theory it's possible to create two event loops (using
>>>>>>>> new_event_loop()), then set one as the default event loop (using
>>>>>>>> set_event_loop()), then run the other one (using run_forever() or
>>>>>>>> run_until_complete()). To tasks running in the latter event loop,
>>>>>>>> get_event_loop() would nevertheless return the former.
>>>>>>>>
>>>>>>>> On Mon, Jun 12, 2017 at 4:39 AM, Pau Freixes <pfreixes at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Sorry a bit of topic, but I would like to figure out why older
>>>>>>>>> python
>>>>>>>>> versions, prior this commit [1], the get_event_loop is not
>>>>>>>>> considered
>>>>>>>>> deterministic
>>>>>>>>>
>>>>>>>>> does anybody know the reason behind this change?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [1] https://github.com/python/cpython/commit/
>>>>>>>>> 600a349781bfa0a8239e1cb95fac29c7c4a3302e
>>>>>>>>>
>>>>>>>>> On Fri, Jun 9, 2017 at 6:07 PM, Ben Darnell <ben at bendarnell.com>
>>>>>>>>> wrote:
>>>>>>>>> > On Fri, Jun 9, 2017 at 11:51 AM Cory Benfield <cory at lukasa.co.uk>
>>>>>>>>> wrote:
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >> My concern with multiple loops boils down to the fact that
>>>>>>>>> urllib3
>>>>>>>>> >> supports being used in a multithreaded context where each
>>>>>>>>> thread can
>>>>>>>>> >> independently make forward progress on one request. To
>>>>>>>>> establish that with a
>>>>>>>>> >> synchronous codebase you either need one event loop per thread
>>>>>>>>> or you need
>>>>>>>>> >> to spawn a background thread on startup that owns the only
>>>>>>>>> event loop in the
>>>>>>>>> >> process.
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > Yeah, one event loop per thread is probably the way to go for
>>>>>>>>> integration
>>>>>>>>> > with synchronous codebases. A dedicated event loop thread may
>>>>>>>>> perform better
>>>>>>>>> > but libraries that spawn threads are problematic.
>>>>>>>>> >
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >> Generally speaking I’ve not had positive results with libraries
>>>>>>>>> spawning
>>>>>>>>> >> their own threads in Python. In my experience this has tended
>>>>>>>>> to lead to
>>>>>>>>> >> programs that deadlock mysteriously or that fail to terminate
>>>>>>>>> in the face of
>>>>>>>>> >> a Ctrl+C. So I tend to prefer to have users spawn their own
>>>>>>>>> threads, which
>>>>>>>>> >> would make me want a “one-event-loop-per-thread” model: hence,
>>>>>>>>> needing a
>>>>>>>>> >> loop parameter to pass around prior to 3.6.
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > You can avoid the loop parameter on older versions of asyncio
>>>>>>>>> (at least as
>>>>>>>>> > long as the default event loop policy is used) by manually
>>>>>>>>> setting your
>>>>>>>>> > event loop as current before calling run_until_complete (and
>>>>>>>>> resetting it
>>>>>>>>> > afterwards).
>>>>>>>>> >
>>>>>>>>> > Tornado's run_sync() method is equivalent to asyncio's
>>>>>>>>> run_until_complete(),
>>>>>>>>> > and Tornado supports multiple IOLoops in this way. We use this
>>>>>>>>> to expose a
>>>>>>>>> > synchronous version of our AsyncHTTPClient:
>>>>>>>>> > https://github.com/tornadoweb/tornado/blob/
>>>>>>>>> 62e47215ce12aee83f951758c96775a43e80475b/tornado/httpclient.py#L54
>>>>>>>>> >
>>>>>>>>> > -Ben
>>>>>>>>> >
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >> I admit that my concerns here regarding libraries spawning
>>>>>>>>> their own
>>>>>>>>> >> threads may be overblown: after my series of negative
>>>>>>>>> experiences I
>>>>>>>>> >> basically never went back to that model, and it may be that the
>>>>>>>>> problems
>>>>>>>>> >> were more user-error than anything else. However, I feel
>>>>>>>>> comfortable saying
>>>>>>>>> >> that libraries spawning their own Python threads is definitely
>>>>>>>>> subtle and
>>>>>>>>> >> hard to get right, at the very least.
>>>>>>>>> >>
>>>>>>>>> >> Cory
>>>>>>>>> >> _______________________________________________
>>>>>>>>> >> Async-sig mailing list
>>>>>>>>> >> Async-sig at python.org
>>>>>>>>> >> https://mail.python.org/mailman/listinfo/async-sig
>>>>>>>>> >> Code of Conduct: https://www.python.org/psf/codeofconduct/
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > _______________________________________________
>>>>>>>>> > Async-sig mailing list
>>>>>>>>> > Async-sig at python.org
>>>>>>>>> > https://mail.python.org/mailman/listinfo/async-sig
>>>>>>>>> > Code of Conduct: https://www.python.org/psf/codeofconduct/
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> --pau
>>>>>>>>> _______________________________________________
>>>>>>>>> Async-sig mailing list
>>>>>>>>> Async-sig at python.org
>>>>>>>>> https://mail.python.org/mailman/listinfo/async-sig
>>>>>>>>> Code of Conduct: https://www.python.org/psf/codeofconduct/
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> --Guido van Rossum (python.org/~guido)
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> --Guido van Rossum (python.org/~guido)
>>>>>> _______________________________________________
>>>>>> Async-sig mailing list
>>>>>> Async-sig at python.org
>>>>>> https://mail.python.org/mailman/listinfo/async-sig
>>>>>> Code of Conduct: https://www.python.org/psf/codeofconduct/
>>>>>>
>>>>> --
>>>>> Thanks,
>>>>> Andrew Svetlov
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> --Guido van Rossum (python.org/~guido)
>>>>
>>> --
>>> Thanks,
>>> Andrew Svetlov
>>>
>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>>
> --
> Thanks,
> Andrew Svetlov
>

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20170612/eb9df428/attachment-0001.html>