[Python-ideas] Making concurrent.futures.Futures awaitable

[Alex Grönholm]

09.08.2015, 03:22, Nick Coghlan kirjoitti:
>
>
> On 8 Aug 2015 22:48, "Alex Grönholm" <alex.gronholm at nextday.fi 
> <mailto:alex.gronholm at nextday.fi>> wrote:
> >
> > That name would and argument placement would be better, but are you 
> suggesting that the ability to pass along extra arguments should be 
> removed? The original method was bad enough in that it only supported 
> positional and not keyword arguments, forcing users to pass partial() 
> objects as callables.
>
> That's a deliberate design decision in many of asyncio's APIs to 
> improve the introspection capabilities and to clearly separate 
> concerns between "interacting with the event loop" and "the operation 
> being dispatched for execution".
>
While I won't pretend to understand what this means, I recognize that 
you've given it considerably more thought than I have.
>
> >> With the suggested change to the method name and signature, the same
> >> example would instead look like:
> >>
> >>      async def handler(self):
> >>          loop = asyncio.get_event_loop()
> >>          result = await
> >> loop.call_in_background(some_blocking_api.some_blocking_call)
> >>          await self.write(result)
> >
> > Am I the only one who's bothered by the fact that you have to get a 
> reference to the event loop first?
> > Wouldn't this be better:
> >
> > async def handler(self):
> >
> >     result = await 
> asyncio.call_in_background(some_blocking_api.some_blocking_call)
> >
> >     await self.write(result)
>
> That was my original suggestion a few weeks ago, but after playing 
> with it for a while, I came to agree with Guido that hiding the event 
> loop in this case likely wasn't helpful to the conceptual learning 
> process. Outside higher level frameworks that place more constraints 
> on your code, you really can't get very far with asyncio without 
> becoming comfortable with interacting with the event loop directly.
>
As long as I can still write a high level framework where boilerplate is 
minimized in user code, I can "yield" on this issue.
>
> I gave a demo using the current spelling as a lightning talk at PyCon 
> Australia last weekend:
> https://www.youtube.com/watch?v=_pfJZfdwkgI
>
> The only part of that demo I really wasn't happy with was the 
> "run_in_executor" call - the rest all felt good for the level asyncio 
> operates at, while still allowing higher level third party APIs that 
> hide more of the underlying machinery (like the event loop itself, as 
> well as the use of partial function application).
> >
> >
> > The call_in_background() function would return an awaitable object 
> that is recognized by the asyncio Task class, which would then submit 
> the function to the default executor of the event loop.
> >
> >> That should make sense to anyone reading the handler, even if they
> >> know nothing about concurrent.futures - the precise mechanics of how
> >> the event loop goes about handing off the call to a background thread
> >> or process is something they can explore later, they don't need to
> >> know about it in order to locally reason about this specific handler.
> >>
> >> It also means that event loops would be free to implement their
> >> *default* background call functionality using something other than
> >> concurrent.futures, and only switch to the latter if an executor was
> >> specified explicitly.
> >
> > Do you mean background calls that don't return objects compatible 
> with concurrent.futures.Futures?
>
> A background call already returns an asyncio awaitable, not a 
> concurrent.futures.Future object.
>
> > Can you think of a use case for this?
>
> Yes, third party event loops like Twisted may have their own 
> background call mechanism that they'd prefer to use by default, rather 
> than the concurrent.futures model.
>
What I don't get is why you say that this name and signature change 
would somehow enable event loops to implement an alternative mechanism 
for background calls. By event loops do you mean something like 
Twisted's reactors or just customized versions of asyncio event loops? 
To me, the former makes no sense at all and with the latter, I don't see 
how this name and signature change changes anything. Could they not 
already use whatever mechanism they please as long as it returns an 
awaitable (or iterable in the case of 3.4 or earlier) object, by having 
their custom implementation of run_in_executor()?
>
> >> There are still some open questions about whether it makes sense to
> >> allow callables to indicate whether or not they expect to be IO bound
> >> or CPU bound,
> >
> > What do you mean by this?
>
> There was a thread on the idea recently, but I don't have a link 
> handy. Indicating CPU vs IO bound directly wouldn't work (that's 
> context dependent), but allowing callables to explicitly indicate 
> "recommended", "supported", "incompatible" for process pools could be 
> interesting.
>
Yeah -- it'll be interesting to see where that goes.
>
> >>   and hence allow event loop implementations to opt to
> >> dispatch the latter to a process pool by default
> >
> > Bad idea! The semantics are too different and process pools have too 
> many limitations.
>
> Yes, that's why I find it an intriguing notion to allow callables to 
> explicitly indicate whether or not they're compatible with them.
>
> Cheers,
> Nick
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150812/f9fa5b54/attachment.html>