[Python-Dev] A more flexible task creation

[Gustavo Carneiro]

On Fri, 15 Jun 2018 at 09:18, Michel Desmoulin <desmoulinmichel at gmail.com>
wrote:

>
> >
> > The strict API compatibility requirements of core Python stdlib, coupled
> > with the very long feature release life-cycles of Python, make me think
> > this sort of thing perhaps is better built in an utility library on top
> > of asyncio, rather than inside asyncio itself?  18 months is a long long
> > time to iterate on these features.  I can't wait for Python 3.8...
> >
>
> A lot of my late requests come from my attempt to group some of that in
> a lib: https://github.com/Tygs/ayo
>

Ah, good idea.


> Most of it works, although I got read of context() recently, but the
> lazy task part really fails.
>
>
> Indeed, the API allows to do:
>
>         async with ayo.scope() as run:
>             task_list = run.all(foo(), foo(), foo())
>             run.asap(bar())
>             await task_list.gather()
>             run.asap(baz())
>
>
>
> scope() return a nursery like object, and this works perfectly, with the
> usual guaranty of Trio's nursery, but working in asyncio right now.
>

To be honest, I see "async with" being abused everywhere in asyncio,
lately.  I like to have objects with start() and stop() methods, but
everywhere I see async context managers.

Fine, add nursery or whatever, but please also have a simple start() /
stop() public API.

"async with" is only good for functional programming.  If you want to go
more of an object-oriented style, you tend to have start() and stop()
methods in your classes, which will call start() & stop() (or close())
methods recursively on nested resources.  So of the libraries (aiopg, I'm
looking at you) don't support start/stop or open/close well.


> However, I tried to add to the mix:
>
>         async with ayo.scope(max_concurrency=2) as run:
>             task_list = run.all(foo(), foo(), foo())
>             run.asap(bar())
>             await task_list.gather()
>             run.asap(baz())
>
> And I can get it to work. task_list will right now contains a list of
> tasks and None, because some tasks are not scheduled immediately. That's
> why I wanted lazytasks. I tried to create my own lazy tasks, but it
> never really worked. I'm going to try to go down the road of wrapping
> the unscheduled coro in a future-like object as suggested by Yuri. But
> having that built-in seems logical, elegant, and just good design in
> general: __init__ should not have side effects.
>

I tend to slightly agree, but OTOH if asyncio had been designed to not
schedule tasks automatically on __init__ I bet there would have been other
users complaining that "why didn't task XX run?", or "why do tasks need a
start() method, that is clunky!".  You can't please everyone...

Also, in
             task_list = run.all(foo(), foo(), foo())

As soon as you call foo(), you are instantiating a coroutine, which
consumes memory, while the task may not even be scheduled for a long time
(if you have 5000 potential tasks but only execute 10 at a time, for
example).

But if you do as Yuri suggested, you'll instead accept a function
reference, foo, which is a singleton, you can have many foo references to
the function, but they will only create coroutine objects when the task is
actually about to be scheduled, so it's more efficient in terms of memory.

-- 
Gustavo J. A. M. Carneiro
Gambit Research
"The universe is always one step beyond logic." -- Frank Herbert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180615/4439b098/attachment.html>