[Async-sig] New blog post: Notes on structured concurrency, or: Go statement considered harmful

Nathaniel Smith njs at pobox.com
Fri Apr 27 00:44:05 EDT 2018

On Thu, Apr 26, 2018 at 7:55 PM, Dima Tisnek <dimaqq at gmail.com> wrote:
> My 2c after careful reading:
> restarting tasks automatically (custom nursery example) is quite questionable:
> * it's unexpected
> * it's not generally safe (argument reuse, side effects)
> * user's coroutine can be decorated to achieve same effect

It's an example of something that a user could implement. I guess if
you go to the trouble of implementing this behavior, then it is no
longer unexpected and you can also cope with handling the edge cases
:-).There may be some reason why it turns out to be a bad idea
specifically in the context of Python, but it's one of the features
that's famously helpful for making Erlang work so well, so it seemed
worth mentioning.

> It's very nice to have the escape hatch of posting tasks to "someone
> else's" nursery.
> I feel there are more caveats to posting a task to parent's or global
> nursery though.
> Consider that local tasks typically await on other local tasks.
> What happens when N1-task1 waits on N2-task2 and N2-task9 encounters an error?
> My guess is N2-task2 is cancelled, which by default cancels N1-task1 too, right?
> That kinda break the abstraction, doesn't it?

"Await on a task" is not a verb that Trio has. (We don't even have
task objects, except in some low-level plumbing/introspection APIs.)
You can do 'await queue.get()' to wait for another task to send you
something, but if the other task gets cancelled then the data will
just... never arrive.

There is some discussion here of moving from a queue.Queue-like model
to a model with separate send- and receive-channels:


If we do this (which I suspect we will), then probably the task that
gets cancelled was holding the only reference to the send-channel (or
even better, did 'with send_channel: ...'), so the channel will get
closed, and then the call to get() will raise an error which it can
handle or not...

But yes, you do need to spend some time thinking about what kind of
task tree topology makes sense for your problem. Trio can give you
tools but it's not a replacement for thoughtful design :-).

> If the escape hatch is available, how about allowing tasks to be moved
> between nurseries?

That would be possible (and in fact there's one special case
internally where we do it!), but I haven't seen a good reason yet to
implement it as a standard feature. If someone shows up with use cases
then we could talk about it :-).

> Is dependency inversion allowed?
> (as in given parent N1 and child N1.N2, can N1.N2.t2 await on N1.t1 ?)
> If that's the case, I guess it's not a "tree of tasks", as in the
> graph is arbitrary, not DAG.

See above re: not having "wait on a task" as a verb.

> I've seen [proprietary] strict DAG task frameworks.
> while they are useful to e.g. perform sub-requests in parallel,
> they are not general enough to be useful at large.
> Thus I'm assuming trio does not enforce DAG...

The task tree itself is in fact a tree, not a DAG. But that tree
doesn't control which tasks can talk to each other. It's just used for
exception propagation, and for enforcing that all children have to
finish before the parent can continue. (Just like how in a regular
function call, the caller stops while the callee is running.) Does
that help?

> Finally, slob programmers like me occasionally want fire-and-forget
> tasks, aka daemonic threads.
> Some are long-lived, e.g. "battery status poller", others short-lived,
> e.g. "tail part of low-latency logging".
> Obv., a careful programmer would keep track of those, but we want
> things simple :)
> Perhaps in line with batteries included principle, trio could include
> a standard way to accomplish that?

Well, what semantics do you want? If the battery status poller
crashes, what should happen? If the "tail part of low-latency logging"
command is still running when you go to shut down, do you want to wait
a bit for it to finish, or cancel it, or ...?

You can certainly implement some helper like:

async with open_throwaway_nursery() as throwaway_nursery:
    # If this crashes, we ignore the problem, maybe log it or something
    # When we exit the with block, it gets cancelled

if that's what you want. Before adding anything like this to trio
itself though I'd like to see some evidence of how it's being used in
real-ish projects.

> Thanks again for the great post!
> I think you could publish an article on this, it would be good to have
> wider discussion, academic, ES6, etc.

Thanks for the vote of confidence :-). And, we'll see...


Nathaniel J. Smith -- https://vorpus.org

More information about the Async-sig mailing list