[Python-ideas] Concurrency Modules
Sven R. Kunze
srkunze at mail.de
Sun Jul 26 12:07:05 CEST 2015
Thanks, Nikolaus. Mostly I refer to things Steve brought up in his
analogies (two recent posts). So, I might interpreted them the wrong way.
On 26.07.2015 02:58, Nikolaus Rath wrote:
> On Jul 25 2015, "Sven R. Kunze" <srkunze-7y4VAllY4QU at public.gmane.org> wrote:
>> startup impact | biggest | medium | smallest
>> cpu impact | biggest | medium | smallest
>> memory impact | biggest | medium | smallest
>> purpose | cpu-bound tasks | i/o-bound tasks | ???
> I don't think any of these is correct. Unfortunately, I also don't think
> there even is a correct version, the differences are simply not so
I think that has already been discussed. We just try to boil it down to
assist people making the decision of which module might be the best for
> On Unix, Process startup-cost can be high if you do fork() + exec(), but
> if you just fork, it's as cheap as a thread.
Didn't know that. Thanks for clarifying. How do multiprocessing.Pool and
multiprocessing.Process work in this regard?
> With asyncio, it's not
> clear to me what exactly you'd define as the "startup impact" (the
> creation of a future maybe? Or setting up the event loop?).
The purpose of survey is to give developers an easy way to decide which
approach might be suitable for them.
So, the definition of 'startup time' should be roughly equivalent across
the approaches. >> What's necessary to get a process up and running a
piece of code compared to what's necessary to get asyncio up and running
the same piece of code.
Steve: "Bakers aren't free, you have to pay for each one (memory, stack
space), it will take time for each one to learn how your bakery works
> "CPU impact" as a category doesn't make any sense to me. If you execute
> the same code it's going to take the same amount of (cumulative) CPU
> time, no matter if this code runs in a separate thread, separate
> process, or asynchronously.
From what I understand, switching contexts impacts cpu whereas the
event loop does not so much.
> "memory impact" is probably highest for separate processes, but I don't
> see an obvious difference when using threads vs asyncio. Where did you
> get this from?
I can imagine that when the os needs to manage threads it creates more
overhead for each thread than what it takes for the Python interpreter
when suspending coroutines. That could be wrong? Do you have any
material on this?
> As far as purpose is concerned, pretty much the only limitation is that
> asyncio is not suitable for cpu-bound tasks. Any other combination is
> possible and also most appropriate in specific circumstances.
What exactly do you mean by any other combination?
I take from this that asyncio is suitable for heavy i/o-bound, threads
are for cpu/io-bound and processes for mainly cpu-bound.
More information about the Python-ideas