Asynchronous processing is more efficient -- surely not?
rosuav at gmail.com
Wed Apr 4 06:45:12 EDT 2018
On Wed, Apr 4, 2018 at 5:27 PM, Steven D'Aprano
<steve+comp.lang.python at pearwood.info> wrote:
> So, I'm, still trying to wrap my brain around async processing, and I
> started reading this tutorial:
> and the very first paragraph broke my brain.
> "Asynchronous programming has been gaining a lot of traction in the past
> few years, and for good reason. Although it can be more difficult than
> the traditional linear style, it is also much more efficient."
> I can agree with the first part of the first sentence (gaining a lot of
> traction), and the first part of the second sentence (more difficult than
> the traditional style), but the second part? Asynchronous processing is
> *more efficient*?
The only way it could be "more efficient" is if you assume that
"sequential processing" == "well, let's parallelize it by going for
multiple processes, each independently started". This is an
assumption, but it's not wholly unreasonable - if you write your code
in a purely sequential way (say, youtube-dl) and want to do more than
one of these at a time, the easiest way is to pop open another
terminal and fire up another copy of the program, with an entire copy
of everything. But that isn't inherent to the design of sequential
processing. It's a cost of a naive form of parallelism, one that could
be alleviated by threading too.
IMO asyncio should be compared to threads, not to sequential
processing. *First* imagine reworking your code to use threads; then
consider cutting out the actual OS-managed threads and using a
cooperative single-threaded system instead. You depend on
application-level scheduling, which means that gethostbyname() can
utterly break you if you aren't careful, and you lose about thirty or
forty years of collective experience in developing operating systems
with intelligent schedulers, but in return, you gain the ability to
manage a hundred thousand concurrent tasks in a single process. You
also lose the ability to context-swap in the middle of a logical
operation, but in return, you lose the ability to context-swap in the
middle of a logical operation. For people who've grown up with actual
real threads, this latter is not an advantage; for people who've grown
up casually mutating global state, that's huge.
IN THEORY, it would be possible to create a lock-free implementation
of Python that is incapable of using multiple threads, but still able
to use asyncio. In practice, since CPython has a highly efficient
locking system (the GIL), the advantage is unlikely to be compelling.
So, no, generally async processing will NOT be more efficient than
sequential processing. It has other advantages, but not that.
More information about the Python-list