[Python-ideas] asyncore: included batteries don't fit

[Guido van Rossum]

On Sat, Oct 6, 2012 at 3:24 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Sat, 6 Oct 2012 15:00:54 -0700
> Guido van Rossum <guido at python.org> wrote:
>>
>> (2) We're at a fork in the road here. On the one hand, we could choose
>> to deeply integrate greenlets/gevents into the standard library. (It's
>> not monkey-patching if it's integrated, after all. :-) I'm not sure
>> how this would work for other implementations than CPython, or even
>> how to address CPython on non-x86 architectures. But users seem to
>> like the programming model: write synchronous code, get async
>> operation for free. It's easy to write protocol parsers that way. On
>> the other hand, we could reject this approach: the integration would
>> never be completely smooth, there's the issue of other implementations
>> and architectures, it probably would never work smoothly even for
>> CPython/x86 when 3rd party extension modules are involved.
>> Callback-based APIs don't have these downsides, but they are harder to
>> program; however we can make programming them easier by using
>> yield-based coroutines. Even Twisted offers those (inline callbacks).
>
> greenlets/gevents only get you half the advantages of single-threaded
> "async" programming: they get you scalability in the face of a high
> number of concurrent connections, but they don't get you the robustness
> of cooperative multithreading (because it's not obvious when reading
> the code where the possible thread-switching points are).

I used to think that too, long ago, until I discovered that as you add
abstraction layers, cooperative multithreading is untenable -- sooner
or later you will lose track of where the threads are switched.

> (I don't actually understand the attraction of gevent, except for
> extreme situations; threads should be cheap on a decent OS)

I think it's the observation that the number of sockets you can
realistically have open in a single process or machine is always 1-2
orders of maginuted larger than the number of threads you can have --
and this makes sense since the total amount of memory (kernel and
user) to represent a socket is just much smaller than needed for a
thread. Just check the configuration limits of your typical Linux
kernel if you don't believe me. :-)

-- 
--Guido van Rossum (python.org/~guido)