[Python-ideas] The async API of the future: Some thoughts from an ignorant Tornado user

[Daniel McDougall]

On Sun, Oct 14, 2012 at 5:32 AM, Laurens Van Houtven <_ at lvh.cc> wrote:
>> import async # The API of the future ;)
>> async.async_call(retrieve_log_playback, settings, tws,
>> mechanism=multiprocessing)
>> # tws == instance of tornado.web.WebSocketHandler that holds the open
>> connection
>
>
> Is this a CPU-bound problem?

It depends on the host.  On embedded platforms (e.g. the BeagleBone)
it is more IO-bound than CPU bound (fast CPU but slow disk and slow
memory).  On regular x86 systems it is mostly CPU-bound.


>> * I should be able to choose the type of event loop/async mechanism
>> that is appropriate for the task:  For CPU-bound tasks I'll probably
>> want to use multiprocessing.  For IO-bound tasks I might want to use
>> threading.  For a multitude of tasks that "just need to be async" (by
>> nature) I'll want to use an event loop.
>
>
> Ehhh, maybe. This sounds like it confounds the tools for different use
> cases. You can quite easily have threads and processes on top of an event
> loop; that works out particularly nicely for processes because you still
> have to talk to your processes.
>
> Examples:
>
> twisted.internet.reactor.spawnProcess (local processes)
> twisted.internet.threads.deferToThread (local threads)
> ampoule (remote processes)
>
> It's quite easy to do blocking IO in a thread with deferToThread; in fact,
> that's how twisted's adbapi, an async wrapper to dbapi, works.

As I understand it, twisted.internet.reactor.spawnProcess is all about
spawning subprocesses akin to subprocess.Popen().  Also, it requires
writing a sophisticated ProcessProtocol.  It seems to be completely
unrelated and wickedly complicated.  The complete opposite of what I
would consider ideal for an asynchronous library since it is anything
but simple.

I mean, I could write a separate program to generate HTML playback
files from logs, spawn a subprocess in an asynchronous fashion, then
watch it for completion but I could do that with termio.Multiplex
(see: https://github.com/liftoff/GateOne/blob/master/gateone/termio.py)
.

deferToThread() does what one would expect but in many situations I'd
prefer something like deferToMultiprocessing().


>> * It should support publish/subscribe-style events (i.e. an event
>> dispatcher).  For example, the ability to watch a file descriptor or
>> socket for changes in state and call a function when that happens.
>> Preferably with the flexibility to define custom events (i.e don't
>> have it tied to kqueue/epoll-specific events).
>
>
> Like connectionMade, connectionLost, dataReceived etc?

Oh there's a hundred different ways to fire and catch events.  I'll
let the low-level async experts decide which is best.  Having said
that, it would be nice if the interface didn't use such
network-specific naming conventions.  I would prefer something more
generic.  It is fine if it uses sockets and whatnot in the background.

-- 
Dan McDougall - Chief Executive Officer and Developer
Liftoff Software ✈ Your flight to the cloud is now boarding.