[Python-ideas] The async API of the future: Some thoughts from an ignorant Tornado user

[Daniel McDougall]

On Sun, Oct 14, 2012 at 5:32 AM, Laurens Van Houtven <_ at lvh.cc> wrote:
>> import async # The API of the future ;)
>> async.async_call(retrieve_log_playback, settings, tws,
>> mechanism=multiprocessing)
>> # tws == instance of tornado.web.WebSocketHandler that holds the open
>> connection
>
>
> Is this a CPU-bound problem?

It depends on the host.  On embedded platforms (e.g. the BeagleBone)
it is more IO-bound than CPU bound (fast CPU but slow disk and slow
memory).  On regular x86 systems it is mostly CPU-bound.


>> * I should be able to choose the type of event loop/async mechanism
>> that is appropriate for the task:  For CPU-bound tasks I'll probably
>> want to use multiprocessing.  For IO-bound tasks I might want to use
>> threading.  For a multitude of tasks that "just need to be async" (by
>> nature) I'll want to use an event loop.
>
>
> Ehhh, maybe. This sounds like it confounds the tools for different use
> cases. You can quite easily have threads and processes on top of an event
> loop; that works out particularly nicely for processes because you still
> have to talk to your processes.
>
> Examples:
>
> twisted.internet.reactor.spawnProcess (local processes)
> twisted.internet.threads.deferToThread (local threads)
> ampoule (remote processes)
>
> It's quite easy to do blocking IO in a thread with deferToThread; in fact,
> that's how twisted's adbapi, an async wrapper to dbapi, works.

As I understand it, twisted.internet.reactor.spawnProcess is all about
spawning subprocesses akin to subprocess.Popen().  Also, it requires
writing a sophisticated ProcessProtocol.  It seems to be completely
unrelated and wickedly complicated.  The complete opposite of what I
would consider ideal for an asynchronous library since it is anything
but simple.

I mean, I could write a separate program to generate HTML playback
files from logs, spawn a subprocess in an asynchronous fashion, then
watch it for completion but I could do that with termio.Multiplex
(see: https://github.com/liftoff/GateOne/blob/master/gateone/termio.py)
.

>
>> * Any async module should support 'basics' like calling functions at
>> an interval and calling functions after a timeout occurs (with the
>> ability to cancel).
>> * Asynchronous tasks should be able to access the same namespace as
>> everything else.  Maybe wishful thinking.
>
>
> With twisted, this is already the case; general caveats for shared mutable
> state across threads of course still apply. Fortunately in most Twisted
> apps, that's a tiny fraction of the total code, and they tend to be
> fractions that are well-isolated or at least easily isolatable.
>
>>
>> * It should support publish/subscribe-style events (i.e. an event
>> dispatcher).  For example, the ability to watch a file descriptor or
>> socket for changes in state and call a function when that happens.
>> Preferably with the flexibility to define custom events (i.e don't
>> have it tied to kqueue/epoll-specific events).
>
>
> Like connectionMade, connectionLost, dataReceived etc?
>
>>
>>
>> Thanks for your consideration; and thanks for the awesome language.
>>
>> --
>> Dan McDougall - Chief Executive Officer and Developer
>> Liftoff Software ✈ Your flight to the cloud is now boarding.
>> 904-446-8323
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> http://mail.python.org/mailman/listinfo/python-ideas
>
>
>
>
> --
> cheers
> lvh
>



-- 
Dan McDougall - Chief Executive Officer and Developer
Liftoff Software ✈ Your flight to the cloud is now boarding.
904-446-8323