[Python-ideas] The async API of the future: Some thoughts from an ignorant Tornado user

[Laurens Van Houtven]

On Sun, Oct 14, 2012 at 12:27 AM, Daniel McDougall <
daniel.mcdougall at liftoffsoftware.com> wrote:

> (This is a response to GVR's Google+ post asking for ideas; I
> apologize in advance if I come off as an ignorant programming newbie)
>

-- snip snip snip --

import async # The API of the future ;)
> async.async_call(retrieve_log_playback, settings, tws,
> mechanism=multiprocessing)
> # tws == instance of tornado.web.WebSocketHandler that holds the open
> connection
>

Is this a CPU-bound problem?

My opinion is that the goal of any async module that winds up in
> Python's standard library should be simplicity and portability.  In
> terms of features, here's my 'async wishlist':
>
> * I should not have to worry about what is and isn't pickleable when I
> decide that a task should be performed asynchronously.
>

Certainly. My above question is important, because this should only matter
for IPC.


> * I should be able to choose the type of event loop/async mechanism
> that is appropriate for the task:  For CPU-bound tasks I'll probably
> want to use multiprocessing.  For IO-bound tasks I might want to use
> threading.  For a multitude of tasks that "just need to be async" (by
> nature) I'll want to use an event loop.
>

Ehhh, maybe. This sounds like it confounds the tools for different use
cases. You can quite easily have threads and processes on top of an event
loop; that works out particularly nicely for processes because you still
have to talk to your processes.

Examples:

twisted.internet.reactor.spawnProcess (local processes)
twisted.internet.threads.deferToThread (local threads)
ampoule (remote processes)

It's quite easy to do blocking IO in a thread with deferToThread; in fact,
that's how twisted's adbapi, an async wrapper to dbapi, works.

* Any async module should support 'basics' like calling functions at
> an interval and calling functions after a timeout occurs (with the
> ability to cancel).
> * Asynchronous tasks should be able to access the same namespace as
> everything else.  Maybe wishful thinking.
>

With twisted, this is already the case; general caveats for shared mutable
state across threads of course still apply. Fortunately in most Twisted
apps, that's a tiny fraction of the total code, and they tend to be
fractions that are well-isolated or at least easily isolatable.


> * It should support publish/subscribe-style events (i.e. an event
> dispatcher).  For example, the ability to watch a file descriptor or
> socket for changes in state and call a function when that happens.
> Preferably with the flexibility to define custom events (i.e don't
> have it tied to kqueue/epoll-specific events).
>

Like connectionMade, connectionLost, dataReceived etc?


>
> Thanks for your consideration; and thanks for the awesome language.
>
> --
> Dan McDougall - Chief Executive Officer and Developer
> Liftoff Software ✈ Your flight to the cloud is now boarding.
> 904-446-8323
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>



-- 
cheers
lvh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20121014/a2caaecd/attachment.html>