On Sun, Oct 14, 2012 at 5:32 AM, Laurens Van Houtven firstname.lastname@example.org wrote:
import async # The API of the future ;) async.async_call(retrieve_log_playback, settings, tws, mechanism=multiprocessing) # tws == instance of tornado.web.WebSocketHandler that holds the open connection
Is this a CPU-bound problem?
It depends on the host. On embedded platforms (e.g. the BeagleBone) it is more IO-bound than CPU bound (fast CPU but slow disk and slow memory). On regular x86 systems it is mostly CPU-bound.
- I should be able to choose the type of event loop/async mechanism
that is appropriate for the task: For CPU-bound tasks I'll probably want to use multiprocessing. For IO-bound tasks I might want to use threading. For a multitude of tasks that "just need to be async" (by nature) I'll want to use an event loop.
Ehhh, maybe. This sounds like it confounds the tools for different use cases. You can quite easily have threads and processes on top of an event loop; that works out particularly nicely for processes because you still have to talk to your processes.
twisted.internet.reactor.spawnProcess (local processes) twisted.internet.threads.deferToThread (local threads) ampoule (remote processes)
It's quite easy to do blocking IO in a thread with deferToThread; in fact, that's how twisted's adbapi, an async wrapper to dbapi, works.
As I understand it, twisted.internet.reactor.spawnProcess is all about spawning subprocesses akin to subprocess.Popen(). Also, it requires writing a sophisticated ProcessProtocol. It seems to be completely unrelated and wickedly complicated. The complete opposite of what I would consider ideal for an asynchronous library since it is anything but simple.
I mean, I could write a separate program to generate HTML playback files from logs, spawn a subprocess in an asynchronous fashion, then watch it for completion but I could do that with termio.Multiplex (see: https://github.com/liftoff/GateOne/blob/master/gateone/termio.py) .
deferToThread() does what one would expect but in many situations I'd prefer something like deferToMultiprocessing().
- It should support publish/subscribe-style events (i.e. an event
dispatcher). For example, the ability to watch a file descriptor or socket for changes in state and call a function when that happens. Preferably with the flexibility to define custom events (i.e don't have it tied to kqueue/epoll-specific events).
Like connectionMade, connectionLost, dataReceived etc?
Oh there's a hundred different ways to fire and catch events. I'll let the low-level async experts decide which is best. Having said that, it would be nice if the interface didn't use such network-specific naming conventions. I would prefer something more generic. It is fine if it uses sockets and whatnot in the background.