On Sat, Oct 13, 2012 at 3:27 PM, Daniel McDougall
(This is a response to GVR's Google+ post asking for ideas; I apologize in advance if I come off as an ignorant programming newbie)
I am the author of Gate One (https://github.com/liftoff/GateOne/) which makes extensive use of Tornado's asynchronous capabilities. It also uses multiprocessing and threading to a lesser extent. The biggest issue I've had trying to write asynchronous code for Gate One is complexity. Complexity creates problems with expressiveness which results in code that, to me, feels un-Pythonic. For evidence of this I present the following example: The retrieve_log_playback() function: http://bit.ly/W532m6 (link goes to Github)
All the function does is generate and return (to the client browser) an HTML playback of their terminal session recording. To do it efficiently without blocking the event loop or slowing down all other connected clients required loads of complexity (or maybe I'm just ignorant of "a better way"--feel free to enlighten me). In an ideal world I could have just done something like this:
import async # The API of the future ;) async.async_call(retrieve_log_playback, settings, tws, mechanism=multiprocessing) # tws == instance of tornado.web.WebSocketHandler that holds the open connection
What you've described is very similar the the concurrent.futures.Executor.submit() method. ProcessPoolExecutor still has multiprocessing's pickle-related limitations, but other than that you're free to create ProcessPoolExecutors and/or ThreadPoolExecutors and submit work to them. Your retrieve_log_playback function could become: # create a global/singleton ProcessPoolExecutor executor = concurrent.futures.ProcessPoolExecutor() def retrieve_log_playback(settings, tws=None): # set up settings dict just like the original io_loop = tornado.ioloop.IOLoop.instance() future = executor.submit(_retrieve_log_playback, settings) def send_message(future): tws.write_message(future.result()) future.add_done_callback(lambda future: io_loop.add_callback(send_message) In Tornado 3.0 there will be some native support for Futures - the last line will probably become "io_loop.add_future(future, send_message)". In _retrieve_log_playback you no longer have a queue argument, and instead just return the result normally. It's also possible to do this just using multiprocessing instead of concurrent.futures - see multiprocessing.Pool.apply_async. -Ben