[Python-ideas] async: switch_to_loop and loop_affinity

[David Foster]

There has been much talk of the use of async libraries (such as Tulip) 
within the general programming community to avoid the "christmas tree" 
pattern of code indentation in code that uses a lot of callbacks. For 
example, consider the following callback-based code:

def duplicate_file_async(filepath, done):
     def exists_async(does_exist):
         if not does_exist:
             done(False)
             return
         def entire_file_read(contents):
             if not contents:
                 done(False)
                 return
             def file_written(success):
                 done(success)
             write_entire_file_async(filepath + ' copy', contents, 
file_written)
         read_entire_file_async(filepath, entire_file_read)
     exists_async(filepath)

This code be rewritten in Tulip as:

@async.coroutine
def duplicate_file_async(filepath):
     if not yield from exists_async(filepath):
         return False
     contents = yield from read_entire_file_async(filepath)
     if not contents:
         return False
     return yield from write_entire_file_async(filepath + ' copy', contents)

Much more clear, no?

Tulip, however, does not appear to address a slightly different problem 
I run into where I need different parts of the same conceptual function 
to run on specific *different* threads. Consider the following code 
which needs different parts executed on a UI thread, database thread, 
and background thread:

def tree_node_clicked(tree_node):
     print('Clicked: ' + repr(tree_node))
     def db_task():
         db_node = fetch_node_from_db(tree_node)

         def process_node(db_node):
             tree_node.set_contents(db_node.contents)
             # (done)

         if db_node is not None:
             def bg_task():
                 db_node = fetch_node_from_internet(tree_node.url)
                 def db_task():
                     insert_into_db(db_node)
                     ui_call_soon(process_node, db_node)
                 db_call_soon(db_task)
             bg_call_soon(bg_task)
         else:
             ui_call_soon(process_node, db_node)
     db_call_soon(db_task)

Imagine if you could write this function like the following:

ui_loop = ...
db_loop = ...
bg_loop = ...

@async.coroutine
def tree_node_clicked(tree_node):
     print('Clicked: ' + repr(tree_node))

     yield from switch_to_loop(db_loop)
     db_node = fetch_node_from_db(tree_node)
     if db_node is not None:
         yield from switch_to_loop(bg_loop)
         db_node = fetch_node_from_internet(tree_node.url)

         yield from switch_to_loop(db_loop)
         insert_into_db(db_node)

     yield from switch_to_loop(ui_loop)
     tree_node.set_contents(db_node.contents)

Or even better: If you created a decorators like @loop_affinity(*_loop) 
that automatically called switch_to_loop(...) if the current event loop 
wasn't correct, you could even write the following even-more simplified 
version:

@async.coroutine
def tree_node_clicked(tree_node):
     print('Clicked: ' + repr(tree_node))
     db_node = yield from fetch_node_from_db(tree_node)
     if db_node is not None:
         db_node = yield from fetch_node_from_internet(tree_node.url)
         yield from insert_into_db(db_node)
     yield from tree_node.set_contents(db_node.contents)

@async.loop_affinity(db_loop)
def fetch_node_from_db(...): ...

@async.loop_affinity(bg_loop)
def fetch_node_from_internet(...): ...

@async.loop_affinity(db_loop)
def insert_into_db(...): ...

@async.loop_affinity(ui_loop)
def set_contents(...): ...

Wouldn't that be just grand? I have prototyped switch_to_loop(...) 
successfully in Tulip, albeit with a race condition I was unable to isolate.

How about some equivalent to switch_to_loop(...) and loop_affinity(...) 
in a future version of Tulip?

-- 
David Foster
http://dafoster.net/