I'm writing a runtime system for doing secure multiparty computation, and here I have used deferreds with great success.
All players run the same program (but with different input) and each player communicate with other players as needed to share intermediate results. These results need to be associated with the right operation and for this I use a program counter.
This minimal example shows my solution without network traffic:
import sys From twisted.internet.defer import Deferred, DeferredList, gatherResults
pc = (0,)
def inc_pc(pc): pc = pc[:-1] + (pc[-1]+1,) return pc
def sub_pc(pc): pc = pc + (1,) return pc
def finish_mul(arg, pc): print "finish_mul(%s): pc = %s" % (arg, pc) return arg
def mul(a, b): global pc pc = inc_pc(pc)
results = gatherResults([a,b]) results.addCallback(finish_mul, sub_pc(pc)) return results
x = Deferred() # Declare variables y = Deferred() z = Deferred() w = Deferred()
mul(x,y) # Schedule multiplications mul(z,w)
x.callback(2) # The value for x arrives over the network z.callback(4)
if len(sys.argv) > 1: y.callback(3) # y gets a value, enabling mul(x,y) to proceed. w.callback(5) else: w.callback(5) # z and w are ready and mul(z,w) can proceed. y.callback(3)
Here mul() schedules calls to finish_mul(), and it is this call which I would like to have a deterministic labelling of. I have solved it by having a "two-dimensional" program counter: the calls to mul() get labels (1,) and (2,). Their calls to the sub-operation finish_mul() get labels (1,1) and (2,1) respectively.
This works, invoking the example with and without a command line argument gives the same labels for finish_mul(). But it is cumbersome to drag this program counter around all the code, and (what is worse) it is easy to forget to increment it and to create sub program counters at the right times.
So have anybody done something similar where you wanted to start a bunch of operations which might start sub-operations themselves and where everything needed to be deterministically labelled?