New subject: [Twisted-Python] pb.cacheable: adjustable cache depth?

Dec. 12, 2006

...
They don't have that capability, no. I guess you could modify them to do  
that, but I can't think of why you would want to.
If you're not sure that's what you want to do, then it might help to  
explain what you're trying to accomplish and someone on this list may be  
able to help you.
This is a statistical algorithm I'm trying to parallelize to run on a 
cluster, and the serial version strongly suggests that the behavior I've 
asked for would be a big help.

I'll try to describe my serial algorithm succinctly, any advice on 
parallelizing it would be extremely welcome.

I have a bunch of objects which are nodes in a  directed acyclic graph. 
I've pseudocoded them in the attached file. What I'd like to do is push 
some of the nodes off to other processes running on the cluster and have 
them retain references to their parents. I'll then make repeated calls 
to all nodes' get_prob() methods and periodically update their values.

It would be nice to make Node.value something like a Cacheable. I'd like 
it to store n remote references, with a timestamp (this just needs to be 
an integer counter, as in the pseudocode) attached to each. When 
queried, it could return its timestamp and value as a tuple.

Thanks again,
Anand

class Node(object):

	value		#A sizeable object, usually a numpy array.
	timestamp	#An integer, incremented every time value is changed.
	parents		#The parent Nodes of self

    parent_timestamp_caches	# Remembered values of parents' timestamps the last n times
							# compute_prob() was called.

    prob_caches				# Remembered outputs of compute_prob() the last n times it
							# was called.

	# It turns out to be optimal to have n = 2.

	def compute_prob():
		"""
		A heavyweight function, should be called as little as possible.
		Returns a float value.
		"""
		return some_function([parent.value for parent in parents])

    def get_prob():
		"""
       	Check my parents' timestamps. 

		If this combination is present in parent_timestamp_caches, 
		return the appropriate value from prob_caches.

		Otherwise,
			- 	Call compute_prob()
			- 	Store this combination of parents' timestamps in
				parent_timestamp_caches
			-	Store the output of compute_prob() in prob_caches
			-	Return the output of compute_prob().
		"""

Re: [Twisted-Python] pb.cacheable: adjustable cache depth?

Anand Patil

Eric Mangold

Eric Mangold

tags

participants (2)