[Twisted-Python] Caching mechanism
hi, is there any cache mechanism for twisted? Could one point me please to that docs? Thank you.
On 07:28 pm, vitaly@synapticvision.com wrote:
There's not really anything in Twisted for this. There are a lot of general Python solutions, though: http://www.google.com/search?q=python+caching&ie=utf-8&oe=utf-8&aq=t&rls=com.ubuntu :en-US:official&client=firefox-a Jean-Paul
Part of the beauty of twisted is that you don't actually need to do anything special to achieve that, just create a global dict or other object of your choice and access it as needed. More likely what you need to look for is a cache expiration mechanism, I've linked in lrucache ( http://pypi.python.org/pypi/lrucache/0.2) in one case, but usually I just write my own constraints as appropriate. It's often misunderstood, but in twisted you are writing an actual server rather than just some methods for servicing requests. The server will run until stopped and all global or even local variables will remain in memory unless deleted or dereferenced and garbage collected. Rather than doing something special to get a persistent variable you need to do something special when you want a non-persistent one. -Andy On Mon, Nov 9, 2009 at 2:28 PM, <vitaly@synapticvision.com> wrote:
-- Blog: http://channel3b.wordpress.com Drinking good coffee makes you wise, drinking bad coffee only makes you awake.
No, clue, but I've sent the code to you off list and if anyone else wants it I'd be glad to do the same. Andy On Mon, Nov 9, 2009 at 5:47 PM, Paul Hubbard <hubbard@sdsc.edu> wrote:
-- Blog: http://channel3b.wordpress.com Drinking good coffee makes you wise, drinking bad coffee only makes you awake.
On Mon, Nov 9, 2009 at 4:47 PM, Paul Hubbard <hubbard@sdsc.edu> wrote:
There's something that's probably similar here: http://code.activestate.com/recipes/498245/ Kevin Horn
Twisted doesn't enforce any, that's part of the effect of you actually writing the server, defining, enforcing, and adhering to those limits is left to the developer. Of course you have a limited amount of space in your machine, and you probably don't want to use swap, but nothing twisted is going to limit you on, it doesn't really do anything on that level of abstraction. Andy Fundinger On Mon, Nov 9, 2009 at 6:24 PM, <vitaly@synapticvision.com> wrote:
-- Blog: http://channel3b.wordpress.com Drinking good coffee makes you wise, drinking bad coffee only makes you awake.
On Mon, Nov 09, 2009 at 06:24:20PM -0500, vitaly@synapticvision.com wrote:
Such global variable(lets say dictionary) will have any size limit?
i don't know what kind of method calls do you need to cache, but what about of memcached? twisted got a really handy client api for memcached and, you should look deeper than i did, with a bit of work you can refactor them as a decorator. memcached server also has got his policies for size limits. the drawback, but "bad luck often brings good luck", of this approach is that your code must be "deferred aware". m.
Hi, you can couple a decorator with the maybeDeferred function. this is a trivial example but it works. This cache is not cleaned and if you write your decorator in a class with a __call__ function, you can isolate the cache dictionary and a configurable lifetime, with a twisted.internet.task.LoopingCall for this. In fact, I have already talk about it in a previous message, too try talking about this technique with more advanced users than me without answer. I supposed it was not so bad. Well, I am using now this kind of code to cache sql select request from a twisted.enterprise.adbapi.ConnectionPool.runQuery and it's work fine. from twisted.internet import defer from twisted.internet import reactor import types cache = {} def memoize(f): def g(*args, **kwargs): key = (tuple(args), frozenset(kwargs.items())) if key not in cache: return f(*args, **kwargs) else: print "from cache" return cache[key] return g def saveInCache(result,key): print "saving to cache" cache[key] = result return result @memoize def asynchronousIsValidUser(user): print "Loading from network" d = defer.Deferred() reactor.callLater(2, d.callback, user in ["Alice", "Angus", "Agnes"]) return d.addCallback(saveInCache, ( (user,), frozenset() )) def printResult(result,user): if result: print "User", user ,"is authenticated" else: print "User", user ,"is not authenticated" def authenticateUser(user): print "Authenticating user ", user d = defer.maybeDeferred(asynchronousIsValidUser, user) return d.addCallback(printResult,user) reactor.callWhenRunning(authenticateUser,"Alice") reactor.callLater(1,authenticateUser,"Alice") reactor.callLater(3,authenticateUser,"Alice") reactor.callLater(5,reactor.stop) reactor.run() vitaly@synapticvision.com a écrit :
Here is the main part of my code class CacheCleanerBaseMixin: def __init__(self, cachetimer,cachelifetime): self._cache = {} self._cachelifetime = timedelta(0,cachelifetime) self._cleaner = task.LoopingCall(self.cleanCache) self._cleaner.start(cachetimer,now=False) self._cleanrun = False def cleanCache(self): if not self._cleanrun: self._cleanrun = True log.msg('Clean the cache...', logLevel=logging.DEBUG) expire = datetime.today() i = 0 for k,v in self._cache.items(): if v[0] < expire: i+=1 del self._cache[k] log.msg('%i items removed' % i, logLevel=logging.DEBUG) self._cleanrun = False class asyncmemoize(CacheCleanerBaseMixin): def __init__(self, function,cachetimer=30,cachelifetime=60): CacheCleanerBaseMixin.__init__(self, cachetimer,cachelifetime) self.function = function def __call__(self, *args, **kwargs): key = (tuple(args), frozenset(kwargs.items())) if key not in self._cache: return self.function(SqlConnection(),*args, **kwargs).addCallback(self.callback,key) else: log.msg('Retrievieng the data from the cache',logLevel=logging.DEBUG) return self._cache[key][1] def callback(self,result,key): try: self._cache[key] = [datetime.today() + self._cachelifetime, result] finally: return result class SqlConnection(object): _instance = None def __new__(cls): if cls._instance is None: cls._instance = object.__new__(cls) return cls._instance def configure(self, dbdriver, dbdriverargs, log_request=False, cache_file='sqlcache.shelve', pool_file='sqloperation.shelve', pool_maxattempt=5, pool_time=30): self.log_request = log_request self.pool = adbapi.ConnectionPool(dbdriver,**dbdriverargs) @asyncmemoize def internal_runQuery(self,query,args=None): rv = None try: if args: rv = self.pool.runQuery(query,args) else: rv = self.pool.runQuery(query) except: log.err() return rv def runQuery(self,query,args=None): try: d = defer.maybeDeferred(self.internal_runQuery, query,args) return d except: log.err() ---------------------------------------------------------------------------- Laposte.net fête ses 10 ans ! Gratuite, garantie à vie et déjà utilisée par des millions d'internautes... vous aussi, pour votre adresse e-mail, choisissez laposte.net. Laposte.net, bien + qu'une messagerie ----------------------------------------------------------------------------
On 07:28 pm, vitaly@synapticvision.com wrote:
There's not really anything in Twisted for this. There are a lot of general Python solutions, though: http://www.google.com/search?q=python+caching&ie=utf-8&oe=utf-8&aq=t&rls=com.ubuntu :en-US:official&client=firefox-a Jean-Paul
Part of the beauty of twisted is that you don't actually need to do anything special to achieve that, just create a global dict or other object of your choice and access it as needed. More likely what you need to look for is a cache expiration mechanism, I've linked in lrucache ( http://pypi.python.org/pypi/lrucache/0.2) in one case, but usually I just write my own constraints as appropriate. It's often misunderstood, but in twisted you are writing an actual server rather than just some methods for servicing requests. The server will run until stopped and all global or even local variables will remain in memory unless deleted or dereferenced and garbage collected. Rather than doing something special to get a persistent variable you need to do something special when you want a non-persistent one. -Andy On Mon, Nov 9, 2009 at 2:28 PM, <vitaly@synapticvision.com> wrote:
-- Blog: http://channel3b.wordpress.com Drinking good coffee makes you wise, drinking bad coffee only makes you awake.
No, clue, but I've sent the code to you off list and if anyone else wants it I'd be glad to do the same. Andy On Mon, Nov 9, 2009 at 5:47 PM, Paul Hubbard <hubbard@sdsc.edu> wrote:
-- Blog: http://channel3b.wordpress.com Drinking good coffee makes you wise, drinking bad coffee only makes you awake.
On Mon, Nov 9, 2009 at 4:47 PM, Paul Hubbard <hubbard@sdsc.edu> wrote:
There's something that's probably similar here: http://code.activestate.com/recipes/498245/ Kevin Horn
Twisted doesn't enforce any, that's part of the effect of you actually writing the server, defining, enforcing, and adhering to those limits is left to the developer. Of course you have a limited amount of space in your machine, and you probably don't want to use swap, but nothing twisted is going to limit you on, it doesn't really do anything on that level of abstraction. Andy Fundinger On Mon, Nov 9, 2009 at 6:24 PM, <vitaly@synapticvision.com> wrote:
-- Blog: http://channel3b.wordpress.com Drinking good coffee makes you wise, drinking bad coffee only makes you awake.
On Mon, Nov 09, 2009 at 06:24:20PM -0500, vitaly@synapticvision.com wrote:
Such global variable(lets say dictionary) will have any size limit?
i don't know what kind of method calls do you need to cache, but what about of memcached? twisted got a really handy client api for memcached and, you should look deeper than i did, with a bit of work you can refactor them as a decorator. memcached server also has got his policies for size limits. the drawback, but "bad luck often brings good luck", of this approach is that your code must be "deferred aware". m.
Hi, you can couple a decorator with the maybeDeferred function. this is a trivial example but it works. This cache is not cleaned and if you write your decorator in a class with a __call__ function, you can isolate the cache dictionary and a configurable lifetime, with a twisted.internet.task.LoopingCall for this. In fact, I have already talk about it in a previous message, too try talking about this technique with more advanced users than me without answer. I supposed it was not so bad. Well, I am using now this kind of code to cache sql select request from a twisted.enterprise.adbapi.ConnectionPool.runQuery and it's work fine. from twisted.internet import defer from twisted.internet import reactor import types cache = {} def memoize(f): def g(*args, **kwargs): key = (tuple(args), frozenset(kwargs.items())) if key not in cache: return f(*args, **kwargs) else: print "from cache" return cache[key] return g def saveInCache(result,key): print "saving to cache" cache[key] = result return result @memoize def asynchronousIsValidUser(user): print "Loading from network" d = defer.Deferred() reactor.callLater(2, d.callback, user in ["Alice", "Angus", "Agnes"]) return d.addCallback(saveInCache, ( (user,), frozenset() )) def printResult(result,user): if result: print "User", user ,"is authenticated" else: print "User", user ,"is not authenticated" def authenticateUser(user): print "Authenticating user ", user d = defer.maybeDeferred(asynchronousIsValidUser, user) return d.addCallback(printResult,user) reactor.callWhenRunning(authenticateUser,"Alice") reactor.callLater(1,authenticateUser,"Alice") reactor.callLater(3,authenticateUser,"Alice") reactor.callLater(5,reactor.stop) reactor.run() vitaly@synapticvision.com a écrit :
Here is the main part of my code class CacheCleanerBaseMixin: def __init__(self, cachetimer,cachelifetime): self._cache = {} self._cachelifetime = timedelta(0,cachelifetime) self._cleaner = task.LoopingCall(self.cleanCache) self._cleaner.start(cachetimer,now=False) self._cleanrun = False def cleanCache(self): if not self._cleanrun: self._cleanrun = True log.msg('Clean the cache...', logLevel=logging.DEBUG) expire = datetime.today() i = 0 for k,v in self._cache.items(): if v[0] < expire: i+=1 del self._cache[k] log.msg('%i items removed' % i, logLevel=logging.DEBUG) self._cleanrun = False class asyncmemoize(CacheCleanerBaseMixin): def __init__(self, function,cachetimer=30,cachelifetime=60): CacheCleanerBaseMixin.__init__(self, cachetimer,cachelifetime) self.function = function def __call__(self, *args, **kwargs): key = (tuple(args), frozenset(kwargs.items())) if key not in self._cache: return self.function(SqlConnection(),*args, **kwargs).addCallback(self.callback,key) else: log.msg('Retrievieng the data from the cache',logLevel=logging.DEBUG) return self._cache[key][1] def callback(self,result,key): try: self._cache[key] = [datetime.today() + self._cachelifetime, result] finally: return result class SqlConnection(object): _instance = None def __new__(cls): if cls._instance is None: cls._instance = object.__new__(cls) return cls._instance def configure(self, dbdriver, dbdriverargs, log_request=False, cache_file='sqlcache.shelve', pool_file='sqloperation.shelve', pool_maxattempt=5, pool_time=30): self.log_request = log_request self.pool = adbapi.ConnectionPool(dbdriver,**dbdriverargs) @asyncmemoize def internal_runQuery(self,query,args=None): rv = None try: if args: rv = self.pool.runQuery(query,args) else: rv = self.pool.runQuery(query) except: log.err() return rv def runQuery(self,query,args=None): try: d = defer.maybeDeferred(self.internal_runQuery, query,args) return d except: log.err() ---------------------------------------------------------------------------- Laposte.net fête ses 10 ans ! Gratuite, garantie à vie et déjà utilisée par des millions d'internautes... vous aussi, pour votre adresse e-mail, choisissez laposte.net. Laposte.net, bien + qu'une messagerie ----------------------------------------------------------------------------
participants (7)
-
Andy Fundinger -
exarkun＠twistedmatrix.com -
Kevin Horn -
Marco Giusti -
mardiros -
Paul Hubbard -
vitaly＠synapticvision.com