On Wed, Apr 29, 2009 at 10:33:27PM +0000, Benjamin Peterson wrote:

I think it would be nice to add a "cache" argument to the property() constructor. When "cache" was True, property would only ask the getter function once for the result. This would simplify properties that require expensive operations to compute.

http://ppa.cvs.sourceforge.net/viewvc/*checkout*/ppa/QPS/qps/qUtils.py See class CachedAttribute. Just use @CachedAttribute instead of @property. Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd@phd.pp.ru Programmers don't die, they just GOSUB without RETURN.

Mathias Panzenböck writes:

You mean like the once methods in eiffel?

I'm not sure. I've never used Eiffel.

On Wed, Apr 29, 2009 at 8:31 PM, Chris Rebert wrote:

On Wed, Apr 29, 2009 at 4:23 PM, Benjamin Peterson wrote:

...

Mathias Panzenböck writes:

...

You mean like the once methods in eiffel?

I'm not sure. I've never used Eiffel.

http://en.wikipedia.org/wiki/Eiffel_(programming_language)#Once_routines

Sounds pretty much like what you're suggesting.

Properties are not plain methods though, they also support setters and deleters. Should the cache be cleared after a set/delete ? Or does cache=True implies fset=fdel=None ? George

On Thu, 30 Apr 2009 08:33:27 am Benjamin Peterson wrote:

I think it would be nice to add a "cache" argument to the property() constructor. When "cache" was True, property would only ask the getter function once for the result. This would simplify properties that require expensive operations to compute.

-1 on an extra parameter to property. +1 on a cache decorator. It is clear and simple enough to write something like: @property @cache # or "once" if you prefer the Eiffel name def value(self): pass Such a decorator could then be used on any appropriate function, not just for properties. -- Steven D'Aprano

Le Thu, 30 Apr 2009 13:57:07 +1000, Steven D'Aprano s'exprima ainsi:

-1 on an extra parameter to property.

+1 on a cache decorator.

It is clear and simple enough to write something like:

@property @cache # or "once" if you prefer the Eiffel name def value(self): pass

I vote +1 for this version. Cache and property are distinct features. Denis ------ la vita e estrany

I think one thing that should be addressed is how to change the cached value, for example after the setter is called. Having multiple decorators would seem to make this slightly more complicated. If we were using only one, it would be easy to just add a cache property to the decorated function, which the setter could easily deal with. @property(cache=True) def foo(self): return(self.bar * 99) @foo.setter def foo(self, bar): self.bar = bar / 99 del foo.cache # or foo.cache = bar I guess this doesn't actually work anyway because of the way setters are specified, since it would refer to the setter instead of the property. Hmm. Perhaps, since the return value of setters is currently meaingless, it could be written into the cache, or used to specify if the cache should be cleared? Or just clear the cache automatically any time the setter is called. I realize that these ideas would probably introduce excessive overhead and have some other problems, but I think this is something that needs to be figured out before too long, so having some more ideas out there probably won't hurt. - JB On 2009-04-30, spir wrote:

Le Thu, 30 Apr 2009 13:57:07 +1000, Steven D'Aprano s'exprima ainsi:

...

-1 on an extra parameter to property.

+1 on a cache decorator.

It is clear and simple enough to write something like:

@property @cache # or "once" if you prefer the Eiffel name def value(self): pass

I vote +1 for this version. Cache and property are distinct features.

Denis ------ la vita e estrany _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

On Thu, 30 Apr 2009 10:36:17 pm Jeremy Banks wrote:

I think one thing that should be addressed is how to change the cached value, for example after the setter is called.

For the use-case being discussed, you generally don't want to cache a value that changes. You generally want to cache a value which is expensive to calculate, but never changes. However, there are uses for caches that expire after some time. I just can't think of any where I'd want them to be accessed via a property instead of a function call.

Having multiple decorators would seem to make this slightly more complicated.

I don't see why. The cache decorator could expose an interface to expire the cached value. Here's a quick&dirty but working solution. (Tested under Python 2.5.) def once(func): class Cache(object): def __call__(self, *args, **kwargs): try: return self._value except AttributeError: result = func(*args, **kwargs) self._value = result return result def expire(self): del self._value return Cache() class Test(object): @property @once def expensive(self): import time time.sleep(20) return 1 Works like a charm :) (At least for a solution I knocked up in 30 seconds.) The only downside is that to expire the cache, you need the not-very-obvious call: Test.expensive.fget.expire() rather than t.expensive.expire(), which can't work for obvious reasons.

If we were using only one, it would be easy to just add a cache property to the decorated function, which the setter could easily deal with. ... I guess this doesn't actually work anyway because of the way setters are specified, since it would refer to the setter instead of the property. Hmm.

I'm not exactly sure that a solution that doesn't work can be said to "easily deal with" anything :) -- Steven D'Aprano

spir writes:

now I wonder whether it's really worth it. Because usually it's a very simple thing to implement -- often two obvious lines of code -- and easy to

understand, even for someone who does

not know the principle yet.

Well, for one, a generic implementation may have to be thread-safe. Also, while it's easy to implement, it's the kind of useful primitive - like enumerate() and others - which is nice to have builtin in the language or the standard library. Regards Antoine.

On 30 Apr 2009, at 20:22, Scott David Daniels wrote:

Steven D'Aprano wrote:

...

Here's a quick&dirty but working solution. (Tested under Python 2.5.) def once(func): class Cache(object): def __call__(self, *args, **kwargs): try: return self._value except AttributeError: result = func(*args, **kwargs) self._value = result return result def expire(self): del self._value return Cache()

This is slightly better (name change as in Antoine Pitrou's comment):

class cached(object):

def __init__(self, function): self._function = function self._cache = {}

def __call__(self, *args): try: return self._cache[args] except KeyError: self._cache[args] = self._function(*args) return self._cache[args]

def expire(self, *args): del self._cache[args]

You're going to run into trouble with unhashable args. -- Arnaud

Le Thu, 30 Apr 2009 12:22:48 -0700, Scott David Daniels s'exprima ainsi:

This is slightly better (name change as in Antoine Pitrou's comment):

class cached(object):

def __init__(self, function): self._function = function self._cache = {}

def __call__(self, *args): try: return self._cache[args] except KeyError: self._cache[args] = self._function(*args) return self._cache[args]

def expire(self, *args): del self._cache[args]

(Aside from the hashable issue pointed by Arnaud) I wonder about having the whole parameter tuple as key for caching. In packrat parsing, you may have more than one parameter (including the source, indeed) but only one is relevant for memoizing (the position). Cache has to be reset anyway when starting a new parse, so that having the source in keys is irrelevant. Typically, if not a single value, I guess saved results form a simple array depending on an ordinal. Denis ------ la vita e estrany

On Thu, Apr 30, 2009 at 4:19 PM, spir wrote:

(Aside from the hashable issue pointed by Arnaud) I wonder about having the whole parameter tuple as key for caching. In packrat parsing, you may have more than one parameter (including the source, indeed) but only one is relevant for memoizing (the position). Cache has to be reset anyway when starting a new parse, so that having the source in keys is irrelevant. Typically, if not a single value, I guess saved results form a simple array depending on an ordinal.

I think It's clear by now that caching in the general case is not trivial, both in terms of API and implementation. That makes the original request - caching properties only - more appealing since most problems go away if there are no parameters. In the simplest case where cache expiration is not supported, cachedproperty can be a 7-line decorator: def cachedproperty(fget): def fget_wrapper(self): try: return fget_wrapper._cached except AttributeError: fget_wrapper._cached = value = fget(self) return value return property(fget_wrapper, doc=fget.__doc__) Cache expiration can be exposed by (ab)using the deleter; "del obj.prop" looks much better than "ObjType.prop.fget.expire(obj)": def cachedproperty(fget): def fget_wrapper(self): try: return fget_wrapper._cached except AttributeError: fget_wrapper._cached = value = fget(self) return value def fdel(self): try: del fget_wrapper._cached except AttributeError: pass return property(fget_wrapper, fdel=fdel, doc=fget.__doc__) And finally here's a general version that supports properly the new in 2.6 getter()/setter()/deleter() methods (note that setting a property expires the cache, just like delete): class cachedproperty(property): def __init__(self, fget=None, fset=None, fdel=None, doc=None): if fget is not None: def fget_wrapper(obj): try: return self._cached except AttributeError: self._cached = value = fget(obj) return value else: fget_wrapper = None if fset is not None: def fset_wrapper(obj, value): fset(obj,value) try: del self._cached except AttributeError: pass else: fset_wrapper = None if fdel is not None: def fdel_wrapper(obj): fdel(obj) try: del self._cached except AttributeError: pass else: def fdel_wrapper(obj): try: del self._cached except AttributeError: pass super(cachedproperty,self).__init__(fget_wrapper, fset_wrapper, fdel_wrapper, doc) self.__doc__ = doc or getattr(fget, '__doc__', None) # store for getter() / setter() / deleter() self._fget, self._fset, self._fdel = fget, fset, fdel def getter(self, getter): return self.__class__(getter, self._fset, self._fdel, self.__doc__) def setter(self, setter): return self.__class__(self._fget, setter, self._fdel, self.__doc__) def deleter(self, deleter): return self.__class__(self._fget, self._fset, deleter, self.__doc__) George

On Thu, Apr 30, 2009 at 7:37 PM, Ben Finney wrote:

George Sakkis writes:

...

I think It's clear by now that caching in the general case is not trivial, both in terms of API and implementation. That makes the original request - caching properties only - more appealing since most problems go away if there are no parameters.

Nevertheless, the “memoize” pattern *is* well-understood, and already implemented for Python as discussed earlier. It covers the “cached property” as a degenerate (i.e. simpler) case. Making a specific decorator that *only* addresses “cached property” is too narrow, IMO.

I think a “memoize” decorator (by whatever name) is the right level of generality to address this use case, and has the advantage of existing implementations.

IMHO no existing implementation is good enough for addition to the standard library. Off the top of my head, some of the issues a general and robust implementation should address are: - Non hashable parameters. - Caching keys based on a subset of the provided parameters (or perhaps more general a key function). - Bounded-length caches. - Different expiration policies for bounded-length caches. - As good performance as possible given the previous constraints. Of course one can write a vanilla version in a few lines, and indeed searching for "memoize" in code.activestate.com returns no less than 125 results (including one I posted 4.5 years ago). I think the sheer number of the posted recipes is an indication that none of them has "solved" the problem for good, otherwise people would just reuse it instead of keep posting alternatives. George

On Fri, 1 May 2009 09:27:13 am George Sakkis wrote:

I think It's clear by now that caching in the general case is not trivial, both in terms of API and implementation. That makes the original request - caching properties only - more appealing since most problems go away if there are no parameters.

In the simplest case where cache expiration is not supported, cachedproperty can be a 7-line decorator: [snip]

In my opinion, properties aren't special enough to need a special cachedproperty() decorator. Especially not get-only properties that never expire, since they're just a function taking a single argument that returns a constant result. A generic cache that runs on any function should be sufficient, and more useful than one that is specific to properties. Properties are not the only application for caching expensive function calls that never change. -- Steven D'Aprano

On Fri, 1 May 2009 06:59:55 am Jared Grubb wrote:

On 30 Apr 2009, at 12:22, Scott David Daniels wrote:

...

This is slightly better (name change as in Antoine Pitrou's comment):

class cached(object):

def __init__(self, function): self._function = function self._cache = {}

def __call__(self, *args): try: return self._cache[args] except KeyError: self._cache[args] = self._function(*args) return self._cache[args]

def expire(self, *args): del self._cache[args]

The only thing I dislike is how many dictionary lookups are required in order to return the value after it's been cached. I count 4 lookups (object.prop, prop.__call__, self._cache, and self._cache[args]). These add up, especially if object.prop could have returned the value immediately without having to go through so much indirection (but this is not currently possible)

But you shouldn't be comparing a cheap attribute requiring one key lookup to a cache requiring four lookups. You should be comparing an expensive function call to four lookups. If the function isn't expensive, there's no value in caching it. -- Steven D'Aprano