[Python-ideas] Fwd: Using functools.lru_cache only on some arguments of a function

Sorry for the late reply everyone. I think relying on closures, while a solution, is messy. I'd still much prefer a way to tell lru_cache to merely ignore certain arguments. I'll use some variant of the storing-the-partial-progress-as-an-attribute-on-the-cached-recursive-function method (either with the cached-recursive hidden from the top level via the try/except stuff, or with a more simple wrapper function). I've further considered my original proposal, and rather than naming it "arg_filter", I realized that builtins like sorted(), min(), max(), etc all already have the exact same thing -- a "key" argument which transforms the elements to the user's purpose. (In the sorted/min/max case, it's called on the elements of the argument rather than the argument itself, but it's still the same concept.) So basically, my original proposal with renaming from arg_filter to key, is tantamount to extending the same functionality from sorted/min/max to lru_cache as well. As has been pointed out, my own use case is almost certainly *not* the only use case. The implementation and interface are both simple, and simpler than the alternatives which I'll rely on for now (wrappers and closures, or worse, global singletons etc). I would still like to see it in the stdlib in the future. I've appended a largely similar patch with the proposed additions (there's some internal variable renaming to avoid confusion, resulting in a longer diff). Thanks again for all the input. -Bill ----------------------------------------------------------------------------------------------------------------- https://hg.python.org/cpython/file/3.5/Lib/functools.py diff functools.py.orig functools.py 363c363 < def _make_key(args, kwds, typed, ---

def _make_key(args, kwds, typed, key, 377c377,379 < key = args

if key is not None: args, kwds = key(args, kwds) cache_key = args

380c382 < key += kwd_mark ---

cache_key += kwd_mark

382c384 < key += item ---

cache_key += item

384c386 < key += tuple(type(v) for v in args) ---

cache_key += tuple(type(v) for v in args)

386,389c388,391 < key += tuple(type(v) for k, v in sorted_items) < elif len(key) == 1 and type(key[0]) in fasttypes: < return key[0] < return _HashedSeq(key) ---

cache_key += tuple(type(v) for k, v in sorted_items) elif len(cache_key) == 1 and type(cache_key[0]) in fasttypes: return cache_key[0] return _HashedSeq(cache_key)

391c393 < def lru_cache(maxsize=128, typed=False): ---

def lru_cache(maxsize=128, typed=False, key=None): 400a403,407 If *key* is not None, it must be a callable which acts on the arguments passed to the function. Its return value is used in place of the actual arguments. It works analogously to the *key* argument to the builtins sorted, max, and min.

421a429,431

if key is not None and not callable(key): raise TypeErrpr('Expected key to be a callable')

423c433,434 < wrapper = _lru_cache_wrapper(user_function, maxsize, typed, _CacheInfo) ---

wrapper = _lru_cache_wrapper(user_function, maxsize, typed, key, _CacheInfo)

428c439 < def _lru_cache_wrapper(user_function, maxsize, typed, _CacheInfo): ---

def _lru_cache_wrapper(user_function, maxsize, typed, key, _CacheInfo): 456,457c467,468 < key = make_key(args, kwds, typed) < result = cache_get(key, sentinel)

cache_key = make_key(args, kwds, typed, key) result = cache_get(cache_key, sentinel)

462c473 < cache[key] = result ---

cache[cache_key] = result

471c482 < key = make_key(args, kwds, typed) ---

cache_key = make_key(args, kwds, typed, key)

473c484 < link = cache_get(key) ---

link = cache_get(cache_key)

487c498 < if key in cache: ---

if cache_key in cache:

496c507 < oldroot[KEY] = key ---

oldroot[KEY] = cache_key

513c524 < cache[key] = oldroot ---

cache[cache_key] = oldroot

517,518c528,529 < link = [last, root, key, result] < last[NEXT] = root[PREV] = cache[key] = link ---

link = [last, root, cache_key, result] last[NEXT] = root[PREV] = cache[cache_key] = link

On Wed, Dec 30, 2015 at 11:10 PM, Michael Selik wrote:

On Tue, Dec 29, 2015 at 2:14 AM Franklin? Lee < leewangzhong+python@gmail.com> wrote:

...

On Sat, Dec 12, 2015 at 1:34 PM, Michael Selik wrote:

...

On Fri, Dec 11, 2015, 8:20 PM Franklin? Lee < leewangzhong+python@gmail.com> wrote:

...

rather than using a closure, depending on how intertwined the code

This whole thing is probably best implemented as two separate functions paths are

...

for the shortcut/non-shortcut versions.

I like the closure because it has semantic ownership: the inner function is a worker for the outer function.

True, a closure has better encapsulation, making it less likely someone will misuse the helper function. On the other hand, that means there's less modularity and it would be difficult for someone to use the inner function. It's hard to know the right choice without seeing the exact problem the original author was working on.

...

...

...

On Fri, Dec 11, 2015 at 8:01 PM, Franklin? Lee wrote:

...

1. Rewrite your recursive function so that the partial state is a nonlocal variable (in the closure), and memoize the recursive part.

I'd flip the rare-case to the except block and put the normal-case in the try block. I believe this will be more compute-efficient and more readable.

The rare case is in the except block, though.

You're correct. Sorry, I somehow misinterpreted the comment, "# To trigger the exception the first time" as indicating that code path would run only once.