Cache-Commit Dictionaries (was: Re: survey: is shelve broken?)

[holger krekel]

Hello Alex, hello everybody,

After some seriously thinking about your
suggestion/patch for fixing the behaviour
of 'shelve' i think i have found a better solution
which

- avoids changing the shelve implementation
  at all and so avoids the previously discussed 
  incompatibilities.

- introduce a more general purpose "cache-commit"
  Dictionary which is useful for many cases.
  It's not much larger than the special-cased
  shelve-patch on sf.

A cache-commit dictionary ('ccdict' for short) is 
a very *simple* transactional Wrapper-Dictionary with
the following (to be further discuessed) properties:

- It wraps a dictionary-like object which
  in turn only needs to provide 
  __getitem__, __setitem__, __delitem__ 
  
- on first access of an element 
  ccdict makes a lookup on the underlying
  dict and caches the item. 

- the next accesses work with the cached thing. 
  dict-semantics as expected (e.g. your case) 
  are provided. 

- commit() transfers the items in the
  cache to the underlying dict and clears
  the cache.

- deleting an item is deferred and actually happens 
  on commit() time. deleting an item and later on
  assigning to it works as expected. 

- deleting an ccdict-instance does *not* commit any changes.
  You have to explicitely call commit().

- clear() only cleares the cache and not the underlying
  dict (my intuition, yours also?)

- setdefault/get work as expected.

- (not implemented) provide iterator semantics?

The name 'cache-commit' refers to the two
prime characteristics:

- caching accesses and 

- committing in one go all changes.

The general ccdict does not call any sync/close/open 
functions. You have to feed it a dictish/shelvish object 
yourself!

But i have included a 15-line ccshelve which
uses ccdict and provides this functionality:

>>> c = ccshelve('/tmp/testshelve', factory=shelve.open)
>>> c['hello']='world'
>>> c['liste']=[1,2,3]
>>> c['liste'].append(42)
>>> c.close()
>>> c.reopen()
>>> print c['liste']
[1, 2, 3, 42]

Just execute the attached proof-of-concept-code 
and you'll (hopefully) see the testexample running and 
can directly start fiddling with it yourself. Hope you like it and
stop thinking that i want to infest the world
with globals, implicitisms and 'exec'-code :-)

regards,

    holger
-------------- next part --------------
#!/usr/bin/env python

# requires python 2.2
class ccdict(dict):
    """ cache-commit dictionary
        must be initialized with a dictish-object.
        self.STATEDICT is the underlying dict
        (in capital letters to visually emphasize)
    """
    def __init__(self, STATEDICT):
        dict.__init__(self)
        self.STATEDICT = STATEDICT
        self.deleted={}

    def get(self, key, default=None):
        if dict.has_key(self,key):
            v=dict.__getitem(self,key)
        else:
            if self.deleted.get(key):
                v=self[key]=default
                del deleted[key]
            else:
                v=self[key]=self.STATEDICT.get(key,default)
        return v

    def __getitem__(self, key):
        if dict.has_key(self,key):
            v = dict.__getitem__(self,key)
        else:
            if self.deleted.has_key(key):
                raise KeyError,"'%s' has been deleted" % key
            v=self[key]=self.STATEDICT[key]
        return v

    def has_key(self, key):
        if dict.has_key(self,key):
            return 1
        if self.deleted.has_key(key):
            return 0
        if self.STATEDICT.has_key(key):
            return 1
        return 0

    def __delitem__(self, key):
        if self.STATEDICT.has_key(key):
            if self.deleted.has_key(key):
                raise KeyError,"'%s' has been deleted" % key
            self.deleted[key]=1
            if not self.has_key(key):
                return
        dict.__delitem__(self, key)

    def clear(self):
        self.deleted.clear()
        dict.clear(self)
    
    def setdefault(self, key, default):
        if self.has_key(key):
            v = dict.__getitem__(self,key)
        else:
            if self.deleted.has_key(key):
                v=self[key]=default
                del self.deleted[key]
            else:
                if self.STATEDICT.has_key(key):
                    v=self[key]=self.STATEDICT[key]
                else:
                    v=self[key]=default
        return v

    def commit(self):
        for key,obj in self.items():
            self.STATEDICT[key]=obj
            self.deleted[key]=0
        for key,value in self.deleted.items():
            if value:
                del self.STATEDICT[key]
        dict.clear(self)
        self.deleted={}

import shelve
class ccshelve(ccdict):
    """ quick example giving cache-commit 
        access to a shelve
    """
    def __init__(self, name, factory=shelve.open):
        self.name=name
        self.factory=factory
        self.shelf=factory(name)
        ccdict.__init__(self, self.shelf)

    def sync(self):
        self.commit()
        self.shelf.sync()

    def __del__(self):
        self.shelf.close()

    def close(self):
        self.sync()
        self.shelf.close()  
        self.shelf=None

    def reopen(self):
        if self.shelf:
            self.shelf.close() 
        self.shelf = self.factory(self.name)
        ccdict.__init__(self,self.shelf)

    def clear(self):
        raise NotImplementError,"clear() semantics not yet defined"

if __name__=='__main__':
    testexample="""
# yes i know about doctest :-)
from __main__ import ccshelve

c = ccshelve('/tmp/testshelve')
c['hello']='world'
c['liste']=[1,2,3]
c['liste'].append(42)
c.commit()
c.reopen()

del c['hello']
try:
    c['hello']
except KeyError:
    pass
else:
    print "never executed!"

print c.setdefault('hello',['python'])
print c.get('nonexisting','nonexisting key, ok')
c['hello'].append('rules')
print c['hello']

c.close()
c.reopen()
print c['hello']
"""
    import code
    ic = code.InteractiveConsole()
    for line in testexample.split('\n'):
        print ">>>",line
        ic.push(line)
    ic.interact('use c.reopen() to continue fiddling around')