shelf-like list?

[kj]

In <mailman.2125.1281849995.1673.python-list at python.org> Chris Rebert <clp2 at rebertia.com> writes:

>On Sat, Aug 14, 2010 at 5:13 PM, kj <no.email at please.post> wrote:
>> In <af7fdb85-8c87-434e-94f3-18d8729bf9be at l25g2000prn.googlegroups.com> Ra=
>ymond Hettinger <python at rcn.com> writes:
>>>On Aug 12, 1:37=3DA0pm, Thomas Jollans <tho... at jollybox.de> wrote:
>>>> On Tuesday 10 August 2010, it occurred to kj to exclaim:
>>>>
>>>> > I'm looking for a module that implements "persistent lists": objects
>>>> > that behave like lists except that all their elements are stored
>>>> > on disk. =3DA0IOW, the equivalent of "shelves", but for lists rather
>>>> > than a dictionaries.
>>> . . .
>>>> You could simply use pickle to save the data every once in a while.
>>
>>>That is a very reasonable solution.
>>
>> Sorry I don't follow. =C2=A0Some sample code would be helpful.

>I would assume something along the lines of (untested):

>from pickle import dump

>MOD_THRESHOLD =3D 42

>class PersistentList(list):
>    def __init__(self, filepath):
>        self.filepath =3D filepath
>        self._mod_count =3D 0

>    def save(self):
>        with open(self.filepath, 'w') as f:
>            dump(self, f)
>        self._mod_count =3D 0

>    def append(self, *args, **kwds):
>        super(PersistentList, self).append(*args, **kwds)
>        self._mod_count +=3D 1
>        if self._mod_count >=3D MOD_THRESHOLD:
>        # obviously time-since-last-dump or other
>        #   more sophisticated metrics might be used instead
>            self.save()

Even though it is saved periodically to disk, it looks like the
whole list remains in memory all the time?  (If so, it's not what
I'm looking for; the whole point of saving stuff to disk is to keep
the list's memory footprint low.)

~K