Sequential Object Store

[Alex Willmer]

On Aug 7, 5:26 pm, GZ <zyzhu2... at gmail.com> wrote:
> I am wondering if there is a module that can persist a stream of
> objects without having to load everything into memory. (For this
> reason, I think Pickle is out, too, because it needs everything to be
> in memory.)

>From the pickle docs it looks like you could do something like:

try:
    import cPickle as pickle
except ImportError
    import pickle

file_obj = open('whatever', 'wb')
p = pickle.Pickler(file_obj)

for x in stream_of_objects:
    p.dump(x)
    p.memo.clear()

del p
file_obj.close()

then later

file_obj = open('whatever', 'rb')
p = pickle.Unpickler(file_obj)

while True:
    try:
        x = p.load()
        do_something_with(x)
    except EOFError:
        break

Your loading loop could be wrapped in a generator function, so only
one object should be held in memory at once.