How to Buffer Serialized Objects to Disk

[Scott McCarty]

Sorry to ask this question. I have search the list archives and googled, but
I don't even know what words to find what I am looking for, I am just
looking for a little kick in the right direction.

I have a Python based log analysis program called petit (
http://crunchtools.com/petit). I am trying to modify it to manage the main
object types to and from disk.

Essentially, I have one object which is a list of a bunch of "Entry"
objects. The Entry objects have date, time, date, etc fields which I use for
analysis techniques. At the very beginning I build up the list of objects
then would like to start pickling it while building to save memory. I want
to be able to process more entries than I have memory. With a strait list it
looks like I could build from xreadlines(), but once you turn it into a more
complex object, I don't quick know where to go.

I understand how to pickle the entire data structure, but I need something
that will manage the memory/disk allocation?  Any thoughts?

Gracias
Scott M
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20110112/597bd14e/attachment.html>