Been digging ever since I posted this. I suspected that the response might be use a database. I am worried I am trying to reinvent the wheel. The problem is I don't want any dependencies and I also don't need persistence program runs. I kind of wanted to keep the use of petit very similar to cat, head, awk, etc. But, that said, I have realized that if I provide the analysis features as an API, you very well, might want persistence between runs.<div>
<br></div><div>What about using an array inside a shelve?<div><br></div><div>Just got done messing with this in python shell:</div><div><br></div><div>import shelve</div><div><br></div><div>d = shelve.open(filename="/root/test.shelf", protocol=-1)</div>
<div><br></div><div>d["log"] = ()</div><div><meta http-equiv="content-type" content="text/html; charset=utf-8"><div>d["log"].append("test1")</div><meta http-equiv="content-type" content="text/html; charset=utf-8"><div>
d["log"].append("test2")</div><meta http-equiv="content-type" content="text/html; charset=utf-8"><div>d["log"].append("test3")</div></div><div><br></div><div>Then, always interacting with d["log"], for example:</div>
<div><br></div><div>for i in d["log"]:</div><div> print i</div><div><br></div><div>Thoughts?<br><br></div><div><br></div><div>I know this won't manage memory, but it will keep the footprint down right?<br>
<div class="gmail_quote">On Wed, Jan 12, 2011 at 5:04 PM, Peter Otten <span dir="ltr"><__<a href="mailto:peter__@web.de">peter__@web.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div><div></div><div class="h5">Scott McCarty wrote:<br>
<br>
> Sorry to ask this question. I have search the list archives and googled,<br>
> but I don't even know what words to find what I am looking for, I am just<br>
> looking for a little kick in the right direction.<br>
><br>
> I have a Python based log analysis program called petit (<br>
> <a href="http://crunchtools.com/petit" target="_blank">http://crunchtools.com/petit</a>). I am trying to modify it to manage the main<br>
> object types to and from disk.<br>
><br>
> Essentially, I have one object which is a list of a bunch of "Entry"<br>
> objects. The Entry objects have date, time, date, etc fields which I use<br>
> for analysis techniques. At the very beginning I build up the list of<br>
> objects then would like to start pickling it while building to save<br>
> memory. I want to be able to process more entries than I have memory. With<br>
> a strait list it looks like I could build from xreadlines(), but once you<br>
> turn it into a more complex object, I don't quick know where to go.<br>
><br>
> I understand how to pickle the entire data structure, but I need something<br>
> that will manage the memory/disk allocation? Any thoughts?<br>
<br>
</div></div>You can write multiple pickled objects into a single file:<br>
<br>
import cPickle as pickle<br>
<br>
def dump(filename, items):<br>
with open(filename, "wb") as out:<br>
dump = pickle.Pickler(out).dump<br>
for item in items:<br>
dump(item)<br>
<br>
def load(filename):<br>
with open(filename, "rb") as instream:<br>
load = pickle.Unpickler(instream).load<br>
while True:<br>
try:<br>
item = load()<br>
except EOFError:<br>
break<br>
yield item<br>
<br>
if __name__ == "__main__":<br>
filename = "tmp.pickle"<br>
from collections import namedtuple<br>
T = namedtuple("T", "alpha beta")<br>
dump(filename, (T(a, b) for a, b in zip("abc", [1,2,3])))<br>
for item in load(filename):<br>
print item<br>
<br>
To get random access you'd have to maintain a list containing the offsets of<br>
the entries in the file.<br>
However, a simple database like SQLite is probably sufficient for the kind<br>
of entries you have in mind, and it allows operations like aggregation,<br>
sorting and grouping out of the box.<br>
<font color="#888888"><br>
Peter<br>
</font><div><div></div><div class="h5"><br>
--<br>
<a href="http://mail.python.org/mailman/listinfo/python-list" target="_blank">http://mail.python.org/mailman/listinfo/python-list</a><br>
</div></div></blockquote></div><br></div></div>