Memory efficient tuple storage

[Aaron Brady]

On Mar 13, 1:13 pm, "psaff... at googlemail.com"
<psaff... at googlemail.com> wrote:
> Thanks for all the replies.
>
> First of all, can anybody recommend a good way to show memory usage? I
> tried heapy, but couldn't make much sense of the output and it didn't
> seem to change too much for different usages. Maybe I was just making
> the h.heap() call in the wrong place. I also tried getrusage() in the
> resource module. That seemed to give 0 for the shared and unshared
> memory size no matter what I did. I was calling it after the function
> call the filled up the lists. The memory figures I give in this
> message come from top.
>
> The numpy solution does work, but it uses more than 1GB of memory for
> one of my 130MB files. I'm using
>
> np.dtype({'names': ['chromo', 'position', 'dpoint'], 'formats': ['S6',
> 'i4', 'f8']})
>
> so shouldn't it use 18 bytes per line? The file has 5832443 lines,
> which by my arithmetic is around 100MB...?
snip

Sorry, did not study your post.  But can you use a ctypes.Structure?
Or, can you use a database or mmap to keep the data out of memory?
Or, how would you feel about a mini extension in C?