save tuple of simple data types to disk (low memory foot print)

[Gelonida N]

On 10/29/2011 01:08 AM, Roy Smith wrote:
> In article <mailman.2293.1319834877.27778.python-list at python.org>,
>  Gelonida N <gelonida at gmail.com> wrote:
> 
>> I would like to save many dicts with a fixed amount of keys
>> tuples to a file  in a memory efficient manner (no random, but only
>> sequential access is required)
> 
> There's two possible scenarios here.  One, which you seem to be 
> exploring, is to carefully study your data and figure out the best way 
> to externalize it which reduces volume.
> 
> The other is to just write it out in whatever form is most convenient 
> (JSON is a reasonable thing to try first), and compress the output.  Let 
> the compression algorithms worry about extracting the entropy.  You may 
> be surprised at how well it works.  It's also an easy experiment to try, 
> so if it doesn't work well, at least it didn't cost you much to find out.


Yes I have to make some more tests to see the defference between
just compressing aplain format (JSON / pickle) and compressing the
'optimized' representation.