fastest data / database format for reading large files
clp2 at rebertia.com
Sun Oct 28 10:26:14 CET 2012
On Tue, Oct 16, 2012 at 11:35 AM, Pradipto Banerjee
<pradipto.banerjee at adainvestments.com> wrote:
> I am working with a series of large files with sizes 4 to 10GB and may need to read these files repeated. What data format (i.e. pickle, json, csv, etc.) is considered the fastest for reading via python?
Pickle /ought/ to be fastest, since it's binary (unless you use the
oldest protocol version) and native to Python. Be sure to specify
HIGHEST_PROTOCOL and use cPickle.
You might consider using SQLite (or some other database) if you will
be doing queries over the data that would be amenable to SQL or
P.S. The verbose disclaimer at the end of your emails is kinda annoying...
More information about the Python-list