[Numpy-discussion] Pickle, pytables, and sqlite - loading and saving recarray's

Vincent Nijs v-nijs at kellogg.northwestern.edu
Thu Jul 19 22:42:42 EDT 2007

I am interesting in using sqlite (or pytables) to store data for scientific
research. I wrote the attached test program to save and load a simulated
11x500,000 recarray. Average save and load times are given below (timeit
with 20 repetitions). The save time for sqlite is not really fair because I
have to delete the data table each time before I create the new one. It is
still pretty slow in comparison. Loading the recarray from sqlite is
significantly slower than pytables or cPickle. I am hoping there may be more
efficient ways to save and load recarray¹s from/to sqlite than what I am now
doing. Note that I infer the variable names and types from the data rather
than specifying them manually.

I¹d luv to hear from people using sqlite, pytables, and cPickle about their

saving recarray with cPickle:       1.448568 sec/pass
saving recarray with pytable:      3.437228 sec/pass
saving recarray with sqlite:         193.286204 sec/pass

loading recarray using cPickle:    0.471365 sec/pass
loading recarray with pytable:     0.692838 sec/pass
loading recarray with sqlite:        15.977018 sec/pass


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070719/20749e95/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: load_tables_test.py
Type: application/octet-stream
Size: 4154 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070719/20749e95/attachment.obj>

More information about the NumPy-Discussion mailing list