I am interesting in using sqlite (or pytables) to store data for scientific research. I wrote the attached test program to save and load a simulated 11x500,000 recarray. Average save and load times are given below (timeit with 20 repetitions). The save time for sqlite is not really fair because I have to delete the data table each time before I create the new one. It is still pretty slow in comparison. Loading the recarray from sqlite is significantly slower than pytables or cPickle. I am hoping there may be more efficient ways to save and load recarray’s from/to sqlite than what I am now doing. Note that I infer the variable names and types from the data rather than specifying them manually.

I’d luv to hear from people using sqlite, pytables, and cPickle about their experiences.

saving recarray with cPickle:       1.448568 sec/pass
saving recarray with pytable:      3.437228 sec/pass
saving recarray with sqlite:         193.286204 sec/pass

loading recarray using cPickle:    0.471365 sec/pass
loading recarray with pytable:     0.692838 sec/pass
loading recarray with sqlite:        15.977018 sec/pass

Best,

Vincent