[DB-SIG] Pickle and sqlite - loading and saving recarray's

[Vincent Nijs]

I posted the message below on the numpy discussion list. Few if any people
there seem to use sqlite.

I am interesting in using sqlite to store data for scientific research. I
wrote the attached test program to save and load a simulated 11x500,000
recarray. Average save and load times are given below (timeit with 20
repetitions). The save time for sqlite is not really fair because I have to
delete the data table each time before I create the new one. It is still
pretty slow in comparison. Loading the recarray from sqlite is significantly
slower than cPickle. I am hoping there may be more efficient ways to save
and load recarray¹s from/to sqlite than what I am now doing. Note that I
infer the variable names and types from the data rather than specifying them
manually.

saving recarray with cPickle:       1.448568 sec/pass
saving recarray with sqlite:         193.286204 sec/pass

loading recarray using cPickle:    0.471365 sec/pass
loading recarray with sqlite:        15.977018 sec/pass

To run the test you do need numpy (http://www.scipy.org/Download)

Best,

Vincent
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/db-sig/attachments/20070720/a6e8e33b/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: load_sqlite_test.py
Type: application/octet-stream
Size: 3391 bytes
Desc: not available
Url : http://mail.python.org/pipermail/db-sig/attachments/20070720/a6e8e33b/attachment.obj