[Pythonmac-SIG] persistence speeds compared

Christopher Barker Chris.Barker at noaa.gov
Tue Jul 17 07:08:13 CEST 2007


David Worrall wrote:
> I'm working w. a dynamic dataset of some 3500 tables - each table  
> grows sequentially.  Total data ~= 5GB

Sorry I've no comparisons for you, but what kind of data is it?

If it's a lot of numbers and short strings, PyTables+ numpy could be 
very fast.

MySQL has a reputation for speed, but interestingly, we found we got 
better performance from Durus (A pure python object persistence system) 
for a read only dataset, though not as big as yours.

Beware the pitfalls of benchmarking -- I think you're probably going to 
need to make a prototype that does the kinds of things you need to do, 
and test a few systems yourself.

good luck,

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception


More information about the Pythonmac-SIG mailing list