Shelve operations are very slow and create huge files

Ole Moller Nielsen ole at
Mon Nov 3 04:29:31 CET 2003

	Hi Everyone
	Yes we used caching of database queries extensively in several projects.
	Our datamining pages are available at 
	and the dm software and at
	Good luck
	On Sun, Nov 02, 2003 at 06:36:00PM +1100, Tim Churches wrote:
> On Sun, 2003-11-02 at 03:38, Eric Wichterich wrote:
> > One script searches the MySQL-database and stores the result, the next 
> > script reads the shelve again and processes the result. But there is a 
> > problem: if the second script is called too early, the error "(11, 
> > 'Resource temporarily unavailable') " occurs.
> The only reason to use shelves is if your query results are too large
> (in total) to fit in memory, and thus have to be retrieved, stored and
> processed row-by-row. 
> > So I took a closer look at the file that is generated by the shelf: The 
> > result-list from MySQL-Query contains 14.600 rows with 7 columns. But, 
> > the saved file is over 3 MB large and contains over 230.000 lines (!), 
> > which seems way too much!
> But that doesn't seem to be the case - your query results can easily fit
> in memory. However, the query may still take a long time to execute, so
> it may be reasonable to want to store or cache the results for further
> processing later. However, it is much quicker to just pickle (cPickle)
> the results to a gzipped file than to use shelve. The use of gzip
> actually speeds things up, provided that your CPU is reasonably fast and
> your disc storage system is mundane (any CPU faster than about 500 Mhz
> sees gains on most result sets). Also it saves disc space.
> Ole Nielsen and Peter Christen have written a neat set of Python
> functions which will automatically handle the caching of query results
> from an MySQL datadase in gzipped pickles - see
> - except the files don't
> seem to be available from that page - Ole and Peter, please fix!
> -- 
> Tim C
> PGP/GnuPG Key 1024D/EAF993D0 available from keyservers everywhere
> or at
> Key fingerprint = 8C22 BF76 33BA B3B5 1D5B  EB37 7891 46A9 EAF9 93D0

Ole Nielsen                     | Email: Ole.Nielsen at
Mathematical Sciences Institute | Phone: +61 2 6125 3873 (Direct)
Australian National University  | Fax:   +61 2 6125 5549
Canberra ACT 0200               | 
Australia                       | 

More information about the Python-list mailing list