[shelve] What are the limitations? Entering too many data crashes it on my machine!

[Josh]

Jeff Epler <jepler at unpythonic.net> wrote in
news:mailman.1040612250.17232.python-list at python.org: 

> I use "shelve" in an irc bot.  There are well over 100k keys (possibly
> more like 1Mkeys) in the shelf, which is 1.7G large on disk.  At peak
> usage, the bot probably makes up to 100 inserts or 400 queries per
> minute, and has been running nearly continuously (with restarts) for
> over a year. No problems (with the exception of running out of
> diskspace).  However, I'm using (according to file(1)) "GNU dbm 1.x or
> ndbm database, little endian".  Oh, and the Python version is almost
> certainly 1.5.2. 
> 
> shelve itself is very unlikely to be where the problem lies.  The
> individual database library or Python database module may be, however.
> 
> Jeff
> 
> 

 I had the same problem when I was using shelve. I was shelving some huge 
CSV files (>20MB) and then was using the shelved data for some sequence 
matching operations. It had about 100,000 keys and the program would work 
flawlessly until say the 10,000th record and then crash with the same 
traceback that Geiger got or an Invalid key traceback. What worked for me 
was using a separate file for the output, meaning I used say shelve_infile 
for the input, matched the sequences and wrote the output out to 
shelve_outfile instead of making changes to shelve_infile. I don't know why 
I couldn't use just one shelved file to read, make changes and write 
changes, but I figured that was probably a bug. Hope that helped.