XML document causes pickle to go into infinite recursion
orest.kozyar at gmail.com
Thu Nov 1 22:25:22 CET 2007
> > I'm working on a CGI script that pulls XML data from a public database
> Ah, I missed that bit on first read. Consider using something different than
> CGI here if you want to do caching. FCGI would allow you to do in-memory
> caching, for example, as would mod_python and a lot of other solutions.
What I'm aiming for is sort of a "permanent" disk cache/mirror that
adds records as needed. The main issue is that the database (PubMed)
requests that we limit requests to once every three seconds. I often
need to access data for hundreds of records, so I've set up a cron job
to cache the needed records to disk overnight. I was using shelve
before, but the size of the file grew to about 500 mB and I started
having issues with shelve performance. So, I just stopped using
shelve and am storing each record separately on disk now.
More information about the Python-list