On Wed, 29 Oct 2003, phil@bubblehouse.org wrote:
well, my only use of the db object directly is for actually updating the table. my particular implementation would have far more views than updates, so i was going to cache the common data, or possibly pre-render html...
At the point where you're caching stuff yourself, it should be obvious that you're writing a database...so why not admit it to yourself and just finish writing it? :) Just cache everything in-memory, and write to files in a fail-safe way (UNIX, luckily, has enough useful primitives for that like writing to a tempfile, then renaming when you're done). This will let you use the operating system's mechanisms for LRU, which is probably more or less the right thing to do in a web application (LRU anticipates stuff like slashdotting, for example :) Twisted even has stuff to do such filesystem-transactions for you: twisted.persisted.dirdbm and twisted.python.filepath.FilePath.setContent
as nice as it would be to live in a world where pickle was a common data format, we're not there yet...
Pickle is just as much a common file format as MySQL's internal format. That is to say, there's just one way to access it, but you can program object viewers easily. In any case, there's no need to write pickles to the files. You can, for example, have each row be a list of netstrings in a file -- Twisted also has code to decode and encode netstrings.