Thanks for the update!
On Oct 9, 2015, at 1:06 PM, Clayton Daley wrote:
FWIW, (sorry, non-twisted!) if you want to avoid mongodb:
1. We use redis for KV caching, and its really, really, really fast.
2. Riak can use LevelDB as a backing store -- leveldb supports snappy compression on documents
3. If you need the document search, i really like Elastic
I don't really like/trust mongodb from past experience myself. I would normally not use it for anything other than a read-through cache... except for my current need. We have a web spider and need to archive the snapshots. WireTiger, which was bought by mongodb and integrated into the 3.x branch, is the only storage backend I've found that will do "bucket level compression". I forget if it's applied on pages or blocks, but it will basically apply zlib or snappy compression onto sets of documents. I looked for weeks on something that could do that, it was the only option. We did a lot of storage tests on our typical data, and bucketed compression saves way more space than document compression or document compression using a seed.