Persistence design [was: RE: OT: why are LAMP sites slow?]

[Robert Brewer]

Jack Diederich wrote:
> *ding*ding*ding*  The biggest mistake I've made most 
> frequently is using
> a database in applications.  YAGNI.  Using a database at all has it's
> own overhead.  Using a database badly is deadly.  Most sites would
> benefit from ripping out the database and doing something simpler.
> Refactoring a database on a live system is a giant pain in the ass,
> simpler file-based approaches make incremental updates easier.
> 
> The Wikipedia example has been thrown around, I haven't looked at the
> code either;  except for search why would they need a database to
> look up an individual WikiWord?  Going to the database 
> requires reading
> an index when pickle.load(open('words/W/WikiWord')) would 
> seem sufficient.
> 
...
> 
> If there is interest I'll follow up with some details on my own LAMP
> software which does live reports on gigs of data and - you 
> guessed it -
> I regret it is database backed.  That story also involves why 
> I started
> using Python (the prototype was in PHP).

I'd be interested, if only selfishly to hear more potential use cases
for *my* projects. ;)

One of my goals for Dejavu* (my ORM) is to abstract persistence to the
point that you can easily test your actual, live dataset against many
potential storage mechanisms (i.e. multiple DB's, but also shelve,
etc.). I need to finish the migration tools, but it's well on the way.


Robert Brewer
MIS
Amor Ministries
fumanchu at amor.org

* http://www.aminus.org/rbre/python