SQL/Python question -- slow... What is the fixed cost?

[Mark Berry]

>	What I am wondering is what the fixed cost for this configuration:
>(python + MySQLdb + MySQL). I am about to design a sophisticated caching
>system, and before I do that, I was hoping someone with some experience
>with this setup would be able to tell me if it is worth my time... Any
>help would be appreciated. I can also share code if it helps.


I would first ask yourself how big is the system going to get.  In my
personal experience I have had much better luck with scaling Postgres
on LARGE tables.  MySQL is a very quick database but under heavy load
it was a bit weaker than PGSQL.  Now, mysql is being improved on a
daily basis and maybe these problems have been resolved, but I have
been able to do things like write stored procedures in C on postgres
and resolve complex queries like the ones you have described and more.

Just becuase a query take .001 seconds, doesn't mean that it will
perform that well with 100k records.

I would (for testing) do the following.  Imagine how big that database
COULD get with your application in a reasonable amount of time. Times
that by 2 and add 10% on top of it in test data into the database
tables.  (you can write a script to do this...)  Then run your tests.

Thats probably the biggest mistake I have made in the past while
normalizing the database, didn't really test it enough under stress
and size.

Another database that would be worth looking at is SAPDB.  It's free
(GPL) and is VERY feature rich (more so than even postgres), though I
am not sure if there are python database drivers.


Good luck.

.mark