well, my only use of the db object directly is for actually updating the table. my particular implementation would have far more views than updates, so i was going to cache the common data, or possibly pre-render html... wouldn't you say that these performance issues are as much of a consequence of poor design as much as the choice to use SQL? obviously db calls are expensive...but some of the benefits of a portable datasource are worthwhile. as nice as it would be to live in a world where pickle was a common data format, we're not there yet...

Date: 29 Oct 2003 06:28:17 -0000 From: Moshe Zadka <m@zadka.site.co.il> Subject: Re: [Twisted-Python] twisted.web and MySQLdb To: twisted-python@twistedmatrix.com Reply-To: twisted-python@twistedmatrix.com

I won't reply to your question here, I'm sure someone more competent than me can do so. I will take a minute to note that for some reason, the instinct of web developers has been to immediately write applications in the LAMP model. While twisted.web is a perfectly good substitute for A, and can indeed be used this way, it is much more capable.

LAMP -- Linux Apache MySQL PHP [each of those has variants]

Linux (and most other things in the "L" place, such as FreeBSD) has an incredibly fast fork() operation. This leads people to write their web code with the process-per-page model. Apache optimizes this by preforking, making such code actually scalable enough to deliver good response times. However, suddenly there is a problem: since different processes are likely to deal with the same user, memory becomes fragile. It is unsafe to put things in memory: processes switch users, get born and die too often. In Linux, and most other Unices, file locking has been a thorny issue. So instead of putting data in files, where the subtle semantics of locking and concurrency should be dealt with, data tends to end up in the database. That means *all* data. Sessions. Temporary "show this message when the user finished his redirect loop". Etc. etc. Once all the data is in MySQL, which is the easiest database to set up, it becomes natural for the average page to be a "glorified select". PHP was designed for this exact scenario: take a select table and spruce it up with HTML.

Unfortunately, what LAMP deals to is exactly to this: inevitably, all pages are just spruced up select-tables. This makes programming somewhat unnatural unless you're programming a database viewer.

With twisted.web, you don't have these problems. It is easy to keep data in files, because locking is not an issue. It is easy to cache data in memory because everything is served from a single process. If you need to attach state to a user's session, you can just keep an object in the session. This means programming twisted.web should be a lot more like writing a GUI application, and a lot less like writing select-with-HTML. You should probably reconsider whether you really want MySQL. It adds complexity to your application, and the gain is usually small. Putting persistent data in files, and using liberal caching schemes, also plays on the core advantage of Linux (and similar) -- it uses the incredibly optimized caching algorithms.