On Oct 28, 2003, at 6:10 PM, Nathan Seven wrote:

--- phil@bubblehouse.org wrote:

...

I'm using MySQLdb to interface with my sql server, but i'm not really sure what to do with my connection object once it's created. i want to put it somewhere that will be easy to access for the rest of the modules, but obviously i don't want it to be persisted when the TAP is saved....

.

Hmmm- well I'm just getting started here as well- but yeah, I don't remember any mentioned way of selectively removing state information for objects. I don't like the whole state-saving stuff at all- I'm doing atomic DB stuff, so my "state" above all is maintained on the DB itself. The Twisted clients are just basically dumb transaction handlers.

This is the way you should be doing it -- the only state in the tap should be configuration information about what services live on what ports, what directory the web server looks at, etc.

...

but i don't know how to make it so the connection object isn't persisted, but created new each time the server is started...

Hmmm- would a simple solution be to simply re-initialize the connection(s) upon any restart? (can a restart even be detected with .app?)

Since the tap is just a pickle, the easiest way to prevent certain objects from getting persisted is to implement a __getstate__ and __setstate__ for your class. See the python documentation for information. dp

On Oct 28, 2003, at 6:10 PM, Nathan Seven wrote:

--- phil@bubblehouse.org wrote:

...

I'm using MySQLdb to interface with my sql server, but i'm not really sure what to do with my connection object once it's created. i want to put it somewhere that will be easy to access for the rest of the modules, but obviously i don't want it to be persisted when the TAP is saved....

.

Hmmm- well I'm just getting started here as well- but yeah, I don't remember any mentioned way of selectively removing state information for objects. I don't like the whole state-saving stuff at all- I'm doing atomic DB stuff, so my "state" above all is maintained on the DB itself. The Twisted clients are just basically dumb transaction handlers.

You should just not use the whole pickled app/saving state thing. Then you don't have to worry about this stuff at all. In addition the next person who comes along and wants to debug something will thank you profusely when he doesn't have to sit there and ponder "gee, I wonder what parameters the last person gave to mktap, cause I really would like to startup a second copy of that app on another port. (or from a different directory, with a new file type handler, whatever)...". I recommend configuring your webserver with python code and starting it with twistd -y. Perhaps not ideal but certainly better than the whole tap mess. All IMO, of course. James

On Oct 28, 2003, at 10:06 PM, James Y Knight wrote:

You should just not use the whole pickled app/saving state thing. Then you don't have to worry about this stuff at all. In addition the next person who comes along and wants to debug something will thank you profusely when he doesn't have to sit there and ponder "gee, I wonder what parameters the last person gave to mktap, cause I really would like to startup a second copy of that app on another port. (or from a different directory, with a new file type handler, whatever)...".

I recommend configuring your webserver with python code and starting it with twistd -y. Perhaps not ideal but certainly better than the whole tap mess.

Now that freeform is done in quotient and passing existing object references to formless methods is next up on the plate (and the implementation is far cleaner than the old quotient formless/configure/webform) we can get started on writing COIL sometime in the near future so you can configure pickled application instances in the tap using a nice web ui. Having a good configuration interface which allows you to load the serialized application, reconfigure it, and serialize it back out to disk has always been an integral part of the tap idea that has never actually been implemented properly yet. The "I wonder what parameters were used" idea is pretty interesting, though. It would be trivial to serialize the command-line options that were given to each invocation of mktap along with everything else so at the very least we could show you which command line options were used to build the application in the first place. dp

On Tue, Oct 28, 2003 at 11:34:37PM -0800, Nathan Seven wrote:

Speaking of- I know we're kinda tied to Python, but what are the thoughts about using any of the OS's provided event-driven polls? (epoll,kqueue) I'm guessing these are essentially untouchable without Python mods.

There's a kqueue reactor already -- see twisted/internet/kqreactor.py (yes, it does rely up on an extension module). I'm not sure how complete it is, it's not tested by the buildbot. epoll shouldn't be too hard to do with pretty much the same approach. Patches accepted :) -Andrew.

Eric C. Newton wrote:

I looked into this for a few hours. I had to build a server that needed to regularly maintain 20K+ connections. Of course, I prototyped it in python first.

It takes a long time just to create and accept 20K+ connections in Python. Epoll wasn't going to help that much.

Can you qualify "long time"? And what's the bottleneck? At the scale of 100K or so function calls, Python isn't egregiously slow, certainly not slow enough to not be able to outrun TCP connection setup...?

Nathan Seven wrote:

Hmmmm I would qualify that- I dont think the filesystem is the place to be handling dynamic data.

The filesystem is a fine place. For example, in the Prevayler persistence model, you just write logfiles to disk, and synchronize your state at a checkpoint. For highly dynamic applications, especially ones which require failover, (You can re-play the transaction log live, after all) this works quite well.

Databases were created *specifically* for this purpose.

I think that databases were specifically designed to store accounting information, actually.

Sure, storing all your static blobs in your database is a really quick way to grind shit to a halt, but locking and concurrency? If you're doing things properly, and your http server is just serving static objects, then these are non-issues.

Databases can be amazingly slow, especially if you have a lot of updates to do. (Even a very fast database can be made slow by I/O bottlenecks if you are trying to make it remote for scalability reasons.) This has an easy solution: you can cache everything! Of course, then you need to be able to easily access the cache from all of the machines, because it may have been updated. Now you have problems with coherency. Then you need to lock the cache, because it could have been updated, and then you need to read from it. Pretty soon you're talking to your caching server as if it were a database. This is _great_ if you are Livejournal:

Yeah- through my line of work I deal with a *lot* of different infrastructures. Everything from "Joe's BBQ Sauce Garage" to Amazon. Literally the only organization I can think of that can keep anything coherent with MySQL is Livejournal- and then I believe only because Brad seems to be a cache-god with memcached and such.

because then you don't have to worry about computation, mutable data, etc - you're basically just storing data and then spitting it back out, and you don't care if the timestamps are a little off. This is the important point about LAMP and Twisted: There are applications which can connect to HTTP which are not blogs. If you are writing a multiplayer game which wants to support lots of concurrent users, you can't afford to spawn a thread and do a database request every time a player picks something up. (Python is quite slow enough already, thanks.) You can't just use a cache because the data changes _all the time_, and you have to care about it from everywhere that you care about your data. Working with your objects directly in memory is close to the only option. If you're writing a real-time financial data system, you do want to use a database, but you want to very carefully control your access to it. Certainly, you don't want to equate 'web hit' with 'database query', as the LAMP model is wont to do. Or maybe you're writing an application that has to operate as a client-side proxy, and you don't have the leisure of a DBA at every desk, so you can't require that an RDBMS gets set up with each installation. This might require some hackish workarounds with the filesystem that you'd rather not do, but nevertheless, it's better than having the user editing pg_hba.conf themselves.

Hello, adding more random thoughts below on LAMP sucking, and spitting. Glyph Lefkowitz wrote:

Nathan Seven wrote:

...

Hmmmm I would qualify that- I dont think the filesystem is the place to be handling dynamic data.

The filesystem is a fine place.

For example, in the Prevayler persistence model, you just write logfiles to disk, and synchronize your state at a checkpoint. For highly dynamic applications, especially ones which require failover, (You can re-play the transaction log live, after all) this works quite well.

...

Databases were created *specifically* for this purpose.

I think that databases were specifically designed to store accounting information, actually.

...

Sure, storing all your static blobs in your database is a really quick way to grind shit to a halt, but locking and concurrency? If you're doing things properly, and your http server is just serving static objects, then these are non-issues.

Databases can be amazingly slow, especially if you have a lot of updates to do. (Even a very fast database can be made slow by I/O bottlenecks if you are trying to make it remote for scalability reasons.) This has an easy solution: you can cache everything! Of course, then you need to be able to easily access the cache from all of the machines, because it may have been updated. Now you have problems with coherency. Then you need to lock the cache, because it could have been updated, and then you need to read from it.

Pretty soon you're talking to your caching server as if it were a database. This is _great_ if you are Livejournal:

...

Yeah- through my line of work I deal with a *lot* of different infrastructures. Everything from "Joe's BBQ Sauce Garage" to Amazon. Literally the only organization I can think of that can keep anything coherent with MySQL is Livejournal- and then I believe only because Brad seems to be a cache-god with memcached and such.

because then you don't have to worry about computation, mutable data, etc - you're basically just storing data and then spitting it back out, and you don't care if the timestamps are a little off.

This is the important point about LAMP and Twisted:

There are applications which can connect to HTTP which are not blogs.

If you are writing a multiplayer game which wants to support lots of concurrent users, you can't afford to spawn a thread and do a database request every time a player picks something up.

New versions of linux have very quick threads. Also different apps have different speeds for threads. Small programs (say less than 100 kilobytes) which are entirely static are lots faster than multi megabyte processes which dynamically load things. Threads and processes also can easily use multiple cpus. Of course there are other reasons people don't like threads. http for games? If your game is sensitive to latency and you can help it, avoid http for games. Http gives you much more latency than a db. So does a centralised server for that matter. In a two player game you can half your latency by talking directly to each other. Assuming you both aren't behind a non configurable firewall, or a proxy server. http is good as a backup protocol though. Because some fascists don't allow anything but access to the internet except through a proxy. Or maybe you are playing a game in a web browser :) Note that some dbs have async interfaces(eg postgres). So you wouldn't need a thread. btw, anyone know if sendfile is in (or going to be in python soon)? http://mail.python.org/pipermail/python-dev/2002-March/021498.html

(Python is quite slow enough already, thanks.) You can't just use a cache because the data changes _all the time_, and you have to care about it from everywhere that you care about your data. Working with your objects directly in memory is close to the only option.

What about berkley dbs? I think bsddb3 was 3-4 times slower than a python dict in general. They are very quick, and you don't need to lock them. You can use transactions. Locking can kill performance. Quickness depends on memory, and data sets really. These are approximate speedinesses of python dictionary like things for different data sizes and memory: say 700MB of memory: 200 MB key value data - python dicts, kjDicts, berklydb3 on disk db. 400 MB key value data - kjDicts(as python dicts use more memory and begin swapping), berkley db3-4 on disk, python dicts.

2 gigs of key value data - berkley db3-4 on disk, kjDicts(the kjdict starts swapping before here), python dicts.

Of course deleting large python dicts is *really* slow. I used to kill my python processes with kill -9 so that I didn't have to wait for the reference counting garbage collecting beast to do lots of free()s on the dicts memory. kjDicts and bsddb dbs were faster for deleting. Some people do use RDBMS for large online games. Check out gamasutra.com for some articles. Seems they have lots of fun performance problems. Compressing the hash would be nice. Maybe memory mapped files on a compressed file system would be quick for this ;) Distributed hashes would also be nice! Any good distruted hashes for python? I think memory is one of the first things that kill apache performance(when using preforking). Especially when it has massive php compiled in! You can get big speed boosts by using different apache configs for different request types(even on the same machine). Eg set one up for static files(eg images), and one for your bloated php.

If you're writing a real-time financial data system, you do want to use a database, but you want to very carefully control your access to it. Certainly, you don't want to equate 'web hit' with 'database query', as the LAMP model is wont to do.

Or maybe you're writing an application that has to operate as a client-side proxy, and you don't have the leisure of a DBA at every desk, so you can't require that an RDBMS gets set up with each installation. This might require some hackish workarounds with the filesystem that you'd rather not do, but nevertheless, it's better than having the user editing pg_hba.conf themselves.

SQLlite and berkley db are good for easy to bundle dbs. One good thing about LAMP though is that lots of servers have it installed, and it can be quite cheap to use as a platform. As you say, not everything is a blog. There are too many different factors for a one size fits all 'this way is best' solution. Besides everyone knows twisted rules ;) Have fun!

--- Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:

Nathan Seven wrote:

...

Databases were created *specifically* for this purpose.

I think that databases were specifically designed to store accounting information, actually.

Exactly, handling and tracking dynamic data :)

If you are writing a multiplayer game which wants to support lots of concurrent users, you can't afford to spawn a thread and do a database request every time a player picks something up. (Python is quite slow enough already, thanks.) You can't just use a cache because the data changes _all the time_, and you have to care about it from everywhere that you care about your data. Working with your objects directly in memory is close to the only option.

I dunno- I guess I just disagree- a multiplayer game is a perfect example of something that absolutely requires atomic transactions. "Buy an item->add item to inventory->deduct $ from player" etc. That transaction either needs to totally complete, or not complete at all. So now you need to build some sort of transaction manager on top of your in-memory objects. Oh and then you probably want some way to ensure that these in-memory objects dont get lost in the case of abnormal shutdown- right? Well- haven't you just built a couple features of any (real) database?

If you're writing a real-time financial data system, you do want to use a database, but you want to very carefully control your access to it. Certainly, you don't want to equate 'web hit' with 'database query', as the LAMP model is wont to do.

Yes hahah- I think my overall point is just that the seperation is key.

Or maybe you're writing an application that has to operate as a client-side proxy, and you don't have the leisure of a DBA at every desk, so you can't require that an RDBMS gets set up with each installation. This might require some hackish workarounds with the filesystem that you'd rather not do, but nevertheless, it's better than having the user editing pg_hba.conf themselves.

Sure- though that specifically is the reason for embeddable databases. ===== -- live- http://www.thedenofsin.org/ to- AIM: IMFDUP _born to violate._

Moshe Zadka wrote:

I won't reply to your question here, I'm sure someone more competent than me can do so. I will take a minute to note that for some reason, the instinct of web developers has been to immediately write applications in the LAMP model. While twisted.web is a perfectly good substitute for A, and can indeed be used this way, it is much more capable.

LAMP -- Linux Apache MySQL PHP [each of those has variants]

Linux (and most other things in the "L" place, such as FreeBSD) has an incredibly fast fork() operation. This leads people to write their web code with the process-per-page model. Apache optimizes this by preforking, making such code actually scalable enough to deliver good response times. However, suddenly there is a problem: since different processes are likely to deal with the same user, memory becomes fragile. It is unsafe to put things in memory: processes switch users, get born and die too often. In Linux, and most other Unices, file locking has been a thorny issue. So instead of putting data in files, where the subtle semantics of locking and concurrency should be dealt with, data tends to end up in the database. That means *all* data. Sessions. Temporary "show this message when the user finished his redirect loop". Etc. etc. Once all the data is in MySQL, which is the easiest database to set up, it becomes natural for the average page to be a "glorified select". PHP was designed for this exact scenario: take a select table and spruce it up with HTML.

Unfortunately, what LAMP deals to is exactly to this: inevitably, all pages are just spruced up select-tables. This makes programming somewhat unnatural unless you're programming a database viewer.

With twisted.web, you don't have these problems. It is easy to keep data in files, because locking is not an issue. It is easy to cache data in memory because everything is served from a single process. If you need to attach state to a user's session, you can just keep an object in the session. This means programming twisted.web should be a lot more like writing a GUI application, and a lot less like writing select-with-HTML. You should probably reconsider whether you really want MySQL. It adds complexity to your application, and the gain is usually small. Putting persistent data in files, and using liberal caching schemes, also plays on the core advantage of Linux (and similar) -- it uses the incredibly optimized caching algorithms.

RANDOM NOTES ON THIS TOPIC: Single process has problems too: - if something blocks or stalls down comes your whole app. This can happen in multi proccess too of course. - you don't use the operating systems privileges. You invent your own which is often broken and needs to be rewritten a few times. Of course both of these can happen with your LAMP too. If the db goes down or whatever. But I guess multiprocess is more resiliant. Some people find select statements easier. Often you can do something in a line or two which is cumbersum in python. Berkley db databases are nice to use. They are quite optimized, and can emulate python dictionaries quite nicely. Then there is zopedb which is an option. Another feature databases have over files is transactions(comming soon to a FS near you!). Making sure stuff is written to files reliably is quite difficult. Databases are accessable from database viewers! This is handy since random python object viewers are not so good :) You can access databases easily from other languages, whereas python objects are harder to do. I guess jelly/xmlrpc/soap helps here if you want to use those. Databases also handle large volumes of data efficiently(or try to). Keeping everything in files is fine as long as you lay it out nicely and you have plenty of time to optimize each new query. However the advantages/disadvantages of dbs are discussed to death all over the place, and I have nothing new to add. Be nice to be able to use python objects as memory mapped files. I guess it may be possible. It has been done with numeric arrays, and with c++ objects. This can be faster than reading files in linux too. A lot of web pages can be cached into static files which improves performance by a lot. Chucking squid, or apache as proxy in front of your normal webserver(s) can be a large help if you are careful. Have fun!