On 03:08 am, lozinski@freerecruiting.com wrote:
I am just choosing a web server, and I thought it would be good to consult with this mailing list, since twisted web was on my short list.
I run a python moinmoin wiki, a static site, a zope 2 site, and a zope 3 site on a VPS.
The thing I care about most, is ease of use. None of my websites are very high traffic. And I have way way too much complexity in my life to manage.
The next thing I care about is reliability.
Twisted Web is rather reliable. http://twistedmatrix.com/ has run on Twisted Web for many years now. From time to time an issue arises, but my experience is that it requires less attention than an Apache deployment. This is just my experience, of course.
The thing I care least about is performance. Would that I had that problem.
That is an excellent attitude! I'll just note that Twisted Web is fast enough for just about anything you might want to do, though. More on this later.
So Apache is the obvious first choice. But it is so complicated. They threw in everything but the kitchen sink. I just do not like it, nor do I want to spend the time wrestling with it to get it to do the needful. And you have to restart apache everytime the config files change.
You'll almost certainly have to restart your Twisted Web server every time you want to change its configuration. Nothing stops you from building a server using Twisted Web for which this is not the case, but it's a lot of careful, detail-oriented work for not a lot of gain.
[snip - lighttpd etc]
TwistedWeb is very interesting. When I read the Apache documentation, my eyes glaze over. When I read the twisted documentation, my eyes light up. What an intelligent object model of networking. Twisted Web is also closely related to Zope. At the very least, it is also using Zope Component Architecture, which I like.
Minor correction, Twisted Web uses Zope Interface (as does the rest of Twisted) and some of the same ideas as ZCA, it doesn't actually use ZCA itself.
So here is my comparison of twisted web and nginx.
With Twisted web, first I wonder about configuration. Nginx is just some configuration files. Perhaps easier then editing a python program.
As I think you've learned, Twisted Web doesn't come with a set of standard configuration files. It is configurable using Python. It's up to you to build a maintainable configuration system out of this base. http://twistedmatrix.com/ uses something very descriptively named "twisted-web" for configuration. This is a vaguely (*very* vaguely) apache-style configuration system, but it's still built out of Python and it's also not currently maintained as far as I know. Fortunately configuration of most features of Twisted Web only takes a couple lines of Python at most. I find this doesn't get too bad. Also, I tend to follow the rule that when configuration does get longer than a few lines, it is time to turn it into *software* with documentation and unit tests, and then *use* it from the configuration.
Then I wonder about reliability. Nginx has a larger installed base, more bug reports presumably more reliable. Plus it includes a process monitor that keeps things running. I think twisted web includes monitoring software, but perhaps not as robust and mature.
Twisted Web doesn't include any monitoring software, although Twisted Runner does include "procmon" which launches and re-launches a program as necessary, so perhaps this is what you're referring to here. There are also lots of other projects that do what "procmon" does (as well or better) and are entirely compatible with Twisted Web, so there's no shortage of options here. Returning to http://twistedmatrix.com/, the Twisted Web server deployed there does *not* run as a managed process controlled by one of these types of tools. It just keeps running and never needs restarting. What kind of reliability concerns do you have apart from the process aborting, segfaulting, or otherwise crashing due to implementation bugs?
The next nice thing about Nginx is that you can drop configuration files and the server recognizes them, and does the right thing. No need to reboot. The twisted web python program I would have to restart.
Yep.
Finally I worry about the performance. You can see a bunch of interesting results for python servers here. http://nichol.as/benchmark-of-python-web-servers
In general twisted web does not seem that fast, nor reliable. High error rates. Bummer.
Please disregard those results. :( First, all of the servers he benchmarked performed remarkably well. 500 requests/second translates to 2ms/request of overhead. Perhaps your moinmoin requests will take 5ms and your zope requests will take 15ms (I just made those number up, I have no idea how moinmoin or zope perform). So your per-request cost is now 7ms and 17ms - that's 142 requests/second and 58 requests/second respectively. I suspect that's fast enough for you, but the real point is to compare this to an ideal server with 0ms/request of overhead. On such a server, you would get 200 requests/second on moinmoin and 66 requests/second on zope. Is the difference between 142 and 200, or 58 and 66 significant for you? Second, no attempt has been made to explore or explain what "errors" mean in the survey. That said, my suspicion is that they're caused by high load on the platform TCP/IP stack and would not typically be observed in real world usage where clients *don't* run on the same host as the server. Regardless, as there's no discussion on the write-up and in real-world use I haven't observed problems like this, I have assumed the errors are spurious and due to the test setup (and await a correction from anyone who can demonstrate otherwise). Third, the benchmark use the WSGI system of Twisted Web. Perhaps WSGI is the only feature you care about in a web server (if so, then you don't need to think too hard about picking a WSGI container, since they are extremely interchangeable - if the one you pick has a problem, switch to another, it's the work of 15 minutes or so at worst). However, no attempt has been made to configure the system for maximum performance (the default threadpool size is used - a value tuned for DNS requests, not WSGI threads) and the benchmark comparison puts this up against several "fake" WSGI implementations ("lightweight threads" and "callback/generator" flavors are not real WSGI containers). If you wanted to push performance higher, there's every reason to expect you could. And if you don't use WSGI, then performance automatically jumps dramatically.
What is my conclusion? Well I think Twisted web is a brilliant object model for networking software. But dedicated software for web serving, like nginx performs way better and is much more reliable.
If you go into more detail about what you mean by "reliable" here, someone may be able to speak to it. As I mentioned above, the deployments of Twisted Web I've managed are quite reliable, they just keep working year after year with minimal maintenance.
So I think I am going to use Nginx in front of my moinmoin, zope 2, zope 3 and static servers. Is that the right decision? Pity, I really like twisted web.
Anyhow I thought I would post this here, and see what people have to say. Presumably I made some mistakes in this document. Please let me know what they are.
Thanks for writing this up and sending it to the list. I hope some of my responses have been useful or at least interesting. Good luck in whatever you choose to deploy! Jean-Paul