I am just choosing a web server, and I thought it would be good to consult with this mailing list, since twisted web was on my short list. I run a python moinmoin wiki, a static site, a zope 2 site, and a zope 3 site on a VPS. The thing I care about most, is ease of use. None of my websites are very high traffic. And I have way way too much complexity in my life to manage. The next thing I care about is reliability. The thing I care least about is performance. Would that I had that problem. So Apache is the obvious first choice. But it is so complicated. They threw in everything but the kitchen sink. I just do not like it, nor do I want to spend the time wrestling with it to get it to do the needful. And you have to restart apache everytime the config files change. Nginx is the next obvious choice. "Apache is like Microsoft Word, it has a million options but you only need six. Nginx does those six things, and it does five of them 50 times faster than Apache." Lightpd is an option, but perhaps it does not support wsgi. Maybe lightpd2 does. Cherokee showed up on my radar, it has a nice web admin GUI, but hard to configure with moinmoin. There are 20 or 30 other web servers, all of which I am ignoring. But one that I have looked at closely is twisted web. TwistedWeb is very interesting. When I read the Apache documentation, my eyes glaze over. When I read the twisted documentation, my eyes light up. What an intelligent object model of networking. Twisted Web is also closely related to Zope. At the very least, it is also using Zope Component Architecture, which I like. So here is my comparison of twisted web and nginx. With Twisted web, first I wonder about configuration. Nginx is just some configuration files. Perhaps easier then editing a python program. Then I wonder about reliability. Nginx has a larger installed base, more bug reports presumably more reliable. Plus it includes a process monitor that keeps things running. I think twisted web includes monitoring software, but perhaps not as robust and mature. The next nice thing about Nginx is that you can drop configuration files and the server recognizes them, and does the right thing. No need to reboot. The twisted web python program I would have to restart. Finally I worry about the performance. You can see a bunch of interesting results for python servers here. http://nichol.as/benchmark-of-python-web-servers In general twisted web does not seem that fast, nor reliable. High error rates. Bummer. What is my conclusion? Well I think Twisted web is a brilliant object model for networking software. But dedicated software for web serving, like nginx performs way better and is much more reliable. So I think I am going to use Nginx in front of my moinmoin, zope 2, zope 3 and static servers. Is that the right decision? Pity, I really like twisted web. Anyhow I thought I would post this here, and see what people have to say. Presumably I made some mistakes in this document. Please let me know what they are.

On 03:08 am, lozinski@freerecruiting.com wrote:

I am just choosing a web server, and I thought it would be good to consult with this mailing list, since twisted web was on my short list.

I run a python moinmoin wiki, a static site, a zope 2 site, and a zope 3 site on a VPS.

The thing I care about most, is ease of use. None of my websites are very high traffic. And I have way way too much complexity in my life to manage.

The next thing I care about is reliability.

Twisted Web is rather reliable. http://twistedmatrix.com/ has run on Twisted Web for many years now. From time to time an issue arises, but my experience is that it requires less attention than an Apache deployment. This is just my experience, of course.

The thing I care least about is performance. Would that I had that problem.

That is an excellent attitude! I'll just note that Twisted Web is fast enough for just about anything you might want to do, though. More on this later.

So Apache is the obvious first choice. But it is so complicated. They threw in everything but the kitchen sink. I just do not like it, nor do I want to spend the time wrestling with it to get it to do the needful. And you have to restart apache everytime the config files change.

You'll almost certainly have to restart your Twisted Web server every time you want to change its configuration. Nothing stops you from building a server using Twisted Web for which this is not the case, but it's a lot of careful, detail-oriented work for not a lot of gain.

[snip - lighttpd etc]

TwistedWeb is very interesting. When I read the Apache documentation, my eyes glaze over. When I read the twisted documentation, my eyes light up. What an intelligent object model of networking. Twisted Web is also closely related to Zope. At the very least, it is also using Zope Component Architecture, which I like.

Minor correction, Twisted Web uses Zope Interface (as does the rest of Twisted) and some of the same ideas as ZCA, it doesn't actually use ZCA itself.

So here is my comparison of twisted web and nginx.

With Twisted web, first I wonder about configuration. Nginx is just some configuration files. Perhaps easier then editing a python program.

As I think you've learned, Twisted Web doesn't come with a set of standard configuration files. It is configurable using Python. It's up to you to build a maintainable configuration system out of this base. http://twistedmatrix.com/ uses something very descriptively named "twisted-web" for configuration. This is a vaguely (*very* vaguely) apache-style configuration system, but it's still built out of Python and it's also not currently maintained as far as I know. Fortunately configuration of most features of Twisted Web only takes a couple lines of Python at most. I find this doesn't get too bad. Also, I tend to follow the rule that when configuration does get longer than a few lines, it is time to turn it into *software* with documentation and unit tests, and then *use* it from the configuration.

Then I wonder about reliability. Nginx has a larger installed base, more bug reports presumably more reliable. Plus it includes a process monitor that keeps things running. I think twisted web includes monitoring software, but perhaps not as robust and mature.

Twisted Web doesn't include any monitoring software, although Twisted Runner does include "procmon" which launches and re-launches a program as necessary, so perhaps this is what you're referring to here. There are also lots of other projects that do what "procmon" does (as well or better) and are entirely compatible with Twisted Web, so there's no shortage of options here. Returning to http://twistedmatrix.com/, the Twisted Web server deployed there does *not* run as a managed process controlled by one of these types of tools. It just keeps running and never needs restarting. What kind of reliability concerns do you have apart from the process aborting, segfaulting, or otherwise crashing due to implementation bugs?

The next nice thing about Nginx is that you can drop configuration files and the server recognizes them, and does the right thing. No need to reboot. The twisted web python program I would have to restart.

Yep.

Finally I worry about the performance. You can see a bunch of interesting results for python servers here. http://nichol.as/benchmark-of-python-web-servers

In general twisted web does not seem that fast, nor reliable. High error rates. Bummer.

Please disregard those results. :( First, all of the servers he benchmarked performed remarkably well. 500 requests/second translates to 2ms/request of overhead. Perhaps your moinmoin requests will take 5ms and your zope requests will take 15ms (I just made those number up, I have no idea how moinmoin or zope perform). So your per-request cost is now 7ms and 17ms - that's 142 requests/second and 58 requests/second respectively. I suspect that's fast enough for you, but the real point is to compare this to an ideal server with 0ms/request of overhead. On such a server, you would get 200 requests/second on moinmoin and 66 requests/second on zope. Is the difference between 142 and 200, or 58 and 66 significant for you? Second, no attempt has been made to explore or explain what "errors" mean in the survey. That said, my suspicion is that they're caused by high load on the platform TCP/IP stack and would not typically be observed in real world usage where clients *don't* run on the same host as the server. Regardless, as there's no discussion on the write-up and in real-world use I haven't observed problems like this, I have assumed the errors are spurious and due to the test setup (and await a correction from anyone who can demonstrate otherwise). Third, the benchmark use the WSGI system of Twisted Web. Perhaps WSGI is the only feature you care about in a web server (if so, then you don't need to think too hard about picking a WSGI container, since they are extremely interchangeable - if the one you pick has a problem, switch to another, it's the work of 15 minutes or so at worst). However, no attempt has been made to configure the system for maximum performance (the default threadpool size is used - a value tuned for DNS requests, not WSGI threads) and the benchmark comparison puts this up against several "fake" WSGI implementations ("lightweight threads" and "callback/generator" flavors are not real WSGI containers). If you wanted to push performance higher, there's every reason to expect you could. And if you don't use WSGI, then performance automatically jumps dramatically.

What is my conclusion? Well I think Twisted web is a brilliant object model for networking software. But dedicated software for web serving, like nginx performs way better and is much more reliable.

If you go into more detail about what you mean by "reliable" here, someone may be able to speak to it. As I mentioned above, the deployments of Twisted Web I've managed are quite reliable, they just keep working year after year with minimal maintenance.

So I think I am going to use Nginx in front of my moinmoin, zope 2, zope 3 and static servers. Is that the right decision? Pity, I really like twisted web.

Anyhow I thought I would post this here, and see what people have to say. Presumably I made some mistakes in this document. Please let me know what they are.

Thanks for writing this up and sending it to the list. I hope some of my responses have been useful or at least interesting. Good luck in whatever you choose to deploy! Jean-Paul

On 2 July 2013 12:26, <exarkun@twistedmatrix.com> wrote:

On 03:08 am, lozinski@freerecruiting.com wrote: <snip>

...

Then I wonder about reliability. Nginx has a larger installed base, more bug reports presumably more reliable. Plus it includes a process monitor that keeps things running. I think twisted web includes monitoring software, but perhaps not as robust and mature.

Twisted Web doesn't include any monitoring software, although Twisted Runner does include "procmon" which launches and re-launches a program as necessary, so perhaps this is what you're referring to here.

There are also lots of other projects that do what "procmon" does (as well or better) and are entirely compatible with Twisted Web, so there's no shortage of options here.

Worth mentioning that Twisted works very well with Systemd for process monitoring. And it supports Systemd socket activation too (unlike Nginx) * https://twistedmatrix.com/documents/current/core/howto/systemd.html * http://trac.nginx.org/nginx/ticket/237 This may also partly satisfy your configuration reload requirement. In so far as you can restart the Twisted process without dropping any new connections. Systemd keeps listening on port 80 queuing up new connections until Twisted starts up again. What I don't know is whether you can shutdown Twisted web gracefully ie wait for established connections to close. I also don't know if systemd would start a new twisted process until the old one had died. Another problem is that Twisted doesn't yet support SSL on adopted sockets. -RichardW.

On 02:25 pm, mithrandi@mithrandi.net wrote:

On Tue, Jul 2, 2013 at 3:25 PM, Richard Wall <m-lists@the-moon.net> wrote:

...

Another problem is that Twisted doesn't yet support SSL on adopted sockets.

Is there a bug report for this? I would have expected it to Just Work (using twisted.protocols.tls, that is).

It works, if you write the code to do it. I think what Richard meant is that there is no endpoint that says "add fd 17 to the reactor and start speaking TLS over it". This is perhaps blocked on someone inventing a way to compose endpoints, so that either the existing systemd endpoint and the existing tls endpoint could be combined to create this feature, or at least new endpoints could be written which could be combined to create this feature (rather than requiring a new endpoint for each possible combination). Jean-Paul

On 2 July 2013 15:41, <exarkun@twistedmatrix.com> wrote: <snip>

It works, if you write the code to do it. I think what Richard meant is that there is no endpoint that says "add fd 17 to the reactor and start speaking TLS over it".

Yes, I was also ignorant of TLS in Twisted. I assumed that it required a new IReactorSocket method to specifically return a t.internet.ssl.Port. I managed to get it working with the following code: * https://gist.github.com/wallrj/5911925 Much simpler than I expected.

This is perhaps blocked on someone inventing a way to compose endpoints, so that either the existing systemd endpoint and the existing tls endpoint could be combined to create this feature, or at least new endpoints could be written which could be combined to create this feature (rather than requiring a new endpoint for each possible combination).

There's just been a short discussion about it on #twisted.web habnabit wrote this: * https://github.com/habnabit/txsocksx/blob/parsley/txsocksx/ssl.py We'll create some tickets. -RichardW

On Jul 7, 2013, at 11:09 AM, zooko <zooko@zooko.com> wrote:

Oh, there are some potential security problems, too, with Twisted Web! In its default configuration it offers to use single-DES for encryption, which is a bad idea even though it isn't clear (to me) whether an attacker could take advantage of that.

http://twistedmatrix.com/trac/ticket/5514

It also has compression turned on, apparently, which could lead to a vulnerability in very specific circumstances (called "CRIME"), and it by default supports RC4, which has recently been condemned by cryptographers as potentially unsafe.

Also, it does not, at least with default configuration, support forward secrecy.

As far as I understand it, these are all just bad defaults that Twisted inherits from OpenSSL, and whoever built your particular OpenSSL. (I'm pretty sure there are compile-time options for OpenSSL to not include DES, or at least to disable it by default.) That's not to say that we shouldn't offer *better* defaults, but Twisted is not a cryptography library, and for better or worse we rely on OpenSSL's judgement because it's currently the only crypto library we support. Twisted should have a better cipher-suite defaults and some better command-line options for 'twistd web' (probably in the form of better options for the SSL string endpoint syntax) for modifying those defaults if the user has a good reason to. But really, it would be nicer to just defer to the judgement of a transport layer security library that has *good* judgement about defaults rather than re-hashing every questionable decision that OpenSSL makes. -glyph

On Oct 7, 2014, at 3:21 PM, Carl D'Halluin <carl@amplidata.com> wrote:

...

Another problem is that Twisted doesn't yet support SSL on adopted sockets.

I did this by hand:

# Suppose your create/bind/list your listen_socket, and # its file descriptor is listen_socket_fd

site = server.Site(MyHttpsSite())

cert = '/path/to/my/cert' key = '/path/to/my/key'

ctx = DefaultOpenSSLContextFactory(key, cert) tlsFactory = tls.TLSMemoryBIOFactory(ctx, False, site) p = tcp.Port._fromListeningDescriptor(reactor, listen_socket_fd, socket.AF_INET, tlsFactory) p._type = 'TLS' p.startListening()

os.close(listen_socket_fd) reactor.run()

There is work underway for addressing this particular use-case (endpoint composition) via string endpoints: <https://twistedmatrix.com/trac/ticket/5642> But even today you don't have to touch unsupported private APIs to do this. As per <https://twistedmatrix.com/trac/wiki/CompatibilityPolicy> we really like to discourage people from touching private (i.e. underscore-prefixed or imported-from-another-module) API, because it may well break in the next release and you'll have no recourse. (Plus, you should really be using CertificateOptions, not DefaultOpenSSLContextFactory, either via PrivateCertificate(...).options() or directly constructed.) from twisted.python.filepath import FilePath site = server.Site(MyHttpsSite()) cert = FilePath('/path/to/my/cert').getContent() key = FilePath('/path/to/my/key').getContent() from twisted.internet.ssl import PrivateCertificate certificateWithKey = PrivateCertificate.loadPEM(b"\n".join([cert, key])) tlsFactory = tls.TLSMemoryBIOFactory(certificateWithKey.options(), False, site) import socket from twisted.internet import reactor reactor.adoptStreamPort(listen_socket_fd, socket.AF_INET, tlsFactory) import os os.close(listen_socket_fd) reactor.run() This code hasn't been tested, but no underscores should be necessary! -glyph