Twisted and reverse proxies
Hi Twisted folks, We have a pure Twisted application that runs on a URL akin to: http://hostname:8080 We wish to use an Apache reverse proxy to allow our web users to transparently view the Twisted application 'within' our existing Apache served site. So that, something like: http://www.companyname.com/tools/twisted_app proxies to: http://hostname:8080 As someone relatively new to Twisted, how will our static (and other) resources react to being reverse proxy served? Will they barf? Is there any online documentation I can look at for case studies? Thanks in advance, - Rob
Hi Rob: We have a number of different installations running precisely that arrangement -- Twisted running behind Apache (or Nginx). Very stable and reliable. One note, you ask how your static resources 'will react' to being reverse proxied. When running Twisted behind Apache, we find it most useful to serve all the static files directly from Apache and leave Twisted to serve only the dynamic content. No reason to use Twisted to serve static files when there's a perfectly good Apache server available. The one small issue we've had is that all the log entries in Twisted come out showing a source IP of 127.0.0.1 (i.e., the Apache server). We turned on the X-Forwarded headers in Apache and ended up hacking some library code in Twisted, but always figured we must have missed something since it seems that grabbing the X-Forwarded header should either a) be something Twisted would do; or b) have a simple setting to indicate the use of X-Forwarded. (perhaps some more knowledgeable person will point me in the right direction.) --Ray ----- Original Message ----- From: "Rob Newman" <rlnewman@ucsd.edu> To: twisted-web@twistedmatrix.com Sent: Tuesday, March 30, 2010 7:52:46 PM GMT -05:00 US/Canada Eastern Subject: [Twisted-web] Twisted and reverse proxies Hi Twisted folks, We have a pure Twisted application that runs on a URL akin to: http://hostname:8080 We wish to use an Apache reverse proxy to allow our web users to transparently view the Twisted application 'within' our existing Apache served site. So that, something like: http://www.companyname.com/tools/twisted_app proxies to: http://hostname:8080 As someone relatively new to Twisted, how will our static (and other) resources react to being reverse proxy served? Will they barf? Is there any online documentation I can look at for case studies? Thanks in advance, - Rob _______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web -- Ray Cote, President Appropriate Solutions, Inc. We Build Software 603.924.6079
On Wednesday 31 March 2010, Ray Cote wrote:
One note, you ask how your static resources 'will react' to being reverse proxied. When running Twisted behind Apache, we find it most useful to serve all the static files directly from Apache and leave Twisted to serve only the dynamic content. No reason to use Twisted to serve static files when there's a perfectly good Apache server available.
In most cases you are right, but there can be a reason to let Twisted serve the static files. For performance it is best to set the expiery date of a static resource a long time (1 year) into the future. That way, the browser can load it once and uses its cached version ever after. However, if you change the resoure, for example to add elements to your style sheet, you want the new version to be used instead of the outdated cached version. To force the new version to be loaded without reducing the expiery time dramatically or having to manually flush the browser's cache, you can give the new version a new URL. You could put the release version in the URL: "app/static/1.0", "app/static/1.1" etc. That works well for deployed apps, but is still a pain for development. So what I did instead was generate a unique ID on startup of the app and use that ID in the URL: "app/static-HDUI89BDY3/main.css", "app/static-HDUI89BDY3/icon.png" etc. So after a content chance, an app restart ensures the browser will refetch everything. An alternative would be to compute a hash of the static content and use that hash the URL. That way you would not need an application restart when static content changes. Bye, Maarten
On Wednesday 31 March 2010, I wrote:
An alternative would be to compute a hash of the static content and use that hash the URL. That way you would not need an application restart when static content changes.
I just realized that this would make it very difficult to have one static resource (style sheet) refer to another static resource (image). Unless you would rewrite the referring resource, but then it's not really static anymore. The approach with an ID generated at startup does not have this problem, since the ID can be put in the URL as a path element and all references between static resources can be done as relative URLs. Bye, Maarten
On 01:50 am, rgacote@appropriatesolutions.com wrote:
The one small issue we've had is that all the log entries in Twisted come out showing a source IP of 127.0.0.1 (i.e., the Apache server). We turned on the X-Forwarded headers in Apache and ended up hacking some library code in Twisted, but always figured we must have missed something since it seems that grabbing the X-Forwarded header should either a) be something Twisted would do; or b) have a simple setting to indicate the use of X-Forwarded. (perhaps some more knowledgeable person will point me in the right direction.)
This is a poor behavior of Twisted Web logging, and something I'd very much like to see fixed. Since the better header handling code appeared in Twisted 8.2 or so, and since the Twisted Web proxy itself is again apparently working in Twisted 10.0, this is probably more feasible to actually fix. http://twistedmatrix.com/trac/ticket/1468 is the ticket for this. Notice that I said I was going to work on it a while ago, but then ran into the various issues above, and never got anywhere. Possibly with those roadblocks out of the way, the plan I described there could be attempted (and perhaps it will even turn out to be a good one, but no promises). Jean-Paul
On Wednesday 31 March 2010, Rob Newman wrote:
We wish to use an Apache reverse proxy to allow our web users to transparently view the Twisted application 'within' our existing Apache served site. So that, something like:
http://www.companyname.com/tools/twisted_app
proxies to:
This is what the Apache configuration would look like: === # Serve static content in Apache: # (the ProxyPass exception must be located before the ProxyPass rule) ProxyPass /tools/twisted_app/static ! Alias /tools/twisted_app/static /some/path/on/disk/static # Act as a proxy for Twisted: <Location /tools/twisted_app> ProxyPass http://hostname:8080 retry=1 ProxyPassReverse http://hostname:8080 ProxyPassReverseCookiePath / /tools/twisted_app/ </Location> === "ProxyPass" sets up the reverse proxy. The "retry=1" ensures that if your Twisted instance returns from being unavailable (for example after an upgrade), Apache will retry contacting it 1 second later. The default is a lot longer than 1 second. "ProxyPassReverse" rewrites headers so redirections continue to work. It does not rewrite content. "ProxyPassReverseCookiePath" rewrites cookies to make their path correspond to the URL of your web app. Without this, if you run multiple Twisted applications reverse proxied on the same host, their (session) cookies will have identical paths and start overwriting each other.
As someone relatively new to Twisted, how will our static (and other) resources react to being reverse proxy served? Will they barf? Is there any online documentation I can look at for case studies?
As Ray wrote, in most cases it is best to let Apache serve the static content and use Twisted for the dynamic content. If your app puts absolute URLs in the content it produces, such as registration conformation e-mails or RSS/Atom feeds, it must use the "/tools/twisted_app" outer URL for this, since Apache does not rewrite content. So the application must be aware that it is reverse proxied and under which URL. One thing to remember is that Apache buffers the data it gets from Twisted before sending it back to the requester. So if you stream events instead of writing a document, the requester will not receive the events until much later. I worked around this by reading streaming data directly from the Twisted HTTP server. Bye, Maarten
participants (4)
-
exarkun＠twistedmatrix.com -
Maarten ter Huurne -
Ray Cote -
Rob Newman