[Twisted-Python] Twisted as a full solution for web hosting [WSGI + other]
Hello, Short form of the question: Are people using Twisted to host WSGI applications AND serve static files AND replace celery/redis/other? Are there any inherent drawbacks in using Twisted for this use case? Long form of the question: These days to get a reasonably feature-full python web stack deployed you need to have a lot of 3rd-party libraries. The bare minimum looks probably like so: 1. Your web-framework of choice, like Django 2. Some kind of WSGI container, like gunicorn 3. A static file server, like nginx 4. Some kind of database [off-topic for this message] Additionally, you might want: * Celery * Redis * Cron * Something for web sockets or similar technology * … and so on In my experience, Twisted can be used to replace a lot of those use cases: * It has a WSGI container * It has a static web server * It can be used for other long-running tasks I'd like to know if there is some kind of inherent drawback of using Twisted to fill those areas. My use case deals with many small intranet-like deployments of web applications, and I'd like to streamline the stack as much as possible. I believe that with the newly-released Crochet someone could even write a Django extension to replace `runserver` with something more production-oriented. Orestis
On Oct 20, 2013, at 2:21 AM, Orestis Markou <orestis@orestis.gr> wrote:
Hello,
Short form of the question:
Are people using Twisted to host WSGI applications AND serve static files AND replace celery/redis/other?
I'm not personally using it as a WSGI host, but otherwise, yes, a full-stack application container speaking multiple protocols.
Are there any inherent drawbacks in using Twisted for this use case?
Nope. Twisted is the best :-).
Long form of the question:
These days to get a reasonably feature-full python web stack deployed you need to have a lot of 3rd-party libraries.
The bare minimum looks probably like so:
1. Your web-framework of choice, like Django 2. Some kind of WSGI container, like gunicorn 3. A static file server, like nginx 4. Some kind of database [off-topic for this message]
Additionally, you might want:
* Celery * Redis * Cron * Something for web sockets or similar technology * … and so on
In my experience, Twisted can be used to replace a lot of those use cases:
* It has a WSGI container * It has a static web server * It can be used for other long-running tasks
I'd like to know if there is some kind of inherent drawback of using Twisted to fill those areas. My use case deals with many small intranet-like deployments of web applications, and I'd like to streamline the stack as much as possible. I believe that with the newly-released Crochet someone could even write a Django extension to replace `runserver` with something more production-oriented.
If you hit any problems, they're bugs, report them and they'll get fixed. Please do this. I even gave a talk about this at DjangoCon a couple of years ago: <http://blip.tv/djangocon/keynote-glyph-lefkowitz-5573264> -glyph
Does the Twisted wsgi runner have support for process rather than threading models to avoid the gil? Because if so I should take a look at it... -- Sent from my phone with, please excuse brevity and typos
On Oct 21, 2013, at 1:16 PM, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
Does the Twisted wsgi runner have support for process rather than threading models to avoid the gil? Because if so I should take a look at it...
Nope. It ought to though: <https://twistedmatrix.com/trac/ticket/5180>. Any interest in tackling that? -glyph
On 21 Οκτ 2013, at 10:32 μ.μ., Glyph <glyph@twistedmatrix.com> wrote:
On Oct 20, 2013, at 2:21 AM, Orestis Markou <orestis@orestis.gr> wrote:
Hello,
Short form of the question:
Are people using Twisted to host WSGI applications AND serve static files AND replace celery/redis/other?
I'm not personally using it as a WSGI host, but otherwise, yes, a full-stack application container speaking multiple protocols.
Any pointers on how to best use this in combination with WSGI/Django? In the past I had a combination of twisted-web (for /static and /media) and wsgi host (for everything else), all running under the same Service. Essentially: os.environ['DJANGO_SETTINGS_MODULE'] = 'myproject.settings' application = django.core.handlers.wsgi.WSGIHandler() wsgi_resource = WSGIResource(reactor, reactor.getThreadPool(), application) resource = Root(wsgi_resource) # this could probably be automatically inferred from settings.py resource.putChild('static', File(...)) resource.putChild('uploaded', File(...)) # other stuff site = server.Site(resource) reactor.listenTCP(8000, site) reactor.run()
Are there any inherent drawbacks in using Twisted for this use case?
Nope. Twisted is the best :-).
Long form of the question:
These days to get a reasonably feature-full python web stack deployed you need to have a lot of 3rd-party libraries.
The bare minimum looks probably like so:
1. Your web-framework of choice, like Django 2. Some kind of WSGI container, like gunicorn 3. A static file server, like nginx 4. Some kind of database [off-topic for this message]
Additionally, you might want:
* Celery * Redis * Cron * Something for web sockets or similar technology * … and so on
In my experience, Twisted can be used to replace a lot of those use cases:
* It has a WSGI container * It has a static web server * It can be used for other long-running tasks
I'd like to know if there is some kind of inherent drawback of using Twisted to fill those areas. My use case deals with many small intranet-like deployments of web applications, and I'd like to streamline the stack as much as possible. I believe that with the newly-released Crochet someone could even write a Django extension to replace `runserver` with something more production-oriented.
If you hit any problems, they're bugs, report them and they'll get fixed.
Please do this. I even gave a talk about this at DjangoCon a couple of years ago:
That was interesting to watch - I wonder if you know whether things have been improved since then. Here's a thought experiment - I'd like to keep URL routing 100% in Django for anything that hits the DB. I have some code that needs to spawn an external process to generate an image on-demand (with a layer of caching on top). In the past I defined a Twisted.Web handler for that. Could I now expose an internal API that (through Crochet?) do the spawnProcess dance and come back with image data that Django could then handle internally (store in a file, whatever). How would the threaded WSGI container deal with the blocking request (not really blocking, but that request would stall until the Deferred would be fired).
-glyph
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On 07:41 am, orestis@orestis.gr wrote:
On 21 Οκτ 2013, at 10:32 μ.μ., Glyph <glyph@twistedmatrix.com> wrote:
On Oct 20, 2013, at 2:21 AM, Orestis Markou <orestis@orestis.gr> wrote:
Hello,
Short form of the question:
Are people using Twisted to host WSGI applications AND serve static files AND replace celery/redis/other?
I'm not personally using it as a WSGI host, but otherwise, yes, a full-stack application container speaking multiple protocols.
Any pointers on how to best use this in combination with WSGI/Django? In the past I had a combination of twisted-web (for /static and /media) and wsgi host (for everything else), all running under the same Service. Essentially:
os.environ['DJANGO_SETTINGS_MODULE'] = 'myproject.settings' application = django.core.handlers.wsgi.WSGIHandler()
wsgi_resource = WSGIResource(reactor, reactor.getThreadPool(), application) resource = Root(wsgi_resource) # this could probably be automatically inferred from settings.py resource.putChild('static', File(...)) resource.putChild('uploaded', File(...)) # other stuff site = server.Site(resource) reactor.listenTCP(8000, site) reactor.run()
This looks about like I'd expect (if my guess that `Root` is an `IResource` that knows how to split dispatch between the WSGI resource and its other children). If you have any suggestions for improving the experience please share them. :)
[snip]
Here's a thought experiment - I'd like to keep URL routing 100% in Django for anything that hits the DB. I have some code that needs to spawn an external process to generate an image on-demand (with a layer of caching on top). In the past I defined a Twisted.Web handler for that. Could I now expose an internal API that (through Crochet?) do the spawnProcess dance and come back with image data that Django could then handle internally (store in a file, whatever). How would the threaded WSGI container deal with the blocking request (not really blocking, but that request would stall until the Deferred would be fired).
It will produce roughly the same results as you'd get if you used any other WSGI container: one of the threads in the thread pool will be kept unavailable as it waits for the result. This will have the usual consequence: if your threadpool has a max of N threads and you receive N requests that need to do this, the N+1th request that needs to be handled by the WSGI part of your server won't be handled until one of the previous requests completes (completion frees up one thread which is then used to handle the N+1th request). The only difference might be that since you also have non-WSGI content (all of your static content) even when your thread pool is completely in use requests for static content will still be satisfied right away. However, if you previously had a WSGI+lighttpd/whatever then you probably already had this property as well. Put another way, Twisted's WSGI container is still just a WSGI container. Fortunately Twisted has other pieces aside from a WSGI container. :) Jean-Paul
On Tue, Oct 22, 2013 at 6:38 AM, <exarkun@twistedmatrix.com> wrote: [snip]
It will produce roughly the same results as you'd get if you used any other WSGI container: one of the threads in the thread pool will be kept unavailable as it waits for the result. This will have the usual consequence: if your threadpool has a max of N threads and you receive N requests that need to do this, the N+1th request that needs to be handled by the WSGI part of your server won't be handled until one of the previous requests completes (completion frees up one thread which is then used to handle the N+1th request).
[snip] It is "roughly" the same, but it may also be better. We recently replaced flup with Twisted as our WSGI container. With traffic spikes, flup would start dropping requests because our WSGI app is too slow, resulting in 500 errors for our users. With Twisted as the WSGI container, we no longer see the same 500 errors. Eventually, all requests finish. I recommend Twisted as a production WSGI container. Matt Haggard
On 22 Οκτ 2013, at 3:38 μ.μ., exarkun@twistedmatrix.com wrote:
On 07:41 am, orestis@orestis.gr wrote:
On 21 Οκτ 2013, at 10:32 μ.μ., Glyph <glyph@twistedmatrix.com> wrote:
On Oct 20, 2013, at 2:21 AM, Orestis Markou <orestis@orestis.gr> wrote:
Hello,
Short form of the question:
Are people using Twisted to host WSGI applications AND serve static files AND replace celery/redis/other?
I'm not personally using it as a WSGI host, but otherwise, yes, a full-stack application container speaking multiple protocols.
Any pointers on how to best use this in combination with WSGI/Django? In the past I had a combination of twisted-web (for /static and /media) and wsgi host (for everything else), all running under the same Service. Essentially:
os.environ['DJANGO_SETTINGS_MODULE'] = 'myproject.settings' application = django.core.handlers.wsgi.WSGIHandler()
wsgi_resource = WSGIResource(reactor, reactor.getThreadPool(), application) resource = Root(wsgi_resource) # this could probably be automatically inferred from settings.py resource.putChild('static', File(...)) resource.putChild('uploaded', File(...)) # other stuff site = server.Site(resource) reactor.listenTCP(8000, site) reactor.run()
This looks about like I'd expect (if my guess that `Root` is an `IResource` that knows how to split dispatch between the WSGI resource and its other children). If you have any suggestions for improving the experience please share them. :)
For anyone following along: class Root(resource.Resource): def __init__(self, wsgi_resource): resource.Resource.__init__(self) self.wsgi_resource = wsgi_resource def getChild(self, path, request): path0 = request.prepath.pop(0) request.postpath.insert(0, path0) return self.wsgi_resource I don't remember now if I wrote this code or if I copied it from somewhere.
[snip]
Here's a thought experiment - I'd like to keep URL routing 100% in Django for anything that hits the DB. I have some code that needs to spawn an external process to generate an image on-demand (with a layer of caching on top). In the past I defined a Twisted.Web handler for that. Could I now expose an internal API that (through Crochet?) do the spawnProcess dance and come back with image data that Django could then handle internally (store in a file, whatever). How would the threaded WSGI container deal with the blocking request (not really blocking, but that request would stall until the Deferred would be fired).
It will produce roughly the same results as you'd get if you used any other WSGI container: one of the threads in the thread pool will be kept unavailable as it waits for the result. This will have the usual consequence: if your threadpool has a max of N threads and you receive N requests that need to do this, the N+1th request that needs to be handled by the WSGI part of your server won't be handled until one of the previous requests completes (completion frees up one thread which is then used to handle the N+1th request).
The only difference might be that since you also have non-WSGI content (all of your static content) even when your thread pool is completely in use requests for static content will still be satisfied right away. However, if you previously had a WSGI+lighttpd/whatever then you probably already had this property as well.
Put another way, Twisted's WSGI container is still just a WSGI container. Fortunately Twisted has other pieces aside from a WSGI container. :)
I wonder if this could be turned on its head - instead of having the Django WSGI app blocking while waiting for a long running thing, field the request beforehand, do whatever long-running thing is required and only then hand over the to the WSGI app to proceed with the HTTP response. This obviously requires a level request processing in Twisted - I'm not sure if it's worth the effort. It could be done by a Twisted-savvy WSGI middleware, perhaps. I'll try to see what I can come up with. Orestis
Jean-Paul
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Oct 22, 2013, at 12:41 AM, Orestis Markou <orestis@orestis.gr> wrote:
Any pointers on how to best use this in combination with WSGI/Django? In the past I had a combination of twisted-web (for /static and /media) and wsgi host (for everything else), all running under the same Service. Essentially:
Have you seen <https://github.com/clemesha/twisted-wsgi-django/blob/master/server.py>? It seems like maybe that repository could use a little maintenance, maybe a couple of command-line options, to do the (fairly default) thing that you seem to want to do :-). -glyph
On 23 Οκτ 2013, at 11:21 π.μ., Glyph <glyph@twistedmatrix.com> wrote:
On Oct 22, 2013, at 12:41 AM, Orestis Markou <orestis@orestis.gr> wrote:
Any pointers on how to best use this in combination with WSGI/Django? In the past I had a combination of twisted-web (for /static and /media) and wsgi host (for everything else), all running under the same Service. Essentially:
Have you seen <https://github.com/clemesha/twisted-wsgi-django/blob/master/server.py>? It seems like maybe that repository could use a little maintenance, maybe a couple of command-line options, to do the (fairly default) thing that you seem to want to do :-).
Bookmarked! I will probably need to really scratch the itch in a couple of weeks time. I'll update the list when I have some results. Orestis
participants (5)
-
exarkun@twistedmatrix.com
-
Glyph
-
Matt Haggard
-
Orestis Markou
-
Phil Mayers