On Oct 21, 2013, at 1:16 PM, Phil Mayers wrote:

Does the Twisted wsgi runner have support for process rather than threading models to avoid the gil? Because if so I should take a look at it...

Nope. It ought to though: https://twistedmatrix.com/trac/ticket/5180. Any interest in tackling that? -glyph

On 07:41 am, orestis@orestis.gr wrote:

On 21 Οκτ 2013, at 10:32 μ.μ., Glyph wrote:

...

On Oct 20, 2013, at 2:21 AM, Orestis Markou wrote:

...

Hello,

Short form of the question:

Are people using Twisted to host WSGI applications AND serve static files AND replace celery/redis/other?

I'm not personally using it as a WSGI host, but otherwise, yes, a full-stack application container speaking multiple protocols.

Any pointers on how to best use this in combination with WSGI/Django? In the past I had a combination of twisted-web (for /static and /media) and wsgi host (for everything else), all running under the same Service. Essentially:

os.environ['DJANGO_SETTINGS_MODULE'] = 'myproject.settings' application = django.core.handlers.wsgi.WSGIHandler()

wsgi_resource = WSGIResource(reactor, reactor.getThreadPool(), application) resource = Root(wsgi_resource) # this could probably be automatically inferred from settings.py resource.putChild('static', File(...)) resource.putChild('uploaded', File(...)) # other stuff site = server.Site(resource) reactor.listenTCP(8000, site) reactor.run()

This looks about like I'd expect (if my guess that `Root` is an `IResource` that knows how to split dispatch between the WSGI resource and its other children). If you have any suggestions for improving the experience please share them. :)

[snip]

Here's a thought experiment - I'd like to keep URL routing 100% in Django for anything that hits the DB. I have some code that needs to spawn an external process to generate an image on-demand (with a layer of caching on top). In the past I defined a Twisted.Web handler for that. Could I now expose an internal API that (through Crochet?) do the spawnProcess dance and come back with image data that Django could then handle internally (store in a file, whatever). How would the threaded WSGI container deal with the blocking request (not really blocking, but that request would stall until the Deferred would be fired).

It will produce roughly the same results as you'd get if you used any other WSGI container: one of the threads in the thread pool will be kept unavailable as it waits for the result. This will have the usual consequence: if your threadpool has a max of N threads and you receive N requests that need to do this, the N+1th request that needs to be handled by the WSGI part of your server won't be handled until one of the previous requests completes (completion frees up one thread which is then used to handle the N+1th request). The only difference might be that since you also have non-WSGI content (all of your static content) even when your thread pool is completely in use requests for static content will still be satisfied right away. However, if you previously had a WSGI+lighttpd/whatever then you probably already had this property as well. Put another way, Twisted's WSGI container is still just a WSGI container. Fortunately Twisted has other pieces aside from a WSGI container. :) Jean-Paul

On 22 Οκτ 2013, at 3:38 μ.μ., exarkun@twistedmatrix.com wrote:

On 07:41 am, orestis@orestis.gr wrote:

...

On 21 Οκτ 2013, at 10:32 μ.μ., Glyph wrote:

...

On Oct 20, 2013, at 2:21 AM, Orestis Markou wrote:

...

Hello,

Short form of the question:

Are people using Twisted to host WSGI applications AND serve static files AND replace celery/redis/other?

I'm not personally using it as a WSGI host, but otherwise, yes, a full-stack application container speaking multiple protocols.

Any pointers on how to best use this in combination with WSGI/Django? In the past I had a combination of twisted-web (for /static and /media) and wsgi host (for everything else), all running under the same Service. Essentially:

os.environ['DJANGO_SETTINGS_MODULE'] = 'myproject.settings' application = django.core.handlers.wsgi.WSGIHandler()

wsgi_resource = WSGIResource(reactor, reactor.getThreadPool(), application) resource = Root(wsgi_resource) # this could probably be automatically inferred from settings.py resource.putChild('static', File(...)) resource.putChild('uploaded', File(...)) # other stuff site = server.Site(resource) reactor.listenTCP(8000, site) reactor.run()

This looks about like I'd expect (if my guess that `Root` is an `IResource` that knows how to split dispatch between the WSGI resource and its other children). If you have any suggestions for improving the experience please share them. :)

For anyone following along: class Root(resource.Resource): def __init__(self, wsgi_resource): resource.Resource.__init__(self) self.wsgi_resource = wsgi_resource def getChild(self, path, request): path0 = request.prepath.pop(0) request.postpath.insert(0, path0) return self.wsgi_resource I don't remember now if I wrote this code or if I copied it from somewhere.

...

[snip]

Here's a thought experiment - I'd like to keep URL routing 100% in Django for anything that hits the DB. I have some code that needs to spawn an external process to generate an image on-demand (with a layer of caching on top). In the past I defined a Twisted.Web handler for that. Could I now expose an internal API that (through Crochet?) do the spawnProcess dance and come back with image data that Django could then handle internally (store in a file, whatever). How would the threaded WSGI container deal with the blocking request (not really blocking, but that request would stall until the Deferred would be fired).

It will produce roughly the same results as you'd get if you used any other WSGI container: one of the threads in the thread pool will be kept unavailable as it waits for the result. This will have the usual consequence: if your threadpool has a max of N threads and you receive N requests that need to do this, the N+1th request that needs to be handled by the WSGI part of your server won't be handled until one of the previous requests completes (completion frees up one thread which is then used to handle the N+1th request).

The only difference might be that since you also have non-WSGI content (all of your static content) even when your thread pool is completely in use requests for static content will still be satisfied right away. However, if you previously had a WSGI+lighttpd/whatever then you probably already had this property as well.

Put another way, Twisted's WSGI container is still just a WSGI container. Fortunately Twisted has other pieces aside from a WSGI container. :)

I wonder if this could be turned on its head - instead of having the Django WSGI app blocking while waiting for a long running thing, field the request beforehand, do whatever long-running thing is required and only then hand over the to the WSGI app to proceed with the HTTP response. This obviously requires a level request processing in Twisted - I'm not sure if it's worth the effort. It could be done by a Twisted-savvy WSGI middleware, perhaps. I'll try to see what I can come up with. Orestis

Jean-Paul

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

On 23 Οκτ 2013, at 11:21 π.μ., Glyph wrote:

On Oct 22, 2013, at 12:41 AM, Orestis Markou wrote:

...

Any pointers on how to best use this in combination with WSGI/Django? In the past I had a combination of twisted-web (for /static and /media) and wsgi host (for everything else), all running under the same Service. Essentially:

Have you seen https://github.com/clemesha/twisted-wsgi-django/blob/master/server.py? It seems like maybe that repository could use a little maintenance, maybe a couple of command-line options, to do the (fairly default) thing that you seem to want to do :-).

Bookmarked! I will probably need to really scratch the itch in a couple of weeks time. I'll update the list when I have some results. Orestis