Hi Folks-
I'm new to this list. (Am beginning to think more about learning twisted... for the third time ;) ). I've got a question for you all and hope that you don't take it the wrong way. It's not meant to be flame bait.
As WSGI is going into python 2.5, I was wondering what the general consensus is among twisted web devs about it? Does the WSGI model map well into twisted's asynchronous model? Does twisted already have everything it needs, so WSGI doesn't bring much to the table?
It appears that as AJAX becomes more popular then the asynchronous model might scale better with many small requests. Say I have a WSGI app and I want to port it to twisted. How easy is that? (I know there is a twisted wsgi module) Do I just need to wrap some logic in deferreds?
Are any twisted people using WSGI?
Again, I'm just interested in hearing your constructive thoughts. Google didn't turn up much on the subject.
thanks,
-matt
On Wed, 05 Apr 2006 14:27:30 -0400, m h sesquile@gmail.com wrote:
Hi Folks-
I'm new to this list. (Am beginning to think more about learning twisted... for the third time ;) ). I've got a question for you all and hope that you don't take it the wrong way. It's not meant to be flame bait.
As WSGI is going into python 2.5, I was wondering what the general consensus is among twisted web devs about it? Does the WSGI model map well into twisted's asynchronous model? Does twisted already have everything it needs, so WSGI doesn't bring much to the table?
In my opinion, no. WSGI doesn't deal with the asynchronous model at all. The original author, Phillip Eby, noted that every time he tried to figure out what async WSGI would look like, he ended up with something like twisted or PEAK anyway.
In general, WSGI seems to have caught on with people who like threaded servers. I honestly see zero value in it if you are going to use an integration toolkit like twisted.
It appears that as AJAX becomes more popular then the asynchronous model might scale better with many small requests. Say I have a WSGI app and I want to port it to twisted. How easy is that? (I know there is a twisted wsgi module) Do I just need to wrap some logic in deferreds?
If you want to do AJAX-y things, then you want Athena, which runs on twisted, and does even more than run-of-the-mill AJAX. Google for "comet" and you'll get an idea of the 2-way web functionality that Athena can provide.
Are any twisted people using WSGI?
Lord, I hope not. The Zope 3 team runs their app-server on top of twisted via WSGI, but they are the only ones I know of.
Again, I'm just interested in hearing your constructive thoughts. Google didn't turn up much on the subject.
thanks,
-matt
Hope this helps,
Daniel
m h wrote:
I'm new to this list. (Am beginning to think more about learning twisted... for the third time ;) ). I've got a question for you all and hope that you don't take it the wrong way. It's not meant to be flame bait.
Don't worry, I don't see how it could remotely be considered anything but an honest and constructive inquiry!
As WSGI is going into python 2.5, I was wondering what the general consensus is among twisted web devs about it? Does the WSGI model map well into twisted's asynchronous model? Does twisted already have everything it needs, so WSGI doesn't bring much to the table?
For a Twisted HTTP server (web2) to serve as a WSGI gateway and run a WSGI application, the application has to either operate under the Twisted non-blocking model or has to be run in a thread. The latter is what's being done with everything run under twisted.web2.wsgi right now; the WSGI Resource object returns the result of a deferToThread(f) where f is the potentially blocking WSGI app.
However, you could write your WSGI app in a way that quickly returns a deferred to its output and then crank away at its leisure with some Twisted-friendly technique like iterations of reactor.callLater(). (The Twisted devs may correct me and suggest more Twisted-friendly techniques for "cranking away" at something...) The advantage of the async approach would be (1) it's more in tune with overall Twisted approach, and (2) you could have partial results start to show up on the page before the WSGI app is completely done producing them.
It appears that as AJAX becomes more popular then the asynchronous model might scale better with many small requests. Say I have a WSGI app and I want to port it to twisted. How easy is that? (I know there is a twisted wsgi module) Do I just need to wrap some logic in deferreds?
Since twisted.web2.wsgi currently runs the WSGI app in a thread (and will no doubt keep the option of doing so even if it broadens to allow for async WSGI apps), it's very simple. You just let your app block all it wants, and the deferred that results from deferToThread will fire when it's done.
Are any twisted people using WSGI?
Yes, I am. I use it to run Trac over Twisted.web2 at my http://foss.eepatents.com site. You can see both an example of it working and some pertinent code at http://foss.eepatents.com/DynamicSite/. That is, by the way, one of several virtual-host sites all running from a single instance of DynamicSite, my same twisted.web2 HTTP server.
Good luck!
-Ed Suominen
m h wrote:
Hi Folks-
I'm new to this list. (Am beginning to think more about learning twisted... for the third time ;) ). I've got a question for you all and hope that you don't take it the wrong way. It's not meant to be flame bait.
I'm not a Twisted developer, so no worries. :)
As WSGI is going into python 2.5, I was wondering what the general consensus is among twisted web devs about it? Does the WSGI model map well into twisted's asynchronous model?
Not as well as it should IMO. There seems to be some disdain for asynchronous servers on the WSGI list. I tried to advocate on the WSGI list for better support for asynchronous servers, especially wrt threading issues.
Does twisted already have everything it needs, so WSGI doesn't bring much to the table?
WSGI makes it much easier for applications to leverage Twisted, so it increases Twisted's potential user base.
It appears that as AJAX becomes more popular then the asynchronous model might scale better with many small requests.
An asynchronous model works well for many requests period, whether they are large or small.
Say I have a WSGI app and I want to port it to twisted. How easy is that? (I know there is a twisted wsgi module)
Trivial.
Do I just need to wrap some logic in deferreds?
No. You don't have to do any twisted programming at all to use the web server via WSGI.
IMO, to get the full scalability benefits you want, you may need to address some threading issues. You might look at the Zope 3 integration, which isn't as clean as it ought to me.
Are any twisted people using WSGI?
I don't know. Zope 3 is using it. Zope 2 will be using it in the future. I don't think we count as Twisted people. (Although we might count as twisted people. ;)
Jim
Jim Fulton wrote:
As WSGI is going into python 2.5, I was wondering what the general consensus is among twisted web devs about it? Does the WSGI model map well into twisted's asynchronous model?
Not as well as it should IMO. There seems to be some disdain for asynchronous servers on the WSGI list. I tried to advocate on the WSGI list for better support for asynchronous servers, especially wrt threading issues.
I think "disdain" is too strong. There's a lack of any complete proposal which would work, especially one that doesn't have Deferred or other particular bits of code as a prerequesite. There was discussion a long time ago, but it drifted off. More generally, there's no larger consensus on how to do asynchronous programming in Python, though there's obvious consensus on how to do synchronous programming (that is: with functions).
I remember some discussion about allowing an asynchronous application to return '' in the WSGI app response iterator, which would signify a yield, but I'm not sure where any ready signal would go.
Say I have a WSGI app and I want to port it to twisted. How easy is that? (I know there is a twisted wsgi module)
Trivial.
Which reminds me -- I started doing this in Paste, but got bogged down in all the setup and imports I didn't understand, and then subscribed here and never followed up. Can someone provide an example of a simple function that would look like:
def serve_with_twisted(hosts, wsgi_app, **kw): """Serve wsgi_app indefinitely
hosts is a list of 'address:port', and wsgi_app is a WSGI application. **kw is... whatever other interesting things you might want to use to configure a Twisted server """
Then I can lightly wrap that and people could use Twisted with their Paste configuration files.
Do I just need to wrap some logic in deferreds?
No. You don't have to do any twisted programming at all to use the web server via WSGI.
IMO, to get the full scalability benefits you want, you may need to address some threading issues. You might look at the Zope 3 integration, which isn't as clean as it ought to me.
Is this related to the threadpool, or are you using backdoors to the underlying Twisted server to do some things in an asynchronous manner?
On Wed, 05 Apr 2006 14:39:36 -0500, Ian Bicking ianb@colorstudy.com wrote:
I think "disdain" is too strong. There's a lack of any complete proposal which would work, especially one that doesn't have Deferred or other particular bits of code as a prerequesite. There was discussion a long time ago, but it drifted off. More generally, there's no larger consensus on how to do asynchronous programming in Python, though there's obvious consensus on how to do synchronous programming (that is: with functions).
There is plenty of consensus on how to do asynchronous programming with Python: use Twisted ;-).
Which reminds me -- I started doing this in Paste, but got bogged down in all the setup and imports I didn't understand, and then subscribed here and never followed up. Can someone provide an example of a simple function that would look like:
Taking this approach restricts your deployment to be web-only, which eliminates a substantial advantage of Twisted. It isn't hard to implement, though:
from twisted.web2 import server, wsgi from twisted.web2.channel import http from twisted.internet import reactor
def serve_with_twisted(hosts, wsgi_app, **kw): """Serve wsgi_app indefinitely
hosts is a list of 'address:port', and wsgi_app is a WSGI application. **kw is... whatever other interesting things you might want to use to configure a Twisted server """
factory = http.HTTPFactory(server.Site(wsgi.WSGIResource(app))) for hostportpair in hosts: host, portstr = hostportpair.split(':') port = int(portstr) reactor.listenTCP(port, factory, interface=host) reactor.run()
This is completely untested, but it should at least get you going in the right direction.
participants (6)
-
Ed Suominen -
glyph＠divmod.com -
Ian Bicking -
Jim Fulton -
L. Daniel Burr -
m h