[Twisted-Python] Deferred SOAP methods
I am making a SOAP server and all (or at least the vast majority) will be returning a deferred. The whole SOAP method is encapsulated in a deferred in most of my cases (I havent implemented this part yet.) Then I got the idea that twisted.web.soap could just wrap my SOAP method (retrieved with lookupFunction) in a deferred and return that deferred right away. For instance I would change twisted.web.soap.render where it says d = defer.maybeDeferred to d = threads.deferToThread I could also use an attribute to specify if I do not want it deferred (kinda the same way to determine whether to use keywords or not) Is this actually a good idea? I figured there might be a reason this wasn't done already considering everything in twisted is pretty well thought out.
On 02:52 pm, landreville@deadtreepages.com wrote:
I am making a SOAP server and all (or at least the vast majority) will be returning a deferred. The whole SOAP method is encapsulated in a deferred in most of my cases (I havent implemented this part yet.) Then I got the idea that twisted.web.soap could just wrap my SOAP method (retrieved with lookupFunction) in a deferred and return that deferred right away.
For instance I would change twisted.web.soap.render where it says d = defer.maybeDeferred to d = threads.deferToThread
It sounds like you're conflating Deferreds and threads a little bit. The reason Twisted doesn't use deferToThread on any method it calls by default is that threads are a really bummer. You have to worry about thread safety, you generally lose repeatability and determinism, your unit tests become a lot harder to write, if it's still possible to write them at all. Then there are the performance considerations. There are many kinds of code that CPython threads don't make any faster (and many kinds which it makes slower). On the other hand, sometimes you really do want to do your work in a thread. That's why deferToThread exists, after all - to make your life a little bit simpler when those cases do arise. :) What is it that you're going to be doing in these SOAP methods? Jean-Paul
On Fri, Nov 13, 2009 at 10:25 AM, <exarkun@twistedmatrix.com> wrote:
On 02:52 pm, landreville@deadtreepages.com wrote:
I am making a SOAP server and all (or at least the vast majority) will be returning a deferred. The whole SOAP method is encapsulated in a deferred in most of my cases (I havent implemented this part yet.) Then I got the idea that twisted.web.soap could just wrap my SOAP method (retrieved with lookupFunction) in a deferred and return that deferred right away.
For instance I would change twisted.web.soap.render where it says d = defer.maybeDeferred to d = threads.deferToThread
It sounds like you're conflating Deferreds and threads a little bit.
The reason Twisted doesn't use deferToThread on any method it calls by default is that threads are a really bummer. You have to worry about thread safety, you generally lose repeatability and determinism, your unit tests become a lot harder to write, if it's still possible to write them at all. Then there are the performance considerations. There are many kinds of code that CPython threads don't make any faster (and many kinds which it makes slower).
On the other hand, sometimes you really do want to do your work in a thread. That's why deferToThread exists, after all - to make your life a little bit simpler when those cases do arise. :)
What is it that you're going to be doing in these SOAP methods?
Jean-Paul
They are doing a bunch of setting and retrieving information from databases. There are a couple that load more modules which add more SOAP methods for interacting with a different database. None of the calls depend on eachother, so I don't want/need any of them to block the application from processing the next soap call. Is there other ways of generating deferreds without threads that I haven't seen?
On 03:42 pm, landreville@deadtreepages.com wrote:
On Fri, Nov 13, 2009 at 10:25 AM, <exarkun@twistedmatrix.com> wrote:
[snip]
What is it that you're going to be doing in these SOAP methods?
Jean-Paul
They are doing a bunch of setting and retrieving information from databases. There are a couple that load more modules which add more SOAP methods for interacting with a different database.
None of the calls depend on eachother, so I don't want/need any of them to block the application from processing the next soap call. Is there other ways of generating deferreds without threads that I haven't seen?
There are lots of ways. Deferreds are just an approach to organizing callbacks. If you're actually asking if there's a way to call blocking functions without blocking - no, threads are more or less it (there's processes too, but that's sort of the same thing). Database access is definitely an area where you often have to deal with blocking APIs. If that's why your methods block, then threads might be the right solution. You might want to take a look at twisted.enterprise.adbapi, a thread-based wrapper for DB-API 2.0 modules. If you're blocking waiting for the network or other events, though, then you can probably find a Twisted API to deal with that without using threads. Jean-Paul
On Fri, Nov 13, 2009 at 3:46 PM, <exarkun@twistedmatrix.com> wrote:
On 03:42 pm, landreville@deadtreepages.com wrote:
On Fri, Nov 13, 2009 at 10:25 AM, <exarkun@twistedmatrix.com> wrote:
[snip]
What is it that you're going to be doing in these SOAP methods?
Jean-Paul
They are doing a bunch of setting and retrieving information from databases. There are a couple that load more modules which add more SOAP methods for interacting with a different database.
None of the calls depend on eachother, so I don't want/need any of them to block the application from processing the next soap call. Is there other ways of generating deferreds without threads that I haven't seen?
There are lots of ways. Deferreds are just an approach to organizing callbacks. If you're actually asking if there's a way to call blocking functions without blocking - no, threads are more or less it (there's processes too, but that's sort of the same thing).
Database access is definitely an area where you often have to deal with blocking APIs. If that's why your methods block, then threads might be the right solution. You might want to take a look at twisted.enterprise.adbapi, a thread-based wrapper for DB-API 2.0 modules.
If you're blocking waiting for the network or other events, though, then you can probably find a Twisted API to deal with that without using threads.
Jean-Paul
Most of the time it is datbase access. I guess I shouldn't have chosen probiscus using the greentrunk api instead of the dbapi. I haven't been pleased by the postgres drivers in python, so I'm wary to change the db api and use adbapi around it. Sometimes the calls are blocking because it is calling another SOAP server (or possibly itself over SOAP). My SOAP calls will all be blocking because twisted.web.soap doesn't support WSDL. Other than db access and accessing soap calls the rest is just formatting the results. I'm thinking maybe I could make an attribute of my soap method called "deferred = True|False" and then if it's True render will wrap the call to the soap method in a deferred thread, otherwise I will defer blocking statements inside the SOAP method.
Landreville wrote:
Sometimes the calls are blocking because it is calling another SOAP server (or possibly itself over SOAP). My SOAP calls will all be blocking because twisted.web.soap doesn't support WSDL.
What are you using for your SOAP client calls, then? There is something called "ZSI" (http://pywebsvcs.sourceforge.net/zsi.html) that can operate with Twisted and supports WSDL. I once used it to write a server that could be used with an existing WSDL as its interface. IIRC its Twised integration has some problems (like not supporting asynchronous handling of requests) but they were relatively simple to overcome.
On Sat, Nov 14, 2009 at 9:15 AM, Tuure Laurinolli < tuure.laurinolli@indagon.com> wrote:
Landreville wrote:
Sometimes the calls are blocking because it is calling another SOAP server (or possibly itself over SOAP). My SOAP calls will all be blocking because twisted.web.soap doesn't support WSDL.
What are you using for your SOAP client calls, then? There is something called "ZSI" (http://pywebsvcs.sourceforge.net/zsi.html) that can operate with Twisted and supports WSDL.
I once used it to write a server that could be used with an existing WSDL as its interface. IIRC its Twised integration has some problems (like not supporting asynchronous handling of requests) but they were relatively simple to overcome.
I'm using soappy for a client because that is what twisted uses for its soap server. Twisted's implementation does not include WSDL support though (so I'm using the normal soappy package for the client).
On 05:21 pm, landreville@deadtreepages.com wrote:
On Sat, Nov 14, 2009 at 9:15 AM, Tuure Laurinolli < tuure.laurinolli@indagon.com> wrote:
Landreville wrote:
Sometimes the calls are blocking because it is calling another SOAP server (or possibly itself over SOAP). My SOAP calls will all be blocking because twisted.web.soap doesn't support WSDL.
What are you using for your SOAP client calls, then? There is something called "ZSI" (http://pywebsvcs.sourceforge.net/zsi.html) that can operate with Twisted and supports WSDL.
I once used it to write a server that could be used with an existing WSDL as its interface. IIRC its Twised integration has some problems (like not supporting asynchronous handling of requests) but they were relatively simple to overcome. I'm using soappy for a client because that is what twisted uses for its soap server. Twisted's implementation does not include WSDL support though (so I'm using the normal soappy package for the client).
Twisted's SOAP support is definitely pretty pathetic. This is probably a result of a few things: * None of the core developers is particularly interested in SOAP. It's a bad protocol that people generally only use because they're forced to. * The Python story for SOAP is confusing. For a while, SOAPpy was the only game in town. Then ZSI came along and that was supposed to be great. Last I heard, ZSI should be avoided and SUDS is the thing to use. Without actually having a real SOAP-related problem to solve, it's hard to tell who to believe. * Various people are using ZSI with Twisted based on the support in ZSI for this. This presumably reduces the motivation to add anything in Twisted to support such integration. Likewise, people are using Twisted and SUDS together with some other custom integration code (I don't think this is part of SUDS proper, though). Again, if it works for them, it's less likely that they'll bother trying to get something better included in Twisted. I don't think that it is necessarily a bad thing that Twisted doesn't have a super awesome SOAP API built in, since it seems that other people are coming up with ways to solve the problem anyway. The biggest drawback is perhaps that it is hard for someone to find all these disparate solutions and select the one that is most suited to solving their problem. Jean-Paul
participants (3)
-
exarkun＠twistedmatrix.com -
Landreville -
Tuure Laurinolli