On Fri, Nov 13, 2009 at 10:25 AM, wrote:

On 02:52 pm, landreville@deadtreepages.com wrote:

...

I am making a SOAP server and all (or at least the vast majority) will be returning a deferred. The whole SOAP method is encapsulated in a deferred in most of my cases (I havent implemented this part yet.) Then I got the idea that twisted.web.soap could just wrap my SOAP method (retrieved with lookupFunction) in a deferred and return that deferred right away.

For instance I would change twisted.web.soap.render where it says d = defer.maybeDeferred to d = threads.deferToThread

It sounds like you're conflating Deferreds and threads a little bit.

The reason Twisted doesn't use deferToThread on any method it calls by default is that threads are a really bummer. You have to worry about thread safety, you generally lose repeatability and determinism, your unit tests become a lot harder to write, if it's still possible to write them at all. Then there are the performance considerations. There are many kinds of code that CPython threads don't make any faster (and many kinds which it makes slower).

On the other hand, sometimes you really do want to do your work in a thread. That's why deferToThread exists, after all - to make your life a little bit simpler when those cases do arise. :)

What is it that you're going to be doing in these SOAP methods?

Jean-Paul

They are doing a bunch of setting and retrieving information from databases. There are a couple that load more modules which add more SOAP methods for interacting with a different database. None of the calls depend on eachother, so I don't want/need any of them to block the application from processing the next soap call. Is there other ways of generating deferreds without threads that I haven't seen?

On Fri, Nov 13, 2009 at 3:46 PM, wrote:

On 03:42 pm, landreville@deadtreepages.com wrote:

...

On Fri, Nov 13, 2009 at 10:25 AM, wrote:

[snip]

...

What is it that you're going to be doing in these SOAP methods?

Jean-Paul

They are doing a bunch of setting and retrieving information from databases. There are a couple that load more modules which add more SOAP methods for interacting with a different database.

None of the calls depend on eachother, so I don't want/need any of them to block the application from processing the next soap call. Is there other ways of generating deferreds without threads that I haven't seen?

There are lots of ways. Deferreds are just an approach to organizing callbacks. If you're actually asking if there's a way to call blocking functions without blocking - no, threads are more or less it (there's processes too, but that's sort of the same thing).

Database access is definitely an area where you often have to deal with blocking APIs. If that's why your methods block, then threads might be the right solution. You might want to take a look at twisted.enterprise.adbapi, a thread-based wrapper for DB-API 2.0 modules.

If you're blocking waiting for the network or other events, though, then you can probably find a Twisted API to deal with that without using threads.

Jean-Paul

Most of the time it is datbase access. I guess I shouldn't have chosen probiscus using the greentrunk api instead of the dbapi. I haven't been pleased by the postgres drivers in python, so I'm wary to change the db api and use adbapi around it. Sometimes the calls are blocking because it is calling another SOAP server (or possibly itself over SOAP). My SOAP calls will all be blocking because twisted.web.soap doesn't support WSDL. Other than db access and accessing soap calls the rest is just formatting the results. I'm thinking maybe I could make an attribute of my soap method called "deferred = True|False" and then if it's True render will wrap the call to the soap method in a deferred thread, otherwise I will defer blocking statements inside the SOAP method.

On Sat, Nov 14, 2009 at 9:15 AM, Tuure Laurinolli < tuure.laurinolli@indagon.com> wrote:

Landreville wrote:

...

Sometimes the calls are blocking because it is calling another SOAP server (or possibly itself over SOAP). My SOAP calls will all be blocking because twisted.web.soap doesn't support WSDL.

What are you using for your SOAP client calls, then? There is something called "ZSI" (http://pywebsvcs.sourceforge.net/zsi.html) that can operate with Twisted and supports WSDL.

I once used it to write a server that could be used with an existing WSDL as its interface. IIRC its Twised integration has some problems (like not supporting asynchronous handling of requests) but they were relatively simple to overcome.

I'm using soappy for a client because that is what twisted uses for its soap server. Twisted's implementation does not include WSDL support though (so I'm using the normal soappy package for the client).