Mailman 3 April 2008 - Twisted

[Twisted-Python] Question on push/pull producers inter-workings, was : "Is there a simple Producer/Consumer example or tutorial?"
by Gabriel Rossetti 21 Apr '08

21 Apr '08

Hello everyone, I am re-posting these questions with a different title, since they have ventured away from the original question I have a question though, I was looking at the way all this works by using a debugger, and I noticed that in twisted.internet.abstract.py in registerProducer() there is the following : if not streaming: producer.resumeProducing() Why is this done only for the pull producer? Shouldn't it also be called for the push producer since to have the data sent one has to call either self.transport.write() or resumeProducing() anyways? If you look at : http://itamarst.org/writings/OSCON03/twisted_internet-112.html it does : transport.registerProducer(self, 1) self.produce() thus starting the writing process and in the pull producer : http://itamarst.org/writings/OSCON03/twisted_internet-111.html it doesn't need to start the writing process explicitly since it's started when the producer is registered. Oh, and also, since from what I see in the code the only difference with a push and pull producer is that the push producer is paused if the data being written/sent is very large (to let the reactor breath and process other things), if it isn't larger than the buffer it behaves like a pull producer, correct? If so, then why have both? Am I mixed up again? Thank you, Gabriel

2 4

[Twisted-Python] Twisted 8.1.0 prerelease 1
by Christopher Armstrong 20 Apr '08

20 Apr '08

Hey all. I've built a prerelease for Twisted 8.1.0. I'd appreciate if some people would download it and check it out. We've fixed a few important regressions in this release: * some reactor re-entrancy problems have been fixed (#3146, #3168) * Constructing failures of Pyrex-raised exceptions works again * User Guide API links now point to twistedmatrix.com instead of to invalid locations (although this prerelease contains API links to the 8.1.0 docs on twistedmatrix.com, which aren't yet there :) There are also a few very minor features and miscellaneous improvements. You can get the prerelease tar.bz2 files from http://twistedmatrix.com/users/radix/8.1.0pre1/ -- Christopher Armstrong International Man of Twistery http://radix.twistedmatrix.com/ http://twistedmatrix.com/ http://canonical.com/

1 0

[Twisted-Python] feeding-worker-loop problem
by Patrick Scharrenberg 20 Apr '08

20 Apr '08

Hi! I've been thinking for some time now about the following problem, but can't find a solution. I have a queue with a fluctuant number of tasks. This queue might be empty at some time but might also become such full, that all workers have work to do and we have to wait until one becomes free again. I'd like to have a number of workers which process these tasks from the queue. What I've got: - the worker, which returns a deferred for further processing - a worker-pool class, which hands out the workers with an attached callback to re-queue them to the "available-pool" My problem: - the connection between the queue and the worker-pool and integrating this into the main reactor-loop My first thought was a twisted.internet.task.LoopingCall, trying to get a worker and on success giving it a task, but that's not very asynchronous, isn't it? Especially no when the queue is empty. I've no clue how to implement this. I've read something about producer-consumer in twisted, but the documentation here is very rare and I think it's not what I'm looking for. Can someone please give me a hint, if twisted offers some techniques to implement this, or point me in the right direction? Thanks Patrick

4 5

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 20 Apr '08

20 Apr '08

1 0

[Twisted-Python] Stackless Python Examples Re: InlineCallback Friendly ?
by Andrew Francis 19 Apr '08

19 Apr '08

Hi Folks: --- Andrew Francis <andrewfr_ice(a)yahoo.com> wrote: [J-P Calderone] >The only kind of dispatcher which is really hostile >towards inlineCallbacks is the kind which is >hostile towards Deferreds in general - ie, one which >requires a return value and does not support >Deferreds. [A Francis] >Once again Jean-Paul, I find your explanation super >helpful and save me time going down the wrong path. Based on Jean-Paul's explanation, here are some code snippets showing how to handle other protocols with Stackless Python and Twisted. This is a continuation of the technique I used in "Adventures" to prevent the reactor from deadlocking. Currently I am rethinking the techniques - I believe there is a way to reduce spawning tasklets - not that tasklets are all that expensive.... Here is an example with resources adapted from the Abe Fettig example on page 48 of "Twisted Network Programming Essentials." I am still getting my feet wet with resources. I intend to use this to help prototype REST support in WS-BPEL. class HomePage(resource.Resource): def doWork(self): message = """ <html> <head> </head> <body> Hello World </body> </html> """ self.request.write(message) self.request.finish() def render(self, request): self.request = request stackless.tasklet(self.doWork)() return server.NOT_DONE_YET and here is one with PyAMF .2 (I remember sketching this one out on a napkin at PyCon 2008) class EchoServer(TwistedGateway): def __init__(self): super(EchoServer, self).__init__() self.request = None return def __echo__(self, request, deferred, y): print "=>", request, deferred, y deferred.callback(y) def echo(self, request, y): print "=>", request, y deferred = defer.Deferred() stackless.tasklet(self.__echo__)(request, deferred, y) return deferred Cheers, Andrew ____________________________________________________________________________________ Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

1 0

[Twisted-Python] "glue" problem
by Andrew Francis 19 Apr '08

19 Apr '08

Hello Dr. Pythoniac: >Additional: Is there any docu about twistd ? How can I >use all its miracles rather than just dumbly guess and >try an err and try and err and Get the Abe Fettig book. There is a section on twistd. >Assume, I want to put sth into the static.Registry what does 'sth' mean? Cheers, Andrew ____________________________________________________________________________________ Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

1 0

[Twisted-Python] Using different cursor types with psycopg+twisted.enterprise.adbapi?
by Nathan 18 Apr '08

18 Apr '08

In my non-twisted scripts, I usually use psycopg2 DictCursors so that my query results come back in dicts instead of lists. curs = db.cursor(cursor_factory = psycopg2.extras.DictCursor) Now I'd like to do the same thing in Twisted. Is there a way to ask t.e.adbapi to use that DictCursor cursor factory? Or maybe there's a way to globally configure psycopg2 so that it used psycopg2.extras.DictCursor by default... /me is searching... ~ Nathan

3 5

[Twisted-Python] Twisted.web2 or Nevow?
by Simon Pickles 18 Apr '08

18 Apr '08

Hi, I want to add a simple http server to my game server to allow monitors and GMs to use a web interface to access game state information. I also want to produce an integrated game web site. My game server is using a twisted.reactor to handle TCP connections. My question is, which two similar technologies should I focus on to provide the HTTP angle? twisted.web2 or Nevow? So far I have mostly research Nevow, and seen how twistd is used to launch the application. How would Nevow then be integrated with my own app and its own reactor? Many thanks Simon -- http://www.squirtual-reality.com -------------------------------- Linux user #458601 - http://counter.li.org.

2 2

Crossing the Chasm [Twisted-Python] documentation
by Andrew Francis 18 Apr '08

18 Apr '08

Hello: >Hi out there >Let me tell you about my experience with TM so far ... >Seeing there is doc. available, I digged through it. >After all TM seems worth the effort. From what I see, >I >love TM. I've been following this thread. I have wondered if the Twisted team are familiar with read Geoffrey Moore's "Crossing the Chasm"? Cheers, Andrew ____________________________________________________________________________________ Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

1 0

[Twisted-Python] Non returning deferred result.
by Crispin Wellington 18 Apr '08

18 Apr '08

Hey everyone, Sorry about the subject line, its a difficult problem to sum up in a single line subject. Give me a little time to explain. I'm using twisted.web2. I have a generic webservice over http. It receives a file via standard POST http upload. It saves this to disk using non blocking stream setup as outlines in web2.static.FileServer. Now I don't immediately return a response, because another thread then processes this file and builds an output file. So I return a deferred from my render() function. When the output file is finished building, the deferred callback is triggered with the http.Response() object and the response is finally returned to the waiting http connection. This DOES work except for a little strange behavoir. When the deferred callback is triggered, the client doesn't see any response, the connection just stays open. Then if I connect to the webservices from anywhere, even a bad URL, I get my response suddenly back. It's as if the callback is not triggered until some TCP activity cause the main reactor to pump the pending requests or something. Is there anyway to get this deferred callback to be recognised and processed immediately? I could do a kind of hack, where the deferred callback is called, and then a quick connection is made to the webservice to "pump" the message through, but it doesn't seem very 'right' to me. Heres some summary of the approach incase the devil is in the details... My Resource class: ================== class UploadFile(resource.PostableResource): # 1 gig is max file upload size maxSize=1 * 1024 * 1024 * 1024 def render(self, ctx): """Create the job for this upload and return the relevent deferred""" request = iweb.IRequest(ctx) job=Job() deferred = job.SubmitInput(request.files) return deferred My deferred callback: ===================== deferred.callback( http.Response(responsecode.OK, {'content-type': http_headers.MimeType('text', 'html')}, "Success!" ) ) My TopLevel: ============ class Toplevel(resource.PostableResource): addSlash = True def render(self, ctx): return http.Response(responsecode.OK, {'content-type': http_headers.MimeType('text', 'html')}, """ <html> <form action="http://localhost:8080/uploadfile" enctype="multipart/form-data" method="post"> filename: <input type="file" name="filename"> filename2: <input type="file" name="filename2"> <input type="submit" value="submit"> </form> </html> """) child_upload = UploadFile(jobqueue) My server startup: ================== # Create the resource we will be serving test = Toplevel() # Setup default common access logging res = log.LogWrapperResource(test) log.DefaultCommonAccessLoggingObserver().start() # Create the site and application objects site = server.Site(res) application = service.Application("demo") # Serve it via standard HTTP on port 8080 s = strports.service('tcp:8080', channel.HTTPFactory(site)) s.setServiceParent(application) Im running the application as: twistd -noy server.py Twisted v8.0.1+, actually, the lastest SVN head. Any one with more knowledge of twisted's internals have any ideas on what is happening? Does anyone have any good ideas on how to make the deferred callback trigger an immediate response (without opening a new TCP connection to kick it into gear)? Thanks Regards Crispin Wellington <cwellington(a)ccg.murdoch.edu.au>

2 5