Mailman 3 April 2008 - Twisted

[Twisted-Python] win32 reactors
by James Mansion April 7, 2008

April 7, 2008

I'm just starting to evaluate Twisted, so apologies if these questions seem naive. It seems to me that the Twisted model for 'reactor' is actually not what you'd expect from the name (at least, if you have a background in other reactor implementations including ACE, basic select etc and Java NIO) since it a) handles queueing outgoing data for you and b) presents incoming data after completion of the IO So its a model that maps well to an AIO implementation so long as inputs from non-blocking sources complete as soon as "some" data is ready, rather than buffer full. This high level approachh should work well on Windows - which is really pleasing. (Albeit there may be buffer starvation issues on all platforms I guess, but that's probably scaled beyond Python ambitions) A quick look at the open bug list shows a lot of noise on the Win32 side though. Is there any reason to have any framework that calls the Win32 select function? And why is the event reactor broken at 64 handles? From what I can see, that's not how to use WaitForMultipleEvents - rather one should regard it as a way to hash completion events from standard overlapped IO requests against 61 of the handles and keep a list of the IOs in progress against each handle. Scales admirably, unlike any attempt to treat the handle array like a select fd_set. Shouldn't broken Win32 implementations be 'retired'? Also - is there any way to determine how much data is queued for write against an ITransport? It appears not, though I would have thought it useful (and, a way to get an event callback when the queued amount falls below a specified threshold) - then in cases where data can be generated on demand we can limit the outgoing buffer usage. James

9 16

Re: [Twisted-Python] Stackless/Twisted integration again
by Andrew Francis April 6, 2008

April 6, 2008

Hi Simon and Colleages: >I note he recommends running a twisted server as a >tasklet. I do this so both the Twisted reactor and the Stackless scheduler can run without blocking each other.... The Twisted Reactor and the Stackless Schedule start their respective systems. Both the Stackless and Twisted run() method assume that the application has finished once the run() method returns. Here is simple example import stackless from twisted.internet import reactor def hello(): print "hello mom" stackless.tasklet(hello)() reactor.run() stackless.run() This programme will hang. Why? The Twisted reactor is running in the main tasklet and returns only after it has been stopped (which is never in this example). Only after the reactor has finished, will the stackless scheduler start. Of course, this is not the behaviour we want. >factory = pb.PBClientFactory >reactor.connectTCP('127.0.0.1', 1234, Factory) ># Load other tasklets here >stackless.tasklet(reactor.iterate)(1) I have never used iterate(). I am having a hard time finding an example of iterate() in action. Perhaps I am wrong but I would assume iterate() is used in conjunction with something like reactor.interleave() so that something like a GUI can function properly. >From my understanding, your example would produce something because in the iteration followed by a one second delay, the client can actually complete a call. However this seems accidental. >factory = pb.PBClientFactory >reactor.connectTCP('127.0.0.1', 1234, Factory) ># Load other tasklets here >while 1: > stackless.tasklet(reactor.iterate)(1) > stackless.schedule() I don't understand iterate() well enough but I think this will accidentally work sometimes. I don't have time to run your programme but off hand, this is what I think is happening: 1) Reactor tasklet is created. Since other tasklets are ahead of it, it is not running. 2) stackless.schedule() kicks off the scheduling, yielding the main tasklet and allowing the next schedulable tasklet to run 3) Eventually the Twisted reactor runs. It does stuff. You get results. However I don't believe it will yield after that. >I want to make the reactor a tasklet so it is included >in the scheduler. Simon, stackless.tasklet(reactor.run)() will make the Twisted reactor into a tasklet. However you need a way to make it yield occasionally. That is why I use task.LoopingCall(stackless.schedule). One nice feature of this approach is that I do not have to alter the Reactor. I remember playing with PB a while ago. Here is the post: http://twistedmatrix.com/pipermail/twisted-python/2007-September/016000.html I am enclosing some sample code so you can play with it. The code may be a bit clunky because I was still learning. Heck I am still learning :-) Cheers, Andrew P.S - You may want to take out the 61 second sleep in the PBServer. And it was neat to actually meet Bruce Eckel at PyCon. ____________________________________________________________________________________ You rock. That's why Blockbuster's offering you one month of Blockbuster Total Access, No Cost. http://tc.deals.yahoo.com/tc/blockbuster/text5.com

1 0

[Twisted-Python] Deferreds and Tasklets
by Simon Pickles April 6, 2008

April 6, 2008

So I am still fiddling with interesting ways to combine stackless and twisted. Its fun, but I am probably naively hacking! (I always seem to want to use the deprecated code!- probably says a lot about me?) I'm using Perspective Broker to do remote calls to an app running Twisted-Stackless. PB is positively brilliant, btw. So say the a remote caller wants to get data from my app. It calls remote_GetData() in my PB app, which uses the actor model(each object has a channel.receive() running in its own tasklet). remote_GetData() has to send a message thru a stackless.channel() to get the required data, then it must stackless.schedule() Now it waits for the result to arrive at its own channel.receive(), which then needs to be passed back as the return value of remote_GetData(). Is this possible? Perhaps like this: from twisted.spread import pb from twisted.internet import reactor import stackless dbRx = stackless.channel() pbRx = stackless.channel() class DataSource: def Listen(self): while 1: msg = dbrx.receive() # a channel will arrive here msg.send(42) stackless.schedule() def __init__(self): stackless.tasklet(self.Listen)() class pbBroker(pb.Referenceable): def remote_GetData(self): dbRx.send(pbRx) result = pbRx.receive() # return 42 to remote caller thru pb return result This isn't ideal since I don't want pbBroker to block during the receive. I'd like it to closer resemble the Listen loop of the first class, so other objects can talk to pbBroker while it is waiting. Maybe like this: class pbBroker(pb.Referenceable): def Listen(self): while 1: msg = self.rx.receive() # do stuff with msg from other local objects stackless.schedule() def __init__(self): self.rx = stackless.channel() stackless.tasklet(self.Listen)() def remote_GetData(self): ch = stackless.channel() dbRx.send(ch) # can block since its a dedicated channel result = ch.receive() # return 42 to remote caller thru pb return result I guess the same problem arise when remote_GetData() has to use a deferred too. from twisted.enterprise import adbapi dbPool = adbapi.ConnectionPool("MySQLdb", db="db", user="user", passwd="pw") class pbBroker(pb.Referenceable): def remote_GetDeferredData(self): d = dbPool.runInteraction(DoQuery, "SELECT * FROM users") # DoQuery just wraps a query with try/except d.addCallback(self.GetDeferredData_Success) def GetDeferredData_Success(self, result): # return to remote caller thru pb return result ## WRONG How can I return a value for a remote_* call when I only get that value thru another callback? Sorry its a long and confusing one! Simon -- Linux user #458601 - http://counter.li.org. -- Linux user #458601 - http://counter.li.org.

2 1

[Twisted-Python] Stackless/Twisted integration again
by Simon Pickles April 6, 2008

April 6, 2008

I was interested in Andrews Paper on this subject. I note he recommends running a twisted server as a tasklet. Is there a problem doing this? factory = pb.PBClientFactory reactor.connectTCP('127.0.0.1', 1234, Factory) # Load other tasklets here stackless.tasklet(reactor.iterate)(1) or should I perhaps do this? factory = pb.PBClientFactory reactor.connectTCP('127.0.0.1', 1234, Factory) # Load other tasklets here while 1: stackless.tasklet(reactor.iterate)(1) stackless.schedule() What does the delay in reactor.iterate(delay=0) do? Is there a reactor.run_once() call? I want to make the reactor a tasklet so it is included in the scheduler. Seems to be working but have I opened a can of worms? Thanks Simon -- Linux user #458601 - http://counter.li.org.

2 3

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com April 6, 2008

April 6, 2008

1 0

[Twisted-Python] Re: Stackless Python/Twisted example
by Andrew Francis April 5, 2008

April 5, 2008

Hi Don and Colleagues: >Thanks much for the example. At a quick scan, it >looks like it'll take some digesting to get my head >around. I'll give it a closer look when I >get the chance. Don, you need to read the code along side the slides. I will probably be giving a revisted version of the talk in early May. I will annotate the example in the week to follow. However I need to change the code somewhat so it is a wee less confusing to a newbie and gets rid of a design flaw concerning daemonisation. With the current programme, if you hit the processor fast enough, you will get back 404s. These really should be 500s. This is not a problem with the technique but with my simple minded implementation of the WS-BPEL specification that warns of race conditions associated with certain activities. Although fine for my purposes, I should conduct two sets of tests 1) More feature torture tests where I test non-trivial message exchange patterns. 2) Performance tests. Test against - Apache ODE - A Stackless /Twisted solution that does not spawn a server instance in a tasklet - A pure Twisted solution - A Twisted solution that uses inlineCallbacks. Cheers, Andrew ____________________________________________________________________________________ You rock. That's why Blockbuster's offering you one month of Blockbuster Total Access, No Cost. http://tc.deals.yahoo.com/tc/blockbuster/text5.com

1 0

[Twisted-Python] Source specific multicasting (SSM) example?
by Rodney Lott April 5, 2008

April 5, 2008

Hi, there. I was wondering if there is any information or examples on how to use Twisted to perform a SSM operation. I am new to Twisted and to Python. In C, I believe one would use the setsocketopt() to specify the joining to a multicast group (G) to a specific source (S): setsockopt(sock,..., IP_ADD_SOURCE_MEMBERSHIP, (S, G, iface), ...) I believe that Python itself has analogous socket calls and could do this using a UDP socket. I read the Multicast UDP example on Twisted here: http://stderr.org/doc/twisted-doc/howto/udp.html#auto3 This shows how to do a normal multicast UDP situation with Twisted (i.e. a process will receive UDP traffic from ALL sources registered to the IGMP group). So, if there were UDP three sources (e.g. A, B, and C) registered to the IGMP group, I would like to find a way for my client to only receive data from source B. I believe that I can get information using a SDP/SAP using a simple UDP server listening on the appropriate port. So, the only thing that I am missing is how to register to a group for a specific source. I am not sure if the joinGroup() method will allow for this. Thanks. -Rodney Lott

2 3

[Twisted-Python] Packaging a Twisted app as an Egg
by Matthew Glubb April 5, 2008

April 5, 2008

Hi All, Apologies if this has been covered before. I am looking to package a twisted application that relies on twistd as an egg but I am concerned as to how twistd will be aware of the my package. How do I make twistd aware of my application if it is packaged as an egg? Is this possible? Do I just create my own stub to twistd that takes care of ensuring that twisted/plugins/myapp.py is in Python's path? Thanks a lot, Matt Matthew Glubb Technical Partner email: matthew.glubb(a)madebykite.com phone: 44 (0) 7715 754017 skype: mglubb Kite http://madebykite.com -- GPG: 96FF DE0E 0B7B 37F0 7F8D C54C E285 3D8F 5625 9244

2 2

[Twisted-Python] Other ways to integrate foreign event loops?
by Nathan April 4, 2008

April 4, 2008

Is there a way to do the opposite of task.coiterate? I'm using pyglet for my UI, which was working great until I upgraded from pyglet 1.0 to 1.1. Now for some reason, the pyglet performance tanked. Since I'm only using twisted to send AMP messages very rarely (on the order of one every few seconds), it seems like I could try to work around the problem by giving pyglet back control of the main loop and occasionally giving twisted some time to breath (occasionally like every 50ms or something). Can that be done? Here's what I'm doing _now_ that technically works, but pyglet 1.1 is really choking on: > class MyWindow(pyglet.window.Window): > def run(self): > while not self.has_exit: > do_some_stuff_like_dispatching_updating_and_drawing() > yield 1 > > my_window = MyWindow() > task.coiterate(my_window.run()) > reactor.run() By the way, the preferred method for the runloop on pyglet 1.1 is to use a builtin one like so: > window = pyglet.window.Window() > > @window.event > def on_draw(): > window.clear() > do_drawing_stuff() > > pyglet.app.run() So if twisted can stand giving up it's run loop, I could periodically call it from that "on_draw()" method. ~ Nathan

6 9

Stackless Python/Twisted example Re: [Twisted-Python] Advice sought on application evolution
by Andrew Francis April 3, 2008

April 3, 2008

Hi Don and Colleagues: >I'll be very interested to learn more about the >melding of the two paradigms. ... >One question: I had the impression that Stackless is >mostly an academic exercise, sort of a proof of >concept. Is this wrong? How stable, portable, etc. is Stackless at this point? Is it a reasonable choice >for a commercial application? Sorry for the wait - I have been very busy. I will probably update my blog on the weekend. Meanwhile, I have enclosed an example of a Toy WS-BPEL processor prototype that more or less implements the I/O parts of WS-BPEL specification: invoke, reply, receive, wait, event handlers, and daemons! In the process, the programme shows all the integration techniques in action. The main thing I want readers to take away is how *little* Stackless Python and Twisted are needed to implement something complex. To me, this is what Python is all about. In your case Don, I try to show how layering helps model business logic. In the example, think of the Process and AlarmProcess belonging to the business logic layer. I have tried my best to strip away the WS-BPEL specific parts to show how I integrate Stackless with Twisted. Still the architecture is influenced by my need to prototype. I am still experimenting. That said, this little prototype starts up 1001 tasklets. If a web client does a HTTP GET on http://localhost:8080/[0-999], the tasklet will wake up, call a web server (in my case Apache), and return a result. Another tasklets simply ticks. Give me a while, and I will show another example with process serialization (at Pycon 2008 - I demonstrated process serialization to a few folks after my talk - hint Twisted does not like to be pickled). Unfortunately I have not had time to set up stress tests. I really don't have access to large computers. That said, the integration techniques reflect my thinking about November/December 2007. I am eager to find the time to try out Christopher Armstrong's coroutine package. If you have any questions, please don't hesitate to ask. Cheers, Andrew ____________________________________________________________________________________ You rock. That's why Blockbuster's offering you one month of Blockbuster Total Access, No Cost. http://tc.deals.yahoo.com/tc/blockbuster/text5.com

2 1