Mailman 3 January 2007 - Twisted

[Twisted-Python] xml
by Lee Connell 28 Jan '07

28 Jan '07

Hey everyone, I have been looking all night and I can’t find any examples of using simple xml to write my own xml protocol for a chat server I intend to build. All I want to do is generate XML from commands and send it across the socket and receive it on the other end and unpack it. Any recommendations on what to use and any tutorials or examples on how to do it. I don’t know a whole lot about XML programming, just it’s basic structure. Thanks! -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.5.432 / Virus Database: 268.17.11/652 - Release Date: 1/25/2007 3:32 PM

5 9

[Twisted-Python] Foolscap-0.0.7 (aka newpb) released
by Brian Warner 27 Jan '07

27 Jan '07

I've made a new release of Foolscap, 0.0.7, which fixes two irritating bugs: in previous versions you could not connect to the same PBURL multiple times, nor could you connect to yourself. The former would return None where it was supposed to return a RemoteReference, the latter could cause a negotiation timeout and failure. Both are fixed now. There are no incompatible protocol changes in this release: 0.0.6 clients will speak to 0.0.7 clients. There likely to be incompatible changes in the future though: the protocol is not yet stabilized. Foolscap is a next-generation RPC protocol for Twisted, under development as "newpb" for the last several years. It is intended to eventually replace PB, but it is not yet ready for production use. My hope is that Foolscap will make it easier for python/twisted applications to use tools and concepts from the object-capability world, in particular from E and the Web-Calculus. Some useful URLs: Foolscap home page: http://twistedmatrix.com/trac/wiki/FoolsCap Download the release: http://twistedmatrix.com/~warner/Foolscap/ Browse the source: http://twistedmatrix.com/trac/browser/sandbox/warner/Foolscap/trunk Read some docs: http://twistedmatrix.com/~warner/Foolscap/using-pb.html E: http://www.erights.org/ Web-Calculus: http://www.waterken.com/dev/Web/ Also I'll probably organize a Foolscap BOF at PyCon this year. share and enjoy, -Brian

2 2

[Twisted-Python] task.LoopingCall: a problem if change the system time
by Roberto Gomez 26 Jan '07

26 Jan '07

Hi everyone, I wrote a simple test trying to find why my app freeze when change the system time: ################# from twisted.internet import reactor, task import time def callback(): print time.ctime(time.time()) t = task.LoopingCall(callback) t.start(2.0) reactor.run() ################# If run this and change the system time, let's say a day forward, everything work fine, but if change the system time backward, the program freeze until you change forward again. This is specially problematic for a computer using ntp, updating its clock time to time. It seems that there is a problem in LoopingCall._reschedule that don't play well when "fromNow = self.starttime - self._seconds()" is negative, and seems to be platform independent: W2k Pro python 2.4.4 twisted 2.4.0 Wxp Pro python 2.4.4 twisted 2.4.0 Ubuntu Linux 5.1 python 2.4.2 twisted 2.0.1-4 I'm not a twisted expert, but and extensive user and fan, and I don´t know how to workaround this. Any clue? TIA ____________________________________________________________________________________ Do you Yahoo!? Everyone is raving about the all-new Yahoo! Mail beta. http://new.mail.yahoo.com

2 2

[Twisted-Python] Losing Deferreds : Re: More Questions about Custom Reactors
by Andrew Francis 25 Jan '07

25 Jan '07

Message: 3 Date: Mon, 22 Jan 2007 00:32:16 +0000 From: Phil Mayers <p.mayers(a)imperial.ac.uk> Subject: Re: [Twisted-Python] More Questions about Custom Reactors To: Twisted general discussion <twisted-python(a)twistedmatrix.com> Message-ID: <45B40610.7020607(a)imperial.ac.uk> Content-Type: text/plain; charset=ISO-8859-1; format=flowed I am sorry that I am tardy in responding. I was trying out your suggestions. Right now, my main question is under which conditions do deferred fail to fire? How do I know that the reactor is still running? Would Manhole help? (Unfortunately I haven't been able to get gtk to work Manhole with Twisted 2.4) >Perhaps you could explain what you're actually trying >to achieve, rather than the solution you have decided >on? An indeed, why you want to use stackless at all - >Twisted is predicated around the assumption of a >stackful i.e. single-threaded Python. I understand your questioning. The main reason I am using Stackless Python is for its process execution state pickling abilities. A second reason is that I find Stackless's communication model very handy for modeling the type of thread synchronisation I wish to do. I know that Twisted assumes a stackful Python and it is an uphill battle. However the gains from getting Twisted to work with Stackless are so great, I think it is worthwhile. For the most part, I have succeeded in having Stackless Python work with Twisted. If you read the Stackless mailing list, I have posted examples. Soon, I will post on the Wiki. My problems stem from situations where I have two or more concurrent tasklets (Stackless threads) that do the following) tasklet[0] - waiting for a Twisted Request Handler tasklet[1] - making a client.getPage call tasklet[2] - making a client.getPage call tasklet[3] - some other operation tasklet[n] - some other operation under these conditions, if tasklet[0] is executed first, the other tasklets will block indefinitely. My problems occur from trying to get around this. Originally I tried taskLoop calling stackless.schedule() to give other tasklets a chance to execute but this would cause reactor problems. Hence the current approach of looking at getting Twisted to run in its own thread. >Alternatively you might be wanting long-running jobs >to be interleaved with other code transparently but >without using real threads? I am not sure how to do this. >If you're proposing using queues to communicate >between a Twisted thread running the reactor and a >stackless thread running your micro-threads, >I don't see why you need to modify Twisted at all. I am a newbie to Twisted. Based on what I am reading, I current assumption is that I would need to >You can just run the stackless microthreads from one >"real" thread (started via callInThread), pass jobs >from Twisted to the "real" thread and probably >a stackless uthread, and pass the replies back to the >twisted thread with a callFromThread. >So, can you not just do this?: Philip, first the good news : thanks a million! This is the exactly the type of example I wanted. It works. The bad news - under certain conditions I am losing deferreds. I think there are deadlock issues but that is not Twisted's fault per se. Unfortunately debuggers work poorly with Stackless so it is difficult for me to know what is happening. I tried telneting but my server is not responding - so I guess that implies that Twisted is still listening but the reactor is not working. Again, any suggestions would be more than welcome. Cheers, Andrew ____________________________________________________________________________________ Bored stiff? Loosen up... Download and play hundreds of games for free on Yahoo! Games. http://games.yahoo.com/games/front

1 0

[Twisted-Python] Pycon Twisted Sprint
by Eric Mangold 25 Jan '07

25 Jan '07

Does anyone have any ideas for a Twisted Sprint? I will probably be staying at least through Monday. Special-rate-registration ends in one week on February 1st. So if there is going to be any kind of organized activity we ought to figure it out in the next few days. Please raise your hand if you want to hack on stuff after the 'con and, assuming enough peer interest, indicate how many days you would like to stay. Personally if there was enough interest I would stay through Tuesday. -- Eric Mangold Twisted/Win32 Co-Maintainer

2 1

[Twisted-Python] Is it a good/recommended practice to pass Instance Method into reactor.callFromThread()/callInThread()?
by jian wu 24 Jan '07

24 Jan '07

Hi Twisted Guru, I'm new to Twisted, I am just experimenting Twisted 2.5 with Python 2.5. It seems that I can pass either a python function or a python object instance method to reactor.callInThread() for blocking operation and reactor.callFromThread() in main event loop. I just want to see whether it is a good/recommended practice to pass Instance Method into reactor.callFromThread()/callInThread(), which implies that the application level context information stored in the python object instance is available when the method is invoked, so I don't have to pass all the context info as parameters. Do I miss anything in Twisted? Any suggestion or tip is really appreciated. Thanks a lot for your help! Jian

2 2

Re: [Twisted-Python] Hanging Deferreds in PB Paging code
by Ed Suominen 24 Jan '07

24 Jan '07

From: "Brian Granger" <ellisonbg.net(a)gmail.com> > Thanks for the code snippet. I had thought about trying to implement > my own, less complicated solution for this issue but hadn't done > anything yet. I will have a look at your code to see if I can adapt > it. I am not dealing with files, but most of the ideas are the same. > > Also, the task queue stuff looks interesting. It is actually pretty > similar to what we have implemented in IPython: > > http://ipython.scipy.org/moin/Parallel_Computing > > We have abstracted things at a slightly lower level to allow various > types of higher level constructs (such as task farming) to be built > easily. We haven't build the task farming interface fully yet, but > that is on our list of todos soon. It might be nice to use some of > your work when we implement the task farming interface, but > unfortunately, IPython is BSD, so your GPL code won't help us much. Well, as it says on the foss.eepatents.com site, "If you want to use this software under different terms ... feel free to contact me about it. Maybe we can work something out, maybe not." :-) I'm interested in the idea of some of the work getting adapted for IPython (a great project!), even if the adaptation winds up being licensed differently. Also, I have already done quite a bit of task farming in another as-yet-private package of mine, "AsynCluster" (is what it sounds like), which I'm planning to incorporate into Twisted-Goodies. Collaborating with someone capable like yourself on this would be great for both of us, I would think. So, let's get together on this off-list soon! Best regards, Ed Suominen Registered Patent Agent & Independent Inventor (cathedral) Open-Source Software Author (bazaar)

2 1

[Twisted-Python] Could anyone send me a copy?
by Arbow 24 Jan '07

24 Jan '07

Hi anyone: I could't download the v2.5 release for the bad network condition (1k~2k, always disconnected), could anyone send me a copy to my mailbox? I need these follows: Twisted 2.5.0 tarball Twisted 2.5 Win32 installer for Python 2.4 Twisted 2.5 Win32 installer for Python 2.5 Thanks Arbow

2 2

Re: [Twisted-Python] Twisted server is 5 times SLOWER on Solaris than Linux?
by glyph＠divmod.com 22 Jan '07

22 Jan '07

On 01:59 am, jarrod(a)vertigrated.com wrote: >We have LOTS and LOTS of test harness code and profiling code to pinpoint >where bottlenecks are. We are going to have process a couple of terabytes a >day thru this system. Latency thru the system is a high priority because of >what kind of system it is. If you do manage to keep Twisted in the loop on this project, please share what strategy you used. It is unfortunately rare that Twisted is itself a really performance-critical component of any system (especially a highly parallel system), so information about optimization is hard to find.

7 11

[Twisted-Python] More Questions about Custom Reactors
by Andrew Francis 22 Jan '07

22 Jan '07

Hello Colleagues : I am trying to write a reactor for Stackless Python. I posted previously here and in the Stackless mailing list but I am still stuck. My problems are in the following areas : 1. How does the custom reactor drive Twisted? 2. How does self.interleave() work? 3. How to use deferreds if Twisted is running in a seperate thread. Is callFromThread() used? 4. Some custom reactor implementations inherit and some reactors simply install say threadedselectorreactor? Which technique is recommended? For now, I am operating under the assumption that Stackless Python and Twisted ought to be executed in seperate threads. This is because if Twisted blocks then Stackless blocks. Since I am taking this approach, I am assuming I will have to use threadedselectorreactor. I want my Stackless Python application to do two things : 1. Process HTTP based SOAP server requests 2. From the Stackless thread, make client.getPage() calls and use a deferred to process the result. To promote decoupling, I want Twisted and Stackless to communicate via queues. So my code looks as follows - def processResponse(self, httpServer): reply = self.myQueue.get() httpServer.write(reply.toxml().encode("utf-8")) httpServer.finish() return example 2. postRequest = client.getPage(self.address, method = 'POST', headers = self.headers, postdata = self.body)postRequest.addCallback(self.__handleResponse__).addErrback(self.__handleError__) def __handleResponse__(self, pageData): self.queue.put(pageData) I have looked at the reactors included in the twisted.internet. I also looked at Blockingdemo.py. I find BlockingDemo.py confusing (i.e., where is poll() called? It does not seem to do anything.) Question about interleave(): """ Taken from threadedselectorreactor In order for Twisted to do its work in the main thread (the thread that interleave is called from), a waker function is necessary. The waker function will be called from a "background" thread with one argument: func. The waker function's purpose is to call func() from the main thread. Many GUI toolkits ship with appropriate waker functions. """ Using wxReactor as a basis, if the following is my custom Stackless reactor's run method def run(self): self.startRunning(...) self.interleave(some_waker_function) self.stacklessApplication.Loop() 1. How does this drive Twisted (in wxReactor there seems to be a second loop that sleeps every millisecond)? . 2. What should the "some_waker_function" be doing? Is the interleave() method always necessary? 3. What is the background thread? Twisted? A workerThread in threadedselectorreactor? 4. To use client.getPage() and callbacks from Stackless to a Twisted reactor running in a thread, must I use callFromThread? Will the Stackless thread block? Once again, any advice would be appreciated. I will happily summarise on the Stackless Wiki. Sorry for the long message. Thank you, Andrew ____________________________________________________________________________________ Do you Yahoo!? Everyone is raving about the all-new Yahoo! Mail beta. http://new.mail.yahoo.com

2 1