Mailman 3 January 2007 - Twisted

[Twisted-Python] Re: Advice on Writing a Custom Twisted Stackless Reactor
by Andrew Francis 12 Jan '07

12 Jan '07

Hello Jean-Paul: Once again, thanks for your response. >Twisted will block while there are no events to >process. As soon as there is an event, though, it will >wake up. So, if your tasklets are I/O bound, as soon >as there is a network event, Twisted will tell you >about it and you can schedule the next one to run. I know this. Since Twisted and Stackless are using the same OS thread, this will block Stackless. Stackless will block under the following condition - some notation: S - tasklet waiting for a server call (waiting for a Twisted request Handler ) C - tasklet waiting for a client call (deferred request) | - running in parallel if you have a sequence of the form (S | C[0] | C[1] | c[n]) and S executes first, there will be no progress on the client tasklets. I tried adding a timer but had limited success. (By the way, when I first mentioned this in the Stackless mailing list, Bob Ipolito pointed this out to me) AF>I thought I got around this problem by AF>using task.LoopingCall/() that would issue a AF>stackless.schedule() at regular intervals. However AF>this approach turns out to be problematic. >What problems did you face? It certainly doesn't >sound ideal, but it does sound like it should give you >something which does more or less work. In some tests I would get a Twisted error : why = getAttr(selectable, method)() others times my processor would simply hang because the deferred function was not called. from my logging, the common demoninator was that the task.LoopingCall stopped. >Better would be to associate network and >timing events with specific tasklets and wake them up >when the reactor fires those events. I do this. >Most Twisted APIs aren't threadsafe, so most likely >what is happening here is that you are using a Twisted >API from the wrong thread and it just isn't working >right, which often manifests as an event which never >fires. What is the wrong thread? How do I correct this? >However, I still suspect bringing in a native thread >is not necessary to accomplish your goals. Maybe you are right. So far my experiences are pointing in the opposite direction. Also I have read posts involving integrating Chandler and IronPython with Twisted and they are running Twisted in a separate thread. That said, these are my deferred calls : postRequest = client.getPage(self.address, method = 'POST', headers = self.headers, postdata = self.body) postRequest.addCallback(self.__handleResponse__).addErrback(self.__handleError__) How should these be done from a thread? >Roughly, yes. blockingdemo.py in the examples >directory gives a fairly short, self-contained example >of integrating another loop using TSR. If you haven't >seen it yet, it'll probably help or help verify your >understanding. If you have, it might be a good basis >to guide further questions. I looked at blockingdemo.py. Blockingdemo is structured different from the reactor examples in the Twisted.internet module. Most of the example reactors i have seen inherit from threadselectreactor and override the run() method. What does TwistedManager.poll() do? The method is defined but called nowhere. I commented out poll() and the programme did the same thing. That said, I have been looking at the wxreactor.py. My questions are: 1. With the customer reactor, it would be Stackless that drives Twisted, not the other way around? 2. How exactly does interleave work? Some reactors don't use it. My Stackless thread and Twisted talk through stackless channels and a variation of a stackless channel, the uthread Queue. My strategy is as follows: 1. Inherit from threadselectreactor and override run() 2. in run(), my interleave calls __hook__() 3. In my Stackless Application, write a method __hook__. __hook__ looks like wxCallAfter in wxPython : def __hook__(self, function, *args, **kw): return function(*args, **kw) 3. Since stackless.schedule() is often called to yield control, wrap the stackless.schedule() """ call the underlying Stackless scheduler """ def __schedule__(self): # this is where I am a little fuzzy but wxCallAfter # has to be called somehow self.__hook__(?) stackless.schedule() 4. In the StacklessReactor.run(), the main loop that is called will look like: def MainLoop(self): while(stackless.getruncount() > 1): self.__schedule__() Again, I really appreciate insights. I really want to get Stackless Python to properly work with Twisted. Cheers, Andrew ____________________________________________________________________________________ 8:00? 8:25? 8:40? Find a flick in no time with the Yahoo! Search movie showtime shortcut. http://tools.search.yahoo.com/shortcuts/#news

1 0

Re: [Twisted-Python] Ticket Hall of Shame - Current Loser: Brian Warner
by glyph＠divmod.com 12 Jan '07

12 Jan '07

On 08:00 pm, exarkun(a)divmod.com wrote: >On Fri, 12 Jan 2007 10:54:10 -0800, Scott Lamb <slamb(a)slamb.org> wrote: >>(1) create a test that likely fails by running many, many times. It'd bog >>down the test suite. >>(2) create a test that might fail sometime, maybe. >>(3) add instrumentation to the main code solely for the purpose of the >>test (break twisted's kneecap before the race). >>(4) just say "clearly[*] if event A happens between lines X and Y, then it >>is handled improperly." >> >>I prefer #4. > >I think this actually varies from case to case. For a problem where all >of the relevant critical points cannot be hooked or mocked or stubbed or >what have you, then a combination of #2 and #4 seems correct to me. You >should be able to demonstrate that a real problem does exist by adding a >test which will fail eventually when run repeatedly (eg, using trial -u) >and which runs for orders of magnitude longer without failing once a fix >has been applied. Race conditions are tricky, but in every case it is worth _trying_ to mock/stub/hook the code as it already is to demonstrate it deterministically. It isn't always possible with a reasonable amount of effort, but very often it is actually quite easy. As you do it more, it gets easier to figure out the trickier cases. In any case, writing a test (which runs quickly) to be used with trial -u is a pretty low bar. There are some extremely rare cases (mostly involving threading infrastructure) where those tests take a long time to fail, but even there it is usually possible to make them *succeed* reliably. >Always feel free to file the ticket before figuring any of this out, by >the way. It is still possible that it will strike someone else's fancy >and the rest of the work will be taken care of for you. :) And of course this is always true.

1 0

[Twisted-Python] Reminder: Early Bird Registration for PyCon Ending Soon
by Jeff Rush 12 Jan '07

12 Jan '07

Greetings. I'm the co-chair for an upcoming volunteer-run conference in the Dallas (Addison) area and would like to get an announcement into the hands of your members in some manner acceptable to your community. The event is the fifth international Python Conference, being held Feb 23-25, 2007 at the Marriott-Quorum in Addison, with early-bird registration ending Jan 15. The conference draws approximately 400-500 attendees from diverse backgrounds such as scientists from national and medical labs, college/K-12 educators, web engineers and the myriad of IT developers and programming hobbyists. Those new to the Python language are welcome, and we're offering a half-day "Python 101" tutorial on the day before the conference, Thursday Feb 22 to help you get up to speed and better enjoy the rest of the conference. While unfortunately no talks specifically on Twisted were submitted this year, some of the presentations that Twisted folk might find interesting are: - Web Frameworks Panel - WSGI: An Introduction - soaplib: an easy-to-use python soap library - Distributing your Project with Python Eggs - Easy Creation of Interactive Tutorials - How I Wrote a Python App and Got $5 Million - Scaling Python for High-Load Web Sites - Software Development with Trac Being run by the Python community as a non-profit event, the conference strives to be inexpensive, with registration being only $260 (or $195 if you register prior to Jan 15th), with a further discount for students. On the day before the conference we are running a full day of classroom tutorials (extra charge per class) and then after the conference is a free four-days of sprints, which are informal gatherings of programmers to work together in coding on various projects. Sprints are excellent opportunities to do agile pair-programming side-by-side with experienced programmers and make new friends. Other activities are lightning talks, which are 5-minute presentations to show off a cool technology or spread the word about a project, open space talks, which are spontaneous gatherings around a topic and, new this year, a Python Lab where experienced and novice programmers will work together to solve challenging problems and then present their solutions. The conference is also running four keynote talks by leaders in the programming field, with a special focus on education this year: "The Power of Dangerous Ideas: Python and One Laptop per Child" by Ivan Krstic, senior member of the One Laptop per Child project "Premise: eLearning does not Belong in Public Schools" by Adele Goldberg, of SmallTalk fame "Python 3000" by Guido van Rossum, creator of Python "The Importance of Programming Literacy" by Robert M. "r0ml" Lefkowitz, a frequent speaker at O'Reilly conferences I believe your members will find the conference educational and enjoyable. More information about the conference along with the full schedule of presentations with abstracts, is available online: http://us.pycon.org/ Thanks for any help you can give in spreading the word, Jeff Rush Co-Chair PyCon 2007

2 1

Re: [Twisted-Python] Advice on Writing a Custom Twisted Stackless Reactor
by Andrew Francis 11 Jan '07

11 Jan '07

Date: Wed, 10 Jan 2007 18:32:43 -0500 From: Jean-Paul Calderone <exarkun(a)divmod.com> Subject: To: Twisted general discussion <twisted-python(a)twistedmatrix.com> Message-ID: <20070110233243.11447.1076809328.divmod.quotient.12503@ohm> Content-Type: text/plain; format=flowed >Okay. I see that you're writing stackless integration >using TSR. I'm not sure what the point of this is. >Can you explain the functionality that's gained by >doing this, rather than using something like what's in >Chris Armstrong's sandbox in threadless.py? Thank you Jean-Paul for your response. My goal is to write a custom Twisted reactor for Stackless Python. Although I have had success writing programmes that use deferred (for client HTTP calls) and HTTP request handlers, I have had problems doing both simultaneously in one programme. Some context : I am prototyping WS-BPEL (Web services Business Process Execution Language) processors. In the WS-BPEL language, it is possible for some process's "threads" to act as a server and other "threads" to act as a client simultaneously. The problem that I have found is that when Twisted blocks waiting for a server request, Stackless will also block, hence no progress on other tasklets that could execute. I thought I got around this problem by using task.LoopingCall/() that would issue a stackless.schedule() at regular intervals. However this approach turns out to be problematic. I thought I could solve the problem by placing Twisted in a thread but no. The main problem with placing Twisted in a thread is that I seem to lose the deferred (and I don't believe I am getting deadlock). >Can you explain the functionality that's gained by >doing this, 1. I solve my problem in a clean fashion. 2. If I write a custom Stackless Reactor, then I have a solution that conforms to the way Twisted does stuff. I think would make it easier for others to use my stuff. >rather than using something like what's in Chris >Armstrong's sandbox in threadless.py? Chris's approach looks like the approach I currently use. >Threaded select reactor actually isn't the same kind >of reactor as the rest of these, since it isn't >intended to be used via the normal reactor API (ie, >run). Right now I am using Twisted 2.4 with Stackless 2.4.3. For now, I would like to build on top of ThreadedSelectReactor. Back to my original question, given my previous example's "main loop", do I simply use interleave() to get my processor to call Twisted? Cheers, Andrew ____________________________________________________________________________________ Want to start your own business? Learn how on Yahoo! Small Business. http://smallbusiness.yahoo.com/r-index

2 1

[Twisted-Python] A PyCon BoF about content repository in Python
by Tarek Ziadé 11 Jan '07

11 Jan '07

Hello, I have added a BoF in the PyCon wiki for the next session in Dallas. It's about having a standard in Python for content repositories, that could be used by any Python framework, like Twisted. http://us.pycon.org/TX2007/PyJCR Please react/participate on the wiki, if you think this could be a good idea, for Twisted and other Python web frameworks. My secret wish would be to see core developers from each framework participate in this discussion, because I believe such a standard could be a huge benefit for Python industrialisation and frameworks maturity Cheers Tarek Ziadé -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/

1 0

[Twisted-Python] Twisted 2.5.0 released
by Christopher Armstrong 11 Jan '07

11 Jan '07

Get Twisted: http://twistedmatrix.com/ Twisted is an event-based framework for internet applications which works on Python 2.3.x, 2.4.x, and 2.5.x. Twisted 2.5.0 is a major feature release, with several interesting new developments and a great number of bug fixes. Some of the highlights follow. * AMP, the Asynchronous Messaging Protocol, was introduced. AMP is a protocol which provides request/response semantics over a persistent connection in a very simple and extensible manner. * An Epoll-based reactor was added, which can be used with twistd or trial by passing "-r epoll" on the command line. This may improve performance of certain high-traffic network applications. * The 'twistd' command can now accept sub-commands which name an application to run. For example, 'twistd web --path .' will start a web server serving files out of the current directory. This functionality is meant to replace the old way of doing things with 'mktap' and 'twistd -f'. * Python 2.5 is now supported. Previous releases of Twisted were broken by changes in the release of Python 2.5. * 'inlineCallbacks' was added, which allows taking advantage of the new 'yield' expression syntax in Python 2.5 to avoid writing callbacks for Deferreds. * Many improvements were made to the Jabber support in twisted.words. Hit http://twistedmatrix.com/trac/wiki/Downloads to see what's new and to get the latest downloads, including tarballs and Windows installers. Again thanks to Jean-Paul Calderone and Cory Dodt for helping get this release out the door, and thanks to all the Twisted contributors whose work went into Twisted 2.5. -- Christopher Armstrong International Man of Twistery http://radix.twistedmatrix.com/ http://twistedmatrix.com/ http://canonical.com/

3 6

[Twisted-Python] Summary of wxPython and Twisted integration status
by Itamar Shtull-Trauring 11 Jan '07

11 Jan '07

As of Twisted 2.1 you should be using twisted.internet.wxreactor for Twisted/wxPython integration. As of the just released 2.5, you should still be using wxreactor, but it should work much better: on Mac OS X Twisted events will be handled even when modal dialogs are open, and shutdown and startup events now work on all platforms. Please reports any bugs you find to our ticket tracker. -- Itamar Shtull-Trauring http://itamarst.org

1 0

[Twisted-Python] Moonfallen's ntsvc code
by Andrew Pratt 11 Jan '07

11 Jan '07

Does anyone know if Moonfallen's ntsvc code (currently written as a distutils extension) has made it into Twisted Core 2.5? If not, what is the likelihood of it being incorporated into twisted rather than remaining sandboxed? _________________________________________________________________ Join the millions of Australians using Live Search. Try live.com.au http://ninemsn.com.au/share/redir/adTrack.asp?mode=click&clientID=740&refer…

1 0

[Twisted-Python] Advice on Writing a Custom Twisted Reactor
by Andrew Francis 10 Jan '07

10 Jan '07

Hello Colleagues: I am a first time poster... For the past few months I have been experimenting with using Stackless Python with Twisted. I still consider myself very much a Twisted newbie. Currently my problems involve deferreds. >From doing online searches, I have run across customized reactors for Twisted. Consequently I am looking at the custom reactors that come with Twisted. Like Stackless, many GUIs have a hard time working with Twisted which blocks. Based on the ThreadedSelectReactor, most of the reactors are for GUIs. The general idea is that custom reactor's event loop calls Twisted in its own thread. This approach seems like it would work for Stackless and provide a decent general solution. However there are a few places where I am not quite clear. Most of my Stackless programmes have the following form (think of this as a MainLoop): ~ set up stuff while (stackless.getruncount() > 1): stackless.schedule() ~ Since it seems my custom reactor has to call Twisted as often as possible. I would wrap stackless.schedule() (since this is often called): stacklessReactor.schedule(): __hook__() # some method that would call Twisted stackless.schedule() Where I start to get a bit confused is: 1) In the Custom reactor.run() method, do I call self.__hook__() in the interleave() method? Is there anything more I have to do in the run() method? Can I avoid signal handler issues by passing False to Twisted's reactor.run() ? 2) From my Stackless code, do I have to use self.callFromThread() to call Twisted methods that involve deferreds? Again, any advice would be welcomed. Cheers, Andrew __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

2 1

[Twisted-Python] twisted and gprs
by John Aherne 10 Jan '07

10 Jan '07

Thanks to Phil, Carl and JPCalderone for the replies. I'll try and answer them by explaining more clearly what I am trying to do. And explaining some terms that I were vague. The sessions are a connection that I need to keep open - a combination of IP address and source port no - so the server can use the source port number that the PDA used when making the initial connection. The PDA will need to block on the TCPIP socket waiting for messages to be sent to it. It will process the message and then wait again. Provided the PDA does not close the socket and lose the connection with the original port no, the firewall will let the conversation keep going. The server will send on the initial source/destination port nos and not close the connection. This way the PDA can still be contacted over a long period of time. Because the PDA is behind the network firewall, if the original port used by the PDA is lost, the server can no longer send data to the PDA, since the firewall will not let the traffic through even though the IP address had not changed. As Carl pointed out I will need more than the IP address to identify the PDA. I will need an ID from the PDA that is unique so that if the PDA comes back in with a different IP address I can pick that up, drop the existing IP/port no if I still have it and register it with it's new IP address and source port no. As i write this down, explaining it to myself as much as anyone else, I think to myself that the mechanics look very much like an irc connection. I might be wrong so please let me know. The server will need to ping the PDA to confirm the connection is still there. If lost it will deregister the PDA and wait for it to reconnect again. What I am not sure about with irc is whether I can keep hold of the port no used by the PDA as its source port and maintain the connection with the IP address/port no combination. What I would like is if I can find a framework that does most of the heavy work with the communications. I can then find a suitable point to make some not too difficult changes to accommodate my requirements. I have done a very simple test using the socket library to confirm that the basics do work. The connection stays open and as long as I do not close the socket connection at either end and use the same source/destination port numbers the traffic flows.And I can push data without the PDA having to initiate new connections and requesting data. I hope this a better explanation than my first post. I am sure there will still be bits that might not be clear so let me know and I will fill in any gaps. If it is clear, what I am looking to find out is which bits of twisted I would need to use and what parts would need changing and to what extent. One final note, there could be around 100 of the PDAs to look after. Thanks for any feedback. John Aherne

4 9