Mailman 3 January 2007 - Twisted

[Twisted-Python] Re: Advice on Writing a Custom Twisted Stackless Reactor
by Andrew Francis Jan. 12, 2007

Jan. 12, 2007

Hello Jean-Paul: Once again, thanks for your response. >Twisted will block while there are no events to >process. As soon as there is an event, though, it will >wake up. So, if your tasklets are I/O bound, as soon >as there is a network event, Twisted will tell you >about it and you can schedule the next one to run. I know this. Since Twisted and Stackless are using the same OS thread, this will block Stackless. Stackless will block under the following condition - some … [View More]notation: S - tasklet waiting for a server call (waiting for a Twisted request Handler ) C - tasklet waiting for a client call (deferred request) | - running in parallel if you have a sequence of the form (S | C[0] | C[1] | c[n]) and S executes first, there will be no progress on the client tasklets. I tried adding a timer but had limited success. (By the way, when I first mentioned this in the Stackless mailing list, Bob Ipolito pointed this out to me) AF>I thought I got around this problem by AF>using task.LoopingCall/() that would issue a AF>stackless.schedule() at regular intervals. However AF>this approach turns out to be problematic. >What problems did you face? It certainly doesn't >sound ideal, but it does sound like it should give you >something which does more or less work. In some tests I would get a Twisted error : why = getAttr(selectable, method)() others times my processor would simply hang because the deferred function was not called. from my logging, the common demoninator was that the task.LoopingCall stopped. >Better would be to associate network and >timing events with specific tasklets and wake them up >when the reactor fires those events. I do this. >Most Twisted APIs aren't threadsafe, so most likely >what is happening here is that you are using a Twisted >API from the wrong thread and it just isn't working >right, which often manifests as an event which never >fires. What is the wrong thread? How do I correct this? >However, I still suspect bringing in a native thread >is not necessary to accomplish your goals. Maybe you are right. So far my experiences are pointing in the opposite direction. Also I have read posts involving integrating Chandler and IronPython with Twisted and they are running Twisted in a separate thread. That said, these are my deferred calls : postRequest = client.getPage(self.address, method = 'POST', headers = self.headers, postdata = self.body) postRequest.addCallback(self.__handleResponse__).addErrback(self.__handleError__) How should these be done from a thread? >Roughly, yes. blockingdemo.py in the examples >directory gives a fairly short, self-contained example >of integrating another loop using TSR. If you haven't >seen it yet, it'll probably help or help verify your >understanding. If you have, it might be a good basis >to guide further questions. I looked at blockingdemo.py. Blockingdemo is structured different from the reactor examples in the Twisted.internet module. Most of the example reactors i have seen inherit from threadselectreactor and override the run() method. What does TwistedManager.poll() do? The method is defined but called nowhere. I commented out poll() and the programme did the same thing. That said, I have been looking at the wxreactor.py. My questions are: 1. With the customer reactor, it would be Stackless that drives Twisted, not the other way around? 2. How exactly does interleave work? Some reactors don't use it. My Stackless thread and Twisted talk through stackless channels and a variation of a stackless channel, the uthread Queue. My strategy is as follows: 1. Inherit from threadselectreactor and override run() 2. in run(), my interleave calls __hook__() 3. In my Stackless Application, write a method __hook__. __hook__ looks like wxCallAfter in wxPython : def __hook__(self, function, *args, **kw): return function(*args, **kw) 3. Since stackless.schedule() is often called to yield control, wrap the stackless.schedule() """ call the underlying Stackless scheduler """ def __schedule__(self): # this is where I am a little fuzzy but wxCallAfter # has to be called somehow self.__hook__(?) stackless.schedule() 4. In the StacklessReactor.run(), the main loop that is called will look like: def MainLoop(self): while(stackless.getruncount() > 1): self.__schedule__() Again, I really appreciate insights. I really want to get Stackless Python to properly work with Twisted. Cheers, Andrew ____________________________________________________________________________________ 8:00? 8:25? 8:40? Find a flick in no time with the Yahoo! Search movie showtime shortcut. http://tools.search.yahoo.com/shortcuts/#news [View Less]

1 0

Re: [Twisted-Python] Ticket Hall of Shame - Current Loser: Brian Warner
by glyph＠divmod.com Jan. 12, 2007

Jan. 12, 2007

On 08:00 pm, exarkun(a)divmod.com wrote: >On Fri, 12 Jan 2007 10:54:10 -0800, Scott Lamb <slamb(a)slamb.org> wrote: >>(1) create a test that likely fails by running many, many times. It'd bog >>down the test suite. >>(2) create a test that might fail sometime, maybe. >>(3) add instrumentation to the main code solely for the purpose of the >>test (break twisted's kneecap before the race). >>(4) just say "clearly[*] if event A happens between … [View More]

1 0

[Twisted-Python] Reminder: Early Bird Registration for PyCon Ending Soon
by Jeff Rush Jan. 12, 2007

Jan. 12, 2007

Greetings. I'm the co-chair for an upcoming volunteer-run conference in the Dallas (Addison) area and would like to get an announcement into the hands of your members in some manner acceptable to your community. The event is the fifth international Python Conference, being held Feb 23-25, 2007 at the Marriott-Quorum in Addison, with early-bird registration ending Jan 15. The conference draws approximately 400-500 attendees from diverse backgrounds such as scientists from national and … [View More]medical labs, college/K-12 educators, web engineers and the myriad of IT developers and programming hobbyists. Those new to the Python language are welcome, and we're offering a half-day "Python 101" tutorial on the day before the conference, Thursday Feb 22 to help you get up to speed and better enjoy the rest of the conference. While unfortunately no talks specifically on Twisted were submitted this year, some of the presentations that Twisted folk might find interesting are: - Web Frameworks Panel - WSGI: An Introduction - soaplib: an easy-to-use python soap library - Distributing your Project with Python Eggs - Easy Creation of Interactive Tutorials - How I Wrote a Python App and Got $5 Million - Scaling Python for High-Load Web Sites - Software Development with Trac Being run by the Python community as a non-profit event, the conference strives to be inexpensive, with registration being only $260 (or $195 if you register prior to Jan 15th), with a further discount for students. On the day before the conference we are running a full day of classroom tutorials (extra charge per class) and then after the conference is a free four-days of sprints, which are informal gatherings of programmers to work together in coding on various projects. Sprints are excellent opportunities to do agile pair-programming side-by-side with experienced programmers and make new friends. Other activities are lightning talks, which are 5-minute presentations to show off a cool technology or spread the word about a project, open space talks, which are spontaneous gatherings around a topic and, new this year, a Python Lab where experienced and novice programmers will work together to solve challenging problems and then present their solutions. The conference is also running four keynote talks by leaders in the programming field, with a special focus on education this year: "The Power of Dangerous Ideas: Python and One Laptop per Child" by Ivan Krstic, senior member of the One Laptop per Child project "Premise: eLearning does not Belong in Public Schools" by Adele Goldberg, of SmallTalk fame "Python 3000" by Guido van Rossum, creator of Python "The Importance of Programming Literacy" by Robert M. "r0ml" Lefkowitz, a frequent speaker at O'Reilly conferences I believe your members will find the conference educational and enjoyable. More information about the conference along with the full schedule of presentations with abstracts, is available online: http://us.pycon.org/ Thanks for any help you can give in spreading the word, Jeff Rush Co-Chair PyCon 2007 [View Less]

2 1

Re: [Twisted-Python] Advice on Writing a Custom Twisted Stackless Reactor
by Andrew Francis Jan. 11, 2007

Jan. 11, 2007

Date: Wed, 10 Jan 2007 18:32:43 -0500 From: Jean-Paul Calderone <exarkun(a)divmod.com> Subject: To: Twisted general discussion <twisted-python(a)twistedmatrix.com> Message-ID: <20070110233243.11447.1076809328.divmod.quotient.12503@ohm> Content-Type: text/plain; format=flowed >Okay. I see that you're writing stackless integration >using TSR. I'm not sure what the point of this is. >Can you explain the functionality that's gained by >doing this, rather than … [View More]using something like what's in >Chris Armstrong's sandbox in threadless.py? Thank you Jean-Paul for your response. My goal is to write a custom Twisted reactor for Stackless Python. Although I have had success writing programmes that use deferred (for client HTTP calls) and HTTP request handlers, I have had problems doing both simultaneously in one programme. Some context : I am prototyping WS-BPEL (Web services Business Process Execution Language) processors. In the WS-BPEL language, it is possible for some process's "threads" to act as a server and other "threads" to act as a client simultaneously. The problem that I have found is that when Twisted blocks waiting for a server request, Stackless will also block, hence no progress on other tasklets that could execute. I thought I got around this problem by using task.LoopingCall/() that would issue a stackless.schedule() at regular intervals. However this approach turns out to be problematic. I thought I could solve the problem by placing Twisted in a thread but no. The main problem with placing Twisted in a thread is that I seem to lose the deferred (and I don't believe I am getting deadlock). >Can you explain the functionality that's gained by >doing this, 1. I solve my problem in a clean fashion. 2. If I write a custom Stackless Reactor, then I have a solution that conforms to the way Twisted does stuff. I think would make it easier for others to use my stuff. >rather than using something like what's in Chris >Armstrong's sandbox in threadless.py? Chris's approach looks like the approach I currently use. >Threaded select reactor actually isn't the same kind >of reactor as the rest of these, since it isn't >intended to be used via the normal reactor API (ie, >run). Right now I am using Twisted 2.4 with Stackless 2.4.3. For now, I would like to build on top of ThreadedSelectReactor. Back to my original question, given my previous example's "main loop", do I simply use interleave() to get my processor to call Twisted? Cheers, Andrew ____________________________________________________________________________________ Want to start your own business? Learn how on Yahoo! Small Business. http://smallbusiness.yahoo.com/r-index [View Less]

2 1

[Twisted-Python] A PyCon BoF about content repository in Python
by Tarek Ziadé Jan. 11, 2007

Jan. 11, 2007

Hello, I have added a BoF in the PyCon wiki for the next session in Dallas. It's about having a standard in Python for content repositories, that could be used by any Python framework, like Twisted. http://us.pycon.org/TX2007/PyJCR Please react/participate on the wiki, if you think this could be a good idea, for Twisted and other Python web frameworks. My secret wish would be to see core developers from each framework participate in this discussion, because I believe such a standard could … [View More]

1 0

[Twisted-Python] Twisted 2.5.0 released
by Christopher Armstrong Jan. 11, 2007

Jan. 11, 2007

Get Twisted: http://twistedmatrix.com/ Twisted is an event-based framework for internet applications which works on Python 2.3.x, 2.4.x, and 2.5.x. Twisted 2.5.0 is a major feature release, with several interesting new developments and a great number of bug fixes. Some of the highlights follow. * AMP, the Asynchronous Messaging Protocol, was introduced. AMP is a protocol which provides request/response semantics over a persistent connection in a very simple and extensible manner. * An … [View More]

3 6

[Twisted-Python] Summary of wxPython and Twisted integration status
by Itamar Shtull-Trauring Jan. 11, 2007

Jan. 11, 2007

As of Twisted 2.1 you should be using twisted.internet.wxreactor for Twisted/wxPython integration. As of the just released 2.5, you should still be using wxreactor, but it should work much better: on Mac OS X Twisted events will be handled even when modal dialogs are open, and shutdown and startup events now work on all platforms. Please reports any bugs you find to our ticket tracker. -- Itamar Shtull-Trauring http://itamarst.org

1 0

[Twisted-Python] Moonfallen's ntsvc code
by Andrew Pratt Jan. 11, 2007

Jan. 11, 2007

Does anyone know if Moonfallen's ntsvc code (currently written as a distutils extension) has made it into Twisted Core 2.5? If not, what is the likelihood of it being incorporated into twisted rather than remaining sandboxed? _________________________________________________________________ Join the millions of Australians using Live Search. Try live.com.au http://ninemsn.com.au/share/redir/adTrack.asp?mode=click&clientID=740&refer…

1 0

[Twisted-Python] Advice on Writing a Custom Twisted Reactor
by Andrew Francis Jan. 10, 2007

Jan. 10, 2007

Hello Colleagues: I am a first time poster... For the past few months I have been experimenting with using Stackless Python with Twisted. I still consider myself very much a Twisted newbie. Currently my problems involve deferreds. >From doing online searches, I have run across customized reactors for Twisted. Consequently I am looking at the custom reactors that come with Twisted. Like Stackless, many GUIs have a hard time working with Twisted which blocks. Based on the … [View More]

2 1

[Twisted-Python] twisted and gprs
by John Aherne Jan. 10, 2007

Jan. 10, 2007

Thanks to Phil, Carl and JPCalderone for the replies. I'll try and answer them by explaining more clearly what I am trying to do. And explaining some terms that I were vague. The sessions are a connection that I need to keep open - a combination of IP address and source port no - so the server can use the source port number that the PDA used when making the initial connection. The PDA will need to block on the TCPIP socket waiting for messages to be sent to it. It will process the … [View More]message and then wait again. Provided the PDA does not close the socket and lose the connection with the original port no, the firewall will let the conversation keep going. The server will send on the initial source/destination port nos and not close the connection. This way the PDA can still be contacted over a long period of time. Because the PDA is behind the network firewall, if the original port used by the PDA is lost, the server can no longer send data to the PDA, since the firewall will not let the traffic through even though the IP address had not changed. As Carl pointed out I will need more than the IP address to identify the PDA. I will need an ID from the PDA that is unique so that if the PDA comes back in with a different IP address I can pick that up, drop the existing IP/port no if I still have it and register it with it's new IP address and source port no. As i write this down, explaining it to myself as much as anyone else, I think to myself that the mechanics look very much like an irc connection. I might be wrong so please let me know. The server will need to ping the PDA to confirm the connection is still there. If lost it will deregister the PDA and wait for it to reconnect again. What I am not sure about with irc is whether I can keep hold of the port no used by the PDA as its source port and maintain the connection with the IP address/port no combination. What I would like is if I can find a framework that does most of the heavy work with the communications. I can then find a suitable point to make some not too difficult changes to accommodate my requirements. I have done a very simple test using the socket library to confirm that the basics do work. The connection stays open and as long as I do not close the socket connection at either end and use the same source/destination port numbers the traffic flows.And I can push data without the PDA having to initiate new connections and requesting data. I hope this a better explanation than my first post. I am sure there will still be bits that might not be clear so let me know and I will fill in any gaps. If it is clear, what I am looking to find out is which bits of twisted I would need to use and what parts would need changing and to what extent. One final note, there could be around 100 of the PDAs to look after. Thanks for any feedback. John Aherne [View Less]

4 9

[Twisted-Python] twisted and gprs networks
by John Aherne Jan. 8, 2007

Jan. 8, 2007

I am looking at using PDA devices talking to a server over gprs. I need to push data to the PDA over elapsed timeframes that might be hours. Naturally during this timespan the device may have been dropped from the network or acquire a different IP address. However to push data to the PDA, I need to record the IP and port used by the PDA for its first contact since it is stuck behind a firewall. The socket needs to be kept open on the PDA so the server can keep responding on the same … [View More]

6 5

[Twisted-Python] Twisted µConf @ linux.conf.au
by Stephen Thorne Jan. 8, 2007

Jan. 8, 2007

It turns out there's going to be a sizable contingent of Twisted hackers at linux.conf.au '07. As such, I'm attempting to organise a Micro Conference on the sunday before the conference. I'm planning on gathering at where we're staying Randwick Lodge, 211 Avoca St because there's wifi access available, and I'd like to be able to sit down and hack. Jonathan has mentioned wanting to hack on trial, and while I'm not resistant to that, I'm open to suggestions. :) http://shiny.thorne.id.au/… [View More]

1 0

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com Jan. 7, 2007

Jan. 7, 2007

1 0

Re: [Twisted-Python] Twisted Adapter
by glyph＠divmod.com Jan. 6, 2007

Jan. 6, 2007

On 02:09 pm, n_draghetta(a)yahoo.com wrote: >I have a wrapper written in Python for a TLS library written in C. I have to write an adapter in Twisted in order to encrypt the communication. Twisted already includes TLS support via PyOpenSSL. See: http://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.… There are some annoying deficiencies, mainly where PyOpenSSL has left out wrapping an API, but in general it works well. It is certainly possible to plug in your own, … [View More]

1 0

Re: [Twisted-Python] Twisted Adapter
by glyph＠divmod.com Jan. 6, 2007

Jan. 6, 2007

On 11:12 am, n_draghetta(a)yahoo.com wrote: >Hi all! >I have a wrapper in Python and I want to make an adapter in Twisted so that the wrapper can be used.I haven't found any informations about adapters in Twisted.Can anyone guide me please/send me to a tutorial? What do you mean by "wrapper"? Depending on what you mean, you might not need to write anything at all. Here is a tutorial on Twisted's components / adapters system: http://twistedmatrix.com/projects/core/documentation/howto/… [View More]

2 1

[Twisted-Python] Twisted Adapter
by n_draghetta Jan. 6, 2007

Jan. 6, 2007

Hi all! I have a wrapper in Python and I want to make an adapter in Twisted so that the wrapper can be used.I haven't found any informations about adapters in Twisted.Can anyone guide me please/send me to a tutorial? Thanks. __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

1 0

[Twisted-Python] writing a newsletter manager in Twisted
by Manlio Perillo Jan. 4, 2007

Jan. 4, 2007

Hi and happy new year. I would like to write a newsletter manager in Twisted, using twisted.mail and twisted.names. I have the following requirements: 1) The sending of mails should be queued. The sending should friendly to server and internet load 2) I should be notified when an email address in unreachable (after n tries) 3) I want to use a caching DNS server, where I should be able to control the TTL (not sure this is a good idea) Is this possible with Twisted and how much … [View More]

3 3

[Twisted-Python] how can i connect curses getch function and a process launched with spawnprocesss?
by Gabriel Santonja Jan. 3, 2007

Jan. 3, 2007

Hello, first : Happy new year 2007. second : I have some difficulties with curses getch() function and process launch with spawnprocess. I'm newbie in Twisted so i'm using the curses example who work fine with irc factory. i'm using the process howto too But when i try to send a message to a process with a >>> self.transport.writeSequence(contents+"\n") It's work fine with deferred but never with the curses.getch() i need a loop like that : - a caracter is push on the … [View More]

2 2

[Twisted-Python] How do I gracefully handle losing a connection?
by fritz Jan. 3, 2007

Jan. 3, 2007

Hello all, I'm building off of the ssh client example in the excellent O'Reilly book on Twisted by Abe Fettig but I've run into some problems and I'm not sure how I should proceed. Basically, I've implemented additional methods in order to handle connection issues with the client. The goal being, exiting of the program and printing out an error message detailing why the connection failed: class ClientCommandFactory(protocol.ClientFactory): ... def clientConnectionFailed(self, … [View More]connector, reason): print reason.getErrorMessage() reactor.stop() class CommandChannel(channel.SSHChannel): ... def closed(self): reactor.stop() def openFailed(self, reason): print reason.getErrorMessage() reactor.stop() class ClientCommandTransport(transport.SSHClientTransport): ... def connectionFailed(self, reason): print reason.getErrorMessage() reactor.stop() def connectionLost(self, reason): print reason.getErrorMessage() reactor.stop() The problem I'm currently facing is that. The connection properly ends and the program exits when bad data is supplied on the command line but when the program exits cleanly CommandChannel.closed() forces a call to ClientCommandTransport.connectionLost() and we see an exception thrown on a connection that I thought should end cleanly: Connection to the other side was lost in a non-clean fashion: Connection lost. Traceback (most recent call last): File "/usr/lib/python2.3/site-packages/twisted/internet/posixbase.py", line 226, in mainLoop self.runUntilCurrent() File "/usr/lib/python2.3/site-packages/twisted/internet/base.py", line 555, in runUntilCurrent call.func(*call.args, **call.kw) File "/usr/lib/python2.3/site-packages/twisted/internet/base.py", line 414, in _continueSystemEvent callable(*args, **kw) File "/usr/lib/python2.3/site-packages/twisted/internet/base.py", line 375, in disconnectAll failure.Failure(main.CONNECTION_LOST)) --- <exception caught here> --- File "/usr/lib/python2.3/site-packages/twisted/python/log.py", line 53, in callWithLogger return callWithContext({"system": lp}, func, *args, **kw) File "/usr/lib/python2.3/site-packages/twisted/python/log.py", line 38, in callWithContext return context.call({ILogContext: newCtx}, func, *args, **kw) File "/usr/lib/python2.3/site-packages/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/lib/python2.3/site-packages/twisted/python/context.py", line 37, in callWithContext return func(*args,**kw) File "/usr/lib/python2.3/site-packages/twisted/internet/tcp.py", line 554, in connectionLost Connection.connectionLost(self, reason) File "/usr/lib/python2.3/site-packages/twisted/internet/tcp.py", line 402, in connectionLost protocol.connectionLost(reason) File "code/python/sshclient/sshclient.py", line 33, in connectionLost reactor.stop() File "/usr/lib/python2.3/site-packages/twisted/internet/base.py", line 342, in stop raise RuntimeError, "can't stop reactor that isn't running" exceptions.RuntimeError: can't stop reactor that isn't running If I remove reactor.stop() from ClientCommandTransport.connectionLost() I then face the issue of some connection problems not being handled correctly and the program will go idle on some bad connections. So my question is, how can I fix my code to handle these exceptions and all cases of bad connections and connections terminating abnormally? Where should I implement connectionLost() and connectionFailed() and how can I cleanly disconnect from the SSH server without seeing ugly errors like the one above? Thanks in advance, Fritz [View Less]

1 0

[Twisted-Python] a question ablout DeferredList
by hzqij Jan. 3, 2007

Jan. 3, 2007

i have a sample code about DeferredList like this: from twisted.internet import reactor, defer def printResult(results): for resultInfo in results: success, result = resultInfo if success: print result reactor.stop() def addTen(result): return result + " ten" deferred1 = defer.Deferred() deferred2 = defer.Deferred() deferred3 = defer.Deferred() dl = defer.DeferredList([deferred1, deferred2, deferred3]).addCallback(printResult) deferred1.callback('one') deferred2.… [View More]

3 2