Mailman 3 May 2008 - Twisted

[Twisted-Python] Maximum recursion depth reached
by Steve Freitas 09 May '08

09 May '08

Hi all, I'm hitting the recursion limit in my code (well, somewhere inside Twisted, that is), and I'd appreciate any insight as to why. I wouldn't think what I'm doing -- returning a callback from method A which calls method B, which then in turn calls method A again -- would be controversial, since I'd assume Python would be letting go of the stack for method A, but I guess there's more to it than that. Code demonstrating the issue follows. Running it results in: File "/usr/lib/python2.5/site-packages/twisted/python/reflect.py", line 550, in accumulateBases accumulateBases(base, l, baseClass) exceptions.RuntimeError: maximum recursion depth exceeded (This doesn't actually perfectly reflect my problem in the real code I'm working with. I'm seeing "'maximum recursion depth exceeded' in <bound method DebugInfo.__del__ of <twisted.internet.defer.DebugInfo instance at 0xb1d99b6c>>", but I assume this is coming from the same basic problem in my code.) =================================== from twisted.internet import defer, reactor def yo_print(s, d): print s d.callback(s) return s def some_deferred(item): d = defer.Deferred() reactor.callLater(0, lambda: yo_print(item, d)) return d def start(items): dl = [] for item in items: dl.append(some_deferred(item)) if len(dl) > 4: break items = items[len(dl):] if dl: return defer.DeferredList(dl).addCallback(next_batch, items) reactor.stop() def next_batch(_, items): return start(items) if __name__ == '__main__': items = [] for i in range(1651): # 1650 doesn't do it items.append(i) print start(items) reactor.run() =================================== So I guess the rule is to never, within a callback chain, call a function which was invoked earlier in the callback chain? Steve

3 4

Re: [Twisted-Python] Twisted Presentation Materials...
by Andrew Francis 09 May '08

09 May '08

Hi Glen: > This is a good opportunity to decompose Twisted into >its core elements... kinda how it would be presented >to mathematics folks. If I don't screw it up, it might >even be useful to a broader audience trying >to figure out what Twisted is and the kinds of problems it addresses. In my PyCon 2008 talk, "Adventures," I have one slide that categorises Protocols, Deferreds, Callbacks, and the Reactor. I think those are the key components of Twisted. For greater completeness, I would add (protocol) factories. I guess a type or set theorist would want to know what sort of operations one performs upon objects of these types.... I have another slide that describes Twisted's control flow (from what I can tell). There I get into the MainLoop and reader/writer sets (there are a few more entries there). Unfortunately I haven't gotten around to properly labelling stuff there on the slide. Cheers, Andrew ____________________________________________________________________________________ Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

2 1

[Twisted-Python] Twisted Presentation Materials...
by Glenn H Tarbox, PhD 08 May '08

08 May '08

Scary thought... I am giving a talk on Twisted at Sage Developer Days. Its a fairly broad audience WRT levels of Pythonism.. but with other talks having titles like "Cylindrical Algebraic Decomposition", I doubt complexity in and of itself is a concern. This is a good opportunity to decompose Twisted into its core elements... kinda how it would be presented to mathematics folks. If I don't screw it up, it might even be useful to a broader audience trying to figure out what Twisted is and the kinds of problems it addresses. I welcome all pointers / content / suggestions. -glenn

3 3

[Twisted-Python] [ANN] Foolscap-0.2.6 released
by Brian Warner 07 May '08

07 May '08

I've just finished releasing Foolscap-0.2.6, the next-generation remote object protocol, intended as the successor to Perspective Broker. This release contains more logging improvements, including the long-awaited "incident-based logging" system (which dumps "black box" records of recent events when something bad happens), and several bugfixes. The full release notes are attached below. Tarballs can be obtained from foolscap.lothar.com and from PyPI. All releases are signed by my GPG public key: 0x1514A7BD . http://pypi.python.org/packages/source/f/foolscap/foolscap-0.2.6.tar.gz#md5… http://pypi.python.org/packages/source/f/foolscap/foolscap-0.2.6.zip#md5=1b… For documentation, bugs, and patches, please visit http://foolscap.lothar.com/trac . share and enjoy, -Brian * Release 0.2.6 (06 May 2008) ** Compatibility All releases between 0.1.3 and 0.2.6 (inclusive) are fully wire-compatible. The saved logfiles produced (by e.g. 'flogtool tail --save-to' and the log-gatherer) in 0.2.6 and beyond are not readable by tools (e.g. 'flogtool dump' and 'flogtool filter') from 0.2.5 and earlier. FURLs which contain "extensions" (described below) will not be tolerated by foolscap 0.2.5 or earlier. If, at some point in the future, we add such extensions to published FURLs, then such an application will require foolscap-0.2.6 or later to interpret those FURLs. ** Logging Changes *** "Incident" Logging This release finally implements the "strangeness-triggered logging" espoused in doc/logging.xhtml . By giving the foolscap logging code a directory to work with, the logging system will automatically save a compressed pickled logfile containing the messages that lead up to sufficiently-severe log event. The docs explain how to control what "sufficiently-severe" means. These events are retrievable through the logport, although no tools have been written yet to actually extract them. They are also retrievable by using 'flogtool dump' directly on the incident files. *** 'flogtool as a subcommand The implementation of the 'flogtool' executable has been rearranged to make it possible to add a 'flogtool' subcommand to some other executable. *** 'flogtool filter' now has --above LEVEL and --from TUBID options *** 'flogtool dump' has --rx-time option, also shows failure tracebacks *** gatherer: don't add erroneous UTC-implying "Z" suffix to filename timestamps *** 'flogtool tail': don't add spurious "0" to timestamps ** constraints no longer reject ('reference',) sequences The foolscap/banana serialization protocol responds to sending two separate copies of the same object in the same callRemote message by emitting one serialized object sequence and one 'reference' sequence: this is the standard way by which cycles are broken in the serialized data. Unfortunately, the wire-level constraint checkers in 0.2.5 and earlier would reject reference sequences with a Violation exception: if they were expecting a tuple, they would reject anything else, even a reference sequence that pointed at a tuple. Worse yet, python's normal constant-object folding code can produce shared references where you might not expect. In the following example, the two tuples are identical objects (and result in a 'reference' sequence on the wire), despite the programmer having typed them separately: rref.callRemote("haveTwoTuples", (0,1), (0,1) ) Foolscap-0.2.6 now allows reference sequence in all wire-level constraint checks, to avoid this false-negative Violation. The post-deserialization check will still enforce the constraint properly. It just does it late enough to be able to tell what the reference points to. ** Twisted/pyopenssl compatibility *** compatible with Twisted-8.0.x Some unit test failures under Twisted-8.0.x (the most recent release) were fixed: tests now pass against Twisted-8.0.x, and a buildbot is used to make sure compatibility is maintained in the future. *** incompatible with pyOpenSSL-0.7 An incompatibility has been discovered with the most recent version of PyOpenSSL. pyopenssl 0.6 works correctly, but pyopenssl 0.7 causes modern versions of Twisted (both 2.5.x and 8.0.x) to follow a code path that breaks the Foolscap unit tests. This may or may not cause a problem in actual application use (the symptom is that the non-winning parallel connections are not disconnected properly, and several negotiation timers are left running). Until a fix is developed for either Twisted or PyOpenSSL, the recommended workaround is to downgrade to PyOpenSSL-0.6 . Twisted bug #3218 and Foolscap bug #62 exist to track this problem. ** setup.py is more setuptools-friendly The foolscap version string is located with a regular expression rather than an import, allowing setuptools to install Foolscap as a build-dependency of some other project without having Twisted available first. If setuptools is available, we also declare a dependency upon Twisted (at least 2.4.0), to give more information to the setuptools dependency-auto-installer. ** Unauthenticated FURLs can now contain multiple connection hints Previously they were limited to a single one ** FURLs can now contain extensions, providing forwards-compatibility The parsing of FURLs has been refined to tolerate (and ignore) certain kinds of extensions. The "tubid" section will be able to have additional identifiers (perhaps stronger hashes for the public key, or an encryption-ready EC-DSA public key). In addition, the "connection hints" section will be able to contain alternate protocol specifiers (for TCP over IPv6, or a less connection-oriented UDP transport). By ignoring such extensions, foolscap-0.2.6 will tolerate FURLs produced (with extensions) by some future version of foolscap. This marks the beginning of a "transition period": when such extensions are introduced, 0.2.6 will be the oldest version still capable of interoperating with the extension-using new version.

3 5

[Twisted-Python] Cascading services (passing events through connected services)
by Matthieu Brucher 07 May '08

07 May '08

Hi, I'm currently trying to implement the JXTA standard (http://www.jxta.org) in Python using twisted (because it seems to be the right choice). I've implemented the basic parts that are not network related, that is, not much ;) To make a correct JXTA peer (well designed to be modular and to be able to add services), I have cascade several services (I think they should be services, but I can be wrong, I'm a real beginner with Twisted). For instance: network -> TCP transport layer service -> endpoint service -> some other high level services -> endpoint service -> TCP transport layer (may not be the same as the first one). Is there a sample somewhere that does something like this ? Each time an event is sent to a listener (the TCP transport layer, the endpoint, ...) that acts upon it. I've tried to find some samples over the Internet, but perhaps someone has a pointer to help me ? Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher

2 2

Re: [Twisted-Python] socket error 22
by glyph＠divmod.com 06 May '08

06 May '08

On 02:43 pm, wthie(a)thiengineering.ch wrote: >Hi all Hi Werner! Thanks for reporting this issue. It's always nice to hear about problems under high load. I hope we can get it fixed. >Running on twisted 2.5 OpenBSD I see the following traceback in the >logs every twenty minutes with up to 200 athena/LivePages connected: > return address.IPv4Address('TCP', *(self.socket.getsockname() + >('INET',))) > File "<string>", line 1, in getsockname > > socket.error: (22, 'Invalid argument') >Changing reactors does not help, the error is very unspecific. Before I >start changing the source sprinkling it with log statements is there >anybody out there who has a cure or knows an explanation. We don't have an OpenBSD buildbot, so I don't know if this is a platform-specific issue. Can you reproduce it on Linux or FreeBSD, for which there are officially-supported buildbots on http://buildbot.twistedmatrix.com/supported?branch=trunk ? One possible issue here is that the POSIX standard says that getsockname may fail with EINVAL if the socket has been shut down: http://www.opengroup.org/onlinepubs/009695399/functions/getsockname.html but Linux says that it may fail with EINVAL only if the namelen parameter is invalid: http://linux.die.net/man/2/getsockname However, it seems wrong that openbsd would shut down the socket without giving some other notification (the usual stuff that Twisted uses to get notified of a socket shutting down; presence in the write-selected set, an error in send() or recv()); i.e. you should see connectionLost called before this error can occur. If you really want to run Twisted on OpenBSD though, the first step should be getting a buildbot going and submitting any patches for failed tests. Otherwise any suggestions from core developers are just going to be guesses...

1 0

[Twisted-Python]The problem afer reactor stoped....
by 李祥曦 06 May '08

06 May '08

Hi,all I want to write a program that can send some e-mails automaticlly,i use the code like this: def sendmail(authenticationUsername, authenticationSecret,fromAddress, toAddress,messageFile,smtpHost, smtpPort=25): resultDeferred = Deferred() senderFactory = ESMTPSenderFactory(authenticationUsername,authenticationSecret,fromAddress,toAddress,messageFile,resultDeferred,requireTransportSecurity=False) reactor.connectTCP(smtpHost, smtpPort, senderFactory) return resultDeferred def sendComplete(result): print "Message sent." reactor.stop() class EmailUtil: def __init__(self,popserver,smptserver,username,password): self.popserver=popserver self.smtpserver=smptserver self.username=username self.password=password def goSend(self,fromaddr, toaddr, subject, body, filenames): message = buildMessage(fromaddr, toaddr, subject, body, filenames) messageData = message.as_string(unixfrom=False) tempfile=open('temp.mai','wb') tempfile.write(messageData) tempfile.close() print 'Sending.....' result = sendmail(self.username,self.password,fromaddr,toaddr,file('temp.mai'),self.smtpserver) result.addCallbacks(sendComplete,handleError) reactor.run() if __name__=='__main__': f=EmailUtil('','smtp.xxx.com','xxxx','1234') filenames=['test2.docx'] f.goSend('xxxx(a)xxx.com','xxx(a)hotmail.com','testSend','aaaaaaaaaa',filenames) f.goSend('xxxx(a)xxx.com','xxx(a)hotmail.com','testSend','aaaaaaaaaa',filenames) f.goSend('xxxx(a)xxx.com','xxx(a)hotmail.com','testSend','aaaaaaaaaa',filenames) I try to send the same mail,but after the first mail was sended,the program deadlock,and i use some tool konws after the first mail sended,no packets was send to the outside world.I guess maybe the problem related with the reactor run and stop,but i am sure,any one can tell me what's problem is?Thanks

3 3

Re: [Twisted-Python] Counting errors in a DeferredList and avoiding Unhandled error in Deferred messages
by Terry Jones 05 May '08

05 May '08

Hi Tristan > You might want to look at the gatherResults() utility function which > returns a deferred that either fires with a list of the results, or > errbacks with the first error to occur. Thanks. I can see how I'd write this, given Jean-Paul's reply. Terry

1 0

[Twisted-Python] EuroPython 2008
by Andrew Francis 05 May '08

05 May '08

Hi Folks: I noticed the deadline for EuroPython is May 22nd, 2008. I have been designing a new talk on Stackless and Twisted and I am thinking about submitting a talk proposal. I am curious if anyone is planning to give a talk? Cheers, Andrew ____________________________________________________________________________________ Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

1 0

[Twisted-Python] Counting errors in a DeferredList and avoiding Unhandled error in Deferred messages
by Terry Jones 05 May '08

05 May '08

If I pass a list of deferreds to DeferredList and want to add a callback to the DeferredList to look at whatever errors have occurred in the underlying deferreds, should I do something like this: deferreds = [] for x in whatever: deferreds.append(funcReturningDeferred(x)) dl = DeferredList(deferreds) dl.addCallback(errorCheckingFunc) for d in deferreds: d.addErrback(lambda _: pass) This ensures that errorCheckingFunc gets defer.FAILURE for those deferreds that finished up in their errback chain. Adding the lambda _: pass errback to each deferred in deferreds after it has been passed to DeferredList makes sure the errback chain doesn't later reach its end with an unhandled error. It that the right/best/approved way to handle needing to deal with errors in the callback of a DeferredList and not trigger the "Unhandled error in Deferred" message? Terry

2 1

Re: [Twisted-Python] Problem Changing Time Format of a Log
by Andrew Francis 05 May '08

05 May '08

Hi Zookoo and Cameron: On May 2, 2008, at 9:54 AM, Andrew Francis wrote: >Does this mean that the the method raised an exception >but the twisted logging infrastructure isn't telling >you what the exception was? >I've done this successfully: >http://allmydata.org/trac/tahoe/browser/src/allmydata/node.py? >rev=2360#L201 Thanks for the pointer. A part of my problem was that I forgot to import datetime. Anyhow, I re-read the documentation and solved my problem by doing: log.FileLogObserver.timeFormat = '%Y/%m/%d %H:%M:%d" However as Cameron suggested, I will upgrade to Twisted 8 when it is convenient. Cheers, Andrew ____________________________________________________________________________________ Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

1 0

[Twisted-Python] How to get port when using application
by Gabriel Rossetti 05 May '08

05 May '08

Hello, I am going to use application and I have a question, I have a server that needs to get a free port, before I would do this : ... port = reactor.listenTCP(0, f).getHost().port ... and now with application all I could find to do the equivilent was : ... t = internet.TCPServer(0, tcpFactory) port = p._getPort()._realPortNumber ... the problem is that this isn't very clean as I'm accessing protected members. Is there a better way of doing this? Thank you, Gabriel

1 0

[Twisted-Python] DNSQueryRefusedError question
by Alvin Delagon 05 May '08

05 May '08

Can anyone explain what is causing this exception? twisted.names.error.DNSQueryRefusedError: <twisted.names.dns.Message instance at 0xb78470cc> I'm using twisted.mail.smtp which subclasses twisted.names.dns. I keep getting this failure during smtp.sendmail(). Thank you in advance. :) --- Alvin Delagon -- http://www.alvinatorsplayground.blogspot.com/

1 2

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 04 May '08

04 May '08

1 0

[Twisted-Python] ldaptor's LDAPClient.send_multiResponse not dealing with chained deferreds, patch
by Michael Torrie 04 May '08

04 May '08

This message is probably for Tv. I have a situation where the callback to a LDAPClient.send_multiResponse() call needs to do a bunch of work that I want to split up over several functions, using deferred chaining. What I want to do is something like this (this in conjunction with an LDAP proxy server I am writing using Twisted and ldaptor): d = self.client.send_multiResponse(request, got_response) then later: def got_response(response): # get a list of other things to do with the response d = do_stuff_with_response(response) d.addCallback(finish) return d def finish(response): # we're done return isinstance(response, ( pureldap.LDAPSearchResultDone, pureldap.LDAPBindResponse, )) However, this doesn't work, because the send_multiResponse() method of LDAPClient isn't expecting the handler to return a deferred, even though if I do return a deferred the reactor does handle it properly, chaining the deferreds and their callbacks. The problem is that in ldapclient.py, line 171, we have: # Return true to mark request as fully handled if handler(msg.value, *args, **kwargs): del self.onwire[msg.id] When handler() returns a deferred object, after the reactor processes the chain the value is a Deferred object, not True or False, even though the value of the deferred object may be True or False. Hence the del self.onwire[msg.id] always executes, which when dealing with search result entries is a problem as they all share the same id. I made a quick hack to fix this: ----------------------------------------------------------- --- /tmp/ldapclient.py 2008-05-03 18:53:26.000000000 -0600 +++ ldaptor/protocols/ldap/ldapclient.py 2008-05-03 18:58:07.000000000 -0600 @@ -168,7 +168,14 @@ assert args is not None assert kwargs is not None # Return true to mark request as fully handled - if handler(msg.value, *args, **kwargs): + result = handler(msg.value, *args, **kwargs) + + try: + result = result.result + except AttributeError: + pass + + if result: del self.onwire[msg.id] ##Bind ---------------------------------------------------------- Is this an acceptable way to do this?

2 2

[Twisted-Python] Problem Changing Time Format of a Log
by Andrew Francis 02 May '08

02 May '08

Hi Folks: I need to change date format of a log - the application I am using wants HH:MM:SS. Twisted seems to give back HH:MM. Based on the example in Abe Fettig's book and looking at Twisted.Python.Log (and I am not sure where to define the timeFormat string), I do the following. class MyLog(log.FileLogObserver): def formatTime(self, when): if self.timeFormat is not None: return time.strftime(self.timeFormat, time.localtime(when)) tzOffset = -self.getTimezoneOffset() when = datetime.datetime.utcfromtimestamp(when + tzOffset) tzHour = int(tzOffset / 60 / 60) tzMin = int(tzOffset / 60 % 60) # added seconds return '%d/%02d/%02d %02d:%02d:%02d %+03d%02d' % ( when.year, when.month, when.day, when.hour, when.minute, when.second, tzHour, tzMin) def emit(self, logEntryDict): log.FileLogObserver.emit(self, logEntryDict) When I run my code in twistd, I get the following error Log observer <bound method InsightLog.emit of <> failed, removing from observer list. Any help would be appreciated in telling me what is going wrong, or pointing out the right way of altering the log. Cheers, Andrew ____________________________________________________________________________________ Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

3 2

[Twisted-Python] From asyncore to twisted
by Daniel Brandt 02 May '08

02 May '08

Hello, I'm looking for some advice on how to get started with an asyncore -> Twisted conversion. Being new to twisted I have some trouble getting started (re)designing my system. All ideas are welcome since I'm also doing this to learn. I am about to convert a message-passing daemon written with asyncore to Twisted, and I have two goals in mind. Code that is easier to maintain and expand, and higher reliability. The last part would probably be achievable while sticking with the present code, but since I keep hearing I really should use Twisted instead, I figured I'd try it out :-). The daemon listens for connections from different peers. Some peers just leave a message and disconnect, some peers connect as receivers and linger waiting for messages. A message will be given to the daemon by a messenger, and forwarded by the daemon to exactly one of the connected receivers. The daemon keeps a data structure (a dict with identifier: channel, actually) of receivers which it cycles through in a round-robin fashion. There are more scenarios as well; (recievers may send status messages back to the daemon that the daemon logs, some messengers linger waiting for confirmation from a receiver and a few other) but they are mostly just variations of the above. I'm a bit confused about what parts should be in the Protocol and what should be separated away from it. Should I make a specialized Factory or make a Service (or several depending on what type of client connection it is)? I realize there is a lot in my description that's open for interpretation, but since I'm mostly after design ideas and good practices, I'm hoping that's OK.. Best regards, Daniel Brandt

3 2

[Twisted-Python] Implement a client to telnet to a Linux machine
by Steven Song 01 May '08

01 May '08

I am new to both Python and Twisted framework, so if this has already been dicussed bofore please excuse me. I am trying to implement a client software using Python to telnet to a Linux machine. The login involves authntication with username and password. Can someone please point me to an example if there is one? I know it has something to do with twisted.conch.telnet but am not sure about the detailed implementation yet. Thanks in advance. _________________________________________________________________ Turn every day into $1000. Learn more at SignInAndWIN.ca http://g.msn.ca/ca55/213

1 0