Mailman 3 April 2005 - Twisted

[Twisted-Python] blocking and threads
by Joachim Boomberschloss 27 Apr '05

27 Apr '05

Hi all, I am just wondering: from the Twisted how-tos, it appears that doing anything that could possibly take arbitrarily long to execute should not be done in the reactor's main thread; i.e. it should be done using an asynchronous library (such as Twisted's network communication facilities), or in a different thread, using the reactor's thread pool. So it would seem that many things that may be considered "primitive" in Python, such as file IO, require some kind of patching if they are to be immediately usable by a Twisted application. I came up with the following solution, which enables calling any function in a different thread with a deferred interface; I just wanted to make sure that I'm not completely missing some point: def deferToThread(func, *args, **kargs): """executes the given function in a thread, and passes the return value to the deferred we return""" d = defer.Deferred() reactor.callInThread(_calledInThread, d, func, *args, **kargs) return d def _calledInThread(d, func, *args, **kargs): try: retval = func(*args, **kargs) except Exception, x: reactor.callFromThread(d.errback, x) else: reactor.callFromThread(d.callback, retval) Thanks, Joe __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

2 1

[Twisted-Python] Dynamic listening to UDP ports?
by Torbjörn Einarsson 27 Apr '05

27 Apr '05

Hi, Im trying to make a server that listens to one TCP socket for commands and then starts listening to more or less UDP sockets for data depending on the commands. The UDP data shall then be forwarded to one or more outgoing UDP ports. For example, here are some commands with explanations: addChannel 10000 should make the server listening to UDP port 10000 addChannel 10002 should make the server listening to UDP port 10002 addConnection 10000 10.0.0.2:5000 start forwarding the packet from UDP port 10000 to port 5000 on another machine addConnection 10000 10.0.0.3:5002 start forwarding the packet from UDP port 10000 to port 5002 on yet another machine delConnection 10000 10.0.0.2:5000 stop forwarding packets to the first guy delConnection 10000 10.0.0.3:5002 stop forwarding packets to the second guy delChannel 10000 stop listening to port 10000 Ive been making some rather advanced servers in twisted, but with fixed ports. How can I deal with this type of dynamic port bindings? For adding a port to listen to: reactor.listenUDP(10000,prot) is working, but how do I stop listening to a port an unbind it? Ive found something called unlistenUDP but it seems to be deprecated? Alternatively, I could start up a service , like u = internet.services.UDPServer(10000,UdpProtocol()) but how do I stop this particular service without stopping all others? Any suggestions are welcome. Regards, Torbjörn Einarsson1

5 11

[Twisted-Python] creating source tarball from svn tree
by Mike Taylor 26 Apr '05

26 Apr '05

I'm trying to build a source distribution of the svn tree using the following command: python setup.py all sdist but I get the error below - any thoughts? running sdist warning: sdist: manifest template 'MANIFEST.in' does not exist (using default file list) Traceback (most recent call last): File "twisted/topfiles/setup.py", line 113, in ? dist.setup(**setup_args) File "./twisted/python/dist.py", line 69, in setup return core.setup(**kw) File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/ python2.3/distutils/core.py", line 149, in setup dist.run_commands() File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/ python2.3/distutils/dist.py", line 907, in run_commands self.run_command(cmd) File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/ python2.3/distutils/dist.py", line 927, in run_command cmd_obj.run() File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/ python2.3/distutils/command/sdist.py", line 143, in run self.get_file_list() File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/ python2.3/distutils/command/sdist.py", line 240, in get_file_list self.add_defaults() File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/ python2.3/distutils/command/sdist.py", line 301, in add_defaults self.filelist.extend(build_ext.get_source_files()) File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/ python2.3/distutils/command/build_ext.py", line 361, in get_source_files self.check_extensions_list(self.extensions) File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/ python2.3/distutils/command/build_ext.py", line 294, in check_extensions_list (ext_name, build_info) = ext TypeError: unpack non-sequence Error: Subprocess exited with result 1 for project core --- Bear http://code-bear.com Open Source Applications Foundation (OSAF) http://www.osafoundation.org PGP Fingerprint = 9996 719F 973D B11B E111 D770 9331 E822 40B3 CD29

2 2

RE: [Twisted-Python] twisted robot ...
by Grant McDonald 25 Apr '05

25 Apr '05

> Twisted does care about this for signal handling. If Twisted is run in a > thread it will not be able to do shutdown based on signals or run > processes correctly (though if you install your own signal handlers that > forward info to twisted it can be made to work). Are there examples of installing signal handlers and forwarding signal information to a twisted thread?

2 1

[Twisted-Python] Help with SMTP Proxy
by Jan Van Uytven 25 Apr '05

25 Apr '05

Hi, I'm a new user of Twisted, I came to it because I need to write an SMTP proxy and want to use Python to do it. The proxy would copy the incoming e-mail (including the envelope headers) and then forward it to Postfix. Googling around I came across an example of an SMTP server (written by the author of twisted.mail I think) and after playing around with it a bit I can get it to receive and copy the e-mail, but I'm not quite sure how to forward it on after that. Right now I have the proxy sitting on port 2025 and the final smtp server on port 2030. Here's the code I have so far: import os, StringIO from zope.interface import implements from twisted.application import service application = service.Application("Postfix SMTP Proxy for CopyUser") from twisted.application import internet from twisted.internet import protocol, defer smtpServerFactory = protocol.ServerFactory() from twisted.mail import smtp class FileMessage(object): implements(smtp.IMessage) def __init__(self, fileObj): self.fileObj = fileObj self.msg = [] def lineReceived(self, line): self.fileObj.write(line + '\n') self.msg.append(line) def eomReceived(self): self.fileObj.close() print "\n".join(self.msg) return defer.succeed(None) def connectionLost(self): self.fileObj.close() os.remove(self.fileObj.name) class Delivery(object): implements(smtp.IMessageDelivery) counter = 0 def validateTo(self, user): fileName = 'smtplog.' + str(self.counter) self.counter += 1 return lambda: FileMessage(file(fileName, 'w')) def validateFrom(self, helo, origin): return origin def receivedHeader(self, helo, origin, recipients): recpts = [] for i in recipients: recpts.append(str(i)) recptstr = "Recipients: " + ",".join(recpts) origin = "Origin: " + str(origin) return recptstr+"\r\n"+origin+"\r\n\r\n" class ESMTPFactory(protocol.ServerFactory): protocol = smtp.ESMTP def buildProtocol(self, addr): p = self.protocol() p.delivery = Delivery() p.factory = self return p class ForwardSMTPClient(smtp.ESMTPClient): def getMailFrom(self): return ["smtp_proxy@localhost"] def getMailTo(self): return "tester@localhost" def getMailData(self): data = """ Subject: Test! Test! """ return StringIO.StringIO(data) def sentMail(self, code, resp, numOk, addresses, log): print 'Sent', numOk, 'messages' class SMTPClientFactory(protocol.ClientFactory): protocol = ForwardSMTPClient def buildProtocol(self, addr): return self.protocol(secret=None, identity='localhost') smtpServerFactory = ESMTPFactory() smtpClientFactory = SMTPClientFactory() smtpServerService = internet.TCPServer(2025, smtpServerFactory) smtpServerService.setServiceParent(application) I've left out actually forwarding the message for now, I just want to forward a test e-mail upon receipt of the message. The whole structure of this program doesn't seem right to me. I thought initially of trying something like adding this to the eomReceived method: smtpForwardService = internet.TCPClient('localhost', 2030, smtpClientFactory) smtpForwardService.setServiceParent(application) but it started an endless loop of sending the test e-mail. Most clients use reactor.stop() to finish after processing, but I need the reactor to keep on going. Documentation on twisted.mail is almost non-existent. Is there a better way of doing this? Thanks, Still Struggling

2 3

[Twisted-Python] Sending unicode strings
by Michal Chruszcz 25 Apr '05

25 Apr '05

Hello, I was undoubtedly surprised when I found out that I cannot pass unicode strings to Twisted. In 1.3 twisted.internet.abstract looks like this: def write(self, data): assert isinstance(data, str), "Data must be a string." if not self.connected: return and in 2.0: def write(self, data): if isinstance(data, unicode): # no, really, I mean it raise TypeError("Data must be not be unicode") Why do you mean it? Why I can't send unicode through twisted? It's ridiculous that I have to convert UTF8 strings to ISO on the client side and then once again from ISO to UTF8 on the server side, so I suppose you've got really good excuse. -- Michal Chruszcz -=- Never seen

5 6

[Twisted-Python] twisted robot ...
by Darran Edmundson 25 Apr '05

25 Apr '05

Hi All, Thanks to everyone for the recent help on integration of twisted with wxPython. We are going to hold off on using Bob's new code at the moment in favour of using the release version of Twisted (easier for the users to install). At the moment we've got Twisted in the main thread, spawning a thread to run wx. It works nicely except we can't figure out how to communicate with wx (ie., generate wxevents) from twisted. (We don't need the converse - it's notification only.) I fly out to San Francisco tommorrow morning so any help would be most appreciated. So about the robot subject line ... we bought a 50cm RC tank on Ebay, ripped it down to the chassis and added a platform for carrying around a 12" Powerbook. The laptop runs our twisted server which drives a USB controllable Phidget (www.phidget.com) to control the tank treads and a servo motor for camera (iSight) tilt. We talk to the server via wifi. The client app (wx GUI is done and working) is what we are working on at the moment... Cheers, Darran. P.S. I'll put some photos and video up on my web site for anyone interested. -- Darran Edmundson (darran.edmundson(a)anu.edu.au) ANU Supercomputer Facility Vizlab Australian National University, Canberra, ACT 2600 tel: +61 2 6125-0517 fax: +61 2 6125-5088

4 3

[Twisted-Python] Weekly Bug Summary
by James Y Knight 25 Apr '05

25 Apr '05

(Or, er, semi-weekly) Bug summary ______________________ Since 2005-04-11 14:20:07.600000: Bugs opened: 29 Bugs closed: 20 Total open bugs: 346 (+9) Urgent: 6 (+2) Bug: 135 (+0) Feature: 135 (+6) Wish: 70 (+1) New / Reopened Bugs ______________________ ==== Urgent ==== [XISH] Erroneous .encode('utf-8') in twisted.xish.domish (yesterday) http://twistedmatrix.com/bugs/issue1002 opened by ralphm [CORE] gtk2 reactor broken for actual GUI apps (4 days ago) http://twistedmatrix.com/bugs/issue994 opened by warner ==== Bug ==== Python 2.4 - exceptions.AttributeError: 'module' object has no attribute 'kwdict' (yesterday) CLOSED http://twistedmatrix.com/bugs/issue1001 opened by cphsu [ENTERPRISE] ConnectionPool allows the pool size to be grown (3 days ago) CLOSED http://twistedmatrix.com/bugs/issue997 opened by mg [CORE, WIN32] Win32 installer rc3 does not install mktap.py (7 days ago) http://twistedmatrix.com/bugs/issue991 opened by exarkun [WEB] request.setHeader should fail if data has already been sent written to the client (3 days ago) http://twistedmatrix.com/bugs/issue998 opened by mg [WEB] [PATCH] web2 CGI syntax error (1 week ago) CLOSED http://twistedmatrix.com/bugs/issue977 opened by warner [WEB] cgi fails to close stdin of the child process (1 week ago) http://twistedmatrix.com/bugs/issue981 opened by warner Excessive deprecation warnings from MetaInterface.__getattr__ (5 days ago) http://twistedmatrix.com/bugs/issue993 opened by jknight session.getComponent doesn't work with 'default' keyword arg (6 days ago) http://twistedmatrix.com/bugs/issue992 opened by justinj AttributeError: 'WSGIHandler' object has no attribute 'stopProducing' (1 week ago) CLOSED http://twistedmatrix.com/bugs/issue987 opened by srichter [WEB] Assertion Error in twisted.internet.abstract (1 week ago) CLOSED http://twistedmatrix.com/bugs/issue986 opened by srichter [WEB] AttributeError: HTTPChannelRequest instance has no attribute 'transport' (1 week ago) CLOSED http://twistedmatrix.com/bugs/issue985 opened by srichter [WEB] twisted.web2.wsgi environment variables (1 week ago) CLOSED http://twistedmatrix.com/bugs/issue984 opened by srichter [CORE] Default log observer chokes on unicode messages (7 days ago) http://twistedmatrix.com/bugs/issue989 opened by exarkun [CORE, DOCUMENTATION] twisted.internet.abstract.FileDescriptor.loseConnection.__doc__ is a stupid lie (1 week ago) http://twistedmatrix.com/bugs/issue988 opened by exarkun [FTP] IFTPShell needs to be completely restructured (1 week ago) http://twistedmatrix.com/bugs/issue983 opened by srichter [patch] Naming consistency in new plugin module (1 week ago) CLOSED http://twistedmatrix.com/bugs/issue980 opened by mg [MAIL, NAMES] fix deprecation warnings in test_mail (1 week ago) CLOSED http://twistedmatrix.com/bugs/issue982 opened by warner backport if-modified-since bugfix from web2 to web (1 week ago) CLOSED http://twistedmatrix.com/bugs/issue976 opened by demaledetti twisted.web.client.getPage never completes files >3.1MB (Windows) (1 week ago) CLOSED http://twistedmatrix.com/bugs/issue974 opened by moonfallen ==== Feature ==== [ENTERPRISE] Add runWithConnection to adbapi.ConnectionPool (1 week ago) CLOSED http://twistedmatrix.com/bugs/issue975 opened by mg Common Access Logging Observer for twisted.web2 (yesterday) http://twistedmatrix.com/bugs/issue1000 opened by srichter twisted.web2 HTTP Server Enhancements (yesterday) http://twistedmatrix.com/bugs/issue999 opened by srichter [CORE] t.plugin caching and read-only package directories (3 days ago) http://twistedmatrix.com/bugs/issue996 opened by mg contribpackages are not updated for Python 2.4 (4 days ago) http://twistedmatrix.com/bugs/issue995 opened by manlio [CORE] Deferred cancellation (7 days ago) http://twistedmatrix.com/bugs/issue990 opened by exarkun [DOCUMENTATION] Rename enterprise howto to rdbms (filename) (1 week ago) http://twistedmatrix.com/bugs/issue979 opened by itamarst [DOCUMENTATION] Document wxPython mess appropriately (1 week ago) http://twistedmatrix.com/bugs/issue978 opened by itamarst Bugs Closed ______________________ ==== Bug ==== Python 2.4 - exceptions.AttributeError: 'module' object has no attribute 'kwdict' (yesterday) http://twistedmatrix.com/bugs/issue1001 opened by cphsu, closed by exarkun [ENTERPRISE] ConnectionPool allows the pool size to be grown (3 days ago) http://twistedmatrix.com/bugs/issue997 opened by mg, closed by davep [WEB] [PATCH] web2 CGI syntax error (1 week ago) http://twistedmatrix.com/bugs/issue977 opened by warner, closed by warner [WEB] [PATCH] http.py stringToDatetime handling RFC850 format (8 months ago) http://twistedmatrix.com/bugs/issue669 opened by db3l, closed by jknight AttributeError: 'WSGIHandler' object has no attribute 'stopProducing' (1 week ago) http://twistedmatrix.com/bugs/issue987 opened by srichter, closed by jknight [WEB] Assertion Error in twisted.internet.abstract (1 week ago) http://twistedmatrix.com/bugs/issue986 opened by srichter, closed by jknight [WEB] AttributeError: HTTPChannelRequest instance has no attribute 'transport' (1 week ago) http://twistedmatrix.com/bugs/issue985 opened by srichter, closed by jknight qtreactor dont works with threads!! (4 months ago) http://twistedmatrix.com/bugs/issue793 opened by nahu, closed by exarkun [WEB] twisted.web2.wsgi environment variables (1 week ago) http://twistedmatrix.com/bugs/issue984 opened by srichter, closed by jknight SSL problems with win32eventreactor (12 months ago) http://twistedmatrix.com/bugs/issue592 opened by justinj, closed by jknight [patch] Naming consistency in new plugin module (1 week ago) http://twistedmatrix.com/bugs/issue980 opened by mg, closed by exarkun [MAIL, NAMES] fix deprecation warnings in test_mail (1 week ago) http://twistedmatrix.com/bugs/issue982 opened by warner, closed by exarkun [WEBSITE] No link to Twisted 2 installer for win32 on website (2 weeks ago) http://twistedmatrix.com/bugs/issue964 opened by ash, closed by exarkun [CORE] gtk reactor not delivering callbacks timely (13 months ago) http://twistedmatrix.com/bugs/issue552 opened by dektwisted, closed by exarkun Weirdness in weakref.WeakValueDictionary casues problems in enterprise/reflector.py (with fix) (8 months ago) http://twistedmatrix.com/bugs/issue682 opened by icepick, closed by exarkun [PB, WEB] twisted.web.distrib uses deprecated pb classes (13 months ago) http://twistedmatrix.com/bugs/issue543 opened by hypatia, closed by warner backport if-modified-since bugfix from web2 to web (1 week ago) http://twistedmatrix.com/bugs/issue976 opened by demaledetti, closed by jknight twisted.web.client.getPage never completes files >3.1MB (Windows) (1 week ago) http://twistedmatrix.com/bugs/issue974 opened by moonfallen, closed by exarkun ==== Feature ==== [ENTERPRISE] Add runWithConnection to adbapi.ConnectionPool (1 week ago) http://twistedmatrix.com/bugs/issue975 opened by mg, closed by davep [DEFERRED] Deferred setTimeout sucks (20 months ago) http://twistedmatrix.com/bugs/issue178 opened by radix, closed by exarkun

1 0

[Twisted-Python] DeferredList and errbacks
by Lenny G Arbage 22 Apr '05

22 Apr '05

I just noticed [what appears to me to be] a contradiction in the twisted documentation. The "Using Deferreds" howto [1] says, "A standard DeferredList will never call errback," while the "Twisted FAQ" [2] states, "Always add errbacks!" in response to the question, "My Deferred or DeferredList never fires, so my program just mysteriously hangs! What's wrong?" So, which is it? I suppose what this is getting at is that each deferred in a DeferredList should have an errback attached separately, but this isn't super-clear to me. Is this the right way to interpret the above statements? thanks, Lenny G. [1] http://twistedmatrix.com/documents/current/howto/defer [2] http://twistedmatrix.com/documents/current/howto/faq#auto19 __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

4 4

[Twisted-Python] Twisted P2P
by van Aswegen, Marinus (ZA - Johannesburg) 22 Apr '05

22 Apr '05

Hi Guys I need some advice before I start hacking about with socket, SocketServer and twisted. I'm planning to write a LAN P2P program to make it easy to share files on the LAN. I'm planning to make it cross platform win / unix. Each P2P client will advertise using UDP multicast that they are out there. If a client wants to send a file or files it will connect TCP to the client and request auth. Upon auth it will tx the files using TCP. I would like the each P2P client to be able to handle multiple connections (send + rec). Suggestions, pointers, help appreciated. Regs Marinus Important Notice: This email is subject to important restrictions, qualifications and disclaimers ("the Disclaimer") that must be accessed and read by visiting our website and viewing the webpage at the following address: http://www.deloitte.com/za/disclaimer. The Disclaimer is deemed to form part of the content of this email in terms of Section 11 of the Electronic Communications and Transactions Act, 25 of 2002. If you cannot access the Disclaimer, please obtain a copy thereof from us by sending an email to ClientServiceCentre(a)Deloitte.co.za.

4 3