Mailman 3 April 2003 - Twisted

[Twisted-Python] Connecting using SOCKS
by David Phillips April 3, 2003

April 3, 2003

I'm new to Twisted, so please forgive me if this is obvious. I read the howtos and looked at some of the example programs, but didn't find anything relevant. I would like to use a protocol with SOCKS. For example, ircLogBot.py calls reactor.connectTCP(). How would I make it connect through SOCKS? If Twisted does not currently have this functionality, how would I go about implemented it the correct way? -- David Phillips <david(a)acz.org> http://david.acz.org/

1 0

[Twisted-Python] non-blocking .read
by Moshe Zadka April 3, 2003

April 3, 2003

It is often convenient in protocol implementations to just .read() as much as you want to. However, it is impossible in Twisted, becase .read() is a blocking operation. Happily, we can make blocking operations seem non-blocking with the use of deferred. The following protocol does just that: class ReadProtocol(protocol.Protocol) toRead = 0 until = '' def rawDataReceived(self, data): pass def dataReceived(self, data): while data: if self.toRead: read, data = data[:self.toRead], data[self.toRead:] self.buffer.write(read) self.toRead -= len(read) if self.toRead == 0: self.deferred.callback(self.buffer.getvalue()) self.buffer.close() elif self.until: idx = data.find(self.until) if idx == -1: self.buffer.write(data) data = '' else: read, data = data[:idx], data[idx:] self.until = '' self.buffer.write(read): self.deferred.callback(self.buffer.getvalue()) self.buffer.close() else: self.rawDataReceived(data) data = '' def readUntil(self, c): self.toRead = 0 self.until = c self.buffer = StringIO() self.deferred = defer.Deferred() return self.deferred def readLength(self, len): self.toRead = len self.until = '' self.buffer = StringIO() self.deferred = defer.Deferred() return self.deferred # How to use, e.g., to implement netstrings: class NetstringProtocol(ReadProtocol): def connectionMade(self): self.readString() def readString(self): self.readUntil(':').callback( lambda s: self.readLength(int(s)).callback( self.stringReceived).callback( lambda _: self.readLength(1).callback( lambda x: x==',' or self.transport.loseConnection()).callback( lambda _: self.readString)) def stringReceived(self, s): pass

4 4

[Twisted-Python] supressess warnings
by Andrew Dalke April 2, 2003

April 2, 2003

I'm just starting to learn how to use Twisted. My first project is to write a simple XML-RPC server. I started with the demo, and that worked fine. I then added an import statement for the module I need. from openeye.oechem import * Now, this uses a shared library extension which was built for Python 2.2, and I'm running a CVS version of Python 2.3, so I usually get /usr/local/lib/python2.3/site-packages/openeye/oechem.py:5: RuntimeWarning: Python C API version mismatch for module _oechem: This Python has API version 1012, module _oechem has version 1011. import _oechem However, that didn't happen when I started my my simple server. I got no warning message. I started to worry that my file system wasn't properly saving files! I played around with the imports, and moved that import to be above the twisted ones, and it started reporting the warning. A bit of searching later and I found twisted/python/log.py which redefines (!) warnings.showwarning to dump the data to a logfile, which is initialized to a NullFile(), which does nothing. So the twisted code is throwing some (potentially useful) diagnostics. Doing a search I found the commit statement http://twistedmatrix.com/pipermail/twisted-commits/2002-October/ 003975.html > 2) implemented a custom warning handler; now warnings look sexy. > (the hackish overriding of warnings.showwarning is the recommended way to > do so, according to the library reference.) I'll grant that's what the docs say. However, why is the logfile for showwarnings initialized to NullFile, and not to sys.stderr? For that matter, I think it's better to delay replacement until startLogging is called. Hmm, and why is os.linesep used in the logging? The log file should have been opened in text mode, in which case "\n" will get translated to the proper os-dependent form. I started to change the code, so I could submit a patch. I got the code out of CVS and am reading the documentation at http://twistedmatrix.com/documents/howto/coding-standard It says that the main test suite builder is in 'twisted.test.test_all' That module doesn't exist (can't import it) nor is the appropriate file present anywhere. And the hyperlink from that page yields "No Such Resource File not found." It then says Acceptance tests are all automated by the bin/accepttests but that program is in admin/accepttests I ran that for my Mac OS X machine, with 'open' as the WEBBROWSER setting (where does the '-b' option go, since open doesn't block .. do I pass it to 'accepttests'? Yes.) and /bin/true as my IRCCLIENT, since I wouldn't know what to do with one even assuming I had one. Anyway, after a while it gave this error /------ |#### |#### Starting mail test. |#### Output should be one email (postmaster(a)foo.bar) |#### and one bounce (postmaster(a)no.such.domain). |#### \------ Running Command: './admin/../bin/mktap mail --domain foo.bar=dump --user postmaster=postmaster --pop 18110' option --pop not a unique prefix Usage: mktap mail [options] Options: -p, --pop3= Port to start the POP3 server on (0 to disable). [default: 8110] -S, --pop3s= Port to start the POP3-over-SSL server on (0 to disable). [default: 0] -s, --smtp= Port to start the SMTP server on (0 to disable). [default: 8025] -r, --relay= relay mail we do not know how to handle to this IP, using the given path as a queue directory -c, --certificate= Certificate file to use for SSL connections -d, --domain= generate an SMTP/POP3 virtual domain which saves to "path" --version --help Display this help and exit. -u, --user= add a user/password to the last specified domains --bounce-to-postmaster undelivered mails are sent to the postmaster This creates a mail.tap file that can be used by twistd. Running Command: './admin/../bin/twistd -f mail.tap' Failed to load application: [Errno 2] No such file or directory: 'mail.tap' Traceback (most recent call last): File "./admin/accepttests", line 271, in ? runAllTests() File "./admin/accepttests", line 247, in runAllTests runMailTest() File "./admin/accepttests", line 204, in runMailTest s = smtplib.SMTP('127.0.0.1', 18026) File "/usr/local/lib/python2.3/smtplib.py", line 240, in __init__ (code, msg) = self.connect(host, port) File "/usr/local/lib/python2.3/smtplib.py", line 302, in connect raise socket.error, msg socket.error: (61, 'Connection refused') Hmmm.... Is 'admin/runtests' the correct script to run? I ran that, and got the following output ======================================================================== ======= FAILURE: testReadLimit (twisted.test.test_policies.ThrottlingTestCase) ------------------------------------------------------------------------ ------- Traceback (most recent call last): File "/Users/dalke/cvses/Twisted/twisted/test/test_policies.py", line 193, in testReadLimit self.assertEquals(tServer.readThisSecond, 0) FailTest: 20 != 0 ======================================================================== ======= FAILURE: testWriteLimit (twisted.test.test_policies.ThrottlingTestCase) ------------------------------------------------------------------------ ------- Traceback (most recent call last): File "/Users/dalke/cvses/Twisted/twisted/test/test_policies.py", line 152, in testWriteLimit self.assert_(abs(p.wrappedProtocol.paused - now - 1.0) < 0.1) FailTest ======================================================================== ======= FAILURE: testPackages (twisted.test.test_setup.CheckingPackagesTestCase) ------------------------------------------------------------------------ ------- Traceback (most recent call last): File "/Users/dalke/cvses/Twisted/twisted/test/test_setup.py", line 44, in testPackages self.failUnlessEqual(l, []) FailTest: ['twisted.bugs', 'twisted.coils', 'twisted.eco', 'twisted.enterprise.userRequests', 'twisted.forum', 'twisted.inetd', 'twisted.issues', 'twisted.issues.ui', 'twisted.issues.ui.templates', 'twisted.lumberjack', 'twisted.metrics', 'twisted.pretzel', 'twisted.protocols.ldap', 'twisted.reality', 'twisted.reality.ui', 'twisted.secsh', 'twisted.test.interactive', 'twisted.web.blog', 'twisted.words.ui', 'twisted.words.ui.gateways'] != [] ======================================================================== ======= FAILURE: testFailing (twisted.test.test_tcp.LoopbackTestCase) ------------------------------------------------------------------------ ------- Traceback (most recent call last): File "/Users/dalke/cvses/Twisted/twisted/test/test_tcp.py", line 171, in testFailing self.assert_(time.time() - start < 0.1) FailTest ======================================================================== ======= ERROR: testUnicodeTolerance (twisted.test.test_xml.MicroDOMTest) ------------------------------------------------------------------------ ------- Traceback (most recent call last): File "/Users/dalke/cvses/Twisted/twisted/trial/unittest.py", line 165, in runOneTest method(testCase) File "/Users/dalke/cvses/Twisted/twisted/test/test_xml.py", line 326, in testUnicodeTolerance ud = microdom.parseString(s.encode('UTF-16')) File "./twisted/web/microdom.py", line 644, in parseString File "./twisted/web/microdom.py", line 609, in parse File "./twisted/protocols/sux.py", line 155, in dataReceived File "./twisted/protocols/sux.py", line 176, in do_begin File "./twisted/protocols/sux.py", line 108, in _parseError ParseError: <xmlfile />:1:1: First char of document ['\xfe'] wasn't < ======================================================================== ======= SKIPPED: test_modules (twisted.test.test_doc.DocCoverage) ------------------------------------------------------------------------ ------- Remove this line when you feel like writing docstrings. ------------------------------------------------------------------------ ------- Ran 494 tests in 227.667s FAILED (failures=4, errors=1, skips=1) Could not import twisted.test.test_conch: No module named Crypto.Cipher So I tried to rewrite the code given that it looks like the self-tests run reasonably well, but then ran into complications with this bit of code # Make sure we have some basic logging setup. This only works in cpython. try: logOwner except NameError: logOwner = LogOwner() ... # Prevent logfile from being erased on reload. This only works in cpython. try: logfile except NameError: logfile = NullFile() logerr = sys.stderr Okay, so we've got some code duplication here, and comment skew. I've fixed that up, and tidied up a few other bits of the code. And for the life of me I can't figure out why the log file should default (when no logging is enabled) to the NullFile rather than sys.stderr -- if you really want the NullFile, just pass it in. So I switched it to use sys.stderr. I then reran the 'admin/runtests' script and got ... some warnings and comments which were being supressed. /Users/dalke/cvses/Twisted/twisted/test/test_delay.py:24: DeprecationWarning: twisted.python.delay is deprecated. Please use reactor methods. from twisted.python import delay /Users/dalke/cvses/Twisted/twisted/protocols/dns.py:46: RuntimeWarning: PyCrypto not available - proceeding with non-cryptographically secure random source Enabling Multithreading. Resolver added ('64.105.156.138', 53) to server list Resolver added ('64.105.132.250', 53) to server list Also, I removed those os.linesep references, and the os module, and the needless string module. And turned 'file_protocol' into '_file_protocol' since it's for internal use only. Why's the Log pickleable? What happens if a cStringIO is passed in instead of a file? Or an outgoing network socket? Or a os.popen("lp")? It looks like Log.synchronized is old code that can be removed. Yes? Why does doc/examples/cursesclient.py set log.logfile directly? Shouldn't it call log.startLogging(log.NullFile()) ? It's the only other piece of code which references log.NullFile. Is 'discardLogs' the true inverse of startLogging? If so, it should add something to restore sys.stdout/sys.stderr if startLogging asked them to be changed. If not, where's the 'stopLogging' function? And 'discard' means to ignore logging? Or should it throw away the log files? The terminology is confusing. Anyway, the diff -u is attached for you all to review and correct. Oh, and with my changes, I now see the expected warning when I start up, and I see two twisted messages ... [Andrew-Dalkes-Computer:~/cvses/Twisted] dalke% python cansmi_rpc.py /usr/local/lib/python2.3/site-packages/openeye/oechem.py:5: RuntimeWarning: Python C API version mismatch for module _oechem: This Python has API version 1012, module _oechem has version 1011. import _oechem twisted.web.server.Site starting on 8080 Starting factory <twisted.web.server.Site instance at 0x640850> which tells me that those two messages ("Site starting" and "Starting factory") were being sent to never-never-land the whole time. Perhaps the latter shouldn't be present? Andrew dalke(a)dalkescientific.com

5 11

[Twisted-Python] win32guireactor
by Thomas Heller April 2, 2003

April 2, 2003

Here's my win32guireactor code, which at least for me, works better than win32eventreactor in my application. I have some problems in my app when using win32eventreactor: If the app displays a message box, no callLater calls are done (because a win32 messagebox has it's own event loop, and doIteration() is not called during this time. There are also other things that the user can do which shows the same behaviour, dragging a scrollbar for example. The second problem is that win32eventreactor uses a lot a cpu time - nearly 100% even on my brandnew 2GHz P4. Looks much like polling behaviour instead of event driven. So I wrote my own reactor implementation, which passes all events (those I've covered so far, reactor.callLater and network events) through a win32 message loop. Deferred's are not yet passed through the loop, so there is still some polling to be done. Looking at the comments at the top of the win32eventreactor code, my win32guireactor module shouldn't have the problems mentioned there. I post this module here asking for peer review and critical comments. This module requires win32all and my ctypes module as well, see the source for details. Thanks, Thomas PS: I tried to run unittests with this and other reactors with the command line py22 bin\trial -p twisted.test.test_internet -r <reactor> -v here are the results: default reactor FAILED (failures=5, errors=18) win32eventreactor twisted.test.test_ftp:FTPClientAndServerTests.testLongFileListings hangs forver. win32guireactor twisted.test.test_ftp:FTPClientAndServerTests.testLongFileListings crashes with this traceback, and then also hangs forever. twisted.test.test_ftp FTPClientAndServerTests testBadLogin ... testLongFileListings ... Traceback (most recent call last): File "C:\sf\Twisted\twisted\internet\win32guireactor.py", line 85, in __wm_timerevent tple.func(*tple.args, **tple.kw) File "C:\sf\Twisted\twisted\test\test_ftp.py", line 124, in errback except self.failureException, e: AttributeError: FTPClientAndServerTests instance has no attribute 'failureException'

2 2

[Twisted-Python] twisted.internet.interfaces.IReactorTime.callAt
by Bob Ippolito April 2, 2003

April 2, 2003

I think it's useful to have a callAt function in the reactor. Itamar mentioned that one of the design goals of the reactor is to have a minimal interface. However, IMHO it should be added to IReactor due to the utility (log rotation, scheduled events, etc.) of such a convenience function and the fact it takes a newbie a lot longer to find something hidden away in a utils package (for example, today, neither Itamar nor I knew that there was a function in t.i.utils to start a process and return a Deferred with the stdout of that process!). The implementation is very simple, it looks like this: def callAt(self, when, f, *args, **kw): """See twisted.internet.interfaces.IReactorTime.callAt. """ return self.callLater(max(0.0, when - time()), f, *args, **kw) def callAt(self, when, callable, *args, **kw): """Call a function at a specified float time @type when: C{float} @param when: The approximate unix time (seconds since the Epoch) to call the callable. It will be called on or after the specified time; immediately if the current or a past time is given. @param callable: the callable object to call later. @param args: the arguments to call it with. @param kw: they keyword arguments to call it with. @returns: An L{IDelayedCall} object that can be used to cancel the scheduled call, by calling its C{cancel()} method. It also may be rescheduled by calling its C{delay()} or C{reset()} methods. """ -bob

1 0

[Twisted-Python] A general note on logging
by Itamar Shtull-Trauring April 2, 2003

April 2, 2003

twisted.python.log is old and crufty. If anyone feels like updating it (requirements include different channels that can be logged to different files), feel free. If you do discuss it on the list first, to make sure everyone has voiced their requirements. -- Itamar Shtull-Trauring http://itamarst.org/ http://www.zoteca.com -- Python & Twisted consulting

1 0

[Twisted-Python] PB+SSH
by Joshua Kifer April 1, 2003

April 1, 2003

Can anyone indicate an example or doc on combining these two pieces? ----------------------------- Democracy is two wolves and a lamb voting on what to have for lunch. Liberty is a well-armed lamb contesting the vote. -- Benjamin Franklin

2 2