Mailman 3 September 2004 - Twisted

[Twisted-Python] retrial, the new-and-improved unittesting framework for twisted
by Jonathan Simms 19 Sep '04

19 Sep '04

Hello twisted people, Many of you know that I've been working on rewriting the twisted-framework's unittesting program "trial". This message is for those of you who don't ;) I've nearly completely redesigned trial in terms of both it's user-facing features, and it's internal workings. In initial testing, it's shown a 4x speed increase over regular trial, running twisted's tests in roughly 220s on a p3-1Ghz machine (as compared to 800+ seconds on the same box). here are some of the features you can look forward to (from the twisted.trial.unittest docstring, so please excuse the epydoc markup): 1. B{Trial now understands deferreds!} - There is no reason to use L{twisted.trial.util.wait} or L{twisted.trial.util.deferredResult}. Write your deferred handling code as you normally would, make your assertions in your callbacks and errbacks, then I{return the deferred from your test method}. Trial will spin the reactor (correctly) and will wait for the results before running the next test. This will allow developers to write more natural-looking tests for their asynchronous code. - there is a new attribute that has been introduced, C{.timeout}. Trial will wait a default 4 seconds for a result from a deferred that is returned from a test method. If you wish to make this value smaller or larger: >>> from twisted.trial.unittest import TestCase >>> from twisted.internet import defer >>> class MyTestCase(TestCase): ... def testThatReturnsADeferred(self): ... return defer.success('Hooray!') ... testThatReturnsADeferred.timeout = 2.8 ... >>> This would cause trial to wait up to 2.8 seconds (quite needlessly in this case) for the deferred to either callback or errback 2. B{Trial is now 100% compatible with new-style classes and zope interfaces} - Some people (the maintainer included), have been bitten in the past by trial's mediocre support for new-style classes (classes which inherit from object). In v2.0, nearly all of the classes that comprise the framework inherit from object, so support is built-in. Whereas before the TestCase finding machinery used a test for inheritance from L{twisted.trial.unittest.TestCase}, the new mechanism tests that L{twisted.trial.interfaces.ITestCaseFactory} is supplied by your class B{type}. You can write a custom TestCase, and trial will detect it and use it as a class to test, if you do: >>> import zope.interface as zi >>> from twisted.trial.interfaces import ITestCaseFactory >>> from twisted.trial.interfaces import ITestCase >>> class MyTestCase(object): ... zi.classProvides(ITestCaseFactory) ... zi.implements(ITestCase) >>> Naturally, the class should actually provide an implementation of L{twisted.trial.interfaces.ITestCase}. - To avoid any possible conflicts (and to provide component de-registration), trial uses it's own private adapter registry, see L{twisted.trial.__init__} for details. - Trial makes use of zope.interface.Interfaces to allow flexibility and adaptation. All objects implement interfaces, and those interfaces are centralized and documented in L{twisted.trial.interfaces}. 3. B{All assert* and fail* methods are now top-level functions of the unittest module} - Previously they were only available as instance-methods on the TestCase. You can now import all of the assert* and fail* variants and use them as functions. This will allow you to use this functionality in helper classes and functions that aren't part of your TestCase (plus less typing ;]) - Note: these methods are no longer part of the ITestCase API, but are provided as a backwards-compatability to classes written to use the original TestCase class. 4. B{The trial script now accepts a --reporter option} - This is to allow for custom reporter classes. If you want to run a trial process remotely, and gain access to the output, or if you would just like to have your reporting formatted differently, you can supply the fully-qualified class name (of a class that implements L{twisted.trial.interfaces.IReporter}) to --reporter, and trial will report results to your class. - The Reporter is now (almost) totally stateless. All stats on the test run are held in the TestSuite and are reported as necessary using the ITestStats interface. This allows for greatly simplified design in the Reporter implementation. - The Reporter API has been greatly simplified by changing the method signatures so that methods are called with a single object that can easily be adapted to provide all information necessary about a given test phase. 5. B{Compatibility for PyUnit tests} - Trial now supports stdlib unittest.TestCase classes transparently. This functionality is unstable, and has not been heavily tested. - Note: Trial accomplishes this by monkey-patching unittest.TestCase in L{twisted.trial.__init__}. - Please report any bugs you find with this feature to the twisted-python mailing list 6. B{Experimental support for doctests} - The trial script now supports a --doctest=[module] option. The argument is a fully-qualified module name, and trial will use a modified version of DocTestSuite to run the doctests it finds. - My support for doctests is broken when using Python 2.4-alpha3, hopefully, i'll get this fixed by the time the first major-release comes out. - Note: you cannot use .skip or .todo attributes with doctests, all tests will be reported as pass/fail - Please report any bugs you find with this feature to the twisted-python mailing list 7. B{expectedAssertions} is no longer supported - it was just too difficult to make radix's clever deferred-doublecheck feature work with this code revision. With his permisison, this feature has been removed. Trial's 'special' attributes: 1. .todo attributes can either be set on the TestCase or on an individual test* method, and indicate that the test is expected to fail. New tests (for which the underlying functionality has not yet been added) should set this flag while the code is being written. Once the feature is added and the test starts to pass, the flag should be removed. 2. Tests of highly-unstable in-development code should consider using .skip to turn off the tests until the code has reached a point where the success rate is expected to be monotonically increasing. 3. Tests that return deferreds may alter the default timeout period of 4.0 seconds by adding a method attribute C{.timeout} which is the number of seconds as a float that trial should wait for a result. I am currently developing this in a branch in the twisted repository. I would like to encourage anyone who is curious to check it out and play with it and its new features. I'd like to make it as bug free as possible before I merge it back into trunk, and the more eyes I can get on it, the better. you can check it out by doing "svn co svn://svn.twistedmatrix.com/svn/Twisted/branches/slyphon/retrial-2" and thank you for your support.... -Jonathan "slyphon" Simms

1 0

[Twisted-Python] Weekly Bug Summary
by James Y Knight 17 Sep '04

17 Sep '04

Bug summary ______________________ Since 2004-09-10 22:45:53.770000: Bugs opened: 9 Bugs closed: 9 Total open bugs: 305 (+0) New / Reopened Bugs ______________________ [WORDS] IRC examples bug fix/improvment (9 hours ago) http://www.twistedmatrix.com/bugs/issue721 opened by yacc cfreactor breaks LoopingCall (yesterday) CLOSED http://www.twistedmatrix.com/bugs/issue717 opened by anthony [MAIL] MaildirMailbox.appendMessage implementation. (4 days ago) http://www.twistedmatrix.com/bugs/issue714 opened by yacc Pluggable Application loaders (yesterday) http://www.twistedmatrix.com/bugs/issue720 opened by exarkun tempfile.mkstemp in twisted.python.compat (4 days ago) http://www.twistedmatrix.com/bugs/issue713 opened by yacc [CORE] better pluggable reactor support for twistd (yesterday) http://www.twistedmatrix.com/bugs/issue719 opened by itamarst [CORE] Allow specifying exit code in generic way (supporting use and non-use of twistd) (yesterday) http://www.twistedmatrix.com/bugs/issue718 opened by itamarst [CONCH] weirdness in conch's connection forwarding (yesterday) http://www.twistedmatrix.com/bugs/issue716 opened by z3p [CORE] No way to override the default signal handler (2 days ago) CLOSED http://www.twistedmatrix.com/bugs/issue715 opened by loic Bugs Closed ______________________ cfreactor breaks LoopingCall (yesterday) http://www.twistedmatrix.com/bugs/issue717 opened by anthony [MAIL] twisted.mail.pop3.POP3 unconditionally chatty (2 weeks ago) http://www.twistedmatrix.com/bugs/issue700 opened by yacc [DOCUMENTATION, RELEASE MANAGEMENT] Twisted 1.3 identifying itself as 1.3.0rc1 (2 months ago) http://www.twistedmatrix.com/bugs/issue641 opened by br1 [DOCUMENTATION, TWISTD] twistd or twistw? (2 months ago) http://www.twistedmatrix.com/bugs/issue640 opened by br1 spec file in Twisted-1.3.0 tarball has wrong version and source information (1 month ago) http://www.twistedmatrix.com/bugs/issue679 opened by jawildman [WEB] SCGI server protocol implementation (1 week ago) http://www.twistedmatrix.com/bugs/issue703 opened by fog Implement spawnProcess for default reactor on Windows (5 months ago) http://www.twistedmatrix.com/bugs/issue591 opened by justinj [CORE] Port t.p.c to Zope3 components (5 months ago) http://www.twistedmatrix.com/bugs/issue572 opened by itamarst [CORE] No way to override the default signal handler (2 days ago) http://www.twistedmatrix.com/bugs/issue715 opened by loic

1 0

[Twisted-Python] PyHeartbeat cookbook recipe
by Nicola Larosa 17 Sep '04

17 Sep '04

My old PyHeartbeat recipe on the Cookbook site, included in the O'Reilly book too, has got a fresh new look, and a new Twisted implementation besides the old threaded one. Its shortness and double implementation may make it a nice first example for those trying to disentangle themselves from those pesky threads. ;-) PyHeartbeat - detecting inactive computers http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/52302 -- Nicola Larosa - nico(a)tekNico.net "Laugh. But you too are going to sound like and old fart one day. And the respect you show or don't show for those that came before you is going to be what you instill in those that come after you." -- EvilTwinSkippy, Slashdot, June 2004

3 4

[Twisted-Python] domish.Element and deepcopy problem
by Arthur Lutz 17 Sep '04

17 Sep '04

Hi, I'm having a problem with domish.Element, here it is (isolated) : from twisted.xish import domish from copy import deepcopy invitation = domish.Element(('jabber:client', 'message')) invitation2 = deepcopy(invitation) gives me : Traceback (most recent call last): File "modules/isolated_bug.py", line 5, in ? invitation2 = deepcopy(invitation) File "/usr/lib/python2.3/copy.py", line 198, in deepcopy rv = reductor(2) TypeError: 'NoneType' object is not callable Python version : Python 2.3.4 (#2, Jul 5 2004, 09:15:05) [GCC 3.3.4 (Debian 1:3.3.4-2)] on linux2 Type "help", "copyright", "credits" or "license" for more information. Hope this is enough for a diagnostic. Thanks a lot for the good work, -- Arthur LUTZ LOGILAB, Paris (France). http://www.logilab.com http://www.logilab.fr http://www.logilab.org Advanced Computing - Knowledge Management - Artificial Intelligence

2 1

[Twisted-Python] Memory leak//problem in twisted write procedures
by Joshua Moore-Oliva 16 Sep '04

16 Sep '04

I believe I have identified a memory problem in twisted. A quick background.. I've been developing a program that has to use mssql for reasons beyond my control. Due to the chaotic nature of mssql tds implementations, I created a program to pipe queries through to a windows box. In the beginning, I was using asyn* for my communications, however asyn* was very inefficient when writing large amounts of data as after each write the string would be spliced to remove the data written. I then moved to twisted to use it's far more efficient buffer implementation, and the performance problems went away. However, I have since been hit with an apparent memory leak. After reading through the twisted source (specifically the write routines in twisted/internet/abstract.py) I believe I have come across the problem. *** the write implementation continually increases it's write databuffer, only freeing up memory when the data buffer is all sent However, successive write operations append data to the data buffer.. So if a program is able to keep the data buffer from ever completely emptying (which mine is) then the data buffer will forever grow, resulting in a memory problem. I have added some output to print out the length of the data buffer and the offset, and after only 10 minutes of running the numbers are already offset == 76906496 dataBuffer len == 120768384 *** Now, reading through the source to fix this problem, the fastest solution (requiring the least change to the existing code) would be to splice//reduce the size of the dataBuffer after offset exceeds a certain number. However, I'm not too intimate with the source.. I will try that above solution to see if it does any good.. otherwise, i think that this is a fairly serious bug that should be addressed by someone more intimate with the source. Thanks, Joshua Moore-Oliva

2 2

[Twisted-Python] Problem with process ?
by Philippe Bouige 14 Sep '04

14 Sep '04

Hello, I have problem with getProcessOutput, I attempt this code (*) and i change the line : return utils.getProcessOutput("ls",["-l", "-a"]) by : return utils.getProcessOutput("ls",["-l, "."]) it's Ok for this... But when i change by : return utils.getProcessOutput("ls",["-l *"]) or : return utils.getProcessOutput("ls",["-l -a"]) or : return utils.getProcessOutput("ls",["-l ."]) and i have allways this error, but I don't undestand ?? *--- Failure #5 --- --- <exception caught here> --- *--- End of Failure #5 --- Traceback (most recent call last): File "/home/foo/twisted/internet/default.py", line 526, in doSelect ....etc.. "/home/foo/twisted/internet/utils.py", line 44, in errReceived self.deferred.errback(failure.Failure(IOError("got stderr"))) exceptions.AttributeError: 'NoneType' object has no attribute 'errback' *--- Failure #9 --- --- <exception caught here> --- *--- End of Failure #9 --- Traceback (most recent call last): File "/home/foo/twisted/internet/default.py", line 526, in doSelect _logrun(selectable, _drdw, selectable, method, dict) ......etc.. File "/home/foo/twisted/internet/utils.py", line 44, in errReceived self.deferred.errback(failure.Failure(IOError("got stderr"))) exceptions.AttributeError: 'NoneType' object has no attribute 'errback' *--- Failure #13 --- --- <exception caught here> --- *--- End of Failure #13 --- Thank you very much for your help. (*)-------------------------------------------------------- from twisted.internet import protocol, reactor, defer, utils from twisted.protocols import basic class FingerProtocol(basic.LineReceiver): def lineReceived(self, user): self.factory.getUser(user ).addCallback(lambda m: (self.transport.write(m+"\r\n\n")) ).addErrback(lambda x : x.printDetailedTraceback()) class FingerFactory(protocol.ServerFactory): protocol = FingerProtocol def getUser(self, user): return utils.getProcessOutput("ls",["-l -a"]) reactor.listenTCP(1079, FingerFactory()) reactor.run() -- Cordialement Philippe

3 3

[Twisted-Python] Weekly Bug Summary
by James Y Knight 13 Sep '04

13 Sep '04

Okay, I made a script that generates this from Roundup -- I'm going to put it in a crontab and have it email out every week from now on. Alternate formatting suggestions welcome. Bug summary ______________________ Since 2004-09-03 22:45:53.767335: Bugs opened: 10 Bugs closed: 5 Total open bugs: 305 (+5) New / Reopened Bugs ______________________ returning None from buildProtocol in ReconnectingClientFactory causes exception (2 hours ago) http://www.twistedmatrix.com/bugs/issue710 opened by djk121 Oscar html function broken (yesterday) CLOSED http://www.twistedmatrix.com/bugs/issue709 opened by jadestorm [CORE] ReactorTime has serious performance issues (yesterday) http://www.twistedmatrix.com/bugs/issue707 opened by stefan Add ClientCreator example to TCP Client howto (yesterday) http://www.twistedmatrix.com/bugs/issue708 opened by mcfletch [DOCUMENTATION] failure.check() docstring (2 days ago) http://www.twistedmatrix.com/bugs/issue706 opened by thomasvs retrial util.deferredError blows up on failure (3 days ago) http://www.twistedmatrix.com/bugs/issue705 opened by warner [CHAT] MSN Protocol bugs (with patches & unit tests) (5 days ago) http://www.twistedmatrix.com/bugs/issue702 opened by delx [XISH] Jabber xmlstream parse bug (with patch and unit test) (5 days ago) http://www.twistedmatrix.com/bugs/issue701 opened by delx SCGI server protocol implementation (3 days ago) http://www.twistedmatrix.com/bugs/issue704 opened by fog SCGI server protocol implementation (3 days ago) http://www.twistedmatrix.com/bugs/issue703 opened by fog Bugs Closed ______________________ Oscar html function broken (yesterday) http://www.twistedmatrix.com/bugs/issue709 opened by jadestorm [PB] callRemote exception handler relies on possibly unbound local `req' (8 months ago) http://www.twistedmatrix.com/bugs/issue459 opened by exarkun [PB] Copyable and Referenceable undefined in pb.py (sandbox) (8 months ago) http://www.twistedmatrix.com/bugs/issue461 opened by exarkun reconnecting ConnectionPools (1 month ago) http://www.twistedmatrix.com/bugs/issue657 opened by davep Error reporting for unslicable objects isn't very informative (8 months ago) http://www.twistedmatrix.com/bugs/issue462 opened by exarkun

3 2

[Twisted-Python] SSH server silently failing to open port
by Jason Stitt 12 Sep '04

12 Sep '04

Hi, I'm still trying to get my SFTP server up. After looking over SFTPGateway in some detail, I think I have an idea of where all the pieces fit, but I'm still having the problem I had before with the sshsimpleserver example. Even though my factory is being instantiated and the log output says it's starting on the port I specified, the port is not, in fact, open to any connections. Because there is no error reported, I'm having a hard time debugging this. I know that my factory's __init__ method completes successfully, and startFactory is also called. After that... nothing. Can anyone help me figure out where to start looking, at least? Thanks, Jason Here's my Factory class, minus a few comments and debug print statements: class Factory(factory.SSHFactory): def __init__(self): self.portal = portal.Portal(Realm()) def makeConnection(*args, **kw): return Connection(*args, **kw) self.services = { 'ssh-userauth': userauth.SSHUserAuthServer, 'ssh-connection': makeConnection, } # TODO: replace this with keys loaded from a file self.__private_keys = { 'ssh-rsa': keys.getPublicKeyString(data=publicKey) } self.__public_keys = { 'ssh-rsa': keys.getPrivateKeyObject(data=privateKey) } def buildProtocol(self, address): # Note: this function does not use its address parameter. I don't know # why it's there. return factory.SSHFactory.buildProtocol(self, address) def getPrivateKeys(self): return self.__private_keys.copy() def getPublicKeys(self): return self.__public_keys.copy()

2 5

[Twisted-Python] Including code under PSF license for backwards compatibility?
by James Y Knight 12 Sep '04

12 Sep '04

Regarding licensing: Is it okay to include Python library code in twisted for purposes of backwards compatibility. Two examples recently: - Heapq is new in python 2.3. Therefore, regardless of whether we modify it or not, if we use it, we either need to depend on python >= 2.3, or include a copy for compatibility purposes. It has a pure python implementation. - yacc on IRC wanted to use tempfile.mkstemp. This only exists on >= 2.3. Again, pure python implementation. Therefore, Glyph: a pronouncement is necessary on whether, and under what conditions, it is acceptable to include PSF-licensed code in twisted. Select one: a) It is always okay. Copy freely from python into twisted. b) It is okay, as long as the modules are included for backwards compatibility only and can be removed as soon as the minimum required version moves up to 2.3 (or whatever the latest and greatest version we need modules from). Also the code under PSF must be segregated in a separate file. c) It is okay <fill in the blank> d) It is never okay. As you can probably tell, my initial inclination is (b). James

2 1

[Twisted-Python] Async Message Queue?
by Tom Samplonius 12 Sep '04

12 Sep '04

Is there an implementation of an async message queue? I would like to be send messages to another server, with the guarrentee that accepted messages can not be lost, if even if the destination server is down. Tom

1 0