[Twisted-Python] Writing unit tests with Trial for Twisted Applications
Hey all, In my recent efforts to update the Beyond code base to use recent, preferred, and non-deprecated Twisted developments such as new cred, I have encountered a few problems in updating the unit tests to match my changes. The unit tests as they stood before my changes used the old pyunit module. After successfully converting the tests on the old code base to use twisted.trial (and fixing a probable bug in the process that pyunit didn't expose), I attempted to get them to work with my changes. That's where the problems started happening. From my experience and looking through the unit tests written for Twisted, I have come up with the following questions. As a background, the client and server that I am testing in Beyond is implemented using PB. 1. Does proper testing necessitate or at least include actually communicating over a port for the client and server? I see this is done for the PB tests and several others, but not for many. 2. Closely related, what is the proper way to connect the client and server? When I made changes, the t.test.test_pb.IOPump that the tests were using didn't seem to work. According to spiv, this may not be the preferred connecting method to use anymore. 3. From that, if I use the twisted.protocols.loopback module, how do I set up callbacks for events such as logging in? As I am using it now, it appears that things just hang and nothing is ever completed. For example, in one of my tests, I have this code, with self.p being my portal and self.c being an instance of my PB client: accum = [] factory = pb.PBClientFactory() deferred = factory.login(UsernamePassword("guest", "guest"), client=self.c) deferred.addCallbacks(accum.append, self.errorReceived) loopback.loopbackTCP(self.p, self.c) It just sits there, not advancing beyond the loopbackTCP call. 4. Regardless of the method used, what is the proper way to have the reactor keep going until there are no more events? I need something analogous to test_pb.IOPump.pump() or flush(), if IOPump is not to be used. trial.unittest.TestCase.runReactor()? 5. When is the proper time to use unittest.deferredResult? I see it used in some cases, yet in other cases regular callbacks and errbacks are used. 6. Realted to that, when I added callbacks and errbacks, they didn't seem to get executed. Yet looking at test_newcred.py, on line 89 for example, they are added and then immediately the result is checked. Does this work because I am listening on a port? Or was my doing this not working originally because the client and server were not connected? Later in the same file deferredResult is used. I'm not understanding what the expected behavior is then. 7. Is it preferred to use unittest.TestCase.assertEqual to the plain assert? I see both used in various places in the Twisted tests. I hope these questions make sense. I felt the mailing list was a better forum for this than IRC because I had detailed questions. Also, would it be appropriate for general cases of answers to these questions (especially ones like #5 and #7) to be added to the "Unit Tests in Twisted" howto? I would be willing to put those appropriate into there after someone answers my questions. :) Thanks a million, and slowly learning, Travis
On Wed, Oct 01, 2003 at 08:24:35PM -0600, Travis B. Hartwell wrote:
1. Does proper testing necessitate or at least include actually communicating over a port for the client and server? I see this is done for the PB tests and several others, but not for many.
Well, this is actually usually the easiest way to test your code; otherwise, you need to deal with crap like fake transports, etc. If you do test over the network, make sure you listen on a local-only interface (i.e., '127.0.0.1') and use a portno of 0. e.g.: self.port = reactor.listenTCP(0, self.svr, interface="127.0.0.1") self.portno = self.port.getHost()[-1]
2. Closely related, what is the proper way to connect the client and server? When I made changes, the t.test.test_pb.IOPump that the tests were using didn't seem to work. According to spiv, this may not be the preferred connecting method to use anymore.
Well, if you do what I said above, then you just need to use regular connectTCP(self.portno, fac).
3. [moot question deleted]
4. Regardless of the method used, what is the proper way to have the reactor keep going until there are no more events? I need something analogous to test_pb.IOPump.pump() or flush(), if IOPump is not to be used.
trial.unittest.TestCase.runReactor()?
Well, you don't really need "until there are no more events" -- usually, you just run the reactor until some state is reached, e.g.: while not l: reactor.iterate() assuming that something triggered by running the reactor will eventually make `l' True (e.g., if l started as an empty list, by appending something to it). runReactor is only useful for timing-related tests.
5. When is the proper time to use unittest.deferredResult? I see it used in some cases, yet in other cases regular callbacks and errbacks are used.
Well, any time you just want to get the result of a Deferred and don't want to add a callback, you can use deferredResult. Some people writing tests that use callbacks might not have known about deferredResult, or don't like it, or needed to take advantage of callback chaining, or whatever.
6. Realted to that, when I added callbacks and errbacks, they didn't seem to get executed. Yet looking at test_newcred.py, on line 89 for example, they are added and then immediately the result is checked. Does this work because I am listening on a port? Or was my doing this not working originally because the client and server were not connected? Later in the same file deferredResult is used. I'm not understanding what the expected behavior is then.
Well, it's probably because you're not running the reactor properly, or something. I can't say more without seeing the code :)
7. Is it preferred to use unittest.TestCase.assertEqual to the plain assert? I see both used in various places in the Twisted tests.
Well, when you use the self.assert* methods, trial marks the tests as "Failures", rather than "Errors", which occur when any other exception is raised from the test. Not a really big difference. Also, assert statements don't get run when python -O is used. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://twistedmatrix.com/users/radix.twistd/
On Wed, Oct 01, 2003 at 10:44:05PM -0400, Christopher Armstrong wrote:
On Wed, Oct 01, 2003 at 08:24:35PM -0600, Travis B. Hartwell wrote:
1. Does proper testing necessitate or at least include actually communicating over a port for the client and server? I see this is done for the PB tests and several others, but not for many.
Well, this is actually usually the easiest way to test your code; otherwise, you need to deal with crap like fake transports, etc. If
It's a shame that fake transports are harder to use; we should be able to do better than that. twisted.protocols.loopback is a good start, though...
you do test over the network, make sure you listen on a local-only interface (i.e., '127.0.0.1') and use a portno of 0. e.g.:
self.port = reactor.listenTCP(0, self.svr, interface="127.0.0.1") self.portno = self.port.getHost()[-1]
This bothers me a little; the details of TCP connections are generally irrelevant to what the test method is trying to test (and TCP, hopefully, is getting fully tested in test_tcp). It'd be nice if this wasn't required. -Andrew.
On Thu, Oct 02, 2003 at 12:53:14PM +1000, Andrew Bennetts wrote:
On Wed, Oct 01, 2003 at 10:44:05PM -0400, Christopher Armstrong wrote:
[snip]
you do test over the network, make sure you listen on a local-only interface (i.e., '127.0.0.1') and use a portno of 0. e.g.:
self.port = reactor.listenTCP(0, self.svr, interface="127.0.0.1") self.portno = self.port.getHost()[-1]
This bothers me a little; the details of TCP connections are generally irrelevant to what the test method is trying to test (and TCP, hopefully, is getting fully tested in test_tcp). It'd be nice if this wasn't required.
I agree. Only a few of the tests you write should actually go over the network -- note that loopback.loopback() and the rest of the fake transport bit fall into this category!). The bulk of your functionality should be testable in isolation of the network code. If it's not, consider trying to rewrite it so it is. This will lead to simpler, more reliable tests as well as simpler and more modular code. Jp -- "The problem is, of course, that not only is economics bankrupt but it has always been nothing more than politics in disguise ... economics is a form of brain damage." -- Hazel Henderson
Jp Calderone wrote:
Only a few of the tests you write should actually go over the network -- note that loopback.loopback() and the rest of the fake transport bit fall into this category!). The bulk of your functionality should be testable in isolation of the network code. If it's not, consider trying to rewrite it so it is. This will lead to simpler, more reliable tests as well as simpler and more modular code.
The ideal situation in this case would be to have a briefer answer like "Use trial.prototest" or something. While I appreciate the "Unit" aspect of unit tests (test only what you're testing) it's still good to actually exercise a little network code under the covers, since the OS can do difficult-to-replicate-and-test things in different environments. I can't think of an easy way to do this, but there should be a 'listenTest' utility function which can listen over TCP or SSL, or over a loopback connection that doesn't consume OS resources, based on a switch to the trial commandline. That way we can exercise both interactions of the higher levels with the lower levels and the higher levels in effective isolation. It occurs to me that a test reactor package would be useful for more than that - testReactor.run() could monitor the event loop for state changes - if no data is sent or received in a given iteration, then you know that the reactor is going to hang and you can raise a test failure to that effect rather than consuming CPU and hiding the traceback. (Am I making sense here?)
Christopher Armstrong <radix@twistedmatrix.com> writes:
On Wed, Oct 01, 2003 at 08:24:35PM -0600, Travis B. Hartwell wrote:
1. Does proper testing necessitate or at least include actually communicating over a port for the client and server? I see this is done for the PB tests and several others, but not for many.
Well, this is actually usually the easiest way to test your code; otherwise, you need to deal with crap like fake transports, etc. If you do test over the network, make sure you listen on a local-only interface (i.e., '127.0.0.1') and use a portno of 0. e.g.:
self.port = reactor.listenTCP(0, self.svr, interface="127.0.0.1") self.portno = self.port.getHost()[-1]
2. Closely related, what is the proper way to connect the client and server? When I made changes, the t.test.test_pb.IOPump that the tests were using didn't seem to work. According to spiv, this may not be the preferred connecting method to use anymore.
Well, if you do what I said above, then you just need to use regular connectTCP(self.portno, fac).
The combination of this answer and the above worked just great. Thanks!
3. [moot question deleted]
4. Regardless of the method used, what is the proper way to have the reactor keep going until there are no more events? I need something analogous to test_pb.IOPump.pump() or flush(), if IOPump is not to be used.
trial.unittest.TestCase.runReactor()?
Well, you don't really need "until there are no more events" -- usually, you just run the reactor until some state is reached, e.g.:
while not l: reactor.iterate()
This worked great for some cases. Sort of. It doesn't quite approximate what pump does so I'm a bit at a loss. I have cases where I don't know what condition to test, or what I am testing might be the only condition. For example: def testCacheObjectToClient(self): myObj = SimObject(id=12345) self.rlm.addSimObject(myObj) myPresence = self.rlm.presences[1] myPresence.addObjectToClientSim(myObj) # Crank assert self.c.localCache[12345], 'Object not received!' The # Crank comment is where self.pump.flush() was located before. If I look based on if there is something in localCache, I would keep looping if the object was not received. This is the very thing I am testing, so when it should fail it would just sit there endlessly instead of failing. So what else could I try? -- Thanks for the answers to the rest of my questions. I am slowly getting this. Travis
Hi, In my current project I'm using python's ftplib to transfer files. This is proving very slow and I need to improve this. I looked at Twisted's protocol.ftp.FTPClient some time ago and there seemed to be no support for uploading files. After trawling through the comp.lang.python news group I saw that this had changed in 1.0.4(?), so I decided to try FTPClient again. I've attached a modified version of the ftpclient.py example I'm using. When I run this I login my ftp server ok, do the pwd ok & then proceed to the storeFile and nothing seems to happen. When I look at the ftp server I find an empty file with the correct name however the store operation never seems to finish. I confess I really don't know how to handle the two deferred object that get returned by storeFile. I just add callback/ errorback for each, this could be the problem. Can anyone enlighten me as to what I'm doing wrong. I would really like to use proper async behavior for doing the uploads, instead of using threads + ftplib. Thanks, om P.S. my system: osx, python 2.2.3, twisted 1.0.7. -- Oisin Mulvihill Engines Of Creation Email: oisin@enginesofcreation.ie Work: +353 1 6791602 Mobile: +353 868191540 # Twisted, the Framework of Your Internet # Copyright (C) 2001 Matthew W. Lefkowitz # # This library is free software; you can redistribute it and/or # modify it under the terms of version 2.1 of the GNU Lesser General Public # License as published by the Free Software Foundation. # # This library is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public # License along with this library; if not, write to the Free Software # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA """ An example of using the FTP client """ # Twisted imports from twisted.protocols.ftp import FTPClient, FTPFileListProtocol from twisted.internet.protocol import Protocol, ClientCreator from twisted.python import usage from twisted.internet import reactor # Standard library imports import string import sys try: from cStringIO import StringIO except ImportError: from StringIO import StringIO # Define some callbacks def success(response): print 'Success! Got response:' print '---' if response is None: print None else: print string.join(response, '\n') print '---' def fail(error): print 'Failed. Error was:' print error from twisted.internet import reactor reactor.stop() class Options(usage.Options): optParameters = [['host', 'h', ''], ['port', 'p', 1221], ['username', 'u', 'anonymous'], ['password', None, 'anonumous@example.com'], ['passive', None, 0], ['debug', 'd', 1], ] def run(): # Get config config = Options() config.parseOptions() config.opts['port'] = int(config.opts['port']) config.opts['passive'] = int(config.opts['passive']) config.opts['debug'] = int(config.opts['debug']) # Create the client FTPClient.debug = config.opts['debug'] creator = ClientCreator(reactor, FTPClient, config.opts['username'], config.opts['password'], passive=config.opts['passive']) creator.connectTCP(config.opts['host'], config.opts['port']).addCallback(connectionMade) print "**** Here 1 -" reactor.run() print "- Here 2 ****" def connectionMade(ftpClient): # Get the current working directory try: print "1.0" ftpClient.pwd().addCallbacks(success, fail) print "1.1" filename = 'myfile.txt' abc = ftpClient.storeFile(filename) print "1.2 - abc:", abc abc[0].addCallbacks(success, fail) abc[1].addCallbacks(success, fail) print "1.3" except: print "Exception -", sys.exc_value reactor.stop() # this only runs if the module was *not* imported if __name__ == '__main__': run() Hello.Was this file transferred?
Oisin Mulvihill wrote:
I confess I really don't know how to handle the two deferred object that get returned by storeFile. I just add callback/ errorback for each, this could be the problem.
If you want to wait for both, use twisted.internet.defer.DeferredList.
Can anyone enlighten me as to what I'm doing wrong. I would really like to use proper async behavior for doing the uploads, instead of using threads + ftplib.
The FTP implementation in Twisted is sub-optimal (although that's mostly the server). If you don't get any satisfactory responses I recommend assigning a bug to spiv in the tracker :).
participants (6)
-
Andrew Bennetts -
Christopher Armstrong -
Glyph Lefkowitz -
Jp Calderone -
Oisin Mulvihill -
Travis B. Hartwell