[Twisted-Python] Testing with trial, adbapi questions
Greetings,
I've begun moving an application I wrote from asynchat to twisted. I've got basic functionality, I've converted my DB stuff to adbapi and have begun writing unit tests. I have a few questions that I haven't been able to get answered through the API docs, twisted.words protocol code, mailing list posts etc. I'd be very grateful for some assistance!
1. Right now I'm defining my dbPool in my factory init, and running a method to load some data from the DB. I've copied the IRC BasicServerFunctionalityTestCase and modified it for my use:
class BasicServerFunctionalityTestCase(unittest.TestCase): def setUp(self): self.f = StringIOWithoutClosing() self.t = FileWrapperThatWorks(self.f) # I need this to return an IPv4Address! self.p = sserverd.SServerProtocol() self.p.factory = sserverd.SServerFactory(self.p) self.p.makeConnection(self.t) self.p.factory.dbPool.start()
def tearDown(self): self.p.factory.dbPool.close()
def check(self, s): self.assertEquals(self.f.getvalue(), s)
def testSendDeveloperID(self): self.p.lineReceived('{%s:"PG"}\0' % sserverson.base10toN(104)) self.check("test\0")
I've gotten past the database threads hanging open by running start() and close() manually. Now, my database callbacks that load necessary data appear to be getting called AFTER my test runs. I know they're getting called, but when the above test runs, the required data structure is empty. What am I missing here?
2. I'm having a bit of trouble wrapping my head around adbapi. Prior to it, one thing I would do is gather data and return a row, as in, the method "developerExists()" would check the DB and return row[0][0] or something (a bool). Now it seems as though I have to preload this data and check against that. Can I replicate this behavior with adbapi? Is there a better way than having to preload ALL my data up front? How can I get any data from the DB like this:
data = getSomethingFromDB()
I'm suspicious that I need to adopt a new paradigm here, but am having difficulty seeing past the old way of doing things!
3. I have some methods (as I pointed out in #1) that load data. With adbapi, it appears that I have to do this:
def _loadDevelopers(self,results): for result in results: developerID = result[2] ... self.addDeveloper(Developer(authHost,authUrl,developerID,self,rc4key, postHost,postUrl))
def loadDevelopers(self): self.dbPool.runQuery("select * from developers where active=1").addCallback( self._loadDevelopers).addErrback(printError) # Before, I would just load my data here!
Is this correct? i.e. a DB method that gets called, each having a corresponding callback method that does the real work? (This seems sort of a pain to me...) I feel as though I'm close to "getting it" (twisted in general, adbapi) but feel that I have yet to connect some key pieces of information.
4. I was digging through the adbapi code, and it appears that the only type of result set I can get is a tuple (because only cursors are used...). Is there a way I can get a dictionary returned, keys are column names? i.e. a MySqlDB.fetch_row(how=1). I'm just not seeing how I can do this with adbapi or, rather, how I can pass through to the MySqlDB module to get this type of result set.
5. I've always run my application using DJB's daemontools (http://cr.yp.to/daemontools.html). Is there an incredibly compelling reason to switch to using twistd? It seems to have some pretty neat features, but I'm just not 100% sure I want to give up daemontools yet.
Thanks!
Brendon Colby
Brendon Colby wrote:
- I was digging through the adbapi code, and it appears that the only
type of result set I can get is a tuple (because only cursors are used...). Is there a way I can get a dictionary returned, keys are column names? i.e. a MySqlDB.fetch_row(how=1). I'm just not seeing how I can do this with adbapi or, rather, how I can pass through to the MySqlDB module to get this type of result set.
This depends on your DBAPI implementation. psycopg2 and pysqlite have optional support for this. I don't know about you mysql, though. Using any of these is not DBAPI2 compliant, but it's extremely useful.
Bye, Karl
On 6/25/07, Karl Bartel karlb@gmx.net wrote:
Brendon Colby wrote:
- I was digging through the adbapi code, and it appears that the only
type of result set I can get is a tuple (because only cursors are used...). Is there a way I can get a dictionary returned, keys are column names? i.e. a MySqlDB.fetch_row(how=1). I'm just not seeing how I can do this with adbapi or, rather, how I can pass through to the MySqlDB module to get this type of result set.
This depends on your DBAPI implementation. psycopg2 and pysqlite have optional support for this. I don't know about you mysql, though. Using any of these is not DBAPI2 compliant, but it's extremely useful.
Aha! This got me thinking...so I dug into the MySQLdb README file (again). To be able to get a tuple of dictionaries returned while using adbabi/MySQLdb, you simply have to do:
self.dbPool = adbapi.ConnectionPool("MySQLdb",host=host,port=port, user=username,passwd=password, db=database, cursorclass=DictCursor) #<-----
Since MySQLdb implements cursors in Python, one can subclass the base cursor class or use one of the several others provided - in this case DictCursor. This just wasn't obvious to me (never was) in the README!
Brendon
Brendon Colby wrote:
Aha! This got me thinking...so I dug into the MySQLdb README file (again).
Friends don't let friends use MySQL.
Sure, you're not my friend yet, but if this doesn't make you one, I don't know what will: ;-)
MySQL-PostgreSQL comparison http://www.teknico.net/devel/myvspg/index.en.html
On 6/25/07, Nicola Larosa nico@teknico.net wrote:
Friends don't let friends use MySQL.
Interesting comparison. I've always used MySQL and for what I'm doing it probably wouldn't matter either way. The most compelling reason for me to switch would be because I've always wanted to learn how to use Postgres. :)
Brendon
* Brendon Colby brendoncolby@gmail.com [2007-06-25 10:17:24 -0500]:
- I've always run my application using DJB's daemontools
(http://cr.yp.to/daemontools.html). Is there an incredibly compelling reason to switch to using twistd? It seems to have some pretty neat features, but I'm just not 100% sure I want to give up daemontools yet.
I'd suggest doing both; just run twistd with the -n flag (--nodaemon) to stop it from doing the daemonization itself.
I think your biggest hurdle is that you need to get used to using Deferred's in adbapi and any functions/methods that call adbapi indirectly. Unless you use a fast embedded database like Sqlite, you have to use Deferred's because there's network traffic under the covers to the database server. I'll try to include a little advice below for each item:
On 6/25/07, Brendon Colby brendoncolby@gmail.com wrote:
Greetings,
I've begun moving an application I wrote from asynchat to twisted. I've got basic functionality, I've converted my DB stuff to adbapi and have begun writing unit tests. I have a few questions that I haven't been able to get answered through the API docs, twisted.words protocol code, mailing list posts etc. I'd be very grateful for some assistance!
- Right now I'm defining my dbPool in my factory init, and running a
method to load some data from the DB. I've copied the IRC BasicServerFunctionalityTestCase and modified it for my use:
class BasicServerFunctionalityTestCase(unittest.TestCase): def setUp(self): self.f = StringIOWithoutClosing() self.t = FileWrapperThatWorks(self.f) # I need this to return an IPv4Address! self.p = sserverd.SServerProtocol() self.p.factory = sserverd.SServerFactory(self.p) self.p.makeConnection(self.t) self.p.factory.dbPool.start()
def tearDown(self): self.p.factory.dbPool.close() def check(self, s): self.assertEquals(self.f.getvalue(), s) def testSendDeveloperID(self): self.p.lineReceived('{%s:"PG"}\0' % sserverson.base10toN(104)) self.check("test\0")I've gotten past the database threads hanging open by running start() and close() manually. Now, my database callbacks that load necessary data appear to be getting called AFTER my test runs. I know they're getting called, but when the above test runs, the required data structure is empty. What am I missing here?
I think you have a race condition because (I assume) self.p.lineReceivedindirectly did a database insert, and then the self.check did a database select. If that's true, you should make your test use Deferred's like this:
d = defer.succeed(None) d.addCallback(lambda out: self.p.lineReceived('{%s:"PG"}\0' % sserverson.base10toN(104)) d.addCallback(lambda out: self.check("test\0")) return d
You could also write this code using the new defer.inlineCallbacks (needs python 2.5) or the older defer.deferredGenerator, but you might as well get used to using callbacks/errbacks and Deferred's.
2. I'm having a bit of trouble wrapping my head around adbapi. Prior
to it, one thing I would do is gather data and return a row, as in, the method "developerExists()" would check the DB and return row[0][0] or something (a bool). Now it seems as though I have to preload this data and check against that. Can I replicate this behavior with adbapi? Is there a better way than having to preload ALL my data up front? How can I get any data from the DB like this:
data = getSomethingFromDB()
I'm suspicious that I need to adopt a new paradigm here, but am having difficulty seeing past the old way of doing things!
You're right that you need a new paradigm, like this:
def cb(data): # use the data here, in the callback function return getSomethingFromDB().addCallback(cb)
This assumes getSomethingFromDB() returns a Deferred which fires with the result you wanted.
3. I have some methods (as I pointed out in #1) that load data. With
adbapi, it appears that I have to do this:
def _loadDevelopers(self,results): for result in results: developerID = result[2] ... self.addDeveloper (Developer(authHost,authUrl,developerID,self,rc4key, postHost,postUrl))
def loadDevelopers(self): self.dbPool.runQuery("select * from developers where active=1").addCallback( self._loadDevelopers).addErrback(printError) # Before, I would just load my data here!
Is this correct? i.e. a DB method that gets called, each having a corresponding callback method that does the real work? (This seems sort of a pain to me...) I feel as though I'm close to "getting it" (twisted in general, adbapi) but feel that I have yet to connect some key pieces of information.
This is basically correct, though loadDevelopers should probably return the result of the runQuery() call.
4. I was digging through the adbapi code, and it appears that the only
type of result set I can get is a tuple (because only cursors are used...). Is there a way I can get a dictionary returned, keys are column names? i.e. a MySqlDB.fetch_row(how=1). I'm just not seeing how I can do this with adbapi or, rather, how I can pass through to the MySqlDB module to get this type of result set.
If you can't find a MySQL-specific way to get a dictionary, you can always write a generic python function which does so, and then always call that in your code. adbapi doesn't do this for you.
5. I've always run my application using DJB's daemontools
(http://cr.yp.to/daemontools.html). Is there an incredibly compelling reason to switch to using twistd? It seems to have some pretty neat features, but I'm just not 100% sure I want to give up daemontools yet.
You really need to run your twisted app with twistd. That's the abstraction for launching twisted programs -- i.e., you define a variable named "application" in your .tac file and then use twistd to start your app. Of course, there's nothing stopping you from using daemontools on top of it.
Thanks!
Brendon Colby
Cheers, Christian
On Mon, 25 Jun 2007 12:28:23 -0400, Christian Simms christian.simms@gmail.com wrote:
[snip]
I think you have a race condition because (I assume) self.p.lineReceivedindirectly did a database insert, and then the self.check did a database select. If that's true, you should make your test use Deferred's like this:
d = defer.succeed(None) d.addCallback(lambda out: self.p.lineReceived('{%s:"PG"}\0' % sserverson.base10toN(104)) d.addCallback(lambda out: self.check("test\0")) return d
You could also write this code using the new defer.inlineCallbacks (needs python 2.5) or the older defer.deferredGenerator, but you might as well get used to using callbacks/errbacks and Deferred's.
Note that the above requires lineReceived to return a Deferred which only fires after the insert has completed and check to return a Deferred which represents the result of the select (if those are indeed the underlying operations).
Jean-Paul
On 6/25/07, Christian Simms christian.simms@gmail.com wrote:
I think your biggest hurdle is that you need to get used to using Deferred's in adbapi and any functions/methods that call adbapi indirectly. Unless you use a fast embedded database like Sqlite, you have to use Deferred's because there's network traffic under the covers to the database server. I'll try to include a little advice below for each item:
Yep this is precisely what I am attempting to do - get used to using deferreds (properly).
I think you have a race condition because (I assume) self.p.lineReceived indirectly did a database insert, and then the self.check did a database select. If that's true, you should make your test use Deferred's like this:
No - I'm just doing a series of runQuerys() when the server starts, the callbacks of which populate dictionaries with objects. The test sends a string identifier to the server which checks whether or not it exists in one of these dictionaries. When the test runs, the dict is empty. However, the load method IS running, just (seemingly) after the test is run. I guess I'm not sure how to code a test for this, as in, "after the database connection has been made and the data loaded, THEN start running tests."
d = defer.succeed(None) d.addCallback(lambda out: self.p.lineReceived('{%s:"PG"}\0' % sserverson.base10toN(104)) d.addCallback(lambda out: self.check("test\0")) return d
You could also write this code using the new defer.inlineCallbacks (needs python 2.5) or the older defer.deferredGenerator, but you might as well get used to using callbacks/errbacks and Deferred's.
Hrm yes, I'm using Debian stock 2.4, but wouldn't be opposed to hand installing a few things (like twisted). I've done it many times. A quick check of deferredGenerator and inlineCallbacks...these look _exactly_ like what I'm asking for. I might just stick with 2.4 and deferredGenerator.
You're right that you need a new paradigm, like this:
def cb(data): # use the data here, in the callback function return getSomethingFromDB().addCallback(cb)
This assumes getSomethingFromDB() returns a Deferred which fires with the result you wanted.
Yep, that's basically exactly what I'm doing now.
You really need to run your twisted app with twistd. That's the abstraction for launching twisted programs -- i.e., you define a variable named "application" in your .tac file and then use twistd to start your app. Of course, there's nothing stopping you from using daemontools on top of it.
Yep, that's exactly what I had considered doing. It doesn't appear to difficult to convert to using twistd run out of daemontools.
Thanks!
Brendon
On Mon, 25 Jun 2007 14:57:50 -0500, Brendon Colby brendoncolby@gmail.com wrote:
[snip]
Hrm yes, I'm using Debian stock 2.4, but wouldn't be opposed to hand installing a few things (like twisted). I've done it many times. A quick check of deferredGenerator and inlineCallbacks...these look _exactly_ like what I'm asking for. I might just stick with 2.4 and deferredGenerator.
I just want to point out that neither deferredGenerator nor inlineCallbacks does anything _new_ - they only provide an alternate API for defining callbacks and errbacks (an API that lets you use code inline within a single function without defining new functions). If your code doesn't execute in the right order without deferredGenerator, it won't execute in the right order with deferredGenerator either.
Jean-Paul
On 6/25/07, Jean-Paul Calderone exarkun@divmod.com wrote:
I just want to point out that neither deferredGenerator nor inlineCallbacks does anything _new_ - they only provide an alternate API for defining callbacks and errbacks (an API that lets you use code inline within a single function without defining new functions). If your code doesn't execute in the right order without deferredGenerator, it won't execute in the right order with deferredGenerator either.
Yep - I meant that I could definitely use this in my code, but not specifically to fix my unittest issues, which I haven't corrected yet.
Brendon
On 6/25/07, Brendon Colby brendoncolby@gmail.com wrote:
On 6/25/07, Christian Simms christian.simms@gmail.com wrote:
[snip]
I think you have a race condition because (I assume) self.p.lineReceived indirectly did a database insert, and then the self.check did a database select. If that's true, you should make your test use Deferred's like
this:
No - I'm just doing a series of runQuerys() when the server starts, the callbacks of which populate dictionaries with objects. The test sends a string identifier to the server which checks whether or not it exists in one of these dictionaries. When the test runs, the dict is empty. However, the load method IS running, just (seemingly) after the test is run. I guess I'm not sure how to code a test for this, as in, "after the database connection has been made and the data loaded, THEN start running tests."
[snip]
Looks like I mis-assumed your problem, sorry. In the setUp method in your tests, you should return a Deferred which fires when all your data is loaded. This can be a pain if you already organized your code into classes inheriting from twisted.application.service.Service which have startService/stopService calls. It's a pain because trial doesn't create an application object like running inside twistd does. So, in the case of writing unit tests for applications, you can abstract out your initialization code into a method, and then call that method directly in your setUp method.
Cheers, Christian
On 6/26/07, Christian Simms christian.simms@gmail.com wrote:
Looks like I mis-assumed your problem, sorry. In the setUp method in your tests, you should return a Deferred which fires when all your data is loaded. This can be a pain if you already organized your code into classes inheriting from twisted.application.service.Service which have startService/stopService calls. It's a pain because trial doesn't create an application object like running inside twistd does. So, in the case of writing unit tests for applications, you can abstract out your initialization code into a method, and then call that method directly in your setUp method.
OK - I will work on this and see if I can figure it out. Thanks for the tip.
Brendon
On 6/26/07, Christian Simms christian.simms@gmail.com wrote:
Looks like I mis-assumed your problem, sorry. In the setUp method in your tests, you should return a Deferred which fires when all your data is loaded. This can be a pain if you already organized your code into classes inheriting from twisted.application.service.Service which have startService/stopService calls. It's a pain because trial doesn't create an application object like running inside twistd does. So, in the case of writing unit tests for applications, you can abstract out your initialization code into a method, and then call that method directly in your setUp method.
I've spent some time working on this, and I see the logic in doing so (firing a deferred when my data is loaded) but I'm stuck at the point of how to do this. In my factory, I call this method in __init__ to load data:
def loadDevelopers(self): def cb(results): for result in results: self.addDeveloper(Developer(result['auth_host'],result['auth_url'], result['devid'],self,result['key'],result['post_host'], result['post_url'])) return self.dbPool.runQuery("select * from developers where active=1").\ addCallback(cb).addErrback(printError)
I created a tac file and am using twistd to run this and it works great. But in my unit tests, what happens is my tests run before my data is loaded (and fail). I haven't been able to figure out how to tell my tests to run after this data is loaded. Here's my setUp method:
def setUp(self): self.file = StringIOWithoutClosing() self.transport = FileWrapperThatWorks(self.file) self.protocol = sserver.SServerProtocol() self.protocol.factory = sserver.SServerFactory(self.protocol) self.protocol.factory.dbPool.start() self.protocol.makeConnection(self.transport)
def testSendDeveloperID(self): print self.protocol.factory.developers # This dictionary prints as empty - it shouldn't!
I guess I'm not making the connection here. How do I code the deferred in setUp to fire when my data is loaded? Also, if I abstract out this logic to an initialization method, how would I have it fire properly when I run under twistd? Here's my tac file:
factory = SServerFactory(protocol=SServerProtocol) application = service.Application("SServer") sserverService = internet.TCPServer(4000,factory) sserverService.setServiceParent(application)
I appreciate the help!
Brendon
On 7/20/07, Brendon Colby brendoncolby@gmail.com wrote:
On 6/26/07, Christian Simms christian.simms@gmail.com wrote:
Looks like I mis-assumed your problem, sorry. In the setUp method in your tests, you should return a Deferred which fires when all your data is loaded. This can be a pain if you already organized your code into classes inheriting from twisted.application.service.Service which have startService/stopService calls. It's a pain because trial doesn't create an application object like running inside twistd does. So, in the case of writing unit tests for applications, you can abstract out your initialization code into a method, and then call that method directly in your setUp method.
I've spent some time working on this, and I see the logic in doing so (firing a deferred when my data is loaded) but I'm stuck at the point of how to do this. In my factory, I call this method in __init__ to load data:
def loadDevelopers(self): def cb(results): for result in results: self.addDeveloper (Developer(result['auth_host'],result['auth_url'], result['devid'],self,result['key'],result['post_host'], result['post_url'])) return self.dbPool.runQuery("select * from developers where active=1").\ addCallback(cb).addErrback(printError)
I created a tac file and am using twistd to run this and it works great. But in my unit tests, what happens is my tests run before my data is loaded (and fail). I haven't been able to figure out how to tell my tests to run after this data is loaded. Here's my setUp method:
def setUp(self): self.file = StringIOWithoutClosing() self.transport = FileWrapperThatWorks(self.file) self.protocol = sserver.SServerProtocol() self.protocol.factory = sserver.SServerFactory(self.protocol) self.protocol.factory.dbPool.start() self.protocol.makeConnection(self.transport) def testSendDeveloperID(self): print self.protocol.factory.developers # This dictionary prints asempty - it shouldn't!
I guess I'm not making the connection here. How do I code the deferred in setUp to fire when my data is loaded? Also, if I abstract out this logic to an initialization method, how would I have it fire properly when I run under twistd? Here's my tac file:
factory = SServerFactory(protocol=SServerProtocol) application = service.Application("SServer") sserverService = internet.TCPServer(4000,factory) sserverService.setServiceParent(application)
I appreciate the help!
Brendon
Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
To fix your unit tests to "wait" until your data is loaded by your loadDevelopers method, you need to have your setUp method return the Deferred created by loadDevelopers's. There are a bunch of ways to do this, the simplest is probably to have your factory's __init__ method store the Deferred created by loadDevelopers into a member variable (say self.deferred), and then add a new line to the end of your setUp method above:
return self.deferred
This way, each unit test will not start until the Deferred from loadDevelopers fires.
Cheers, Christian
On 7/23/07, Christian Simms christian.simms@gmail.com wrote:
To fix your unit tests to "wait" until your data is loaded by your loadDevelopers method, you need to have your setUp method return the Deferred created by loadDevelopers's. There are a bunch of ways to do this, the simplest is probably to have your factory's __init__ method store the Deferred created by loadDevelopers into a member variable (say self.deferred), and then add a new line to the end of your setUp method above:
return self.deferredThis way, each unit test will not start until the Deferred from loadDevelopers fires.
That worked! I was so close to this over the weekend too. I'm starting to feel as though I need an MIT degree to understand this stuff. :)
Brendon
On 09:06 pm, brendoncolby@gmail.com wrote:
That worked! I was so close to this over the weekend too. I'm starting to feel as though I need an MIT degree to understand this stuff. :)
Just as a data point for you: almost everyone who has worked extensively on the Twisted core is a college dropout. (Although some of us apparently feel obscurely guilty about that and attend night school at Harvard. Not me though! Auto-pedagogy for the win!)
Both my fianc�e and my father are programmers with course 6 degrees from MIT, though. It's doubtful that this helps, since Ying responded to me telling her about this message by saying she can't remember how Twisted works "for more than a day at a time", and to my knowledge, Dad only did one thing with it, and that at a sprint which I did not attend - so I haven't seen the code :).
participants (7)
-
Brendon Colby -
Christian Simms -
glyph＠divmod.com -
Jean-Paul Calderone -
Karl Bartel -
Nicola Larosa -
Tristan Seligmann