I think your biggest hurdle is that you need to get used to using Deferred's in adbapi and any functions/methods that call adbapi indirectly. Unless you use a fast embedded database like Sqlite, you have to use Deferred's because there's network traffic under the covers to the database server. I'll try to include a little advice below for each item: On 6/25/07, Brendon Colby <brendoncolby@gmail.com> wrote:
Greetings,
I've begun moving an application I wrote from asynchat to twisted. I've got basic functionality, I've converted my DB stuff to adbapi and have begun writing unit tests. I have a few questions that I haven't been able to get answered through the API docs, twisted.words protocol code, mailing list posts etc. I'd be very grateful for some assistance!
1. Right now I'm defining my dbPool in my factory init, and running a method to load some data from the DB. I've copied the IRC BasicServerFunctionalityTestCase and modified it for my use:
class BasicServerFunctionalityTestCase(unittest.TestCase): def setUp(self): self.f = StringIOWithoutClosing() self.t = FileWrapperThatWorks(self.f) # I need this to return an IPv4Address! self.p = sserverd.SServerProtocol() self.p.factory = sserverd.SServerFactory(self.p) self.p.makeConnection(self.t) self.p.factory.dbPool.start()
def tearDown(self): self.p.factory.dbPool.close()
def check(self, s): self.assertEquals(self.f.getvalue(), s)
def testSendDeveloperID(self): self.p.lineReceived('{%s:"PG"}\0' % sserverson.base10toN(104)) self.check("test\0")
I've gotten past the database threads hanging open by running start() and close() manually. Now, my database callbacks that load necessary data appear to be getting called AFTER my test runs. I know they're getting called, but when the above test runs, the required data structure is empty. What am I missing here?
I think you have a race condition because (I assume) self.p.lineReceivedindirectly did a database insert, and then the self.check did a database select. If that's true, you should make your test use Deferred's like this: d = defer.succeed(None) d.addCallback(lambda out: self.p.lineReceived('{%s:"PG"}\0' % sserverson.base10toN(104)) d.addCallback(lambda out: self.check("test\0")) return d You could also write this code using the new defer.inlineCallbacks (needs python 2.5) or the older defer.deferredGenerator, but you might as well get used to using callbacks/errbacks and Deferred's. 2. I'm having a bit of trouble wrapping my head around adbapi. Prior
to it, one thing I would do is gather data and return a row, as in, the method "developerExists()" would check the DB and return row[0][0] or something (a bool). Now it seems as though I have to preload this data and check against that. Can I replicate this behavior with adbapi? Is there a better way than having to preload ALL my data up front? How can I get any data from the DB like this:
data = getSomethingFromDB()
I'm suspicious that I need to adopt a new paradigm here, but am having difficulty seeing past the old way of doing things!
You're right that you need a new paradigm, like this: def cb(data): # use the data here, in the callback function return getSomethingFromDB().addCallback(cb) This assumes getSomethingFromDB() returns a Deferred which fires with the result you wanted. 3. I have some methods (as I pointed out in #1) that load data. With
adbapi, it appears that I have to do this:
def _loadDevelopers(self,results): for result in results: developerID = result[2] ... self.addDeveloper (Developer(authHost,authUrl,developerID,self,rc4key, postHost,postUrl))
def loadDevelopers(self): self.dbPool.runQuery("select * from developers where active=1").addCallback( self._loadDevelopers).addErrback(printError) # Before, I would just load my data here!
Is this correct? i.e. a DB method that gets called, each having a corresponding callback method that does the real work? (This seems sort of a pain to me...) I feel as though I'm close to "getting it" (twisted in general, adbapi) but feel that I have yet to connect some key pieces of information.
This is basically correct, though loadDevelopers should probably return the result of the runQuery() call. 4. I was digging through the adbapi code, and it appears that the only
type of result set I can get is a tuple (because only cursors are used...). Is there a way I can get a dictionary returned, keys are column names? i.e. a MySqlDB.fetch_row(how=1). I'm just not seeing how I can do this with adbapi or, rather, how I can pass through to the MySqlDB module to get this type of result set.
If you can't find a MySQL-specific way to get a dictionary, you can always write a generic python function which does so, and then always call that in your code. adbapi doesn't do this for you. 5. I've always run my application using DJB's daemontools
(http://cr.yp.to/daemontools.html). Is there an incredibly compelling reason to switch to using twistd? It seems to have some pretty neat features, but I'm just not 100% sure I want to give up daemontools yet.
You really need to run your twisted app with twistd. That's the abstraction for launching twisted programs -- i.e., you define a variable named "application" in your .tac file and then use twistd to start your app. Of course, there's nothing stopping you from using daemontools on top of it. Thanks!
Brendon Colby
Cheers, Christian