[Twisted-Python] db connections
Hi, I've wrote an daemon which does some queries to db and sends response back to client. Do I need to make for every request from client (new instance of MyProtocol) a new connection to DB? Or can I somehow prepare connection, so I could save some time? Maybe make connection in Factory and pass it to Protocol? But what happens if too much clients are connected to server? What is the way to control it? Regards, Pet class MyProtocol(basic.LineReceiver): def __init__(self): print "new connection" self.dbcon = adbapi.ConnectionPool("pyPgSQL.PgSQL", database="data", user='pet', host='local', password='some') class MyFactory(protocol.ServerFactory): protocol = MyProtocol class MyService(internet.TCPServer): def __init__(self): internet.TCPServer.__init__(self,PORT,MyFactory()) def main(): reactor.listenTCP(PORT, MyFactory()) reactor.run() if __name__ == '__main__': main()
Put self.dbcon in the MyFactory class. MyProtocol instances can access it via self.factory. On Tue, Mar 24, 2009 at 6:10 PM, Pet <petshmidt@googlemail.com> wrote:
You'd want to use a connection pool to manage the db interaction. Your server is pretty unusable after a db connection failure. I've used the connection pool before with a cp_min of 1 and a cp_max of 2. http://twistedmatrix.com/documents/8.2.0/api/twisted.enterprise.adbapi.Conne... On Tue, Mar 24, 2009 at 6:45 AM, Alvin Delagon <adelagon@gmail.com> wrote:
Something like this: from twisted.protocols import basic from twisted.internet import protocol, reactor from twisted.enterprise import adbapi class MyProtocol(basic.LineReceiver): def __init__(self): pass def lineReceived(self, line): ### dbcon can be accessed via self.factory print dir(self.factory.dbcon) class MyFactory(protocol.ServerFactory): protocol = MyProtocol def __init__(self): self.dbcon = adbapi.ConnectionPool("pyPgSQL.PgSQL", database="data", user='pet', host='local', password='some') if __name__ == "__main__": reactor.listenTCP(8080, MyFactory()) reactor.run() I usually put persistent data on the factory so that protocol instances can access them such as {username:<protocol instance>} dictionary for chatroom server. On Wed, Mar 25, 2009 at 12:22 AM, Pet <petshmidt@googlemail.com> wrote:
Sorry I apologize... that email I wrote wasn't very clear. I've done it in the factory like below. Slightly different from Alvin. I've used the factory __init__ to pass on the logical_db information... then I have a dbpool created later. That way I can do development and production based on ENV vars. The db.return_kwargs function just returns the appropriate environment args. I was also mentioning the reconnection functionality which by default is turned off. Just make sure you handle ConnectionLost exceptions. Also in the stopFactory I shutdown the dbpool by closing it. You'll also notice the openFun... which i use to set the displayed host name and app name. But this is db vendor specific. def __init__(self, baseDir, validDomains, logical_db, app_name ): ''' factory init ''' self.baseDir = baseDir self.validDomains = validDomains self.logical_db = logical_db self.app_name = app_name def startFactory(self): ''' In startFactory start up the dbpool ''' self.db = db( self.logical_db, self.app_name ) db_kwargs = self.db.return_kwargs() #set the delay_connect and #twisted cp_openfun callable db_kwargs['delay_connect'] = 1 db_kwargs['cp_openfun'] = self.db._dbOptions db_kwargs['cp_min'] = 1 # one thread min db_kwargs['cp_max'] = 1 # one thread max = only one db connection db_kwargs['cp_reconnect'] = 1 # reconnect if die db_kwargs['cp_noisy'] = 1 #print out the db_kwargs print ",".join( ["%s=%s" % (k, v) for k, v in db_kwargs.items()] ) self.dbpool = adbapi.ConnectionPool(self.db.return_import_name(),**db_kwargs) On Tue, Mar 24, 2009 at 10:10 AM, Alvin Delagon <adelagon@gmail.com> wrote:
On Wed, Mar 25, 2009 at 10:22 AM, Pet <petshmidt@googlemail.com> wrote:
Ups! I didn't followed exactly your example and tried to access self.factory.dbcon in __init__ of MyProtocol, so I've got an error. If I access self.factory.dbcon in lineReceived it seems to work. But I don't really understand, why can I access factory in lineReceived and not in __init__. Pet
On Tue, Mar 24, 2009 at 2:45 PM, Alvin Delagon <adelagon@gmail.com> wrote:
Put self.dbcon in the MyFactory class. MyProtocol instances can access it via self.factory.
Thanks for your help! Could you give me an example? I'm getting an error MyProtocol instance has no attribute 'factory' if I do as you suggested.
Put self.dbcon in the MyFactory class. MyProtocol instances can access it via self.factory. On Tue, Mar 24, 2009 at 6:10 PM, Pet <petshmidt@googlemail.com> wrote:
You'd want to use a connection pool to manage the db interaction. Your server is pretty unusable after a db connection failure. I've used the connection pool before with a cp_min of 1 and a cp_max of 2. http://twistedmatrix.com/documents/8.2.0/api/twisted.enterprise.adbapi.Conne... On Tue, Mar 24, 2009 at 6:45 AM, Alvin Delagon <adelagon@gmail.com> wrote:
Something like this: from twisted.protocols import basic from twisted.internet import protocol, reactor from twisted.enterprise import adbapi class MyProtocol(basic.LineReceiver): def __init__(self): pass def lineReceived(self, line): ### dbcon can be accessed via self.factory print dir(self.factory.dbcon) class MyFactory(protocol.ServerFactory): protocol = MyProtocol def __init__(self): self.dbcon = adbapi.ConnectionPool("pyPgSQL.PgSQL", database="data", user='pet', host='local', password='some') if __name__ == "__main__": reactor.listenTCP(8080, MyFactory()) reactor.run() I usually put persistent data on the factory so that protocol instances can access them such as {username:<protocol instance>} dictionary for chatroom server. On Wed, Mar 25, 2009 at 12:22 AM, Pet <petshmidt@googlemail.com> wrote:
Sorry I apologize... that email I wrote wasn't very clear. I've done it in the factory like below. Slightly different from Alvin. I've used the factory __init__ to pass on the logical_db information... then I have a dbpool created later. That way I can do development and production based on ENV vars. The db.return_kwargs function just returns the appropriate environment args. I was also mentioning the reconnection functionality which by default is turned off. Just make sure you handle ConnectionLost exceptions. Also in the stopFactory I shutdown the dbpool by closing it. You'll also notice the openFun... which i use to set the displayed host name and app name. But this is db vendor specific. def __init__(self, baseDir, validDomains, logical_db, app_name ): ''' factory init ''' self.baseDir = baseDir self.validDomains = validDomains self.logical_db = logical_db self.app_name = app_name def startFactory(self): ''' In startFactory start up the dbpool ''' self.db = db( self.logical_db, self.app_name ) db_kwargs = self.db.return_kwargs() #set the delay_connect and #twisted cp_openfun callable db_kwargs['delay_connect'] = 1 db_kwargs['cp_openfun'] = self.db._dbOptions db_kwargs['cp_min'] = 1 # one thread min db_kwargs['cp_max'] = 1 # one thread max = only one db connection db_kwargs['cp_reconnect'] = 1 # reconnect if die db_kwargs['cp_noisy'] = 1 #print out the db_kwargs print ",".join( ["%s=%s" % (k, v) for k, v in db_kwargs.items()] ) self.dbpool = adbapi.ConnectionPool(self.db.return_import_name(),**db_kwargs) On Tue, Mar 24, 2009 at 10:10 AM, Alvin Delagon <adelagon@gmail.com> wrote:
On Wed, Mar 25, 2009 at 10:22 AM, Pet <petshmidt@googlemail.com> wrote:
Ups! I didn't followed exactly your example and tried to access self.factory.dbcon in __init__ of MyProtocol, so I've got an error. If I access self.factory.dbcon in lineReceived it seems to work. But I don't really understand, why can I access factory in lineReceived and not in __init__. Pet
On Tue, Mar 24, 2009 at 2:45 PM, Alvin Delagon <adelagon@gmail.com> wrote:
Put self.dbcon in the MyFactory class. MyProtocol instances can access it via self.factory.
Thanks for your help! Could you give me an example? I'm getting an error MyProtocol instance has no attribute 'factory' if I do as you suggested.
participants (4)
-
Alvin Delagon -
Enrique Samson Jr. -
Pet -
Rob Hoadley