[Twisted-Python] Merging databases the Twisted way
Hi, I got a server running a SQLite-database. Sometimes I need to merge data from another SQLite database into the master. So far I've done blocking old-style read-some from source, write-some to destination stuff using the standard pysqlite-database module, but as mentioned, it blocks the server. AFAIK SQLite only allows one connection to the database as well so that might make it harder to implement a non-blocking solution. If anybody has tips/hints or examples directly related to this or any documentation on DB-operations using Twisted that's a bit more detailed and rich than the one on www.twistedmatrix.com I'd be very grateful. I'm having a hard time getting the hang of how to use twisted and databases in a non-blocking way. Everything seems overly complicated when done the Twisted way. Perhaps it's just me ... NB! I mostly do web-server stuff using SQLite as database. -- Mvh/Best regards, Thomas Weholt http://www.weholt.org
twisted.enterprise.adbapi is the traditional way - it presents a non-blocking API to databases by using a thread-pool. http://itamarst.org/writings/etech04/twisted_internet-91.html has some examples.
On Wed, Oct 13, 2004 at 02:36:18PM +0200, Thomas Weholt wrote:
If you haven't done so already, maybe you can try to ATTACH (SQLITE SQL command) the other database and then merge using SQL commands. This should be faster and easier than going through Python.
Basically, one open transaction per database. Concurrent access will be blocked, for as long as the timeout parameter of the connect() call. With PySQLite2/SQLite3 it will be much better, because SQLite3 has much more fine-grained locking. SQLite also features an API call to register a callback that will be invoked every n SQLite VM operations. I'll wrap it one day, maybe the Twisted folks can make some use of it for a new SQLiteReactor or whatever. -- Gerhard (subscribed, but still hasn't used Twisted for real, yet)
twisted.enterprise.adbapi is the traditional way - it presents a non-blocking API to databases by using a thread-pool. http://itamarst.org/writings/etech04/twisted_internet-91.html has some examples.
On Wed, Oct 13, 2004 at 02:36:18PM +0200, Thomas Weholt wrote:
If you haven't done so already, maybe you can try to ATTACH (SQLITE SQL command) the other database and then merge using SQL commands. This should be faster and easier than going through Python.
Basically, one open transaction per database. Concurrent access will be blocked, for as long as the timeout parameter of the connect() call. With PySQLite2/SQLite3 it will be much better, because SQLite3 has much more fine-grained locking. SQLite also features an API call to register a callback that will be invoked every n SQLite VM operations. I'll wrap it one day, maybe the Twisted folks can make some use of it for a new SQLiteReactor or whatever. -- Gerhard (subscribed, but still hasn't used Twisted for real, yet)
participants (3)
-
Gerhard Haering -
Itamar Shtull-Trauring -
Thomas Weholt