[Twisted-Python] long-lived database transaction
Hi *, attached to this mail there are two files that implement long-lived database transactions for twisted (xdbapi.py, to be dropped in twisted.enterprise) and the necessary unit tests (test_xdbapi.py). some months ago i needed database transactions that span more than a single deferred chain (i.e., runInteraction) and i noting that twisted was missing such a feature i implemented them in my application. now that i have some free time, i though it would be nice to let other people use my code directly from twisted. note that everything you can do with xdbapi can be done with adbapi and long chains of deferred, but for some kind of applications keeping around an open transaction and committing at the end of a complex process is much more simplier. here is a little example, note that all the methods are completely asynchronous and return deferreds. also note that the example just show the usage but not the "reason" for xdbapi. (the original need was that sometimes you can't commit some information because the whole stuff comes in pieces but accumulating it makes the code much more complicated than just writing to the db and the commit the whole at the end.) class MyAppUsingLongLivedTransactions: def __init__(self): self.dbpool = xdbapi.ConnectionPool('psycopg', database='test') def doSomething(self): # react to some user input, open the transaction and save it away, # also do a select and present the user some data d = self.dbpool.begin() d.addCallback(self.doSomething_cb) def doSomething_cb(self, trans): self.trans = trans d = self.trans.runQuery("SELECT ...") d.addCallback(self.doSomethingWithTheDataAndAskUserNextStep) def doSomthingElse(self, data): # the user gave us some data and we need to write to db and commit self.trans.runOperation("INSERT...", data) # no callback/errback? ouch! self.trans.commit() # ouch! again, but this is just an example -- Federico Di Gregorio http://people.initd.org/fog Debian GNU/Linux Developer fog@debian.org INIT.D Developer fog@initd.org L'unica cosa che riesco a produrre con una certa precisione nella mia vita sono i dubbi. -- Natale Titotto
Federico Di Gregorio wrote:
Hi *,
attached to this mail there are two files that implement long-lived database transactions for twisted (xdbapi.py, to be dropped in twisted.enterprise) and the necessary unit tests (test_xdbapi.py).
This looks like a gigantic rape'n'paste of adbapi.py. Isn't there a way you can integrate with/refactor adbapi such that you don't duplicate all that code? Also, please create a "Feature"-level issue on the bug tracker for this patch (especially since the patch needs discussion): http://twistedmatrix.com/bugs/ -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://radix.twistedmatrix.com/
Il sab, 2003-12-13 alle 21:55, Christopher Armstrong ha scritto:
Federico Di Gregorio wrote:
Hi *,
attached to this mail there are two files that implement long-lived database transactions for twisted (xdbapi.py, to be dropped in twisted.enterprise) and the necessary unit tests (test_xdbapi.py).
This looks like a gigantic rape'n'paste of adbapi.py. Isn't there a way you can integrate with/refactor adbapi such that you don't duplicate all that code?
the original code was very different from adbapi, but i wanted to provide a similar interface. that's why i rape'n'pasted. :) unfortunately the adbapi code is based on a *thread* pool and the connection pool is just a side-effect of using one connection per thread (this is completely wrong, imho, see my comments of about one year ago about adbapi on this very list.) anyway, this obviously makes impossible to have long-lived transactions. i would be very happy to convert adbapi to a *real connection pool* and add long-lived transactions to it (very much like xdbapi incorporates one-shot transactions that mimic ConnectionPool.runXXX, look at the code) but i tought it was less invasive to first provide a completely separate implementation, starting with xdbapi.
Also, please create a "Feature"-level issue on the bug tracker for this patch (especially since the patch needs discussion):
ok. let's discuss it and i'll resubmit the patch after. federico -- Federico Di Gregorio http://people.initd.org/fog Debian GNU/Linux Developer fog@debian.org INIT.D Developer fog@initd.org Lasciate che i furetti vengano a me. -- Maria Luisa Benedetta Panzani
On Sat, 2003-12-13 at 13:50, Federico Di Gregorio wrote:
Il sab, 2003-12-13 alle 21:55, Christopher Armstrong ha scritto:
Federico Di Gregorio wrote:
Hi *,
attached to this mail there are two files that implement long-lived database transactions for twisted (xdbapi.py, to be dropped in twisted.enterprise) and the necessary unit tests (test_xdbapi.py).
This looks like a gigantic rape'n'paste of adbapi.py. Isn't there a way you can integrate with/refactor adbapi such that you don't duplicate all that code?
the original code was very different from adbapi, but i wanted to provide a similar interface. that's why i rape'n'pasted. :)
unfortunately the adbapi code is based on a *thread* pool and the connection pool is just a side-effect of using one connection per thread (this is completely wrong, imho, see my comments of about one year ago about adbapi on this very list.) anyway, this obviously makes impossible to have long-lived transactions.
Evidently not impossible, as Bill Gribble has a patch adding this support to the existing adbapi.py. But it still may be simpler to support an extended transaction api by going away from a thread pool. In either case, we should not have two separate modules. dave
Il dom, 2003-12-14 alle 17:48, Dave Peticolas ha scritto: [snip]
Evidently not impossible, as Bill Gribble has a patch adding
eheh. yesss. i should have added "without leaving a thread suspended". my fault.
this support to the existing adbapi.py. But it still may be simpler to support an extended transaction api by going away from a thread pool. In either case, we should not have two separate modules.
agreed. see my later mail; Bill's patch is fine if we can get it in. federico -- Federico Di Gregorio http://people.initd.org/fog Debian GNU/Linux Developer fog@debian.org INIT.D Developer fog@initd.org Viviamo in un mondo reale, Ciccio. -- Lucy
On Sun, 2003-12-14 at 09:21, Federico Di Gregorio wrote:
Il dom, 2003-12-14 alle 17:48, Dave Peticolas ha scritto: [snip]
Evidently not impossible, as Bill Gribble has a patch adding
eheh. yesss. i should have added "without leaving a thread suspended". my fault.
this support to the existing adbapi.py. But it still may be simpler to support an extended transaction api by going away from a thread pool. In either case, we should not have two separate modules.
agreed. see my later mail; Bill's patch is fine if we can get it in.
I'd like to have some tests for it before it goes in. I might have some time over holiday break to work on some. dave
Il lun, 2003-12-15 alle 15:34, Dave Peticolas ha scritto:
On Sun, 2003-12-14 at 09:21, Federico Di Gregorio wrote:
Il dom, 2003-12-14 alle 17:48, Dave Peticolas ha scritto: [snip]
Evidently not impossible, as Bill Gribble has a patch adding
eheh. yesss. i should have added "without leaving a thread suspended". my fault.
this support to the existing adbapi.py. But it still may be simpler to support an extended transaction api by going away from a thread pool. In either case, we should not have two separate modules.
agreed. see my later mail; Bill's patch is fine if we can get it in.
I'd like to have some tests for it before it goes in. I might have some time over holiday break to work on some.
i can do the tests too. can we coordinate over irc or IM? -- Federico Di Gregorio http://people.initd.org/fog Debian GNU/Linux Developer fog@debian.org INIT.D Developer fog@initd.org Viviamo in un mondo reale, Ciccio. -- Lucy
On Sat, 2003-12-13 at 14:55, Christopher Armstrong wrote:
This looks like a gigantic rape'n'paste of adbapi.py. Isn't there a way you can integrate with/refactor adbapi such that you don't duplicate all that code?
Also, please create a "Feature"-level issue on the bug tracker for this patch (especially since the patch needs discussion):
There's already a bug about this, # 303. I submitted a patch for similar functionality a while back. After a couple go-rounds with Dave P, I arrived at a satisfactory solution, which I then dropped the ball on submitting :( I've been using it here for a couple of months with no problems. I just added a few calls to the existing adbapi api to get/commit transactions and to call operations/queries in them. Attached is MY current patch to adbapi.py,, which I'm 92% happy with (there are probably cleaner ways to handle the per-thread work queue). Furthermore there aren't any tests. I'd be happy to do a little more work to clean it up and add tests if it's a decent base to work from. Thanks, b.g.
Il sab, 2003-12-13 alle 22:53, Bill Gribble ha scritto:
There's already a bug about this, # 303. I submitted a patch for similar functionality a while back. After a couple go-rounds with Dave P, I arrived at a satisfactory solution, which I then dropped the ball on submitting :(
sorry, i went over it. :(
I've been using it here for a couple of months with no problems. I just added a few calls to the existing adbapi api to get/commit transactions and to call operations/queries in them.
Attached is MY current patch to adbapi.py,, which I'm 92% happy with
I don't like it very much, mainly because it sticks with the "let's use a thread pool" and then needs semaphores and locks to do what is better done by a simple list (see my patch)... *but* ...it provides what i need(ed) and if it has any possibility of going in i will happily help writing the unit tests. no problems leaving xdbapi out in this case. federico -- Federico Di Gregorio http://people.initd.org/fog Debian GNU/Linux Developer fog@debian.org INIT.D Developer fog@initd.org I terminali seriali sono in via di estinzione (infatti quello che c'era si è estinto) -- Simone Caldana
participants (4)
-
Bill Gribble -
Christopher Armstrong -
Dave Peticolas -
Federico Di Gregorio