Hi! What is the best way to implement synchronous DB access in a twisted application? I've read about defer.deferredGenerator, and various threads about how it's a stupid idea to even attempt this. Do I have restructure my code such that everything that depends on results from a database query is in a callback to a deferred object? -martin
What is the best way to implement synchronous DB access in a twisted application? I've read about defer.deferredGenerator, and various
On Tue, 25 Nov 2008 00:28:21 -0500, "Martin Bright" <Martin.Bright@casero.com> wrote: threads
about how it's a stupid idea to even attempt this. Do I have restructure my code such that everything that depends on results from a database query is in a callback to a deferred object?
You can use runInteraction(). This will execute your DB query in a separate thread and in a transaction.
Hi I thought the twisted enterprise was meant to be more suited... http://twistedmatrix.com/projects/core/documentation/howto/enterprise.html (Sorry Martin for hijacking your question but I think we want to know the same thing). Anyway, the method on this page sounds good but I wasn't actually sure how I fit this in with the renderers. Especially, there are some SQL queries I want to do before rendering anything on the page and perhaps before even sending HTTP header information. If I am calling the deffered from renderHTTP how do I use the result within renderHTTP. An example would be nice :) Regards, AW On Tue, Nov 25, 2008 at 12:18 PM, Vincent Bernat <bernat@luffy.cx> wrote:
On Tue, 25 Nov 2008 00:28:21 -0500, "Martin Bright" <Martin.Bright@casero.com> wrote:
What is the best way to implement synchronous DB access in a twisted application? I've read about defer.deferredGenerator, and various threads about how it's a stupid idea to even attempt this. Do I have restructure my code such that everything that depends on results from a database query is in a callback to a deferred object?
You can use runInteraction(). This will execute your DB query in a separate thread and in a transaction.
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
Abdul-Wahid Paterson wrote:
Hi
I thought the twisted enterprise was meant to be more suited...
http://twistedmatrix.com/projects/core/documentation/howto/enterprise.html
(Sorry Martin for hijacking your question but I think we want to know the same thing). Anyway, the method on this page sounds good but I wasn't actually sure how I fit this in with the renderers. Especially, there are some SQL queries I want to do before rendering anything on the page and perhaps before even sending HTTP header information. If I am calling the deffered from renderHTTP how do I use the result within renderHTTP.
An example would be nice :)
That's a pretty elementary question. Perhaps someone could add a simple example like this to the twisted.web docs? from twisted.web import resource, server from twisted.enterprise import adbapi mypool = adbapi.ConnectionPool(...) class page(resource.Resource) def render_GET(self, request): d = mypool.runInteraction(self.query, request_args) d.addCallback(self.ok, request) d.addErrback(self.err, request) return server.NOT_DONE_YET def query(self, cursor, request_args): """I run in a thread. Do database stuff here""" cursor.execute(...) return cursor.fetchall() def ok(self, results, request): request.write("""<html>...""") for row in results: request.write(...row...) request.finish() def ok(self, failure, request): request.write("""<html><body><h2>Error</h2>""") request.write(failure.getErrorMessage()) request.write("""</body></html>""") request.finish()
Martin Bright wrote:
Hi!
What is the best way to implement synchronous DB access in a twisted application?
For newly created software I would use Divmod Axiom. It uses SQLite as a backend. For already existing databases (in MySQL or PostgreSQL) I would use Canonical Storm. It even has a branch for integreation with Twisted, which uses separate thread to access database.
Do I have restructure my code such that everything that depends on results from a database query is in a callback to a deferred object?
I believe it was Glyph Lefkowitz who once said, that he finds asynchronous data access APIs extremely painful (and that's why Divmod Axiom has synchronous API). I could not agree more; I think this is also a concern for you, that's why you asked this question. Even if using SQL with Twisted is not a problem, basing on my experience I would rather suggest one of mentioned approaches, than anything else. -- m
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi there, Michał Pasternak wrote:
Martin Bright wrote:
Hi!
What is the best way to implement synchronous DB access in a twisted application?
For newly created software I would use Divmod Axiom. It uses SQLite as a backend. For already existing databases (in MySQL or PostgreSQL) I would use Canonical Storm. It even has a branch for integreation with Twisted, which uses separate thread to access database. Has anyone tested this branch? I'd love to see a smallish usage example.
Do I have restructure my code such that everything that depends on results from a database query is in a callback to a deferred object?
I believe it was Glyph Lefkowitz who once said, that he finds asynchronous data access APIs extremely painful (and that's why Divmod Axiom has synchronous API). I could not agree more; I think this is also a concern for you, that's why you asked this question. Even if using SQL with Twisted is not a problem, basing on my experience I would rather suggest one of mentioned approaches, than anything else.
AFAIK some of the db interfaces (at least MysqlDB) will not block other threads when running in one so it makes a lot of sense to throw those operations into a separate thread. I usually try to separate out the code doing db operations into threads to keep the application responsive - instead of just the call to the db operation. This makes for easier coding but with more code in the thread. regards, Tarjei
-- m
------------------------------------------------------------------------
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJLSEbYVRKCnSvzfIRAkX7AKCqg3RB2LAAlaGjbZMPYz8m1jws6QCfRic5 KHX3nnnPr+C6rUdwUqY3isg= =4Vdg -----END PGP SIGNATURE-----
On Wed, 2008-11-26 at 11:12 +0100, tarjei wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi there, Michał Pasternak wrote:
Martin Bright wrote:
Hi!
What is the best way to implement synchronous DB access in a
twisted
application?
For newly created software I would use Divmod Axiom. It uses SQLite as a backend. For already existing databases (in MySQL or PostgreSQL) I would use Canonical Storm. It even has a branch for integreation with Twisted, which uses separate thread to access database. Has anyone tested this branch? I'd love to see a smallish usage example.
I have, and I'm a big fan. It's been on my list (of a billion things to do) for a while to put together a mini tutorial for storm.twisted and Nevow... However, Thomas' tests for this branch are really good; they show nice usage examples (for unit tests). d
On Wed, Nov 26, 2008 at 5:12 AM, tarjei <tarjei@nu.no> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi there, Michał Pasternak wrote:
Martin Bright wrote:
Hi!
What is the best way to implement synchronous DB access in a twisted application?
For newly created software I would use Divmod Axiom. It uses SQLite as a backend. For already existing databases (in MySQL or PostgreSQL) I would use Canonical Storm. It even has a branch for integreation with Twisted, which uses separate thread to access database. Has anyone tested this branch? I'd love to see a smallish usage example.
I've been using the twisted-integration branch on a project at work. So far I have used with sqlite (for unit tests) and MySQL and have not found any issues. Per Duncan's comments, the unit tests tests/twistorm.py are the best place to look now to see how to use the API. I've been working on adding some unit tests and minor features + more tests as well, in hopes that we can get this branch merged into storm proper. A small howto might be in order as well - some external documentation doesn't hurt.
Do I have restructure my code such that everything that depends on results from a database query is in a callback to a deferred object?
I believe it was Glyph Lefkowitz who once said, that he finds asynchronous data access APIs extremely painful (and that's why Divmod Axiom has synchronous API). I could not agree more; I think this is also a concern for you, that's why you asked this question. Even if using SQL with Twisted is not a problem, basing on my experience I would rather suggest one of mentioned approaches, than anything else.
AFAIK some of the db interfaces (at least MysqlDB) will not block other threads when running in one so it makes a lot of sense to throw those operations into a separate thread.
I usually try to separate out the code doing db operations into threads to keep the application responsive - instead of just the call to the db operation. This makes for easier coding but with more code in the thread.
twisted-integration uses threads of course - most importantly it couples a single thread to a store (StoreThread), which is the only safe way I know of to do transactions.
On 25 Nov, 02:09 pm, michal.dtz@gmail.com wrote:
I believe it was Glyph Lefkowitz who once said, that he finds asynchronous data access APIs extremely painful (and that's why Divmod Axiom has synchronous API).
This isn't exactly how I feel, but it took me a really long time to frame my response. You can find it here: <http://glyph.twistedmatrix.com/2008/12/databases-and-twisted-when- threads-are.html>
Martin Bright wrote:
What is the best way to implement synchronous DB access in a twisted application? I've read about defer.deferredGenerator, and various threads about how it's a stupid idea to even attempt this. Do I have restructure my code such that everything that depends on results from a database query is in a callback to a deferred object?
I'm using deferredGenerator (and the yummier inlineCallbacks) all over with my database calls. My database is an RDF database that I talk to over HTTP (with my http://projects.bigasterisk.com/sparqlhttp/). The inlineCallbacks API is so great, you won't miss synchronous code at all. My setup doesn't have transactions, which is one issue with twisted db access. I try to arrange my updates so only the last atomic piece matters. If something goes wrong, my worst case should be some orphaned data. Ebay might work like this too: http://martinfowler.com/bliki/Transactionless.html
participants (10)
-
Abdul-Wahid Paterson -
Drew Perttula -
Drew Smathers -
Duncan McGreggor -
glyph＠divmod.com -
Martin Bright -
Michał Pasternak -
Phil Mayers -
tarjei -
Vincent Bernat