[Twisted-Python] SQLAlchemy, Twisted, and sAsync
I have recently started a project using Twisted with a MySQL backend. From previous posts to this and other lists, it seems that ORMs and threads don't get along too well... and, as far as I can tell, there's no way to get away from threads if you don't want longish queries to block your entire application. At this point, I am using SQLAlchemy's SQL builder functionality (but not ORM layer) and using sAsync to manage the threading. Does anyone have experience with this technique and, if so, with what results? It took me quite some time to piece together everything I could find related to sAsync (which seems to be a dead but functional project), so I threw up a quick Trac page for it at http://sasync.org. Thanks in advance for any feedback! Matthew
Matthew Williams wrote:
From previous posts to this and other lists, it seems that ORMs and threads don't get along too well...
What makes you think that?
and, as far as I can tell, there's no way to get away from threads if you don't want longish queries to block your entire application.
Right, SQLAlchemy doesn't play nicely with *non-threaded* environments, from my understanding, which may well be wrong ;-)
It took me quite some time to piece together everything I could find related to sAsync (which seems to be a dead but functional project), so I threw up a quick Trac page for it at http://sasync.org.
Cool. What is it you're doing that needs to mix Twisted and SQLAlchemy? cheers, Chris PS: I've CC'ed the SQLAlchemy list in as well as I think people there may want to add to this discussion... -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
On Mar 26, 2010, at 3:20 AM, Chris Withers wrote:
Matthew Williams wrote:
From previous posts to this and other lists, it seems that ORMs and threads don't get along too well...
What makes you think that?
First of all, most of my impressions about ORMs come from SQLAlchemy. This quote from this list (http://twistedmatrix.com/pipermail/twisted-python/2009-March/019359.html ) sums up what I have found as well: "It's much trickier if you want to use the ORM, unless you are very careful to fully eager load every thing in any possible database operation if you have need of the information subsequently in your twisted code. Otherwise you may block unexpectedly simply when accessing your objects, and end up with database operations from the main twisted thread." So perhaps I should have said "SQL Alchemy's ORM and threads don't get along too well"... that's not to say it's impossible, you just have to be exceedingly careful how you use it.
and, as far as I can tell, there's no way to get away from threads if you don't want longish queries to block your entire application.
Right, SQLAlchemy doesn't play nicely with *non-threaded* environments, from my understanding, which may well be wrong ;-)
It took me quite some time to piece together everything I could find related to sAsync (which seems to be a dead but functional project), so I threw up a quick Trac page for it at http://sasync.org .
Cool. What is it you're doing that needs to mix Twisted and SQLAlchemy?
The project (an internal project) doesn't really *need* to mix them... I could just use mysqldb. Matthew
Matthew Williams wrote:
"It's much trickier if you want to use the ORM, unless you are very careful to fully eager load every thing in any possible database operation if you have need of the information subsequently in your twisted code. Otherwise you may block unexpectedly simply when accessing your objects, and end up with database operations from the main twisted thread."
So perhaps I should have said "SQL Alchemy's ORM and threads don't get along too well"... that's not to say it's impossible, you just have to be exceedingly careful how you use it.
I think you have the wrong end of the stick. SQLAlchemy and threads play fine, SQLAlchemy and Twisted's asynchronous model, which refuses to use threads on principle, do not, for the reasons you describe.
It took me quite some time to piece together everything I could find related to sAsync (which seems to be a dead but functional project), so I threw up a quick Trac page for it at http://sasync.org.
Cool. What is it you're doing that needs to mix Twisted and SQLAlchemy?
The project (an internal project) doesn't really *need* to mix them... I could just use mysqldb.
Heh, wrong end of the stick again; my question was why you needed to use Twisted ;-) cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
participants (2)
-
Chris Withers
-
Matthew Williams