Twisted and SQLAlchemy Cache Problem
Hi, When executing the following program, changes in the database (other than those made by this program) aren't visible until it is restarted, due to some form of SQLAlchemy caching: from twisted.web import server, resource import model # my database model class MyRoot(resource.Resource): def render(self, request): m = model.MyModel.get(request.args['id'][0]) return m.some_column # Connect to database. model.metadata.connect('some_dburi') # Create site structure. root = resource.Resource() root.putChild('', MyRoot()) site = server.Site(root) # Switch user, start service. from twisted.application import service, strports application = service.Application("myexample") s = strports.service('tcp:8080', site) s.setServiceParent(application) So if I visit http://localhost:8080/?id=1, then alter the some_column value for the row who's id is 1 and reload, the change is not reflected. Problem is, I need the update to be visible. How would I go about implementing this? Regards, Magnus
Magnus Hult ha scritto:
Hi,
When executing the following program, changes in the database (other than those made by this program) aren't visible until it is restarted, due to some form of SQLAlchemy caching:
Are you using PostgreSQL? If this is the case, then give a look at: http://initd.org/tracker/psycopg/wiki/psycopg2_documentation#setting-transac... psycopg1 has a SERIALIZED default isolation level, psycopg2 has a READ COMMITED default isolation level. The solution is: make sure to add a commit after every select. You can read a discussion with the psycopg author in the it.comp.lang.python newsgroup: http://groups.google.com/group/it.comp.lang.python/browse_thread/thread/bf98...
[...]
Regards Manlio Perillo
Thanks for your swift reply!
On 6/13/07, Manlio Perillo
Magnus Hult ha scritto:
When executing the following program, changes in the database (other than those made by this program) aren't visible until it is restarted, due to some form of SQLAlchemy caching:
Are you using PostgreSQL?
Yes, I am. With psycopg2.
[...] The solution is: make sure to add a commit after every select.
It doesn't seem to help though. In fact, the following little program t = objectstore.create_transaction() m = model.MyModel.get(1) print m.some_column t.commit() sys.stdin.readline() t = objectstore.create_transaction() m = model.MyModel.get(1) print m.some_column t.commit() only seems to execute one select statement. The other result seems to be taken from the cache. If this is the case, then this question would rather belong in an SQLAlchemy mailing list, I guess.
You can read a discussion with the psycopg author in the it.comp.lang.python newsgroup: http://groups.google.com/group/it.comp.lang.python/browse_thread/thread/bf98...
Yes, I could, if I could read Italian. :)
Magnus Hult ha scritto:
Thanks for your swift reply!
On 6/13/07, Manlio Perillo
wrote: Magnus Hult ha scritto:
When executing the following program, changes in the database (other than those made by this program) aren't visible until it is restarted, due to some form of SQLAlchemy caching:
Are you using PostgreSQL?
Yes, I am. With psycopg2.
[...] The solution is: make sure to add a commit after every select.
It doesn't seem to help though. In fact, the following little program
t = objectstore.create_transaction() m = model.MyModel.get(1) print m.some_column t.commit() sys.stdin.readline() t = objectstore.create_transaction() m = model.MyModel.get(1) print m.some_column t.commit()
only seems to execute one select statement. The other result seems to be taken from the cache. If this is the case, then this question would rather belong in an SQLAlchemy mailing list, I guess.
The problem can be this: http://www.sqlalchemy.org/docs/documentation.html#unitofwork_identitymap After the first "select" you do not reset the Unif Of Work. In my queries I always do: def _sessionTransaction(self, callable_, *args, **kwargs): from sqlalchemy import orm conn = self.contextual_connect() sess = orm.create_session(bind_to=conn) # This cannot fail try: trans = sess.create_transaction() try: ret = callable_(conn, sess, *args, **kwargs) trans.commit() return ret except: trans.rollback() raise finally: sess.close() # This cannot fail conn.close() I create and close a new session object for every request. You can give a look at my nadbapi: http://developer.berlios.de/projects/nadbapi/ Regards Manlio Perillo
On 6/13/07, Manlio Perillo
The problem can be this: http://www.sqlalchemy.org/docs/documentation.html#unitofwork_identitymap
After the first "select" you do not reset the Unif Of Work.
Yes, that seems to be the problem. For this my tiny problem, I can get away by simply not using activemapper, though. mytable.select().execute(id=1) mytable.select().execute(id=1) yields two select statements towards the database, just as I want it. Thanks!
only seems to execute one select statement. The other result seems to be taken from the cache.
AFAIK, SQLAlchemy does perform the select statements, but, the second time, does not populate your mapped object since it's already present in the identity_map cache. You can see these by turning SA's engine.echo = True
Yes, that seems to be the problem. For this my tiny problem, I can get away by simply not using activemapper, though.
Have you tried myobject.refresh(), before the second time ? This will force SA to repopulate your instance Also, depending on your application behavior, you might want to consider a request to your server <=> a session in SA (that's usually what used for web apps). This means each time you end a request, you would have to session.clear(), preventing any caching. Hope that helps. Cheers, Seb -- Sébastien LELONG http://sirloon.net http://sirbot.org sebastien.lelong[at]sirloon.net
On 13/06/2007 15:56 Magnus Hult wrote:
only seems to execute one select statement. The other result seems to be taken from the cache. If this is the case, then this question would rather belong in an SQLAlchemy mailing list, I guess.
Have you tried using the sAsync wrapper for Twisted? I have not experienced caching problems with anything wrapped with it's transaction decorator. -- Colin Alston ~ http://www.karnaugh.za.net/ Expecting one person to deal with all your problems is like praying each time you require an ambulance: You'll eventually be dead long before you get a response.
On 6/13/07, Colin Alston
On 13/06/2007 15:56 Magnus Hult wrote:
only seems to execute one select statement. The other result seems to be taken from the cache. If this is the case, then this question would rather belong in an SQLAlchemy mailing list, I guess.
Have you tried using the sAsync wrapper for Twisted? I have not experienced caching problems with anything wrapped with it's transaction decorator.
No, I stumbled upon it while investigating this problem though. I'll definitely give it a closer look!
On Wed, 13 Jun 2007 14:40:11 +0200, Magnus Hult
Hi,
When executing the following program, changes in the database (other than those made by this program) aren't visible until it is restarted, due to some form of SQLAlchemy caching:
from twisted.web import server, resource import model # my database model
class MyRoot(resource.Resource): def render(self, request): m = model.MyModel.get(request.args['id'][0]) return m.some_column
Here, you're arranging to render whatever the expression `m.some_column' evaluates to. However that is implemented and whatever value it results in, that's what will be rendered on the page.
[snip - db and service setup]
So if I visit http://localhost:8080/?id=1, then alter the some_column value for the row who's id is 1 and reload, the change is not reflected.
Problem is, I need the update to be visible. How would I go about implementing this?
It would seem that `m.some_column' evaluates to the same value each time the page is rendered. Could it be that SQLAlchemy is performing some undesirable caching which is preventing the new value from showing up? I am not particularly familiar with SQLAlchemy, so I'm not sure why this might be. It does seem like you want to be looking into SQLAlchemy behavior to figure this out, though. Hope this helps, Jean-Paul
participants (5)
-
Colin Alston
-
Jean-Paul Calderone
-
Magnus Hult
-
Manlio Perillo
-
Sébastien LELONG