[Twisted-Python] Question about using the adbapi
Hi!
In the application I'm currently working on, we gather data from different sources and store them in a database, but only if the values have changed.
One example is GPS information. We don't want to store hundreds of identical values that are sent by the receiver, so we have to compare the last written row with the current latitude / longitude pair.
I use a subclass of NMEARreceiver class to determine the current latitude and longitude. Then I query the database using a ConnectionPool instance to get the previous values for latitude and longitude. If the result differs from the current values, a new entry is stored, again using the ConnectionPool.
The problem is: How do I "connect" the current latitude and longitude to the database query? The query returns a Deferred object, but this expects a callback with only one argument, the result of the query.
Storing them as instance variables of NMEAReceiver does not seem to be the right way, since I don't know how long it takes to query the database - the relevant GPS information might be overwritten by a new NMEA telegram when the connection pool finally triggers the callback
One workaround I've found is to add this information to the SQL query, but this seems quite weird to me (and doesn't work in the case that the query does not return a result).
A different approach would be to implement a callback with three arguments (result, latitude and longitude), and wrap it in a lambda function that fills latitude and longitude with the current values. Would this would be a better way to handle this requirement?
How would you implement this?
Thanks & best regards,
Albert
On 10:50 am, albert.brandl@weiermayer.com wrote:
Hi!
In the application I'm currently working on, we gather data from different sources and store them in a database, but only if the values have changed.
One example is GPS information. We don't want to store hundreds of identical values that are sent by the receiver, so we have to compare the last written row with the current latitude / longitude pair.
I use a subclass of NMEARreceiver class to determine the current latitude and longitude. Then I query the database using a ConnectionPool instance to get the previous values for latitude and longitude. If the result differs from the current values, a new entry is stored, again using the ConnectionPool.
The problem is: How do I "connect" the current latitude and longitude to the database query? The query returns a Deferred object, but this expects a callback with only one argument, the result of the query.
d.addCallback(f, x, y) will result in f(resultOfD, x, y)
Jean-Paul
Storing them as instance variables of NMEAReceiver does not seem to be the right way, since I don't know how long it takes to query the database
- the relevant GPS information might be overwritten by a new NMEA
telegram when the connection pool finally triggers the callback
One workaround I've found is to add this information to the SQL query, but this seems quite weird to me (and doesn't work in the case that the query does not return a result).
A different approach would be to implement a callback with three arguments (result, latitude and longitude), and wrap it in a lambda function that fills latitude and longitude with the current values. Would this would be a better way to handle this requirement?
How would you implement this?
Thanks & best regards,
Albert
Albert Brandl Weiermayer Solutions GmbH | Abteistra�e 12, A-4813 Altm�nster phone: +43 (0) 720 70 30 14 | fax: +43 (0) 7612 20 3 56 web: http://www.weiermayer.com
Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Mon, May 09, 2011 at 11:03:12AM -0000, exarkun@twistedmatrix.com wrote:
d.addCallback(f, x, y) will result in f(resultOfD, x, y)
Cool - no need to wrap the method call :-). The documentation for runQuery does not mention this feature, though:
"Returns a Deferred which will fire the return value of a DB-API cursor's 'fetchall' method, or a Failure."
Maybe it's obvious for people who know how Deferreds work :-/
Thanks, and best regards,
I see 2 other options:
a) keep the last (longitude, latitude) pair for every receiver stored in your script, too, besides the db. When a new pair of values arrives, compare it with the current and only if it is different, send the "INSERT" query to the database.
b) When a new pair of values arrives, send a query (or stored procedure) that tells the database to check and "INSERT" only if the pair is different that the latest.
Pandelis Theodosiou
On Mon, May 9, 2011 at 1:50 PM, Albert Brandl albert.brandl@weiermayer.comwrote:
Hi!
In the application I'm currently working on, we gather data from different sources and store them in a database, but only if the values have changed.
One example is GPS information. We don't want to store hundreds of identical values that are sent by the receiver, so we have to compare the last written row with the current latitude / longitude pair.
I use a subclass of NMEARreceiver class to determine the current latitude and longitude. Then I query the database using a ConnectionPool instance to get the previous values for latitude and longitude. If the result differs from the current values, a new entry is stored, again using the ConnectionPool.
The problem is: How do I "connect" the current latitude and longitude to the database query? The query returns a Deferred object, but this expects a callback with only one argument, the result of the query.
Storing them as instance variables of NMEAReceiver does not seem to be the right way, since I don't know how long it takes to query the database
- the relevant GPS information might be overwritten by a new NMEA
telegram when the connection pool finally triggers the callback
One workaround I've found is to add this information to the SQL query, but this seems quite weird to me (and doesn't work in the case that the query does not return a result).
A different approach would be to implement a callback with three arguments (result, latitude and longitude), and wrap it in a lambda function that fills latitude and longitude with the current values. Would this would be a better way to handle this requirement?
How would you implement this?
Thanks & best regards,
Albert
Albert Brandl Weiermayer Solutions GmbH | Abteistraße 12, A-4813 Altmünster phone: +43 (0) 720 70 30 14 | fax: +43 (0) 7612 20 3 56 web: http://www.weiermayer.com
Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
http://int.ask.com/web?siteid=10000861&webqsrc=999&l=dis&q=UPDATE http://int.ask.com/web?siteid=10000861&webqsrc=999&l=dis&q=triple%20
On Mon, May 09, 2011 at 02:15:04PM +0300, Pandelis Theodosiou wrote:
a) keep the last (longitude, latitude) pair for every receiver stored in your script, too, besides the db. When a new pair of values arrives, compare it with the current and only if it is different, send the "INSERT" query to the database.
This will only work if I can guarantee that the database query is finished before the next (longitude, latitude) pair arrives. Otherwise, I'll lose the previous pair. Storing a list of (longitude, latitude) pairs won't work either: the database queries are executed in separate threads, and I can't control the order in which they will finish. So I have no correspondence between the query result and a (longitude, latitude) pair.
For GPS data, this might not be an issue, since a new position arrives only about once a second. But other data (e.g. the current speed) are sent every ten ms or so. For these, I have to make sure that the current data corresponds to the query result.
b) When a new pair of values arrives, send a query (or stored procedure) that tells the database to check and "INSERT" only if the pair is different that the latest.
That is an interesting option, since it moves the responsibility from the Twisted server to the database.
Thanks & best regards,
Albert
participants (3)
-
Albert Brandl -
exarkun＠twistedmatrix.com -
Pandelis Theodosiou