[Twisted-Python] advice sought - two database connections in a RunInteraction (adbapi.ConnectionPool)
I've got some database transactional code running within a thread via RunInteraction (it needs to rollback on error). One of the features on the logic is that it interacts with a 3rd party API, which may or may not have a ratelimit in effect. Most 3rd party APIs i consume have a minute/hourly based limit -- so I just do a graceful fail on the transaction to requeue it, and set a local timeout. This particular API has a per-second meter in effect. It is considerably more efficient to just sleep for a second and "try again" a second later. I end up being able to process over twice as many API calls in a given period with that strategy. The problem I ran into, is that I need a second database connection to note that a ratelimit has been created in the database -- and i'd like to pull that db connection out of the twisted connection pool to simplify my connection management. - I can't do it within the existing connection, because I am in a transaction and the limit won't appear to other twisted jobs. - I can't use runInteraction() off the pool, because that will defer to a thread and I am in a thread, and the last time I did something like that... everything broke. - the stopgap I have right now is just creating/closing a raw pscycopg2 connection and using that. it works, but it requires a bit more management of postgres to ensure I can balance the right number of connections. has anyone had a need like this before? if so, what did you do? i basically just need to make two "INSERT" statements from a second connection.
On Sep 22, 2014, at 9:00 AM, Jonathan Vanasco
has anyone had a need like this before? if so, what did you do?
Yes. I re-wrote twisted.enterprise entirely, with a better design. You can see the result here: http://trac.calendarserver.org/browser/twext/trunk/twext/enterprise/adbapi2.... And you can 'pip install' it from here: https://pypi.python.org/pypi/twextpy/0.1a12469 Hopefully it will see a non-alpha release at some point. -glyph
oh great! I'll jump through the docs this weekend! thanks so much for the reply and for writing that! On Sep 23, 2014, at 1:45 AM, Glyph wrote:
On Sep 22, 2014, at 9:00 AM, Jonathan Vanasco
wrote: has anyone had a need like this before? if so, what did you do?
Yes. I re-wrote twisted.enterprise entirely, with a better design.
You can see the result here:
http://trac.calendarserver.org/browser/twext/trunk/twext/enterprise/adbapi2....
And you can 'pip install' it from here:
https://pypi.python.org/pypi/twextpy/0.1a12469
Hopefully it will see a non-alpha release at some point.
-glyph _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
// Jonathan Vanasco c. 646.729.6436 | 415.501.9815 e. jonathan@2xlp.com w. http://findmeon.com/user/jvanasco linkedin. http://linkedin.com/in/jonathanvanasco blog. http://destructuring.net
participants (2)
-
Glyph
-
Jonathan Vanasco