[Twisted-Python] Twisted + Redis-py + Connection Pooling
Hello everyone
I am currently working on creating a complex python script which will be
using connection pooling from redis-py library (it will be working as a
backport Unix Domain Socket for redis 1.4, which doesn't support unix
sockets)
The simplest version of my code was included in the attachment.
I have a problem thought. Even if traffic is high (500+ requests per
second) there are no new connections started to redis (added print to
connection.py to trace it).
Here's my output:
2012-09-28 23:00:31+0200 [-] Log opened.
2012-09-28 23:00:31+0200 [-] Factory starting on '/tmp/redis_test.sock'
2012-09-28 23:00:31+0200 [-] Starting factory
One problem in the code: it kinda seems you want to receive lines, but you're overriding dataReceived instead of lineReceived in RedisWorkProtocol. At the very minimum you shouldn't assume dataReceived is called with the exact bytes that were written on the other side. More importantly, I assume redis-py is blocking. Calling a blocking API from Twisted will block the reactor, preventing any other code from running until it the API call returns. As such it means you can't have two queries to redis running at the same time, unless you use something like deferToThread, or a redis client that supports Twisted directly. -- Itamar Turner-Trauring, Future Foundries LLC http://futurefoundries.com/ — Twisted consulting, training and support.
Thank you very much for your reply Itamar. I have switched to txredisapi
which is non-blocking library and my problem was solved.
2012/9/29 Itamar Turner-Trauring
One problem in the code: it kinda seems you want to receive lines, but you're overriding dataReceived instead of lineReceived in RedisWorkProtocol. At the very minimum you shouldn't assume dataReceived is called with the exact bytes that were written on the other side.
More importantly, I assume redis-py is blocking. Calling a blocking API from Twisted will block the reactor, preventing any other code from running until it the API call returns. As such it means you can't have two queries to redis running at the same time, unless you use something like deferToThread, or a redis client that supports Twisted directly.
-- Itamar Turner-Trauring, Future Foundries LLC http://futurefoundries.com/ — Twisted consulting, training and support.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
participants (2)
-
Arkadiusz Rzadkowolski
-
Itamar Turner-Trauring