[Twisted-Python] Persistent Queue with subscriber worker
Daniel de la Cuesta wrote:
Hi,
I want to develop a video conversion service. My idea is to offer a REST api and save all the conversion request in a database to track the state of each process (pending, doing, done, error, etc.)
I want to use this database table as a persistent queue and a twisted daemon that takes its task from this table. This daemon has to take the task, update the state of the task and do the work.
How can I subscribe the twisted daemon to the database inserts?
The easiest way is to use twisted.internet.task.LoopingCall and check for new entries every N seconds (e.g. n=1, n=30) If your database supports things like postgres' NOTIFY and you have an async-aware db adapter (e.g. pgasync) you could use a trigger on the table to issue a NOTIFY and respond immediately, but it's been my experience that's more trouble than it's worth.
Is the thread pool a good approach to this problem?
Since most DB adapters are blocking you'll want to use something like a thread pool to access them - twisted's adbapi module does this for you.
Hi Daniel On Monday 03 November 2008 09:40:14 Daniel de la Cuesta wrote:
I want to develop a video conversion service. My idea is to offer a REST api and save all the conversion request in a database to track the state of each process (pending, doing, done, error, etc.)
Do you plan to expose this REST api using Twisted?
I want to use this database table as a persistent queue and a twisted daemon that takes its task from this table. This daemon has to take the task, update the state of the task and do the work.
How can I subscribe the twisted daemon to the database inserts?
Is the thread pool a good approach to this problem?
Do you know a more simple solution to this problem?
Well, I wouldn't use a database at all (except for logging purposes only maybe). Actually you're already using the proper terms (queue, subscriber, etc.), so I would use a message-driven architecture: - A messaging broker, either using AMQP (RabbitMQ, Qpid and OpenAMQ) or Stomp (RabbitMQ, ActiveMQ) - You have a subscriber (waiting for messages on a queue), which dispatches tasks from the queue - The web application sends messages to the messaging broker, which will route them to the right subscriber One of the advantages of this scenario is that you can add more subscribers at runtime if needed, which will distribute the load between several conversion servers. You can choose either AMQP [1] or Stomp [2], each has its strengths and weaknesses. Cheers. 1 - https://launchpad.net/txamqp (disclaimer, I'm one of the developers) 2 - http://code.google.com/p/stomper/
participants (3)
-
Daniel de la Cuesta -
Esteve Fernandez -
Phil Mayers