Persistent Queue implementations?

Karl A. Krueger kkrueger at example.edu
Thu Dec 26 15:07:43 EST 2002


Aahz <aahz at pythoncraft.com> wrote:
> In article <auf9a4$ovp$1 at baldur.whoi.edu>,
> Karl A. Krueger <kkrueger at example.edu> wrote:
>>That's a very good point.  I may ponder it for a bit and come back. :)
>>I'm not resistant to databases -- I have a mod_python / PostgreSQL app
>>currently in maintenance -- but I'd kind of like something faster and
>>more transparent here.
> 
> If re-processing already completed items isn't a problem, you could have
> a separate thread that monitors the queue and checkpoints every X items.
> It's less transparent in principle, but if the queue hides the
> checkpoint thread from other threads, it doesn't look that way from the
> outside.

That's an interesting design.  Here's another (surely not original):

When a queue is created, it's given a timeout interval.

When an item is enqueued, it gets assigned a serial number or other
magic cookie.

When an item is dequeued by a client, the client gets back the tuple
(item, cookie) ... and the item is not deleted from the queue, it's just
marked "handed out" with a timestamp.  (Meanwhile, other clients can get
things off the queue, and the queue is considered "empty" if all items
have been handed out.)

When the client thread is finished processing the item, it calls a
release method with the cookie, and the item is actually deleted from
the queue.

A daemon thread enforces the timeout -- every timeout interval, it runs
through the queue and clears the "handed out" flags from items that have
been open for that amount of time or more.  (If everything was handed
out before, this also clears the emptiness sema.)

This is obviously no good if the queue must be processed in strict
order.  It should be OK for a multithreaded mail filter, which is one of
the applications I'm thinking of using this with.  :)

-- 
Karl A. Krueger <kkrueger at example.edu>
Woods Hole Oceanographic Institution
Email address is spamtrapped.  s/example/whoi/
"Outlook not so good." -- Magic 8-Ball Software Reviews



More information about the Python-list mailing list