[Tutor] Shared FIFO?

[Kent Johnson]

On Sun, May 17, 2009 at 11:36 PM, Allen Fowler <allen.fowler at yahoo.com> wrote:
>
> Hello,
>
> I have several CGI scripts that I would like coordinate via a "First In / First Out" style buffer.    That is, some processes are adding work units, and some take the oldest and start work on them.
>
> What is the right way to do this?    I suppose I could use an SQL server or SQlite , but that seems very complex for just a FIFO.

Are the worker processes CGI scripts too? That doesn't make sense to me.

You could consolidate all the user-facing CGI scripts into one process
using something like CherryPy or Django, then use the multiprocessing
module in the std lib to farm out the work to other processes.

One advantage of using a disk-based queue vs multiprocessing.Queue is
that it is more robust against system failures; a catastrophic failure
will lose work in progress and work not yet queued but items in the
queue should be preserved.

Kent