At 2:33 PM -0800 12/14/00, J C Lawrence wrote:

I'm currently going back and forth on usine an external queue system. Its a nice idea, and there are definite administrative and control benefits, but its also moderately complex, [...]

you forgot compatibility and stability issues. Tie yourself to a queue system, you only run on systems that queue system runs on, and you're only as stable as that queue system is. So you end up adding a lot of dependencies onto a piece of code you don't control that's core to the program.

My current noodling is tending towards implementing a queue-like system using light-weight self-discovering processes that could easily be individually removed and their interconnects replaced by calls to a real queueing system (whatever that may be) on an ad-hoc/per-site basis.

one thing I'm doodling with for my SMTP back end is starting up a server that places a socket, the n starting up "N" SMTP processes as clients that grab addresses from the server one at a time for delivery. This gets me completely away fro this "slow DNS" stuff, since any one slow address slows only itself, and since the system I'm looking at is 100% verped/customized (ala Lyris's footers, at the minimum), I'm not worrying about the added overhead (you could potentially do batches through an interface). using sockets means the clients can go off-machine for free, as long as they know where to look.

now, maybe it could be something like that, a controlling process that uses both threads and forks (and perhaps remote commands through rsh or ssh) to spawn instances as needed...