[Mailman-Developers] Mailman queue design problem?

[Barry A. Warsaw]

>>>>> "CG" == Carson Gaspar <carson@taltos.org> writes:

    CG> So I recommend a quick fix of splitting the queue into 3
    CG> parts, and a medium term fix of adopting one of the tiered
    CG> directory structures (postfix, new sendmail, etc.).

Mailman 2.1's queue is split into at least 7 subdirectories,
specifically for:

- incoming posts from the MTA

- incoming commands (i.e. bounces, -request, -owner, -admin, -join, -leave)

- outgoing mail to the MTA

- outgoing mail to the archiver

- outgoing to the nntpd

- the "shunt" queue (for uncaught system errors)

- the "virgin" queue (for messages crafted by Mailman)

It's fairly easy to add more queues, although I'm not sure what else
would be useful.  Some other features of the 2.1 queue system:

1) FIFO order is guaranteed

2) Files are "randomly" distributed in a 120 bit hash space for
   no-lock cooperative multiprocessing.

3) Different qrunners can be assigned different priorities (i.e. you
   can run your incoming posts and MTA-bound queues more often then
   your archiver, nntpd, or command processing queues).

4) The qrunners are long running processes, monitored by a cron
   spawned watchdog.  Pros: on a warmed up system, delivery occurs
   almost immediately, with start up delays amortized over the life of
   the qrunner (Python 2.0's cyclic garbage collector helps keep
   memory usage under control).  Cons: debugging is, er, more
   challenging because you have to kill the master qrunner whenever
   you (really, I ;) make a change to the code.

All in all, while I have field tested it yet, I think the 2.1 queue
runner is miles ahead of what's in 2.0.

Cheers,
-Barry