[Redirecting part of this discussion from mailman-users to mailman-developers. Mark does a great job of explaining the gory details of the queue runners.]
On Oct 16, 2012, at 09:23 PM, Mark Sapiro wrote:
First of all, The actual physical size of the queue directory impacts processing. Every time an entry is added to the queue, and every time a .pck file is renamed to .bak, the entire physical directory must be searched to ensure this isn't a duplicate name. Depending on OS settings, cache sizes and the physical directory size, this may actually involve multiple disk reads each time. Thus, if the qfiles/out/ directory has grown large because 8000+ messages were added to the queue when the runner couldn't handle them (and there may have been more in the retry/ queue because of SMTP failures), it would benefit from shrinking. This is accomplished by moving (mv) or renaming the queue directory itself aside, not just its contents and then letting the runner recreate it when it starts. Then, if necessary, move messages back a few at a time so the directory doesn't grow large again.
I'd like to begin to explore ways to make this automatically more manageable, so that when problems occur (e.g. upstream SMTPd not responding or slow), Mailman can recognize and response more efficiently to the problem.
The first question to ask is whether this mostly affects the outgoing queue, or whether other queues *in practice* can suffer the same overloading? I suspect that the incoming queue could get filled quickly if Mailman is slow to handling an onslaught of new messages. My sense is that in general, it's the outgoing queue that gives people the most problems though.
The switchboard in Mailman 3 is largely the same as in Mailman 2 (modulo updating the code to more modern Pythons).
Before describing my own thoughts, I'd like to open it up and get a sense of your experiences, and strategies you think might help manage big queues.
Cheers, -Barry