Hi
After rebooting my server, several mails from some weeks ago were resent to lists and also a bunch of moderator requests to me.
How may I prevent this from happening again?
Eric
On 08/27/2015 10:53 PM, Eric Smith wrote:
There was a bug in Mailman 2.1.9 that would not unlink the qfiles/QUEUE/*.bak files used for recovery processing under some conditions. These files would then be 'recovered' when Mailman is restarted.
How may I prevent this from happening again?
If the above is the reason, update to a more recent version, or possibly run a periodic cron to remove qfiles/QUEUE/*.bak files older than some age.
It the above is not the reason, We'd need more information to diagnose how this happened.
Perhaps id you examine your 'qrunner' log leading up to the reboot, you'll find that some runner or runners had died and not restarted, and then started and processed their queues after the reboot.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Thank Mark
Mark Sapiro wrote on Sun-30-Aug 15 3:54AM
% dpkg -l|grep mailman. ii mailman 1:2.1.18-1ubuntu1
I started reboot on 27 August at 16:14; # zcat qrunner.3.gz Aug 27 16:15:25 2015 (3092) CommandRunner qrunner started. Aug 27 16:15:25 2015 (3094) NewsRunner qrunner started. Aug 27 16:15:25 2015 (3090) ArchRunner qrunner started. Aug 27 16:15:25 2015 (3091) BounceRunner qrunner started. Aug 27 16:15:25 2015 (3093) IncomingRunner qrunner started. Aug 27 16:15:25 2015 (3097) RetryRunner qrunner started. Aug 27 16:15:26 2015 (3096) VirginRunner qrunner started. Aug 27 16:15:26 2015 (3095) OutgoingRunner qrunner started. # zcat qrunner.2.gz Aug 28 08:04:29 2015 (3097) RetryRunner qrunner caught SIGHUP. Reopening logs. Aug 28 08:04:29 2015 (3089) Master watcher caught SIGHUP. Re-opening log files. Aug 28 08:04:29 2015 (3092) CommandRunner qrunner caught SIGHUP. Reopening logs. Aug 28 08:04:29 2015 (3096) VirginRunner qrunner caught SIGHUP. Reopening logs. Aug 28 08:04:29 2015 (3091) BounceRunner qrunner caught SIGHUP. Reopening logs. Aug 28 08:04:29 2015 (3095) OutgoingRunner qrunner caught SIGHUP. Reopening logs. Aug 28 08:04:29 2015 (3094) NewsRunner qrunner caught SIGHUP. Reopening logs. Aug 28 08:04:29 2015 (3093) IncomingRunner qrunner caught SIGHUP. Reopening logs. Aug 28 08:04:29 2015 (3090) ArchRunner qrunner caught SIGHUP. Reopening logs. # zcat qrunner.1.gz pepper:/var/log/mailman# cat qrunner.1 Aug 29 07:53:39 2015 (3096) VirginRunner qrunner caught SIGHUP. Reopening logs. Aug 29 07:53:39 2015 (3095) OutgoingRunner qrunner caught SIGHUP. Reopening logs. Aug 29 07:53:40 2015 (3089) Master watcher caught SIGHUP. Re-opening log files. Aug 29 07:53:42 2015 (3091) BounceRunner qrunner caught SIGHUP. Reopening logs. Aug 29 07:53:42 2015 (3094) NewsRunner qrunner caught SIGHUP. Reopening logs. Aug 29 07:53:43 2015 (3092) CommandRunner qrunner caught SIGHUP. Reopening logs. Aug 29 07:53:43 2015 (3097) RetryRunner qrunner caught SIGHUP. Reopening logs. Aug 29 07:53:43 2015 (3093) IncomingRunner qrunner caught SIGHUP. Reopening logs. Aug 29 07:53:43 2015 (3090) ArchRunner qrunner caught SIGHUP. Reopening logs.
huh? # zcat qrunner.4.gz # zcat qrunner.5.gz # ls -lrt qr* -rw-rw-r-- 1 list list 20 Aug 23 07:58 qrunner.7.gz -rw-rw-r-- 1 list list 20 Aug 24 08:06 qrunner.6.gz -rw-rw-r-- 1 list list 20 Aug 25 07:48 qrunner.5.gz -rw-rw-r-- 1 list list 20 Aug 26 07:39 qrunner.4.gz -rw-rw-r-- 1 list list 151 Aug 27 16:15 qrunner.3.gz -rw-rw-r-- 1 list list 194 Aug 28 08:04 qrunner.2.gz -rw-rw-r-- 1 list list 729 Aug 29 07:53 qrunner.1 -rw-rw-r-- 1 list list 729 Aug 30 07:37 qrunner
Eric
On 08/29/2015 11:01 PM, Eric Smith wrote:
And 8 runners were started.
Then the next morning Mailman's logs were rotated.
And again the following Morning.
You haven't shown 'cat qrunner', but it presumably is just another logrotate set like 'cat qrunner.1' only a day later.
The strange (interesting?) thing is all of qrunner.4.gz through qrunner.7.gz are gzips of empty files. logrotate ran because the log was rotated, but no SIGHUP entries were written.
My best guess is this is because Mailman wasn't running between Aug 23 07:58 and the reboot time and probably for some time before that which is why messages were just queued and not processed until you rebooted which started Mailman.
If you have no older Mailman logs, we don't know why or exactly when Mailman stopped, but I'm reasonably sure that's what happened.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 08/27/2015 10:53 PM, Eric Smith wrote:
There was a bug in Mailman 2.1.9 that would not unlink the qfiles/QUEUE/*.bak files used for recovery processing under some conditions. These files would then be 'recovered' when Mailman is restarted.
How may I prevent this from happening again?
If the above is the reason, update to a more recent version, or possibly run a periodic cron to remove qfiles/QUEUE/*.bak files older than some age.
It the above is not the reason, We'd need more information to diagnose how this happened.
Perhaps id you examine your 'qrunner' log leading up to the reboot, you'll find that some runner or runners had died and not restarted, and then started and processed their queues after the reboot.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Thank Mark
Mark Sapiro wrote on Sun-30-Aug 15 3:54AM
% dpkg -l|grep mailman. ii mailman 1:2.1.18-1ubuntu1
I started reboot on 27 August at 16:14; # zcat qrunner.3.gz Aug 27 16:15:25 2015 (3092) CommandRunner qrunner started. Aug 27 16:15:25 2015 (3094) NewsRunner qrunner started. Aug 27 16:15:25 2015 (3090) ArchRunner qrunner started. Aug 27 16:15:25 2015 (3091) BounceRunner qrunner started. Aug 27 16:15:25 2015 (3093) IncomingRunner qrunner started. Aug 27 16:15:25 2015 (3097) RetryRunner qrunner started. Aug 27 16:15:26 2015 (3096) VirginRunner qrunner started. Aug 27 16:15:26 2015 (3095) OutgoingRunner qrunner started. # zcat qrunner.2.gz Aug 28 08:04:29 2015 (3097) RetryRunner qrunner caught SIGHUP. Reopening logs. Aug 28 08:04:29 2015 (3089) Master watcher caught SIGHUP. Re-opening log files. Aug 28 08:04:29 2015 (3092) CommandRunner qrunner caught SIGHUP. Reopening logs. Aug 28 08:04:29 2015 (3096) VirginRunner qrunner caught SIGHUP. Reopening logs. Aug 28 08:04:29 2015 (3091) BounceRunner qrunner caught SIGHUP. Reopening logs. Aug 28 08:04:29 2015 (3095) OutgoingRunner qrunner caught SIGHUP. Reopening logs. Aug 28 08:04:29 2015 (3094) NewsRunner qrunner caught SIGHUP. Reopening logs. Aug 28 08:04:29 2015 (3093) IncomingRunner qrunner caught SIGHUP. Reopening logs. Aug 28 08:04:29 2015 (3090) ArchRunner qrunner caught SIGHUP. Reopening logs. # zcat qrunner.1.gz pepper:/var/log/mailman# cat qrunner.1 Aug 29 07:53:39 2015 (3096) VirginRunner qrunner caught SIGHUP. Reopening logs. Aug 29 07:53:39 2015 (3095) OutgoingRunner qrunner caught SIGHUP. Reopening logs. Aug 29 07:53:40 2015 (3089) Master watcher caught SIGHUP. Re-opening log files. Aug 29 07:53:42 2015 (3091) BounceRunner qrunner caught SIGHUP. Reopening logs. Aug 29 07:53:42 2015 (3094) NewsRunner qrunner caught SIGHUP. Reopening logs. Aug 29 07:53:43 2015 (3092) CommandRunner qrunner caught SIGHUP. Reopening logs. Aug 29 07:53:43 2015 (3097) RetryRunner qrunner caught SIGHUP. Reopening logs. Aug 29 07:53:43 2015 (3093) IncomingRunner qrunner caught SIGHUP. Reopening logs. Aug 29 07:53:43 2015 (3090) ArchRunner qrunner caught SIGHUP. Reopening logs.
huh? # zcat qrunner.4.gz # zcat qrunner.5.gz # ls -lrt qr* -rw-rw-r-- 1 list list 20 Aug 23 07:58 qrunner.7.gz -rw-rw-r-- 1 list list 20 Aug 24 08:06 qrunner.6.gz -rw-rw-r-- 1 list list 20 Aug 25 07:48 qrunner.5.gz -rw-rw-r-- 1 list list 20 Aug 26 07:39 qrunner.4.gz -rw-rw-r-- 1 list list 151 Aug 27 16:15 qrunner.3.gz -rw-rw-r-- 1 list list 194 Aug 28 08:04 qrunner.2.gz -rw-rw-r-- 1 list list 729 Aug 29 07:53 qrunner.1 -rw-rw-r-- 1 list list 729 Aug 30 07:37 qrunner
Eric
On 08/29/2015 11:01 PM, Eric Smith wrote:
And 8 runners were started.
Then the next morning Mailman's logs were rotated.
And again the following Morning.
You haven't shown 'cat qrunner', but it presumably is just another logrotate set like 'cat qrunner.1' only a day later.
The strange (interesting?) thing is all of qrunner.4.gz through qrunner.7.gz are gzips of empty files. logrotate ran because the log was rotated, but no SIGHUP entries were written.
My best guess is this is because Mailman wasn't running between Aug 23 07:58 and the reboot time and probably for some time before that which is why messages were just queued and not processed until you rebooted which started Mailman.
If you have no older Mailman logs, we don't know why or exactly when Mailman stopped, but I'm reasonably sure that's what happened.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (2)
-
Eric Smith -
Mark Sapiro