I'm not sure why I am getting these errors but it seems to be associated with backups. Is there a backup cronjob that didn't run, fail or is something else wrong? The directories do exist but not the indicated pck files. TIA
Feb 25 12:02:06 2020 (14100) Uncaught runner exception: [Errno 2] No such file or directory Feb 25 12:02:06 2020 (5133) Uncaught runner exception: [Errno 2] No such file or directory: '/var/spool/mailman/out/1582657325.390982+504a3666a91ef8722e1af700669bf0190e00417d.pck' Feb 25 12:02:06 2020 (5133) Traceback (most recent call last): File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 99, in _oneloop msg, msgdata = self._switchboard.dequeue(filebase) File "/usr/lib/mailman/Mailman/Queue/Switchboard.py", line 154, in dequeue fp = open(filename) IOError: [Errno 2] No such file or directory: '/var/spool/mailman/out/1582657325.390982+504a3666a91ef8722e1af700669bf0190e00417d.pck'
Feb 25 12:02:06 2020 (5133) Skipping and preserving unparseable message: 1582657325.390982+504a3666a91ef8722e1af700669bf0190e00417d Feb 25 12:02:06 2020 (14100) Traceback (most recent call last): File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 99, in _oneloop msg, msgdata = self._switchboard.dequeue(filebase) File "/usr/lib/mailman/Mailman/Queue/Switchboard.py", line 158, in dequeue os.rename(filename, backfile) OSError: [Errno 2] No such file or directory
Feb 25 12:02:06 2020 (14100) Skipping and preserving unparseable message: 1582657325.390982+30007f7a3ce65a426ba60c1ab46d996c33d2bb9a Feb 25 12:02:06 2020 (14102) Failed to unlink/preserve backup file: /var/spool/mailman/out/1582657325.390982+504a3666a91ef8722e1af700669bf0190e00417d.bak Feb 25 12:02:07 2020 (5128) Failed to unlink/preserve backup file: /var/spool/mailman/archive/1582657325.390982+30007f7a3ce65a426ba60c1ab46d996c33d2bb9a.bak Feb 25 12:07:35 2020 (5134) Uncaught runner exception: [Errno 2] No such file or directory: '/var/spool/mailman/virgin/1582657654.150712+53c19a1ee899509786cc9beb40990d4c631f65d4.pck' Feb 25 12:07:35 2020 (5134) Traceback (most recent call last): File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 99, in _oneloop msg, msgdata = self._switchboard.dequeue(filebase) File "/usr/lib/mailman/Mailman/Queue/Switchboard.py", line 154, in dequeue fp = open(filename) IOError: [Errno 2] No such file or directory: '/var/spool/mailman/virgin/1582657654.150712+53c19a1ee899509786cc9beb40990d4c631f65d4.pck'
Feb 25 12:07:35 2020 (5134) Skipping and preserving unparseable message: 1582657654.150712+53c19a1ee899509786cc9beb40990d4c631f65d4 Feb 25 12:07:35 2020 (14098) Failed to unlink/preserve backup file: /var/spool/mailman/virgin/1582657654.150712+53c19a1ee899509786cc9beb40990d4c631f65d4.bak Feb 25 12:29:07 2020 (14099) Uncaught runner exception: [Errno 2] No such file or directory Feb 25 12:29:07 2020 (14099) Traceback (most recent call last): File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 99, in _oneloop msg, msgdata = self._switchboard.dequeue(filebase) File "/usr/lib/mailman/Mailman/Queue/Switchboard.py", line 158, in dequeue os.rename(filename, backfile) OSError: [Errno 2] No such file or directory
Feb 25 12:29:07 2020 (14099) Skipping and preserving unparseable message: 1582658947.082654+30ee2b8943fe331a15218871760c1c8d3a5d8a90 Feb 25 12:29:07 2020 (5131) Failed to unlink/preserve backup file: /var/spool/mailman/in/1582658947.082654+30ee2b8943fe331a15218871760c1c8d3a5d8a90.bak Feb 25 13:04:00 2020 (14099) Uncaught runner exception: [Errno 2] No such file or directory Feb 25 13:04:00 2020 (14099) Traceback (most recent call last): File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 99, in _oneloop msg, msgdata = self._switchboard.dequeue(filebase) File "/usr/lib/mailman/Mailman/Queue/Switchboard.py", line 158, in dequeue os.rename(filename, backfile) OSError: [Errno 2] No such file or directory
Feb 25 13:04:00 2020 (14099) Skipping and preserving unparseable message: 1582661039.25433+fcd57e8ec68cce26d6ec44748d97987a885b8f3c Feb 25 13:04:00 2020 (5131) Failed to unlink/preserve backup file: /var/spool/mailman/in/1582661039.25433+fcd57e8ec68cce26d6ec44748d97987a885b8f3c.bak Feb 25 13:04:01 2020 (14104) Uncaught runner exception: [Errno 2] No such file or directory: '/var/spool/mailman/bounces/1582661040.887134+c53e72b90df57867b3447d0e85584d819d2f14b4.pck' Feb 25 13:04:01 2020 (14104) Traceback (most recent call last): File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 99, in _oneloop msg, msgdata = self._switchboard.dequeue(filebase) File "/usr/lib/mailman/Mailman/Queue/Switchboard.py", line 154, in dequeue fp = open(filename) IOError: [Errno 2] No such file or directory: '/var/spool/mailman/bounces/1582661040.887134+c53e72b90df57867b3447d0e85584d819d2f14b4.pck'
Feb 25 13:04:01 2020 (14104) Skipping and preserving unparseable message: 1582661040.887134+c53e72b90df57867b3447d0e85584d819d2f14b4 Feb 25 13:04:01 2020 (5129) Failed to unlink/preserve backup file: /var/spool/mailman/bounces/1582661040.887134+c53e72b90df57867b3447d0e85584d819d2f14b4.bak
On 2/26/20 6:56 AM, Dennis Putnam wrote:
I'm not sure why I am getting these errors but it seems to be associated with backups. Is there a backup cronjob that didn't run, fail or is something else wrong? The directories do exist but not the indicated pck files. TIA
I haven't looked at this in full detail, but is there more than one BounceRunner running processing the same slice. What does
ps -fwwA|grep BounceRunner
give? If there is more than one process showing "--runner=BounceRunner:0:1" see the article at https://wiki.list.org/x/4030715 for advice on completely stopping Mailman and starting only one instance.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 2/26/2020 11:09 AM, Mark Sapiro wrote:
On 2/26/20 6:56 AM, Dennis Putnam wrote:
I'm not sure why I am getting these errors but it seems to be associated with backups. Is there a backup cronjob that didn't run, fail or is something else wrong? The directories do exist but not the indicated pck files. TIA
I haven't looked at this in full detail, but is there more than one BounceRunner running processing the same slice. What does
ps -fwwA|grep BounceRunner
give? If there is more than one process showing "--runner=BounceRunner:0:1" see the article at https://wiki.list.org/x/4030715 for advice on completely stopping Mailman and starting only one instance.
mailman 5129 5125 0 Feb24 ? 00:00:16 /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=BounceRunner:0:1 -s
I think that means there is only 1 process.
On 2/26/20 8:18 AM, Dennis Putnam wrote:
mailman 5129 5125 0 Feb24 ? 00:00:16 /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=BounceRunner:0:1 -s
I think that means there is only 1 process.
Yes, but maybe that wasn't the case at the time of those messages. Are the errors continuing? Also, I indicated BounceRunner as I was looking at the last messages you posted. There are also ones involving OutgoingRunner and VirginRunner
Also, there should be files in /var/spool/mailman/bad/ which you may be able to examine with Mailman's dumpdb which are the unparseable message(s).
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 2/26/2020 9:14 PM, Mark Sapiro wrote:
On 2/26/20 8:18 AM, Dennis Putnam wrote:
mailman 5129 5125 0 Feb24 ? 00:00:16 /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=BounceRunner:0:1 -s
I think that means there is only 1 process.
Yes, but maybe that wasn't the case at the time of those messages. Are the errors continuing? Also, I indicated BounceRunner as I was looking at the last messages you posted. There are also ones involving OutgoingRunner and VirginRunner
Also, there should be files in /var/spool/mailman/bad/ which you may be able to examine with Mailman's dumpdb which are the unparseable message(s).
Hi Mark,
It has not happened in 2 days however, there are no files in any of those directories. Does that not imply the backups are not working? Is that handled by a cronjob?
On 2/27/20 6:27 AM, Dennis Putnam wrote:
It has not happened in 2 days however, there are no files in any of those directories. Does that not imply the backups are not working? Is that handled by a cronjob?
This has nothing to do with backups per se. The error messages are somewhat different.
Feb 25 12:02:06 2020 (14100) Uncaught runner exception: [Errno 2] No such file or directory
The above actually goes with a set below (pid 14100).
Feb 25 12:02:06 2020 (5133) Uncaught runner exception: [Errno 2] No such file or directory: '/var/spool/mailman/out/1582657325.390982+504a3666a91ef8722e1af700669bf0190e00417d.pck' Feb 25 12:02:06 2020 (5133) Traceback (most recent call last): File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 99, in _oneloop msg, msgdata = self._switchboard.dequeue(filebase) File "/usr/lib/mailman/Mailman/Queue/Switchboard.py", line 154, in dequeue fp = open(filename) IOError: [Errno 2] No such file or directory: '/var/spool/mailman/out/1582657325.390982+504a3666a91ef8722e1af700669bf0190e00417d.pck'
The above says OutgoingRunner has listed its queue and is now trying to retrieve the indicated .pck file from the queue, but it is gone. The only way this can happen is if another instance of OutgoingRunner has retrieved the message in the mean time.
Feb 25 12:02:06 2020 (5133) Skipping and preserving unparseable message: 1582657325.390982+504a3666a91ef8722e1af700669bf0190e00417d
This goes with the above, but is spurious. The missing file exception is assumed to be an unparseable message, but it's not.
Feb 25 12:02:06 2020 (14100) Traceback (most recent call last): File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 99, in _oneloop msg, msgdata = self._switchboard.dequeue(filebase) File "/usr/lib/mailman/Mailman/Queue/Switchboard.py", line 158, in dequeue os.rename(filename, backfile) OSError: [Errno 2] No such file or directory
Feb 25 12:02:06 2020 (14100) Skipping and preserving unparseable message: 1582657325.390982+30007f7a3ce65a426ba60c1ab46d996c33d2bb9a
Here again, we are trying to retrieve a queued message that another instance has already retrieved.
The rest of the messages are similar.
As best as I can tell, this is all due to the issue discussed at https://wiki.list.org/x/4030715. If the server has been rebooted since this happened, that would have fixed it.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 2/27/2020 12:47 PM, Mark Sapiro wrote:
On 2/27/20 6:27 AM, Dennis Putnam wrote:
It has not happened in 2 days however, there are no files in any of those directories. Does that not imply the backups are not working? Is that handled by a cronjob?
This has nothing to do with backups per se. The error messages are somewhat different.
Feb 25 12:02:06 2020 (14100) Uncaught runner exception: [Errno 2] No such file or directory The above actually goes with a set below (pid 14100).
Feb 25 12:02:06 2020 (5133) Uncaught runner exception: [Errno 2] No such file or directory: '/var/spool/mailman/out/1582657325.390982+504a3666a91ef8722e1af700669bf0190e00417d.pck' Feb 25 12:02:06 2020 (5133) Traceback (most recent call last): File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 99, in _oneloop msg, msgdata = self._switchboard.dequeue(filebase) File "/usr/lib/mailman/Mailman/Queue/Switchboard.py", line 154, in dequeue fp = open(filename) IOError: [Errno 2] No such file or directory: '/var/spool/mailman/out/1582657325.390982+504a3666a91ef8722e1af700669bf0190e00417d.pck'
The above says OutgoingRunner has listed its queue and is now trying to retrieve the indicated .pck file from the queue, but it is gone. The only way this can happen is if another instance of OutgoingRunner has retrieved the message in the mean time.
Feb 25 12:02:06 2020 (5133) Skipping and preserving unparseable message: 1582657325.390982+504a3666a91ef8722e1af700669bf0190e00417d This goes with the above, but is spurious. The missing file exception is assumed to be an unparseable message, but it's not.
Feb 25 12:02:06 2020 (14100) Traceback (most recent call last): File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 99, in _oneloop msg, msgdata = self._switchboard.dequeue(filebase) File "/usr/lib/mailman/Mailman/Queue/Switchboard.py", line 158, in dequeue os.rename(filename, backfile) OSError: [Errno 2] No such file or directory
Feb 25 12:02:06 2020 (14100) Skipping and preserving unparseable message: 1582657325.390982+30007f7a3ce65a426ba60c1ab46d996c33d2bb9a
Here again, we are trying to retrieve a queued message that another instance has already retrieved.
The rest of the messages are similar.
As best as I can tell, this is all due to the issue discussed at https://wiki.list.org/x/4030715. If the server has been rebooted since this happened, that would have fixed it.
Hi Mark,
Thanks. I'll just keep an eye on it to see if it recurs.
participants (2)
-
Dennis Putnam
-
Mark Sapiro