Attachments are unexpectedly re-created.
Dear list,
I am kind of new to use mailman on administrative, so if this is an known issue I'm sorry. I would very much appreciate it if you would kindly answer my questions.
Firtst of all, one of my customer wanted to delete attachments to make more disk spaces on the server. The server's disk resource was getting full because the size of the directory for attachments was very big. So the customer did the following works first.
Digest options -> Disabled(Digestable 'NO') Archiving options -> Disabled(Archive 'NO')
Secondly, after the above, the customer deleted directories located under the following directory such as 20091201, 20091202 and so on.
/var/lib/mailman/archives/private/<list name>/attachments/<date>
# rm -rf 20091201,20091202,.....
Finaly, the customer got the "Digest options" and "Archiving options" back to enabled as follows.
Digest options -> Enabled(Digestable 'YES') Archiving options -> Enabled(Archive 'YES')
With this operations, the directories under attachments directory were deleted and the disk space is now enough, however, by unknown timing, directories that are named with past's date, which were manually deleted, are re-created automatically. And in that directories, it seems that the same attachment files are repeatly created.
One thing that I realized to stop this weird action is to disable both "Digest options" and "Archiving options". Once those options are disabled, no more unexpected attachment files would be created under that directory.
To know what was going on to the mailman server at that moment (in the meantime of the unexpected action), I asked the customer for logs such as error log in /var/log/mailman direcoty, and in the error log, I saw the following errors regarding "send_digests".
xxx xx xx:xx:xx 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character ...... xxx xx xx:xx:xx 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character ...... xxx xx xx:xx:xx 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character ......
I have not yet been sure if these errors are related to the cuase, but it seems that "send_digests()" appearently fails and retries repeatly.
My questions this time are as follows.
1.Why are attachment files re-created? Is it a normal action whenever "Digest options" and "Archiving options" are available?
2.If the answer for the question 1 is "NO", what could be the cause? The fail of send_digests() is it?
3.Does the send_digests() repeat(retry) if the action fails?
4.Is there anything else that should be done after deleting attachments manually to avoid being suffered this weird behavior?
---Environment--- OS:RHEL5 postfix-2.3.3-2.1.el5_2 mailman-2.1.9-4.el5
Any comments/suggestions would be greatly appreciated.
Sincerely,
--
Masaharu Kawada Associate Technical Support Engineer Red Hat K K Ebisu Neonato 5F 1-18 Ebisu 4-chome, Shibuya-ku Tokyo 150-0013, Japan Direct: +81-3-5798-8482
Masaharu Kawada wrote:
Firtst of all, one of my customer wanted to delete attachments to make more disk spaces on the server. The server's disk resource was getting full because the size of the directory for attachments was very big. So the customer did the following works first.
Digest options -> Disabled(Digestable 'NO') Archiving options -> Disabled(Archive 'NO')
Secondly, after the above, the customer deleted directories located under the following directory such as 20091201, 20091202 and so on.
/var/lib/mailman/archives/private/<list name>/attachments/<date>
# rm -rf 20091201,20091202,.....
Finaly, the customer got the "Digest options" and "Archiving options" back to enabled as follows.
Digest options -> Enabled(Digestable 'YES') Archiving options -> Enabled(Archive 'YES')
There was no need to turn off digestable and archive if they were only going to be turned on again.
With this operations, the directories under attachments directory were deleted and the disk space is now enough, however, by unknown timing, directories that are named with past's date, which were manually deleted, are re-created automatically. And in that directories, it seems that the same attachment files are repeatly created.
One thing that I realized to stop this weird action is to disable both "Digest options" and "Archiving options". Once those options are disabled, no more unexpected attachment files would be created under that directory.
To know what was going on to the mailman server at that moment (in the meantime of the unexpected action), I asked the customer for logs such as error log in /var/log/mailman direcoty, and in the error log, I saw the following errors regarding "send_digests".
xxx xx xx:xx:xx 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character ...... xxx xx xx:xx:xx 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character ...... xxx xx xx:xx:xx 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character ......
I have not yet been sure if these errors are related to the cuase, but it seems that "send_digests()" appearently fails and retries repeatly.
Yes, I think this is the problem. Somewhere in the lists/LISTNAME/digest.mbox file there is a bad message that causes this error. Every time there is a post to the list, A digest is triggered because of the size of the digest.mbox. The mbox is processed and attachments are scrubbed (from the 'plain' digest) and stored up to the point of the exception which aborts the process until the next time, but each time those attachments prior to the bad message get stored again.
My questions this time are as follows.
1.Why are attachment files re-created? Is it a normal action whenever "Digest options" and "Archiving options" are available?
Non text/plain MIME parts and text/plain MIME parts with unknown character set are removed and stored aside and replaced by links in both the archive and the plain format digest. Thus there normally are two copies of each in the archive.
In your case, the same messages in the digest.mbox are processed repeatedly causing the attachments to accumulate.
2.If the answer for the question 1 is "NO", what could be the cause? The fail of send_digests() is it?
3.Does the send_digests() repeat(retry) if the action fails?
4.Is there anything else that should be done after deleting attachments manually to avoid being suffered this weird behavior?
The underlying cause is a message in the lists/LISTNAME/digest.mbox file with a body or sub part with declared as charset=EUC-JP with an invalid character or something related.
Removing or moving aside the lists/LISTNAME/digest.mbox or finding the bad message and removing it from the file should also fix this.
I consider this behavior to be a bug. I'll look into fixing it. I think what I'll need to do is just move the digest.mbox aside and log an error message. It would be better if I could just bypass the bad message, but that may be difficult.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro wrote:
I consider this behavior to be a bug. I'll look into fixing it. I think what I'll need to do is just move the digest.mbox aside and log an error message. It would be better if I could just bypass the bad message, but that may be difficult.
I have created a bug report for this at <https://bugs.launchpad.net/mailman/+bug/531081>.
I still have a couple of issues. Unfortunately, when we catch the exception that aborts the digest, we don't log a traceback, so I don't really know what went wrong. If possible I would like to get a copy of the offending digest.mbox to see if the actual problem is one of a couple of bugs in Scrubber.py that have already been fixed since 2.1.9. If it is possible to get your customer to send it to me off list, I promise to treat it with complete confidentiality and destroy it when I'm done.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark-san,
Thank you very much for your help.
I will try to get all of the digest.mbox located on under lists/LISTNAME/ from the customer. Though it might be difficult since digest.mbox consists confidential information for the customer. I will update as soon as I get the result from the customer.
Best Regards,
Mark Sapiro wrote:
Mark Sapiro wrote:
I consider this behavior to be a bug. I'll look into fixing it. I think what I'll need to do is just move the digest.mbox aside and log an error message. It would be better if I could just bypass the bad message, but that may be difficult.
I have created a bug report for this at <https://bugs.launchpad.net/mailman/+bug/531081>.
I still have a couple of issues. Unfortunately, when we catch the exception that aborts the digest, we don't log a traceback, so I don't really know what went wrong. If possible I would like to get a copy of the offending digest.mbox to see if the actual problem is one of a couple of bugs in Scrubber.py that have already been fixed since 2.1.9. If it is possible to get your customer to send it to me off list, I promise to treat it with complete confidentiality and destroy it when I'm done.
--
Masaharu Kawada Associate Technical Support Engineer Red Hat K K Ebisu Neonato 5F 1-18 Ebisu 4-chome, Shibuya-ku Tokyo 150-0013, Japan Direct: +81-3-5798-8482
Mark-san,
Thank you very much for your help.
There was no need to turn off digestable and archive if they were only going to be turned on again.
I did some research and found information that it is necessary to disable those options if you want to delete attachments manually, otherwise something unclaim might happen. However, even if you delete attachments, no need to turn off those?
Removing or moving aside the lists/LISTNAME/digest.mbox or finding the bad message and removing it from the file should also fix this.
Yes, this is what I asked the customer for doing, but I am wondering that even if I could find the bad message and delete it, how can I reject it to be sent all the time. I mean what if the same happens again? My guessing for this behavior is that it is necessary to removing or moving aside the digest.mbox whenever bad message is in that mbox as long as the digestable options turned on. Is my guessing correct?
I appreciated your help!
Best Regards,
Mark Sapiro wrote:
Masaharu Kawada wrote:
Firtst of all, one of my customer wanted to delete attachments to make more disk spaces on the server. The server's disk resource was getting full because the size of the directory for attachments was very big. So the customer did the following works first.
Digest options -> Disabled(Digestable 'NO') Archiving options -> Disabled(Archive 'NO')
Secondly, after the above, the customer deleted directories located under the following directory such as 20091201, 20091202 and so on.
/var/lib/mailman/archives/private/<list name>/attachments/<date>
# rm -rf 20091201,20091202,.....
Finaly, the customer got the "Digest options" and "Archiving options" back to enabled as follows.
Digest options -> Enabled(Digestable 'YES') Archiving options -> Enabled(Archive 'YES')
There was no need to turn off digestable and archive if they were only going to be turned on again.
With this operations, the directories under attachments directory were deleted and the disk space is now enough, however, by unknown timing, directories that are named with past's date, which were manually deleted, are re-created automatically. And in that directories, it seems that the same attachment files are repeatly created.
One thing that I realized to stop this weird action is to disable both "Digest options" and "Archiving options". Once those options are disabled, no more unexpected attachment files would be created under that directory.
To know what was going on to the mailman server at that moment (in the meantime of the unexpected action), I asked the customer for logs such as error log in /var/log/mailman direcoty, and in the error log, I saw the following errors regarding "send_digests".
xxx xx xx:xx:xx 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character ...... xxx xx xx:xx:xx 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character ...... xxx xx xx:xx:xx 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character ......
I have not yet been sure if these errors are related to the cuase, but it seems that "send_digests()" appearently fails and retries repeatly.
Yes, I think this is the problem. Somewhere in the lists/LISTNAME/digest.mbox file there is a bad message that causes this error. Every time there is a post to the list, A digest is triggered because of the size of the digest.mbox. The mbox is processed and attachments are scrubbed (from the 'plain' digest) and stored up to the point of the exception which aborts the process until the next time, but each time those attachments prior to the bad message get stored again.
My questions this time are as follows.
1.Why are attachment files re-created? Is it a normal action whenever "Digest options" and "Archiving options" are available?
Non text/plain MIME parts and text/plain MIME parts with unknown character set are removed and stored aside and replaced by links in both the archive and the plain format digest. Thus there normally are two copies of each in the archive.
In your case, the same messages in the digest.mbox are processed repeatedly causing the attachments to accumulate.
2.If the answer for the question 1 is "NO", what could be the cause? The fail of send_digests() is it?
3.Does the send_digests() repeat(retry) if the action fails?
4.Is there anything else that should be done after deleting attachments manually to avoid being suffered this weird behavior?
The underlying cause is a message in the lists/LISTNAME/digest.mbox file with a body or sub part with declared as charset=EUC-JP with an invalid character or something related.
Removing or moving aside the lists/LISTNAME/digest.mbox or finding the bad message and removing it from the file should also fix this.
I consider this behavior to be a bug. I'll look into fixing it. I think what I'll need to do is just move the digest.mbox aside and log an error message. It would be better if I could just bypass the bad message, but that may be difficult.
--
Masaharu Kawada Associate Technical Support Engineer Red Hat K K Ebisu Neonato 5F 1-18 Ebisu 4-chome, Shibuya-ku Tokyo 150-0013, Japan Direct: +81-3-5798-8482
Masaharu Kawada wrote:
There was no need to turn off digestable and archive if they were only going to be turned on again.
I did some research and found information that it is necessary to disable those options if you want to delete attachments manually, otherwise something unclaim might happen. However, even if you delete attachments, no need to turn off those?
I suppose that something unexpected could happen if you removed one of the attachment directories while it was in the process of being written, so what you did was safest.
Removing or moving aside the lists/LISTNAME/digest.mbox or finding the bad message and removing it from the file should also fix this.
Yes, this is what I asked the customer for doing, but I am wondering that even if I could find the bad message and delete it, how can I reject it to be sent all the time. I mean what if the same happens again? My guessing for this behavior is that it is necessary to removing or moving aside the digest.mbox whenever bad message is in that mbox as long as the digestable options turned on. Is my guessing correct?
It is my thought that when an exception occurs in this process, that I would produce a traceback as well as the existing log message and also move the digest.mbox aside and log that action too.
However, I'm not convinced that the problem in this case is not due to a Scrubber.py bug that either has already been fixed since 2.1.9 or could be fixed, thus avoiding the issue. That's why I would like to get the offending digest.mbox if possible.
In any case, the attached ToDigest.patch.txt contains a patch to Mailman/Handlers/ToDigest.py. If your customer can install that patch (just apply the patch to the file and restart mailman), we can at least get a traceback from the exception which may help pinpoint the problem.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark-san,
I am sorry that I would like to know one more thing about digestable in advance.
xxx xx xx:xx:xx 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character ...... xxx xx xx:xx:xx 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character ...... xxx xx xx:xx:xx 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character ......
Yes, I think this is the problem. Somewhere in the lists/LISTNAME/digest.mbox file there is a bad message that causes this error.
It seems that the operation of the "send_digests()" starts via corn at noon(12:00) every day. However, re-creation attachment files problem happens even other time(not only at noon but also other time), so the "send_digests()" operation seems to be done sometime except at the specified time of cron. If my understanding is right, the "send_digests()" operation repeats(retries) sevral times after the fail of its first action. On this matter, what I would like to know is that what the interval of the retry, and how many time does it repeat.
Could you please let me know if you know about this.
Best Regards,
Mark Sapiro wrote:
Masaharu Kawada wrote:
Firtst of all, one of my customer wanted to delete attachments to make more disk spaces on the server. The server's disk resource was getting full because the size of the directory for attachments was very big. So the customer did the following works first.
Digest options -> Disabled(Digestable 'NO') Archiving options -> Disabled(Archive 'NO')
Secondly, after the above, the customer deleted directories located under the following directory such as 20091201, 20091202 and so on.
/var/lib/mailman/archives/private/<list name>/attachments/<date>
# rm -rf 20091201,20091202,.....
Finaly, the customer got the "Digest options" and "Archiving options" back to enabled as follows.
Digest options -> Enabled(Digestable 'YES') Archiving options -> Enabled(Archive 'YES')
There was no need to turn off digestable and archive if they were only going to be turned on again.
With this operations, the directories under attachments directory were deleted and the disk space is now enough, however, by unknown timing, directories that are named with past's date, which were manually deleted, are re-created automatically. And in that directories, it seems that the same attachment files are repeatly created.
One thing that I realized to stop this weird action is to disable both "Digest options" and "Archiving options". Once those options are disabled, no more unexpected attachment files would be created under that directory.
To know what was going on to the mailman server at that moment (in the meantime of the unexpected action), I asked the customer for logs such as error log in /var/log/mailman direcoty, and in the error log, I saw the following errors regarding "send_digests".
xxx xx xx:xx:xx 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character ...... xxx xx xx:xx:xx 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character ...... xxx xx xx:xx:xx 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character ......
I have not yet been sure if these errors are related to the cuase, but it seems that "send_digests()" appearently fails and retries repeatly.
Yes, I think this is the problem. Somewhere in the lists/LISTNAME/digest.mbox file there is a bad message that causes this error. Every time there is a post to the list, A digest is triggered because of the size of the digest.mbox. The mbox is processed and attachments are scrubbed (from the 'plain' digest) and stored up to the point of the exception which aborts the process until the next time, but each time those attachments prior to the bad message get stored again.
My questions this time are as follows.
1.Why are attachment files re-created? Is it a normal action whenever "Digest options" and "Archiving options" are available?
Non text/plain MIME parts and text/plain MIME parts with unknown character set are removed and stored aside and replaced by links in both the archive and the plain format digest. Thus there normally are two copies of each in the archive.
In your case, the same messages in the digest.mbox are processed repeatedly causing the attachments to accumulate.
2.If the answer for the question 1 is "NO", what could be the cause? The fail of send_digests() is it?
3.Does the send_digests() repeat(retry) if the action fails?
4.Is there anything else that should be done after deleting attachments manually to avoid being suffered this weird behavior?
The underlying cause is a message in the lists/LISTNAME/digest.mbox file with a body or sub part with declared as charset=EUC-JP with an invalid character or something related.
Removing or moving aside the lists/LISTNAME/digest.mbox or finding the bad message and removing it from the file should also fix this.
I consider this behavior to be a bug. I'll look into fixing it. I think what I'll need to do is just move the digest.mbox aside and log an error message. It would be better if I could just bypass the bad message, but that may be difficult.
--
Masaharu Kawada Associate Technical Support Engineer Red Hat K K Ebisu Neonato 5F 1-18 Ebisu 4-chome, Shibuya-ku Tokyo 150-0013, Japan Direct: +81-3-5798-8482
Masaharu Kawada wrote:
I am sorry that I would like to know one more thing about digestable in advance.
xxx xx xx:xx:xx 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character ...... xxx xx xx:xx:xx 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character ...... xxx xx xx:xx:xx 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character ......
Yes, I think this is the problem. Somewhere in the lists/LISTNAME/digest.mbox file there is a bad message that causes this error.
It seems that the operation of the "send_digests()" starts via corn at noon(12:00) every day. However, re-creation attachment files problem happens even other time(not only at noon but also other time), so the "send_digests()" operation seems to be done sometime except at the specified time of cron. If my understanding is right, the "send_digests()" operation repeats(retries) sevral times after the fail of its first action. On this matter, what I would like to know is that what the interval of the retry, and how many time does it repeat.
cron/senddigests runs every day at noon and is responsible for the exception that happens at noon or just thereafter.
There are no retries per se. The list has a setting digest_size_threshhold (on its Digest options page). When a post arrives that makes the digest.mbox bigger than digest_size_threshhold, a digest is triggered immediately. In this case, the digest.mbox is bigger than digest_size_threshhold and the digest is never successfully produced so every post to the list tries to send a digest and causes the exception to be triggered again. Thus, all the other logged exceptions are the result of a post to the list triggering the digest process.
If there were no posts to the list, the process would only run once a day.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark-san,
Thank you very much for your respose.
There are no retries per se. The list has a setting digest_size_threshhold (on its Digest options page). When a post arrives that makes the digest.mbox bigger than digest_size_threshhold, a digest is triggered immediately. In this case, the digest.mbox is bigger than digest_size_threshhold and the digest is never successfully produced so every post to the list tries to send a digest and causes the exception to be triggered again. Thus, all the other logged exceptions are the result of a post to the list triggering the digest process.
If there were no posts to the list, the process would only run once a day.
So, it doesn't matter if the senddigest fais or not, there are no retries. And, senddigest via cron runs only once a day at noon. However, whenever messages posted to the list, send_digest() will called based on the size of digest_size_threshhold, and try to send digest mails. But in my customer's case, every time message comes to the list, digest process(send_digest()) fails due to the bad message in the digest.mbox. This is the cause of the error message in /var/log/mailman/error repeatly appear. Is my understanding right?
Thanks!
Mark Sapiro wrote:
Masaharu Kawada wrote:
I am sorry that I would like to know one more thing about digestable in advance.
xxx xx xx:xx:xx 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character ...... xxx xx xx:xx:xx 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character ...... xxx xx xx:xx:xx 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character ......
Yes, I think this is the problem. Somewhere in the lists/LISTNAME/digest.mbox file there is a bad message that causes this error.
It seems that the operation of the "send_digests()" starts via corn at noon(12:00) every day. However, re-creation attachment files problem happens even other time(not only at noon but also other time), so the "send_digests()" operation seems to be done sometime except at the specified time of cron. If my understanding is right, the "send_digests()" operation repeats(retries) sevral times after the fail of its first action. On this matter, what I would like to know is that what the interval of the retry, and how many time does it repeat.
cron/senddigests runs every day at noon and is responsible for the exception that happens at noon or just thereafter.
There are no retries per se. The list has a setting digest_size_threshhold (on its Digest options page). When a post arrives that makes the digest.mbox bigger than digest_size_threshhold, a digest is triggered immediately. In this case, the digest.mbox is bigger than digest_size_threshhold and the digest is never successfully produced so every post to the list tries to send a digest and causes the exception to be triggered again. Thus, all the other logged exceptions are the result of a post to the list triggering the digest process.
If there were no posts to the list, the process would only run once a day.
--
Masaharu Kawada Associate Technical Support Engineer Red Hat K K Ebisu Neonato 5F 1-18 Ebisu 4-chome, Shibuya-ku Tokyo 150-0013, Japan Direct: +81-3-5798-8482
Masaharu Kawada wrote:
So, it doesn't matter if the senddigest fais or not, there are no retries. And, senddigest via cron runs only once a day at noon. However, whenever messages posted to the list, send_digest() will called based on the size of digest_size_threshhold, and try to send digest mails. But in my customer's case, every time message comes to the list, digest process(send_digest()) fails due to the bad message in the digest.mbox. This is the cause of the error message in /var/log/mailman/error repeatly appear. Is my understanding right?
Yes, that is exactly correct.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark-san,
Thank you very much for your response. I was kind of misunderstanding about what the send_digest() does before, but I now got better knowledge on it because of your explanations for me. I very much appreciate it!!
I have asked the customer for providing the digest.mbox(s) to us and waited for the reply , so once I get the result, I will update it with replying to this mail.
Best Regards,
Mark Sapiro wrote:
Masaharu Kawada wrote:
So, it doesn't matter if the senddigest fais or not, there are no retries. And, senddigest via cron runs only once a day at noon. However, whenever messages posted to the list, send_digest() will called based on the size of digest_size_threshhold, and try to send digest mails. But in my customer's case, every time message comes to the list, digest process(send_digest()) fails due to the bad message in the digest.mbox. This is the cause of the error message in /var/log/mailman/error repeatly appear. Is my understanding right?
Yes, that is exactly correct.
--
Masaharu Kawada Associate Technical Support Engineer Red Hat K K Ebisu Neonato 5F 1-18 Ebisu 4-chome, Shibuya-ku Tokyo 150-0013, Japan Direct: +81-3-5798-8482
Hi Mark-san,
I got a reply from the customer, but it was not possbile for the customer to provide the whole digest.mbox to us due to their policy. However, sevral parts of the error log and digest.mbox's content have been provided. As for the error messages, please see the attachment. And as for the digest.mbox which might be the problematic one, I see lots of messages look like below.
ESC$B?7%"%s%A%&%$%k%9=8Cf4IM}%7%9%F%`ESC(B 2009/11/07(7:25:58) ESC$B%&%$%k%9$,8+$D$+$j$^$7$?ESC(B (ESC$B4m81ESC(B) Trojan.Pandex A32Z02V (A32G29V)(MG740762) (UPS Delivery Problem>>inv.zip>>inv.exe) ESC$BMW5a=hM}"*%/%j!<%K%s%0ESC(B ESC$B<B:]=hM}"*:o=|$K$h$C$F%/%j!<%K%s%0$7$^$7$?ESC(B
I know that I got just a few information to investigate, but can you be able to guess what the cause of this is?
Thanks in advance,
Best Regards,
Masaharu Kawada wrote:
Mark-san,
Thank you very much for your response. I was kind of misunderstanding about what the send_digest() does before, but I now got better knowledge on it because of your explanations for me. I very much appreciate it!!
I have asked the customer for providing the digest.mbox(s) to us and waited for the reply , so once I get the result, I will update it with replying to this mail.
Best Regards,
Mark Sapiro wrote:
Masaharu Kawada wrote:
So, it doesn't matter if the senddigest fais or not, there are no retries. And, senddigest via cron runs only once a day at noon. However, whenever messages posted to the list, send_digest() will called based on the size of digest_size_threshhold, and try to send digest mails. But in my customer's case, every time message comes to the list, digest process(send_digest()) fails due to the bad message in the digest.mbox. This is the cause of the error message in /var/log/mailman/error repeatly appear. Is my understanding right?
Yes, that is exactly correct.
--
Masaharu Kawada Associate Technical Support Engineer Red Hat K K Ebisu Neonato 5F 1-18 Ebisu 4-chome, Shibuya-ku Tokyo 150-0013, Japan Direct: +81-3-5798-8482
Feb 08 06:55:04 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 06:55:20 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 06:55:35 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 07:30:45 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 07:35:57 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 07:37:19 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 07:37:34 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 07:38:17 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 07:38:33 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 07:38:48 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 07:39:03 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 07:39:19 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 07:48:04 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 07:48:20 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 07:49:13 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 07:49:29 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 07:58:04 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 07:58:20 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 07:58:35 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 07:58:59 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 07:59:15 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 08:01:52 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 08:02:08 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 08:02:24 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 08:02:39 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 08:02:54 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 08:06:16 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 08:08:00 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 08:10:58 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 08:13:01 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 08:24:12 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 08:25:10 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 08:25:26 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 08:25:42 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 08:25:57 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 08:44:50 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 08:45:57 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 08:46:39 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 08:46:54 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 08:47:11 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 08:47:27 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 08:47:43 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 08:47:59 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 08:48:16 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 08:48:31 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 08:48:46 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 08:49:02 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 08:49:18 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 08:49:35 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 08:53:01 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 08:54:35 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 08:58:01 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 08:58:18 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 08:58:34 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 08:58:50 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 08:59:05 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 08:59:22 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 08:59:38 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 08:59:54 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:00:10 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:00:26 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:00:43 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:00:59 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:01:15 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:01:31 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:01:46 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:02:02 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:02:20 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:02:36 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:02:53 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:03:09 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:03:26 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:03:43 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:03:59 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:04:15 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:04:31 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:04:47 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:05:03 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:05:19 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:05:35 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:05:51 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:06:07 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:09:16 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:09:33 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:09:49 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:12:45 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:13:02 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:16:59 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:17:16 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:17:32 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:17:49 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:18:06 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:18:22 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:18:38 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:18:55 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:19:12 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:19:28 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:19:44 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:20:01 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:20:18 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:20:35 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:24:29 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:25:54 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:26:10 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:26:26 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:26:42 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:27:09 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:27:25 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:28:17 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:28:33 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:28:49 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:29:05 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:29:21 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:29:38 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:29:54 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:38:34 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0xfff5 in JIS X 0208 Feb 08 09:44:06 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:44:22 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:44:39 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:44:55 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:45:12 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:45:28 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:45:44 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:46:00 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:46:16 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:46:33 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:46:49 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 09:59:13 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 10:06:47 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 10:07:03 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 10:07:19 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 10:07:36 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 10:23:40 2010 (2129) send_digests() failed: ISO-2022-JP encoding error: invalid character \u2116 Feb 08 10:25:01 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 10:25:18 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 10:25:34 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 10:25:50 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 10:26:07 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 10:26:24 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 10:32:13 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 10:32:29 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 10:32:46 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 10:33:02 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 10:33:19 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 10:34:49 2010 (2129) send_digests() failed: ISO-2022-JP encoding error: invalid character \u2116 Feb 08 10:53:46 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0xfff5 in JIS X 0208 Feb 08 10:56:05 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 10:59:11 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 10:59:27 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 11:01:32 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 11:01:48 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 11:02:05 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 11:02:21 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 12:04:01 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 15:06:17 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 16:36:19 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0xfff5 in JIS X 0208 Feb 08 17:04:16 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 17:04:55 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 17:05:11 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 17:05:26 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 17:05:43 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 17:05:59 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 17:06:15 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 17:06:31 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0x8357 in JIS X 0208 Feb 08 22:27:37 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0xfff5 in JIS X 0208 Feb 08 22:28:12 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0xfff5 in JIS X 0208 Feb 08 22:51:35 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0xfff5 in JIS X 0208 Feb 08 22:51:53 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0xfff5 in JIS X 0208 Feb 08 22:52:09 2010 (2129) send_digests() failed: EUC-JP decoding error: invalid character 0xfff5 in JIS X 0208
(10/03/03 16:54), Masaharu Kawada wrote:
Hi Mark-san,
I got a reply from the customer, but it was not possbile for the customer to provide the whole digest.mbox to us due to their policy. However, sevral parts of the error log and digest.mbox's content have been provided. As for the error messages, please see the attachment. And as for the digest.mbox which might be the problematic one, I see lots of messages look like below.
Hi, Kawada san, The problem is that the Japanese mail users/MUA developers use CP-932 charset as Shift-JIS and its derivatives (ISO-2022-JP/EUC-JP). CP-932 contains more (extended) characters than Shift-JIS while Python codec is strict on the latter. The characters like circled numbers fail to be decoded in unicode and cause error. Similar errors are reported in Japanese Mailman users, like the thread starting from: http://mm.tkikuchi.net/pipermail/mmjp-users/2009-February/002487.html A workaround is to patch charset.py in Python email library as: --- Lib/email/charset.py 2009-09-22 08:59:56.000000000 +0900 +++ Lib/email/charset.py.orig 2009-09-22 08:58:36.000000000 +0900 @@ -264,8 +264,7 @@ def convert(self, s): """Convert a string from the input_codec to the output_codec.""" if self.input_codec != self.output_codec: - return unicode(s, self.input_codec, 'replace' - ).encode(self.output_codec, 'replace') + return unicode(s, self.input_codec).encode(self.output_codec) else: return s -- Tokio Kikuchi, tkikuchi@is.kochi-u.ac.jp http://weather.is.kochi-u.ac.jp/
Hello kikuchi-san, Thank you for your response with a precious information. Since I am a beginner for mailman, any comments/suggestions such as this kind of thing are very helpful for me. Including what Mark-san have been doing for me and you have done for me as well, I really appreciate the cooperation from this list. As for the problem on my question, I will look into the infomation that you just gave me, and later on I will compare my customer's with one posted in that thread you provided. Thanks a lot! Best Regards, Tokio Kikuchi wrote:
(10/03/03 16:54), Masaharu Kawada wrote:
Hi Mark-san,
I got a reply from the customer, but it was not possbile for the customer to provide the whole digest.mbox to us due to their policy. However, sevral parts of the error log and digest.mbox's content have been provided. As for the error messages, please see the attachment. And as for the digest.mbox which might be the problematic one, I see lots of messages look like below.
Hi, Kawada san,
The problem is that the Japanese mail users/MUA developers use CP-932 charset as Shift-JIS and its derivatives (ISO-2022-JP/EUC-JP). CP-932 contains more (extended) characters than Shift-JIS while Python codec is strict on the latter. The characters like circled numbers fail to be decoded in unicode and cause error.
Similar errors are reported in Japanese Mailman users, like the thread starting from: http://mm.tkikuchi.net/pipermail/mmjp-users/2009-February/002487.html
A workaround is to patch charset.py in Python email library as:
--- Lib/email/charset.py 2009-09-22 08:59:56.000000000 +0900 +++ Lib/email/charset.py.orig 2009-09-22 08:58:36.000000000 +0900 @@ -264,8 +264,7 @@ def convert(self, s): """Convert a string from the input_codec to the output_codec.""" if self.input_codec != self.output_codec: - return unicode(s, self.input_codec, 'replace' - ).encode(self.output_codec, 'replace') + return unicode(s, self.input_codec).encode(self.output_codec) else: return s
-- ------------------- Masaharu Kawada Associate Technical Support Engineer Red Hat K K Ebisu Neonato 5F 1-18 Ebisu 4-chome, Shibuya-ku Tokyo 150-0013, Japan Direct: +81-3-5798-8482
Hello Kikuchi-san, As far as I checked the error messages and compared with each one, The problem of my customer's seems to be almost the same as one posted to the thread written in the last Kikuchi-san's email. There is one thing that I wonder, is that if the charset.py is done with that patch, what the current digest.mbox exsisted under lists/<listname> are supposed to be? Do they need to be deleted or not? My point on this is that once the patch is done, is there nothing else to do to fix this problem? Thanks in advance, Best Regards, Masaharu Kawada wrote:
Hello kikuchi-san,
Thank you for your response with a precious information. Since I am a beginner for mailman, any comments/suggestions such as this kind of thing are very helpful for me. Including what Mark-san have been doing for me and you have done for me as well, I really appreciate the cooperation from this list.
As for the problem on my question, I will look into the infomation that you just gave me, and later on I will compare my customer's with one posted in that thread you provided.
Thanks a lot!
Best Regards,
Tokio Kikuchi wrote:
(10/03/03 16:54), Masaharu Kawada wrote:
Hi Mark-san,
I got a reply from the customer, but it was not possbile for the customer to provide the whole digest.mbox to us due to their policy. However, sevral parts of the error log and digest.mbox's content have been provided. As for the error messages, please see the attachment. And as for the digest.mbox which might be the problematic one, I see lots of messages look like below.
Hi, Kawada san,
The problem is that the Japanese mail users/MUA developers use CP-932 charset as Shift-JIS and its derivatives (ISO-2022-JP/EUC-JP). CP-932 contains more (extended) characters than Shift-JIS while Python codec is strict on the latter. The characters like circled numbers fail to be decoded in unicode and cause error.
Similar errors are reported in Japanese Mailman users, like the thread starting from: http://mm.tkikuchi.net/pipermail/mmjp-users/2009-February/002487.html
A workaround is to patch charset.py in Python email library as:
--- Lib/email/charset.py 2009-09-22 08:59:56.000000000 +0900 +++ Lib/email/charset.py.orig 2009-09-22 08:58:36.000000000 +0900 @@ -264,8 +264,7 @@ def convert(self, s): """Convert a string from the input_codec to the output_codec.""" if self.input_codec != self.output_codec: - return unicode(s, self.input_codec, 'replace' - ).encode(self.output_codec, 'replace') + return unicode(s, self.input_codec).encode(self.output_codec) else: return s
-- ------------------- Masaharu Kawada Associate Technical Support Engineer Red Hat K K Ebisu Neonato 5F 1-18 Ebisu 4-chome, Shibuya-ku Tokyo 150-0013, Japan Direct: +81-3-5798-8482
Masaharu Kawada wrote:
There is one thing that I wonder, is that if the charset.py is done with that patch, what the current digest.mbox exsisted under lists/<listname> are supposed to be? Do they need to be deleted or not? My point on this is that once the patch is done, is there nothing else to do to fix this problem?
That is correct. Assuming the underlying issue is fixed by the patch, all you need to do is apply the patch to the Python email library charset.py module (probably at /usr/lib/pythonx.x/lib/email/charset.py) and restart Mailman. Then the next time the digest is triggered, it will be sent with all messages and no more errors. If you apply the patch and restart Mailman and the errors continue, then they are caused by something else. Note that the patch: --- Lib/email/charset.py 2009-09-22 08:59:56.000000000 +0900 +++ Lib/email/charset.py.orig 2009-09-22 08:58:36.000000000 +0900 @@ -264,8 +264,7 @@ def convert(self, s): """Convert a string from the input_codec to the output_codec.""" if self.input_codec != self.output_codec: - return unicode(s, self.input_codec, 'replace' - ).encode(self.output_codec, 'replace') + return unicode(s, self.input_codec).encode(self.output_codec) else: return s is reversed. The '+' is the original code and the '-' is the new code. -- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Hi, (10/03/04 1:58), Mark Sapiro wrote:
Masaharu Kawada wrote:
There is one thing that I wonder, is that if the charset.py is done with that patch, what the current digest.mbox exsisted under lists/<listname> are supposed to be? Do they need to be deleted or not? My point on this is that once the patch is done, is there nothing else to do to fix this problem?
That is correct. Assuming the underlying issue is fixed by the patch, all you need to do is apply the patch to the Python email library charset.py module (probably at /usr/lib/pythonx.x/lib/email/charset.py) and restart Mailman. Then the next time the digest is triggered, it will be sent with all messages and no more errors.
If you apply the patch and restart Mailman and the errors continue, then they are caused by something else.
In addition, you should have a lot of shunted messages in qfiles/shunt. They are safely deleted.
Note that the patch:
--- Lib/email/charset.py 2009-09-22 08:59:56.000000000 +0900 +++ Lib/email/charset.py.orig 2009-09-22 08:58:36.000000000 +0900 @@ -264,8 +264,7 @@ def convert(self, s): """Convert a string from the input_codec to the output_codec.""" if self.input_codec != self.output_codec: - return unicode(s, self.input_codec, 'replace' - ).encode(self.output_codec, 'replace') + return unicode(s, self.input_codec).encode(self.output_codec) else: return s
is reversed. The '+' is the original code and the '-' is the new code.
Yes. Thanks Mark. -- Tokio Kikuchi, tkikuchi@is.kochi-u.ac.jp http://weather.is.kochi-u.ac.jp/
Hi Mark-san, Kikuchi-san, Thank you very much for your help. Just to make sure, about the patch, is it just need to be modified in the way you mentioned? Which means that it should be look like below. # vi /usr/lib/python2.4/email/Charset.py ---<Before modifying>--- 243 def convert(self, s): 244 """Convert a string from the input_codec to the output_codec.""" 245 if self.input_codec <> self.output_codec: 246 return unicode(s, self.input_codec).encode(self.output_codec) 247 else: 248 return s ---<After modifying>--- 243 def convert(self, s): 244 """Convert a string from the input_codec to the output_codec.""" 245 if self.input_codec != self.output_codec: 246 return unicode(s, self.input_codec, 'replace').encode(self.output_codec, 'replace') 247 return unicode(s, self.input_codec).encode(self.output_codec) 248 else: 249 return s Best Regards, Tokio Kikuchi wrote:
Hi,
(10/03/04 1:58), Mark Sapiro wrote:
Masaharu Kawada wrote:
There is one thing that I wonder, is that if the charset.py is done with that patch, what the current digest.mbox exsisted under lists/<listname> are supposed to be? Do they need to be deleted or not? My point on this is that once the patch is done, is there nothing else to do to fix this problem?
That is correct. Assuming the underlying issue is fixed by the patch, all you need to do is apply the patch to the Python email library charset.py module (probably at /usr/lib/pythonx.x/lib/email/charset.py) and restart Mailman. Then the next time the digest is triggered, it will be sent with all messages and no more errors.
If you apply the patch and restart Mailman and the errors continue, then they are caused by something else.
In addition, you should have a lot of shunted messages in qfiles/shunt. They are safely deleted.
Note that the patch:
--- Lib/email/charset.py 2009-09-22 08:59:56.000000000 +0900 +++ Lib/email/charset.py.orig 2009-09-22 08:58:36.000000000 +0900 @@ -264,8 +264,7 @@ def convert(self, s): """Convert a string from the input_codec to the output_codec.""" if self.input_codec != self.output_codec: - return unicode(s, self.input_codec, 'replace' - ).encode(self.output_codec, 'replace') + return unicode(s, self.input_codec).encode(self.output_codec) else: return s
is reversed. The '+' is the original code and the '-' is the new code.
Yes. Thanks Mark.
-- ------------------- Masaharu Kawada Associate Technical Support Engineer Red Hat K K Ebisu Neonato 5F 1-18 Ebisu 4-chome, Shibuya-ku Tokyo 150-0013, Japan Direct: +81-3-5798-8482
(10/03/04 11:47), Masaharu Kawada wrote:
Hi Mark-san, Kikuchi-san,
Thank you very much for your help.
Just to make sure, about the patch, is it just need to be modified in the way you mentioned? Which means that it should be look like below.
Yes, its OK.
# vi /usr/lib/python2.4/email/Charset.py
---<Before modifying>--- 243 def convert(self, s): 244 """Convert a string from the input_codec to the output_codec.""" 245 if self.input_codec <> self.output_codec: 246 return unicode(s, self.input_codec).encode(self.output_codec) 247 else: 248 return s
---<After modifying>--- 243 def convert(self, s): 244 """Convert a string from the input_codec to the output_codec.""" 245 if self.input_codec != self.output_codec: 246 return unicode(s, self.input_codec, 'replace').encode(self.output_codec, 'replace') 247 return unicode(s, self.input_codec).encode(self.output_codec) 248 else: 249 return s
Best Regards,
Tokio Kikuchi wrote:
Hi,
(10/03/04 1:58), Mark Sapiro wrote:
Masaharu Kawada wrote:
There is one thing that I wonder, is that if the charset.py is done with that patch, what the current digest.mbox exsisted under lists/<listname> are supposed to be? Do they need to be deleted or not? My point on this is that once the patch is done, is there nothing else to do to fix this problem?
That is correct. Assuming the underlying issue is fixed by the patch, all you need to do is apply the patch to the Python email library charset.py module (probably at /usr/lib/pythonx.x/lib/email/charset.py) and restart Mailman. Then the next time the digest is triggered, it will be sent with all messages and no more errors.
If you apply the patch and restart Mailman and the errors continue, then they are caused by something else.
In addition, you should have a lot of shunted messages in qfiles/shunt. They are safely deleted.
Note that the patch:
--- Lib/email/charset.py 2009-09-22 08:59:56.000000000 +0900 +++ Lib/email/charset.py.orig 2009-09-22 08:58:36.000000000 +0900 @@ -264,8 +264,7 @@ def convert(self, s): """Convert a string from the input_codec to the output_codec.""" if self.input_codec != self.output_codec: - return unicode(s, self.input_codec, 'replace' - ).encode(self.output_codec, 'replace') + return unicode(s, self.input_codec).encode(self.output_codec) else: return s
is reversed. The '+' is the original code and the '-' is the new code.
Yes. Thanks Mark.
-- Tokio Kikuchi tkikuchi@is.kochi-u.ac.jp http://weather.is.kochi-u.ac.jp/
Kikuchi-san, Mark-san, I will ask the customer to consider patching it. Thanks a million!!! Best Regards, Tokio Kikuchi wrote:
(10/03/04 11:47), Masaharu Kawada wrote:
Hi Mark-san, Kikuchi-san,
Thank you very much for your help.
Just to make sure, about the patch, is it just need to be modified in the way you mentioned? Which means that it should be look like below.
Yes, its OK.
# vi /usr/lib/python2.4/email/Charset.py
---<Before modifying>--- 243 def convert(self, s): 244 """Convert a string from the input_codec to the output_codec.""" 245 if self.input_codec <> self.output_codec: 246 return unicode(s, self.input_codec).encode(self.output_codec) 247 else: 248 return s
---<After modifying>--- 243 def convert(self, s): 244 """Convert a string from the input_codec to the output_codec.""" 245 if self.input_codec != self.output_codec: 246 return unicode(s, self.input_codec, 'replace').encode(self.output_codec, 'replace') 247 return unicode(s, self.input_codec).encode(self.output_codec) 248 else: 249 return s
Best Regards,
Tokio Kikuchi wrote:
Hi,
(10/03/04 1:58), Mark Sapiro wrote:
Masaharu Kawada wrote:
There is one thing that I wonder, is that if the charset.py is done with that patch, what the current digest.mbox exsisted under lists/<listname> are supposed to be? Do they need to be deleted or not? My point on this is that once the patch is done, is there nothing else to do to fix this problem?
That is correct. Assuming the underlying issue is fixed by the patch, all you need to do is apply the patch to the Python email library charset.py module (probably at /usr/lib/pythonx.x/lib/email/charset.py) and restart Mailman. Then the next time the digest is triggered, it will be sent with all messages and no more errors.
If you apply the patch and restart Mailman and the errors continue, then they are caused by something else.
In addition, you should have a lot of shunted messages in qfiles/shunt. They are safely deleted.
Note that the patch:
--- Lib/email/charset.py 2009-09-22 08:59:56.000000000 +0900 +++ Lib/email/charset.py.orig 2009-09-22 08:58:36.000000000 +0900 @@ -264,8 +264,7 @@ def convert(self, s): """Convert a string from the input_codec to the output_codec.""" if self.input_codec != self.output_codec: - return unicode(s, self.input_codec, 'replace' - ).encode(self.output_codec, 'replace') + return unicode(s, self.input_codec).encode(self.output_codec) else: return s
is reversed. The '+' is the original code and the '-' is the new code.
Yes. Thanks Mark.
-- ------------------- Masaharu Kawada Associate Technical Support Engineer Red Hat K K Ebisu Neonato 5F 1-18 Ebisu 4-chome, Shibuya-ku Tokyo 150-0013, Japan Direct: +81-3-5798-8482
Masaharu Kawada wrote:
Kikuchi-san, Mark-san,
I will ask the customer to consider patching it.
Thanks a million!!!
Best Regards,
Tokio Kikuchi wrote:
(10/03/04 11:47), Masaharu Kawada wrote:
Hi Mark-san, Kikuchi-san,
Thank you very much for your help.
Just to make sure, about the patch, is it just need to be modified in the way you mentioned? Which means that it should be look like below.
Yes, its OK.
# vi /usr/lib/python2.4/email/Charset.py
---<Before modifying>--- 243 def convert(self, s): 244 """Convert a string from the input_codec to the output_codec.""" 245 if self.input_codec <> self.output_codec: 246 return unicode(s, self.input_codec).encode(self.output_codec) 247 else: 248 return s
---<After modifying>--- 243 def convert(self, s): 244 """Convert a string from the input_codec to the output_codec.""" 245 if self.input_codec != self.output_codec: 246 return unicode(s, self.input_codec, 'replace').encode(self.output_codec, 'replace') 247 return unicode(s, self.input_codec).encode(self.output_codec) 248 else: 249 return s
Actually, that's not quite right. It should be
---<After modifying>--- 243 def convert(self, s): 244 """Convert a string from the input_codec to the output_codec.""" 245 if self.input_codec != self.output_codec: 246 return unicode(s, self.input_codec, 'replace' 247 ).encode(self.output_codec, 'replace') 248 else: 249 return s
The exact line numbers will depend on the Python version.
Although, what you have would work because the first return at line 246 is correct although it is all on one line and the second return at line 246 which is the problem will never be reached.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (3)
-
Mark Sapiro -
Masaharu Kawada -
Tokio Kikuchi