[Mailman-Users] Attachments are unexpectedly re-created.

[Masaharu Kawada]

Mark-san,

I am sorry that I would like to know one more thing about
digestable in advance.

>>---
>>xxx xx xx:xx:xx 2010 (2129) send_digests() failed: EUC-JP decoding error:
>>invalid character ......
>>xxx xx xx:xx:xx 2010 (2129) send_digests() failed: EUC-JP decoding error:
>>invalid character ......
>>xxx xx xx:xx:xx 2010 (2129) send_digests() failed: EUC-JP decoding error:
>>invalid character ......
>>---

 >Yes, I think this is the problem. Somewhere in the
 >lists/LISTNAME/digest.mbox file there is a bad message that causes
 >this error.

It seems that the operation of the "send_digests()" starts via corn
at noon(12:00) every day. However, re-creation attachment files
problem happens even other time(not only at noon but also other time),
so the "send_digests()" operation seems to be done sometime
except at the specified time of cron. If my understanding is right,
the "send_digests()" operation repeats(retries) sevral times after
the fail of its first action. On this matter, what I would like to know
is that what the interval of the retry, and how many time does it
repeat.

Could you please let me know if you know about this.

Best Regards,

Mark Sapiro wrote:
> Masaharu Kawada wrote:
>   
>> Firtst of all, one of my customer wanted to delete attachments to
>> make more disk spaces on the server. The server's disk resource
>> was getting full because the size of the directory for attachments
>> was very big. So the customer did the following works first.
>>
>> Digest options -> Disabled(Digestable 'NO')
>> Archiving options -> Disabled(Archive 'NO')
>>
>> Secondly, after the above, the customer deleted directories located
>> under the following directory such as 20091201, 20091202 and so on.
>>
>> /var/lib/mailman/archives/private/<list name>/attachments/<date>
>>
>> # rm -rf 20091201,20091202,.....
>>
>> Finaly, the customer got the "Digest options" and "Archiving options"
>> back to enabled as follows.
>>
>> Digest options -> Enabled(Digestable 'YES')
>> Archiving options -> Enabled(Archive 'YES')
>>     
>
>
> There was no need to turn off digestable and archive if they were only
> going to be turned on again.
>
>
>   
>> With this operations, the directories under attachments directory
>> were deleted and the disk space is now enough, however, by unknown
>> timing, directories that are named with past's date, which were
>> manually deleted, are re-created automatically. And in that 
>> directories, it seems that the same attachment files are repeatly
>> created.
>>
>> One thing that I realized to stop this weird action is to disable
>> both "Digest options" and "Archiving options". Once those options
>> are disabled, no more unexpected attachment files would be created
>> under that directory. 
>>
>> To know what was going on to the mailman server at that moment
>> (in the meantime of the unexpected action), I asked the customer
>> for logs such as error log in /var/log/mailman direcoty, and 
>> in the error log, I saw the following errors regarding "send_digests".
>>
>> ---
>> xxx xx xx:xx:xx 2010 (2129) send_digests() failed: EUC-JP decoding error:
>> invalid character ......
>> xxx xx xx:xx:xx 2010 (2129) send_digests() failed: EUC-JP decoding error:
>> invalid character ......
>> xxx xx xx:xx:xx 2010 (2129) send_digests() failed: EUC-JP decoding error:
>> invalid character ......
>> ---
>>
>> I have not yet been sure if these errors are related to the cuase,
>> but it seems that "send_digests()" appearently fails and retries
>> repeatly.
>>     
>
>
> Yes, I think this is the problem. Somewhere in the
> lists/LISTNAME/digest.mbox file there is a bad message that causes
> this error. Every time there is a post to the list, A digest is
> triggered because of the size of the digest.mbox. The mbox is
> processed and attachments are scrubbed (from the 'plain' digest) and
> stored up to the point of the exception which aborts the process until
> the next time, but each time those attachments prior to the bad
> message get stored again.
>
>
>   
>> My questions this time are as follows.
>>
>> 1.Why are attachment files re-created? Is it a normal action whenever
>>  "Digest options" and "Archiving options" are available?
>>     
>
>
> Non text/plain MIME parts and text/plain MIME parts with unknown
> character set are removed and stored aside and replaced by links in
> both the archive and the plain format digest. Thus there normally are
> two copies of each in the archive.
>
> In your case, the same messages in the digest.mbox are processed
> repeatedly causing the attachments to accumulate.
>
>
>   
>> 2.If the answer for the question 1 is "NO", what could be the cause?
>>  The fail of send_digests() is it?
>>
>> 3.Does the send_digests() repeat(retry) if the action fails?
>>
>> 4.Is there anything else that should be done after deleting attachments
>>   manually to avoid being suffered this weird behavior?
>>     
>
>
>
> The underlying cause is a message in the lists/LISTNAME/digest.mbox
> file with a body or sub part with declared as charset=EUC-JP with an
> invalid character or something related.
>
> Removing or moving aside the lists/LISTNAME/digest.mbox or finding the
> bad message and removing it from the file should also fix this.
>
> I consider this behavior to be a bug. I'll look into fixing it. I think
> what I'll need to do is just move the digest.mbox aside and log an
> error message. It would be better if I could just bypass the bad
> message, but that may be difficult.
>
>   


-- 
-------------------
Masaharu Kawada
Associate Technical Support Engineer
Red Hat K K
Ebisu Neonato 5F
1-18 Ebisu 4-chome, Shibuya-ku
Tokyo 150-0013, Japan
Direct: +81-3-5798-8482