S. Patrick Eaton wroye:

I've been having a number of problems with the distribution of digest messages that I am hoping someone here can help me with.

The first is that, unless the Mailman administrative interface is changed to Japanese, any Japanese characters included in digest messages get changed into question mark characters when the digest is sent out, even if those characters appeared correctly in the message that was sent out to non-digest subscribers. If the interface is changed to Japanese, this does not occur; Japanese characters then appear in the digests the way they should.

This occurs with 'plain' format digests because they are produced in the character set of the list's language which for English is us-ascii. It does not occur with the MIME format digest because in that digest, the individual messages are separate MIME parts that retain the original character set and encoding.

You could try setting the digest members' digest format to MIME (uncheck plain in Membership Management... -> Membership List for digest members and set Digest options -> mime_is_default_digest to MIME so new members get MIME digests by default.

This may or may not be satisfactory for any particular digest member depending on the member's MUA. Some MUAs including those on smart phones do not handle MIME format digests well.

This, however, inevitably runs into another problem: When the interface is changed to Japanese, the distribution of digests continues for a while, but inevitably it stops at some point and does not resume until the interface is changed back to English, at which point all of the messages that had not been distributed in digests since the last digest was sent get included in the next digest message, which can be quite massive if the digests have not been getting sent out regularly for a while.

[...]

I have encountered this very same problem in two Mailman installations, one running Mailman 2.1.9 and another running Mailman 2.1.12, on two entirely separate hosted environments, so I am reasonably sure that the problem is specific to Mailman and not to installation or environment-related issues.

There should be a message in Mailman's error log saying "send_digests() failed:" with a specific exception. Beginning with Mailman 2.1.14, a traceback from the exception is also logged.

The exception itself occurs because there is some message among those accumulated for the digest (in the lists/LISTNAME/digest.mbox file) that is causing some exception to be thrown in processing the messages for the digest. It is not possible to say what the problem is without seeing the exception and the offending message or digest.mbox.

This problem used to also stop list delivery when the digest.mbox reached the threshold size that would trigger a digest, but that bug has been fixed. However,your situation is more difficult. Ideally, Mailman would just remove the offending message from the digest and store it aside somewhere, but doing this is not straightforward,and it currently isn't done, so what happens is the exception aborts the digest processing, but the digest.mbox is unchanged, so the exception continues each time there is an attempt to produce a digest.

In your situation, the exception depends on the list's language so changing the language allows a digest to be produced and the digest.mbox removed so the problem doesn't recur even if you change the language back until another offending message arrives.

If you are able to report the exception from the error log, please do. Also, if you are able and willing to send me (off list) a copy of an offending lists/LISTNAME/digest.mbox file or put it on the web somewhere and post or send me a link, I will try to find just what it is about certain messages that causes this.

A possible avoidance is to change Mailman's character set for English from us-ascii to utf-8 and leave the list's language set to English. To do this, you would add the line

add_language('en', 'English (USA)', 'utf-8')

to mm_cfg.py. You could possibly also avoid this by changing Mailman's character set for Japanese from euc-jp to utf-8, but this is more difficult as it also requires transliterating the Japanese templates and message catalog.

-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan