[Mailman-Users] Mailman locking up, seems to be Spamassassin related

[Richard Barrett]

Further to my earlier comments, you might be able to alleviate some  
of the problems with your integration of Spamassassin with Mailman by  
having multiple IncomingRunners. To do this, see the comments  
adjacent to the definition of QRUNNERS in your Defaults.py. You could  
redefine QRUNNERS in you mm_cfg.py and increase the count of  
IncomingRunners in that redefinition. But beware, multiple instances  
of the same Runner type simply split the contents of the queue it  
works on between them by filename slices. A stalled IncomingRunner  
instance will still act as a road block for all those messages in the  
incoming queue that the it is responsible for; another runner  
instance cannot take over those blocked files. Spotting a jammed  
IncomingRunner will be quite difficult. At least when a single  
IncomingRunner is jammed it is more obvious.

On 27 Jun 2006, at 12:41, Richard Barrett wrote:

>
> On 27 Jun 2006, at 10:33, Chris Croome wrote:
>
>> Hi
>>
>> One a CentOS 4 box, running Mailman that comes with the distro, I'm
>> seeing locks up every now and then -- all the lists on the machine
>> stop
>> processing mail.
>>
>> When this happens a restart gets things moving again.
>>
>> This is the error that is written to /var/log/mailman/error:
>>
>>  Jun 26 12:04:58 2006 (28052) Uncaught runner exception: (4,
>> 'Interrupted system call')
>>  Jun 26 12:04:58 2006 (28052) Traceback (most recent call last):
>>   File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 111, in
>> _oneloop
>>     self._onefile(msg, msgdata)
>>   File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 167, in
>> _onefile
>>     keepqueued = self._dispose(mlist, msg, msgdata)
>>   File "/usr/lib/mailman/Mailman/Queue/IncomingRunner.py", line
>> 130, in _dispose
>>     more = self._dopipeline(mlist, msg, msgdata, pipeline)
>>   File "/usr/lib/mailman/Mailman/Queue/IncomingRunner.py", line
>> 153, in _dopipeline
>>     sys.modules[modname].process(mlist, msg, msgdata)
>>   File "/usr/lib/mailman/Mailman/Handlers/SpamAssassin.py", line
>> 76, in process
>>     score, symbols = check_message(mlist, str(msg))
>>   File "/usr/lib/mailman/Mailman/Handlers/SpamAssassin.py", line
>> 62, in check_message
>>     res = connection.check(spamd.SYMBOLS, message)
>>   File "/usr/lib/mailman/Mailman/Handlers/spamd.py", line 123, in
>> check
>>     response = fp.readline()
>>   File "/usr/lib/python2.3/socket.py", line 338, in readline
>>     data = self._sock.recv(self._rbufsize)
>>  error: (4, 'Interrupted system call')
>>
>>  Jun 26 12:04:58 2006 (28052) SHUNTING: 1151298261.692024
>> +45c6f4028caac564c9880d1ac3f73dca4b88bfb1
>>
>> Mailman has been set up to work with Spamassassin using the
>> instructions
>> here:
>>
>>   http://www.jamesh.id.au/articles/mailman-spamassassin/
>>
>> As far as I can tell the only thing that this involves is this line
>> being added to /etc/mailman/mm_cfg.py:
>>
>>   GLOBAL_PIPELINE.insert(1, 'SpamAssassin')
>>
>> (I didn't set this machine up however...).
>>
>> Does anyone have any ideas what could be going wrong here and what
>> might
>> be don't to fix it...?
>>
>
> I do not use this myself but a quick look at the code available at
> the site you reference above shows that the addition to the Mailman
> pipeline is synchronously passing the message to an external/
> separate process for checking and then awaiting its response.
>
> That pipeline is operated by the IncomingRunner of which there is
> usually only one such queue runner. Hence if there is any delay in
> the external SpamAssassin processing a message all other incoming
> messages will just wait in the queue.
>
> The code communicating with the checking process does not appear to
> protect itself with any timeouts so if the external process fails to
> respond for prolonged periods then your Mailman system will simply
> appear to lock up as far as processing incoming messages is
> concerned. In effect it will lockup the entire Mailman system as
> without new messages to process, the other queue runners have nothing
> to do.
>
> Personally, I would not adopt this approach to running SpamAssassin.
> But if I were to use this approach I would re-engineer the queue
> runner/pipeline to allow multiple instances of the external checking
> process to be utilised and add some timeout protection.
>
> But better, I would integrate SpamAssassin into the MTA delivering to
> the Mailman server. We use the latter approach so that unchecked mail
> never gets near Mailman (or any other MTA/MDA/MUA on our site). This
> should give much higher levels of concurrency when checking incoming
> than this serial approach with Spamassassin in the Mailman pipeline.
> This is in fact what we do.
>
> Incidentally, we also use this check all the mail going off-site from
> our systems via the MTA so that we can avoid any infected machine in
> our network passing its problem on to the outside world.
>
> Maybe you should discuss the problem with the author of the
> Spamassassin integration code.
>
>> Thanks
>>
>> Chris
>>
>> -- 
>> Chris Croome                                
>> <chris at webarchitects.co.uk>
>> web design                             http:// 
>> www.webarchitects.co.uk/
>> web content management                               http:// 
>> mkdoc.com/
>