[Mailman-Developers] re: Much cpu/memory load

Dan Mick Dan.Mick@sun.com
Fri Nov 1 20:20:11 2002


Simone Piunno wrote:
> On Thu, Oct 31, 2002 at 07:03:50PM -0800, Dan Mick wrote:
> 
> 
>>The problem's affecting me badly right now too.  So far it seems that the 
>>holdup is in the 'date.html' index file processing; everything else is 
>>finished, but the "bin/qrunner -r Arch -o" process has lost its little mind 
>>trying to write the date index.  It's in a loop, chewing up 409600 bytes of 
>>memory (brk), and then opening, reading, and closing archidxentry.html,
>>hundreds of times in a row.
> 
> 
> Ok, I believe this is because of my patch for i18n in archives.
> Do you feel like trying this patch?  
> It should speed up things a lot....
> 
> Index: HyperArch.py
> ===================================================================
> RCS file: /cvsroot/mailman/mailman/Mailman/Archiver/HyperArch.py,v
> retrieving revision 2.22
> diff -u -r2.22 HyperArch.py
> --- HyperArch.py        19 Oct 2002 20:59:27 -0000      2.22
> +++ HyperArch.py        1 Nov 2002 10:13:21 -0000
> @@ -923,10 +923,14 @@
>              'sequence': article.sequence,
>              'author':   author
>          }
> -        print Utils.maketext(
> -            'archidxentry.html', d, raw=1,
> -            lang=self.maillist.preferred_language,
> -            mlist=self.maillist)
> +        print """<LI><A HREF="%(filename)s">%(subject)s
> +        </A><A NAME="%(sequence)i">&nbsp;</A>
> +        <I>%(author)s
> +        </I>""" % d
> +        #print Utils.maketext(
> +        #    'archidxentry.html', d, raw=1,
> +        #    lang=self.maillist.preferred_language,
> +        #    mlist=self.maillist
> 

That's definitely the cause of the many opens of archidxentry.html, but it
doesn't explain the 400K-per-iteration growth.

Barry, wasn't there some Python instrumentation trick for using the GC to 
find unreferenced objects and complain, or invoking GC manually in a loop 
to try to alleviate such problems?....was it in the gc module?...