[Mailman-Users] Really Fustrating - Searchable Archives

Peter Russell prussell at mteliza.com.au
Fri Jul 11 00:28:33 CEST 2003

Richard, thanks for your guide - i followed it step by step and everything 
appears to work now, first time. This wasnt even really that hard, just 
needed the guide to follow.

Maybe you could get this on a webpage and have a link where ever your 
patches live? Or is it just me who couldnt do it? :)

I think i will just swap the production machine for my test machine rather 
than do all that remotely.

Many thanks

Richard Barrett <r.barrett at openinfo.co.uk>
10/07/2003 09:21 AM

        To:     "Peter Russell" <prussell at mteliza.com.au>
        Subject:        Re: [Mailman-Users] Really Fustrating - Searchable Archives


I did the notes below for somebody else wanting to upgrade Mailman 2.1.2 
installed from source. If you have installed Mailman from source on your 
test machine why not take a look at my notes and see what you think. Try 
adding the htdig integration to your test machine to see how much of a 
problem you find it.

If you like I will take a look at a procedure for de-installing the RPM 
installed MM on your production machine and replacing it with MM from 
source. One thing you could do is send me the output from the following 

rpm -ql mailman

I run Suse Linux rather than RedHat so I cannot readily check the exact 
details of the files RedHat's mailman RPM package installs. The command 
I've given provides that information.




The following notes are applicable to MM 2.1.2 installed from source.

Phase 1 - Initial preparation of Mailman

1. Download the following patch files and save them in the directory above 

your Mailman build directory.

templates-2.1.2-0.1.patch from 

tcache-2.1.2-0.3.patch.gz from 

tz-2.1.2-0.2.patch from 

indexing-2.1.2-0.1.patch.gz from 

htdig-2.1.2-0.3.patch.gz from 

2. gunzip all of the patch files that are .gz suffix. For example use the 

gunzip indexing-2.1.2-0.1.patch.gz

3. With your MM build directory as current working directory apply the 

patch -p1 < ../templates-2.1.2-0.1.patch
patch -p1 < ../tcache-2.1.2-0.3.patch
patch -p1 < ../tz-2.1.2-0.2.patch
patch -p1 < ../indexing-2.1.2-0.1.patch
patch -p1 < ../htdig-2.1.2-0.3.patch

4. Run the Mailman ./configure using exactly the same command as you ran 
when you originally installed Mailman.

5. Use mailmanctl to stop Mailman's daemons. For example use the command:

$prefix/bin/mailmanctl stop

6. Remove Mailman's crontab. For example use the following command as 

crontab -r -u mailman

7. Stop your MTA. For example you might run, as root, the following 
if you were running sendmail:

/etc/rc.d/init.d/sendmail stop

8. Stop your web server. For example you might run, as root, the following 


/etc/rc.d/init.d/apache stop

9. Run the Mailman make install:

make install

10. Run the Mailman checkperms with $prefix as the current working 
director, re-running it with the -f option if any errors are reported:


11. Restart your web server.

12. Reinstall the Mailman crontab, which should now have an extra entry 
running nightly_htdig in it.

13. Use mailmanctl to restart Mailman's daemons.

Mailman should now be patched to support the use of Htdig for searching 
nothing will be activated until Htdig is installed and some Mailman 
configuration variables have been set up. Check Mailman is working 
in all other respects.

Phase 2 - Installing Htdig

I strongly recommend using Htdig 3.1.6. I have not tested my mailman-htdig 

patches with other than the latest stable release which is 3.1.6. RedHat 
seems to distribute htdig 3.2b3 as if it were a stable release and I have 
had several reports of people having problems of one sort or another with 

1. Getting the htdig 3.1.6 source via this page http://htdig.org/where.html 
works for me.

2. Having unpacked the source package I then run commands, as root, to 
configure and install it, basically following the instructions here 
http://htdig.org/install.html, with the following commands:

./configure  --with-cgi-bin-dir=/usr/local/httpd/cgi-bin 
make install

3. In my case this installs htdig into the default /opt/www/htdig/ 
directory with some of the web related stuff under the ServerRoot 
for my Apache server. If ServerRoot in your Apache server's 
/etc/httpd/conf/httpd.conf is not /usr/local/httpd then change the 
./configure line above to be what your system is setup for.

4. To overcome permissions problems when rundig is first used to build 
search indexes you should change the group ownership of some of the htdig 
installation directories to allow the mailman user and group write access 
to those directories and the files in them. In my case I say:

chgrp -R mailman /opt/www/htdig/conf /opt/www/htdig/db

Phase 3 - Configuring Mailman to use Htdig

1. Add the following lines to your $prefix/Mailman/mm-cfg.py file. Note 
again that the paths/URL on three of these lines need to fit in with the 
way you installed htdig in Phase 2.

# Mailman-htdig configuration
HTDIG_RUNDIG_PATH = '/opt/www/htdig/bin/rundig'
HTDIG_HTSEARCH_PATH = '/opt/www/htdig/bin/htsearch'
HTDIG_FILES_URL = '/htdig/'

2. Use mailmanctl to restart the Mailman daemons.

3. From now on, the first new message sent to each list will trigger the 
creation of list-specific htdig.conf files and add a serach form to the 
list's archive TOC page. Until such a message is received by a list no 
search facility will be enabled for it.

4. Until the nightly_htdig cron script is run, after the first new message 

to a list, the list will have no search indexes. You can run, as the 
mailman user, the nightly_htdig script from the command line.

For detailed information see the INSTALL.htdig-mm file added to the MM 
build directory by the patches.

With luck and a fair wind you should now be up and running with htdig 
searchable list archives.

At 23:18 09/07/2003, you wrote:

>MANY thanks for taking the time to try and help me - much appreciated.
>I suppose io want to emulate what list.org have on the mailman-users 
>archive. I hope i dont sound unappreciative, i realise that this software 

>(and all the patches and the support) are people investing thier own time 

>for little/no financial return - sounds like my job), i really do think 
>mailman is fantatsic, but these things are fustrating, especiually when 
>you see how good it can work when you see the searchabkle archives at 
>list.org - kind of expected to get that when i installed it.
>Again many thanks in advance for trying to help me - answers to your 
>questiosn follow.
>1. Both. Internet/international users and local 'staff'.
>2. All public - we only have a handlfull of lists in mind.
>3. Preferably i would like to keep the search specific to each list. But 
>dont think this is essentail.
>4. I dont have anything like that installed - i have mailman, sendmail 
>apache. Except for a little bit of mail relaying for machines on our 
>private network, the mailman machine is dedicated to mailman
>5. We only have a handful of Linux machines, everything else in win32 - 
>boss beleives that linux is not mature or robust enough for enterprise 
>:( In fact he says same about win2k and insists on using NT4 - i am sure 
>he doesnt even like that and would prefer we had %100 DOS environment. We 

>dont have any other linux machines running any kind of search software.
>6. I dont completely understand this question - i want to be able to 
>search on the contents of the archive - the body/subject of messages.
>7. Red Hat 9 and the mailman version that came with it, RPM - which is a 
>little bit of a hassle, because i am not an expert i dont know where to 
>begin in regards to recompile and apply tio RPM instalation - where as 
>i know that with a little bit of effort compiling was easy enough on my 
>test machine. (i have a test machine i am trying to get this all working 
>on first)
>8. I typed python and it returns 2.2.2 - it came with red hat 9
>Richard Barrett <r.barrett at openinfo.co.uk>
>10/07/2003 12:40 AM
>         To:        "Peter Russell" <prussell at mteliza.com.au>
>         cc:        mailman-users at python.org
>         Subject:        Re: [Mailman-Users] Really Fustrating - 
> Searchable Archives
>At 14:34 09/07/2003, Peter Russell wrote:
> >Not being a Linux guru, just getting mailman and sendmail or postfix 
> >set up and working properly is a fair task. Attacking a working 
> >to get mhonarc and htdig working and MM patched is just an excercise in
> >fustration. (without a detailed howto, slightly beyond my linux skills)
>You do not have to install Mhonarc to get searchable archives. Mhonarc is
>an external archiver and some people prefer that to using the Mailman
>internal pipermail archiver. Adding Mhonarc to the equation just
>complicates the issues for you rather than simplifying adding search.
>With the following information available, more directed advice will be
>1. are you making the list archives available over the internet or only
>within your own domain?
>2. are all your list archives public or do you have some private archives
>whose privacy you want to preserve over search?
>3. do you want per-list search indexes that will only return search hits
>relevant to individual lists or do you want site-wide search indexes that
>may return hits from many lists?
>4. do you have any search engine installed on the server running Mailman?
>If so what is it?
>5. do you have any search engine installed on any other server in your
>network? If so what is it and can you change its configuration files so
>that it can be directed to also index your list archives?
>6. do you want search forms on the TOC pages of individual list archives?
>7. what OS and version are you running?
>8. what version of Python are you running?
>8. what version of MM are you running and how was it installed: from 
>or was it packaged?
>With the above information it will be easier to produce a directed guide 
>setting up search for your list archives.
> >Is there anywhere that a 1, 2, 3 - press this then type that type guide
> >exists to achieve threaded archiving and a good search feature?
>Answer the above questions and you may get your wish. btw: Mailman's
>internal archiver produces threaded archives out of the box.
> >OR is there any chance that these types of features are going to be
> >avilable in the MM release? Or is there a CVS i can download and 
>It is being considered.
> >Can anyone tell me why archive search (or even threaded archives) are 
> >used out of the box with this product?
>A fair number of people are running htdig with or without using the 
>integration patches. It really is not that difficult to install and 
>Like a lot of Open Software, there may be a certain lack of polish but 
>base technology is pretty good and comes at a remarkably good price.
>btw: did I mention that Mailman's internal archiver produces threaded
>archives out of the box.
> >Is there a product that does EVERYTHING that mailman does and does have 
> >search and archives features too?
>Maybe, but a mailing list supporting no-cost Open Software is probably 
>the place to look for it.
> >The punters who wanted mailman are starting to get mad as hell at me 
> >not having these features going, i appreciate ANY input/suggestions.
>Richard Barrett                                      http://www.openinfo.co.uk

Richard Barrett                                      http://www.openinfo.co.uk

More information about the Mailman-Users mailing list