[Mailman-Users] Really Fustrating - Searchable Archives
prussell at mteliza.com.au
Fri Jul 11 00:28:33 CEST 2003
Richard, thanks for your guide - i followed it step by step and everything
appears to work now, first time. This wasnt even really that hard, just
needed the guide to follow.
Maybe you could get this on a webpage and have a link where ever your
patches live? Or is it just me who couldnt do it? :)
I think i will just swap the production machine for my test machine rather
than do all that remotely.
Richard Barrett <r.barrett at openinfo.co.uk>
10/07/2003 09:21 AM
To: "Peter Russell" <prussell at mteliza.com.au>
Subject: Re: [Mailman-Users] Really Fustrating - Searchable Archives
I did the notes below for somebody else wanting to upgrade Mailman 2.1.2
installed from source. If you have installed Mailman from source on your
test machine why not take a look at my notes and see what you think. Try
adding the htdig integration to your test machine to see how much of a
problem you find it.
If you like I will take a look at a procedure for de-installing the RPM
installed MM on your production machine and replacing it with MM from
source. One thing you could do is send me the output from the following
rpm -ql mailman
I run Suse Linux rather than RedHat so I cannot readily check the exact
details of the files RedHat's mailman RPM package installs. The command
I've given provides that information.
The following notes are applicable to MM 2.1.2 installed from source.
Phase 1 - Initial preparation of Mailman
1. Download the following patch files and save them in the directory above
your Mailman build directory.
2. gunzip all of the patch files that are .gz suffix. For example use the
3. With your MM build directory as current working directory apply the
patch -p1 < ../templates-2.1.2-0.1.patch
patch -p1 < ../tcache-2.1.2-0.3.patch
patch -p1 < ../tz-2.1.2-0.2.patch
patch -p1 < ../indexing-2.1.2-0.1.patch
patch -p1 < ../htdig-2.1.2-0.3.patch
4. Run the Mailman ./configure using exactly the same command as you ran
when you originally installed Mailman.
5. Use mailmanctl to stop Mailman's daemons. For example use the command:
6. Remove Mailman's crontab. For example use the following command as
crontab -r -u mailman
7. Stop your MTA. For example you might run, as root, the following
if you were running sendmail:
8. Stop your web server. For example you might run, as root, the following
9. Run the Mailman make install:
10. Run the Mailman checkperms with $prefix as the current working
director, re-running it with the -f option if any errors are reported:
11. Restart your web server.
12. Reinstall the Mailman crontab, which should now have an extra entry
running nightly_htdig in it.
13. Use mailmanctl to restart Mailman's daemons.
Mailman should now be patched to support the use of Htdig for searching
nothing will be activated until Htdig is installed and some Mailman
configuration variables have been set up. Check Mailman is working
in all other respects.
Phase 2 - Installing Htdig
I strongly recommend using Htdig 3.1.6. I have not tested my mailman-htdig
patches with other than the latest stable release which is 3.1.6. RedHat
seems to distribute htdig 3.2b3 as if it were a stable release and I have
had several reports of people having problems of one sort or another with
1. Getting the htdig 3.1.6 source via this page http://htdig.org/where.html
works for me.
2. Having unpacked the source package I then run commands, as root, to
configure and install it, basically following the instructions here
http://htdig.org/install.html, with the following commands:
3. In my case this installs htdig into the default /opt/www/htdig/
directory with some of the web related stuff under the ServerRoot
for my Apache server. If ServerRoot in your Apache server's
/etc/httpd/conf/httpd.conf is not /usr/local/httpd then change the
./configure line above to be what your system is setup for.
4. To overcome permissions problems when rundig is first used to build
search indexes you should change the group ownership of some of the htdig
installation directories to allow the mailman user and group write access
to those directories and the files in them. In my case I say:
chgrp -R mailman /opt/www/htdig/conf /opt/www/htdig/db
Phase 3 - Configuring Mailman to use Htdig
1. Add the following lines to your $prefix/Mailman/mm-cfg.py file. Note
again that the paths/URL on three of these lines need to fit in with the
way you installed htdig in Phase 2.
# Mailman-htdig configuration
USE_HTDIG = 1
HTDIG_RUNDIG_PATH = '/opt/www/htdig/bin/rundig'
HTDIG_HTSEARCH_PATH = '/opt/www/htdig/bin/htsearch'
HTDIG_FILES_URL = '/htdig/'
2. Use mailmanctl to restart the Mailman daemons.
3. From now on, the first new message sent to each list will trigger the
creation of list-specific htdig.conf files and add a serach form to the
list's archive TOC page. Until such a message is received by a list no
search facility will be enabled for it.
4. Until the nightly_htdig cron script is run, after the first new message
to a list, the list will have no search indexes. You can run, as the
mailman user, the nightly_htdig script from the command line.
For detailed information see the INSTALL.htdig-mm file added to the MM
build directory by the patches.
With luck and a fair wind you should now be up and running with htdig
searchable list archives.
At 23:18 09/07/2003, you wrote:
>MANY thanks for taking the time to try and help me - much appreciated.
>I suppose io want to emulate what list.org have on the mailman-users
>archive. I hope i dont sound unappreciative, i realise that this software
>(and all the patches and the support) are people investing thier own time
>for little/no financial return - sounds like my job), i really do think
>mailman is fantatsic, but these things are fustrating, especiually when
>you see how good it can work when you see the searchabkle archives at
>list.org - kind of expected to get that when i installed it.
>Again many thanks in advance for trying to help me - answers to your
>1. Both. Internet/international users and local 'staff'.
>2. All public - we only have a handlfull of lists in mind.
>3. Preferably i would like to keep the search specific to each list. But
>dont think this is essentail.
>4. I dont have anything like that installed - i have mailman, sendmail
>apache. Except for a little bit of mail relaying for machines on our
>private network, the mailman machine is dedicated to mailman
>5. We only have a handful of Linux machines, everything else in win32 -
>boss beleives that linux is not mature or robust enough for enterprise
>:( In fact he says same about win2k and insists on using NT4 - i am sure
>he doesnt even like that and would prefer we had %100 DOS environment. We
>dont have any other linux machines running any kind of search software.
>6. I dont completely understand this question - i want to be able to
>search on the contents of the archive - the body/subject of messages.
>7. Red Hat 9 and the mailman version that came with it, RPM - which is a
>little bit of a hassle, because i am not an expert i dont know where to
>begin in regards to recompile and apply tio RPM instalation - where as
>i know that with a little bit of effort compiling was easy enough on my
>test machine. (i have a test machine i am trying to get this all working
>8. I typed python and it returns 2.2.2 - it came with red hat 9
>Richard Barrett <r.barrett at openinfo.co.uk>
>10/07/2003 12:40 AM
> To: "Peter Russell" <prussell at mteliza.com.au>
> cc: mailman-users at python.org
> Subject: Re: [Mailman-Users] Really Fustrating -
> Searchable Archives
>At 14:34 09/07/2003, Peter Russell wrote:
> >Not being a Linux guru, just getting mailman and sendmail or postfix
> >set up and working properly is a fair task. Attacking a working
> >to get mhonarc and htdig working and MM patched is just an excercise in
> >fustration. (without a detailed howto, slightly beyond my linux skills)
>You do not have to install Mhonarc to get searchable archives. Mhonarc is
>an external archiver and some people prefer that to using the Mailman
>internal pipermail archiver. Adding Mhonarc to the equation just
>complicates the issues for you rather than simplifying adding search.
>With the following information available, more directed advice will be
>1. are you making the list archives available over the internet or only
>within your own domain?
>2. are all your list archives public or do you have some private archives
>whose privacy you want to preserve over search?
>3. do you want per-list search indexes that will only return search hits
>relevant to individual lists or do you want site-wide search indexes that
>may return hits from many lists?
>4. do you have any search engine installed on the server running Mailman?
>If so what is it?
>5. do you have any search engine installed on any other server in your
>network? If so what is it and can you change its configuration files so
>that it can be directed to also index your list archives?
>6. do you want search forms on the TOC pages of individual list archives?
>7. what OS and version are you running?
>8. what version of Python are you running?
>8. what version of MM are you running and how was it installed: from
>or was it packaged?
>With the above information it will be easier to produce a directed guide
>setting up search for your list archives.
> >Is there anywhere that a 1, 2, 3 - press this then type that type guide
> >exists to achieve threaded archiving and a good search feature?
>Answer the above questions and you may get your wish. btw: Mailman's
>internal archiver produces threaded archives out of the box.
> >OR is there any chance that these types of features are going to be
> >avilable in the MM release? Or is there a CVS i can download and
>It is being considered.
> >Can anyone tell me why archive search (or even threaded archives) are
> >used out of the box with this product?
>A fair number of people are running htdig with or without using the
>integration patches. It really is not that difficult to install and
>Like a lot of Open Software, there may be a certain lack of polish but
>base technology is pretty good and comes at a remarkably good price.
>btw: did I mention that Mailman's internal archiver produces threaded
>archives out of the box.
> >Is there a product that does EVERYTHING that mailman does and does have
> >search and archives features too?
>Maybe, but a mailing list supporting no-cost Open Software is probably
>the place to look for it.
> >The punters who wanted mailman are starting to get mad as hell at me
> >not having these features going, i appreciate ANY input/suggestions.
>Richard Barrett http://www.openinfo.co.uk
Richard Barrett http://www.openinfo.co.uk
More information about the Mailman-Users