I am trying to get htdig working with my mailman installation. Searched the archives and did not find the answer. I was hoping someone could point me in the right direction.
I have applied the patches and everything seem to be ok with the install, night rundig works fine, search form shows up on list info pages, but whenever I click the search button I am getting this error:
ht://Dig error
htsearch detected an error. Please report this to the webmaster of this site. The error message is: Unable to read configuration file
I am a little confused about the HTDIG_CONF_LINK_DIR and HTDIG_MAILMAN_LINK vars though and I suspect this may be where the error lies. My htdig is in /var/www/htdig/, and my mailman dir is /usr/local/mailman/. I have created a symbolic link htdig-mailman -> /usr/local/mailman/archives/htdig.
Does anyone have any suggestions?
At 07:13 26/02/2003, C. Posey wrote:
I'll guess from what you've said that there is a directory called /var/www/htdig/conf which contains a file called htdig.conf as part of your htdig installation. This is the dault directory where htdig expect to find config files and the default htdig.conf file.
If that is so, then you need to create the symlink called htdig-mailman in the /var/www/htdig/conf, pointing at /usr/local/mailman/archives/htdig.
Thanks for your reply:
What you described is exactly what I have...
[/var/www/htdig/conf] root# ls -la
total 24
drwxr-xr-x 2 root root 4096 Feb 25 23:49 .
drwxr-xr-x 6 root root 4096 May 19 2002 ..
lrwxrwxrwx 1 root root 33 Feb 25 23:49 htdig-mailman
-> /usr/local/mailman/archives/htdig
-rw-r--r-- 1 root root 8411 Feb 6 02:36 htdig.conf
-rw-r--r-- 1 root root 3102 Feb 25 23:48 mailman.conf.tgz
and in the dir pointed to by the symlink...
[/usr/local/mailman/archives/htdig/] root# ls -la
total 8
drwxrwsr-x 2 mailman mailman 4096 Feb 26 01:52 .
drwxrwsr-x 5 mailman mailman 4096 Feb 6 01:47 ..
lrwxrwxrwx 1 root mailman 74 Feb 6 07:33
macvoodoolist.conf ->
/usr/local/mailman/archives/private/macvoodoolist/htdig/
macvoodoolist.conf
lrwxrwxrwx 1 root mailman 62 Feb 6 02:24 mailman.conf ->
/usr/local/mailman/archives/private/mailman/htdig/mailman.conf
Any other ideas??
On Wednesday, February 26, 2003, at 04:00 PM, Richard Barrett wrote:
Just to be sure I patched everything correctly - I followed the
instructions in the Mailman FAQ...
http://www.python.org/cgi-bin/faqw-mm.py?req=all
Is this correct?
The patches to handle this are on the Mailman sourceforge patches
pages, with ids:-
[ #661138 ] corrects defects in some HTML templates (only for MM 2.1)
[ #668685 ] corrects error in sys.exit handling by script/driver
(only for MM 2.1)
[ #444879 ] Archive indexer control to improve indexing
[ #444884 ] Integration of Mailman & htdig for archive search
The URLs for the patches are
http://sourceforge.net/tracker/
index.php?func=detail&aid=661138&group_id=103&atid=300103
http://sourceforge.net/tracker/
index.php?func=detail&aid=444879&group_id=103&atid=300103
http://sourceforge.net/tracker/
index.php?func=detail&aid=668685&group_id=103&atid=300103
http://sourceforge.net/tracker/
index.php?func=detail&aid=444884&group_id=103&atid=300103
At 06:36 27/02/2003, C. Posey wrote:
The URLs are correct. The file names shown below are the correct ones for MM 2.1.1:
http://sourceforge.net/tracker/ index.php?func=detail&aid=661138&group_id=103&atid=300103
templates-2.1-0.1.patch.gz
http://sourceforge.net/tracker/ index.php?func=detail&aid=444879&group_id=103&atid=300103
indexing-2.1.1-0.1.patch.gz
http://sourceforge.net/tracker/ index.php?func=detail&aid=668685&group_id=103&atid=300103
driver-2.1-0.1.patch.gz
http://sourceforge.net/tracker/ index.php?func=detail&aid=444884&group_id=103&atid=300103
htdig-2.1.1-0.1.patch.gz
Actually I may have spoken too soon...
I am able to actually perform searches off the TOC now, but I am still getting this error generated from the cronjob...
DB2 problem...: /var/www/htdig/db/word2root.db: Permission denied
DB2 problem...: /var/www/htdig/db/root2word.db: Permission denied
/var/www/htdig/bin/rundig: line 68: 30671 Segmentation fault
$BINDIR/htfuzzy $opts endings
DB2 problem...: /var/www/htdig/common/synonyms.db: Permission denied
On Thursday, February 27, 2003, at 11:33 AM, C. Posey wrote:
Thx for all your help Richard!
The recompile of htdig did the trick....
This problem is discussed to some extent in the INSTALL.htdig-mm under the heading "htdig Permissions Considerations".
The problem is that the rundig process ends up trying to generate some shareable (across the entire intallation) db files in htdig's COMMON_DIR. If rundig is first run by the root user then the files are created OK. But running as mailman the shared file generation fails because of access permissions and is redone (and fails again) each time rundig is run on behalf of a list by nightly_htdig.
I solved the problem by making /var/www/htdig/common owned by root:mailman and giving the group write privileges over the directory. If you are concerned about the security of this then I think you could revert the privileges over that directory after the db file in common have been created. Personally, I do not bother.
At 16:35 27/02/2003, C. Posey wrote:
At 21:31 26/02/2003, C. Posey wrote:
On the face of it this looks OK. What I am puzzled by is that the symlinks are owned root:mailman. I would have expected them to be owned by mailman:mailman. Ths links are normally automatically created as mail posted to the list is being dealt with by Mailman and I would expect the process concerned to be running under the mailman uid.
What version of MM are you running? If MM 2.0.x is the qrunner cron job running under the mailman user? If MM 2.1 or 2.1.1 how are you starting mailmanctl?
How did you install htdig and can you confirm that the installation's default CONFIG_DIR is /var/www/htdig/conf?
I'm assuming that either your /usr/local/mailman/Mailman/Defaults.py or mm_cfg.py contains the following line:
HTDIG_MAILMAN_LINK = htdig-mailman
Any other ideas??
Your previous post indicated that nightly_htdig was running OK. I take it that the /usr/local/mailman/archives/private/<listname>/htdig/ directories contain the <listname>.conf files that are the subject of the symlinks. Also that, following nightly_htdig they contain files like this:
mailman@mailman2:/mailman/run/archives/private/rbtest/htdig> ls -l total 20 -rw-rw-r-- 1 mailman mailman 2048 Feb 25 17:45 db.docdb -rw-rw-r-- 1 mailman mailman 2048 Feb 25 17:45 db.docs.index -rw-rw-r-- 1 mailman mailman 128 Feb 25 17:45 db.wordlist -rw-rw-r-- 1 mailman mailman 2048 Feb 25 17:45 db.words.db -rw-rw-rw- 1 mailman mailman 3879 Feb 25 17:38 rbtest.conf -rw-rw-r-- 1 mailman mailman 0 Feb 25 17:45 rundig_last_run
You could try the following and see if it has any beneficial effects:
Select one of your lists and run /usr/local/mailman/bin/blowaway_htdig for that list
The search form should now be gone from the list's TOC page
3, Send a message to the list
The search form should now be back on the list's TOC page. The /usr/local/mailman/archives/private/<listname>/htdig/<listname>.conf file should be back and alone in the dirctory. Also the symlink in /usr/local/mailman/archives/htdig/ should be back.
Run nightly_htdig -v <listname> to rebuild the list's search indexes and confirm the script says it is doing so. This should add the extra files to /usr/local/mailman/archives/private/<listname>/htdig/ and update the list's TOC page.
Searching should now work.
If it doesn't then let me know.
I am running Mailman 2.1 - fresh install last week
All qrunner processes are running under UID mailman.
Htdig came preinstalled with my Slackware install. It was initially installed in /opt/var .. but I am old skool and prefer my www in /var/www. I moved reqd htdig to /var/www/htdig.
/var/www/htdig/conf does contain config files, but where do I check to see if CONFIG_DIR is set to that location
How did you install htdig and can you confirm that the installation's default CONFIG_DIR is /var/www/htdig/conf?
Yes ...
USE_HTDIG = 1 # 0 - don't use integrated htdig, 1 - use it HTDIG_ARCHIVE_URL = '/mailman/htdig/' # must end in a slash HTDIG_SEARCH_URL = '/cgi-bin/htsearch' HTDIG_FILES_URL = '/htdig/' HTDIG_CONF_LINK_DIR = os.path.join(VAR_PREFIX, 'archives', 'htdig') HTDIG_MAILMAN_LINK = 'htdig-mailman' HTDIG_RUNDIG_PATH = '/usr/local/bin/rundig'
Yes - my /usr/local/mailman/archives/private/<listname>/htdig/ contains this...
drwxrwsr-x 2 mailman mailman 4096 Feb 25 23:14 . drwxrwsr-x 6 mailman mailman 4096 Feb 20 17:04 .. -rw-rw-r-- 1 root mailman 110592 Feb 25 23:14 db.docdb -rw-rw-r-- 1 root mailman 6144 Feb 25 23:14 db.docs.index -rw-rw-r-- 1 root mailman 151894 Feb 25 23:14 db.wordlist -rw-rw-r-- 1 root mailman 227328 Feb 25 23:14 db.words.db -rw-rw-rw- 1 mailman mailman 3850 Feb 6 07:33 macvoodoolist.conf -rw-rw-r-- 1 root mailman 0 Feb 25 23:15 rundig_last_run
Although I notice - here also root is the owner of some of the files
Tried your recommendation, I can verify that all the files are cleared with blowaway, and the serachform is removed from TOC. After sending a msg to the list, the search form is available again and the respective files have been recreated. Running the nightly cronjob within the mailman crontab - gives me the following errors though:
Subject: cron: /usr/local/bin/python -S /usr/local/mailman/cron/nightly_htdig mailman. Note the last line - it is still pointing to the old /opt/www/htdig location. Can that be changed in config file or is that compiled into htdig. Maybe I just need to recompile htdig to my preferred location.
DB2 problem...: /var/www/htdig/db/word2root.db: Permission denied
DB2 problem...: /var/www/htdig/db/root2word.db: Permission denied
/var/www/htdig/bin/rundig: line 68: 23659 Segmentation fault
$BINDIR/htfuzzy $opts endings
DB2 problem...: /opt/www/htdig/common/synonyms.db: Permission denied
I tried changing the group on the file in /var/www/htdig/db/ to mailman, but that had no effect.
Any attempts to use the search page yields:
ht://Dig error htsearch detected an error. Please report this to the webmaster of this site. The error message is: Unable to read configuration file
Thank you VERY much for all your help. I am reinstalling htdig 3.1.6 as we speak. I'll let you know if that works any better. Or if you have any suggestions - please send.
Ok - I completely rebuilt htdig 3.1.6 and I can see now from the ./configure - that a lot is dependent on the build process. So I rebuilt with the proper values for my htdig install. Ran the blow away again, sent another msg to the list, form appears on TOC, ran the cronjob, and now when I try to search I am getting a new msg:
ht://Dig error
htsearch detected an error. Please report this to the webmaster of this site. The error message is: Unable to read word database file Did you run htmerge?
I wa</spt aware that I was supposed to run htmerge after the cronjob?
At 07:13 26/02/2003, C. Posey wrote:
I'll guess from what you've said that there is a directory called /var/www/htdig/conf which contains a file called htdig.conf as part of your htdig installation. This is the dault directory where htdig expect to find config files and the default htdig.conf file.
If that is so, then you need to create the symlink called htdig-mailman in the /var/www/htdig/conf, pointing at /usr/local/mailman/archives/htdig.
Thanks for your reply:
What you described is exactly what I have...
[/var/www/htdig/conf] root# ls -la
total 24
drwxr-xr-x 2 root root 4096 Feb 25 23:49 .
drwxr-xr-x 6 root root 4096 May 19 2002 ..
lrwxrwxrwx 1 root root 33 Feb 25 23:49 htdig-mailman
-> /usr/local/mailman/archives/htdig
-rw-r--r-- 1 root root 8411 Feb 6 02:36 htdig.conf
-rw-r--r-- 1 root root 3102 Feb 25 23:48 mailman.conf.tgz
and in the dir pointed to by the symlink...
[/usr/local/mailman/archives/htdig/] root# ls -la
total 8
drwxrwsr-x 2 mailman mailman 4096 Feb 26 01:52 .
drwxrwsr-x 5 mailman mailman 4096 Feb 6 01:47 ..
lrwxrwxrwx 1 root mailman 74 Feb 6 07:33
macvoodoolist.conf ->
/usr/local/mailman/archives/private/macvoodoolist/htdig/
macvoodoolist.conf
lrwxrwxrwx 1 root mailman 62 Feb 6 02:24 mailman.conf ->
/usr/local/mailman/archives/private/mailman/htdig/mailman.conf
Any other ideas??
On Wednesday, February 26, 2003, at 04:00 PM, Richard Barrett wrote:
Just to be sure I patched everything correctly - I followed the
instructions in the Mailman FAQ...
http://www.python.org/cgi-bin/faqw-mm.py?req=all
Is this correct?
The patches to handle this are on the Mailman sourceforge patches
pages, with ids:-
[ #661138 ] corrects defects in some HTML templates (only for MM 2.1)
[ #668685 ] corrects error in sys.exit handling by script/driver
(only for MM 2.1)
[ #444879 ] Archive indexer control to improve indexing
[ #444884 ] Integration of Mailman & htdig for archive search
The URLs for the patches are
http://sourceforge.net/tracker/
index.php?func=detail&aid=661138&group_id=103&atid=300103
http://sourceforge.net/tracker/
index.php?func=detail&aid=444879&group_id=103&atid=300103
http://sourceforge.net/tracker/
index.php?func=detail&aid=668685&group_id=103&atid=300103
http://sourceforge.net/tracker/
index.php?func=detail&aid=444884&group_id=103&atid=300103
At 06:36 27/02/2003, C. Posey wrote:
The URLs are correct. The file names shown below are the correct ones for MM 2.1.1:
http://sourceforge.net/tracker/ index.php?func=detail&aid=661138&group_id=103&atid=300103
templates-2.1-0.1.patch.gz
http://sourceforge.net/tracker/ index.php?func=detail&aid=444879&group_id=103&atid=300103
indexing-2.1.1-0.1.patch.gz
http://sourceforge.net/tracker/ index.php?func=detail&aid=668685&group_id=103&atid=300103
driver-2.1-0.1.patch.gz
http://sourceforge.net/tracker/ index.php?func=detail&aid=444884&group_id=103&atid=300103
htdig-2.1.1-0.1.patch.gz
Actually I may have spoken too soon...
I am able to actually perform searches off the TOC now, but I am still getting this error generated from the cronjob...
DB2 problem...: /var/www/htdig/db/word2root.db: Permission denied
DB2 problem...: /var/www/htdig/db/root2word.db: Permission denied
/var/www/htdig/bin/rundig: line 68: 30671 Segmentation fault
$BINDIR/htfuzzy $opts endings
DB2 problem...: /var/www/htdig/common/synonyms.db: Permission denied
On Thursday, February 27, 2003, at 11:33 AM, C. Posey wrote:
Thx for all your help Richard!
The recompile of htdig did the trick....
This problem is discussed to some extent in the INSTALL.htdig-mm under the heading "htdig Permissions Considerations".
The problem is that the rundig process ends up trying to generate some shareable (across the entire intallation) db files in htdig's COMMON_DIR. If rundig is first run by the root user then the files are created OK. But running as mailman the shared file generation fails because of access permissions and is redone (and fails again) each time rundig is run on behalf of a list by nightly_htdig.
I solved the problem by making /var/www/htdig/common owned by root:mailman and giving the group write privileges over the directory. If you are concerned about the security of this then I think you could revert the privileges over that directory after the db file in common have been created. Personally, I do not bother.
At 16:35 27/02/2003, C. Posey wrote:
At 21:31 26/02/2003, C. Posey wrote:
On the face of it this looks OK. What I am puzzled by is that the symlinks are owned root:mailman. I would have expected them to be owned by mailman:mailman. Ths links are normally automatically created as mail posted to the list is being dealt with by Mailman and I would expect the process concerned to be running under the mailman uid.
What version of MM are you running? If MM 2.0.x is the qrunner cron job running under the mailman user? If MM 2.1 or 2.1.1 how are you starting mailmanctl?
How did you install htdig and can you confirm that the installation's default CONFIG_DIR is /var/www/htdig/conf?
I'm assuming that either your /usr/local/mailman/Mailman/Defaults.py or mm_cfg.py contains the following line:
HTDIG_MAILMAN_LINK = htdig-mailman
Any other ideas??
Your previous post indicated that nightly_htdig was running OK. I take it that the /usr/local/mailman/archives/private/<listname>/htdig/ directories contain the <listname>.conf files that are the subject of the symlinks. Also that, following nightly_htdig they contain files like this:
mailman@mailman2:/mailman/run/archives/private/rbtest/htdig> ls -l total 20 -rw-rw-r-- 1 mailman mailman 2048 Feb 25 17:45 db.docdb -rw-rw-r-- 1 mailman mailman 2048 Feb 25 17:45 db.docs.index -rw-rw-r-- 1 mailman mailman 128 Feb 25 17:45 db.wordlist -rw-rw-r-- 1 mailman mailman 2048 Feb 25 17:45 db.words.db -rw-rw-rw- 1 mailman mailman 3879 Feb 25 17:38 rbtest.conf -rw-rw-r-- 1 mailman mailman 0 Feb 25 17:45 rundig_last_run
You could try the following and see if it has any beneficial effects:
Select one of your lists and run /usr/local/mailman/bin/blowaway_htdig for that list
The search form should now be gone from the list's TOC page
3, Send a message to the list
The search form should now be back on the list's TOC page. The /usr/local/mailman/archives/private/<listname>/htdig/<listname>.conf file should be back and alone in the dirctory. Also the symlink in /usr/local/mailman/archives/htdig/ should be back.
Run nightly_htdig -v <listname> to rebuild the list's search indexes and confirm the script says it is doing so. This should add the extra files to /usr/local/mailman/archives/private/<listname>/htdig/ and update the list's TOC page.
Searching should now work.
If it doesn't then let me know.
I am running Mailman 2.1 - fresh install last week
All qrunner processes are running under UID mailman.
Htdig came preinstalled with my Slackware install. It was initially installed in /opt/var .. but I am old skool and prefer my www in /var/www. I moved reqd htdig to /var/www/htdig.
/var/www/htdig/conf does contain config files, but where do I check to see if CONFIG_DIR is set to that location
How did you install htdig and can you confirm that the installation's default CONFIG_DIR is /var/www/htdig/conf?
Yes ...
USE_HTDIG = 1 # 0 - don't use integrated htdig, 1 - use it HTDIG_ARCHIVE_URL = '/mailman/htdig/' # must end in a slash HTDIG_SEARCH_URL = '/cgi-bin/htsearch' HTDIG_FILES_URL = '/htdig/' HTDIG_CONF_LINK_DIR = os.path.join(VAR_PREFIX, 'archives', 'htdig') HTDIG_MAILMAN_LINK = 'htdig-mailman' HTDIG_RUNDIG_PATH = '/usr/local/bin/rundig'
Yes - my /usr/local/mailman/archives/private/<listname>/htdig/ contains this...
drwxrwsr-x 2 mailman mailman 4096 Feb 25 23:14 . drwxrwsr-x 6 mailman mailman 4096 Feb 20 17:04 .. -rw-rw-r-- 1 root mailman 110592 Feb 25 23:14 db.docdb -rw-rw-r-- 1 root mailman 6144 Feb 25 23:14 db.docs.index -rw-rw-r-- 1 root mailman 151894 Feb 25 23:14 db.wordlist -rw-rw-r-- 1 root mailman 227328 Feb 25 23:14 db.words.db -rw-rw-rw- 1 mailman mailman 3850 Feb 6 07:33 macvoodoolist.conf -rw-rw-r-- 1 root mailman 0 Feb 25 23:15 rundig_last_run
Although I notice - here also root is the owner of some of the files
Tried your recommendation, I can verify that all the files are cleared with blowaway, and the serachform is removed from TOC. After sending a msg to the list, the search form is available again and the respective files have been recreated. Running the nightly cronjob within the mailman crontab - gives me the following errors though:
Subject: cron: /usr/local/bin/python -S /usr/local/mailman/cron/nightly_htdig mailman. Note the last line - it is still pointing to the old /opt/www/htdig location. Can that be changed in config file or is that compiled into htdig. Maybe I just need to recompile htdig to my preferred location.
DB2 problem...: /var/www/htdig/db/word2root.db: Permission denied
DB2 problem...: /var/www/htdig/db/root2word.db: Permission denied
/var/www/htdig/bin/rundig: line 68: 23659 Segmentation fault
$BINDIR/htfuzzy $opts endings
DB2 problem...: /opt/www/htdig/common/synonyms.db: Permission denied
I tried changing the group on the file in /var/www/htdig/db/ to mailman, but that had no effect.
Any attempts to use the search page yields:
ht://Dig error htsearch detected an error. Please report this to the webmaster of this site. The error message is: Unable to read configuration file
Thank you VERY much for all your help. I am reinstalling htdig 3.1.6 as we speak. I'll let you know if that works any better. Or if you have any suggestions - please send.
Ok - I completely rebuilt htdig 3.1.6 and I can see now from the ./configure - that a lot is dependent on the build process. So I rebuilt with the proper values for my htdig install. Ran the blow away again, sent another msg to the list, form appears on TOC, ran the cronjob, and now when I try to search I am getting a new msg:
ht://Dig error
htsearch detected an error. Please report this to the webmaster of this site. The error message is: Unable to read word database file Did you run htmerge?
I wa</spt aware that I was supposed to run htmerge after the cronjob?
participants (8)
-
C. Posey -
C. Posey -
C. Posey -
C. Posey -
C. Posey -
C. Posey -
Richard Barrett -
Richard Barrett