[ mailman-Patches-444884 ] Integration of Mailman & htdig for archi

SourceForge.net noreply at sourceforge.net
Tue Sep 30 16:14:08 EDT 2003


Patches item #444884, was opened at 2001-07-26 18:27
Message generated for change (Comment added) made by ppsys
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=300103&aid=444884&group_id=103

Category: Unofficial 2.0 patch
Group: Mailman 2.2 / 3.0
Status: Open
Resolution: None
Priority: 3
Submitted By: Richard Barrett (ppsys)
Assigned to: Barry A. Warsaw (bwarsaw)
Summary: Integration of Mailman & htdig for archi

Initial Comment:
This patch is applicable to Mailman 2.0.6 release that 

has had search enhancement patch 444879 patch 

installed - if your Defaults.py has the 

ARCHIVE_INDEXING_ENABLE and ARCHIVE_INDEXING_DISABLE 

in it then you've got that patch.



It replaces earlier patches 401670 and 402423 and is 

mainly to correct some problems arising from fixes 

introduced into Mailman by bug fix releases since the 

402423 patch. 



This patch integrates htdig with Mailman and provides:



1. per list search facility with a search form on the 

list's TOC page.



2. maintenance of privacy of private archives which 

requires the user to establish their credentials via 

the normal private archive access before any access 

via htdig is allowed.

   

3. a common base URL for both public and private 

archive access via htsearch results so that htdig 

indices are unaffected by changingan archive from 

private to public and vice versa. All access to 

archives via htdig is controlled by a new wrapped cgi-

bin script called htdig.py.

   

4. a new cron activated script and extra crontab entry 

which runs htdig regularly to maintain the per list 

search indices.

   

5. automatic creation, deletion and maintenance of 

htdig configuration files and such. Beyond installing 

htdig and telling Mailman where it is via mm_cfg you 

do not have to do any other setup. Well not quite you 

do have to set up a single per installation symlink to 

allow htdig to find the automatically generated per 

list htdig configuration files.



You probably want to run this patch as follows:



cd <mailman 2.0.6 untarred and unzipped directory>

patch -p1 < <this patch file>  

----------------------------------------------------------------------

>Comment By: Richard Barrett (ppsys)
Date: 2003-09-30 20:14

Message:
Logged In: YES 
user_id=75166

htdig-2.1.3-0.1.patch is a MM 2.1.3 compatible version of

the patch

----------------------------------------------------------------------

Comment By: Richard Barrett (ppsys)
Date: 2003-06-06 17:47

Message:
Logged In: YES 
user_id=75166

last comment should have read: 



htdig-2.1.2-0.4.patch.gz corrects an error in 2 scripts, 

mmsearch.py and remote_mmsearch, which caused an 

exception if list archives were being accessed via HTTPS and 

a search was performed.

----------------------------------------------------------------------

Comment By: Richard Barrett (ppsys)
Date: 2003-06-06 17:45

Message:
Logged In: YES 
user_id=75166

htdig-2.1.2-0.3.patch.gz corrects an error in 2 scripts, 

mmsearch.py and remote_mmsearch, which caused an 

exception if list archives were being accessed via HTTPS and 

a search was performed.

----------------------------------------------------------------------

Comment By: Richard Barrett (ppsys)
Date: 2003-05-01 19:00

Message:
Logged In: YES 
user_id=75166

htdig-2.1.2-0.3.patch.gz adds some minor performance 

improvement in template handling in MM 2.1.2



You should consider also applying this bug-fis patch:



[ 730769 ] template access hierarchy is broken



http://sourceforge.net/tracker/index.php?

func=detail&aid=730769&group_id=103&atid=100103

----------------------------------------------------------------------

Comment By: Richard Barrett (ppsys)
Date: 2003-04-28 15:44

Message:
Logged In: YES 
user_id=75166

htdig-2.1.2-0.2.patch.gz corrects error in file uploaded as 

htdig-2.1.2-0.1.patch.gz. Sorry for any inconvenience.

----------------------------------------------------------------------

Comment By: Richard Barrett (ppsys)
Date: 2003-04-28 14:46

Message:
Logged In: YES 
user_id=75166

htdig-2.1.2-0.1.patch.gz is a revised version for MM 2.1.2 

compatibility.



It also incoporates a previosuly unpublished change to 

overcome a potential problem with htdig excluced urls - see 

the INSTALL.htdig-mm file for more information

----------------------------------------------------------------------

Comment By: Richard Barrett (ppsys)
Date: 2003-03-21 18:29

Message:
Logged In: YES 
user_id=75166

htdig-2.1.1-0.4.patch.gz fixes a problem with mmsearch 

handling multi-page search results from htsearch.

----------------------------------------------------------------------

Comment By: Richard Barrett (ppsys)
Date: 2003-03-21 17:54

Message:
Logged In: YES 
user_id=75166

htdig-2.1.1-0.3.patch.gz fixes a fault when mmsearch.py is 

rasing an excpetion because it has had a problem running 

htsearch

----------------------------------------------------------------------

Comment By: Richard Barrett (ppsys)
Date: 2003-03-20 14:10

Message:
Logged In: YES 
user_id=75166

htdig-2.1.1-0.2.patch.gz close a security exploit which allows 

leakage of information held in htdig's per-list search indexes 

to users not authorized to view private list archives.



Read file INSTALL.htdig-mm installed by this patch for details 

and instructions for upgrading MM installations using earlier 

versions of this patch





----------------------------------------------------------------------

Comment By: Richard Barrett (ppsys)
Date: 2003-02-10 15:50

Message:
Logged In: YES 
user_id=75166

htdig-2.1.1-0.1.patch.gz introduces no functional change but 

applies without offset warnings to MM 2.1.1

----------------------------------------------------------------------

Comment By: Richard Barrett (ppsys)
Date: 2003-02-05 18:12

Message:
Logged In: YES 
user_id=75166

It seems it is possible, if this patch is installed, for a list's 

htdig conf file and the list specific htdig index db files to be 

read directly through the web interface for list archives. 



Even if this patch isn't installed it seems a list's pipermail.pck 

file can also be read directly through the web interface for list 

archives.



This seems to be true for accesses via /pipermail for public 

lists and via /mailman/private for private lists.



The problem does not occur for htdig search results 

accessed via /mailman/htdig as the htdig.py script is more 

protective than private.py



Broadly speaking the data affected is availble to a user in 

normal operation which is why I do not consider the issue to 

be a security breach as such.



Adding the following RewriteRule to Apache's httpd.conf 

prevents the situation, assuming you got the RewriteEngine 

On:



RewriteRule ^(/pipermail/.*)/(pipermail.pck|htdig/[^/]*)$ 

$1/index.html [F]



RewriteRule ^(/mailman/private/.*)/(pipermail.pck|htdig/[^/]*)$ 

$1/index.htm

l [F]



You could, of course, substitute an R flag for the F flag on the 

RewriteRules and be more hacker friendly.

----------------------------------------------------------------------

Comment By: Richard Barrett (ppsys)
Date: 2003-01-22 14:20

Message:
Logged In: YES 
user_id=75166

htdig-2.1-0.3.patch corrects yet another bug in htdig.py. Hope 

that all of them!



Stops use of obsolete config variable DEFAULT_HOST in 

several files.

----------------------------------------------------------------------

Comment By: Richard Barrett (ppsys)
Date: 2003-01-15 20:18

Message:
Logged In: YES 
user_id=75166

htdig-2.1-0.2.patch corrects a bug in htdig.py and deals with 

an adverse interaction between htdig.py and a bug in 

$prefix/scripts/driver (see #668685 for a patch to fix this).



It also improves the content type and security handling by 

htdig.py for MM 2.1 version of patch

----------------------------------------------------------------------

Comment By: Richard Barrett (ppsys)
Date: 2003-01-15 20:14

Message:
Logged In: YES 
user_id=75166

Uploaded wrong file mailer-2.0.13-0.4.patch on last attempt.



Should have been htdig-2.0.13-0.4.patch which improves the 

content type and security handling by htdig.py for MM 2.0.13 

version of patch.



Please ignore mailer-2.0.13-0.4.patch file

----------------------------------------------------------------------

Comment By: Richard Barrett (ppsys)
Date: 2003-01-15 20:09

Message:
Logged In: YES 
user_id=75166

mailer-2.0.13-0.4.patch improves the content type and 

security handling by htdig.py for MM 2.0.13 version of patch

----------------------------------------------------------------------

Comment By: Richard Barrett (ppsys)
Date: 2003-01-02 16:07

Message:
Logged In: YES 
user_id=75166

htdig-2.1-0.1.patch is a revised version of the patch that is 

compatible with MM 2.1

----------------------------------------------------------------------

Comment By: Richard Barrett (ppsys)
Date: 2002-12-11 11:48

Message:
Logged In: YES 
user_id=75166

htdig-2.1b6-0.1.patch is a revised version of the patch that is 

compatible with MM 2.1b6

----------------------------------------------------------------------

Comment By: Richard Barrett (ppsys)
Date: 2002-12-04 10:53

Message:
Logged In: YES 
user_id=75166

htdig-2.0.13-0.3.patch corrects a minor typo in text appearing 

in the list TOC after the patch is applied.

----------------------------------------------------------------------

Comment By: Richard Barrett (ppsys)
Date: 2002-11-27 10:24

Message:
Logged In: YES 
user_id=75166

htdig-2.1b5-0.1.patch is a revised version of the patch that is 

compatible with MM 2.1b5

----------------------------------------------------------------------

Comment By: Richard Barrett (ppsys)
Date: 2002-10-30 11:43

Message:
Logged In: YES 
user_id=75166

htdig-2.1b4-0.1.patch is a revised version of the patch that is 

compatible with MM 2.1b4

----------------------------------------------------------------------

Comment By: Richard Barrett (ppsys)
Date: 2002-10-14 11:50

Message:
Logged In: YES 
user_id=75166

htdig-2.1b3-0.3.patch removes use of the file() function, used 

instead of the open() function, in three cron scripts added by 

the patch. Use of the file() function created an unnecessary 

dependency on Python 2.2

----------------------------------------------------------------------

Comment By: Colin Mackinlay (cmackinlay)
Date: 2002-10-12 16:51

Message:
Logged In: YES 
user_id=624179

Got a workaround!



The line referred to in the traceback:

 file(rundig_run_file, 'w').close()

is used to create a 'rundig_last_run' file of lenght 0 bytes

Creating this manually (or copying it) means the line isn't 

called and everything seems to work.



Either file() is not a valid function call or my python is broken - 

I'm not literate enough in python to know the answer though! 

----------------------------------------------------------------------

Comment By: Colin Mackinlay (cmackinlay)
Date: 2002-10-06 14:18

Message:
Logged In: YES 
user_id=624179

Just rebuilt MM as 2.1b3 with htdig.

Upgraded lists which had htdig before work fine

New lists give the obvious error:

  Unable to read word database file

  Did you run htmerge?

Running the cronjob doesn't fix as it used to, message is:

  Output from command /usr/bin/python -

S /usr/local/mailman/cron/nightly_htdig ..



Traceback (most recent call last):

  File "/usr/local/mailman/cron/nightly_htdig", line 153, in ?

    main()

  File "/usr/local/mailman/cron/nightly_htdig", line 118, in main

    file(rundig_run_file, 'w').close()

NameError: global name 'file' is not defined



The archive/htdig folder only contains the xx.conf file, but no 

db.xx files



If I copy in db.xx files from another list then the problem goes 

away (except I've now got an invalid set of references!)



Is this my elementary error or is it more sinister?!



----------------------------------------------------------------------

Comment By: Richard Barrett (ppsys)
Date: 2002-08-15 11:02

Message:
Logged In: YES 
user_id=75166

htdig-2.1b3-0.2.patch corrects a dumb syntax error in htdig-

2.1b3-0.1.patch which will typically show up as logged errors 

in the operation of the ArchRunner qrunner at line 721 of 

HyperArch.py

----------------------------------------------------------------------

Comment By: Richard Barrett (ppsys)
Date: 2002-08-14 16:51

Message:
Logged In: YES 
user_id=75166

htdig-2.1b3-0.1.patch is a revised version of the patch that is 

compatible with MM 2.1b3

----------------------------------------------------------------------

Comment By: Barry A. Warsaw (bwarsaw)
Date: 2002-08-08 16:33

Message:
Logged In: YES 
user_id=12800

I've sent Richard some comments off-line about this patch.



Meta comments: the 2.0.x patches can't be officially

supported, but I'm going to create an unofficial patches

page off the wiki for where the 2.0 patches can be migrated.



I think this patch set is too big for MM2.1, but if it's

cleaned up as per my private message, let's re-evaluate it

for MM2.2 (or 3.0).

----------------------------------------------------------------------

Comment By: Richard Barrett (ppsys)
Date: 2002-08-05 10:11

Message:
Logged In: YES 
user_id=75166

htdig-2.0.13-0.2.patch just adds a GPL notice to the patch



----------------------------------------------------------------------

Comment By: Richard Barrett (ppsys)
Date: 2002-08-01 16:35

Message:
Logged In: YES 
user_id=75166

htdig-2.1b2-0.1.patch is a revised version of the patch 

that is compatible with MM 2.1b2

----------------------------------------------------------------------

Comment By: Richard Barrett (ppsys)
Date: 2002-07-30 11:25

Message:
Logged In: YES 
user_id=75166

htdig-2.0.13-0.1.patch is purely cosmetic to get no mumble 

application to MM 2.0.13

----------------------------------------------------------------------

Comment By: Richard Barrett (ppsys)
Date: 2002-07-25 15:07

Message:
Logged In: YES 
user_id=75166

Do not use htdig-2.0.12-0.1.patch there is an error in it.

Use htdig-2.0.12-0.2.patch instead

----------------------------------------------------------------------

Comment By: Richard Barrett (ppsys)
Date: 2002-07-25 14:10

Message:
Logged In: YES 
user_id=75166

htdig-2.0.12-0.1.patch is a revised version of the patch that 

applies without complaint to MM 2.0.12.



It also add a facility for adding site wide htdig configuration 

attributes to all list specific htdig configuration files. 





----------------------------------------------------------------------

Comment By: Richard Barrett (ppsys)
Date: 2002-05-23 09:59

Message:
Logged In: YES 
user_id=75166

htdig-2.0.11-0.1.patch is a revised version of the patch that 

is compatible with MM 2.0.11



This version removes an incompatibility with Python 2.2 

which caused warning messages to be generated when any 

of the family cron/nightly_htdig scripts were run.



Some guidance on file access permissions for some htdig 

database files needed by rundig have been added to 

installation notes.

----------------------------------------------------------------------

Comment By: Richard Barrett (ppsys)
Date: 2002-04-19 10:59

Message:
Logged In: YES 
user_id=75166

htdig-2.0.10-0.1.patch is a revised version of the patch 

that is compatible with MM 2.0.10

----------------------------------------------------------------------

Comment By: Richard Barrett (ppsys)
Date: 2002-04-08 17:46

Message:
Logged In: YES 
user_id=75166

htdig-2.0.9-0.1.patch is a revised version of the patch 

that is compatible with MM 2.0.9

----------------------------------------------------------------------

Comment By: Richard Barrett (ppsys)
Date: 2002-03-06 16:22

Message:
Logged In: YES 
user_id=75166

htdig-2.1cvs-20020306.patch is a revised version of the patch that is compatible with the code published in 

mailman CVS on sourceforge as 12:30 GMT 6 Mar 2002



Known deficiency is that the non-English versions of files under $build/templates still contain text in English 

and need translations I cannot do. Also the necessary pygettext activity and subsequent translations in 

files under $build/messages remain to be done.

----------------------------------------------------------------------

Comment By: Richard Barrett (ppsys)
Date: 2001-12-17 16:56

Message:
Logged In: YES 
user_id=75166

htdig-2.1cvs-20011217.patch is a revised version of the 

patch that is compatible with the code published in mailman 

CVS on sourceforge as 11:50 GMT 17 Dec 2001



The only known deficiency is that the non-English versions 

of files under $build/templates still contain text in 

English and need translations I cannot do. Also the 

necessary pygettext activity and subsequent translations in 

files under $build/messages remain to be done.

----------------------------------------------------------------------

Comment By: Richard Barrett (ppsys)
Date: 2001-12-13 16:58

Message:
Logged In: YES 
user_id=75166

htdig-2.1a3-0.1.patch is a revised version of the patch that is compatible with the code published in 

mailman-2.1a3.tgz on sourceforge.



The only known deficiency is that the non-English versions of files under $build/templates still contain text 

in English and need translations I cannot do. Also the necessary pygettext activity and subsequent 

translations in files under $build/messages remain to be done. 

----------------------------------------------------------------------

Comment By: Richard Barrett (ppsys)
Date: 2001-11-28 17:33

Message:
Logged In: YES 
user_id=75166

The htdig-2.0.8-0.1.patch version of the patch resolves a problem that can arise with htdig indexing if the 

web_page_url for a list uses other than the http addressing (some folks want to use https). While specified 

as for MM 2.0.8 the revised patch should work OK with 2.0.7, 2.0.6 and probably back as far as 2.0.3. If 

you do not have the requirement for using other than http addressing in you lists web_page_urls it probably 

isn't worth the trouble of upgrading to this patch level.

----------------------------------------------------------------------

Comment By: Richard Barrett (ppsys)
Date: 2001-11-28 11:08

Message:
Logged In: YES 
user_id=75166

This patch should also apply without problems to MM 2.0.8

----------------------------------------------------------------------

Comment By: Richard Barrett (ppsys)
Date: 2001-11-27 12:00

Message:
Logged In: YES 
user_id=75166

This patch should also apply without problems to Mm 2.0.7

----------------------------------------------------------------------

Comment By: Richard Barrett (ppsys)
Date: 2001-11-09 11:54

Message:
Logged In: YES 
user_id=75166

The htdig-2.0.6-03.patch version of the patch makes some 

previously hard-coded things configurable and enhances the 

capability to run the htdig searches and indexing on a 

different machine to the one delivering Mailman and 

Mailman's web UI.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=300103&aid=444884&group_id=103



More information about the Mailman-coders mailing list