Mailman 3 December 2002 - Mailman-Developers

unicode / archive problem followup (scrubber.py too)
by Ron Brogden 04 Dec '02

04 Dec '02

Howdy. Since no one replied with any suggestions for me I decided to take a chance with a crude little hack to deal with the archiver getting confused when the unicode type returned was "None". This also brought up a bug with the attachment handling code in the archiver though which again, I have done a crude hack to but which I want to make sure I haven't done a big no no. First off, the specific changes: In HyperArchive.py the following lines were munged to always return the following charset if none could be discerned: 450: return unicode(subj, "iso-8859-1") 1010: return unicode(result, "iso-8859-1") This got me past the initial error but this led to a new error to with dates and attachement folders. The sanity checking is not too bright for the folder name selection and so what happens is that if the date cannot be culled from the message due to a logic problem (it falls through in a funny way), the net result is that the archiver dies instead of either just picking a date or tossing the attachment. Since I am not totally clear on what all the date efffects, I chickened out and just assumed that there would always be a date in the received header which probably isn't wise but is better than it is now. In Scrubber.py around line 80: def calculate_attachments_dir(mlist, msg, msgdata): # Calculate the directory that attachments for this message will go # under. To avoid inode limitations, the scheme will be: # archives/private/<listname>/attachments/YYYYMMDD/<msgid-hash>/<files> # Start by calculating the date-based and msgid-hash components. msgdate = msg.get('Date') if msgdate is None: now = time.gmtime(msgdata.get('received_time', time.time())) else: now = parsedate(msgdate) This a problem since it appears that parsedate() is not guaranteed to return something useful and so again, the archiver was dying. To get around this I had to add the following catchall afterwards: if now is None: now = time.gmtime(msgdata.get('received_time', time.time())) So two questions. First, is the above a reasonable thing to do (i.e. am I introducing anything nasty by doing this)? Secondly, I don't want attachments archived at all - how do how do you disable this behaviour? Thanks for any suggestions or feedback. Cheers, Ron

1 0

[ mailman-Patches-644810 ] Sendmail mailer in Python
by noreply＠sourceforge.net 03 Dec '02

03 Dec '02

Patches item #644810, was opened at 2002-11-27 16:27 You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=300103&aid=644810&group_i… Category: mail delivery Group: Mailman 2.1 Status: Open Resolution: None Priority: 5 Submitted By: Richard Barrett (ppsys) Assigned to: Nobody/Anonymous (nobody) Summary: Sendmail mailer in Python Initial Comment: This patch provide a Python implementation (with C wrapper) of a Sendmail mailer, if Sendmail is your MTA of choice, which removes the necessity of maintining an aliases database on your Mailman server. All subject to a bunch of conditions most prominent of which is that the server has to be pretty much dedicated to MM. This patch draws on an original contribution by David Champion <dgc(a)uchicago.edu> which is included in the contrib directory of the Mailman 2.1b distribution as mm-handler. See the README.SENDMAIL.mailer installed by this patch for differences between this and David's Perl original and more information of what you are getting into if you use this patch. Versions of the patch for MM 2.0.13 and MM 2.1b5 are available. This patch requires patch 644797 to be installed before it. In the MM build directory say: patch -p1 < patch-filename ---------------------------------------------------------------------- >Comment By: Richard Barrett (ppsys) Date: 2002-12-03 17:37 Message: Logged In: YES user_id=75166 Fix for permissions related exceptions being generated by the subprocess spawned by mm_mailer script ---------------------------------------------------------------------- Comment By: Richard Barrett (ppsys) Date: 2002-12-03 17:36 Message: Logged In: YES user_id=75166 Fix for permissions related exceptions being generated by the subprocess spawned by mm_mailer script. ---------------------------------------------------------------------- Comment By: Richard Barrett (ppsys) Date: 2002-11-27 16:29 Message: Logged In: YES user_id=75166 version for MM 2.1b5 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=300103&aid=644810&group_i…

1 0

[ mailman-Patches-644810 ] Sendmail mailer in Python
by noreply＠sourceforge.net 03 Dec '02

03 Dec '02

Patches item #644810, was opened at 2002-11-27 16:27 You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=300103&aid=644810&group_i… Category: mail delivery Group: Mailman 2.1 Status: Open Resolution: None Priority: 5 Submitted By: Richard Barrett (ppsys) Assigned to: Nobody/Anonymous (nobody) Summary: Sendmail mailer in Python Initial Comment: This patch provide a Python implementation (with C wrapper) of a Sendmail mailer, if Sendmail is your MTA of choice, which removes the necessity of maintining an aliases database on your Mailman server. All subject to a bunch of conditions most prominent of which is that the server has to be pretty much dedicated to MM. This patch draws on an original contribution by David Champion <dgc(a)uchicago.edu> which is included in the contrib directory of the Mailman 2.1b distribution as mm-handler. See the README.SENDMAIL.mailer installed by this patch for differences between this and David's Perl original and more information of what you are getting into if you use this patch. Versions of the patch for MM 2.0.13 and MM 2.1b5 are available. This patch requires patch 644797 to be installed before it. In the MM build directory say: patch -p1 < patch-filename ---------------------------------------------------------------------- >Comment By: Richard Barrett (ppsys) Date: 2002-12-03 17:36 Message: Logged In: YES user_id=75166 Fix for permissions related exceptions being generated by the subprocess spawned by mm_mailer script. ---------------------------------------------------------------------- Comment By: Richard Barrett (ppsys) Date: 2002-11-27 16:29 Message: Logged In: YES user_id=75166 version for MM 2.1b5 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=300103&aid=644810&group_i…

1 0

[ mailman-Feature Requests-581103 ] do not moderate mails from nntp-groups
by noreply＠sourceforge.net 03 Dec '02

03 Dec '02

Feature Requests item #581103, was opened at 2002-07-13 20:35 You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=350103&aid=581103&group_i… Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Markus Mandalka (mandalka) Assigned to: Nobody/Anonymous (nobody) Summary: do not moderate mails from nntp-groups Initial Comment: i wish to have an setting for the nntp-gateway that allows posting the nntp-articles without moderating on lists, that need confirmation for member-postings. The moderation should only be for the emails sendet to the list, not for nntp-postings. ---------------------------------------------------------------------- Comment By: Philip Lowman (plowman) Date: 2002-12-03 07:43 Message: Logged In: YES user_id=78148 I've also been looking for this feature to be added, i'm kinda upset it hasn't been assigned to anyone :( This is a good idea! I don't care who posts on the NNTP side of things, reflecting that to the list is fine. I am worried about spammers on the email side of things and there is no way to add everyone who would post to the newsgroup to the explicit allow list. This is not good :)... luckily I haven't been spammed... yet ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=350103&aid=581103&group_i…

1 0

MHonArc integration
by Prakash Subramanian 03 Dec '02

03 Dec '02

I am trying to use a simple method to include MHonArc archiver enabled with monthly archive functionality. I found your script that does, but I am not sure how to use it. I tried replacing ~mailman/Mailman/Archiver/HyperArch.py with your MHonArc.py but it gives the following error message. ~mailman/bin/arch mylist Traceback (most recent call last): File "/usr/local/mailman/bin/arch", line 43, in ? from Mailman.Archiver.HyperArch import HyperArchive, Article ImportError: cannot import name Article The method Article is been used by various script and I am not able to get rid of this . Please advice on how to use your script. Another question, why Many thanks in advance. Prakash

2 1

"Site list is missing"
by NOW Website Coordinator 03 Dec '02

03 Dec '02

I can't find this answer in the archives, or in the scripts. The first time that 2.1 (right now I'm using 2.1b5) would send out password reminders, it failed with this message: Your "cron" job on now /usr/local/bin/python -S /usr/local/mailman/cron/mailpasswds produced the following output: Site list is missing: mailman When I look at the mailpasswds script, I find the section it came from: # This is the list that all the reminders will look like they come from, # but with the host name coerced to the virtual host we're processing. try: sitelist = MailList.MailList(mm_cfg.MAILMAN_SITE_LIST, lock=0) except Errors.MMUnknownListError: # Do it this way for I18n's _() sitelistname = mm_cfg.MAILMAN_SITE_LIST print >> sys.stderr, _('Site list is missing: %(sitelistname)s') syslog('error', 'Site list is missing: %s', mm_cfg.MAILMAN_SITE_LIST) sys.exit(1) but I'm not sure what exactly the problem is. Is there something in mm_cfg.py I should set? Thanks in advance. By the way, this worked for 2.0. Also: 2.1b5 fixed my earlier problem of not being able to see the members! Thanks. Sarah Stapleton-Gray -------------------------------------------- Web Site Coordinator National Organization for Women http://www.now.org

2 1

[ mailman-Patches-647083 ] compress dies w/ MemoryError
by noreply＠sourceforge.net 03 Dec '02

03 Dec '02

Patches item #647083, was opened at 2002-12-02 10:38 You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=300103&aid=647083&group_i… Category: command line scripts Group: None Status: Open Resolution: None Priority: 5 Submitted By: John-Mark Gurney (jmg) Assigned to: Nobody/Anonymous (nobody) Summary: compress dies w/ MemoryError Initial Comment: If you have a large mailing list, it is very easy to quickly exhast memory in many conditions. nightly_gzip script tries to load the entire file into memory before compressing it, not only is this terribly ineffecient, it also means you are limited to file sizes that can fit in memory. Some machines also further restrict how much memory each process can access at a time. If that limit is 512megs, and you have a 400+meg file, compress will die. Solution: read/write the file in blocks. ---------------------------------------------------------------------- >Comment By: John-Mark Gurney (jmg) Date: 2002-12-02 16:22 Message: Logged In: YES user_id=27410 Well, from the looks of it, it is best to use the gzip module if we want to be able to use gunzip (cmd line) to uncompress the file. zlib only provides for compression objects, and doesn't say if the output is compatible for direct input to the gunzip program. I also went with the easiest modification, least source modifications. Now that I'm thinking about it, it might of been implemented as a call to writelines(xreadlines()) w/o having to use my own while loop resulting in even less modification of the code. The only problem is that this may produce very long lines for binary code. Also, this requires more scanning for the new lines, while the block while does not have to scan the input. To bad write doesn't support file objects to copy in a block manner. so the code could simply be: outfp.write(infp) P.S. I forgot to mention in the body that this work is supported by Vernier Networks, Inc. ---------------------------------------------------------------------- Comment By: Bryan Fullerton (fehwalker) Date: 2002-12-02 10:53 Message: Logged In: YES user_id=660772 would it make more sense to do this with zlib routines on a stream than with gzip on blocks? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=300103&aid=647083&group_i…

1 0

[ mailman-Bugs-647102 ] MemoryError when generating indexes
by noreply＠sourceforge.net 02 Dec '02

02 Dec '02

Bugs item #647102, was opened at 2002-12-02 11:08 You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=100103&aid=647102&group_i… Category: Pipermail Group: 2.0.x Status: Open Resolution: None Priority: 5 Submitted By: John-Mark Gurney (jmg) Assigned to: Nobody/Anonymous (nobody) Summary: MemoryError when generating indexes Initial Comment: Vernier Networks has a relatively small/medium mailing list archive (only ~238megs) that can not be regenerated in 512megs of memory. (Once the machine needs a reboot, this will increase to 1gig.) It dies with a MemoryError supplied below: Updating index files for archive [2002-May] Traceback (most recent call last): File "/home/mailman/bin/arch", line 129, in ? main() File "/home/mailman/bin/arch", line 119, in main archiver.close() File "/home/mailman/Mailman/Archiver/pipermail.py", line 304, in close self.update_dirty_archives() File "/home/mailman/Mailman/Archiver/HyperArch.py", line 934, in update_dirty_archives self.__super_update_dirty_archives() File "/home/mailman/Mailman/Archiver/pipermail.py", line 513, in update_dirty_archives self.update_archive(i) File "/home/mailman/Mailman/Archiver/HyperArch.py", line 937, in update_archive self.__super_update_archive(archive) File "/home/mailman/Mailman/Archiver/pipermail.py", line 417, in update_archive self.__set_parameters(archive) File "/home/mailman/Mailman/Archiver/pipermail.py", line 332, in __set_parameters firstdate = self.database.firstdate(archive) File "/home/mailman/Mailman/Archiver/HyperDatabase.py", line 207, in firstdate self.__openIndices(archive) File "/home/mailman/Mailman/Archiver/HyperDatabase.py", line 238, in __openIndices self.__closeIndices() File "/home/mailman/Mailman/Archiver/HyperDatabase.py", line 259, in __closeIndices index.close() File "/home/mailman/Mailman/Archiver/HyperDatabase.py", line 186, in close fp.write(marshal.dumps(self.dict)) MemoryError and I also have an IOError on a lock file. I will be looking at it, but since I do not know Pipermail, I won't progress very far. ---------------------------------------------------------------------- >Comment By: John-Mark Gurney (jmg) Date: 2002-12-02 15:56 Message: Logged In: YES user_id=27410 Found that in pipermail.T.processUnixMailbox, it will read in ALL articles w/ their bodies attached. I have fixed pipermail.Article to use an instance of a new class I wrote for body. The class will emulate a simple sequence and an interator object and just store the offsets to the lines, and reread the lines as necessary. I'm waiting the nightly run to see how things will go. The patches currently work for small archives. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=100103&aid=647102&group_i…

1 0

[ mailman-Bugs-647102 ] MemoryError when generating indexes
by noreply＠sourceforge.net 02 Dec '02

02 Dec '02

Bugs item #647102, was opened at 2002-12-02 11:08 You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=100103&aid=647102&group_i… Category: Pipermail Group: 2.0.x Status: Open Resolution: None Priority: 5 Submitted By: John-Mark Gurney (jmg) Assigned to: Nobody/Anonymous (nobody) Summary: MemoryError when generating indexes Initial Comment: Vernier Networks has a relatively small/medium mailing list archive (only ~238megs) that can not be regenerated in 512megs of memory. (Once the machine needs a reboot, this will increase to 1gig.) It dies with a MemoryError supplied below: Updating index files for archive [2002-May] Traceback (most recent call last): File "/home/mailman/bin/arch", line 129, in ? main() File "/home/mailman/bin/arch", line 119, in main archiver.close() File "/home/mailman/Mailman/Archiver/pipermail.py", line 304, in close self.update_dirty_archives() File "/home/mailman/Mailman/Archiver/HyperArch.py", line 934, in update_dirty_archives self.__super_update_dirty_archives() File "/home/mailman/Mailman/Archiver/pipermail.py", line 513, in update_dirty_archives self.update_archive(i) File "/home/mailman/Mailman/Archiver/HyperArch.py", line 937, in update_archive self.__super_update_archive(archive) File "/home/mailman/Mailman/Archiver/pipermail.py", line 417, in update_archive self.__set_parameters(archive) File "/home/mailman/Mailman/Archiver/pipermail.py", line 332, in __set_parameters firstdate = self.database.firstdate(archive) File "/home/mailman/Mailman/Archiver/HyperDatabase.py", line 207, in firstdate self.__openIndices(archive) File "/home/mailman/Mailman/Archiver/HyperDatabase.py", line 238, in __openIndices self.__closeIndices() File "/home/mailman/Mailman/Archiver/HyperDatabase.py", line 259, in __closeIndices index.close() File "/home/mailman/Mailman/Archiver/HyperDatabase.py", line 186, in close fp.write(marshal.dumps(self.dict)) MemoryError and I also have an IOError on a lock file. I will be looking at it, but since I do not know Pipermail, I won't progress very far. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=100103&aid=647102&group_i…

1 0

[ mailman-Patches-647083 ] compress dies w/ MemoryError
by noreply＠sourceforge.net 02 Dec '02

02 Dec '02

Patches item #647083, was opened at 2002-12-02 13:38 You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=300103&aid=647083&group_i… Category: command line scripts Group: None Status: Open Resolution: None Priority: 5 Submitted By: John-Mark Gurney (jmg) Assigned to: Nobody/Anonymous (nobody) Summary: compress dies w/ MemoryError Initial Comment: If you have a large mailing list, it is very easy to quickly exhast memory in many conditions. nightly_gzip script tries to load the entire file into memory before compressing it, not only is this terribly ineffecient, it also means you are limited to file sizes that can fit in memory. Some machines also further restrict how much memory each process can access at a time. If that limit is 512megs, and you have a 400+meg file, compress will die. Solution: read/write the file in blocks. ---------------------------------------------------------------------- Comment By: Bryan Fullerton (fehwalker) Date: 2002-12-02 13:53 Message: Logged In: YES user_id=660772 would it make more sense to do this with zlib routines on a stream than with gzip on blocks? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=300103&aid=647083&group_i…

1 0