...

...

...

...

"LM" == Luca Maranzano liuk@publinet.it writes:

LM> It seems that all is OK now, both gate_news and the Web U/I :)

LM> I've still to report this:

LM> after issuing /etc/init.d/mailman start

| "/home/mailman/Mailman/Archiver/pipermail.py", line 210, in
| __init__
|     s = StringIO(message.get_payload())
| TypeError: expected string, list found

Fixing this is a bit more complicated than I thought. Watch CVS tonight (hopefully). I might have to spin an alpha4 in the next couple of days though.

-Barry

...

...

...

...

"BAW" == Barry A Warsaw barry@zope.com writes:

BAW> If you're watching the CVS log messages, you might see some
BAW> checkins to address the problems with Pipermail in 2.1a3.
BAW> Had an all day meeting today, and I'm beat so I'll email more
BAW> about it tomorrow, but I think I have a neat solution that
BAW> will also address Ben's patch to clean attachments out of the
BAW> archives, and may serve as a basis for a built-in de-mimer.

So here's the scoop. I've been thinking about Ben Gertzfield's code to sanitize the archives, and I've been mulling about the de-mime stuff. It all came to a head when 2.1a3 broke archiving for multipart messages.

Here's what I've now got in cvs and it seems to work fairly well. Only more testing will tell for sure.

There's a new handler module called Scrubber.py, but it's not in the primary pipeline. Only Pipermail is going to call it, and that via the new mm_cfg.py/Default.py variable ARCHIVE_SCRUBBER.

This module hardcodes the following de-mime decisions:

• text/plain parts are passed through unchanged

• text/html parts are removed completely. If the outer message is of type text/html then the whole message is discarded (i.e. DiscardMessage is raised).

• For all other non-multipart parts, we treat them as "attachments" by pulling the decoded payload out of the message, storing it in a file inside the list's private archive directory (e.g. archives/private/mylist/attachments) and rewriting the payload of the part to include a description of the attachment.

  Included in this description is a url to the attachment file, which Pipermail will hyperlink. One drawback here is that if archives are switched from public to private, or vice versa, all the attachment urls will break. But you could re-run bin/arch to regenerate the whole thing -- the key being that Scrubber works only on a copy of the message being prepped for the archiver, /not/ on the message being saved in the mbox.

• multiparts are ignored for the first pass, but are recursed to perform the above cleaning.

Then the entire scrubbed message is converted into a flat message, where only the headers are parsed and the body is slurped in one gulp; it isn't parsed recursively. Along the way, we throw out the headers for any internal parts, and we play games with the inter-part boundary strings so they are move useful (yes, this is a kludge).

There's even more kludgery involved to get Pipermail to archive scrubbed message without having to rewrite huge chunks of inscrutable code. But it seems to work.

Now, the interesting thing is that Scrubber.py is written so that it /could/ be used in the main pipeline. E.g. it supports the proper signature and semantics for use in the pipeline. But I'm not adding it there for now primarily because it isn't configurable via the web. All its decisions above are hardcoded because getting the u/i right is more work than I want to do right now.

But if you were interested in mainlining Scrubber.py, here's how you might do it: Add it to GLOBAL_PIPELINE in your mm_cfg.py. I would suggest sticking it after ToArchives so that the mbox gets the original unscrubbed message (this lets you adjust the scrubber's behavior for archive purposes and regenerate from the raw mbox). In fact, what I'd do is move ToArchive to just after the Hold module, and stick Scrubber just between Hold and Tagger. This is untested.

I think this will give us a foothold into providing a cleaner archive with Pipermail, and to experimenting with Mailman supported de-mime-ification. Probably the best that'll happen for MM2.1.

Enjoy, -Barry

On 2001.10.25, in 87vgh3ayv3.fsf@nausicaa.interq.or.jp, "Ben Gertzfield" che@debian.org wrote:

Here's a patch that actually throws out all-HTML emails, but just removes HTML parts.

Actually, why don't we just decode HTML attachments like any other, and let the user beware if they want to click on it? There are lots of legitimate reasons to allow HTML attachments. I can't think of any to allow all-HTML messages. *grin* We could treat all-HTML messages in the same way, just provide a link and let the user beware if they click on it.

Unfortunately, I think there are legitimate reasons for allowing HTML messages (as well as parts) into the record. But I don't think that legitimizes passing the HTML through literally -- this poses a big potential threat to archive viewers.

I don't care to make a full-blown rendering of HTML; I'd argue that it's not Mailman's job -- but it is Mailman's job (or, more precisely, the archiver's job) to provide any text available to the archive viewer. Whether its display is true to the intentions of the poster is subject to endless debate, but HTML is widely expected to be legible even if it's not rendered per specification -- and it almost always is, if you try hard enough -- so I think that the content should be available.

I suggested transliterating the HTML with < and > tokens, to make it harmless but legible, in case there's significant text inside. But, admittedly, that is pretty ugly. What about simply stripping out ALL markup, leaving only bare text -- and perhaps doing some minor interpretation for <br> and <p> tags, just to improve readability? Then throw in a link to the original, as Ben suggests, for good measure.

The patch also adds a filename to the replacement payload, so that users can have an idea of what they're going to see if a description was not provided (VERY common).

Ah, filenames. I'd actually like to see the filename stored on the server as requested in the MIME content-disposition. I don't think the archiver needs to guarantee literalism here; a good-faith effort is sufficient. But I think it's significant in many cases, where the transmission filename is really how the file needs to be saved locally. Minimally I'd like the filename to be shown on the archive display, but it'd be nice if I don't need to change the filename in my browser's "save as..." dialog each time I save an attachment.

I'd suggest a very basic sanitizing of the basename of the MIME filename. Something like s!.*[:/\\]!! to remove pathname components for all three major pathname separators, and then (optionally) to either hex-encode the non-alphanumeric symbols, a la HTML, or to replace them with some other token.

-- -D. dgc@uchicago.edu NSIT University of Chicago

On 2001.10.26, in 87hesmbysl.fsf@nausicaa.interq.or.jp, "Ben Gertzfield" che@debian.org wrote:

Sure, but duplicates will come in quite quickly; it will be pretty useless as soon as 40 people send in "map.gif", don't you think? We'd have to do filename munging in any case, and that's sticky. What do you suggest, prefix the filename with a number if the original filename is taken?

Oh, I forgot to mention that part: make a subdirectory of the attachments/ directory, whose name is unique and preferably based on some characteristic of the message (message-id, sequence number, etc.) combined with something unlikely to repeat (a hash, a sequence number, etc.).

All that message's attachments would go in there.

That might be desirable anyway -- do you suppose a list might see enough attachments and enough activity that all the attachments make readdirs on the attachments/ directory uncomfortably slow?

-- -D. dgc@uchicago.edu NSIT University of Chicago

...

...

...

...

"David" == David Champion dgc@uchicago.edu writes:

David> Oh, I forgot to mention that part: make a subdirectory of
David> the attachments/ directory, whose name is unique and
David> preferably based on some characteristic of the message
David> (message-id, sequence number, etc.)  combined with
David> something unlikely to repeat (a hash, a sequence number,
David> etc.).

Hm.. It'd need more testing. Either that or *some* kind of hashing based on the message-id will be necessary on any list with a good amount of traffic with attachments. But yes, that's a good idea and we should test it.

Ben

-- Brought to you by the letters N and L and the number 3. "It should be illegal to yell 'Y2K' in a crowded economy." Debian GNU/Linux maintainer of Gimp and GTK+ -- http://www.debian.org/

Folks,

Thanks for the really great feedback. I'm about to check in a new version of Scrubber.py that addresses the many issues brought up. Apologies for not quoting everything.

• permission problems: fixed

• problems with multipart/mixed containing gif, html, and jpeg parts: fixed.

• text/html decoding: there's now a new global variable ARCHIVE_HTML_SANITIZER which can be 0, 1, or a string.

This variable defines what happens to text/html subparts. They can be

# stripped completely, escaped, or filtered through an external program. The # legal values are: # 0 - Strip out text/html parts completely, leaving a notice of the removal in # the message. If the outer part is text/html, the entire message is # discarded. # 1 - Remove any embedded text/html parts, leaving them as HTML-escaped # attachments which can be separately viewed. Outer text/html parts are # simply HTML-escaped. # # The value can also be a string, in which case it is the name of a command to # filter the HTML page through. The resulting output is left in an attachment # or as the entirety of the message when the outer part is text/html. The # format of the string must include a "%(filename)s" which will contain the # name of the temporary file that the program should operate on. It should # write the processed message to stdout. ARCHIVE_HTML_SANITIZER = '/usr/bin/lynx -dump %(filename)s'

This seems to work pretty well (will provide examples shortly). As with the rest of Scrubber, it's a bit of a kludge, but perhaps not horrible. It could definitely use more testing by you guys.

It's actually rather difficult to get Pipermail to /not/ HTML-escape attachments, so I'm punting on that for now. Plus, I just feel it's way too dangerous to support.

• storing in get_filename() if available: fixed, and I've also implemented the idea of sticking each message's attachments in a separate subdir off of archives/private/mylist/attachments. The subdir is based on the Message-ID: and files inside there are uniquified if necessary.

• problems with the attachment url: what we really needed was a more elaborate PUBLIC_ARCHIVE_URL format string. It now accepts %(hostname)s as well as %(listname)s, and the former gets interpolated with the list's web host name (as looked up in the inverted VIRTUAL_HOSTS dictionary, and defaulting to DEFAULT_URL_HOST).

Watch for checkins shortly. -Barry

On Fri, Oct 26, 2001 at 04:32:51AM -0400, Dale Newfield wrote:

How 'bout if there's a directory created for each message that has separate files, and the files are placed inside that directory?

Just for the record, Sourceforge just hit the 16,000 lists limit today and broke because ext2fs doesn't support more than 32,000 links to a directory (archives/private had 32,000 archive dirs) I'm told freebsd's UFS has similar problems.

Of course, that can be fixed with Residerfs/XFS/name your FS here or archives/private/l/li/listname{,.mbox} but my point was that if you create a directory for each attachment, you're going to hit the 32,000 limit very quickly.

As far as I know, most FS don't handle hundreds of thousands of files in the same dir very well, but at least they handle it.

Marc

Microsoft is to operating systems & security .... .... what McDonalds is to gourmet cooking

Home page: http://marc.merlins.org/ | Finger marc_f@merlins.org for PGP key

...

...

...

...

"MM" == Marc MERLIN marc_news@valinux.com writes:

MM> Just for the record, Sourceforge just hit the 16,000 lists
MM> limit today and broke because ext2fs doesn't support more than
MM> 32,000 links to a directory (archives/private had 32,000
MM> archive dirs) I'm told freebsd's UFS has similar problems.

Wow, neat! Not for you, but neat. :)

MM> Of course, that can be fixed with Residerfs/XFS/name your FS
MM> here or archives/private/l/li/listname{,.mbox} but my point
MM> was that if you create a directory for each attachment, you're
MM> going to hit the 32,000 limit very quickly.

Ah, so /that's/ why we have /home/groups/m/ma/mailman... :)

MM> As far as I know, most FS don't handle hundreds of thousands
MM> of files in the same dir very well, but at least they handle
MM> it.

Don't worry, we're not talking about a directory for each attachment, but a directory for each message with an attachment. Hmm, maybe we /should/ worry! 32k messages with attachments sure doesn't seem all that many.

Hmm, have to think about it. -Barry

On Sat, 2001-10-27 at 03:43, Barry A. Warsaw wrote:

Don't worry, we're not talking about a directory for each attachment, but a directory for each message with an attachment. Hmm, maybe we /should/ worry! 32k messages with attachments sure doesn't seem all that many.

Or are the message attachment directories within the normal per-period archive directories? In that case you would have a limit of 16K messages with attachments per archive period and presumably if you have busy lists you decrease the archive period accordingly (ie for the exim lists I rotate the archives weekly).

Nigel.

On Fri, Nov 09, 2001 at 05:34:47PM -0800, Marc MERLIN wrote:

On Fri, Oct 26, 2001 at 10:43:54PM -0400, Barry A. Warsaw wrote:

...

MM> Just for the record, Sourceforge just hit the 16,000 lists
MM> limit today and broke because ext2fs doesn't support more than
MM> 32,000 links to a directory (archives/private had 32,000
MM> archive dirs) I'm told freebsd's UFS has similar problems.

Wow, neat! Not for you, but neat. :)

BTW, I ended up removing all the HTML archive directories since I've turned off HTML archiving anyway. That gives us a little while (more than six months if we are lucky :-D) before we hit the 32,000 lists mark.

I'm thinking about an optional modification to mailman which only affects list creation that creates the mailman/foo/l/li/listname dirs, and then symlinks all this to mailman/foo/listname Yeah, symlinks aren't great, but the advantage is that it requires no other changes to the mailman code

I haven't yet had the time to work on this, but I wanted to know what your preference was between a hack that optionally creates mailman/foo/l/li/listname dirs at the time the list is created, and then sets symlinks back to where the rest of the mailman code expects to see the files and dirs, or a bigger change to teach all of mailman about the new file and dir location (no more symlinks) Of course, if we go with the no symlink route, making the deep subdir config optional would be a bit more work.

What's your take on this, and which route would you prefer?

Marc

Microsoft is to operating systems & security .... .... what McDonalds is to gourmet cooking

Home page: http://marc.merlins.org/ | Finger marc_f@merlins.org for PGP key