Or Re: [Mailman-Developers 10417] Improving the archives
I would like to interject and highlight some use cases for stable
and predictable IDs. For us, "message IDs" are directly used both by
people and ignorant programs. Our mailing lists serve as a permanent
and concise record of our discussions, decisions, and operations, and
we find it invaluable to be able to refer to individual messages in a
simple and memorable way: "message 1210 in the calibration list", say.
Other people can then easily jot that info down or directly find the
message. Some message IDs even become shorthands for a particular
topic or decision. We have also added trac InterWiki templates
pointing into our mail archives (as listname:number), which encourages
desirable cross-referencing (PRs, wiki pages, and SVN change logs can
refer to mail messages, just as wiki pages could always refer to
changesets and PRs, etc, etc.) But trac InterWiki templates can only
interpolate $1,$2,... arguments into strings, and could not possibly
calculate anything based on the _content_ of the messages.
Globally unique IDs, hashed IDs, etc., are very appealing from
various CS-y and techie points of view, but are simply not memorable
to humans or knowable by dumb external programs. I think as much, or
more, effort should be put into delivering a straightforwardly useable
naming scheme as goes into making an arbitrary message recoverable
from anywhere. Basically, "friendly URLs" should be a primary
requirement, not an optional afterthought for careless geeks like me
to get wrong later....
We long ago added an extremely simple ID handoff between MM 2.1.8
and pipermail, and though imperfect it has served us well. Basically,
we hijacked the .post_id member in mailman (otherwise basically
unused, and mysteriously a floating point number); CookHeaders stuffed
it into a X-Mailman-Sequence-ID header line, and AfterDelivery
incremented it. In turn, pipermail uses the header to feed a sequence
ID into make_article, and the message is squirreled away as
$mailinglist/all/%d.html. There are a few other minor matters (e.g.
post_id was added to Decorators, a couple of templates were changed,
we lost having 'ls' sort chronologically [did we have to add .last
and .prev to the HyperDatabase classes?]), but it really was a minor
bit of work. And for stability, as long as the archive files aren't
lost, pipermail rebuilds should yield the same URLs even if junk
messages have been deleted. [Oh, we did also add a "never rotate"
policy to our archives, but that is finesseable. ]
As an aside on other discussions, can you get away without using
Message-ID or Date? I.e., aren't those just more of those tokens which
were standardized back before the Internet got tricky enough to
invalidate the standards? Mailing lists serialize incoming messages,
and so can generate their own unique and trustworthy IDs. "UUIDs"
would work, but if you can trust yourself to generate them,
consecutive integers provide minimal, order-preserving, perfect
hashing, too!
Anyhow, we have found that people will enthusiastically refer by
name to individual messages within mail archives if they can.
- craig
Hello.
If I use "--with-user" I got some errors with qmail-to-mailman.py.
That's why it has got on line 67:
local = re.sub("^mailman-","",local)
But "mailman" isn't correct, because of "--with-user". Can you please
fix that with something like:
local = re.sub("^" + os.environ['USER'] + "-","",local)
or even better
re.escape(os.environ["USER"])
Lots of greetings
Sebastian 'kickino' Wieseler, one of the Savannah admins,
--
,= ,-_-. =. /"\
((_/)o o(\_)) \ / ASCII Ribbon Campaign
`-'(. .)`-' && X against HTML e-mail
\_/ / \
I'd like to write a filter for the Courier MTA which will run mailman's
approval and spam tests before accepting messages to mailing lists. My
filter framework is in python, so I think I can import the relevant bits
of mailman (I'd like to start with MM2.1 compatibility). Can I get any
pointers to the functions that check the sender for approval, and for
the spam-specific posting filters?
I'm sorry if this issue was already addressed in later versions of mailman.
I'm running debian etch and the mailman deb:
dpkg -l | grep mailman
ii mailman 2.1.9-7 Powerful,
web-based mailing list manager
I wrote a majordomo like interface for mailman which allows users to perform
some backwards compatible functions via email, and access some information,
like subscribed lists and owned lists via a web interface.
When I released the interface to users, one of the first things a user did
was to use all caps in the list they were attempting to create. My script
verifies that the list doesn't already exist:
if Utils.list_exists(listName):
which it didn't and then passes it into the mailman Create function.
mlist.Create(listName, listOwner, listPasswordHash )
There were no errors in the create process. The problem was that the list
was actually a duplicate of an existing list which obviously wasn't in all
caps. Since the process by which mailman gets the list configuration seems
to be case insensitive, both lists would access the old config, so
functionally I had two references to the same list.
The solution to the problem was to just .lower() the listName string at the
top of the list create function, but it might make sense to lower the string
in the Create function as mailman is not case sensitive after that point.
Let me know if you'd like me to open a bug for this issue...
Thanks,
-Lawren Quigley-Jones
lawrenqj(a)gmail.com
I had a little problem because my provider does not allow running daemons.
However, crontab is fine.
Since setting up aliases does not work either and I did not want to
change .procmailrc all the time, I'm using a catchall user and Maildir
delivery.
In order to make this work, I changed two things.
diff -ur mailman-2.1.9/Mailman/Queue/MaildirRunner.py mailman-2.1.9-mod/Mailman/Queue/MaildirRunner.py
--- mailman-2.1.9/Mailman/Queue/MaildirRunner.py 2005-08-27 01:40:17.000000000 +0000
+++ mailman-2.1.9-mod/Mailman/Queue/MaildirRunner.py 2007-10-06 16:08:29.000000000 +0000
@@ -123,7 +123,7 @@
# message was destined for. See verp_bounce() in
# BounceRunner.py for why we do things this way.
vals = []
- for header in ('delivered-to', 'envelope-to', 'apparently-to'):
+ for header in ('x-original-to','delivered-to', 'envelope-to', 'apparently-to'):
vals.extend(msg.get_all(header, []))
for field in vals:
to = parseaddr(field)[1]
diff -ur mailman-2.1.9/bin/qrunner mailman-2.1.9-mod/bin/qrunner
--- mailman-2.1.9/bin/qrunner 2006-01-19 01:07:40.000000000 +0000
+++ mailman-2.1.9-mod/bin/qrunner 2007-10-09 10:33:11.000000000 +0000
@@ -119,7 +119,8 @@
# Subclass to hack in the setting of the stop flag in _doperiodic()
class Once(qrclass):
def _doperiodic(self):
- self.stop()
+ if self._oneloop()<1:
+ self.stop()
qrunner = Once(slice, range)
else:
qrunner = qrclass(slice, range)
First I added the x-original-to field since some mails got stuck.
The second change in qrunner is the short hack to get all pending
stuff done with one run from crontab:
qrunner -v -r All -o
Would that be something that could be included in future releases
maybe as a separate option?
Regards,
Maximilian
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Those of you who have been watching the commit messages can see I've
been making some good progress. I'm actually hoping to have a
Mailman 3.0 alpha some time RSN which will almost allow you to run
the system from the command line, but without a web u/i.
So one of the things I'm looking at is the MM2.1 concept of an
Approved header. If a message comes into a list with an Approved
header (or an Approved line at the start of the message body), and
that header has a password that matches the list admin or moderator
password, the message is pre-approved and short-circuits the posting
tests.
The concept doesn't translate well in a Mailman 3 world where there
is no shared admin or moderator password. Web access will be control
via roles and protected by user authentication much like any modern
web application.
So the question is, what do we do about the Approved header?
1. We can drop the concept altogether. This means there'd be no way
to post a message as coming from an approved source, with a bypass of
the posting filters. Maybe because few people have MUAs that support
adding custom headers, this feature just isn't used much in the real
world these days. You'd still have the moderation bit for announce-
only lists though.
2. Replace the concept with some other email authentication
mechanism, e.g. something more secure like a signature check. The
problem with this is that I still don't think message signing is
common practice outside our small community of geeks.
3. Allow an owner or moderator to use their own password in the
Approved header. I'm not crazy about this because it has to be sent
in the clear and if (when?) it gets compromised, their account is
compromised, and this includes their administration of the mailing list.
4. Add a new shared password just for this purpose. You'd still have
to communicate it to all your moderators, probably via the web page,
but at least this password wouldn't have any other purpose so if
(when?) it gets compromised, the only asset it protects is approved
postings. Bad yes, if a spammer gets it, but easily changed and
hopefully fairly limited in the damage it can do.
5. Your suggestion.
Comments? I think my preference would be for #1 with future support
for #2 and just accepting the fact that message signatures are for
power users. Maybe that set is pretty close to the set of people
currently using Approved anyway.
Cheers,
- -Barry
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
iD8DBQFHAwd62YZpQepbvXERAmMiAKCm3EyxA1CWxWyz4zWkzNwIDpCNKQCbBSXz
hGqwpKEGmUScNjov68TUdgs=
=gUiT
-----END PGP SIGNATURE-----
Since I've largely finished up the coding contract that was eating up
a lot of my time, I'm thinking that I'd like to do some coding for
fun. And nothing says fun like trying to fix the Mailman archives! ;)
I'm trying to remember all the things people have suggested for the
archives in the past so I can figure out what needs to be done and
what might be nice to have, and see if this is doable in the time I
have in the foreseeable future.
The big things people wanted most, if I recall correctly, included:
- modernized HTML/CSS/Themes (preferably to match a modernized web
interface... is that all set up now?)
- archive links that won't break if the archive is rebuilt
- better address obfuscation (maybe by generating pages through cgi)
- search
- not adding a billion dependencies to Mailman
Here's the list from the wiki's Mailman 2.2 page: http://
wiki.list.org/display/DEV/Mailman+2.2
* Reconsider using a 3rd-party archiver
* Perhaps URLs to messages should be based on message-ids
instead of message numbers so that regenerating archives can't break
links. This must include backward compatible links
* Ditch direct access and vend all archive messages through CGI
so that we can do address obfuscation, and message deletion, etc. on
the fly (with caching of course, but have to worry about web crawlers).
* Add RSS feed
* Allow for admins to remove or edit messages through the web.
* Move archive threads into another list?
* Put archives in the list/mylist directory.
* Add a search option
* Make archives default template look and feel similar to Web UI
(whatever it looks like after the Summer of Code project is done)
* Make archive templatable (at least by changing CSS) so they
can match people's existing site look-and-feel
* MUAs usually make URLs clickable. An new Archive could be used
when posts are distributed, in the footer, so that each message has a
link to the whole thread in the Archive.
* Present all messages in a thread at once, and offer plaintext
download of the whole thread
* Put messages into a database and/or move away from mbox as the
canonical storage format.
So the questions are:
(1) Is anyone working on this already?
(2) What else is on people's wish lists for a pipermail replacement?
Terri
