Hi,
For those of you that don't know I am currently working on a archive
component for Mailman as part of my degree. The interface to the archive
shall be based on the ideas in Ka-Ping Yee's paper on his Zest
prototype. Over the past two weeks I have been looking at requirements
and have the following. These are in no specific order. Due to the time
constraints on my project (I am to only spend 200 hours in total on it,
including writing reports, presentations etc) there is a limit to the
amount I can do.
Functional Requirements
The archive component should
1. store email discussions.
2. integrate with Mailman.
3. provide a web-based interface to those email-discussions.
4. provide an interface that threads discussions by their content. (ZEST)
5. provide an interface that threads discussions by e-mail replies.
6. allow for full-text searching of the archives.
7. allow for filtering by date, author, and/or topic.
8. be MIME aware.
9. allow archives to be set as public or private.
10. allow posts to be added, deleted, and modified through web interface.
11. allow archives to be locked to prevent modification.
12. allow postings to be emailed.
13. allow postings to be referenced externally.
Non-Functional Requirements
1. Maintainable
2. Secure
3. Scalable
The minimum I am planning on doing is the first 5 functional
requirements restricted by the first 2 non-functional requirements.
There are a two reasons I am posting this.
Is there anything obvious that I have missed?
Which of the functional requirements, 6 to 13, do you feel are the most
important? (As part of my report I have to analyse the requirements
captured)
Any feedback is very much appreciated. Thanks in advance.
Iain
Hello again,
anyone didn't reply to me so I fixed this by myself, here is the patch:
--- Mailman/Cgi/options.py~ Thu Oct 30 12:02:41 2003
+++ Mailman/Cgi/options.py Thu Oct 30 11:44:55 2003
@@ -156,9 +156,22 @@ def main():
if cgidata.has_key('login-unsub'):
# Because they can't supply a password for unsubscribing, we'll need
# to do the confirmation dance.
+
if mlist.isMember(user):
- mlist.ConfirmUnsubscription(user, userlang)
- doc.addError(_('The confirmation email has been sent.'), tag='')
+ # If we're doing admin-approved unsubs, don't worry about the password
+ if mlist.unsubscribe_policy:
+ try:
+ mlist.Lock()
+ try:
+ mlist.DeleteMember(user, 'via the listinfo page', userack=1)
+ except Errors.MMNeedApproval:
+ doc.addError(_('Your unsubscription request has been forwarded to the list administrator for approval.'), tag='')
+ mlist.Save()
+ finally:
+ mlist.Unlock()
+ else:
+ mlist.ConfirmUnsubscription(user, userlang)
+ doc.addError(_('The confirmation email has been sent.'), tag='')
else:
# Not a member
if mlist.private_roster == 0:
On Thu, 30 Oct 2003, Pasi Sjoholm wrote:
> This is possible when user is not logged in but when logged there will be
> a request to unsubscribe user "x" for list admin. So.. it's a bug =) I
> also tested it on 2.2.3.
>
> --
> Pasi Sjöholm
>
>
> On Thu, 30 Oct 2003, Pasi Sjoholm wrote:
>
> > Hello,
> >
> > I set up a mailman today and now I have a little problem. I have this
> > corporate mailing list and I have set unsubscribe_policy to yes for that
> > list but still users can unsubscribe via web gui after they have clicked
> > the unsubscribe and confirmed that they really want to unsubscribe via
> > email.
> >
> > I'm using Mailman 2.1.2... is this a bug or what?
> >
> > --
> > Pasi Sjöholm
Hello,
Please update the URL for my essay "Reply-To Munging Considered Useful" in
the help item for setting reply-to. The new URL is:
http://www.metasystema.net/essays/reply-to.mhtml
Note that the new URL is at metasystema.net rather than metasystema.org.
There is no hurry, as metasystema.org will redirect to metasystema.net for
another year. However, after that metasystema.org will be retired, so the
sooner the change is made, the less users will get a dead link.
I'd like to take the opportunity to thank the developers of Mailman for
their efforts. I admin several lists and Mailman is my mailing list
manager of choice. I'd also like to thank you all for including a link to
my essay in Mailman - and an especial thanks to those of you who disagree
with my position but decided to include it anyway.
Best regards,
Simon Hill
On Fri, 31 Oct 2003 23:26:08 +0100
Simone Piunno <pioppo(a)ferrara.linux.it> wrote:
> On Friday 31 October 2003 22:30, Brad Knowles wrote:
>>> Right now we're receiving messages over a pipe and as far as I see
>>> we're not using any environment or command line parameter (beside
>>> the script name) so where's this SMTP magic?
>> You mean the envelope sender address? The IP address of the envelope
>> sender?
> yes, I mean exactly this, we're not retrieving them, neither by
> command line options nor by environment variables.
Ahh, yes. I'd forgotten that as I wrap Mailman's wrapper in a larger
filter which does take and processes the envelope information.
However, there remains a distinct difference if several addresses are
conflated to the same POP/IMAP box. In that case you lose the target
side of the envelope, and thus can't determine reliably which
Mailman-specific address/wrapper/alias to deliver the message to.
> Doing it would probably be MTA-specific (i.e. less portable).
As of last I checked the Debian Exim packages ship with the
no-alias-file-required Mailman configs by default.
--
J C Lawrence
---------(*) Satan, oscillate my metallic sonatas.
claw(a)kanga.nu He lived as a devil, eh?
http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.
On Fri, 31 Oct 2003 10:21:20 +0100
Simone Piunno <pioppo(a)ferrara.linux.it> wrote:
> Alle 01:51, venerdì 31 ottobre 2003, J C Lawrence ha scritto:
>>> A test installation (or a poor man's installation) will fetch
>>> messages from pop3 mailboxes...
>> Hell no. Mailman is a conformant well behaved and very standard mail
>> system, not a hack on top of a kludge that deliberately flouts the
>> standards just because it wants to.
> Hehe, I knew you'd have screamed :)
Aye, its a lot of fragile and unnecessary work that encourages a use of
mailman which I'd rather never occurred.
> Try to think it the other way: a test installation with pop3 is an
> hack on top of a solid conformant well behaved and very standard mail
> system.
Right, but the information lost across the non-SMTP translation is
irretrievable and necessary.
> Skilled people can do it right now with Mailman 2.1: just configure
> fetchmail to act as an MTA and pipe messages to Mailman. The only
> difference I proposed is that non-skilled people should be able to do
> this...
I account this one of those areas where the people who know how to do
it, also have some hope of knowing better than to.
>> Why? Even ignoring the abuse possibilities, what possible reason
>> could we have for that time and effort investment when those problems
>> are already far more competently and easily handled than we ever
>> could, and there are so many other, more rewarding and demanding
>> problems and features on the burner?
> We all know CGI is sub-optimal. We're also planning to stop vending
> archives directly from disk, increasing the CGI load.
True, however the processing delta can be kept quite small.
> mod_python or a web runner would perform better.
The problem there is that they are insufficiently portable.
> I feel the reverse-proxy configuration (similar to zope) would be the
> better choice.
Why would it be better? What are the disadvantages of those benefits?
What specifically useful or necessary function would it add which is not
being served today? What interesting and valuable deployment cases
would it prevent or make more difficult? Which would it open to the
product that are not current addressed?
Let's get a cost and value statement going.
--
J C Lawrence
---------(*) Satan, oscillate my metallic sonatas.
claw(a)kanga.nu He lived as a devil, eh?
http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.
On Fri, 31 Oct 2003 16:04:43 +0100
Brad Knowles <brad.knowles(a)skynet.be> wrote:
> At 8:20 PM -0500 2003/10/30, J C Lawrence wrote:
>> While I don't disagree, this is really an MTA's job, not Mailman's.
>> This is why I've been doing log analysis of MXes and routing mail to
>> customised outbound MTAs on the basis of responsiveness, since early
>> 2000. Adaptive MX routing is great stuff.
> There is a need for this function, and no MTA available today does it.
> MLMs throughout the history of the Internet have incorporated a
> variety of features for SMTP performance enhancement that are unique
> to mailing lists or are usually found primarily in mailing lists, and
> this is no different.
True. Its not a very difficult process, and is absurdly expensive the
way I handle it. At some point in my copious spare time I should whack
another couple config tokens into Exim, just to up the ante.
> If you want to externalize all these functions outside of mailman,
> that's fine. But then someone has to pick up the ball and start
> hacking on bulk_mailer or some other program to provide these
> features.
Aye, but some care should be taken here defining who the people are,
between the Good-For-Mailman, and Good-For-Large-Mail-Systems camps.
They're related, but not synonymous.
>> Yup. I did it at the first level with an initial SMTP proxy which
>> routed based on MX response records pulled from a DB.
> Again, this is a feature which is not found on any MTA available
> today, and which is known to have a huge impact on mailing list
> performance. This feature needs to be provided somewhere, by someone.
True.
> If you go back to Barry's message, he was talking about getting
> even further involved, by doing a mail-merge process. Since there is
> no MMTP (something that Bryan Costales, Eric Allman, and I had worked
> on for a while, before we realized that it would just make the spam
> problem worse and then dropped all further efforts), there is a need
> for an intermediate program that is called by mailman and then hands
> the messages off to the MTA.
<nod>
Mailmerge and VERP customisation, and the standards for the
communication of those things to the MTA are areas that need attention,
both for Mailman and the rest of the market (tho the IronPort and
related guys might argue). This would be a good point to get some
cross-MTA discussion going on.
>> We're not in the game of second guessing the MTAs. That way lies
>> wasted time and madness.
> If there were MLTAs which were optimized for this function,
IIRC QMail has a (typically DJB) VERP/rewrite handoff method. I also
recall that it is very bound into QMail's process and IO model, but
perhaps this should be examined?
> I would agree with you. Since we're trying to take standard MTAs
> which may have only some optimizations that might be generally
> applicable to most situations (including mailing lists), I must
> disagree.
There's that audience problem again. I actually agree with you in the
general case, and am willing to spend time and effort in that direction.
However I see this as somewhat disjoint from Mailman in specific.
--
J C Lawrence
---------(*) Satan, oscillate my metallic sonatas.
claw(a)kanga.nu He lived as a devil, eh?
http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.
On Thu, 30 Oct 2003 18:20:56 +0100
Brad Knowles <brad.knowles(a)skynet.be> wrote:
> At 8:41 AM -0800 2003/10/30, Chuq Von Rospach wrote:
> One of them is recipient sorting by average delivery time over the
> past week (probably want a decaying geometric mean), which would
> require tracking log data on a per-recipient basis.
While I don't disagree, this is really an MTA's job, not Mailman's.
This is why I've been doing log analysis of MXes and routing mail to
customised outbound MTAs on the basis of responsiveness, since early
2000. Adaptive MX routing is great stuff.
> Another is two-level message handling, by configuring the MTA for the
> initial delivery attempt to use very low timeouts, but then to fall
> back to a secondary MTA (or MTA pool) that uses more standard timeouts
> for those sites that are slower.
Yup. I did it at the first level with an initial SMTP proxy which
routed based on MX response records pulled from a DB.
> Perhaps in its current form, that is true. However, not all sites are
> using sendmail 8.12, and of the ones that are, most are probably not
> using it in a manner that is more suitable for mailing lists.
I'm generally of the view that Mailman should do opportunistic domain
sorting and per-MTA customised VERP handoffs (because nobody has
standardised VERP across MTAs), and beyond that to back off. Mailman's
job is to get the outbound mail into the MTA's spool as quickly as
possible, wrapped in transactions (ie RCPT TO bundles) that are friendly
to efficient processing, and that's it.
We're not in the game of second guessing the MTAs. That way lies wasted
time and madness.
> However, given the issues you've mentioned, it would probably be a
> good idea to be able to turn off selected "bulk_mailer" type features,
> so that you can let the MTA do more of it's job better -- if it is
> configured to do so.
There are thresholds for covering up for broken software. There are
also thresholds for covering up for SysAdm negligence or oversight.
You've got to pick where you stop accepting the problem. Ideally we
should be resilient and friendly to both. Realistically we need to do
something reasonable and not worry too hard about the rest.
Priorities.
Mailman's primary performance problems are not at the MTA hand off. MTA
configuration and tuning for mailing lists is only a minor art. There
is not-inconsiderable documentation and understanding of the field. A
US$2K commodity box subjected to moderate tuning efforts using readily
available documentation can sustain 2,400 outbound deliveries per
minute. You do the arithmetic. In a perfect world that maps out to 3.4
million per day. Cut that under half for queue injection overhead other
crap and you're still talking a million deliveries per day for a US$2K
host.[1] A million messages a day already puts us above the 99th
percentile for list server audiences. I'm not really concerned about
that problem.
Where Mailman's performance hurts is in the handling of the list
configs, especially for lists with very large memberships rosters and in
queue runner performance and overhead (try watching queue runner's
system resource profile in v2.1 for lists with > 50,000 members). For
me those are the obvious low hanging fruit, and those are the points
that will help not just the performance hounds, but also the lower 80%
who are running under-provisioned under-configured under-admined
multi-purpose boxes who want Mailman to be a bit more reasonable and
forgiving about their not-so-brilliant systems.
[1] That's of course assuming reasonable sustained queue size and
responsive MXes. However, those are separate problems and ignoring
MTA-specific behaviours (like Exim's active hatred of large queues),
the methods and systems to segment and tame those problems are fairly
well known.
--
J C Lawrence
---------(*) Satan, oscillate my metallic sonatas.
claw(a)kanga.nu He lived as a devil, eh?
http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.
On Fri, 31 Oct 2003 00:45:32 +0100
Simone Piunno <pioppo(a)ferrara.linux.it> wrote:
> On Thursday 30 October 2003 05:17, J C Lawrence wrote:
>>> ...as well as implement a bulk mailer to eliminate the need for an
>>> outgoing mail server.
>> Eeeek! I trust this would be for immediate handoff to a "real" MTA
>> versus handling final delivery directly? Quite the Pandora's box if
>> not.
> I believe the best approach is to cover all options:
No. This is an one-size-doesn't-fit-anybody argument.
> A test installation (or a poor man's installation) will fetch messages
> from pop3 mailboxes...
Hell no. Mailman is a conformant well behaved and very standard mail
system, not a hack on top of a kludge that deliberately flouts the
standards just because it wants to.
> ... (polling! I can hear you scream while you read this!) and send
> them directly to the internet (no real MTA involved) and will probably
> serve web pages directly, controlling port 80 (no real web server
> involved).
Why? Even ignoring the abuse possibilities, what possible reason could
we have for that time and effort investment when those problems are
already far more competently and easily handled than we ever could, and
there are so many other, more rewarding and demanding problems and
features on the burner?
--
J C Lawrence
---------(*) Satan, oscillate my metallic sonatas.
claw(a)kanga.nu He lived as a devil, eh?
http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.
On Thu, 30 Oct 2003 09:53:19 -0500
Barry Warsaw <barry(a)python.org> wrote:
> - The desire/requirement that Mailman chunk and sort recipients
This shouldn't be any more complex than domain sorting, and need not be
perfect.
> - The ability for Mailman to swamp the mail server or cause the mail
> server to consume all available cpu
Rate limiting.
> - The fact that failures in upstream mail server are reported to
> Mailman as bounces instead of as error codes
I don't know that Mailman can do anything about this. We can't reliably
distinguish between system errors and delivery failures for MTAs beyond
Mailman's borders. There's a protocol hole here I don't know we can or
should attempt to fix.
> - Inefficiencies in VERP/personalization/mail-merge because of the
> lack of cooperation
Oh yeah.
> - The need for Mailman to queue outgoing messages that aren't
> completely delivered
Queue runner could do with some more intelligence in that dept.
> Mailman wants to simply hand that data off to some agent and forget
> about it. It wants to know that the agent will make best effort to
> mail merge and deliver. It wants to be informed of any final delivery
> failures. And that's it. Mailman doesn't want to chunkify
> recipients, and it doesn't want to sort them. It doesn't want to
> worry about a mail server effectively managing system resources. I'd
> rather not have to hand it a couple of meg of recipient or
> substitution data, but there seems to be no other way.
> So what can we do here to improve matters?
Start yelling at DJB, Wietse, Phillip, and Eric about a standardised
SMTP extension for VERP. With a little luck and minor work we can
probably get some of the other commercial mail people involved as well.
--
J C Lawrence
---------(*) Satan, oscillate my metallic sonatas.
claw(a)kanga.nu He lived as a devil, eh?
http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.
On Thu, 30 Oct 2003 08:41:17 -0800
Chuq Von Rospach <chuqui(a)plaidworks.com> wrote:
> On Oct 30, 2003, at 7:48 AM, Brad Knowles wrote:
> One big reason: increasing spam blocking (stupid or otherwise) of
> non-individually addressed email. The old list server setup of:
> to: subscribers of list <list@foo>
> bcc: bulk_drop(a)of.subscribes
> is increasingly risky as far as delivery is concerned.
I've seen a couple mail BCPs and internal spam-handling plans at large
ISPs and corporates which explicitly include the line item:
Discard all mail with more than one address in the envelope.
Scary, stupid, true: They want the pain to stop. I find it hard to
blame them.
> I also don't think it allows for the kind of personalization that's
> needed for your general audiences (help URLs, unsub URls, etc).
Aye, such VERPish attributes is becoming a necessity.
--
J C Lawrence
---------(*) Satan, oscillate my metallic sonatas.
claw(a)kanga.nu He lived as a devil, eh?
http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.