Mailman 3 December 2000 - Mailman-Developers

Re: [Mailman-Developers] Re: Components and pluggablility
by J C Lawrence Dec. 15, 2000

Dec. 15, 2000

On Thu, 14 Dec 2000 23:00:02 -0800 J C Lawrence <claw(a)kanga.nu> wrote: > What's this host-specific stuff? More later. Oops, I forgot to paste a particular bit from my notes: The basic pattern of execution: Messages arrive from the MTA and are put in files in ~/inbound with the name of the file matching both the list address it was send to and what needs to be done to it (bounce processing, list post, etc). A cron job runs and reads ~/config.<hostname>, and on the basis of the contents of that file, forks up to three child processes and then silently dies: i) Process inbound messages ii) Process messages in ~/lists/<listname>/pending iii) Process messages in ~/outbound The ~/config.hostname file indicates which if any or all of the above three children should be forked, and further, can specify both which groups or individual lists to process for all three. It is this which allows functionality to be distributed across hosts with individual hosts assuming specific named loads. Note: Its actually more than three children as that doesn't (yet) account for command and bounce processing (which I've put into seperate queues), but it gives you the general idea. -- J C Lawrence claw(a)kanga.nu ---------(*) http://www.kanga.nu/~claw/ --=| A man is as sane as he is dangerous to his environment |=--

1 0

Re: [Mailman-Developers] (no subject)
by J C Lawrence Dec. 14, 2000

Dec. 14, 2000

On Thu, 14 Dec 2000 14:52:00 -0800 Chuq Von Rospach <chuqui(a)plaidworks.com> wrote: > one thing I'm doodling with for my SMTP back end is starting up a > server that places a socket, then starting up "N" SMTP processes > as clients that grab addresses from the server one at a time for > delivery. This gets me completely away fro this "slow DNS" stuff, > since any one slow address slows only itself, and since the system > I'm looking at is 100% verped/customized (ala Lyris's footers, at > the minimum), I'm not worrying about the added overhead (you could > potentially do batches through an interface). using sockets means > the clients can go off-machine for free, as long as they know > where to look. If the number of slow addresses at any instant exceed N th entire system stops for that period. The benefit of parallelisation in this case is that in *general* traffic will cotniue to flow given one or more bad addresses, and the assumption is that the rate of bad/slow addresses will never/rarely coincide to the point that the entire queue is bogged. > now, maybe it could be something like that, a controlling process > that uses both threads and forks (and perhaps remote commands > through rsh or ssh) to spawn instances as needed... I see this as orthogonal to Mailman or the MLM process. You could drop such a solution transparently into the outbound queue process. -- J C Lawrence claw(a)kanga.nu ---------(*) http://www.kanga.nu/~claw/ --=| A man is as sane as he is dangerous to his environment |=--

1 0

Re: Slow Performance on semi-large lists
by David Champion Dec. 14, 2000

Dec. 14, 2000

I shifted this to mailman-developers because I want to talk about changes in qrunner that D.J. Atkinson brought up. On 2000.12.13, in <Pine.SOL.4.05.10012131455360.22660-100000(a)babu.pcisys.net>, "D.J. Atkinson" <dj(a)pcisys.net> wrote: > > I posted a message over the weekend where I saw qrunner only processing > part of the queue. It turned out that there were three messages in the > queue with 3 unresolvable names each. (3 messages to the same list) > Each of these queued files took 400 seconds to time out, by which time, we > were past the default max qrunner process length (15 minutes), and qrunner > exited. > > I've of course now increased the process length to 30 minutes, and > everything seems to be OK. But that's only temporary, I'm sure. As list > volume builds, it will become a problem again. It would be great if there > were a more graceful way of dealing with this than currently exists. How about altering qrunner's algorithm to split the queue on timeout, appending the head of the queue to the tail? A - fails B - succeeds C - fails D - fails/unprocessed; qrunner times out E - unprocessed F - unprocessed With this change, your next queue runner will process this queue: E F A C D Eventually (ahem) the queue will contain only those batches which are hard to deliver, and they'll be re-ordered with each run to give equal attempts over time. Actually, that's not true if the queue is reduced to containing only A, C, and D, and qrunner always times out on D; D will never get the same time as A and C. Leaving D at the head of the queue (that is, splitting the queue ahead of the current batch, rather than behind it) solves that problem until the case occurs in which D contains enough bad or slow addresses to stop the queue even though it's first. Two solutions to this: 1) never stop qrunner during the first queued batch (always wait for it to exit); or 2) split the queue ahead or behind of the current batch randomly. Does this seem to anyone else to solve the problem? I haven't looked at the code yet, so this is just cursory thought. -- -D. dgc(a)uchicago.edu NSIT University of Chicago

3 4

Re: Mailman-Developers digest, Vol 1 #741 - 8 msgs
by Ken Manheimer Dec. 14, 2000

Dec. 14, 2000

This is one case where i really mean for the message subject to refer to the digest as a whole - because today's material, this digest, is *chock-full* of material, and i want to make a suggestion about it: maybe it's time to do some harvesting - take a look at carving out some space in barry's post-2.0 mailman wiki: http://www.zope.org/Members/bwarsaw/MailmanDesignNotes/FrontPage and filling in some areas covered by these postings. (It may be worthwhile to do a top-down, "what are the topic areas" first kind of approach - and then list various variations being discussed, for controversial areas; it looks like there's a *lot* of agreement about some substantial portions of a Mailman Next Generation world view.) I know i'm unable to do more than scan most of it, and even if you are savoring every bit, it's flowing by - tomorrow you'll either have to rehash it or have it fade - unless you get it in some stable, growable home... Ken klm(a)digicool.com

1 0

Re: [Mailman-Developers] (no subject)
by J C Lawrence Dec. 14, 2000

Dec. 14, 2000

On Mon, 11 Dec 2000 20:49:36 -0800 Chuq Von Rospach <chuqui(a)plaidworks.com> wrote: > At 7:51 PM -0800 12/11/00, J C Lawrence wrote: >> ObTheme: All config files should be human readable unless those >> files are dynamically created and contain data which will be >> easily and automatically recreated. > ObTheme: All configuration should be possible via the web, even if > the system is misconfigured and non-functional. Anything that can > NOT be safely reconfigured without breaking the system should not > be configurable via the web. (in other words, anything you can > change, you should be able to change remotely, unless you can > break the ssytem. If you cna break the system, you shouldn't be > allowed near it trivially...) Agreed. I'm currently working under the generous assumption that its possible to cook up a web interface design for almost anything, so I'm punting there for now. >> 1) Using multiple simultaneous processes/threads to parallelise a >> given task. >> >> 2) Using multiple systems running parallel to parallelise a given >> task. >> >> 3) Using multiple systems, each one dedicated to some portion(s) >> or sub-set of the overall task (might be all working in parallel >> on the entire problem (lock contention! failure modes!)). > that's my model perfectly, althought I think 2 and 3 are > reversed. <nod> > it's cleaner architecturally to go to divesting and distributing > functionality before 'clustering'. In fact, I'm not sure > clustering (which I'll use to term multiple mailman systems > running in parallel) implies a system really, really large, when > you realize that the primary resource eaters (like delivery) can > effectively be infinitely distributed. Yup, I've accounted for that so far in the design. > I'm not sure how big a Mailman system you'd need ot require > parallelizing the core process, as long as you can divest off > other pieces to a farm that could grow without bounds. So maybe we > don't need that next (complicated) step, and make it parallelized > and distributable for everything except that core control process, > but manage the complexity of that control process to keep > everyting out of it exect the absolute necessity. I'm working on the principle that there is no core process, and thre are no musical conductors or other time beaters, just discrete nodes and processes competing for resources. >> Observation: MLMs are primarily IO bound devices, and are >> specifically IO bound on output. Internal processing on mail >> servers, even given crypto authentication and expensive >> membership generation processes (eg heavy SQL DB joins etc) are >> an order of magnitude smaller problem than just getting the >> outbound mail off the system. > some of that is the MUA's problem, actually, but they get tied > together. you don't, for instance, want an MLM who will dump 50K > pieces of email an hour into the queues of an MUA that can only > process 40K... I think you mean MTA above and below. > But in general, you're correct. Especially if you define DNS > delays and SMTP protocol delays caused by the receiving machine to > be "output" (grin) Or just the simple FS commit requirements for MTA spools. Its a heavy process. >> Sites with large numbers of lists with large numbers of members >> (and presumably large numbers of messages per list) are the >> pessimal case, and is not one Mailman is currently targeting to >> solve. > but if you define the distribution capabilities correctly, this > case is solved by throwing even more hardware at it, and the > owners of this pessimal case presumably have a budget for it. If > you see someone tryting to run Sourceforge on a 486 and a 128K DSL > line, you laugh at them. True, except that lock contention becomes a major problem and scheduling strategies become critical. >> Observation: Traffic bursts are bad. Minimally the MLM should >> attempt to smooth out delivery rates to a given MTA to be no >> higher than N messages/time. > The obverse of that is that end-users seriously dislike delays, > especially on conversational lists. It turns into the old "user > expectation" problem -- it's better to hold ALL mail for 15 > minutes so users come to expect it than to normally deliver mail > in 2 minutes, except during the worst bulges... But in general, > the MLM should deliver as fast as it reasonable can without > overloading the MUA, which implies some kind of monitoring setup > for the MUA, or some user-controlled throttling system. the latter > unfortunately, implies teaching admins how to monitor and adjust, > a support issue. The former implies writing an interface for every > MTA -- a development AND support issue. My intent so far is just "deliver no more than N mesages per minute" per outbound aueue runner. It knocks the peaks off the problem, and the base structure ie easy to extend from there (and I don't want to think about that now). >> 1) Receipt of message by local MTA > 1a) passthrough of message via a security wrapper from MTA to list > server... (I think it's important we remember that, because we > can't lose it, and it involves a layer of passthrough and a > process spawning, so it's somewhat heavyweight -- but > indispensable) I should note that my base design is very heavy in terms of process forks (which happen to be quite light weight under Linux, but that's another matter). The general structural approach I'm taking is: There's a directory full of scripts/programs. Run them all, in directory sort order, on this message to determine if we should do XXX with it. Now the default case could have those directories empty, meaning that Mailman will default to internal/cheap implementations, but its much easier to just have default implementations of the scripts for those directories and then punt normally. ObNote; Of course the default scripts could by python scripts and could be processed in-line as modules rather than forking children. >> 2) Receipt by list server 3) Approval/editing/moderation 4) >> Processing of message and emission of any resultant message(s) 5) >> Delivery of message to MTA for final delivery. > 6) delivery of message to non-MTA recipients (the archiver, > the logging thing, the digester, the bounce processor....) I'm actually doing archiving by injecting new messages back into the inbound queue which are addressed to the archiver. Digest processing should probably be handled this way, but I've currently got it as a pre-post script (not entirely keen on that). Bounces are handled entirely OOB to the rest of the MLM, rather similarly in fact to request messages. >> #1 is significant only because we can can rely on the MTA to >> distinguish between valif list-related addresses and non-list >> addresses. > although one thing I've toyed with is to give a subdomain to the > MLM, and simply pass everything to it (in sendmail terms, using > virtusertable to pass @list.foo.bar to mailman(a)foo.bar). Then you > take the MLM out of having to know what lists exist and > administrative needs to keep that interface in sync. There are a couple other list servers that demand that approach. The problem is that it really doesn't fit well with people/sites that don't control their own DNS. > I've more or less decided than when I rewrite my internal > corporate mail list, I'll do that rather than generate alias > listings (for, oh, 12,000 groups) and teh hassles and overheads of > all that. That'll be especially useful if we do waht I hope, which > is set it up so the server has no data at all, but authenticates > via LDAP to get list information on demand out of the corporate > databases. There are some definite advantages to not knowing > whether something exists until the need to know exists -- and as > Mailman starts edging towards interfacing to non-Mailman data > sources for list information, that ability grows in importance. FWIW the design can do that right now. A message comes in, various parameters are extracted from it, and the parameters are handled to a directory of scripts the accumulated stdout of which forms the distribution list for that message. The distribution list can be passed thru a pre-processor (dupe removal, domain sorting, MX sorting, whatever) to do any final processing of the distribution list before attaching it to the message and putting it in the outbound queue. So, want LDAP? Want SQL? Want local DBM? Want all three? No problem. > 6) is the processesing needed to support other functions that act > on messages. The idea is that instead of delivering to the MTA, we > have a suite of functions that deliver the message ot whatever > needs to process it. Those can be asynchronous and don't need to > be as timely as (5), and have different enough design needs that I > split them out from the MTA delivery (although traditionally, > stuff like digests are managed by doing an MTA transfer out of the > MLM and back in to a different program...) I have one set of queue processing functions geared solely for list posts. There are then several parallel queue sequences dor processing non-posts (such as bounces, command requests, etc). Additionally, you can trivially set things up so that post explosions occur on N machines, while command processing and bounce processing occur only on say machine X (which is perhaps on the other side of the DMZ and access rights to your internal backing stores). I don't see the different queues needing markedly different designs, but needing to be able to have their processes supports cleanly divisible. The base structures end up markedly similar after that. > It also assumes that these non-delivery things are separate > processes from teh act of making them available to those things, > to keep (6) lightweight as possible. Process fork overhead is a problem I've not confronted yet. Its going to need looking at. That and distributed lock contention. Bother are pretty ugly with my current model. > and besides, they are basically independent, asynchronous > processes that don't need to be managed by any of the core logic, > other than handing messages into their queue and making sure they > stay running. same with, IMHO, storing messages for archives, BTW I'd like to have the MLM archive messages such that a member can request, "SEND ME POST XXX" and have the MLM send it to him. Ditto for digests. This is in addition to any web archiving. > storing messages for digests, updating archives, processing > digests (but the processed digest is fed back into the core logic > for delivery), and whatever else we decide it needs to do that > isn't part of the core, time-sensitive code base. I've been thinking about this. I *REALLY* don't think there's much time sensitive code in a MLM. There's a lot of data you want to get out of the way as quickly as possible, because if you don't its just going to build up and make a bigger problem, but the actual speed with which individual bits prior to the explosion occur seems arbitrary outside of a latency viewpoint. Okay, it takes N ticks for a post to start exploding versus it takes 5N ticks to start exploding? Am I really going to care when handling the explosion takes several hundred N? Even with say 50 forks per inbound list post prior to explosion, the comparitive overhead compared to explosion is still trivial. > (in fact, there's no reason why you couldn't have multiple flavors > of these things, feeding archives into an mbox, another archiver > into mhonarc or pipermail, something that updates the search > engine indexes, and text adn mime digesters... by turning them > into their own logic streams with their own queues, you > effectivley have just made them all plug-in swappable, because > you're writing to a queue, and not worrying about what happens > once its there. you merely need to make sure it goes in the right > queue, in the approved format. Hurm. Good point. I like that idea. Just inject messages back into the system targetted for the apppropriate stream. Nice. Heck, perhaps I should shove this thing as-is into Barry's ZWiki. > We're visiting the relatives. Cover us. I missed you. Please wait while I reload. -- J C Lawrence claw(a)kanga.nu ---------(*) : http://www.kanga.nu/~claw/ --=| A man is as sane as he is dangerous to his environment |=--

3 2

Re: [Mailman-Developers] (no subject)
by J C Lawrence Dec. 14, 2000

Dec. 14, 2000

On Thu, 14 Dec 2000 13:42:37 -0800 Chuq Von Rospach <chuqui(a)plaidworks.com> wrote: > At 9:21 PM -0800 12/13/00, J C Lawrence wrote: >> >> http://www.gnqs.org/ >> What's performance like? > Seems okay, but I'm not really stressing it. I'm currently going back and forth on usine an external queue system. Its a nice idea, and there are definite administrative and control benefits, but its also moderately complex, is likely unfamiliar to many/most SysAdms, adds a new level of fault intolerance/dependency to distributed systems and exposes several security concerns that aren't entirely attractive. My current noodling is tending towards implementing a queue-like system using light-weight self-discovering processes that could easily be individually removed and their interconnects replaced by calls to a real queueing system (whatever that may be) on an ad-hoc/per-site basis. The implementation complexity isn't that bad (the documentation would be), but the number of interface abstractions is kinda scary. -- J C Lawrence claw(a)kanga.nu ---------(*) : http://www.kanga.nu/~claw/ --=| A man is as sane as he is dangerous to his environment |=--

2 1

Re: [Mailman-Developers] (no subject)
by J C Lawrence Dec. 14, 2000

Dec. 14, 2000

On Wed, 13 Dec 2000 21:06:38 -0800 Chuq Von Rospach <chuqui(a)plaidworks.com> wrote: > At 6:41 PM -0800 12/12/00, J C Lawrence wrote: >> >> >> Any experience with GNQS? >> >> http://www.gnqs.org/ > it's what I'm using now (I think I called it QPS before. my > bad). I'm pretty happy with it, but Im' not distributing anything > right now -- one machine, a number of queues, solaris and it's > nicely stable. What's performance like? I'm also looking at MQM: http://www.cs.orst.edu/~pancake/ptools/mqm/ Which looks a little more mature and comes with perhaps the most abbreviated BSD-style license I've seen, -- J C Lawrence claw(a)kanga.nu ---------(*) : http://www.kanga.nu/~claw/ --=| A man is as sane as he is dangerous to his environment |=--

2 1

Re: [Mailman-Developers] Re: Slow Performance on semi-large lists
by J C Lawrence Dec. 14, 2000

Dec. 14, 2000

On Thu, 14 Dec 2000 08:22:22 -0700 (MST) D J Atkinson <dj(a)pcisys.net> wrote: >> You are talking about different queue runners. He's talking >> about the queue runner in sendmail, and you are talking about >> Mailman. > No. I was talking about the qrunner in Mailman. With synchronous > DNS sendmail won't accept messages if it can't pick out an MX > host. Those sit in the mailman queue and bog things down. In which case the solution is to configure your MTA to not DNS verify deliveries from localhost. -- J C Lawrence claw(a)kanga.nu ---------(*) : http://www.kanga.nu/~claw/ --=| A man is as sane as he is dangerous to his environment |=--

1 0

Re: [Mailman-Developers] Re: Slow Performance on semi-large lists
by J C Lawrence Dec. 14, 2000

Dec. 14, 2000

On Wed, 13 Dec 2000 20:28:18 -0600 David Champion <dgc(a)uchicago.edu> wrote: > On 2000.12.13, in > <Pine.SOL.4.05.10012131455360.22660-100000(a)babu.pcisys.net>, > "D.J. Atkinson" <dj(a)pcisys.net> wrote: > How about altering qrunner's algorithm to split the queue on > timeout, appending the head of the queue to the tail? You are talking about different queue runners. He's talking about the queue runner in sendmail, and you are talking about Mailman. -- J C Lawrence claw(a)kanga.nu ---------(*) : http://www.kanga.nu/~claw/ --=| A man is as sane as he is dangerous to his environment |=--

2 1

Re: [Mailman-Developers] Access to admin page times out - what to do?
by J C Lawrence Dec. 14, 2000

Dec. 14, 2000

On Thu, 14 Dec 2000 00:47:22 -0500 John A Martin <jam(a)jamux.com> wrote: > Access to a Mailman 2.0 list times out without the browser > displaying anything. This list has more than 450 held messages > amounting to 8MB in mailman/data. How should this be repaired > without disrupting this or other lists? It looks like another > list on the same host with about 50 held messages amounting to > about 700KB can be accessed OK while another with 290 messages > amounting to 3MB cannot. One of two problems: 1) A stale lock file in ~/locks 2) It really is a timeout (unlikely). For the #1 remove the locks in ~/locks. If it really is the latter, move some of the held messages (~/heldmsg-<listname>-*) off to the side, and they try accessing the admin interface again. -- J C Lawrence claw(a)kanga.nu ---------(*) : http://www.kanga.nu/~claw/ --=| A man is as sane as he is dangerous to his environment |=--

1 0