At 7:51 PM -0800 12/11/00, J C Lawrence wrote:

ObTheme: All config files should be human readable unless those files are dynamically created and contain data which will be easily and automatically recreated.

ObTheme: All configuration should be possible via the web, even if the system is misconfigured and non-functional. Anything that can NOT be safely reconfigured without breaking the system should not be configurable via the web. (in other words, anything you can change, you should be able to change remotely, unless you can break the ssytem. If you cna break the system, you shouldn't be allowed near it trivially...)

1. Using multiple simultaneous processes/threads to parallelise a given task.

2. Using multiple systems running parallel to parallelise a given task.

3. Using multiple systems, each one dedicated to some portion(s) or sub-set of the overall task (might be all working in parallel on the entire problem (lock contention! failure modes!)).

that's my model perfectly, althought I think 2 and 3 are reversed. it's cleaner architecturally to go to divesting and distributing functionality before 'clustering'. In fact, I'm not sure clustering (which I'll use to term multiple mailman systems running in parallel) implies a system really, really large, when you realize that the primary resource eaters (like delivery) can effectively be infinitely distributed. I'm not sure how big a Mailman system you'd need ot require parallelizing the core process, as long as you can divest off other pieces to a farm that could grow without bounds. So maybe we don't need that next (complicated) step, and make it parallelized and distributable for everything except that core control process, but manage the complexity of that control process to keep everyting out of it exect the absolute necessity.

Observation: MLMs are primarily IO bound devices, and are specifically IO bound on output. Internal processing on mail servers, even given crypto authentication and expensive membership generation processes (eg heavy SQL DB joins etc) are an order of magnitude smaller problem than just getting the outbound mail off the system.

some of that is the MUA's problem, actually, but they get tied together. you don't, for instance, want an MLM who will dump 50K pieces of email an hour into the queues of an MUA that can only process 40K...

But in general, you're correct. Especially if you define DNS delays and SMTP protocol delays caused by the receiving machine to be "output" (grin)

Sites with large numbers of lists with large numbers of members (and presumably large numbers of messages per list) are the pessimal case, and is not one Mailman is currently targeting to solve.

but if you define the distribution capabilities correctly, this case is solved by throwing even more hardware at it, and the owners of this pessimal case presumably have a budget for it. If you see someone tryting to run Sourceforge on a 486 and a 128K DSL line, you laugh at them.

Observation: Traffic bursts are bad. Minimally the MLM should attempt to smooth out delivery rates to a given MTA to be no higher than N messages/time.

The obverse of that is that end-users seriously dislike delays, especially on conversational lists. It turns into the old "user expectation" problem -- it's better to hold ALL mail for 15 minutes so users come to expect it than to normally deliver mail in 2 minutes, except during the worst bulges... But in general, the MLM should deliver as fast as it reasonable can without overloading the MUA, which implies some kind of monitoring setup for the MUA, or some user-controlled throttling system. the latter unfortunately, implies teaching admins how to monitor and adjust, a support issue. The former implies writing an interface for every MTA -- a development AND support issue.

20Million messages sitting in the outbound queue), that the MLM will give the MTA the opportunity to try and react intelligently rather than overwhelming it near instantly with all 20M messages dumped in the MTA spool over 30 seconds while the spool filesystem gags.

I will not make comments about qmail. I will not make comments about qmail. I will be good. I will be good. (grin)

1. Receipt of message by local MTA

1a) passthrough of message via a security wrapper from MTA to list server... (I think it's important we remember that, because we can't lose it, and it involves a layer of passthrough and a process spawning, so it's somewhat heavyweight -- but indispensable)

2. Receipt by list server
3. Approval/editing/moderation
4. Processing of message and emission of any resultant message(s)
5. Delivery of message to MTA for final delivery.

6) delivery of message to non-MTA recipients (the archiver, the 

logging thing, the digester, the bounce processor....)

#1 is significant only because we can can rely on the MTA to distinguish between valif list-related addresses and non-list addresses.

although one thing I've toyed with is to give a subdomain to the MLM, and simply pass everything to it (in sendmail terms, using virtusertable to pass @list.foo.bar to mailman@foo.bar). Then you take the MLM out of having to know what lists exist and administrative needs to keep that interface in sync. The downside is it doesn't fit the design of some users (but that can be fixed by education if we can prove why it's better), and you get into having to handle some MTA functions, such as DSN compatible bounce messages. I've more or less decided than when I rewrite my internal corporate mail list, I'll do that rather than generate alias listings (for, oh, 12,000 groups) and teh hassles and overheads of all that. That'll be especially useful if we do waht I hope, which is set it up so the server has no data at all, but authenticates via LDAP to get list information on demand out of the corporate databases. There are some definite advantages to not knowing whether something exists until the need to know exists -- and as Mailman starts edging towards interfacing to non-Mailman data sources for list information, that ability grows in importance.

6. is the processesing needed to support other functions that act on messages. The idea is that instead of delivering to the MTA, we have a suite of functions that deliver the message ot whatever needs to process it. Those can be asynchronous and don't need to be as timely as (5), and have different enough design needs that I split them out from the MTA delivery (although traditionally, stuff like digests are managed by doing an MTA transfer out of the MLM and back in to a different program...)

It also assumes that these non-delivery things are separate processes from teh act of making them available to those things, to keep (6) lightweight as possible.

Note: Bounce processing and request processing re not detailed at this point as their rate of occurance outside of DoS attacks is comparitively low and are far cheaper than list broadcasts in general.

and besides, they are basically independent, asynchronous processes that don't need to be managed by any of the core logic, other than handing messages into their queue and making sure they stay running. same with, IMHO, storing messages for archives, storing messages for digests, updating archives, processing digests (but the processed digest is fed back into the core logic for delivery), and whatever else we decide it needs to do that isn't part of the core, time-sensitive code base. (in fact, there's no reason why you couldn't have multiple flavors of these things, feeding archives into an mbox, another archiver into mhonarc or pipermail, something that updates the search engine indexes, and text adn mime digesters... by turning them into their own logic streams with their own queues, you effectivley have just made them all plug-in swappable, because you're writing to a queue, and not worrying about what happens once its there. you merely need to make sure it goes in the right queue, in the approved format.

We don't want an over-arching API, or the attempt to solve the entire peoblem with either one hammer, or one sort of hammer.

I like hammers! My thumb doesn't, not since the divorce, at least...

kewl. good stuff here.

-- Chuq Von Rospach - Plaidworks Consulting (mailto:chuqui@plaidworks.com) Apple Mail List Gnome (mailto:chuq@apple.com)

We're visiting the relatives. Cover us.

CVR> ObTheme: All configuration should be possible via the web,
CVR> even if the system is misconfigured and
CVR> non-functional. Anything that can NOT be safely reconfigured
CVR> without breaking the system should not be configurable via
CVR> the web. (in other words, anything you can change, you should
CVR> be able to change remotely, unless you can break the
CVR> ssytem. If you cna break the system, you shouldn't be allowed
CVR> near it trivially...)

Agreed completely. web_page_url comes to mind here. :(

-Barry

At 7:18 PM -0500 12/14/00, Barry A. Warsaw wrote:

Let me change that a little bit. The MLM should /process/ messages as fast as possible, getting them through the moderate-and-munge and into the outbound queue at top speed possible. Once that message is sitting in that outbound queue, it's that queue's runner process that can be configured to throttle, distribute, batch, whatever it takes.

buy that man a beer.

It's not the MLM's problem (not to say it isn't the problem of the simple-minded qrunner script we distribute and enable by default). All I need to do is document the file format for the outbound queue files and site administrators can take it from there.

except I see this as still part of the MLM, since it's the tool doing the MLM->MTA handoff, not part of the MTA itself.

chuq

Chuq Von Rospach - Plaidworks Consulting (mailto:chuqui@plaidworks.com) Apple Mail List Gnome (mailto:chuq@apple.com)

We're visiting the relatives. Cover us.

On Thu, 14 Dec 2000 19:05:25 -0500 Barry A Warsaw <barry@digicool.com> wrote:

I like the idea of process queues, but I don't want to take the federation-of-processes architecture too far. Yes, we want a component architecture, but where I see the process boundaries is at the message queue level.

There are in essence seven queues:

1. inbound Message arrives at the MLM
2. authentication Do I accept it?
3. moderation Does the list accept it?
4. pending Associate a distribution list with message
5. outbound Send it.
6. bounce Demote the subscriber
7. command A combo of #2, #3, and command processing.

There's a possible eighth for OOB stuff like archiving and digests, which I mostly see as a fork off the side of the pending queue.

So in my view, when Mailman decides that a message can be delivered to a membership list, it's dropped fully formed in an outbound queue.

Not exactly. It drops a mesasge, any relevant meta data, and a distribution list in the outbound queue. A delivery process then takes that and does what it will with them (eg VERP, application of templates, etc).

Process pipes...

The file formats are the interface b/w Mailman and the queue runners and should be platform (i.e. Python) independent.

Bingo. This is a point I've invested considerable time into.

That way, I can ship a simple queue runner that takes messages from the outbound queue and hands them off to the smtpd, but /you/ could drop in a different runner process that uses GNQS to distribute load across an infinitely expandable smtpd server farm.

If you continue the same abstraction across all queues and the staging processes of queus, you build something that isn't inherently a queue run-system, it merely looks like one and can in fact be fairly trivially hung off a queue based system (MQM or whatever).

Consider the following setup:

Three machines:

HostA is the primary MX and receives the list mail along with
all mail for the rest of the site.

HostB has a private hole in the firewall and is the only host to
have access to the backing stores for authentication and
membership data.

HostC has a nicely tuned MTA built for outbound processing.

Given a queue bases system supporting that, or something several dozen times more complex, becomes trivial. The problem is in making the architecture that runs on a single host without an external queue manager the same as the system above where different hosts each take responsibility for different queues in the message system.

It can be don, it just requires a little elegance.

[Side note. Here's another reason why I'm keen on ZODB/ZEO as the underlying persistency mechanism for internal Mailman data: I believe we can parallelize the moderate-and-munge part of message processing. Because the ZEO protocols serialize writes at commit time, you could have multiple moderate-and-munge processes running on a server farm and guarantee db consistency across them.

There are problems with this due to the fact that external transactions (such as SMTP sends) are asynchonous and not nested in ZODB transactions.

...

...

...

...

...

"JCL" == J C Lawrence <claw@kanga.nu> writes: "CVR" == Chuq Von Rospach <chuqui@plaidworks.com> writes:

These processes are not completely independent of Mailman though, e.g. for handling hard errors at smtp transaction time or URL generation for summary digests. Some of these can be handled by re-injection into the message queues (i.e. generate a bounce message and stick it in the bounce queue), but some may need an rpc interface.

Thus the pending queue above -- it allows a mesasge to undergo a set of pre-post filters prior to landing in the outbound queue. Archiving, digests, all sorts of things can happen at that point.

-- J C Lawrence claw@kanga.nu ---------(*) http://www.kanga.nu/~claw/ --=| A man is as sane as he is dangerous to his environment |=--

On Thu, 14 Dec 2000 22:37:51 -0500 Barry A Warsaw <barry@digicool.com> wrote:

"JCL" == J C Lawrence <claw@kanga.nu> writes:

...

1. inbound Message arrives at the MLM 2) authentication Do I > accept it? 3) moderation Does the list accept it?

Remind me again about the difference between 2 and 3, and why 2 is under the purview of the MLM.

...

4. pending Associate a distribution list with message 5) outbound Send it. 5) bounce Demote the subscriber 7) command A combo of #2, #3, and command processing.

I'm not totally sold that 4 needs a process boundary, or that 2, 3, and 4 aren't part of the same process, structured as interfaces in the framework.

Oddly enough I didn't write what I meant to write, and what I meant also wasn't what my own notes and documents say (which happen to be more correct than I was). I'm looking at the following process. First the base model:

Configuration of what exactly happens to a message is done by dropping scrpts/program in specially named directories (it is expected that typically only SymLinks will be dropped (which makes the web interface easy -- just creates and moves symlinks about)).

In general processing of any item will consist of taking that item and iteratively passing it to every script in the appropriately named directory, invoking those scripts in directory sort order (cf SysV init scripts with /etc/rc.# etc). This makes the web config interface easy -- it just varies the numerical prefix on the scripts to enforce a processing order.

Of course each script must follow a defined contract as far arguments, IO, return codes, and other behaviour.

The processing sequence for a message:

Message arrives in inbound queue.

Message is picked up it is passed on to moderation which consists of:

a) extracting a set of meta data from the message and any
associated resources and then associating that meta data with
the message (this is done as an efficiency support ala
pre-processing (very useful for later template expansion))

b) Iteratively passing the message thru all the scripts in the
moderation directory, in order, until either one returns
non-zero or all scripts have been run.  All scripts returning
zero means that the message is moved to the pending queue.
Various non-zero returns have other effects ranging from instant
deletion, leaving the message in the inbound queue, moving to
the moderation queue ("holding pen" is more accurate),
auto-bouncing some sort of reply...

Some event happens to move a mesasge from the moderation queue to the pending queue (or it goes straight there, bypassing moderation).

Message is found in the pending queue and two things happen in order:

a) It is passed iteratively, with its meta data through every
script in the membership directory.  A non-zero return means
that the message stays in pending.  The combined output (stdout)
of all membership scripts forms the distribution list for that
message.  Upon getting a membership (everything returns zero),
the resultant list is passed thru a specially anemd script (if
it exists) for post-processing (dupe removal, VERP instruction
insertion, domain/MX sorting, etc).  The final distribution list
is associated with the message much like the meta data already
is.

b) The message is passed iteratively, with its meta data and
distribution list through the contents of the pre-post
directory.  This does whatever it wants to do (archiving,
digests, templating, VERP spit outs, whatever).  A non zero
return will leave the message in the pending queue for a later
pass.

Finally, having gained a distribution list and been pre-post processed (if there's anything there), the message is moved to the outbound queue with its distribution list.

The mesasge is found in the outbound queue and is handed off to whatever the transport is (MTA, NNTP, whatever).

Now where this gets interesting is that every script and tool along the path above is free to edit the message, edit the meta data, edit the distribution list, to cause the current message to be silently deleted, and/or to crete new or derived messages and to inject them into any other mesasge queue in the system.

...

So in my view, when Mailman decides that a message can be delivered to a membership list, it's dropped fully formed in an outbound queue.

Yes, with its associated distribution list.

JCL> Not exactly. It drops a mesasge, any relevant meta data, and a JCL> distribution list in the outbound queue. A delivery process JCL> then takes that and does what it will with them (eg VERP, JCL> application of templates, etc).

In Mailman 2.0 the distribution list is also metadata -- it lives in the .db file. Do you see that differently?

I see it as a class of meta data, but specifically one that is bound to and unique to the message in question, not to the list or the list config (consider the previous discussion of of nosy lists). As such a distribution list is genned for every message, and then attached to that message for later editing/delivery.

...

These processes are not completely independent of Mailman though, e.g. for handling hard errors at smtp transaction time or URL generation for summary digests. Some of these can be handled by re-injection into the message queues (i.e. generate a bounce message and stick it in the bounce queue), but some may need an rpc interface.

JCL> Thus the pending queue above -- it allows a mesasge to undergo JCL> a set of pre-post filters prior to landing in the outbound JCL> queue. Archiving, digests, all sorts of things can happen at JCL> that point.

Maybe I'm taking the term "distribution list" too narrowly in your description about. Do you "list of recipient addresses" or do you mean "internal queue routing destinations", or something else?

I don't understand your question. Given the above, could you rephrase?

-- J C Lawrence claw@kanga.nu ---------(*) http://www.kanga.nu/~claw/ --=| A man is as sane as he is dangerous to his environment=--

On Fri, 15 Dec 2000 01:17:58 -0500 Barry A Warsaw <barry@digicool.com> wrote:

...

...

...

...

...

"JCL" == J C Lawrence <claw@kanga.nu> writes:

JCL> Configuration of what exactly happens to a message is done by JCL> dropping scrpts/program in specially named directories (it is JCL> expected that typically only SymLinks will be dropped (which JCL> makes the web interface easy -- just creates and moves symlinks JCL> about)).

At a high level, what you're describing is a generalization of MM2's message handler pipeline. In that respect, I'm in total agreement. It's a nice touch to have separate pipelines between each queue boundary, with return codes directing the machinery as to the future disposition of the message.

<nod>

But I don't like the choice of separate scripts/programs as the basic components of this pipeline. Let me rotate that just a few degrees to the left, squint my eyes, and change the scripts to Python modules, and return codes to return values or exceptions. Then I'm sold, and I think you can do everything you want (including using separate scripts if you want), and are more efficient for the common situations.

Fair dinkum, given the below caveat.

First, we don't need to mess with symlinks to make processing order configurable. We simply change the order of entries in a sequence (read: Python list). It's a trivial matter to allow list admins to select the names of the components they want, the order, etc. and to keep this information on a per-list basis.

<nod>

Actually, the web interface I imagine doesn't give list admins configurability at that fine a grain. Instead, a site administrator can set up list "styles" or patterns, one of which includes canned filter sets; i.e. predefined component orderings created, managed, named, and made available by the site administrator.

I'll discuss this later below (it comes down to a multi-level list setup/definition deal).

Second, it's more efficient because I imagine Mailman 3.0 will be largely a long running server process, so modules need only be imported once as the system warms up.

I have been working specifically on the assumption that it will not be a long running process, and that instead it will be automated by cron starting up a helper app periodically which will fork an appropriate number of sub-processes to run the various queues (with simple checks to make sure that the total number of queue running processes of a given type on a given host don't exceed some configured value. The base reason for this assumption is that it makes the queue processing more analagous to traditional queue managers, allowing the potential transition from Mailman's internal (cron based) automation to a real queue manager semi-transparent. The assumption in this was that the tool used to move a message between queues was an external explicitly standa-alone script. The supporting reason being a that simple replacement of that script by something that called the appropriate queue management tools for the queue manager de jour would allow the removal of the Mailman "listmom" and its replacement by the queue manager, be it LSF, QPS MQM, GNU queu, or something else.

This is what I mean by "light weight self-discovering processes that behave in a queue-like manner". The processes are small and light. They figure out what needs to be done locally per their host-specific configurations, and then do that in a queue-like manner.

What's this host-specific stuff? More later.

ObNote: There actually need to be seperate and discrete tools for both moving a given message into a specific queue (ie different tools for inbound, pending, oubound, etc) and different tools for injecting messages (that didn't exist before) into each queu. Doing it this way allows a site to roll part of the system to a queue manager and allow the rest to remain default. This could be done by a single tool linked to different names, or passing the queue name as an argument and allowing an easy call-out as above to a module-wrapped external tool.

Even re-importing in a one-shot architecture will be more efficient than starting and stopping scripts all the time, because of the way Python modules cache their bytecodes (pyc files).

I'm sold given the comment on the next paragraph.

Third, you can still do separate scripts/programs if you want or need. Say there's something you can only do by writing a separate Java program to interface with your corporate backend Subject: header munger. You should be able to easily write a pipeline module that hides all that in the implementation. You can even design your own efficient backend IPC protocol to talk to whatever external resource you need to talk to. I contend that the overhead and complexity of forking off scripts, waiting for their exit codes, process management, etc. etc. just isn't necessary in the common case, where 5 or 50 lines of Python will do the job nicely.

Then we should provide a template python module that accepts the approriate arguments passes them the a template external program, and grabs its stdout and RC. Configuring users could/would then merely take this, rename it, and customise it and roll it in transparently.

Fourth, yes, maybe it's a little harder to write these components in Perl, bash, Icon or whatever. That doesn't bother me. I'm not going to make it impossible, and in fact, I think if that if that were to become widely necessary, a generic process-forking module could be written and distributed.

Umm, yeah. Shame nobody thought of that.

I don't think this is very far afield of what your describing, and it has performance and architectural benefits IMO. We still formalize the interface that pipeline modules must conform to, probably spelled like a Python class definition, with elaborations accomplished through subclassing.

Bingo.

Does this work for you? Is there something a script/program component model gives you that the class/module approach does not?

Not inherently given a mathod for easy call outs as mentioned above.

Now onto the business of the host-specific configurations, what I've been looking at is something as below. The global list configuration consists of the following directories and files:

~/cgi-bin/* (MLM CGIs) ~/config (global MLM config) ~/config.force (global MLM config (can't change) ~/config.<hostname> (config specifics for this host) ~/scripts/* (all the tools and scripts that do things) ~/scripts/member/* (membership scripts) ~/scripts/moderate/* (moderation scripts) ~/scripts/pre-post/* (scripts run before posting) ~/inbound/* (messages awaiting processing by the MLM) ~/outbound/* (messages to be sent my the MLM) ~/services/* (the processes that actually run mailman) ~/templates/* (well, templates) ~/groups/ (groups of list configs) ~/groups/default/ (There has to be a default) ~/groups/default/... (Basically a full duplicate of the root setup, mostly done as symlinks) ~/groups/<groupname>/config (deltas from ~/config) ...etc

Then on the list base:

~lists/<listname>/config (list config as deltas from group config) ~lists/<listname>/group (symlink to ~/groups/<something>) ~lists/<listname>/moderate/* (messages held for moderation) ~lists/<listname>/pending/* (messages waiting to be processed) ~lists/<listname>/scripts/* (what does all the work)

The assumption so far is that the queues were represented as discrete files on disk, much like the current held messages in v2, with file names mapping to address/function of the message (ie list name plus command/request/post/bounce/reject/something) with filename extentions for various meta data sets, etc, (this helps keep things human readable). There are aspects of this I'm not happy with (eg for distribution lists on account of size (consider a 1M member list).

The idea is that the config files are simple collections of variable assignments much like the current Defaults.py or mm_cfg.py. Further, they are read in the following order:

~/config ~/groups/<groupID>/config ~/lists/<listname>/config ~/groups/<groupID>/config.force ~/config.force

Where the web interface would present the options that are locked by a higher level config (ie in a force file) as present but unconfigurable.

Now, the next thing, outside of populating the initial root directory with files (such as the various configures python modules etc), everything else gets gone from the web. One account has access to the root and can create and edit groups etc. Another account has access to the list configs, and then of course there are mdoerator-only accounts. All of this of course gets exported thru the standard authentication methods so thta it can get replaced by <whatever>.

-- J C Lawrence claw@kanga.nu ---------(*) http://www.kanga.nu/~claw/ --=| A man is as sane as he is dangerous to his environment |=--