[Mailman-Developers] Re: Components and pluggablility

[J C Lawrence]

On Thu, 14 Dec 2000 22:37:51 -0500 
Barry A Warsaw <barry@digicool.com> wrote:

> "JCL" == J C Lawrence <claw@kanga.nu> writes:

>>   1) inbound Message arrives at the MLM 2) authentication Do I >
>>   accept it?  3) moderation Does the list accept it?

> Remind me again about the difference between 2 and 3, and why 2 is
> under the purview of the MLM.

>>   4) pending Associate a distribution list with message 5)
>> outbound Send it.  5) bounce Demote the subscriber 7) command
>> A combo of #2, #3, and command processing.

> I'm not totally sold that 4 needs a process boundary, or that 2,
> 3, and 4 aren't part of the same process, structured as interfaces
> in the framework.

Oddly enough I didn't write what I meant to write, and what I meant
also wasn't what my own notes and documents say (which happen to be
more correct than I was).  I'm looking at the following process.
First the base model:

  Configuration of what exactly happens to a message is done by
  dropping scrpts/program in specially named directories (it is
  expected that typically only SymLinks will be dropped (which makes
  the web interface easy -- just creates and moves symlinks about)).

  In general processing of any item will consist of taking that item
  and iteratively passing it to every script in the appropriately
  named directory, invoking those scripts in directory sort order
  (cf SysV init scripts with /etc/rc.# etc).  This makes the web
  config interface easy -- it just varies the numerical prefix on
  the scripts to enforce a processing order.

  Of course each script must follow a defined contract as far
  arguments, IO, return codes, and other behaviour.

The processing sequence for a message:

  Message arrives in inbound queue.

  Message is picked up it is passed on to moderation which consists
  of:

    a) extracting a set of meta data from the message and any
    associated resources and then associating that meta data with
    the message (this is done as an efficiency support ala
    pre-processing (very useful for later template expansion))

    b) Iteratively passing the message thru all the scripts in the
    moderation directory, in order, until either one returns
    non-zero or all scripts have been run.  All scripts returning
    zero means that the message is moved to the pending queue.
    Various non-zero returns have other effects ranging from instant
    deletion, leaving the message in the inbound queue, moving to
    the moderation queue ("holding pen" is more accurate),
    auto-bouncing some sort of reply...

  Some event happens to move a mesasge from the moderation queue to
  the pending queue (or it goes straight there, bypassing
  moderation).

  Message is found in the pending queue and two things happen in
  order:

    a) It is passed iteratively, with its meta data through every
    script in the membership directory.  A non-zero return means
    that the message stays in pending.  The combined output (stdout)
    of all membership scripts forms the distribution list for that
    message.  Upon getting a membership (everything returns zero),
    the resultant list is passed thru a specially anemd script (if
    it exists) for post-processing (dupe removal, VERP instruction
    insertion, domain/MX sorting, etc).  The final distribution list
    is associated with the message much like the meta data already
    is.

    b) The message is passed iteratively, with its meta data and
    distribution list through the contents of the pre-post
    directory.  This does whatever it wants to do (archiving,
    digests, templating, VERP spit outs, whatever).  A non zero
    return will leave the message in the pending queue for a later
    pass.

 Finally, having gained a distribution list and been pre-post
 processed (if there's anything there), the message is moved to the
 outbound queue with its distribution list.

  The mesasge is found in the outbound queue and is handed off to
  whatever the transport is (MTA, NNTP, whatever).  

Now where this gets interesting is that every script and tool along
the path above is free to edit the message, edit the meta data, edit
the distribution list, to cause the current message to be silently
deleted, and/or to crete new or derived messages and to inject them
into any other mesasge queue in the system.

>> So in my view, when Mailman decides that a message can be
>> delivered to a membership list, it's dropped fully formed in an
>> outbound queue.

Yes, with its associated distribution list.

JCL> Not exactly.  It drops a mesasge, any relevant meta data, and a
JCL> distribution list in the outbound queue.  A delivery process
JCL> then takes that and does what it will with them (eg VERP,
JCL> application of templates, etc).

> In Mailman 2.0 the distribution list is also metadata -- it lives
> in the .db file.  Do you see that differently?

I see it as a class of meta data, but specifically one that is bound
to and unique to the message in question, not to the list or the
list config (consider the previous discussion of of nosy lists).  As
such a distribution list is genned for every message, and then
attached to that message for later editing/delivery.

>> These processes are not completely independent of Mailman though,
>> e.g. for handling hard errors at smtp transaction time or URL
>> generation for summary digests.  Some of these can be handled by
>> re-injection into the message queues (i.e. generate a bounce
>> message and stick it in the bounce queue), but some may need an
>> rpc interface.

JCL> Thus the pending queue above -- it allows a mesasge to undergo
JCL> a set of pre-post filters prior to landing in the outbound
JCL> queue. Archiving, digests, all sorts of things can happen at
JCL> that point.

> Maybe I'm taking the term "distribution list" too narrowly in your
> description about.  Do you "list of recipient addresses" or do you
> mean "internal queue routing destinations", or something else?

I don't understand your question.  Given the above, could you
rephrase?

-- 
J C Lawrence                                       claw@kanga.nu
---------(*)                          http://www.kanga.nu/~claw/
--=| A man is as sane as he is dangerous to his environment=--