I'm working on a design proposal for Mailman v3 and have arrived at a few questions:
Is there a GPL distributed queue processing system ala IBM's MQ about? I've not been able to find one. The specific problem I'm attempting to solve in a performant manner is:
How does a given delivery process on a given system obtain access to a message it is qualified to deliver with a minimum of lock contention or other overhead while preventing any other process from attempting to also access the message it has acquired from that point forward, but while also allowing another process to recover that message should the original process or host fail?
Note that the queue directory and the delivery process may be on different systems (eg NFS) and that there may be multiple competing queue delivery processes on any given system as well as across multiple systems.
NB Needs to be GPL as Mailman is a GNU project.
How much interest is there in optionally supporting VERP? I've no interest in making Mailman v3 VERP-only, but I'd like to see it as an option, and specifically, an option which can be turned on and off dynamically on a configurable percentage of posts basis (eg N% of the messages exploded from an arbitrary list post are VERPed). I've got the basics worked out (assuming the local MTA supports plus+addressing), but some of the smaller bits are fiddly.
The current design requires messages to have MessageIDs which are unique within the range of MessageIDs currently within the new Mailman queue system (I'm using MessageIDs as the tag oc choice for moderation by email). To do this, I'm proposing that Mailman do a couple new things:
Insert MessageID headers with created values in messages that don't contain any MessageID.
Detect collisions within its rather small/arbitrary window, and auto-discard/reject messages subsequent messages with a duplicate MessageID. This would not a rigorous dupe check, but would only check for dupes against the messages already in the Mailman queue (ie received and not yet sent back out).
Any problems in messing with MessageIDs in this way?
I'm not keen on MessageID re-writing, however its worth noting
that the above two rules would only come into effect when the
mail system has already broken down at some level (MUA emitting
non-unique or no IDs, mail dupes, etc).4) While it seems a subtlesmall point, its bugging me. Given user account support, and messages to a given user bouncing, should that user be unsubscribed from only that list, or from all lists at that site? Where this is actually bugging me most is for virtual domains and whether or not lists in a virtual domains should be transparent or opaque to a bounce on a list in a different virtual domain?
The admin in me says, "Hell yes!" The commercial reality nut
in me demurrs (think about list hosting for small companies and
their PR image given transparent virtual hosting).For those interested the basic model is built upon arbitrary process queues and pipes. Also please remember I'm coming up with a proposal, not a mandate (ie Barry etc haven't seen or commented on this yet).
-- J C Lawrence claw@kanga.nu ---------(*) : http://www.kanga.nu/~claw/ --=| A man is as sane as he is dangerous to his environment |=--