[Mailman-Developers] Speaking about kitties (or archivers)

[Stephen J. Turnbull]

On Tue, Apr 24, 2012 at 7:20 AM, Barry Warsaw <barry at list.org> wrote:
> Thanks for posting this Pierre-Yves!
>
> On Apr 23, 2012, at 08:17 PM, Pierre-Yves Chibon wrote:
>
>>  mm-core (handles the lists themselves) --send emails to archivers-->
>
> Note that the core doesn't *have* to send an email to the archiver.  From the
> core's perspective, the `IArchiver` interface has three functions:
>
>  - add a message to the archive
>  - get a 'permalink' to the message in the archive
>  - get the url to the "top" of the list's archive

Maybe it would be better to call that the archive's "index",
"directory", or "table of contents".  The archive may not be
hierarchically organized.

> The important things are 1) calculating the 'permalink' should not require a
> round-trip with the archiver; 2) the details of adding a message to the
> archiver are irrelevant to the core.

Yes, yes, Yes, YES, yesyesyesyesyes!  I mean, FTW. ;-)

>>The questions are then:
>>- how do we store the emails ?

That's not an appropriate question.  The archive backend will decide
that, and will provide an IArchive function that can be registered
with the core and with front ends.

It would be nice if an IArchive-compatible archive provided a way for
new frontends to discover it, but I guess that's kinda bootstrappy --
if we have that, then why don't we just serve the results over that
channel?

>>- how do we expose the API ?
>>- how to make it such that it becomes easy to extend ? (ie: the stats
>>module wants to read the db, but probably also to store information on
>>it)

No storing, please.  The stats module can keep its own db if it wants
to, and should be using on-line algorithms in any case so the expense
of hitting the archive should be minimal.

> I think we've pretty much come to agreement that the core itself doesn't need
> a full copy of all the messages after it's sent them, but of course, the
> "prototype" archiver could be used to keep a local copy of everything in a
> maildir.  That could be shared at the lower level (maildir) or through some
> kind of API in minikitty.

I don't like the idea of having a "minikitty".  As is probably
apparent (and I apologize for that, my opinion there is really
irrelevant) I am not a fan of turning ML archives into a social
network.  However, I think Pingou and other HyperKitty worker should
just do whatever it is they want to do, and do it right.  If you
really want a solid base set of functionality and only then
extensions, maybe a plugin architecture would be the way to go.  Or
you can specify and implement that base first, then add the
extensions.  (But the mockup already sports UI for the extensions!)

But if that's not really what you want to do, Clearsilver provides a
perfectly good base set for us, and I'll be happy to maintain the
GPL3-ed distro-in-the-Mailman-distro if that's how it needs to be.
*You* do what makes *you* happy.

>>On the other hand, having the archiver-core relying on the same system
>>as the core itself would be nicer from a sysadmin pov.

IMHO, premature optimization.  Among other things, there isn't going
to be a "the" archiver-core.  Mailman should provide "a"
archiver-core, and I think it should be based on maildir (which is
apparently Barry's intuition, too).  Theory and implementation of
maildir are simple and robust, and that allows us to concentrate on
the archiver interface.

>>The challenge will be speed

IMHO, Mailman should not take responsibility for speed of any archiver
backend distributed with Mailman.  It just needs to provide a robust
storage, and the two points Barry mentions above.