On Dec 27, 2014, at 12:42 PM, Stephen J. Turnbull wrote:
Remember, Mailman core is going to be distributing Internet mail. Except in the case where 100.00% of the users are on one host, that's going to be the bottleneck on message processing. Archiving simply does not need to be fast. The archiver can implement LMTP even, although that would be overkill if we didn't already have an LMTP server. The simplest approach would be to simply put the Archive-Permalink header in the message and stream it to the archiver which would parse it out.
Remember too that in MM3, messages only get fed to the registered IArchiver interfaces by a separate archive runner. So they aren't a bottleneck for delivery to the user, but on heavily trafficked sites, they can potential consume a lot of resources if the archiver is local and relatively inefficient.
I tend to agree that a good design for any archiver is to be able to accept messages over an IPC channel. A site may for example want to run the core on one system and HK on another system (e.g. separate VMs perhaps). This would only really be possible if the core can feed HK messages over a configurable IPC. As I mentioned, I think LMTP *could* work, but REST (inside HK) could work too. Aurelien, what do you think?
I'm also using the custom Message class a lot in the tests.
Can't avoid that with Python 2, I guess, but using Message will be *so* much less painful with Python 3.
See my other follow up. I don't think there's much in the py3 branche's Message class that is going to be super helpful to HK. Even the core *could* use the standard email.message.Message class with a couple of utility functions, but it's just a little easier to add a few properties and methods in the subclasses.
In the long run (ie, when nobody who's anybody uses Python 2 at all) I think everybody would be happier if you refactor to keep KittyStore at arm's length from Mailman core.
Agreed, with of course the caveat that we'll need a thin HK IArchiver implementation in the core to generate the permalink and communicate with HK over IPC. Generally we want the permalink to be able to be generated without direction communication with the archiver (see the motivation for X-Message-ID-Hash), but if the core *has* to talk to HK to generate the permalink, then I don't think an LMTP channel will work. In that case REST or some homegrown protocol may be the answer.
Cheers, -Barry