Stephen J. Turnbull <stephen@xemacs.org> wrote:

On Wed, Mar 28, 2012 at 4:21 AM, Terri Oda <terri@zone12.com> wrote:

...

...

Looks like archiver for mm3 is still in development stage. As far as I understand searcher depends on the srchiver, right? Not completely but it somewhat depends on archiver. I am not sure if searcher can be implemented without archiver. If possible I can implement for mm3 also.

Searcher and archiver are interdependent *if* we want to share caches and data stores, which we probably do for any installation with larger archives where storing 2 copies vs 4 of each message would make a difference.  Plus, many archive views may be basically searches "messages in the last month" "messages which are replies to messageid $foo" etc.

Actually, as far as I can see, the summary/search/index/retrieval functions depend only on the API for the message store. If you want, you can split this into the database layer and a presentation layer, of course. However, the database layer is surely going to have its own schema optimized for the kinds of retrieval its designer considers important. If the designer emphasizes threads, however, she is *not* going to try to store messages in thread order or anything like that. Rather, any reasonable store will be message-ID-addressable.

Right. UpLib has a 'message-store', which the threading code interacts with to generate threads as data referring to document IDs. The message-store API can take both message-IDs or UpLib document IDs and resolve them.

The only tricky issue is that we *do* have to worry about message-ID collisions of truly different messages and about messages without message IDs, especially for converted historical archives. So the API needs to be able to deal with these issues, probably by returning a set or sequence of messages.

Right. UpLib takes a message and creates multiple 'documents' (one for the message, and one for each attachment), each of which have their own unique 'doc ID', the assigned UpLib ID. In addition, the email is assigned a 'mail-guid', which is calculated from some of the header information and may also include the doc ID. The metadata of each attachment refers back to the 'mail-guid' of the message it was part of.

Message-ID, mail-guid, and document ID are all separately indexed for each document, and any of them can be searched on.

Oh, and we probably ought to have a more general notion of retrievable "object" rather than just messages, as some archive/retrieval backends may store some types of MIME part separately. Hopefully these would be presented to us as MIME parts with external bodies and content IDs.

Here's how I do it. In UpLib, a multipart email is analyzed into a message plus possible attachments. The parts that are the 'message' are unified and presented as a document. The parts that are attachments are broken out and processed as independent documents, iconified links to which are then put back into the 'message' document.

See http://uplib.parc.com/misc/noguchi.png for an example of the UpLib reader, ReadUp, showing a plain-text email with an attached PDF file. Most of the things that can be links there (like "Reply" or the email addresses or my name or the URL or the attachment icon and name) are in fact links.

And that's all we want to say about the archiver and the associated message-retrieval logic, I think. (In fact, it occurs to me that maybe we should say "RFC 3501" and be done with it. I don't mean that we necessarily implement IMAP protocol per se, but some subset of its functionality probably is what we need from an archiver.)

Yes, there's an IMAP server that runs in UpLib, and can export any document via IMAP (including archived email). Though it currently doesn't scale well; I need to re-write it with Tornado, too.

Bill