Mailman 3 October 2003 - Mailman-Developers

Re: [Mailman-Developers] Requirements for a new archiver
by J C Lawrence Oct. 30, 2003

Oct. 30, 2003

On Wed, 29 Oct 2003 20:59:56 -0800 Chuq Von Rospach <chuqui(a)plaidworks.com> wrote: > On Oct 29, 2003, at 8:53 PM, Barry Warsaw wrote: >> Chuq, do you think it would be feasible for Mailman to try to handle >> that 90% itself, and then only hand-off to a Real MTA when it runs >> into trouble with the other 10% -- assuming it could know when it >> runs into trouble. > I think you have enough on your plate to not re-invent what others > have already done pretty well. When you run out of features to > implement, then think about this. Not until. Seconded, in spades. -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

1 0

Re: [Mailman-Developers] Requirements for a new archiver
by J C Lawrence Oct. 30, 2003

Oct. 30, 2003

On Wed, 29 Oct 2003 23:27:46 -0500 Barry Warsaw <barry(a)python.org> wrote: > On Wed, 2003-10-29 at 22:06, Chuq Von Rospach wrote: > We should just throw everything into a ZODB FileStorage Data.fs file, > and let it grow to gigs in size <1/2 wink>. There are good reasons I use DirectoryStorage: $ find /var/lib/zope/instance/default/var/Data_fs_dir -type f | wc -l 499266 Lotsa little teensy files! -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

2 1

Re: [Mailman-Developers] Requirements for a new archiver
by J C Lawrence Oct. 30, 2003

Oct. 30, 2003

On Wed, 29 Oct 2003 20:37:49 -0800 Chuq Von Rospach <chuqui(a)plaidworks.com> wrote: > On Oct 29, 2003, at 8:27 PM, Barry Warsaw wrote: > and don't forget, a single monolithic storage file gets backed up > fully every time you change it. The guy in charge of buying tapes to > back up your system just screamed in agony, since there's no > possibility of an incremental backup for what is 99.9999999% static > data. Ha! So just why do you think I moved off FileStorage for Data.fs? That said there's some value in getting a versioning data store with rollback support for list configs. The data volume isn't huge, but it is highly sensitive. I'd also like to see flat text logging of all configuration changes in addition to moderation activity. It would save the help and support desks a lot of hurt. -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

2 1

Re: [Mailman-Developers] Requirements for a new archiver
by J C Lawrence Oct. 30, 2003

Oct. 30, 2003

On Thu, 30 Oct 2003 04:45:37 +0100 Brad Knowles <brad.knowles(a)skynet.be> wrote: > At 10:27 PM -0500 2003/10/29, J C Lawrence wrote: >> Actually the two cases are considerably different. In the delete >> case I have to do pool management, with some eye toward fragmentation >> control and optimisations of average latency for free heap searches, >> as well as heap integrity audits. In the write-only case I just >> build on the end and need pay no mind to prior data once it is >> allocated. > Not really. You still have to maintain all the indexes, make sure > that if things get moved around that all the links get updated, > etc.... With a write-once system you don't actually need to ever move anything. At its core it is: Open one file, repetitively append to end until file size exceeds size N, create new file, repeat. You can do object size clustering across files or other optimisation techniques, but the basic pattern remains the same. For the few cases you have to support delete you either just NULL the byte stream for the pointed-to object, or you invalidate the key. As the frequency and number of such deletes is infinitesimal, they require no special management complexity. You can afford to just swallow the lost free space as the cost of attempting to manage it is simply never rewarded. > True, you don't have to worry about fragementation control or other > more complex aspects of heap management, but that's a further cost > savings over other techniques and not a "drawback" to using this > technique for this purpose. True. I'm not lableing it a drawback, just a boon of dubious advantage. > Now, if you want to consider what would happen to you if the > Scientologists ever came after you, or if you had court orders to > remove postings that linked to bomb-making instructions, you'd > probably want to keep all those other tools related to heap management > around anyway. Not really. The percentage of such deleted posts over the lifetime of the store can be generally assumed to be less than 1 in 10^5, and is probably considerably lower, if not in the 1:10^8 range. Add a simple invalid key semantic and you're done. Caveat: Continual addition and deletion of SPAM from an archive would change this balance. > They'd be less likely to be used, but at least you wouldn't have to > take the entire site down while you went and wrote the tools from > scratch to handle a situation that you had not foreseen. You're going to need tools when the percentage of such deleted postings is sufficiently high that the cost of the lost free space and its overhead exceeds the cost of managing that free space. That's not a quick thing. -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

2 1

Re: [Mailman-Developers] Requirements for a new archiver
by J C Lawrence Oct. 30, 2003

Oct. 30, 2003

On Wed, 29 Oct 2003 11:38:33 -0800 Chuq Von Rospach <chuqui(a)plaidworks.com> wrote: > Hint: look at what INN did when they implmented cycbufs. Aye, its a cute system. > Effectively, you create 1-N files, or create files as needed. Each > file is N bytes long, pre-allocated on file creation. When you store > messages, they're written into the file sequentially (or any other way > you want. If you want to get into best fit allocations and turn this > into a malloc() style heap, be my guest). > Metadata to access the info is then a filename, and an lseek() pointer > into the file, and # of bytes to read, plus your normal identifying > info. It's fast, it's efficient use of file pointers, it avoids the > worst aspects of the unix file system, and I'm amazed nobody ever > thinks to use it for other purposes (or that it took that long for > usenet people to discover it, I suggested a simpler variant of it back > in the 80s and was told inodes are our friends...) Small caveat: Some modern fileystems make operating on the one-file-per-message stores extremely efficient. Admittedly they aren't in wide cross-platform deployment, but the filesystems and file op behaviour of today and yesteryear are not quite the same. > I've even thought of using it as the backing store for a picture > library. With a nice relational database and a series of these "data > boxes", I think you have store data in the best and fastest possible > way... Some years back I talked to Mike Belshe (used to be at Remarq) about their storage techniques (I caught him shortly after Critical Path bought Remarq). Keying off other LISA papers they segmented their storage space by object size, customising and configuring each segment to suit (things like RAID strip size, number of spindles, FS tuning parameters, etc). He asserted that the rewards were very significant. However, these are very large archive problems and are a bit outside of Mailman's home turf. -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

4 3

Re: [Mailman-Developers] Requirements for a new archiver
by J C Lawrence Oct. 30, 2003

Oct. 30, 2003

On Thu, 30 Oct 2003 04:08:45 +0100 Brad Knowles <brad.knowles(a)skynet.be> wrote: > At 9:22 PM -0500 2003/10/29, J C Lawrence wrote: >> cycbufs implement a filesystem-based heap with pool semantics. >> (There's a fair bit of literature on that space in the OS and >> application realm) As such they are specifically tuned for the case >> where the number of calls to malloc() are of a similar magnitude to >> the calls to free(). This makes sense in a netnews world where news >> articles expire regularly, and in general as much data is added to >> the spool as is removed from it. > So long as the calls to malloc() are kept reasonably small (which is > typically true in this case), it shouldn't matter whether or not there > are any free() calls. I've written several heap managers including several pool based systems as well as other sorts of custom allocators. There are a great many simplifications that come along with the write-once approach, especially in terms of the trade-offs between allocation expense and free space management. > Yes, you slowly build up more disk space in utilization, but all > archive solutions will have the same problem, and this solution will > scale as well as, or better than, any other that I know of. Which is not exactly my point. cycbufs are a useful technique to be sure, much as Chuq has discussed from a management perspective. My point is more that I don't see that they add anything essentially different to the storage space in terms of storage semantics. You get a higher rate of file handle re-use, a more friendly filesystem behaviour for older filesystem designs (pleasant optimisations), but exactly the same single key -> byte stream without adding any more interesting verbs of transforms to the solution space. This is not a Bad Thing, just not something that seems applicable at this state in the design discussion. First come ontology and semantics, then comes implementation. > Consider the case where you are trying to store all news articles that > have ever been posted -- not really much difference. Actually the two cases are considerably different. In the delete case I have to do pool management, with some eye toward fragmentation control and optimisations of average latency for free heap searches, as well as heap integrity audits. In the write-only case I just build on the end and need pay no mind to prior data once it is allocated. In both cases I have to do predictive work on the distribution of allocation sizes, but that's far cheaper in the write-only case as the multiple-pool search overhead can be entirely skipped. There's a considerable difference in complexity between the two. -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

2 1

Re: [Mailman-Developers] Requirements for a new archiver
by J C Lawrence Oct. 30, 2003

Oct. 30, 2003

On Wed, 29 Oct 2003 13:59:06 -0800 Peter C Norton <spacey-mailman(a)lenin.nu> wrote: > On Wed, Oct 29, 2003 at 10:14:52PM +0100, Brad Knowles wrote: >> I don't believe that there are any databases in existence that >> "... can be made more suitable to the problem." > In theory you can add data types to postgresql. Not that I've done it > myself, but its been done. True, but that doesn't answer the question of whether an RDBMS is a good storage tool for messages. I spent a couple months of spare time last year building an archiving system I liked atop PostgresQL using fully decomposed SQL structures for all the message bits. It was not a pretty exercise, and the results were worse. Brad makes excellent points in his comments on poor BLOB support, the value if DBs for meta-data, and disaster recovery ease. -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

4 6

Re: [Mailman-Developers] Requirements for a new archiver
by J C Lawrence Oct. 30, 2003

Oct. 30, 2003

On Wed, 29 Oct 2003 16:12:50 -0800 Chuq Von Rospach <chuqui(a)plaidworks.com> wrote: > On Oct 29, 2003, at 2:28 PM, Peter C. Norton wrote: >> I may not have made it clear, but I'm focusing on the metadata. Once >> you've parsed rfc822/2822, then it may become easier to have things >> in the database that can manipulate those types. I.e. to do be able >> to do simple searches for a property of given arbitrary headers (w/o >> having to have a database schema that consists of a few known headers >> and "others" which you then have to treat as a blob or as text). > my only real worry is that from what I've seen, 99.99% of the time, > the user is going to want content searches. header stuff is fine, but > of really low priority in the scheme of things (necessary to put > useful things together, meaningless if you can't content/context > search in fulltext). I see two needs, for significantly different populations. The first wants a browsing interface with keyed and indexed by date, thread, and author. The second wands full text search with rapid location and retrieval of matching messages. Often a single user will move between the access methods, reading by thread, bouncing over to a search, then reading all an author has written that match, then searching again, etc. As such two distinct sets of indexes seem called for: full text and message meta-data. > that's why I'm leaning, blob issues or no, towards full-text storage > in MySQL 4. Because if you can't easily chop up the message body > content and find the messages you want to deal with, elegant storage > of the headers is irrelevant... True. However, but this seems to conflate two distinct problems. If you're going to do unindexed searches then this makes sense, however except for minimal cases that's an interesting space. It scales like crap and has an even worse feature set. It is more interesting to split storage and indexing into distinct solution designs, and to build or pick something tailored for that smaller problem. That way you don't do full text searching, you do full text indexing and then search the indexes. > I think you need that, too. But until you get a reasonable context > search for the message body, designing the rest is silly. Is searching message bodies really interesting, or is building indexes of message bodies such that you can later search those indexes the actually interesting point? > And it seems to me there are few better methods than dumping the text > into MySQL and letting it do the work. Compromises, tradeoffs and etc > notwithstanding... How does MySQL help you in building language-sensitive rapid response indexes of large text blobs? -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

2 1

Re: [Mailman-Developers] Requirements for a new archiver
by J C Lawrence Oct. 30, 2003

Oct. 30, 2003

On Thu, 30 Oct 2003 02:52:52 +0100 Brad Knowles <brad.knowles(a)skynet.be> wrote: > I think you could do full word indexing per message, and then store > that index information in the database. Searching for phrases would > require hitting the message bodies themselves, but searching for > individual words could be done on indexed fields. Consider an index which records not just the fact of a token's presence in an entity, but also the offsets at which it occurs within the entity. Searching for phrases then consists of searching for objects which satisfy the boolean "X AND Y", as well as the smaller clause "offset(X) + length (X) + 1|2 == offset (Y)". Larger phrases extend the equivalence language linearly, tho they create exponential search costs. -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

1 0

Re: [Mailman-Developers] Requirements for a new archiver
by J C Lawrence Oct. 29, 2003

Oct. 29, 2003

On Wed, 29 Oct 2003 16:29:09 +0100 Brad Knowles <brad.knowles(a)skynet.be> wrote: > At 12:41 AM -0500 2003/10/28, J C Lawrence wrote: >> Quite, this is how/why NNTP uses Message-IDs are unique indexing >> qualifiers. > Problem is that client-assigned message-ids are not guaranteed > unique. Right, and that was the point. If we do nothing to Message IDs we don't change external behaviour. If we use a netnews backing store for the archives and we don't dick with the message IDs we run the risk of some messages never reaching the archives. If we use a netnews backing store and dick with message IDs we can offer various levels of guarantee that messages reach the archives, and of pissing off users because we messed with the Message IDs. As always, you get to pick. > Everything else could quite feasibly collide, and you'd wind up with > multiple non-unique message-ids. In which case the many people currently using ID-based dupe collapsing (eg default Exchange config) will lose messages, and the archives will lose messages....OR...we offer some level of guarantee (see yesterday's discussion) with the matching trade-offs. > You need a guaranteed unique id to be used as a primary index field. "Need" is a strong word. Its very deployment and use-case sensitive. There are a large number of cases where I'm content to rest on the assurance that the Message IDs arriving at my lists will always be unique. There are also a large number of cases where I'm not willing to make that assessment, as well as a large number of cases where I'm willing to simply discard anu duplicated Message ID messages at the archiver level. Similarly, there are cases where re-writing the Message IDs in any form is significantly troubling, and cases where its not. "Need"? No. It is a deployment choice with easily understood ramifications. -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

3 2