[Mailman-Developers] Requirements for a new archiver

[J C Lawrence]

On Wed, 29 Oct 2003 11:38:33 -0800 
Chuq Von Rospach <chuqui at plaidworks.com> wrote:

> Hint: look at what INN did when they implmented cycbufs.

Aye, its a cute system.

> Effectively, you create 1-N files, or create files as needed. Each
> file is N bytes long, pre-allocated on file creation. When you store
> messages, they're written into the file sequentially (or any other way
> you want. If you want to get into best fit allocations and turn this
> into a malloc() style heap, be my guest).

> Metadata to access the info is then a filename, and an lseek() pointer
> into the file, and # of bytes to read, plus your normal identifying
> info. It's fast, it's efficient use of file pointers, it avoids the
> worst aspects of the unix file system, and I'm amazed nobody ever
> thinks to use it for other purposes (or that it took that long for
> usenet people to discover it, I suggested a simpler variant of it back
> in the 80s and was told inodes are our friends...)

Small caveat: Some modern fileystems make operating on the
one-file-per-message stores extremely efficient.  Admittedly they aren't
in wide cross-platform deployment, but the filesystems and file op
behaviour of today and yesteryear are not quite the same.

> I've even thought of using it as the backing store for a picture
> library. With a nice relational database and a series of these "data
> boxes", I think you have store data in the best and fastest possible
> way...

Some years back I talked to Mike Belshe (used to be at Remarq) about
their storage techniques (I caught him shortly after Critical Path
bought Remarq).  Keying off other LISA papers they segmented their
storage space by object size, customising and configuring each segment
to suit (things like RAID strip size, number of spindles, FS tuning
parameters, etc).  He asserted that the rewards were very significant.

However, these are very large archive problems and are a bit outside of
Mailman's home turf.

-- 
J C Lawrence                
---------(*)                Satan, oscillate my metallic sonatas. 
claw at kanga.nu               He lived as a devil, eh?		  
http://www.kanga.nu/~claw/  Evil is a name of a foeman, as I live.