[Mailman-Developers] Requirements for a new archiver
davidb at chelsea.net
Wed Oct 29 16:05:43 EST 2003
A few comments from the peanut gallery:
1. grep, perl, awk, vim, emacs, cp, mv, tar, etc. don't work too well on
SQL databases, but are nice for admins who want quick and dirty
searching, move mailman to a new machine, or need to poke around.
2. third-party add-ons make it that much harder to install. If I have to
set up a Mysql or Postgres database to use Mailman, it's a step that
will put off people who don't already have it going.
On Wed, 29 Oct 2003, J C Lawrence wrote:
> On Wed, 29 Oct 2003 11:38:33 -0800
> Chuq Von Rospach <chuqui at plaidworks.com> wrote:
> > Hint: look at what INN did when they implmented cycbufs.
> Aye, its a cute system.
> > Effectively, you create 1-N files, or create files as needed. Each
> > file is N bytes long, pre-allocated on file creation. When you store
> > messages, they're written into the file sequentially (or any other way
> > you want. If you want to get into best fit allocations and turn this
> > into a malloc() style heap, be my guest).
> > Metadata to access the info is then a filename, and an lseek() pointer
> > into the file, and # of bytes to read, plus your normal identifying
> > info. It's fast, it's efficient use of file pointers, it avoids the
> > worst aspects of the unix file system, and I'm amazed nobody ever
> > thinks to use it for other purposes (or that it took that long for
> > usenet people to discover it, I suggested a simpler variant of it back
> > in the 80s and was told inodes are our friends...)
> Small caveat: Some modern fileystems make operating on the
> one-file-per-message stores extremely efficient. Admittedly they aren't
> in wide cross-platform deployment, but the filesystems and file op
> behaviour of today and yesteryear are not quite the same.
> > I've even thought of using it as the backing store for a picture
> > library. With a nice relational database and a series of these "data
> > boxes", I think you have store data in the best and fastest possible
> > way...
> Some years back I talked to Mike Belshe (used to be at Remarq) about
> their storage techniques (I caught him shortly after Critical Path
> bought Remarq). Keying off other LISA papers they segmented their
> storage space by object size, customising and configuring each segment
> to suit (things like RAID strip size, number of spindles, FS tuning
> parameters, etc). He asserted that the rewards were very significant.
> However, these are very large archive problems and are a bit outside of
> Mailman's home turf.
> J C Lawrence
> ---------(*) Satan, oscillate my metallic sonatas.
> claw at kanga.nu He lived as a devil, eh?
> http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.
> Mailman-Developers mailing list
> Mailman-Developers at python.org
More information about the Mailman-Developers