[Mailman-Developers] Braindump on new archiver

[Thomas Wouters]

On Sat, Jun 16, 2001 at 12:35:08PM -0700, J C Lawrence wrote:
> On Sat, 16 Jun 2001 13:34:02 -0400 (EDT) 
> Dale Newfield <Dale@Newfield.org> wrote:
> 
> > On Sat, 16 Jun 2001, Thomas Wouters wrote:
> >> Maildir works by making every message a file on its own. A
> >> mailbox is a directory with three subdirectories, 'cur', 'new'
> >> and 'tmp'.  Messages in 'new' are unread, messages in 'cur' are
> >> read, and messages in 'tmp' are in transit.
> 
> > What about file system limitations to the maximum number of files
> > in a directory?

> Typically this is an inode limit for the whole filesystem, not a
> directory size issue.  Directory size is a performance issue that
> different filesystems handle variously well (eg Ext2FS badly,
> ReiserFS fairly well).  

Exactly. The NetApps we use have their own filesystem, which doesn't have an
inode limit (the WAFL filesystem is a very cool thing). They also use btree
directory indexes, so large directories are not a problem. I believe
ReiserFS and XFS (SGI's journalling filesystem, which is also ported to the
Linux kernel) also use btree directories.

That said, yes, for lists with a lot of small messages, inodes could become
a problem. I think most UNIX filesystems default to around
8k-bytes-per-inode nowadays, whereas the median size in my mailbox is about
2k. (Trust me.. it's a big enough sample size :-) I forgot to mention this,
but the number of files was one of the reasons not to go for the
directories-with-symlinks for archive data. Still, the problem can be
avoided by removing the maildir mailbox (converting it to an mbox one) when
the month (or week, or year, or whatever) ends and the archive is up to
date. If it turns out to be a problem.

Don't forget that the current system is also pretty inode-heavy, since every
archived message is already a file on its own. And don't think I'm
forgetting about non-Linux, non-NetApp-attached systems :) Our listservers
run BSDI and FreeBSD, both not brilliant in filesystem performance.

-- 
Thomas Wouters <thomas@xs4all.net>

Hi! I'm a .signature virus! copy me into your .signature file to help me spread!