On Thu, Oct 30, 2003 at 05:00:48AM +0100, Brad Knowles wrote:
SGIs XFS on Irix does a pretty good job, with hashed directory structures, and an extent-based journaling filesystem. Regretfully, I don't think that all of these features are fully supported under the Linux version of XFS, and that work has basically ground to a halt with the lay-offs of all the key SGI people who had been working on XFS. Veritas VxFS also does a good job in this area.
[ A cursory google search indicates that hashed dirs, extents, and journalling are all in linux xfs. I can't imagine an unsupported feature making its way into the filesystem that SGI is putting on its latest and greatest systems, but if you know about this, please share ]
In the case of a one-file-per-message approach, my experience with vxfs is that it creates a rather slow filesystem when you get your filesystem to the point of haing with a few hundred thousand small files (lots of wasted space in the extents and I believe, though I may be wrong, that there were lots of metadata lookups through multiple layers of indirections slowing things down).
However reiserfs was built to handle a mix of lots of small files, ala maildir or mh spools.
I'm not too current on current bsd going-ons, but I'd bet that ffs2 has something to offer in this arena, too, since it looks like it almost does extent-based allocation now.
Kirk McKusick and Eric Allman agree with you that this is a proper filesystem problem that should be solved at the filesystem level (at least, that's what they've said to me when I brought this issue up to them), and they feel you should not attempt to solve filesystem problems with "tricks" like INN timecaf/timehash cycbufs.
Err... then to relate this to a prior post, why not just use maildirs on filesystems that are engineered to handle that sort of thing?
However, while that's nice in theory, that doesn't necessarily help us here in the real world.
Unless you are using a filesystem that works for this, right? Like xfs, vxfs, reiserfs, and probably ffs2. I believe that linux's ext3 has support for hashing directories (or soon will - I don't precisely know as I've been focusing on other things)
-Peter
-- The 5 year plan: In five years we'll make up another plan. Or just re-use this one.