[Mailman-Developers] Requirements for a new archiver

[Brad Knowles]

At 11:01 PM -0500 2003/10/29, J C Lawrence wrote:

>  With a write-once system you don't actually need to ever move anything.

	Depends on how you manage the storage of those large files.  If 
you have an infinitely large filesystem that is guaranteed 100% 
reliable in all possible circumstances, you're right.  Otherwise, you 
might find that the filesystem is getting full and things need to be 
moved around, or you suffer a disk or storage system crash and you 
have to restore from backups, or you use an HSM solution to move 
older files to slower/higher capacity storage, or you have issues 
with too many large files in a single directory and need to implement 
your own directory hashing scheme, etc....

>  Not really.  The percentage of such deleted posts over the lifetime of
>  the store can be generally assumed to be less than 1 in 10^5, and is
>  probably considerably lower, if not in the 1:10^8 range.  Add a simple
>  invalid key semantic and you're done.

	It depends on whether or not the court order allows you to just 
mark things as "deleted" and be done with it.  If they force you to 
actually expunge all copies of that data from your systems, you will 
have to do more work.

>  You're going to need tools when the percentage of such deleted postings
>  is sufficiently high that the cost of the lost free space and its
>  overhead exceeds the cost of managing that free space.  That's not a
>  quick thing.

	True enough, but as you've pointed out, there have been a number 
of implementations of this sort of solution, and you've worked on at 
least a couple yourself.  These sorts of tools should already be 
reasonably well understood and not too difficult to write or "borrow" 
from other sources.

-- 
Brad Knowles, <brad.knowles at skynet.be>

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
     -Benjamin Franklin, Historical Review of Pennsylvania.

GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)