[Mailman-Developers] Requirements for a new archiver

[J C Lawrence]

On Thu, 30 Oct 2003 05:59:48 +0100 
Brad Knowles <brad.knowles at skynet.be> wrote:
> At 11:43 PM -0500 2003/10/29, J C Lawrence wrote:

>> Not a whole lot more complexity.  You're just invalidating the
>> pointed-to data as well as the key.  You're still not doing free
>> space management.

> What about your backups?  And your off-site backups?  And your mirror
> sites around the world?  Any other copies of those files that might
> have been copied off somewhere else

I'm not going to touch the aspects of attempting to rewrite the data in
backup sets without invalidating the backups.  Uhh uhh.  No deal.  I'm
also not going to touch the management of data that has been copied
outside of the store's purview.  Its no longer in the store's scope and
so isn't really under discussion.  I can run strings on my Oracle tables
as well, but that really doesn't make the resulting data files part of
Oracle's data-management model.

At its core this is a snapshot issue.  What you're really arguing for is
the ability to revert, recover, or synchronise (they're all the same
thing under the covers) the state of the store in a logically consistent
fashion.  As such you're interested in logical consistency for not just
one Big File, but across files, and across the meta-indexes; logical
consistency of the store as a whole.  This really isn't a storage format
problem.  Its a transaction framing problem and a snapshotting problem
(which is really jut a transaction framing problem).  You need to not
only know the state of the data files, but the state of the
meta-indexes, and that they are synchronised with each other.

This is not a trivial space, but its also not an unknown space.  File
versioning systems have been messing here for years with change keys and
and signatures.  Ultimately it comes down to a shared transaction key.
The old AT&T SCCS papers are a particularly good read in this regard.

-- 
J C Lawrence                
---------(*)                Satan, oscillate my metallic sonatas. 
claw at kanga.nu               He lived as a devil, eh?		  
http://www.kanga.nu/~claw/  Evil is a name of a foeman, as I live.