[Borgbackup] Further development idea/proposal: Heal the damaged repository by replacing damaged blocks from other repositories created from same sources

[JK]

Hi,

Further development idea/proposal:

So borg clears damaged blocks to zero but keeps the original hash 
calculated from the original content.

Could it be possible to "heal" the damaged repository by replacing these 
damaged blocks from other repositories created from same sources?

I keep most of my repositories on USB disks. There are several USB 
disks, usually 3 which I switch daily. The most critical data is also 
backed up to a local disk repository. In these critical cases the backup 
always runs twice, first the local disk repo and then the USB disk repo. 
Less critical repositories are only on USB disks.

So although the repositories on different USB disks and on the local 
disk are not identical, they have lots of common content because they 
all have identical source directory settings and are pruned with same 
policy.

Now imagine I find that one (or more) of these repositories are 
partially corrupted. (Most likely the damage is only in one repository 
but if there are damages also in other repositories, they are most 
likely not concerning same source files or same source file areas.)

Could it be possible to repair the damaged repository by replacing 
zeroed blocks in the damaged repository from other repository if the 
hashes in both repositories are identical? This way we could heal the 
damaged repository or atleast decrease the number of zeroed blocks even 
with another partially damaged repository.

JK



On 3.4.2019 1.46, Marcin Zajączkowski wrote:
> Thanks for your comprehensive explanations!
>
> Marcin
>
>
> On 2019-04-02 23:47, Thomas Waldmann wrote:
>>> Changing a little bit my question, would I know after a repair operation
>>> or while getting my files from the backup (also after a repair
>>> operation) that those accessed files are corrupted?
>>>
>>> Or those files would be read as any other files, just having
>>> occasionally some zeros inside?
>> If you try to extract stuff from a corrupt repo, you will get exceptions
>> like ObjectNotFound or IntegrityError, so you'll definitely notice
>> something is wrong.
>>
>> borg check --repair tries to get a repo into a consistent state.
>>
>> That doesn't mean that data which is lost can be magically brought back,
>> but it will either delete corrupt archives or replace missing/corrupt
>> content blocks in files by all-zero blocks of same size (and also it
>> will remember the correct block hashes).
>>
>> Repo objects that have invalid contents (invalid crc or invalid MAC)
>> will be removed from the repo.
>>
>> If you extract such a "zero-patched" file, borg will warn you about it.
>> borg mount will reject reading such files, except when mounting with a
>> special option.
>>
>> If you do a backup again after such a repair that reproduces objects
>> which were lost / corrupted and you run borg check --repair again
>> afterwards, borg might be able to heal some "patched" files (because it
>> notices that lost blocks are there again and it still knows the correct
>> hash of the previously missing blocks).
>
> _______________________________________________
> Borgbackup mailing list
> Borgbackup at python.org
> https://mail.python.org/mailman/listinfo/borgbackup