[Borgbackup] what do I do with inconsistencies?

[Thorsten von Eicken]

On 6/19/2016 4:12 PM, Thomas Waldmann wrote:
>> == Starting check for repo home
>> Repository: backup at backup:/big/h/home
>> borg check --show-rc --last 3 backup at backup:/big/h/home
>> borg.repository Remote: Index object count mismatch. 537180 != 537185
>> borg.repository Remote: Completed repository check, errors found.
>> terminating with warning status, rc 1
>
> That should not be. Can you reproduce this on different hardware?
>
> You could e.g. try almost the same commands on your x86_64 machine, 
> just using a local repo. If that works try again to the ARM machine, 
> but use a different repo storage medium.
>
So I did one test, which is to create the almost-identical backup on the 
x86_64 machine locally and from the x86_64 machine remotely onto the ARM 
box. I did the same init/create/check sequence in both cases with just 
the repo location being different. The result is that the local backup 
checked out OK and the remote one failed. This is how the good 
create/check looked:

# borg create --show-rc --stats -v -e .cache -C lzma 
home-test::home-2016-06-19 /big/home /etc /root /big/usr-local
------------------------------------------------------------------------------
Archive name: home-2016-06-19
Archive fingerprint: 
4a2f0549b8c15f1ffc5f1cb76bcbecb3eb1845149b2bd234f18de05a1bd8ab73
Time (start): Sun, 2016-06-19 17:43:51
Time (end):   Sun, 2016-06-19 22:17:22
Duration: 4 hours 33 minutes 31.50 seconds
Number of files: 917266
------------------------------------------------------------------------------
                        Original size      Compressed size Deduplicated size
This archive:               38.08 GB             19.39 GB             
16.77 GB
All archives:               38.08 GB             19.39 GB             
16.77 GB

                        Unique chunks         Total chunks
Chunk index:                  537453               935954
------------------------------------------------------------------------------
terminating with success status, rc 0
# borg check -v home-test
Starting repository check
Completed repository check, no problems found.
Starting archive consistency check...
Analyzing archive home-2016-06-19 (1/1)
Archive consistency check complete, no problems found.

This is how the bad create/check looked:

# borg create --show-rc --stats -v -e .cache -C lzma 
backup at backup:/big/h/home2::home-2016-06-19 /big/
------------------------------------------------------------------------------
Archive name: home-2016-06-19
Archive fingerprint: 
75c8ef99a5fb4a9d8c0c699c9e85abf7d0a989e1209fa5bead9965144d69b214
Time (start): Sun, 2016-06-19 17:56:20
Time (end):   Sun, 2016-06-19 22:27:56
Duration: 4 hours 31 minutes 36.10 seconds
Number of files: 917266
------------------------------------------------------------------------------
                        Original size      Compressed size Deduplicated size
This archive:               38.08 GB             19.41 GB             
16.76 GB
All archives:               38.08 GB             19.41 GB             
16.76 GB

                        Unique chunks         Total chunks
Chunk index:                  537312               935857
------------------------------------------------------------------------------
terminating with success status, rc 0
# borg check -v backup at backup:/big/h/home2
borg.repository Remote: Starting repository check
borg.repository Remote: Index object count mismatch. 537313 != 537318
borg.repository Remote: Completed repository check, errors found.

I was hacking on the machine while the backups were running and the 
second create started a few minutes later, so the minor difference in 
the chunk counts could be explained by that.

I then rsync'd the broken repository to the x86_64 box and ran a check 
there and got the same error report. So the issue is not in the check 
code but the create code.

I'm wondering what to test next. Some thoughts:
- rsync the data to the ARM box and perform a local create/check there
- nfs mount the data onto the ARM box and perform a local create/check 
this way

Suggestions?
Thorsten