[Borgbackup] 550GB rsync-tree will not dump

[Thomas Waldmann]

Hi Gerrit,

thanks for reporting this and congrats, it looks like you triggered the
discovery of at least 3 issues.

> directories with many files
> occurring just once but referenced many times by hard links to them.

The hardlinks do not matter (would happen in the same way if you had a
lot of 0 byte files), but I assume you have a huge count of filesystem
items due to this.

Could you count them (files + directories), e.g. using find . | wc -l in
the toplevel dir of the rsync backup dir?

> I tried this using lz4 or no compression to no avail.

As it was already asked by Adrian: we also need the error msg and
traceback for the "no compression" case.

> Exception: lz4 compress failed
> Any ideas what's happening?

Yes, I analyzed it:

https://github.com/borgbackup/borg/issues/1453


The root cause for this is very likely that:

https://github.com/borgbackup/borg/issues/1452

If you give us the uncompressed traceback, I guess we could verify that.


Also, we noticed that:

https://github.com/borgbackup/borg/issues/1451


We'll try to fix or relax some of the issues ASAP.


BTW, if you transform a rsync-hardlink-method backup into a borg backup,
instead of just backing up the whole thing into 1 repo, you could create
1 archive per rsync-snapshot. There is even an option to set the
timestamp of the archive to some past date, if you like.

This would likely already solve the issue you encountered, because the
per-archive item (files/dirs) count will be much less then.

If you could keep your rsync backup until 1.0.7 (rc2?), it would be cool
if you could check by then if our fix works for you.

Cheers,

Thomas

-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393