[Borgbackup] Fw: pve-zsync + borgbackup "changed size" discrepancy
devzero at web.de
devzero at web.de
Sat Mar 2 08:59:58 EST 2019
Hi,
i was wondering why borg repo was growing so fast where zfs replication was transferring only 10 times less data (see initial post at the bottom of this mail) - and got this resolved after intensively reading into borg docs and in the end getting across https://github.com/borgbackup/borg/blob/master/docs/misc/create_chunker-params.txt
shouldn't we need some "backup efficiency advisor" which should be able to give a better hint on "how to get the most out of borg" ?
what about a tool which analyses the borg repo and giving a hint for more space efficient usage ?
at least i would have found it useful to have such information at hands (in the manpage for example) what tuning knob we need to look at to make backups of large-files-with-tiny-changes much more space efficient...
mind that even with "--chunker-params 10,23,16,4095" borg backup diff grows up to 7.98MB where xdelta3 reports a diff of only 158218 . so this is still not optimal, but a value i can live with....
thought this could be interesting issue for many, as borg is getting more and more mainstream backup tool, especially for large data files...
regards
roland
let's have a look on the "real diff" between the two VM snapshots
# xdelta3 -v -c -s ./zfs-auto-snap_hourly-2019-03-02-1109/images/103/vm-103-disk-0.qcow2 ./zfs-auto-snap_hourly-2019-03-02-1117/images/103/vm-103-disk-0.qcow2 2>&1 >/dev/null |grep bytes
xdelta3: finished in 230 sec; input 34365243392 output 158218 bytes (0.00%)
ok - it's only 160kb binary diff
apparently, the diff in borg (even with compression + dedup) grows to nearly 100mb in size because of sub-optimal default chunker-params for this scenario
# /root/backup/borg-linux64 init -e=none /rpool/borg-backup-replicated/testrepo
# /root/backup/borg-linux64 create --stats -C lz4 /rpool/borg-backup-replicated/testrepo::initial ./zfs-auto-snap_hourly-2019-03-02-1109/images/103/vm-103-disk-0.qcow2
------------------------------------------------------------------------------
Archive name: initial
Archive fingerprint: d5f9ace56a2908723f7b288f013dc11415a2fc7608cabc6bb8b663c9000d2083
Time (start): Sat, 2019-03-02 13:36:45
Time (end): Sat, 2019-03-02 13:48:08
Duration: 11 minutes 22.85 seconds
Number of files: 1
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
Original size Compressed size Deduplicated size
This archive: 34.37 GB 8.95 GB 8.86 GB
All archives: 34.37 GB 8.95 GB 8.86 GB
Unique chunks Total chunks
Chunk index: 5446 7781
------------------------------------------------------------------------------
# /root/backup/borg-linux64 create --stats -C lz4 /rpool/borg-backup-replicated/testrepo::secondrun ./zfs-auto-snap_hourly-2019-03-02-1117/images/103/vm-103-disk-0.qcow2
------------------------------------------------------------------------------
Archive name: secondrun
Archive fingerprint: b60bf8590aca0e772f9300f9f09e177715f55e0722d33cddf462629232d8aa07
Time (start): Sat, 2019-03-02 13:50:54
Time (end): Sat, 2019-03-02 13:57:13
Duration: 6 minutes 18.75 seconds
Number of files: 1
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
Original size Compressed size Deduplicated size
This archive: 34.37 GB 8.95 GB 94.95 MB
All archives: 68.73 GB 17.91 GB 8.95 GB
Unique chunks Total chunks
Chunk index: 5501 15562
------------------------------------------------------------------------------
so, let's try to tune this in a better way accoring to https://github.com/borgbackup/borg/blob/master/docs/misc/create_chunker-params.txt :
# rm -rf /rpool/borg-backup-replicated/testrepo/
# /root/backup/borg-linux64 init -e=none /rpool/borg-backup-replicated/testrepo
# /root/backup/borg-linux64 create --stats -C lz4 --chunker-params 10,23,16,4095 /rpool/borg-backup-replicated/testrepo::initial ./zfs-auto-snap_hourly-2019-03-02-1109/images/103/vm-103-disk-0.qcow2
------------------------------------------------------------------------------
Archive name: initial
Archive fingerprint: 6c412446c52547c5cfd6e213bba049d442c1bf8545e1286ca6389c1f4319f376
Time (start): Sat, 2019-03-02 14:03:53
Time (end): Sat, 2019-03-02 14:16:34
Duration: 12 minutes 41.91 seconds
Number of files: 1
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
Original size Compressed size Deduplicated size
This archive: 34.37 GB 9.04 GB 8.39 GB
All archives: 34.37 GB 9.04 GB 8.39 GB
Unique chunks Total chunks
Chunk index: 190164 210448
------------------------------------------------------------------------------
# /root/backup/borg-linux64 create --stats -C lz4 --chunker-params 10,23,16,4095 /rpool/borg-backup-replicated/testrepo::second ./zfs-auto-snap_hourly-2019-03-02-1117/images/103/vm-103-disk-0.qcow2
------------------------------------------------------------------------------
Archive name: second
Archive fingerprint: 05b6f69b89aba8dd7f90da341ca847b1f35f4f7fd76b93d60d409fc5393df6f0
Time (start): Sat, 2019-03-02 14:35:18
Time (end): Sat, 2019-03-02 14:41:40
Duration: 6 minutes 22.33 seconds
Number of files: 1
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
Original size Compressed size Deduplicated size
This archive: 34.37 GB 9.04 GB 7.98 MB
All archives: 68.75 GB 18.08 GB 8.40 GB
Unique chunks Total chunks
Chunk index: 190271 420890
------------------------------------------------------------------------------
> Gesendet: Freitag, 01. März 2019 um 12:50 Uhr
> Von: devzero at web.de
> An: pve-user at pve.proxmox.com
> Betreff: pve-zsync + borgbackup "changed size" discrepancy
>
> hi,
> i'm syncing a zfs dataset with pve-zsync to another proxmox host and also do backup with borg-backup of the same
> dataset via temporary backup snapshot.
>
> i have shutdown all VMs on the dataset and only leave a single win10 vm up and running.
>
> i see a huge discrepancy between what pve-zsync is reporting as "changed data" to be synced to the other node
> and what borbackup is reporting regarding backup growth (see below). ("deduplicated size" means: this is the size
> the archive grew after the data has been checksummed, deduplicated and compressed)
>
> does anybody have a clue why this differs by "roughly a power of 10" ?
>
> is pve-zsync estimation wrong ?
> is borgbackup "overhead" so big when backing up large single qcow2 files ?
>
> regards
> roland
>
> pve-zsync estimated size:
> total estimated size is 1.79M
> total estimated size is 22.9M
> total estimated size is 13.5M
> total estimated size is 24.4M
> total estimated size is 22.9M
> total estimated size is 24.1M
> total estimated size is 10.2M
> total estimated size is 20.9M
> total estimated size is 81.0M
> total estimated size is 53.3M
> total estimated size is 10.8M
> total estimated size is 14.1M
> total estimated size is 10.9M
> total estimated size is 189M
> total estimated size is 29.0M
> total estimated size is 21.2M
> total estimated size is 14.1M
> total estimated size is 13.1M
> total estimated size is 11.0M
> total estimated size is 14.1M
> total estimated size is 11.3M
> total estimated size is 14.9M
> total estimated size is 10.5M
> total estimated size is 14.2M
> total estimated size is 20.4M
>
> BorgBackup growth
> Original size Compressed size Deduplicated size
> This archive: 111.69 GB 19.86 GB 95.31 MB
> This archive: 111.69 GB 19.86 GB 197.58 MB
> This archive: 111.69 GB 19.86 GB 119.40 MB
> This archive: 111.69 GB 19.86 GB 138.81 MB
> This archive: 111.69 GB 19.86 GB 180.30 MB
> This archive: 111.69 GB 19.86 GB 205.75 MB
> This archive: 111.69 GB 19.86 GB 76.22 MB
> This archive: 111.69 GB 19.86 GB 159.60 MB
> This archive: 111.69 GB 19.86 GB 636.18 MB
> This archive: 111.69 GB 19.86 GB 413.14 MB
> This archive: 111.69 GB 19.86 GB 83.86 MB
> This archive: 111.69 GB 19.86 GB 122.86 MB
> This archive: 111.69 GB 19.86 GB 108.17 MB
> This archive: 111.69 GB 19.86 GB 153.89 MB
> This archive: 111.69 GB 19.89 GB 946.72 MB
> This archive: 111.69 GB 19.89 GB 126.48 MB
> This archive: 111.69 GB 19.89 GB 139.23 MB
> This archive: 111.69 GB 19.89 GB 113.23 MB
> This archive: 111.69 GB 19.89 GB 67.53 MB
> This archive: 111.69 GB 19.89 GB 129.25 MB
> This archive: 111.69 GB 19.89 GB 74.28 MB
> This archive: 111.69 GB 19.89 GB 110.28 MB
> This archive: 111.69 GB 19.89 GB 80.15 MB
> This archive: 111.69 GB 19.89 GB 92.86 MB
> This archive: 111.69 GB 19.89 GB 160.57 MB
More information about the Borgbackup
mailing list