[Borgbackup] Deduplication not efficient on single file VM images

Thu Dec 7 11:16:47 EST 2017

2017-12-04 21:32 GMT+01:00 Thomas Waldmann <tw at waldmann-edv.de>:
>> The backup image files are created by another tool (so these are
>> proper backups, not live disk images) and I am piping them into borg
>> stdin in my wrapper script.
>
> Likely because of the data format used.

Indeed it turned out so.
>
>> The biggest problem right now is that Borg seems to fail to
>> deduplicate most of the data:
>>
>> # du -sh {zbackup,borg}/vm-100
>> 1,9G    zbackup/vm-100
>> 8,0G    borg/vm-100
>
> I don't know zbackup details, but maybe you need finer granularity for
> borg's chunker?

Also correct, details follow :-)

>
>> Borg stats output for first, second and last borg create for vm-100:
>>  ------------------------------------------------------------------------------
>>  Archive name: vzdump-qemu-100-2017_11_20-15_52_32.vma
>
> Ah, you use proxmox? So guess one needs to research that .vma format...
>
> https://git.proxmox.com/?p=pve-qemu.git;a=blob;f=vma_spec.txt
>
> It puts a UUID into the VMA extent headers.
> Looks like this is always a different UUID in each .vma file.
> So that spoils dedup for the chunks containing that UUID.

Ouch, for some reason I thought that the VMAs do not have any internal
structure and that they are just block device copies. Thanks for
finding this as I would probably resist acknowledging that there is
some backup-specific metadata sprayed all over the VM dump. I would
rather assume that all metadata is at the beginning of the file and
then the raw blockdevice contents. After reading [1] I also kinda
understand why this format is cool and might have out-of-order data.

> extent = 59 clusters a 64kiB = ~ 3.8MB
>
> borg's default target block size is 2MiB - so borg's chunks will often
> contain that UUID (and thus not dedup with other .vma) and every 2nd
> chunk without an UUID inside might as well not match chunks from other
> .vma due different cutting places.
>
> So, you need to lower target chunk size significantly. You could check
> what zbackup uses or just try some target chunk sizes >= 64kiB.

AFAIU zbackup has a upper bound of 64k on chunk size [2] which fits
this kind of data nicely. After doing some thought experiments it
seems to me that for VMA the best chunk size would be 4k (because it
seems that the image data itself is kept in 4k blocks within the 64
clusters), but this will produce way too many chunks (and the indices
would be too large).

>
>> The machine itself is a simple shorewall based router and the image
>> doesn't change much. The only content that is changing are the logs,
>> so I am truly amazed why the deduplication performs so weakly.
>
> You could try doing a snapshot manually and reading the raw image data
> (from the blockdevice or whatever) into borg.

Yeah, but then I wouldn't backup the vm configuration. Restoring would
be a lot more manual work I then.

>> Is there something I am missing from the documentation regarding
>> tuning for my use-case?
>
> --chunker-params maybe. See also docs/misc/...
>
> But be aware the small chunks means also more chunks and more management
> overhead.

Yeah so I did a couple of calculations and tests with chunk size from
16k up to 64k:

-rw------- 1 root backup 30304298 Dec  6 01:24
borg_test_16k_10,18,14,4095/vm-100/index.468
-rw------- 1 root backup 17825978 Dec  6 11:25
borg_test_32k_10,18,15,4095/vm-100/index.458
-rw------- 1 root backup 17825978 Dec  6 15:21
borg_test_32k_12,20,15,4095/vm-100/index.455
-rw------- 1 root backup 10485898 Dec  6 15:10
borg_test_64k_10,18,16,4095/vm-100/index.478
-rw------- 1 root backup 10485898 Dec  6 00:44
borg_test_64k_12,20,16,4095/vm-100/index.477
1.9G    zbackup/vm-100
2.3G    borg_test_16k_10,18,14,4095/vm-100
2.2G    borg_test_32k_10,18,15,4095/vm-100
2.2G    borg_test_32k_12,20,15,4095/vm-100
2.3G    borg_test_64k_10,18,16,4095/vm-100
2.3G    borg_test_64k_12,20,16,4095/vm-100
-rw------- 1 root backup 17825978 Dec  6 02:48
borg_test_16k_10,18,14,4095/vm-101/index.357
-rw------- 1 root backup 10485898 Dec  6 12:30
borg_test_32k_10,18,15,4095/vm-101/index.352
-rw------- 1 root backup 10485898 Dec  6 16:35
borg_test_32k_12,20,15,4095/vm-101/index.355
-rw------- 1 root backup  5244058 Dec  6 16:19
borg_test_64k_10,18,16,4095/vm-101/index.378
-rw------- 1 root backup  5244058 Dec  6 01:54
borg_test_64k_12,20,16,4095/vm-101/index.381
1.5G    zbackup/vm-101
1.7G    borg_test_16k_10,18,14,4095/vm-101
1.7G    borg_test_32k_10,18,15,4095/vm-101
1.7G    borg_test_32k_12,20,15,4095/vm-101
1.8G    borg_test_64k_10,18,16,4095/vm-101
1.8G    borg_test_64k_12,20,16,4095/vm-101
-rw------- 1 root backup 171652778 Dec  6 23:48
borg_test_16k_10,18,14,4095/vm-200/index.5285
-rw------- 1 root backup  87578378 Dec  7 07:24
borg_test_32k_10,18,15,4095/vm-200/index.5535
-rw------- 1 root backup  87578378 Dec  7 12:01
borg_test_32k_12,20,15,4095/vm-200/index.5538
-rw------- 1 root backup  51516698 Dec  7 12:32
borg_test_64k_10,18,16,4095/vm-200/index.6146
-rw------- 1 root backup  51516698 Dec  6 19:33
borg_test_64k_12,20,16,4095/vm-200/index.6180
26G     zbackup/vm-200
26G     borg_test_16k_10,18,14,4095/vm-200
28G     borg_test_32k_10,18,15,4095/vm-200
28G     borg_test_32k_12,20,15,4095/vm-200
31G     borg_test_64k_10,18,16,4095/vm-200
31G     borg_test_64k_12,20,16,4095/vm-200
-rw------- 1 root backup 17825978 Dec  7 03:35
borg_test_16k_10,18,14,4095/vm-999/index.640
-rw------- 1 root backup 10485898 Dec  7 10:47
borg_test_32k_10,18,15,4095/vm-999/index.722
-rw------- 1 root backup 10485898 Dec  7 15:35
borg_test_32k_12,20,15,4095/vm-999/index.724
-rw------- 1 root backup  5244058 Dec  7 15:58
borg_test_64k_10,18,16,4095/vm-999/index.869
-rw------- 1 root backup  5244058 Dec  6 23:00
borg_test_64k_12,20,16,4095/vm-999/index.882
3.2G    zbackup/vm-999
3.0G    borg_test_16k_10,18,14,4095/vm-999
3.4G    borg_test_32k_10,18,15,4095/vm-999
3.4G    borg_test_32k_12,20,15,4095/vm-999
4.1G    borg_test_64k_10,18,16,4095/vm-999
4.2G    borg_test_64k_12,20,16,4095/vm-999

So it seems that borg is on par with zbackup dedupe/compression. Also
the processing times are more reasonable (even with lzma enabled).

I think that I will use 32k chunksize because the saved space seems to
be at least 10% and my largest projected (original) repository size is
somewhere around 7 TiB. Based on the index size (90MB) of vm-200 which
has (original) size somewhere around 250 GB the index size for 7TiB
repo should be around 2.5 GB. Assuming that the 2.1 factor for RAM
usage from docs/misc/create_chunker-params.txt holds it should take a
bit more than 5 GiB memory and I can live with that. Maybe I am doing
some obvious miscalculation here, if you see it please let me know.

One assumption that might be off-mark is that the index size depends
on the original repo size and not the actual repo size.

>> Another problem is that the backup takes way longer (zbackup takes
>> around 8 minutes to process the non-initial 14GB images, borg takes
>> more than 2 hours every time).
>
> That's likely the consequence of dedup not kicking in as much as expected.

Yeah, now it went down to 14 mins or so, so again it's on par with zbackup.

Thanks again for pointing out that the chunk size might be the problem.

[1] https://git.proxmox.com/?p=pve-qemu.git;a=blob;f=backup.txt;
[2] http://zbackup.org/#scalability