[Borgbackup] tag A or U when files are updated ?

[Thorsten Schöning]

Guten Tag Ernest CHIARELLO,
am Freitag, 3. September 2021 um 15:32 schrieben Sie:

> I want to clarify my request.[...]

No, you are changing it and I storngly suggest that further discussion
should be made in a new, indiviodual topic. Remember that the former
topic was about why BorgBackup considers your unchanged files changed.
You know the reason now and a workaround.

What you are describing/answering for additionally is more about a
suggestion about your overall backup use-case. That is a totally
different topic, otherwise you wouldn't need to describe it. :-)

> my question is: would it be possible to replace the "cp -u" by
> "borg create" (from /var/lib/libvirt/images to
> /var/lib/libvirt/backups), in order to benefit from the
> deduplication and to shorten the copy times ?

No, because you are mixing two different topics. If you want to backup
the individual files created by "fi-backup", then of course you can
use Borg to do so. Simply because Borg doesn't care too much about the
purpose of the files you are backing up.

But is that logically the same as letting "fi-backup" copy its own
files, create larger images and back those up? No. It's different. Is
it what you need/want? Don't know, that's something only you can
decide on your own.

Anyway, the result of using "borg create" in different steps won't
ever be the same as some "cp -u", because on one case you have content
added to a Borg repo always, while in the other case you have a
completely different file format maintained by "fi-backup". Most
likely QCOW2 if I understand the docs correctly.

> the creation of dummy files would then also be managed by fi-backup.

If those are necessary at all, in my opinion those dummy files a an
imlpementation detail for Borg and should therefore be managed by Borg
or, as suggested, BorgMatic instead. "fi-backup" doesn't know or need
to care about what Borg consideres changed or unchanged, when it does
read which files again for wehatever reason etc.

> i suppose you can not save the qcow2 file of a running VM. you must
> stop it before.

That totally depends on the content of the image. A lot of operating
systems, file systems and even apps like databases or else can handle
and survive a crashing host pretty well. just think about your own
Windows/Linux computer you use for your daily work or privately, they
crashed most likely at some point in the past. With whcih results,
completely data loss, new hardware necessary? Most likely not, only
some unsaved changes might have been lost, some FSCK/CHECKDISK being
executed and afterwards you were good to go again.

That's not much different with your VMs most likely and allows you to
backup running VMs as well. The important thing is to freeze the
running VM for how long the backup takes and you have multiple ways to
do so: Suspending the VM or something like file system level snapshots
of e.g. BTRFS and ZFS.

In my opinion, most people these days prefer the latter: Putting VMs
into BTRFS/ZFS, create a snapshot, mount that snapshots and let
BorgBackup directly backup that snapshot. This doesn't interrupt the
oepration of the VM, while at the same time what you get is a so
called "crash consistent" backup. That is enough for many needs, but
of course you need to decide that on your own.

And here's the point: When doing that approach, while BorgBackup reads
the whole image, in that use-case it directly applies de-duplication,
compression etc. and really only writes the changed data to its own
repo. If that is faster or not than what "fi-backup" does in two steps
is something you need to test on your own. Though, without knowing
"fi-backup", in my opinion having only one step is easier in the end.
Remember that each run of "borg create" will provide you a working VM,
without additional steps to copy things around, merge QCOW2 images
etc.

Additionally have a look at what "fi-backup" writes on its own:

> By default fi-backup.sh guarantees that qcow2 images format is
> consistent, but this doesn't imply that the contained filesystems
> are consistent too. In order to perform consistent backups, you can
> use two different strategies:

https://github.com/dguerri/LibVirtKvm-scripts#note

So, depending on your setup, you might already backup only "crash
consistent" state anyway. Additionally, even with the guest agend of
QEMU installed, individual apps like databases won't know anything
about that "fi-backup" is doing some backup "right now" and therefore
won't do anything to make their own files consistent. The only thing
the agent changes is that "fi-backup" is able to tell the guest-VM to
e.g. sync pending file system changes on disk before suspending. that
DOES NOT guarantee any consistent file format state in the VM, though.

Mit freundlichen Grüßen

Thorsten Schöning

-- 
AM-SoFT IT-Service - Bitstore Hameln GmbH
Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister für IT und TK

E-Mail: Thorsten.Schoening at AM-SoFT.de
Web:    http://www.AM-SoFT.de/

Tel:   05151-  9468- 0
Tel:   05151-  9468-55
Fax:   05151-  9468-88
Mobil:  0178-8 9468-04

AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 221853 - Geschäftsführer: Janine Galonska