[Borgbackup] Test : Borg vs Restic

[Thomas Waldmann]

Hi Melkor,

> The backup data consists of a live mail repository using the maildir
> format and holding 139 GB (2327 dirs, 665456 files).

== lots of little files in a huge directory, performance will be
relatively bad in any case.

> This archive:              137.47 GB            112.55 GB       102.30 GB

Compression is relatively bad because it's only small files.

> This archive:              137.47 GB            112.55 GB      14.57 MB

Hmm, maybe your first run was not as root, thus it changed atime, thus
not deduplicating the metadata stream.

Maybe try running as root (then it can open using a no-atime-changing
mode) or borg create --no-atime to not store the atime.

> (restic) duration: 0:47, 2920.94MiB/s

Hah, so much for meaningless performance metrics (it did not really READ
the files, so the 2.9GiB/s didn't really happen). :D

> - Borg is way faster on first pass (1h15m vs 1h48m) but significantly
> slower on second pass (2m1s vs 47s)

As you see at the deduped size, borg wrote some new information to the
repo (likely changed atime timestamps).

Not sure whether restic deals with changed atime / does store it at all.

> - Borg repo size (103 GB) is smaller than Restic repo size (121 GB)

Expected. Restic can't compress, borg 1.1 does lz4 by default.

> BORG vs RESTIC on mounted archives :
> ====================================
> borg real    0m22.383s
> restic real  0m0.003s

Expected.

A borg archive is somehow like a tar archive (but only the metadata as
one stream, having the content data stored separately). So it is one
linear stream of file metadata for all the stuff. There is no separate
directory information, directory contents are just implicit from all the
file paths.

When doing the FUSE mount, borg needs to read the whole archive and
build up the hierarchical filesystem structure in memory.

Restic stores directory information in a separate / structured way, so I
guess it loads just what it needs on demand when you enter a new directory.

Maybe run a ls -lR on both mounts and include that in the timing.

> * Extract some path from the mounted repositories :
> borg real    3m36.534s
> restic real  6m23.970s

The revenge of the lazy loading? :D

> - Borg is faster by a factor of 2 to restore the exact same data using
> about 2x more CPU.

BTW, when extracting bigger amounts, borg extract should be faster than
copying from the fuse mount.

> * Fetch deep info on mounted repositories :
> Shell# time du -s --si ...
> restic real  1m18.590s
> borg real    5m30.143s

Hmm, that is strange. For old borg (< 1.1) I'ld have said "ok, it needs
to compute size information by summing up chunk lengths". But IIRC we
now store that information into the metadata (1.1), so that should be
fast now.
But maybe it is something completely different.

> BORG vs RESTIC trying to backup while having mounted archives :

We play safe. Whenever you do something, the repo is locked. This is a
quite coarse granular way, but simple and safe.

For multiple writers, a lock is rather obviously needed to avoid
corrupting shared data structures.

But even for a one writer multiple reader scenario it is unsafe to read
data while a writer could modify it...

> NOTE: Typical use case would be trying to restore a very big file in a
> very nested/complex directory hierarchy that would make this impractical
> using the "extract/restore" command. Retrieving the said file would be
> so time consuming that it would overlap with the next scheduled backup
> for example.

We might be able to do less / more fine granular locking in the future,
but next priority is multihreading to increase performance and crypto
enhancements, so locking will have to wait.

> NOTE: Here the Restic design have a clear advantage. Quoting the doc :
> "All files in a repository are only written once and never modified
> afterwards. This allows accessing and even writing to the repository
> with multiple clients in parallel".

For all the content data chunks and backup archives, this is quite the
same for borg. The manifest (list of all archives) is read-modify-write
though.

Also, when writing with multiple writers at the same time, there is
obviously a cache coherency issue (if one uses a local cache, like borg
does). Not necessarily fatal, but maybe less efficient.

> Features I like in Restic :
> ===========================
> 
> - Nothing is written outside the repository

If you mean borg's cache with "outside the repository": this is for
performance reasons. We could always query the repo, but that would be
slower.

> - The "views" on mounted repositories (host, snapshots and tags)

Yes, borg just has archive names and comments. Maybe a little less
versatile / structured, but does its job (assuming you use sane names).

> - Multiple "keys" per repository (like LUKS)

So it is the same repo encryption key, just encrypted with different
passphrases?

> Features I *DISLIKE* in Restic :
> ================================
> 
> - The "mount" command blocks the shell and waits for "CTRL-C" to end.

Likely: Just not forking to the background. Good for debugging, bad for UX.

> - No support for sparse files (AFAIK) which makes it not usable for VM
> images and such.

Oh, really?

I'ld have expected that since it is easy to do (at least the simple
implementation).

> Features I *DISLIKE* in Borg :
> ==============================
> 
> - Writes several files OUTSIDE the repository, ~/.config/borg and
> ~/.cache/borg and AFAIK, there's no option to use another paths for
> these files.

There is, see environment vars.

> - The "several seconds or more" delays when mounting repositories and
> scanning deeper directories.

Lazy loading for all directory levels is not efficiently possible for
borg due to the linear archive structure.

> I can live with the delays but I really wish there was an option to
> relocate the ".config" and ".cache" data. I need this because it makes
> it easier to copy the data offsite without forgetting anything!

You don't need to copy the cache, it can be reconstructed (just takes
some time). But make sure you do not lose passphrase and key / make a
backup of them (borg key ...).

> I know
> that ".cache" is disposable bug having this data available when
> restoring in case of disaster recovery is a huge gain of time.

Hmm, did you actually try that?

> Features I *DISLIKE* in BOTH tools :
> ====================================
> 
> - Their design geared at "backup-and-push-to-repository" which is nice
> but not desired in my environment. I need a
> "repository-pulls-backup-from-agent" design.

Matter of taste / threat model.

We have an issue in the issue tracker about it with some suggestion how
to implement pull (based on existing code). It just needs somebody to
confirm it and do a docs pull request.

Cheers, Thomas

-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393