[Borgbackup] some performance numbers...

[Antoine Beaupré]

Yesterday I did my first purge ever of my repository. I had been doing
daily backups of my home server for almost 3 years now and had 945
archives. It was an interesting experiment and I figured I would share
the results I found.

I ran the following command:

    time borg prune --verbose --list -d 30 -w 52 -y 10 $repo

That deleted 860 archives and kept 85. It freed about 500GB of disk
space and took well over 12 hours:

real    798m5,477s
user    341m9,688s
sys     18m16,624s

The last archive saved before the purge had the following stats:

------------------------------------------------------------------------------
Archive name: marcos-2017-09-15
Archive fingerprint: dbd7121462701d4916f3c4a6c9369154d75090d93be0f1a0c04a269b4902b435
Time (start): Fri, 2017-09-15 00:07:40
Time (end):   Fri, 2017-09-15 00:28:10
Duration: 20 minutes 29.83 seconds
Number of files: 1920931
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:              231.90 GB            212.51 GB              1.11 GB
All archives:              239.20 TB            211.46 TB              1.23 TB

                       Unique chunks         Total chunks
Chunk index:                18349863           4937580530
------------------------------------------------------------------------------

The next one had those:

------------------------------------------------------------------------------
Archive name: marcos-auto-2017-09-17
Archive fingerprint: 51637fc22df29c9e0dffe814de547e8c2c5045e3d41e0c318de5fbc37d442103
Time (start): Sun, 2017-09-17 00:09:45
Time (end):   Sun, 2017-09-17 00:26:39
Duration: 16 minutes 54.47 seconds
Number of files: 1919597
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:              231.51 GB            212.15 GB              1.20 GB
All archives:               19.31 TB             17.37 TB            550.97 GB

                       Unique chunks         Total chunks
Chunk index:                 8065589            397239183
------------------------------------------------------------------------------

So an important takeaway here is that removing a significant portion of
the archives improved the execution time by about 25% at almost zero
cost in terms of information loss: notice how the number of files didn't
really change, for example. This means we still track all files all the
way back to the very beginning of the history here, and we can restore
really old files. What we can't do is choose between two *days* for
archives made 3 years ago, but the odds or needing that are dim. This
also confirms we saved about 500GB of disk space during the purge, or,
more accurately, about 650GB or 50% of the archive.

Then I also had to rename archives to follow the new naming convention I
use regarding logging (see previous mail). This also took a surprising
amount of time... I had to rename the remaining 85 archives, and that
took about an hour.

So, there you go... Obviously, those are just some anectdotal results,
but I figured it might be interesting to the casual readers here.

A.

-- 
We must shift America from a needs- to a desires-culture. People must
be trained to desire, to want new things, even before the old have
been entirely consumed. Man's desires must overshadow his needs.
                         - Paul Mazur, Lehman Brothers