[Borgbackup] A small compression test

Tue Mar 7 03:31:40 EST 2023

Guten Tag Bzzzz,
am Freitag, 3. März 2023 um 00:21 schrieben Sie:

> So, if this ML accepts attachments (?) and that may help somebody to
> make a choice, here they are.

Hi everyone,

thanks for sharing these numbers, found them pretty interesting and
made me reconsider which compression mode to use. Tried with "zstd,3"
now and see somewhat strange results with my two use-cases of backing
up many infividual files of within a host vs. few large files of
VM-images and database dumps.

According to your numbers I would have expected increased backup times
of about factor 4, but for some hosts I see factor 16! What makes me
especially wonder is that things seem to scale with the number of
files being backed up and not with the amount of actual data to store.
The latter is MUCH more for my VM-images and database dumps, but their
processing times mostly didn't increase at all.

From my understanding, compression times shouldn't depend on the
number of files or stuff, but only on the amount of data to be
compressed and how good it can be compressed, shouldn't it?

Beyond changing compression for new archives, I've recompressed all
existing ones as well to test how much space is saved, how slow things
are etc. Especially the repos containing many individual host files
weren't that slow to recompress.

> sudo borgmatic --config ~/.config/borgmatic.d/[...].yaml --verbosity 2 borg recreate --recompress

Though, might that have introduced some problem? E.g. because creating
new archives additionally scales with the compression mode of existing
archives? From my understanding of how Borg works I would say no,
because deduplication is independent of compression and compression
really only applied when reading or writing actual chunks of data.

The following is an example of processing times with new compression
mode. All the mentioned files are pretty small, mostly text stuff and
the multiple hours of this log excerpt is already FAR LONGER than the
overall backup of that host took before changing to "zstd,3". :-)

> borg create --exclude-from /tmp/tmp4fo42514 --compression zstd,3
> --numeric-ids --files-cache ctime,size --remote-path borg-1.2
> --umask 7 --list --filter AME- --debug --show-rc --umask 0007
> bts.ams-sbox:bak_borg/hosts/bts.ams::de.am-soft.potsdam.potsdam-{utcnow:%Y%m%dT%H%M%SZ}
> /home/ams_d_bak_borg/.borgmatic
> /mnt/ams_d_bak_borg/backup/ams.pdm.pdm/root_wo_dbs

> [2023-03-06 11:22:49,502] INFO: M /[...]/home/jenkins/jobs/Libs Java AMS/scm-polling.log
> [2023-03-06 11:26:04,195] INFO: M /[...]/home/jenkins/jobs/Libs Perl AMS/scm-polling.log
> [2023-03-06 11:32:40,381] INFO: M /[...]/home/jenkins/jobs/Smart-Metering (Bin)/scm-polling.log
> [2023-03-06 11:55:09,610] INFO: M /[...]/home/jenkins/jobs/Smart-Metering (Src)/scm-polling.log
> [2023-03-06 11:55:09,782] INFO: Remote: check_free_space: required bytes 1035718058, free bytes 1639959844864
> [2023-03-06 11:55:10,259] INFO: security: saving state for c0bb9740b53cd69ac70b2414879f0d80280b3a1c08353e56693a1a6f578b7251 to /home/ams_d_bak_borg/.config/borg/security/c0bb9740b53cd69ac70b2414879f0d80280b3a1c08353e56693a1a6f578b7251
> [2023-03-06 11:55:10,259] INFO: security: current location   ssh://bts.ams-sbox/./bak_borg/hosts/bts.ams
> [2023-03-06 11:55:21,159] INFO: security: key type           0
> [2023-03-06 11:55:21,159] INFO: security: manifest timestamp 2023-03-06T10:55:09.625553
> [2023-03-06 11:55:21,159] INFO: Remote: Verified integrity of /home/bak_borg/hosts/bts.ams/index.59682
> [2023-03-06 11:55:21,159] INFO: Remote: Cleaned up 0 uncommitted segment files (== everything after segment 59682).
> [2023-03-06 11:55:21,161] INFO: Remote: Verified integrity of /home/bak_borg/hosts/bts.ams/hints.59682
> [2023-03-06 11:58:43,238] INFO: M /[...]/home/jenkins/jobs/de.am_soft.docbeam.raw/scm-polling.log
> [2023-03-06 12:03:14,950] INFO: M /[...]/home/jenkins/jobs/fwbuilder/scm-polling.log
> [2023-03-06 12:08:14,870] INFO: M /[...]/home/jenkins/jobs/de.am_soft.docbeam.egvp_int/scm-polling.log
> [2023-03-06 12:08:52,546] INFO: M /[...]/home/jenkins/jobs/de.am_soft.docbeam.printing.d/scm-polling.log
> [2023-03-06 12:08:54,241] INFO: M /[...]/home/jenkins/logs/tasks/Workspace clean-up.log
> [2023-03-06 12:08:54,285] INFO: M /[...]/home/jenkins/logs/tasks/Fingerprint cleanup.log.5
> [2023-03-06 12:08:54,329] INFO: M /[...]/home/jenkins/logs/tasks/Fingerprint cleanup.log.4
> [2023-03-06 12:08:54,373] INFO: M /[...]/home/jenkins/logs/tasks/Fingerprint cleanup.log.3
> [2023-03-06 12:08:54,417] INFO: M /[...]/home/jenkins/logs/tasks/Fingerprint cleanup.log.2
> [2023-03-06 12:08:54,461] INFO: M /[...]/home/jenkins/logs/tasks/Fingerprint cleanup.log.1
> [2023-03-06 12:08:54,505] INFO: M /[...]/home/jenkins/logs/tasks/Fingerprint cleanup.log
> [2023-03-06 12:08:54,549] INFO: M /[...]/home/jenkins/logs/tasks/Periodic background build discarder.log.5
> [2023-03-06 12:08:54,597] INFO: M /[...]/home/jenkins/logs/tasks/Periodic background build discarder.log.4
> [2023-03-06 12:08:54,645] INFO: M /[...]/home/jenkins/logs/tasks/Periodic background build discarder.log.3
> [2023-03-06 12:08:54,650] INFO: M /[...]/home/jenkins/logs/tasks/Periodic background build discarder.log.2
> [2023-03-06 12:08:54,697] INFO: M /[...]/home/jenkins/logs/tasks/Periodic background build discarder.log.1
> [2023-03-06 12:08:54,745] INFO: M /[...]/home/jenkins/logs/tasks/Periodic background build discarder.log
> [2023-03-06 12:10:35,270] INFO: M /[...]/home/jenkins/updates/default.json
> [2023-03-06 12:10:35,348] INFO: M /[...]/home/jenkins/updates/hudson.tasks.Maven.MavenInstaller
> [2023-03-06 12:10:35,397] INFO: M /[...]/home/jenkins/updates/hudson.tasks.Ant.AntInstaller
> [2023-03-06 12:10:35,445] INFO: M /[...]/home/jenkins/updates/hudson.tools.JDKInstaller
> [2023-03-06 12:10:36,130] INFO: M /[...]/home/jenkins/.owner
> [2023-03-06 12:38:17,991] INFO: M /[...]/home/sm-mtg/sp1/de.am_soft.sm_mtg/de.am_soft.sm_mtg.backend.open_vpn/data/.svn/wc.db
> [2023-03-06 12:38:18,166] INFO: Remote: check_free_space: required bytes 1035718138, free bytes 1639958956032
> [2023-03-06 12:38:18,895] INFO: security: saving state for c0bb9740b53cd69ac70b2414879f0d80280b3a1c08353e56693a1a6f578b7251 to /home/ams_d_bak_borg/.config/borg/security/c0bb9740b53cd69ac70b2414879f0d80280b3a1c08353e56693a1a6f578b7251
> [2023-03-06 12:38:18,895] INFO: security: current location   ssh://bts.ams-sbox/./bak_borg/hosts/bts.ams
> [2023-03-06 12:38:30,078] INFO: security: key type           0
> [2023-03-06 12:38:30,078] INFO: security: manifest timestamp 2023-03-06T11:38:18.014040
> [2023-03-06 12:38:30,078] INFO: Remote: Verified integrity of /home/bak_borg/hosts/bts.ams/index.59686
> [2023-03-06 12:38:30,078] INFO: Remote: Cleaned up 0 uncommitted segment files (== everything after segment 59686).
> [2023-03-06 12:38:30,080] INFO: Remote: Verified integrity of /home/bak_borg/hosts/bts.ams/hints.59686
> [2023-03-06 12:38:30,129] INFO: M /[...]/home/sm-mtg/sp1/de.am_soft.sm_mtg/de.am_soft.sm_mtg.backend.open_vpn/data/.svn/wc.db-journal
> [2023-03-06 12:38:30,497] INFO: M /[...]/home/sm-mtg/sp1/de.am_soft.sm_mtg/de.am_soft.sm_mtg.backend.open_vpn/data/IPC-CL/1.lock
> [2023-03-06 15:49:35,014] INFO: M /[...]/tmp/ams_cookies/db_login/referenz/system/ZMAIWOVEQVVLWIFYUNSFRDUKCJTQMKWIITNCQGQDJVSQKHBGKD.pag
> [2023-03-06 15:49:35,145] INFO: Remote: check_free_space: required bytes 1035718218, free bytes 1639958796800
> [2023-03-06 15:49:35,718] INFO: security: saving state for c0bb9740b53cd69ac70b2414879f0d80280b3a1c08353e56693a1a6f578b7251 to /home/ams_d_bak_borg/.config/borg/security/c0bb9740b53cd69ac70b2414879f0d80280b3a1c08353e56693a1a6f578b7251
> [2023-03-06 15:49:35,718] INFO: security: current location   ssh://bts.ams-sbox/./bak_borg/hosts/bts.ams
> [2023-03-06 15:49:47,080] INFO: security: key type           0
> [2023-03-06 15:49:47,081] INFO: security: manifest timestamp 2023-03-06T14:49:35.026277
> [2023-03-06 15:49:47,081] INFO: Remote: Verified integrity of /home/bak_borg/hosts/bts.ams/index.59690
> [2023-03-06 15:49:47,081] INFO: Remote: Cleaned up 0 uncommitted segment files (== everything after segment 59690).
> [2023-03-06 15:49:47,082] INFO: Remote: Verified integrity of /home/bak_borg/hosts/bts.ams/hints.59690
> [2023-03-06 15:49:47,785] INFO: A /[...]/tmp/hsperfdata_tomcat/1024
> [2023-03-06 15:49:47,834] INFO: A /[...]/tmp/hsperfdata_root/1324
> [2023-03-06 15:49:48,329] INFO: A /[...]/tmp/hsperfdata_sm-mtg/1717
> [2023-03-06 15:49:48,377] INFO: A /[...]/tmp/hsperfdata_sm-mtg/1716

Does anyone have any idea what might be the problem here? Thanks!

Mit freundlichen Grüßen

Thorsten Schöning

-- 
AM-SoFT IT-Service - Bitstore Hameln GmbH
Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister für IT und TK

E-Mail: Thorsten.Schoening at AM-SoFT.de
Web:    http://www.AM-SoFT.de/

Tel:   +49 5151-  9468- 0
Tel:   +49 5151-  9468-55
Mobil: +49  178-8 9468-04

AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 221853 - Geschäftsführer: Janine Galonska

Für Rückfragen stehe ich Ihnen jederzeit zur Verfügung. 

Mit freundlichen Grüßen, 

Thorsten Schöning

Telefon: +49 5151 9468-55
Fax: 
E-Mail: TSchoening at am-soft.de

AM-Soft IT-Service - Bitstore Hameln GmbH
Brandenburger Straße 7c
31789 Hameln

Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen und ist ausschliesslich für den Adressaten bestimmt. Jeglicher Zugriff auf diese E-Mail durch andere Personen als den Adressaten ist untersagt. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Sollten Sie nicht der für diese E-Mail bestimmte Adressat sein, ist Ihnen jede Veröffentlichung, Vervielfältigung oder Weitergabe wie auch das Ergreifen oder Unterlassen von Massnahmen im Vertrauen auf erlangte Information untersagt. 

This e-mail may contain confidential and/or privileged information and is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. 

Hinweise zum Datenschutz: bitstore.group/datenschutz