From ngoonee.talk at gmail.com  Tue Jan  1 14:18:42 2019
From: ngoonee.talk at gmail.com (Oon-Ee Ng)
Date: Wed, 2 Jan 2019 03:18:42 +0800
Subject: [Borgbackup] 'Multiple repos with the same ID' error
In-Reply-To: <IFHK0UDO3U_sfnnbdbElT0uOJ41lem0iyfnu38k7C1R-0pEvsKn0TOIz8uAr-aKTm4oWP6wW2J4EZNSRIOCrJI4SIBJd3qGanHrQRtgznIM=@protonmail.com>
References: <IFHK0UDO3U_sfnnbdbElT0uOJ41lem0iyfnu38k7C1R-0pEvsKn0TOIz8uAr-aKTm4oWP6wW2J4EZNSRIOCrJI4SIBJd3qGanHrQRtgznIM=@protonmail.com>
Message-ID: <CAGQ70evgzDiTTbAoCc8HWxpmH5qjeZ9xhr4pVs897Y5nauZGRg@mail.gmail.com>

You could probably just clear the local cache (if you're confident there's
no attack) and it will re-sync to the older information from the repo, I
believe (have not tried this myself).

This would be fundamentally similar to the warning you get when accessing
an SSH host which has changed public key.

The SOURCE of the problem would be interesting though. Where is your backup
being done to (sounds like an external USB drive)? Perhaps there has been
data loss from that drive (sync not completed before you unplugged it)?

On Tue, Jan 1, 2019 at 1:23 AM ypm.tmp via Borgbackup <borgbackup at python.org>
wrote:

> [Note that English is not my native language, please excuse any typing
> errors
> and awful phrasing]
>
> Hi beautiful people!
>
> My name is ypm I've been using borg for some months now and I love how all
> the
> features it has makes it my ideal backup solution so far: incremental,
> compressed and encrypted backups <3
>
> Last week I've been having issues with it with my normal workflow. I would
> do
> the same thing and it would backup normally, but next time I would connect
> the
> drive, it would return the following error, with my usual 'borg create'
> command, as well as with 'borg list':
>
>     Cache, or information obtained from the security directory is newer
>     than repository - this is either an attack or unsafe
>     (multiple repos with same ID)
>
> Rendering the backup pretty much unusable, so I can't access my data even
> if I
> have my passwords, and I don't know what should my next course of action
> should
> be in order to both access my data and prevent this error from happening
> in the
> future. I'm familiar with some borg commands and performing simple tasks,
> but
> not with every single one of them. I can provide more details if any is
> needed.
>
> Thanks in advance and happy holidays!
>
> Regards,
> ypm
>
>
> Details
> *******
>
> borg version:      1.1.8
>
> _______________________________________________
> Borgbackup mailing list
> Borgbackup at python.org
> https://mail.python.org/mailman/listinfo/borgbackup
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20190102/08616bce/attachment.html>

From ypm.tmp at protonmail.com  Tue Jan  1 15:37:56 2019
From: ypm.tmp at protonmail.com (ypm.tmp)
Date: Tue, 01 Jan 2019 20:37:56 +0000
Subject: [Borgbackup] 'Multiple repos with the same ID' error
In-Reply-To: <CAGQ70evgzDiTTbAoCc8HWxpmH5qjeZ9xhr4pVs897Y5nauZGRg@mail.gmail.com>
References: <IFHK0UDO3U_sfnnbdbElT0uOJ41lem0iyfnu38k7C1R-0pEvsKn0TOIz8uAr-aKTm4oWP6wW2J4EZNSRIOCrJI4SIBJd3qGanHrQRtgznIM=@protonmail.com>
 <CAGQ70evgzDiTTbAoCc8HWxpmH5qjeZ9xhr4pVs897Y5nauZGRg@mail.gmail.com>
Message-ID: <W8BNUoaHn8Mx-_e9gLrWKsA4U5M5VXurpLhY0V2wWnsVhhmYBUq26JOUPIRIOHWPGF-8D-NdKmtVfEpDL2RH2Bk5iClFYc10EqpA6nTu6yE=@protonmail.com>

I had deleted the cache before writing my first message in order to try to fix
it. Now I did it again and the same error (Cache, or information obtained from
the security directory is newer than repository - this is either an attack or
unsafe (multiple repos with same ID) is being returned.

Yes, I do make backups to several external USB drives, and I have a script that
iterates to each drive sequentially, but never making multiple backups at once.
All syncs were always completed before I unplugged the drives.

I had read the wiki and the only info I found about this error on the wiki was
on the "All about JSON: How to develop frontends" page [1]

Does anybody know if there is any way that I can access the data of my backup
now?

Regards,
ypm

*************************************************************************************
[1]: https://borgbackup.readthedocs.io/en/stable/internals/frontends.html#message-ids


From billk at iinet.net.au  Sun Jan  6 02:01:21 2019
From: billk at iinet.net.au (Bill Kenworthy)
Date: Sun, 6 Jan 2019 15:01:21 +0800
Subject: [Borgbackup] include and exclude patterns
Message-ID: <591da1ef-da34-dee7-4cac-82bac85b0820@iinet.net.au>

Hi,

??? I am having problems figuring out how to include a directory within
another directory where I do not want anything else.

e.g, create a whole system backup for root "/" on a gentoo system

??? ??? exclude /usr/portage (contains many files/directories)

??? ??? include only /usr/portage/packages from that directory

Can anyone suggest a simple way to do this?

BillK




From devzero at web.de  Sun Jan  6 05:24:20 2019
From: devzero at web.de (devzero at web.de)
Date: Sun, 6 Jan 2019 11:24:20 +0100
Subject: [Borgbackup] limit local  cpu/bandwidth/iops usage
Message-ID: <trinity-e4f26ed6-a593-47d7-a5bc-3f1ed6d43589-1546770260020@3c-app-webde-bs42>

Hello, 

this does apply to remote/network bandwidth only - is that correct ?

https://borgbackup.readthedocs.io/en/stable/faq.html#is-there-a-way-to-limit-bandwidth-with-project-name

if yes, how to limit cpu/disk bandwidth/iops when using borg only on local machine ?

i have a backup job where borgbackup directly reads from zfs snapshot (virtual machines in proxmox) and i don't want the running VMs to suffer from the backup job's cpu/disk ressource usage.

regards
roland

From dave at gasaway.org  Sun Jan  6 12:12:52 2019
From: dave at gasaway.org (David Gasaway)
Date: Sun, 6 Jan 2019 09:12:52 -0800
Subject: [Borgbackup] limit local cpu/bandwidth/iops usage
In-Reply-To: <trinity-e4f26ed6-a593-47d7-a5bc-3f1ed6d43589-1546770260020@3c-app-webde-bs42>
References: <trinity-e4f26ed6-a593-47d7-a5bc-3f1ed6d43589-1546770260020@3c-app-webde-bs42>
Message-ID: <CAGsJ5AX=wyj4ma8d2=fiUeUO6Pff0inrXLv67=6H3OjC_Rd-ug@mail.gmail.com>

On Sun, Jan 6, 2019 at 2:24 AM <devzero at web.de> wrote:

> Hello,
>
> this does apply to remote/network bandwidth only - is that correct ?
>
>
> https://borgbackup.readthedocs.io/en/stable/faq.html#is-there-a-way-to-limit-bandwidth-with-project-name
>
> if yes, how to limit cpu/disk bandwidth/iops when using borg only on local
> machine ?
>
> i have a backup job where borgbackup directly reads from zfs snapshot
> (virtual machines in proxmox) and i don't want the running VMs to suffer
> from the backup job's cpu/disk ressource usage.
>

Have you looked into cgroups?

-- 
-:-:- David K. Gasaway
-:-:- Email: dave at gasaway.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20190106/75248bd7/attachment.html>

From dave at gasaway.org  Sun Jan  6 12:21:49 2019
From: dave at gasaway.org (David Gasaway)
Date: Sun, 6 Jan 2019 09:21:49 -0800
Subject: [Borgbackup] include and exclude patterns
In-Reply-To: <591da1ef-da34-dee7-4cac-82bac85b0820@iinet.net.au>
References: <591da1ef-da34-dee7-4cac-82bac85b0820@iinet.net.au>
Message-ID: <CAGsJ5AXGx=spYeZW+jFnmQx8FzjYugsmSSDr-5g3jWSO=JJRDg@mail.gmail.com>

On Sat, Jan 5, 2019 at 11:05 PM Bill Kenworthy <billk at iinet.net.au> wrote:

> Hi,
>
>     I am having problems figuring out how to include a directory within
> another directory where I do not want anything else.
>
> e.g, create a whole system backup for root "/" on a gentoo system
>
>         exclude /usr/portage (contains many files/directories)
>
>         include only /usr/portage/packages from that directory
>
> Can anyone suggest a simple way to do this?
>

Using --patterns-from format:

R /
+ /usr/portage/packages
- /usr/portage

It's important to use "-" not "!" as the latter will not recurse the
excluded folder, while "-" will.

-- 
-:-:- David K. Gasaway
-:-:- Email: dave at gasaway.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20190106/cb24253b/attachment.html>

From devzero at web.de  Sun Jan  6 17:46:21 2019
From: devzero at web.de (devzero at web.de)
Date: Sun, 6 Jan 2019 23:46:21 +0100
Subject: [Borgbackup] limit local cpu/bandwidth/iops usage
In-Reply-To: <CAGsJ5AX=wyj4ma8d2=fiUeUO6Pff0inrXLv67=6H3OjC_Rd-ug@mail.gmail.com>
References: <trinity-e4f26ed6-a593-47d7-a5bc-3f1ed6d43589-1546770260020@3c-app-webde-bs42>
 <CAGsJ5AX=wyj4ma8d2=fiUeUO6Pff0inrXLv67=6H3OjC_Rd-ug@mail.gmail.com>
Message-ID: <trinity-ecf4880a-d0f8-450d-83ee-529b4be4d117-1546814781512@3c-app-webde-bap62>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20190106/760c9448/attachment.html>

From tw at waldmann-edv.de  Mon Jan  7 05:15:56 2019
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Mon, 7 Jan 2019 11:15:56 +0100
Subject: [Borgbackup] limit local cpu/bandwidth/iops usage
In-Reply-To: <trinity-ecf4880a-d0f8-450d-83ee-529b4be4d117-1546814781512@3c-app-webde-bap62>
References: <trinity-e4f26ed6-a593-47d7-a5bc-3f1ed6d43589-1546770260020@3c-app-webde-bs42>
 <CAGsJ5AX=wyj4ma8d2=fiUeUO6Pff0inrXLv67=6H3OjC_Rd-ug@mail.gmail.com>
 <trinity-ecf4880a-d0f8-450d-83ee-529b4be4d117-1546814781512@3c-app-webde-bap62>
Message-ID: <0f0cf48b-459f-17c5-288f-f64407e94580@waldmann-edv.de>

>     this does apply to remote/network bandwidth only - is that correct ?
> 
>     https://borgbackup.readthedocs.io/en/stable/faq.html#is-there-a-way-to-limit-bandwidth-with-project-name
> 
>     if yes, how to limit cpu/disk bandwidth/iops when using borg only on
>     local machine ?

You can use ssh://user at localhost/... for the repo.

>     i have a backup job where borgbackup directly reads from zfs
>     snapshot (virtual machines in proxmox) and i don't want the running
>     VMs to suffer from the backup job's cpu/disk ressource usage.

borg will not use more than 1 core anyway as it is still single-threaded.

the default lz4 compression is also very quick and usually saves more
resources than it needs for compressing.

of course there is some I/O impact, especially if you back up the VMs
from the outside, so borg has to read the complete disk image, chunk and
hash it before it can know about what actually has changed.

sparse files help here as the sparse part does not really cause disk i/o.

From eric at in3x.io  Mon Jan  7 07:44:33 2019
From: eric at in3x.io (eric)
Date: Mon, 7 Jan 2019 07:44:33 -0500
Subject: [Borgbackup] Remote monitoring of Borg results
Message-ID: <dc5a0e05-2346-459c-b1ac-fa1b45efa204@edison.tech>

 

 I have a few instances of borg+borgmatic running at different client sites and wonder how others monitor the success or failure of each run? 

 
 

 
I was sending the results of running borgmatic via email but unfortunately, that's not possible/reliable anymore.
 

 
How do others solve this problem?
 

 
Thanks in advance for your insight.
 
---eric
 

 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20190107/971b16b1/attachment.html>

From qzwx2007 at gmail.com  Tue Jan  8 07:05:48 2019
From: qzwx2007 at gmail.com (JK)
Date: Tue, 8 Jan 2019 14:05:48 +0200
Subject: [Borgbackup] Remote monitoring of Borg results
In-Reply-To: <dc5a0e05-2346-459c-b1ac-fa1b45efa204@edison.tech>
References: <dc5a0e05-2346-459c-b1ac-fa1b45efa204@edison.tech>
Message-ID: <0e26e638-e437-4030-1cc7-fa69a27e017d@gmail.com>

I use bash scripts to launch borg. Scripts are launced manually when 
needed and from cron according a schedule.

Prune same way: manually script launch when needed and scheduled from cron.

With cron I use chronic:? https://github.com/docwhat/chronic , chronic 
emails me only in case of error.

As a double check I monitor the execution of backups with pandora: 
https://pandorafms.org/en/? Most of the bash scripts I use in a 
scheduled way (from cron) update a task specific timestamp file as a 
sign of success. Pandora then alarms me if any of these timestamps are 
too old.

jk

On 7.1.2019 14.44, eric wrote:
>
> I have a few instances of borg+borgmatic running at different client 
> sites and wonder how others monitor the success or failure of each run?
>
> I was sending the results of running borgmatic via email but 
> unfortunately, that's not possible/reliable anymore.
>
> How do others solve this problem?
>
> Thanks in advance for your insight.
> ---eric
>
>
> _______________________________________________
> Borgbackup mailing list
> Borgbackup at python.org
> https://mail.python.org/mailman/listinfo/borgbackup
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20190108/ab45dc92/attachment.html>

From eric at in3x.io  Tue Jan  8 09:28:42 2019
From: eric at in3x.io (eric)
Date: Tue, 8 Jan 2019 09:28:42 -0500
Subject: [Borgbackup] Remote monitoring of Borg results
In-Reply-To: <0e26e638-e437-4030-1cc7-fa69a27e017d@gmail.com>
 <dc5a0e05-2346-459c-b1ac-fa1b45efa204@edison.tech>
References: <dc5a0e05-2346-459c-b1ac-fa1b45efa204@edison.tech>
Message-ID: <@localhost>

 
 

 Thanks for the great references to a couple of tools. the one thing I haven't solved yet is how to get results from cron without using email. Ideally it should be some sort of a centralized service / channel so I can get all of my chrome results in one place but still have a history so I can go back and look at them later.  

 
Yes, I'm quite aware that I'm replicating the behavior of update by email but unfortunately, at more than one of my clients sites, sending messages out by email is not possible as I don't control the mail system.
 

 
I'm half tempted to set up dedicated email server just to receive system notifications from different clients.
 
 

 
 
 
 
>  
> On Jan 8, 2019 at 7:05 AM,  <JK>  wrote:
>  
>
> With cron I use chronic:     https://github.com/docwhat/chronic  , chronic emails me only in case of error.
>
>  
>
> As a double check I monitor the execution of backups with pandora:  https://pandorafms.org/en/    Most of the bash scripts I use in a scheduled way (from cron) update a task specific timestamp file as a sign of success. Pandora then alarms me if any of these timestamps are too old.
>
>  
>
> jk
>
>
>  
> On 7.1.2019 14.44, eric wrote:
>  
> >    
> >
> >  I have a few instances of borg+borgmatic running at different client sites and wonder how others monitor the success or failure of each run?  
> >
> >  
> > I was sending the results of running borgmatic via email but unfortunately, that's not possible/reliable anymore.
> >  
> >
> >  
> > How do others solve this problem?
> >  
> >
> >  
> > Thanks in advance for your insight.
> >  
> > ---eric
> >  
> >
> >  
> >    
> >  _______________________________________________ Borgbackup mailing list  Borgbackup at python.org   https://mail.python.org/mailman/listinfo/borgbackup  
>
>  
>  
>  _______________________________________________ Borgbackup mailing list Borgbackup at python.org https://mail.python.org/mailman/listinfo/borgbackup 
>  
   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20190108/7b537935/attachment.html>

From jruiz at octanio.com  Wed Jan  9 05:54:18 2019
From: jruiz at octanio.com (=?utf-8?Q?Jos=C3=A9_Ruiz?=)
Date: Wed, 9 Jan 2019 11:54:18 +0100
Subject: [Borgbackup] Remote monitoring of Borg results
In-Reply-To: <@localhost>
References: <dc5a0e05-2346-459c-b1ac-fa1b45efa204@edison.tech> <@localhost>
Message-ID: <9F838175-7906-432F-A9E5-A781D150AE6A@octanio.com>

Hi Eric,

You could check the status of your borg process and then send the result to an InfluxDB (with a simple curl). Then, you could create a simple Grafana panel to see the results.

Cheers,
Jose

> On 8 Jan 2019, at 15:28, eric <eric at in3x.io <mailto:eric at in3x.io>> wrote:
> 
> 
> Thanks for the great references to a couple of tools. the one thing I haven't solved yet is how to get results from cron without using email. Ideally it should be some sort of a centralized service / channel so I can get all of my chrome results in one place but still have a history so I can go back and look at them later.
> 
> Yes, I'm quite aware that I'm replicating the behavior of update by email but unfortunately, at more than one of my clients sites, sending messages out by email is not possible as I don't control the mail system.
> 
> I'm half tempted to set up dedicated email server just to receive system notifications from different clients.
> 
>> On Jan 8, 2019 at 7:05 AM, <JK <mailto:qzwx2007 at gmail.com>> wrote:
>> With cron I use chronic:  https://github.com/docwhat/chronic <https://github.com/docwhat/chronic> , chronic emails me only in case of error.
>> 
>> As a double check I monitor the execution of backups with pandora: https://pandorafms.org/en/ <https://pandorafms.org/en/>  Most of the bash scripts I use in a scheduled way (from cron) update a task specific timestamp file as a sign of success. Pandora then alarms me if any of these timestamps are too old.
>> 
>> jk
>> 
>> On 7.1.2019 14.44, eric wrote:
>>> 
>>> I have a few instances of borg+borgmatic running at different client sites and wonder how others monitor the success or failure of each run?
>>> 
>>> I was sending the results of running borgmatic via email but unfortunately, that's not possible/reliable anymore.
>>> 
>>> How do others solve this problem?
>>> 
>>> Thanks in advance for your insight.
>>> ---eric
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> Borgbackup mailing list
>>> Borgbackup at python.org <mailto:Borgbackup at python.org>
>>> https://mail.python.org/mailman/listinfo/borgbackup <https://mail.python.org/mailman/listinfo/borgbackup>
>> 
>> _______________________________________________
>> Borgbackup mailing list
>> Borgbackup at python.org <mailto:Borgbackup at python.org>
>> https://mail.python.org/mailman/listinfo/borgbackup <https://mail.python.org/mailman/listinfo/borgbackup>
>> 
> _______________________________________________
> Borgbackup mailing list
> Borgbackup at python.org <mailto:Borgbackup at python.org>
> https://mail.python.org/mailman/listinfo/borgbackup <https://mail.python.org/mailman/listinfo/borgbackup>




 <https://www.octanio.com/>
Jose Ruiz Garc?a ? Administrador de sistemas
Octanio Sistemas Inform?ticos
 +34 952020575 <tel:+34 952020575> ? jruiz at octanio.com <mailto:jruiz at octanio.com>
 www.octanio.com <https://www.octanio.com/>
 jruiz.octanio <http://jruiz.octanio/>





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20190109/cefd4a40/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20190109/cefd4a40/attachment.sig>

From a_kemper at gmx.de  Mon Jan 14 07:30:37 2019
From: a_kemper at gmx.de (Andreas Kemper)
Date: Mon, 14 Jan 2019 13:30:37 +0100
Subject: [Borgbackup] Borg verify fails for big archive on webdav share
Message-ID: <5e8716e8-eca4-25fb-ca1a-ed29cab727c9@gmx.de>

Hi,

I'm using Borgbackup for quite some time on different machines without 
any problems so far, while in particular appreciating the compact 
archive size.

Now I've created a new archive for approx. ~ 25GB of files from my 
ownCloud installation. Creating / updating / pruning the archive works 
well, but when running a "borg check -v (--repository-only)" it 
repetitively crashes as printed below. Doing the same check on a local 
copy of the respective archive works well, while even increasing the 
lock-wait value doesn't help with the webdav mount (1&1 Online-Speicher).

Is this something I should file as a bug, alternatively any tips for a 
workaround (apart from rsyncing a local archive for instance)?

Thx
Andreas


root at vserver:/# borg check -v --repository-only /media/1und1/backup/www
Starting repository check
Local Exception
Traceback (most recent call last):
   File "/usr/lib/python3/dist-packages/borg/repository.py", line 1283, 
in get_fd
     return self.fds[segment]
   File "/usr/lib/python3/dist-packages/borg/lrucache.py", line 21, in 
__getitem__
     value = self._cache[key]  # raise KeyError if not found
KeyError: 25

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
   File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4434, in 
main
     exit_code = archiver.run(args)
   File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4366, in run
     return set_ec(func(args))
   File "/usr/lib/python3/dist-packages/borg/archiver.py", line 152, in 
wrapper
     return method(self, args, repository=repository, **kwargs)
   File "/usr/lib/python3/dist-packages/borg/archiver.py", line 313, in 
do_check
     if not repository.check(repair=args.repair, 
save_space=args.save_space):
   File "/usr/lib/python3/dist-packages/borg/repository.py", line 911, 
in check
     objects = list(self.io.iter_objects(segment))
   File "/usr/lib/python3/dist-packages/borg/repository.py", line 1328, 
in iter_objects
     fd = self.get_fd(segment)
   File "/usr/lib/python3/dist-packages/borg/repository.py", line 1285, 
in get_fd
     fd = open(self.segment_filename(segment), 'rb')
OSError: [Errno 5] Input/output error: '/media/1und1/backup/www/data/0/25'

Platform: Linux vserver 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 
14:45:28 UTC 2018 x86_64
Linux: Ubuntu 18.04 bionic
Borg: 1.1.7  Python: CPython 3.6.7
PID: 3204  CWD: /
sys.argv: ['/usr/bin/borg', 'check', '-v', '--repository-only', 
'/media/1und1/backup/www']
SSH_ORIGINAL_COMMAND: None

From imperator at jedimail.de  Mon Jan 14 07:44:30 2019
From: imperator at jedimail.de (Imperator)
Date: Mon, 14 Jan 2019 13:44:30 +0100
Subject: [Borgbackup] Borg verify fails for big archive on webdav share
In-Reply-To: <5e8716e8-eca4-25fb-ca1a-ed29cab727c9@gmx.de>
References: <5e8716e8-eca4-25fb-ca1a-ed29cab727c9@gmx.de>
Message-ID: <b3af97bd-b4cb-973c-f779-bf320f3b22a0@jedimail.de>

Hi Andreas,

this is a WebDAV/davfs2 issue. You need to have enough free space left
on /var/cache/davfs2 for caching your whole borg repository.

My work-around is to mount additional partition on /var/cache/davfs2
since my root fs is too small.

Sascha

Am 14.01.19 um 13:30 schrieb Andreas Kemper:
> Hi,
>
> I'm using Borgbackup for quite some time on different machines without
> any problems so far, while in particular appreciating the compact
> archive size.
>
> Now I've created a new archive for approx. ~ 25GB of files from my
> ownCloud installation. Creating / updating / pruning the archive works
> well, but when running a "borg check -v (--repository-only)" it
> repetitively crashes as printed below. Doing the same check on a local
> copy of the respective archive works well, while even increasing the
> lock-wait value doesn't help with the webdav mount (1&1 Online-Speicher).
>
> Is this something I should file as a bug, alternatively any tips for a
> workaround (apart from rsyncing a local archive for instance)?
>
> Thx
> Andreas
>
>
> root at vserver:/# borg check -v --repository-only /media/1und1/backup/www
> Starting repository check
> Local Exception
> Traceback (most recent call last):
> ? File "/usr/lib/python3/dist-packages/borg/repository.py", line 1283,
> in get_fd
> ??? return self.fds[segment]
> ? File "/usr/lib/python3/dist-packages/borg/lrucache.py", line 21, in
> __getitem__
> ??? value = self._cache[key]? # raise KeyError if not found
> KeyError: 25
>
> During handling of the above exception, another exception occurred:
>
> Traceback (most recent call last):
> ? File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4434,
> in main
> ??? exit_code = archiver.run(args)
> ? File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4366,
> in run
> ??? return set_ec(func(args))
> ? File "/usr/lib/python3/dist-packages/borg/archiver.py", line 152, in
> wrapper
> ??? return method(self, args, repository=repository, **kwargs)
> ? File "/usr/lib/python3/dist-packages/borg/archiver.py", line 313, in
> do_check
> ??? if not repository.check(repair=args.repair,
> save_space=args.save_space):
> ? File "/usr/lib/python3/dist-packages/borg/repository.py", line 911,
> in check
> ??? objects = list(self.io.iter_objects(segment))
> ? File "/usr/lib/python3/dist-packages/borg/repository.py", line 1328,
> in iter_objects
> ??? fd = self.get_fd(segment)
> ? File "/usr/lib/python3/dist-packages/borg/repository.py", line 1285,
> in get_fd
> ??? fd = open(self.segment_filename(segment), 'rb')
> OSError: [Errno 5] Input/output error:
> '/media/1und1/backup/www/data/0/25'
>
> Platform: Linux vserver 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6
> 14:45:28 UTC 2018 x86_64
> Linux: Ubuntu 18.04 bionic
> Borg: 1.1.7? Python: CPython 3.6.7
> PID: 3204? CWD: /
> sys.argv: ['/usr/bin/borg', 'check', '-v', '--repository-only',
> '/media/1und1/backup/www']
> SSH_ORIGINAL_COMMAND: None
> _______________________________________________
> Borgbackup mailing list
> Borgbackup at python.org
> https://mail.python.org/mailman/listinfo/borgbackup

From mhossbach at gmail.com  Thu Jan 17 03:10:36 2019
From: mhossbach at gmail.com (=?UTF-8?Q?Martin_Ho=C3=9Fbach?=)
Date: Thu, 17 Jan 2019 09:10:36 +0100
Subject: [Borgbackup] Possible Bug?
Message-ID: <CAPSx4k7rTH4THDGV2nSmemq=E_ZazVevgvvu7KB5Y2ZUxeddoQ@mail.gmail.com>

Dear community,

I've run into a problem creating a backup. Borg 1.1.8 crashes with this
error report when creating a certain archive:


Processing files ...


Cleaned up 0 uncommitted segment files (== everything after segment
614750).

Local Exception
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 4436, in
main
    exit_code = archiver.run(args)
  File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 4368, in
run
    return set_ec(func(args))
  File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 152, in
wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 555, in
do_create
    create_inner(archive, cache)
  File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 518, in
create_inner
    read_special=args.read_special, dry_run=dry_run, st=st)
  File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 632, in
_process
    read_special=read_special, dry_run=dry_run)
  File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 632, in
_process
    read_special=read_special, dry_run=dry_run)
  File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 606, in
_process
    status = archive.process_file(path, st, cache)
  File "/usr/lib/python3.7/site-packages/borg/archive.py", line 1028, in
process_file
    self.chunk_file(item, cache, self.stats,
backup_io_iter(self.chunker.chunkify(fd, fh)))
  File "/usr/lib/python3.7/site-packages/borg/archive.py", line 956, in
chunk_file
    item.chunks.append(chunk_processor(data))
  File "/usr/lib/python3.7/site-packages/borg/archive.py", line 944, in
chunk_processor
    chunk_entry = cache.add_chunk(self.key.id_hash(data), data, stats,
wait=False)
  File "/usr/lib/python3.7/site-packages/borg/cache.py", line 898, in
add_chunk
    self.repository.put(id, data, wait=wait)
  File "/usr/lib/python3.7/site-packages/borg/repository.py", line 1076, in
put
    self.prepare_txn(self.get_transaction_id())
  File "/usr/lib/python3.7/site-packages/borg/repository.py", line 504, in
prepare_txn
    hints = msgpack.unpack(fd)
  File "/usr/lib/python3.7/site-packages/msgpack/__init__.py", line 57, in
unpack
    return unpackb(data, **kwargs)
  File "msgpack/_unpacker.pyx", line 187, in msgpack._cmsgpack.unpackb
ValueError: 338108 exceeds max_map_len(32768)

Platform: Linux vault82 4.20.1-arch1-1-ARCH #1 SMP PREEMPT Wed Jan 9
20:25:43 UTC 2019 x86_64
Linux: arch
Borg: 1.1.8  Python: CPython 3.7.2
PID: 5997  CWD: /mnt/backup/clients/mirror/root
sys.argv: ['/usr/bin/borg', 'create', '--debug', '-p', '-s', '-C', 'zlib',
'--numeric-owner', '-x', '-e', 're:/swapfile1', '-e', 're:/video-tablet/',
'-e', 're:/pagefile.sys', '-e', 're:/hiberfil.sys', '-e',
're:/swapfile.sys', '--exclude-caches', '--keep-exclude-tags',
'::mirror-root-{now:%s}', '.']
SSH_ORIGINAL_COMMAND: None


The same backup runs smoothly when I use a newly-created repository, and as
well when I install the git version of borg (1.2.0.dev737+g4d29747f). Also,
borg check finds no problems in my repository.

So, I could now simply wait until borg 1.2.0 is released, but I could also
help to track that issue in borg. However, I'm not really fluent in Python,
so I could use some pointers what the developers would prefer, and where to
start?

Martin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20190117/103f14e9/attachment.html>

From tw at waldmann-edv.de  Thu Jan 17 08:26:25 2019
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Thu, 17 Jan 2019 14:26:25 +0100
Subject: [Borgbackup] Possible Bug?
In-Reply-To: <CAPSx4k7rTH4THDGV2nSmemq=E_ZazVevgvvu7KB5Y2ZUxeddoQ@mail.gmail.com>
References: <CAPSx4k7rTH4THDGV2nSmemq=E_ZazVevgvvu7KB5Y2ZUxeddoQ@mail.gmail.com>
Message-ID: <3942a904-d1d9-c8ff-07de-c028a6af8660@waldmann-edv.de>


On 1/17/19 9:10 AM, Martin Ho?bach wrote:

> ? File "msgpack/_unpacker.pyx", line 187, in msgpack._cmsgpack.unpackb
> ValueError: 338108 exceeds max_map_len(32768)
> 
> Platform: Linux vault82 4.20.1-arch1-1-ARCH #1 SMP PREEMPT Wed Jan 9
> 20:25:43 UTC 2019 x86_64
> Linux: arch??
> Borg: 1.1.8? Python: CPython 3.7.2

See there:

https://github.com/borgbackup/borg/issues/4251


-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393

From mhossbach at gmail.com  Thu Jan 17 08:30:35 2019
From: mhossbach at gmail.com (=?UTF-8?Q?Martin_Ho=C3=9Fbach?=)
Date: Thu, 17 Jan 2019 14:30:35 +0100
Subject: [Borgbackup] Possible Bug?
In-Reply-To: <3942a904-d1d9-c8ff-07de-c028a6af8660@waldmann-edv.de>
References: <CAPSx4k7rTH4THDGV2nSmemq=E_ZazVevgvvu7KB5Y2ZUxeddoQ@mail.gmail.com>
 <3942a904-d1d9-c8ff-07de-c028a6af8660@waldmann-edv.de>
Message-ID: <CAPSx4k5a2_tUbyKNPeR8XSxTYiT4rKDfrsPSsW9peXES+=bpAQ@mail.gmail.com>

I don't know how I could have missed that. Thank you!

On Thu, Jan 17, 2019 at 2:26 PM Thomas Waldmann <tw at waldmann-edv.de> wrote:

>
> On 1/17/19 9:10 AM, Martin Ho?bach wrote:
>
> >   File "msgpack/_unpacker.pyx", line 187, in msgpack._cmsgpack.unpackb
> > ValueError: 338108 exceeds max_map_len(32768)
> >
> > Platform: Linux vault82 4.20.1-arch1-1-ARCH #1 SMP PREEMPT Wed Jan 9
> > 20:25:43 UTC 2019 x86_64
> > Linux: arch
> > Borg: 1.1.8  Python: CPython 3.7.2
>
> See there:
>
> https://github.com/borgbackup/borg/issues/4251
>
>
> --
>
> GPG ID: 9F88FB52FAF7B393
> GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393
> _______________________________________________
> Borgbackup mailing list
> Borgbackup at python.org
> https://mail.python.org/mailman/listinfo/borgbackup
>


-- 
Martin Ho?bach
mhossbach at gmail.com
Am Filmlager 10, 12555 Berlin
Tel. +49-30-37468271, Mobil: +49-1525-6535234
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20190117/44b60833/attachment.html>

From ngoonee.talk at gmail.com  Fri Jan 18 05:40:33 2019
From: ngoonee.talk at gmail.com (Oon-Ee Ng)
Date: Fri, 18 Jan 2019 18:40:33 +0800
Subject: [Borgbackup] Possible Bug?
In-Reply-To: <CAPSx4k5a2_tUbyKNPeR8XSxTYiT4rKDfrsPSsW9peXES+=bpAQ@mail.gmail.com>
References: <CAPSx4k7rTH4THDGV2nSmemq=E_ZazVevgvvu7KB5Y2ZUxeddoQ@mail.gmail.com>
 <3942a904-d1d9-c8ff-07de-c028a6af8660@waldmann-edv.de>
 <CAPSx4k5a2_tUbyKNPeR8XSxTYiT4rKDfrsPSsW9peXES+=bpAQ@mail.gmail.com>
Message-ID: <CAGQ70esarjuXSaP4F3k=rBO2gFcWeG8H=8Gz4zRhu=MAdRMTog@mail.gmail.com>

Interestingly enough, borg create works but borg mount doesn't here (Arch
system, so with python-msgpack=0.6.0). Is there some threshold to trigger
this, archive size or index size wise?

On Thu, Jan 17, 2019 at 9:30 PM Martin Ho?bach <mhossbach at gmail.com> wrote:

> I don't know how I could have missed that. Thank you!
>
> On Thu, Jan 17, 2019 at 2:26 PM Thomas Waldmann <tw at waldmann-edv.de>
> wrote:
>
>>
>> On 1/17/19 9:10 AM, Martin Ho?bach wrote:
>>
>> >   File "msgpack/_unpacker.pyx", line 187, in msgpack._cmsgpack.unpackb
>> > ValueError: 338108 exceeds max_map_len(32768)
>> >
>> > Platform: Linux vault82 4.20.1-arch1-1-ARCH #1 SMP PREEMPT Wed Jan 9
>> > 20:25:43 UTC 2019 x86_64
>> > Linux: arch
>> > Borg: 1.1.8  Python: CPython 3.7.2
>>
>> See there:
>>
>> https://github.com/borgbackup/borg/issues/4251
>>
>>
>> --
>>
>> GPG ID: 9F88FB52FAF7B393
>> GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393
>> _______________________________________________
>> Borgbackup mailing list
>> Borgbackup at python.org
>> https://mail.python.org/mailman/listinfo/borgbackup
>>
>
>
> --
> Martin Ho?bach
> mhossbach at gmail.com
> Am Filmlager 10, 12555 Berlin
> Tel. +49-30-37468271, Mobil: +49-1525-6535234
> _______________________________________________
> Borgbackup mailing list
> Borgbackup at python.org
> https://mail.python.org/mailman/listinfo/borgbackup
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20190118/526ba404/attachment.html>

From stefan at sbuehler.de  Sat Jan 19 07:23:36 2019
From: stefan at sbuehler.de (Stefan Buehler)
Date: Sat, 19 Jan 2019 13:23:36 +0100
Subject: [Borgbackup] Mac OS Mojave
Message-ID: <F9A2EAD8-8A91-4D6B-8478-2972E9C86E83@sbuehler.de>

Dear Borg gurus,

I run borg on a Mac. Since I updated to Mojave, I get error messages of this kind:

/users/xxx/Pictures/Photos Library.photoslibrary: scandir: [Errno 1] Operation not permitted: '/users/xxx/Pictures/Photos Library.photoslibrary?

I suspect this has to do with Mojave being more restrictive about letting apps access certain files or directories. Do you think giving ?full disk access? to borg will solve this? Or should I give that to the shell script that runs borg for me, or??

Perhaps there are other Mac users out there that have also run into this and already solved it.

Best wishes,

Stefan


From devzero at web.de  Sat Jan 19 10:51:55 2019
From: devzero at web.de (devzero at web.de)
Date: Sat, 19 Jan 2019 16:51:55 +0100
Subject: [Borgbackup] Mac OS Mojave
In-Reply-To: <F9A2EAD8-8A91-4D6B-8478-2972E9C86E83@sbuehler.de>
References: <F9A2EAD8-8A91-4D6B-8478-2972E9C86E83@sbuehler.de>
Message-ID: <trinity-d3183df4-7164-4364-a245-0f5ac5f00aa1-1547913115458@3c-app-webde-bs45>

you may have a look at https://github.com/restic/restic/issues/2051


> Gesendet: Samstag, 19. Januar 2019 um 13:23 Uhr
> Von: "Stefan Buehler" <stefan at sbuehler.de>
> An: borgbackup at python.org
> Betreff: [Borgbackup] Mac OS Mojave
>
> Dear Borg gurus,
> 
> I run borg on a Mac. Since I updated to Mojave, I get error messages of this kind:
> 
> /users/xxx/Pictures/Photos Library.photoslibrary: scandir: [Errno 1] Operation not permitted: '/users/xxx/Pictures/Photos Library.photoslibrary?
> 
> I suspect this has to do with Mojave being more restrictive about letting apps access certain files or directories. Do you think giving ?full disk access? to borg will solve this? Or should I give that to the shell script that runs borg for me, or??
> 
> Perhaps there are other Mac users out there that have also run into this and already solved it.
> 
> Best wishes,
> 
> Stefan
> 
> _______________________________________________
> Borgbackup mailing list
> Borgbackup at python.org
> https://mail.python.org/mailman/listinfo/borgbackup
>

From stefan at sbuehler.de  Sat Jan 19 11:58:20 2019
From: stefan at sbuehler.de (Stefan Buehler)
Date: Sat, 19 Jan 2019 17:58:20 +0100
Subject: [Borgbackup] Mac OS Mojave
In-Reply-To: <trinity-d3183df4-7164-4364-a245-0f5ac5f00aa1-1547913115458@3c-app-webde-bs45>
References: <F9A2EAD8-8A91-4D6B-8478-2972E9C86E83@sbuehler.de>
 <trinity-d3183df4-7164-4364-a245-0f5ac5f00aa1-1547913115458@3c-app-webde-bs45>
Message-ID: <CFC4CF77-2E4B-4C91-8ADA-E360A5329763@sbuehler.de>

Hi Devzero,

thanks for pointing me to this. Oh dear, it confirms my fear that there probably is no simple solution. It took me a lot of trial and error to get my script setup to work initially, but since then I has been wonderfully stable. I hate the idea of going through a similar trial and error setup and debugging process again.

I wish somebody would make a high-level Mac tool based on borg, that I could just install and use (and that has this issue figured out, of course). Perhaps this exists already and I missed it?

Anyway, of course I realise that this is not in principle a borg issue. But in some way it is, since it probably makes the program almost useless for many typical users on Mac.

All the best,

Stefan

> On 19. Jan 2019, at 16:51, devzero at web.de wrote:
> 
> you may have a look at https://github.com/restic/restic/issues/2051
> 
> 
>> Gesendet: Samstag, 19. Januar 2019 um 13:23 Uhr
>> Von: "Stefan Buehler" <stefan at sbuehler.de>
>> An: borgbackup at python.org
>> Betreff: [Borgbackup] Mac OS Mojave
>> 
>> Dear Borg gurus,
>> 
>> I run borg on a Mac. Since I updated to Mojave, I get error messages of this kind:
>> 
>> /users/xxx/Pictures/Photos Library.photoslibrary: scandir: [Errno 1] Operation not permitted: '/users/xxx/Pictures/Photos Library.photoslibrary?
>> 
>> I suspect this has to do with Mojave being more restrictive about letting apps access certain files or directories. Do you think giving ?full disk access? to borg will solve this? Or should I give that to the shell script that runs borg for me, or??
>> 
>> Perhaps there are other Mac users out there that have also run into this and already solved it.
>> 
>> Best wishes,
>> 
>> Stefan
>> 
>> _______________________________________________
>> Borgbackup mailing list
>> Borgbackup at python.org
>> https://mail.python.org/mailman/listinfo/borgbackup
>> 


From devzero at web.de  Sat Jan 19 12:51:39 2019
From: devzero at web.de (devzero at web.de)
Date: Sat, 19 Jan 2019 18:51:39 +0100
Subject: [Borgbackup] Mac OS Mojave
In-Reply-To: <CFC4CF77-2E4B-4C91-8ADA-E360A5329763@sbuehler.de>
References: <F9A2EAD8-8A91-4D6B-8478-2972E9C86E83@sbuehler.de>
 <trinity-d3183df4-7164-4364-a245-0f5ac5f00aa1-1547913115458@3c-app-webde-bs45>
 <CFC4CF77-2E4B-4C91-8ADA-E360A5329763@sbuehler.de>
Message-ID: <trinity-6e29c347-e02f-48c8-b92d-762d6bea50b5-1547920299961@3c-app-webde-bs62>

mhh, apple knows what's good for you! ;)

did you know that apple store employees are forbidden to say words like "problem" ? :D  ( https://www.theguardian.com/technology/2018/dec/03/claps-and-cheers-apple-stores-carefully-managed-drama )

i hate such things..  breaking functionality because of security and not telling how to properly handle those broken things and simply being ignorant. apple is a very ignorant and patronzing company, imho... 

maybe this taks you a step further?

https://n8henrie.com/2018/11/how-to-give-full-disk-access-to-a-binary-in-macos-mojave/

please keep us posted...

regards
roland

> Gesendet: Samstag, 19. Januar 2019 um 17:58 Uhr
> Von: "Stefan Buehler" <stefan at sbuehler.de>
> An: devzero at web.de
> Cc: borgbackup at python.org
> Betreff: Re: [Borgbackup] Mac OS Mojave
>
> Hi Devzero,
> 
> thanks for pointing me to this. Oh dear, it confirms my fear that there probably is no simple solution. It took me a lot of trial and error to get my script setup to work initially, but since then I has been wonderfully stable. I hate the idea of going through a similar trial and error setup and debugging process again.
> 
> I wish somebody would make a high-level Mac tool based on borg, that I could just install and use (and that has this issue figured out, of course). Perhaps this exists already and I missed it?
> 
> Anyway, of course I realise that this is not in principle a borg issue. But in some way it is, since it probably makes the program almost useless for many typical users on Mac.
> 
> All the best,
> 
> Stefan
> 
> > On 19. Jan 2019, at 16:51, devzero at web.de wrote:
> > 
> > you may have a look at https://github.com/restic/restic/issues/2051
> > 
> > 
> >> Gesendet: Samstag, 19. Januar 2019 um 13:23 Uhr
> >> Von: "Stefan Buehler" <stefan at sbuehler.de>
> >> An: borgbackup at python.org
> >> Betreff: [Borgbackup] Mac OS Mojave
> >> 
> >> Dear Borg gurus,
> >> 
> >> I run borg on a Mac. Since I updated to Mojave, I get error messages of this kind:
> >> 
> >> /users/xxx/Pictures/Photos Library.photoslibrary: scandir: [Errno 1] Operation not permitted: '/users/xxx/Pictures/Photos Library.photoslibrary?
> >> 
> >> I suspect this has to do with Mojave being more restrictive about letting apps access certain files or directories. Do you think giving ?full disk access? to borg will solve this? Or should I give that to the shell script that runs borg for me, or??
> >> 
> >> Perhaps there are other Mac users out there that have also run into this and already solved it.
> >> 
> >> Best wishes,
> >> 
> >> Stefan
> >> 
> >> _______________________________________________
> >> Borgbackup mailing list
> >> Borgbackup at python.org
> >> https://mail.python.org/mailman/listinfo/borgbackup
> >> 
> 
>

From stefan at sbuehler.de  Fri Jan 25 08:39:37 2019
From: stefan at sbuehler.de (Stefan Buehler)
Date: Fri, 25 Jan 2019 14:39:37 +0100
Subject: [Borgbackup] Mac OS Mojave
In-Reply-To: <trinity-6e29c347-e02f-48c8-b92d-762d6bea50b5-1547920299961@3c-app-webde-bs62>
References: <F9A2EAD8-8A91-4D6B-8478-2972E9C86E83@sbuehler.de>
 <trinity-d3183df4-7164-4364-a245-0f5ac5f00aa1-1547913115458@3c-app-webde-bs45>
 <CFC4CF77-2E4B-4C91-8ADA-E360A5329763@sbuehler.de>
 <trinity-6e29c347-e02f-48c8-b92d-762d6bea50b5-1547920299961@3c-app-webde-bs62>
Message-ID: <FC23A578-6356-455C-ABF1-500EC93529BF@sbuehler.de>

Dear Roland, dear all,

I have tried the solution described in Roland?s link and can confirm that it works. (To make a small binary executable with full disk access that then executes the backup shell script.)

Thanks for your help!

Stefan

> On 19. Jan 2019, at 18:51, devzero at web.de wrote:
> 
> mhh, apple knows what's good for you! ;)
> 
> did you know that apple store employees are forbidden to say words like "problem" ? :D  ( https://www.theguardian.com/technology/2018/dec/03/claps-and-cheers-apple-stores-carefully-managed-drama )
> 
> i hate such things..  breaking functionality because of security and not telling how to properly handle those broken things and simply being ignorant. apple is a very ignorant and patronzing company, imho... 
> 
> maybe this taks you a step further?
> 
> https://n8henrie.com/2018/11/how-to-give-full-disk-access-to-a-binary-in-macos-mojave/
> 
> please keep us posted...
> 
> regards
> roland
> 
>> Gesendet: Samstag, 19. Januar 2019 um 17:58 Uhr
>> Von: "Stefan Buehler" <stefan at sbuehler.de>
>> An: devzero at web.de
>> Cc: borgbackup at python.org
>> Betreff: Re: [Borgbackup] Mac OS Mojave
>> 
>> Hi Devzero,
>> 
>> thanks for pointing me to this. Oh dear, it confirms my fear that there probably is no simple solution. It took me a lot of trial and error to get my script setup to work initially, but since then I has been wonderfully stable. I hate the idea of going through a similar trial and error setup and debugging process again.
>> 
>> I wish somebody would make a high-level Mac tool based on borg, that I could just install and use (and that has this issue figured out, of course). Perhaps this exists already and I missed it?
>> 
>> Anyway, of course I realise that this is not in principle a borg issue. But in some way it is, since it probably makes the program almost useless for many typical users on Mac.
>> 
>> All the best,
>> 
>> Stefan
>> 
>>> On 19. Jan 2019, at 16:51, devzero at web.de wrote:
>>> 
>>> you may have a look at https://github.com/restic/restic/issues/2051
>>> 
>>> 
>>>> Gesendet: Samstag, 19. Januar 2019 um 13:23 Uhr
>>>> Von: "Stefan Buehler" <stefan at sbuehler.de>
>>>> An: borgbackup at python.org
>>>> Betreff: [Borgbackup] Mac OS Mojave
>>>> 
>>>> Dear Borg gurus,
>>>> 
>>>> I run borg on a Mac. Since I updated to Mojave, I get error messages of this kind:
>>>> 
>>>> /users/xxx/Pictures/Photos Library.photoslibrary: scandir: [Errno 1] Operation not permitted: '/users/xxx/Pictures/Photos Library.photoslibrary?
>>>> 
>>>> I suspect this has to do with Mojave being more restrictive about letting apps access certain files or directories. Do you think giving ?full disk access? to borg will solve this? Or should I give that to the shell script that runs borg for me, or??
>>>> 
>>>> Perhaps there are other Mac users out there that have also run into this and already solved it.
>>>> 
>>>> Best wishes,
>>>> 
>>>> Stefan
>>>> 
>>>> _______________________________________________
>>>> Borgbackup mailing list
>>>> Borgbackup at python.org
>>>> https://mail.python.org/mailman/listinfo/borgbackup
>>>> 
>> 
>> 


From ed at edgewood.to  Mon Feb  4 15:29:36 2019
From: ed at edgewood.to (Ed Blackman)
Date: Mon, 4 Feb 2019 15:29:36 -0500
Subject: [Borgbackup] Separating repos
Message-ID: <1549308203.e5301f@strabo.garrett>

I have a set of systems backing up to a single repository on an internal 
borg server.  Initially all the archives came from the same system, and 
there was a good chance that as files moved around in that system that 
they'd be backed up in one set of archives, then show up in another.  So 
I used a single repository to maximize deduplication.

Over time, I've separated the functions of that single system into 
multiple systems, and also improved procedures so that a given file 
should show up in at most one type of archive.  But since they're going 
into a shared repo, the backups have to spend time syncing the cache.

I'd like to separate retain the archives but separate them into 
different repos, but it doesn't look like borg has a native way to copy 
archives from one repo to another.

Is the only way to do it to make a copy of the repo directory outside of 
borg, and then selectively remove archives from one copy or the other?  
Would the repos shrink automatically as the archives are removed, such 
that they would approximately sum to the original repo size, less some 
for chunks that are now duplicated in two repos?

-- 
Ed?Blackman

From anarcat at debian.org  Mon Feb  4 16:10:49 2019
From: anarcat at debian.org (=?utf-8?Q?Antoine_Beaupr=C3=A9?=)
Date: Mon, 04 Feb 2019 16:10:49 -0500
Subject: [Borgbackup] Separating repos
In-Reply-To: <1549308203.e5301f@strabo.garrett>
References: <1549308203.e5301f@strabo.garrett>
Message-ID: <87zhrbjm3q.fsf@curie.anarc.at>

On 2019-02-04 15:29:36, Ed Blackman wrote:
> I have a set of systems backing up to a single repository on an internal 
> borg server.  Initially all the archives came from the same system, and 
> there was a good chance that as files moved around in that system that 
> they'd be backed up in one set of archives, then show up in another.  So 
> I used a single repository to maximize deduplication.
>
> Over time, I've separated the functions of that single system into 
> multiple systems, and also improved procedures so that a given file 
> should show up in at most one type of archive.  But since they're going 
> into a shared repo, the backups have to spend time syncing the cache.
>
> I'd like to separate retain the archives but separate them into 
> different repos, but it doesn't look like borg has a native way to copy 
> archives from one repo to another.
>
> Is the only way to do it to make a copy of the repo directory outside of 
> borg, and then selectively remove archives from one copy or the other?  
> Would the repos shrink automatically as the archives are removed, such 
> that they would approximately sum to the original repo size, less some 
> for chunks that are now duplicated in two repos?

I don't know, but what I do know is how *not* to do this:

https://github.com/borgbackup/borg/issues/4272

A.

-- 
Striving for social justice is the most valuable thing to do in life
                       - Albert Einstein

From eric at in3x.io  Mon Feb  4 18:11:07 2019
From: eric at in3x.io (Eric S. Johansson)
Date: Mon, 4 Feb 2019 18:11:07 -0500
Subject: [Borgbackup] Separating repos
In-Reply-To: <1549308203.e5301f@strabo.garrett>
References: <1549308203.e5301f@strabo.garrett>
Message-ID: <fcbc325c-a5c9-84c7-1318-dda4af9c8839@in3x.io>

what I did when faced with similar problem was to start new repository
from scratch and leave the old repository untouched except for scheduled
purges.

On 2/4/2019 3:29 PM, Ed Blackman wrote:
> I have a set of systems backing up to a single repository on an internal 
> borg server.  Initially all the archives came from the same system, and 
> there was a good chance that as files moved around in that system that 
> they'd be backed up in one set of archives, then show up in another.  So 
> I used a single repository to maximize deduplication.
>
> Over time, I've separated the functions of that single system into 
> multiple systems, and also improved procedures so that a given file 
> should show up in at most one type of archive.  But since they're going 
> into a shared repo, the backups have to spend time syncing the cache.
>
> I'd like to separate retain the archives but separate them into 
> different repos, but it doesn't look like borg has a native way to copy 
> archives from one repo to another.
>
> Is the only way to do it to make a copy of the repo directory outside of 
> borg, and then selectively remove archives from one copy or the other?  
> Would the repos shrink automatically as the archives are removed, such 
> that they would approximately sum to the original repo size, less some 
> for chunks that are now duplicated in two repos?
>
-- 
Eric S. Johansson
eric at in3x.io
http://www.in3x.io
978-512-0272



From ed at edgewood.to  Mon Feb  4 19:59:02 2019
From: ed at edgewood.to (Ed Blackman)
Date: Mon, 4 Feb 2019 19:59:02 -0500
Subject: [Borgbackup] Separating repos
In-Reply-To: <87zhrbjm3q.fsf@curie.anarc.at>
References: <1549308203.e5301f@strabo.garrett> <87zhrbjm3q.fsf@curie.anarc.at>
Message-ID: <1549328135.10012f@strabo.garrett>

On Mon, Feb 04, 2019 at 04:10:49PM -0500, Antoine Beaupr? wrote:
> > Is the only way to do it to make a copy of the repo directory outside of 
> > borg, and then selectively remove archives from one copy or the other?  
> > Would the repos shrink automatically as the archives are removed, such 
> > that they would approximately sum to the original repo size, less some 
> > for chunks that are now duplicated in two repos?
> 
> I don't know, but what I do know is how *not* to do this:
> 
> https://github.com/borgbackup/borg/issues/4272

My repo is on btrfs, and I was thinking of using the CoW feature (cp 
--reflink=always) to save space.  I expect it to work better than hard 
links, but that issue gives me a good test case.  Thanks!

-- 
Ed?Blackman

From ed at edgewood.to  Mon Feb  4 20:08:05 2019
From: ed at edgewood.to (Ed Blackman)
Date: Mon, 4 Feb 2019 20:08:05 -0500
Subject: [Borgbackup] Separating repos
In-Reply-To: <fcbc325c-a5c9-84c7-1318-dda4af9c8839@in3x.io>
References: <1549308203.e5301f@strabo.garrett>
 <fcbc325c-a5c9-84c7-1318-dda4af9c8839@in3x.io>
Message-ID: <1549328368.8968df@strabo.garrett>

On Mon, Feb 04, 2019 at 06:11:07PM -0500, Eric S. Johansson wrote:
> what I did when faced with similar problem was to start new repository
> from scratch and leave the old repository untouched except for scheduled
> purges.

I thought about that, but the problem is that the "keep" purges won't 
purge anything if you're not adding anything.  The last X 
daily/weekly/monthly backups keep being the last X if you're not adding 
any new ones.  I did think about adding a flag (--keep-absolute-time ??  
naming is hard) that would change the sense of --keep-daily etc such 
that --keep-absolute-time --keep-daily 7 would purge any daily backups 
older than 7 days from "now" even if that would leave less than 7.

But I haven't done any investigation deeper than thinking about what to 
name the flag <grin>

Did you hardcode purges somehow?

-- 
Ed?Blackman

From dave at gasaway.org  Mon Feb  4 22:20:12 2019
From: dave at gasaway.org (David Gasaway)
Date: Mon, 4 Feb 2019 19:20:12 -0800
Subject: [Borgbackup] Separating repos
In-Reply-To: <1549328368.8968df@strabo.garrett>
References: <1549308203.e5301f@strabo.garrett>
 <fcbc325c-a5c9-84c7-1318-dda4af9c8839@in3x.io>
 <1549328368.8968df@strabo.garrett>
Message-ID: <CAGsJ5AVmtKpNnqFuuwiZAz-R3qziRxFTSPyo-tGW4ZDzAF2oyQ@mail.gmail.com>

On Mon, Feb 4, 2019 at 5:11 PM Ed Blackman <ed at edgewood.to> wrote:

> I thought about that, but the problem is that the "keep" purges won't
> purge anything if you're not adding anything.  The last X
> daily/weekly/monthly backups keep being the last X if you're not adding
> any new ones.  I did think about adding a flag (--keep-absolute-time ??
> naming is hard) that would change the sense of --keep-daily etc such
> that --keep-absolute-time --keep-daily 7 would purge any daily backups
> older than 7 days from "now" even if that would leave less than 7.
>

This option already exists, see --keep-within.

-- 
-:-:- David K. Gasaway
-:-:- Email: dave at gasaway.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20190204/3e33108d/attachment-0001.html>

From ed at edgewood.to  Tue Feb  5 11:06:53 2019
From: ed at edgewood.to (Ed Blackman)
Date: Tue, 5 Feb 2019 11:06:53 -0500
Subject: [Borgbackup] Separating repos
In-Reply-To: <CAGsJ5AVmtKpNnqFuuwiZAz-R3qziRxFTSPyo-tGW4ZDzAF2oyQ@mail.gmail.com>
References: <1549308203.e5301f@strabo.garrett>
 <fcbc325c-a5c9-84c7-1318-dda4af9c8839@in3x.io>
 <1549328368.8968df@strabo.garrett>
 <CAGsJ5AVmtKpNnqFuuwiZAz-R3qziRxFTSPyo-tGW4ZDzAF2oyQ@mail.gmail.com>
Message-ID: <1549381885.91bec9@strabo.garrett>

On Mon, Feb 04, 2019 at 07:20:12PM -0800, David Gasaway wrote:
> On Mon, Feb 4, 2019 at 5:11 PM Ed Blackman <ed at edgewood.to> wrote:
> > I thought about that, but the problem is that the "keep" purges won't
> > purge anything if you're not adding anything.  The last X
> > daily/weekly/monthly backups keep being the last X if you're not adding
> > any new ones.  I did think about adding a flag (--keep-absolute-time ??
> > naming is hard) that would change the sense of --keep-daily etc such
> > that --keep-absolute-time --keep-daily 7 would purge any daily backups
> > older than 7 days from "now" even if that would leave less than 7.
> 
> This option already exists, see --keep-within.

That would work for the simple example above, but if I have a prune that 
keeps, eg, the last 7 dailies, the last 4 weeklies, and the last 3 
monthlies, there's no way I can think of to *gradually* prune those 
backups.  --keep-within 3m would keep them all until the end of the 3 
months, then drop them all at once.

-- 
Ed?Blackman

From qzwx2007 at gmail.com  Tue Feb  5 16:33:46 2019
From: qzwx2007 at gmail.com (JK)
Date: Tue, 5 Feb 2019 23:33:46 +0200
Subject: [Borgbackup] Separating repos
In-Reply-To: <1549381885.91bec9@strabo.garrett>
References: <1549308203.e5301f@strabo.garrett>
 <fcbc325c-a5c9-84c7-1318-dda4af9c8839@in3x.io>
 <1549328368.8968df@strabo.garrett>
 <CAGsJ5AVmtKpNnqFuuwiZAz-R3qziRxFTSPyo-tGW4ZDzAF2oyQ@mail.gmail.com>
 <1549381885.91bec9@strabo.garrett>
Message-ID: <8beaca37-0b2e-6b29-5122-e1813dc9bb29@gmail.com>

How to prune gradually the old repo: Take backup into the old repo as 
usually, but change the content of the backup. Archive could even be 
empty. Then run your prune algorithm as usually. Prune works gradually 
because there are daily new archives.

On 5.2.2019 18.06, Ed Blackman wrote:
> On Mon, Feb 04, 2019 at 07:20:12PM -0800, David Gasaway wrote:
>> On Mon, Feb 4, 2019 at 5:11 PM Ed Blackman <ed at edgewood.to> wrote:
>>> I thought about that, but the problem is that the "keep" purges won't
>>> purge anything if you're not adding anything.  The last X
>>> daily/weekly/monthly backups keep being the last X if you're not adding
>>> any new ones.  I did think about adding a flag (--keep-absolute-time ??
>>> naming is hard) that would change the sense of --keep-daily etc such
>>> that --keep-absolute-time --keep-daily 7 would purge any daily backups
>>> older than 7 days from "now" even if that would leave less than 7.
>> This option already exists, see --keep-within.
> That would work for the simple example above, but if I have a prune that
> keeps, eg, the last 7 dailies, the last 4 weeklies, and the last 3
> monthlies, there's no way I can think of to *gradually* prune those
> backups.  --keep-within 3m would keep them all until the end of the 3
> months, then drop them all at once.
>

From dave at gasaway.org  Tue Feb  5 21:14:57 2019
From: dave at gasaway.org (David Gasaway)
Date: Tue, 5 Feb 2019 18:14:57 -0800
Subject: [Borgbackup] Separating repos
In-Reply-To: <1549381885.91bec9@strabo.garrett>
References: <1549308203.e5301f@strabo.garrett>
 <fcbc325c-a5c9-84c7-1318-dda4af9c8839@in3x.io>
 <1549328368.8968df@strabo.garrett>
 <CAGsJ5AVmtKpNnqFuuwiZAz-R3qziRxFTSPyo-tGW4ZDzAF2oyQ@mail.gmail.com>
 <1549381885.91bec9@strabo.garrett>
Message-ID: <CAGsJ5AVfd-UBKXHQYhNM1cQK1s-suBB4BG2TiOwvqS1QNNJDjA@mail.gmail.com>

On Tue, Feb 5, 2019 at 8:12 AM Ed Blackman <ed at edgewood.to> wrote


> That would work for the simple example above, but if I have a prune that
> keeps, eg, the last 7 dailies, the last 4 weeklies, and the last 3
> monthlies, there's no way I can think of to *gradually* prune those
> backups.  --keep-within 3m would keep them all until the end of the 3
> months, then drop them all at once.
>

'--keep-within 3m' would keep anything created in the last three months.
If run daily in your example, it would first prune the oldest monthly.
After a month, it would prune another monthly.  After about another month,
it would prune the last monthly and the oldest weekly.  After another week,
another weekly would get pruned.  After running for a week short of three
months, it would prune the last weekly and start pruning the dailies.  In
other words, it would gradually roll them off.

How does this differ from the new option that you had in mind?

-- 
-:-:- David K. Gasaway
-:-:- Email: dave at gasaway.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20190205/52587f53/attachment.html>

From tw at waldmann-edv.de  Mon Feb 11 03:09:39 2019
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Mon, 11 Feb 2019 09:09:39 +0100
Subject: [Borgbackup] borgbackup 1.1.9 released!
Message-ID: <b64b6f41-4e02-4ef4-1fc3-00cd37cd0fe1@waldmann-edv.de>

borgbackup 1.1.9 released with security and bug fixes.

https://github.com/borgbackup/borg/releases/tag/1.1.9


-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393

From ndbecker2 at gmail.com  Tue Feb 12 06:52:00 2019
From: ndbecker2 at gmail.com (Neal Becker)
Date: Tue, 12 Feb 2019 06:52:00 -0500
Subject: [Borgbackup] msgpack-0.6.1 not supported
Message-ID: <CAG3t+pEGAPuWpp-WsBLKounCP-N6p41JCkRtG2-QYehaeG8wnw@mail.gmail.com>

I just updated to latest borgbackup, and it refused to run, saying my
version of msgpack, which is the latest 0.6.1, is not supported.

Downgrading msgpack fixed it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20190212/76fc375f/attachment.html>

From imperator at jedimail.de  Fri Feb 15 07:11:49 2019
From: imperator at jedimail.de (Sascha Ternes)
Date: Fri, 15 Feb 2019 13:11:49 +0100
Subject: [Borgbackup] What to do if borg check fails due to Input/Output
 Error 5?
Message-ID: <246aed0e-cfa6-e53f-bec2-7bd1d1af7564@jedimail.de>

Hi!

I use BorgBackup with repository on davfs for quite a while now. After
several month of having no problems I now get an error (OSError 5 /
Input/output error) with one of the data chunk files when doing a "borg
check" (see log below).

I read some issues in GitHub concerning davfs. I do not consider this
error a BorgBackup issue, but my question is: What can I possibly do
now? Is there a way to delete this file or truncate it, or do I have to
throw away the whole backup (since I cannot run a check anymore) and
create it from scratch?

Sascha

----

Starting repository check
Local Exception
Traceback (most recent call last):
? File "borg/repository.py", line 1295, in get_fd
? File "borg/lrucache.py", line 21, in __getitem__
KeyError: 103

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
? File "borg/archiver.py", line 4455, in main
? File "borg/archiver.py", line 4387, in run
? File "borg/archiver.py", line 154, in wrapper
? File "borg/archiver.py", line 315, in do_check
? File "borg/repository.py", line 923, in check
? File "borg/repository.py", line 1340, in iter_objects
? File "borg/repository.py", line 1297, in get_fd
OSError: [Errno 5] Input/output error: '/srv/borg/data/0/103'

Platform: Linux v22018041585863736 4.9.0-8-amd64 #1 SMP Debian 4.9.130-2
(2018-10-27) x86_64
Linux: debian 9.7
Borg: 1.1.9? Python: CPython 3.5.6
PID: 8601? CWD: /root
sys.argv: ['/usr/local/bin/borg', 'check', '-v', '/srv/borg']
SSH_ORIGINAL_COMMAND: None


From panayotis at panayotis.com  Sat Feb 16 05:29:16 2019
From: panayotis at panayotis.com (Panayotis Katsaloulis)
Date: Sat, 16 Feb 2019 12:29:16 +0200
Subject: [Borgbackup] borg verbose leaves visual garbage on screen
In-Reply-To: <0164af5d-7014-4a16-9cf9-a3bc3d5e5436@Spark>
References: <0164af5d-7014-4a16-9cf9-a3bc3d5e5436@Spark>
Message-ID: <6dc6664b-9740-40eb-8f11-7849e143eed1@Spark>

Hello people and thank you for this excellent backup tool.
Though, it seems that there is a small glitch on the display of verbose backup.

When it displays a file which is not candidate for a backup, it ?recycles? the line and displays the next file on top of it.
If the file needs to be backed up, it leaves the line as is and goes down to the next line.
This is perfect behaviour.

BUT, if the file needs to be backed up in length, it is smaller than the file being displayed previously (or any other file displayed previously), then the display of the previous file is still on screen, and instead garbage is displayed after the name of the file being actually backed up. This makes it difficult to understand what is actually backed up, if you don?t know it in advance.

As an example I can give you what I constantly see if I run this command:

# borg create -v --progress --compression lzma,9 --list --filter=AME "$HOST:$BORGLOC::${NAME}" $FDIRS
Creating archive at ????....:.borg::ierax_20190215-171646"
M Works/Development/Java/libs/appbundler/target/antrun/build-main.xmly/.git/objects/75/0777a3b5006ffe73e0e2f2e99c173ab63bc675
M Works/Development/Mobile/SDK/xmlvm/target/antrun/build-main.xmlK/xmlvm/tutorial/ios/sensor/camera/nbproject/configs
M Works/.last_syncC 6.32 MB D 116318 N Works/System/EFI/config-orig.plist

The three actual files are
Works/Development/Java/libs/appbundler/target/antrun/build-main.xml
Works/Development/Mobile/SDK/xmlvm/target/antrun/build-main.xml
Works/.last_sync

It is interesting that the last line not only has garbage, but also the backup statistics.

My system is OSX Mohave and borg 1.1.8 (I know that 1.1.9 already exists, but I am still waiting for brew to catch up)




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20190216/ef3b7d71/attachment.html>

From tw at waldmann-edv.de  Sat Feb 16 13:12:14 2019
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Sat, 16 Feb 2019 19:12:14 +0100
Subject: [Borgbackup] borg verbose leaves visual garbage on screen
In-Reply-To: <6dc6664b-9740-40eb-8f11-7849e143eed1@Spark>
References: <0164af5d-7014-4a16-9cf9-a3bc3d5e5436@Spark>
 <6dc6664b-9740-40eb-8f11-7849e143eed1@Spark>
Message-ID: <7e99ceab-436a-3268-8800-92a4ac1d242b@waldmann-edv.de>

> # borg create -v --progress --list ...

--progress and --list are not intended to be used together.

> --compression lzma,9

Everything > 6 is pointless.

Also, check zstd, it is a more modern algorithm with a wide scale from
very fast to high compression.


-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393

From panayotis at panayotis.com  Sat Feb 16 15:50:00 2019
From: panayotis at panayotis.com (Panayotis Katsaloulis)
Date: Sat, 16 Feb 2019 22:50:00 +0200
Subject: [Borgbackup] borg verbose leaves visual garbage on screen
In-Reply-To: <0eb0082f-ca43-40d7-8dda-0178ceb05142@Spark>
References: <0164af5d-7014-4a16-9cf9-a3bc3d5e5436@Spark>
 <6dc6664b-9740-40eb-8f11-7849e143eed1@Spark>
 <7e99ceab-436a-3268-8800-92a4ac1d242b@waldmann-edv.de>
 <0eb0082f-ca43-40d7-8dda-0178ceb05142@Spark>
Message-ID: <09e239fb-1258-4a4f-a94e-ef886705073c@Spark>

On 16 Feb 2019, 20:12 +0200, Thomas Waldmann <tw at waldmann-edv.de>, wrote:
> > # borg create -v --progress --list ...
>
> --progress and --list are not intended to be used together.

Indeed, if only one of these is used, then there?s no visual conflict.
But, for my eyes, both are vital information fo how well a backup is running.
I don?t even understand why irrelevant files should be displayed there, since only changed files are visible.
And why it should be a conflict.

Sorry for asking this, but I really love this software and, even if this behaviour is ?not as intended?, maybe it should be though over again?


>
> > --compression lzma,9
>
> Everything > 6 is pointless.
>
> Also, check zstd, it is a more modern algorithm with a wide scale from
> very fast to high compression.

Since my backup is already old, do I just change the command line switch, or should I do something more?

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20190216/15de42aa/attachment.html>

From sitaramc at gmail.com  Sat Feb 16 23:10:32 2019
From: sitaramc at gmail.com (Sitaram Chamarty)
Date: Sun, 17 Feb 2019 09:40:32 +0530
Subject: [Borgbackup] borg verbose leaves visual garbage on screen
In-Reply-To: <09e239fb-1258-4a4f-a94e-ef886705073c@Spark>
References: <0164af5d-7014-4a16-9cf9-a3bc3d5e5436@Spark>
 <6dc6664b-9740-40eb-8f11-7849e143eed1@Spark>
 <7e99ceab-436a-3268-8800-92a4ac1d242b@waldmann-edv.de>
 <0eb0082f-ca43-40d7-8dda-0178ceb05142@Spark>
 <09e239fb-1258-4a4f-a94e-ef886705073c@Spark>
Message-ID: <20190217041032.GA8111@sita-dell>

On Sat, Feb 16, 2019 at 10:50:00PM +0200, Panayotis Katsaloulis wrote:
> On 16 Feb 2019, 20:12 +0200, Thomas Waldmann <tw at waldmann-edv.de>, wrote:
> > > # borg create -v --progress --list ...
> >
> > --progress and --list are not intended to be used together.
> 
> Indeed, if only one of these is used, then there?s no visual conflict.
> But, for my eyes, both are vital information fo how well a backup is running.
> I don?t even understand why irrelevant files should be displayed there, since only changed files are visible.
> And why it should be a conflict.
> 
> Sorry for asking this, but I really love this software and, even if this behaviour is ?not as intended?, maybe it should be though over again?

hate to be a "me too", but... "me too".  I've always had this
problem, but never asked.

Would using a BORG_LOGGING_CONF help separate the two streams?

If it does, can I please request a bare bones minimal example in
the man page, instead of a link to the much more generic
**python** documentation.

To paraphrase Larry Wall (or someone like him anyway), that conf
format makes hard things possible, but it also makes easy things
hard.

A quick example that just gets INFO messages into a file, right
in the borg man page, would be *highly* appreciated by anyone
who is not already familiar with python's logging conf file
format and the arcana of "handlers", "loggers", and
"formatters".

regards
sitaram


From felix.schwarz at oss.schwarz.eu  Sun Feb 17 17:15:44 2019
From: felix.schwarz at oss.schwarz.eu (Felix Schwarz)
Date: Sun, 17 Feb 2019 23:15:44 +0100
Subject: [Borgbackup] borg cache / security aspects
Message-ID: <c4ca7850-eca4-e3b6-cd4b-a23e2c8bc718@oss.schwarz.eu>

Hi,

I'd like to know more about security aspects of borg's cache when using
encrypted repos.

If I use an encrypted borg repo and store borg's cache right next to the
actual repo (by setting "BORG_CACHE_DIR") are there any security considerations?

So if I assume an attacker gets access to the (encrypted) borg repo and the
cache files (but not the repo key) can he deduce anything from the cache?

Background:
- I'd like to back up a somewhat bigger dataset (~2 TB) from a remote server
  with limited connectivity (100 Mbit).
- The borg repo should be on external hard drives.
- I would like to store the borg cache on the same drive so I can run the
  backup from multiple work stations. Rebuilding the cache on a new machine
  takes quite a while to rebuild (just 100 Mbit).
- I don't want to worry if someone steals the hard drive.

Thank you very much,
Felix

From tw at waldmann-edv.de  Tue Feb 19 02:50:11 2019
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Tue, 19 Feb 2019 08:50:11 +0100
Subject: [Borgbackup] borg cache / security aspects
In-Reply-To: <c4ca7850-eca4-e3b6-cd4b-a23e2c8bc718@oss.schwarz.eu>
References: <c4ca7850-eca4-e3b6-cd4b-a23e2c8bc718@oss.schwarz.eu>
Message-ID: <2cced7a6-bf14-0051-652b-9a29522ff0ba@waldmann-edv.de>

> If I use an encrypted borg repo and store borg's cache right next to the
> actual repo (by setting "BORG_CACHE_DIR") are there any security considerations?

Don't do that, that is not secure.

The cache is assumed to be on the system that also has access to the
backed up original files, so the same security assumptions apply for the
cache and the original files.

> So if I assume an attacker gets access to the (encrypted) borg repo and the
> cache files (but not the repo key) can he deduce anything from the cache?

The cache is not encrypted and has quite some metadata about your files.

> Background:
> - I'd like to back up a somewhat bigger dataset (~2 TB) from a remote server
>   with limited connectivity (100 Mbit).
> - The borg repo should be on external hard drives.
> - I would like to store the borg cache on the same drive so I can run the
>   backup from multiple work stations. Rebuilding the cache on a new machine
>   takes quite a while to rebuild (just 100 Mbit).

You can use different repos for different machines to avoid the cache
coherency issue (and the related rebuilding of the cache).


-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393

From felix.schwarz at oss.schwarz.eu  Tue Feb 19 03:29:06 2019
From: felix.schwarz at oss.schwarz.eu (Felix Schwarz)
Date: Tue, 19 Feb 2019 09:29:06 +0100
Subject: [Borgbackup] borg cache / security aspects
In-Reply-To: <2cced7a6-bf14-0051-652b-9a29522ff0ba@waldmann-edv.de>
References: <c4ca7850-eca4-e3b6-cd4b-a23e2c8bc718@oss.schwarz.eu>
 <2cced7a6-bf14-0051-652b-9a29522ff0ba@waldmann-edv.de>
Message-ID: <0056f545-2e7f-91bd-f822-5dd6adb695a8@oss.schwarz.eu>

Hi,

Am 19.02.19 um 08:50 schrieb Thomas Waldmann:
>> If I use an encrypted borg repo and store borg's cache right next to the
>> actual repo (by setting "BORG_CACHE_DIR") are there any security considerations?
> 
> Don't do that, that is not secure.

Ok, that's what I expected (but wanted to be sure). I'll create an encrypted
volume on the external drive which holds the borg cache then.

>> Background:
>> - I'd like to back up a somewhat bigger dataset (~2 TB) from a remote server
>>   with limited connectivity (100 Mbit).
>> - The borg repo should be on external hard drives.
>> - I would like to store the borg cache on the same drive so I can run the
>>   backup from multiple work stations. Rebuilding the cache on a new machine
>>   takes quite a while to rebuild (just 100 Mbit).
> 
> You can use different repos for different machines to avoid the cache
> coherency issue (and the related rebuilding of the cache).

Unfortunately this won't work in my case (without vastly increased storage
space): borg should be run on different machines but the backup data is remote
(exposed via a network file system) with limited bandwidth and I can not
install borg on the remote machine (yet).

So I want to avoid that each workstation has to rebuild the cache for the same
backup source.

Thanks for your help,
Felix

From sitaramc at gmail.com  Tue Feb 19 08:31:05 2019
From: sitaramc at gmail.com (Sitaram Chamarty)
Date: Tue, 19 Feb 2019 19:01:05 +0530
Subject: [Borgbackup] borg verbose leaves visual garbage on screen
In-Reply-To: <20190217041032.GA8111@sita-dell>
References: <0164af5d-7014-4a16-9cf9-a3bc3d5e5436@Spark>
 <6dc6664b-9740-40eb-8f11-7849e143eed1@Spark>
 <7e99ceab-436a-3268-8800-92a4ac1d242b@waldmann-edv.de>
 <0eb0082f-ca43-40d7-8dda-0178ceb05142@Spark>
 <09e239fb-1258-4a4f-a94e-ef886705073c@Spark>
 <20190217041032.GA8111@sita-dell>
Message-ID: <20190219133105.GA28957@sita-dell>

On Sun, Feb 17, 2019 at 09:40:32AM +0530, Sitaram Chamarty wrote:
> On Sat, Feb 16, 2019 at 10:50:00PM +0200, Panayotis Katsaloulis wrote:
> > On 16 Feb 2019, 20:12 +0200, Thomas Waldmann <tw at waldmann-edv.de>, wrote:
> > > > # borg create -v --progress --list ...
> > >
> > > --progress and --list are not intended to be used together.
> > 
> > Indeed, if only one of these is used, then there?s no visual conflict.
> > But, for my eyes, both are vital information fo how well a backup is running.
> > I don?t even understand why irrelevant files should be displayed there, since only changed files are visible.
> > And why it should be a conflict.
> > 
> > Sorry for asking this, but I really love this software and, even if this behaviour is ?not as intended?, maybe it should be though over again?
> 
> hate to be a "me too", but... "me too".  I've always had this
> problem, but never asked.
> 
> Would using a BORG_LOGGING_CONF help separate the two streams?
> 
> If it does, can I please request a bare bones minimal example in
> the man page, instead of a link to the much more generic
> **python** documentation.
> 
> To paraphrase Larry Wall (or someone like him anyway), that conf
> format makes hard things possible, but it also makes easy things
> hard.
> 
> A quick example that just gets INFO messages into a file, right
> in the borg man page, would be *highly* appreciated by anyone
> who is not already familiar with python's logging conf file
> format and the arcana of "handlers", "loggers", and
> "formatters".

Panayotis

As I suspected, using BORG_LOGGING_CONF helps.  This is the
minimal conf I could come up with (see end of this mail).  As
you can see, it's pretty verbose and roundabout, for what should
have been as simple as "--logfile=/tmp/borg.log".

For people who will respond saying "aah but python's logging
library is much more powerful and it can [...]", I would like to
humbly state that a **user** of borg should not need to know how
to use one of the libraries that borg **developers** chose to
use.  Certainly, allow power users to avail this extra power and
flexibility, but the simplest and most basic use of it should
not require such knowledge.

rgds
sitaram

Copy the below text in a file, and point the environment
variable BORG_LOGGING_CONF to this file before running borg.
With that, the --progress comes to STDERR as before, while the
--list goes to the file you designated in the "args" line.  I
suggest you also use "--filter=AME".

    [loggers]
    keys=root

    [handlers]
    keys=hand01

    [formatters]
    keys=form01

    [logger_root]
    level=NOTSET
    handlers=hand01

    [handler_hand01]
    class=FileHandler
    level=INFO
    formatter=form01
    args=('/tmp/borg.log', 'w')

    [formatter_form01]
    format=%(asctime)s %(levelname)s %(message)s
    datefmt=
    class=logging.Formatter

From kramski at hoernle-marbach.de  Thu Feb 21 14:16:13 2019
From: kramski at hoernle-marbach.de (Heinz Werner Kramski-Grote)
Date: Thu, 21 Feb 2019 20:16:13 +0100
Subject: [Borgbackup] How often do you run "borg check"?
Message-ID: <5315159.rS7cyfeXXH@sandl>

I'm currently setting up a Helios4[1] box as my private backup server following the great deployment chapter[2]. The server eventually will be located at a friends home, which I'm temporarily simulating with "--remote-ratelimit 3125".

I'm using borgmatic[3] as a declarative wrapper. Borgmatic's default is to check each repository and each archive on every "create" run, which turns out to be extremely time consuming (minimal backup only: 14 minutes, backup and default consistency checks: 7 hours).

How often do you check your repos and archives? Will once a week be ok, assuming that backup runs take place once a day?

Thanks for advice,
   Heinz

[1] https://kobol.io/helios4/
[2] https://borgbackup.readthedocs.io/en/stable/deployment.html
[3] https://torsion.org/borgmatic/



From imperator at jedimail.de  Fri Feb 22 04:48:02 2019
From: imperator at jedimail.de (Sascha Ternes)
Date: Fri, 22 Feb 2019 10:48:02 +0100
Subject: [Borgbackup] What to do if borg check fails due to Input/Output
 Error 5?
In-Reply-To: <246aed0e-cfa6-e53f-bec2-7bd1d1af7564@jedimail.de>
References: <246aed0e-cfa6-e53f-bec2-7bd1d1af7564@jedimail.de>
Message-ID: <6b78fec2-b034-950f-9eaf-a88946f532e0@jedimail.de>

Hi!

I have read the data structures in BorgBackup docs. My conclusion is
that it has a very big impact on the repository consistency if one of
the segment files is lost.

@ThomasWaldmann: Do you see a possibility for a "borg check --repair"
that is able to repair the consistency of a repository (truncate the
lost file, remove chunks and corresponding archives files or whole
archives)?

Sascha

> Hi!
>
> I use BorgBackup with repository on davfs for quite a while now. After
> several month of having no problems I now get an error (OSError 5 /
> Input/output error) with one of the data chunk files when doing a "borg
> check" (see log below).
>
> I read some issues in GitHub concerning davfs. I do not consider this
> error a BorgBackup issue, but my question is: What can I possibly do
> now? Is there a way to delete this file or truncate it, or do I have to
> throw away the whole backup (since I cannot run a check anymore) and
> create it from scratch?
>
> Sascha
>
> ----
>
> Starting repository check
> Local Exception
> Traceback (most recent call last):
> ? File "borg/repository.py", line 1295, in get_fd
> ? File "borg/lrucache.py", line 21, in __getitem__
> KeyError: 103
>
> During handling of the above exception, another exception occurred:
>
> Traceback (most recent call last):
> ? File "borg/archiver.py", line 4455, in main
> ? File "borg/archiver.py", line 4387, in run
> ? File "borg/archiver.py", line 154, in wrapper
> ? File "borg/archiver.py", line 315, in do_check
> ? File "borg/repository.py", line 923, in check
> ? File "borg/repository.py", line 1340, in iter_objects
> ? File "borg/repository.py", line 1297, in get_fd
> OSError: [Errno 5] Input/output error: '/srv/borg/data/0/103'
>
> Platform: Linux v22018041585863736 4.9.0-8-amd64 #1 SMP Debian 4.9.130-2
> (2018-10-27) x86_64
> Linux: debian 9.7
> Borg: 1.1.9? Python: CPython 3.5.6
> PID: 8601? CWD: /root
> sys.argv: ['/usr/local/bin/borg', 'check', '-v', '/srv/borg']
> SSH_ORIGINAL_COMMAND: None
>

From tw at waldmann-edv.de  Fri Feb 22 12:46:50 2019
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Fri, 22 Feb 2019 18:46:50 +0100
Subject: [Borgbackup] How often do you run "borg check"?
In-Reply-To: <5315159.rS7cyfeXXH@sandl>
References: <5315159.rS7cyfeXXH@sandl>
Message-ID: <d7d3b447-3ff5-f223-ea11-2647e139c629@waldmann-edv.de>

> How often do you check your repos and archives? Will once a week be ok, assuming that backup runs take place once a day?

Guess that is fine.

Of course it depends a bit in the reliability of your stuff, so there
can be no general advice.

-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393

From tw at waldmann-edv.de  Fri Feb 22 12:41:23 2019
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Fri, 22 Feb 2019 18:41:23 +0100
Subject: [Borgbackup] What to do if borg check fails due to Input/Output
 Error 5?
In-Reply-To: <6b78fec2-b034-950f-9eaf-a88946f532e0@jedimail.de>
References: <246aed0e-cfa6-e53f-bec2-7bd1d1af7564@jedimail.de>
 <6b78fec2-b034-950f-9eaf-a88946f532e0@jedimail.de>
Message-ID: <966399d4-f375-a665-e32b-71f878e1c0b8@waldmann-edv.de>

> I have read the data structures in BorgBackup docs. My conclusion is
> that it has a very big impact on the repository consistency if one of
> the segment files is lost.

Yes, due to deduplication, you have everything just once.

> @ThomasWaldmann: Do you see a possibility for a "borg check --repair"
> that is able to repair the consistency of a repository (truncate the
> lost file, remove chunks and corresponding archives files or whole
> archives)?

Well, esp. for I/O Errors this is rather problematic.

While borg check --repair could try to make everything consistent again
(even if this means deleting a lot of stuff), it might make things worse
- e.g. if the I/O error is temporary and caused by some hardware, OS,
filesystem, network issue.

>> I use BorgBackup with repository on davfs for quite a while now. After
>> several month of having no problems I now get an error (OSError 5 /
>> Input/output error) with one of the data chunk files when doing a "borg
>> check" (see log below).

Try to find out why.

>> ? File "borg/repository.py", line 1340, in iter_objects
>> ? File "borg/repository.py", line 1297, in get_fd
>> OSError: [Errno 5] Input/output error: '/srv/borg/data/0/103'

So, what's up with that file?


-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393

From imperator at jedimail.de  Sun Feb 24 03:47:34 2019
From: imperator at jedimail.de (Sascha Ternes)
Date: Sun, 24 Feb 2019 09:47:34 +0100
Subject: [Borgbackup] What to do if borg check fails due to Input/Output
 Error 5?
In-Reply-To: <966399d4-f375-a665-e32b-71f878e1c0b8@waldmann-edv.de>
References: <246aed0e-cfa6-e53f-bec2-7bd1d1af7564@jedimail.de>
 <6b78fec2-b034-950f-9eaf-a88946f532e0@jedimail.de>
 <966399d4-f375-a665-e32b-71f878e1c0b8@waldmann-edv.de>
Message-ID: <75469a1a-867b-5c19-3d33-1dd94f430275@jedimail.de>

>>> I use BorgBackup with repository on davfs for quite a while now. After
>>> several month of having no problems I now get an error (OSError 5 /
>>> Input/output error) with one of the data chunk files when doing a "borg
>>> check" (see log below).
> 
> Try to find out why.
> 
>>> ? File "borg/repository.py", line 1340, in iter_objects
>>> ? File "borg/repository.py", line 1297, in get_fd
>>> OSError: [Errno 5] Input/output error: '/srv/borg/data/0/103'
> 
> So, what's up with that file?

That file was perfectly fine when I manually downloaded it. SO I've done
some experiments with davfs debugging and increased davfs cache and got
this OSError with other data files too. Increasing the cache did help.

So this is again a davfs issue, caused by "borg check" only when davfs
cache is too small.

From tw at waldmann-edv.de  Sun Feb 24 07:49:03 2019
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Sun, 24 Feb 2019 13:49:03 +0100
Subject: [Borgbackup] What to do if borg check fails due to Input/Output
 Error 5?
In-Reply-To: <75469a1a-867b-5c19-3d33-1dd94f430275@jedimail.de>
References: <246aed0e-cfa6-e53f-bec2-7bd1d1af7564@jedimail.de>
 <6b78fec2-b034-950f-9eaf-a88946f532e0@jedimail.de>
 <966399d4-f375-a665-e32b-71f878e1c0b8@waldmann-edv.de>
 <75469a1a-867b-5c19-3d33-1dd94f430275@jedimail.de>
Message-ID: <99f3f102-ea0c-a3eb-475d-3bb7550b3155@waldmann-edv.de>

> So this is again a davfs issue, caused by "borg check" only when davfs
> cache is too small.

If you take this to github or to our IRC channel, we could try some code
modification (disabling our "open files" lru cache), whether that
improves it for davfs.

You need a borg installation where you can edit the python files (*.py
files below the borg/ directory).

-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393

From imperator at jedimail.de  Sun Feb 24 08:55:57 2019
From: imperator at jedimail.de (Sascha Ternes)
Date: Sun, 24 Feb 2019 14:55:57 +0100
Subject: [Borgbackup] What to do if borg check fails due to Input/Output
 Error 5?
In-Reply-To: <99f3f102-ea0c-a3eb-475d-3bb7550b3155@waldmann-edv.de>
References: <246aed0e-cfa6-e53f-bec2-7bd1d1af7564@jedimail.de>
 <6b78fec2-b034-950f-9eaf-a88946f532e0@jedimail.de>
 <966399d4-f375-a665-e32b-71f878e1c0b8@waldmann-edv.de>
 <75469a1a-867b-5c19-3d33-1dd94f430275@jedimail.de>
 <99f3f102-ea0c-a3eb-475d-3bb7550b3155@waldmann-edv.de>
Message-ID: <b286f96b-e82c-3511-2997-932d74d9df3a@jedimail.de>

Am 24.02.19 um 13:49 schrieb Thomas Waldmann:
>> So this is again a davfs issue, caused by "borg check" only when davfs
>> cache is too small.
> 
> If you take this to github or to our IRC channel, we could try some code
> modification (disabling our "open files" lru cache), whether that
> improves it for davfs.
> 
> You need a borg installation where you can edit the python files (*.py
> files below the borg/ directory).

I'm on both GitHub and IRC. Shall I create a new issue or will you do that?

I have installed BorgBackup via Git now and have a 1.2.0dev working.
Currently I'm trying to reproduce the error with a smaller davfs cache.

From imperator at jedimail.de  Mon Feb 25 02:10:05 2019
From: imperator at jedimail.de (Sascha Ternes)
Date: Mon, 25 Feb 2019 08:10:05 +0100
Subject: [Borgbackup] What to do if borg check fails due to Input/Output
 Error 5?
In-Reply-To: <99f3f102-ea0c-a3eb-475d-3bb7550b3155@waldmann-edv.de>
References: <246aed0e-cfa6-e53f-bec2-7bd1d1af7564@jedimail.de>
 <6b78fec2-b034-950f-9eaf-a88946f532e0@jedimail.de>
 <966399d4-f375-a665-e32b-71f878e1c0b8@waldmann-edv.de>
 <75469a1a-867b-5c19-3d33-1dd94f430275@jedimail.de>
 <99f3f102-ea0c-a3eb-475d-3bb7550b3155@waldmann-edv.de>
Message-ID: <0b5495b5-367a-b67f-4ed9-c13e729d41b9@jedimail.de>

@ThomasWaldmann: Master branch works with davfs perfectly!

Some additions for the ML archive:

>> So this is again a davfs issue, caused by "borg check" only when davfs
>> cache is too small.
> 
> If you take this to github or to our IRC channel, we could try some code
> modification (disabling our "open files" lru cache), whether that
> improves it for davfs.

We did some experimenting and were successful. First, some words about
my setup:

- WebDAV mounted via davfs2 module
- custom davfs settings: cache_size 50000 MiByte, table_size 4096
- davfs cache on /var/cache/davfs2 with 120 GiB free space
- BorgBackup repo mounted on /srv/borg, 7.63 TB original size, 117 GB
dedup total, 2225 segments

1. BorgBackup 1.1.9
"borg check --debug -p /srv/borg" crashes when reaching segment 103,
which is ~ 50 GB when davfs cache is full.

2. BorgBackup 1.1.10.dev (branch 1.1-maint), lru cache disabled
"borg check --debug -p /srv/borg" does not crash when reaching cache
full. davfs starts to delete data files 1, 2, 3... while the check
continues.
I aborted the check when I was sure that it does not crash, since a
whole check would take several hours.

3. BorgBackup 1.2.0.dev (branch master), unmodified
Same as 2., borg check does not crash anymore.

Sascha

From tw at waldmann-edv.de  Mon Feb 25 06:35:07 2019
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Mon, 25 Feb 2019 12:35:07 +0100
Subject: [Borgbackup] What to do if borg check fails due to Input/Output
 Error 5?
In-Reply-To: <0b5495b5-367a-b67f-4ed9-c13e729d41b9@jedimail.de>
References: <246aed0e-cfa6-e53f-bec2-7bd1d1af7564@jedimail.de>
 <6b78fec2-b034-950f-9eaf-a88946f532e0@jedimail.de>
 <966399d4-f375-a665-e32b-71f878e1c0b8@waldmann-edv.de>
 <75469a1a-867b-5c19-3d33-1dd94f430275@jedimail.de>
 <99f3f102-ea0c-a3eb-475d-3bb7550b3155@waldmann-edv.de>
 <0b5495b5-367a-b67f-4ed9-c13e729d41b9@jedimail.de>
Message-ID: <06690ccf-82b3-4eff-5da2-a316254b0857@waldmann-edv.de>

> 3. BorgBackup 1.2.0.dev (branch master), unmodified
> Same as 2., borg check does not crash anymore.

I guess this is the relevant change in master that avoids the crash:

https://github.com/borgbackup/borg/pull/3901

The problem is that some filesystems give borg file handles but somehow
do not accept these filehandles at a later time any more (although they
never were closed by borg).

I marked this PR for backporting to 1.1-maint now.

If somebody could do that and also test it heavily, that would be
welcome! I initially did not intend to backport it because this could
also introduce new issues.


-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393

From devzero at web.de  Sat Mar  2 08:59:58 2019
From: devzero at web.de (devzero at web.de)
Date: Sat, 2 Mar 2019 14:59:58 +0100
Subject: [Borgbackup] Fw: pve-zsync + borgbackup "changed size" discrepancy
References: <trinity-689d96d2-effe-440f-a3d6-90c545844807-1551441054846@3c-app-webde-bap36>
Message-ID: <trinity-feceebce-af80-4a7a-bc03-60a10f163191-1551535198002@3c-app-webde-bs09>

Hi, 

i was wondering why borg repo was growing so fast where zfs replication was transferring only 10 times less data (see initial post at the bottom of this mail) - and got this resolved after intensively reading into borg docs and in the end getting across https://github.com/borgbackup/borg/blob/master/docs/misc/create_chunker-params.txt

shouldn't we need some "backup efficiency advisor" which should be able to give a better hint on "how to get the most out of borg" ?

what about a tool which analyses the borg repo and giving a hint for more space efficient usage ?

at least i would have found it useful to have such information at hands (in the manpage for example) what tuning knob we need to look at to make backups of large-files-with-tiny-changes much more space efficient...

mind that even with "--chunker-params 10,23,16,4095" borg backup diff grows up to 7.98MB where xdelta3 reports a diff of only 158218 . so this is still not optimal, but a value i can live with....

thought this could be interesting issue for many, as borg is getting more and more mainstream backup tool, especially for large data files...

regards
roland 


let's have a look on the "real diff" between the two VM snapshots

# xdelta3 -v -c  -s ./zfs-auto-snap_hourly-2019-03-02-1109/images/103/vm-103-disk-0.qcow2 ./zfs-auto-snap_hourly-2019-03-02-1117/images/103/vm-103-disk-0.qcow2 2>&1 >/dev/null |grep bytes
xdelta3: finished in 230 sec; input 34365243392 output 158218 bytes (0.00%)

ok - it's only 160kb binary diff

apparently, the diff in borg (even with compression + dedup) grows to nearly 100mb in size because of sub-optimal default chunker-params for this scenario

# /root/backup/borg-linux64 init -e=none /rpool/borg-backup-replicated/testrepo

# /root/backup/borg-linux64 create --stats -C lz4 /rpool/borg-backup-replicated/testrepo::initial ./zfs-auto-snap_hourly-2019-03-02-1109/images/103/vm-103-disk-0.qcow2
------------------------------------------------------------------------------
Archive name: initial
Archive fingerprint: d5f9ace56a2908723f7b288f013dc11415a2fc7608cabc6bb8b663c9000d2083
Time (start): Sat, 2019-03-02 13:36:45
Time (end):   Sat, 2019-03-02 13:48:08
Duration: 11 minutes 22.85 seconds
Number of files: 1
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:               34.37 GB              8.95 GB              8.86 GB
All archives:               34.37 GB              8.95 GB              8.86 GB

                       Unique chunks         Total chunks
Chunk index:                    5446                 7781
------------------------------------------------------------------------------

# /root/backup/borg-linux64 create --stats -C lz4 /rpool/borg-backup-replicated/testrepo::secondrun ./zfs-auto-snap_hourly-2019-03-02-1117/images/103/vm-103-disk-0.qcow2
------------------------------------------------------------------------------
Archive name: secondrun
Archive fingerprint: b60bf8590aca0e772f9300f9f09e177715f55e0722d33cddf462629232d8aa07
Time (start): Sat, 2019-03-02 13:50:54
Time (end):   Sat, 2019-03-02 13:57:13
Duration: 6 minutes 18.75 seconds
Number of files: 1
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:               34.37 GB              8.95 GB             94.95 MB
All archives:               68.73 GB             17.91 GB              8.95 GB

                       Unique chunks         Total chunks
Chunk index:                    5501                15562
------------------------------------------------------------------------------


so, let's try to tune this  in a better way accoring to https://github.com/borgbackup/borg/blob/master/docs/misc/create_chunker-params.txt :


# rm -rf /rpool/borg-backup-replicated/testrepo/

# /root/backup/borg-linux64 init -e=none /rpool/borg-backup-replicated/testrepo

# /root/backup/borg-linux64 create --stats -C lz4 --chunker-params 10,23,16,4095 /rpool/borg-backup-replicated/testrepo::initial ./zfs-auto-snap_hourly-2019-03-02-1109/images/103/vm-103-disk-0.qcow2
------------------------------------------------------------------------------
Archive name: initial
Archive fingerprint: 6c412446c52547c5cfd6e213bba049d442c1bf8545e1286ca6389c1f4319f376
Time (start): Sat, 2019-03-02 14:03:53
Time (end):   Sat, 2019-03-02 14:16:34
Duration: 12 minutes 41.91 seconds
Number of files: 1
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:               34.37 GB              9.04 GB              8.39 GB
All archives:               34.37 GB              9.04 GB              8.39 GB

                       Unique chunks         Total chunks
Chunk index:                  190164               210448
------------------------------------------------------------------------------

# /root/backup/borg-linux64 create --stats -C lz4 --chunker-params 10,23,16,4095 /rpool/borg-backup-replicated/testrepo::second ./zfs-auto-snap_hourly-2019-03-02-1117/images/103/vm-103-disk-0.qcow2
------------------------------------------------------------------------------
Archive name: second
Archive fingerprint: 05b6f69b89aba8dd7f90da341ca847b1f35f4f7fd76b93d60d409fc5393df6f0
Time (start): Sat, 2019-03-02 14:35:18
Time (end):   Sat, 2019-03-02 14:41:40
Duration: 6 minutes 22.33 seconds
Number of files: 1
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:               34.37 GB              9.04 GB              7.98 MB
All archives:               68.75 GB             18.08 GB              8.40 GB

                       Unique chunks         Total chunks
Chunk index:                  190271               420890
------------------------------------------------------------------------------



> Gesendet: Freitag, 01. M?rz 2019 um 12:50 Uhr
> Von: devzero at web.de
> An: pve-user at pve.proxmox.com
> Betreff: pve-zsync + borgbackup "changed size" discrepancy
>
> hi, 
> i'm syncing a zfs dataset with pve-zsync to another proxmox host and also do backup with borg-backup of the same
> dataset via temporary backup snapshot.
> 
> i have shutdown all VMs on the dataset and only leave a single win10 vm up and running.
> 
> i see a huge discrepancy between what pve-zsync is reporting as "changed data" to be synced to the other node
> and what borbackup is reporting regarding backup growth (see below). ("deduplicated size" means: this is the size 
> the archive grew after the data has been checksummed, deduplicated and compressed)
> 
> does anybody have a clue why this differs by "roughly a power of 10" ?
> 
> is pve-zsync estimation wrong ?
> is borgbackup "overhead" so big when backing up large single qcow2 files ?
> 
> regards
> roland
> 
> pve-zsync estimated size:
> total estimated size is 1.79M
> total estimated size is 22.9M
> total estimated size is 13.5M
> total estimated size is 24.4M
> total estimated size is 22.9M
> total estimated size is 24.1M
> total estimated size is 10.2M
> total estimated size is 20.9M
> total estimated size is 81.0M
> total estimated size is 53.3M
> total estimated size is 10.8M
> total estimated size is 14.1M
> total estimated size is 10.9M
> total estimated size is 189M
> total estimated size is 29.0M
> total estimated size is 21.2M
> total estimated size is 14.1M
> total estimated size is 13.1M
> total estimated size is 11.0M
> total estimated size is 14.1M
> total estimated size is 11.3M
> total estimated size is 14.9M
> total estimated size is 10.5M
> total estimated size is 14.2M
> total estimated size is 20.4M
> 
> BorgBackup growth
>                        Original size      Compressed size    Deduplicated size
> This archive:              111.69 GB             19.86 GB             95.31 MB
> This archive:              111.69 GB             19.86 GB            197.58 MB
> This archive:              111.69 GB             19.86 GB            119.40 MB
> This archive:              111.69 GB             19.86 GB            138.81 MB
> This archive:              111.69 GB             19.86 GB            180.30 MB
> This archive:              111.69 GB             19.86 GB            205.75 MB
> This archive:              111.69 GB             19.86 GB             76.22 MB
> This archive:              111.69 GB             19.86 GB            159.60 MB
> This archive:              111.69 GB             19.86 GB            636.18 MB
> This archive:              111.69 GB             19.86 GB            413.14 MB
> This archive:              111.69 GB             19.86 GB             83.86 MB
> This archive:              111.69 GB             19.86 GB            122.86 MB
> This archive:              111.69 GB             19.86 GB            108.17 MB
> This archive:              111.69 GB             19.86 GB            153.89 MB
> This archive:              111.69 GB             19.89 GB            946.72 MB
> This archive:              111.69 GB             19.89 GB            126.48 MB
> This archive:              111.69 GB             19.89 GB            139.23 MB
> This archive:              111.69 GB             19.89 GB            113.23 MB
> This archive:              111.69 GB             19.89 GB             67.53 MB
> This archive:              111.69 GB             19.89 GB            129.25 MB
> This archive:              111.69 GB             19.89 GB             74.28 MB
> This archive:              111.69 GB             19.89 GB            110.28 MB
> This archive:              111.69 GB             19.89 GB             80.15 MB
> This archive:              111.69 GB             19.89 GB             92.86 MB
> This archive:              111.69 GB             19.89 GB            160.57 MB

From tw at waldmann-edv.de  Mon Mar  4 11:06:05 2019
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Mon, 4 Mar 2019 17:06:05 +0100
Subject: [Borgbackup] Fw: pve-zsync + borgbackup "changed size"
 discrepancy
In-Reply-To: <trinity-feceebce-af80-4a7a-bc03-60a10f163191-1551535198002@3c-app-webde-bs09>
References: <trinity-689d96d2-effe-440f-a3d6-90c545844807-1551441054846@3c-app-webde-bap36>
 <trinity-feceebce-af80-4a7a-bc03-60a10f163191-1551535198002@3c-app-webde-bs09>
Message-ID: <4032e784-9764-24b0-d744-463ccc054636@waldmann-edv.de>

> shouldn't we need some "backup efficiency advisor" which should be able to give a better hint on "how to get the most out of borg" ?

feel free to code one. :)

just be aware that not everybody has the goal of "take as little repo
space as possible".

there are also:
- "do not make my machine run out of memory"
- "be as fast as possible".
- "be able to process a huge amount of data"

> what about a tool which analyses the borg repo and giving a hint for more space efficient usage ?

i guess if you want to determine good chunker params you would need to
backup the source data using different chunker params and compare the
results.

which is a rather expensive operation if you want to try a lot of
different combinations (and maybe your input data is also not small).

also, the chunker is seeded with a random value, so the results might
not be 100% reproducable.

> at least i would have found it useful to have such information at hands (in the manpage for example) what tuning knob we need to look at to make backups of large-files-with-tiny-changes much more space efficient...

if you can improve the docs, do a pull request.

> mind that even with "--chunker-params 10,23,16,4095" borg backup diff grows up to 7.98MB where xdelta3 reports a diff of only 158218 . so this is still not optimal, but a value i can live with....

compared to other tools, borg's goal not finding the minimal diff
between two files, but rather:
- speed
- logically stable chunk cutting points (good if data gets inserted /
removed)
- produce a manageable amount of chunks (with default chunker params)

borg tries to cut chunks of roughly some give target size (default 2MB).

so if your file is 100MB, that is ~ 50 chunks.

you can totally spoil the dedup by 50 times changing 1 byte in each chunk.

if you produce 500 chunks and have the same 50 changes, your dedup still
works in 90% of chunks. but you'll need 10x as much memory to manage the
chunks.

a specialised tool (i don't know xdelta3, but i assume this might be
one) can produce a very small diff by comparing the 2 files - but that
is not how borg works.


-- 


GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393

From devzero at web.de  Mon Mar  4 20:11:19 2019
From: devzero at web.de (devzero at web.de)
Date: Tue, 5 Mar 2019 02:11:19 +0100
Subject: [Borgbackup] Fw: pve-zsync + borgbackup "changed size"
 discrepancy
In-Reply-To: <4032e784-9764-24b0-d744-463ccc054636@waldmann-edv.de>
References: <trinity-689d96d2-effe-440f-a3d6-90c545844807-1551441054846@3c-app-webde-bap36>
 <trinity-feceebce-af80-4a7a-bc03-60a10f163191-1551535198002@3c-app-webde-bs09>
 <4032e784-9764-24b0-d744-463ccc054636@waldmann-edv.de>
Message-ID: <trinity-a761e0e0-2219-45c8-bb39-ea01eeb73c05-1551748279509@3c-app-webde-bap34>

thanks for great explanation!


> Gesendet: Montag, 04. M?rz 2019 um 17:06 Uhr
> Von: "Thomas Waldmann" <tw at waldmann-edv.de>
> An: borgbackup at python.org
> Betreff: Re: [Borgbackup] Fw: pve-zsync + borgbackup "changed size" discrepancy
>
> > shouldn't we need some "backup efficiency advisor" which should be able to give a better hint on "how to get the most out of borg" ?
> 
> feel free to code one. :)
> 
> just be aware that not everybody has the goal of "take as little repo
> space as possible".
> 
> there are also:
> - "do not make my machine run out of memory"
> - "be as fast as possible".
> - "be able to process a huge amount of data"
> 
> > what about a tool which analyses the borg repo and giving a hint for more space efficient usage ?
> 
> i guess if you want to determine good chunker params you would need to
> backup the source data using different chunker params and compare the
> results.
> 
> which is a rather expensive operation if you want to try a lot of
> different combinations (and maybe your input data is also not small).
> 
> also, the chunker is seeded with a random value, so the results might
> not be 100% reproducable.
> 
> > at least i would have found it useful to have such information at hands (in the manpage for example) what tuning knob we need to look at to make backups of large-files-with-tiny-changes much more space efficient...
> 
> if you can improve the docs, do a pull request.
> 
> > mind that even with "--chunker-params 10,23,16,4095" borg backup diff grows up to 7.98MB where xdelta3 reports a diff of only 158218 . so this is still not optimal, but a value i can live with....
> 
> compared to other tools, borg's goal not finding the minimal diff
> between two files, but rather:
> - speed
> - logically stable chunk cutting points (good if data gets inserted /
> removed)
> - produce a manageable amount of chunks (with default chunker params)
> 
> borg tries to cut chunks of roughly some give target size (default 2MB).
> 
> so if your file is 100MB, that is ~ 50 chunks.
> 
> you can totally spoil the dedup by 50 times changing 1 byte in each chunk.
> 
> if you produce 500 chunks and have the same 50 changes, your dedup still
> works in 90% of chunks. but you'll need 10x as much memory to manage the
> chunks.
> 
> a specialised tool (i don't know xdelta3, but i assume this might be
> one) can produce a very small diff by comparing the 2 files - but that
> is not how borg works.
> 
> 
> -- 
> 
> 
> GPG ID: 9F88FB52FAF7B393
> GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393
> _______________________________________________
> Borgbackup mailing list
> Borgbackup at python.org
> https://mail.python.org/mailman/listinfo/borgbackup
>

From jminer74 at post.com  Tue Mar  5 01:01:24 2019
From: jminer74 at post.com (John Miner)
Date: Tue, 5 Mar 2019 07:01:24 +0100
Subject: [Borgbackup] borg check and related bandwidth usage
Message-ID: <trinity-0952238d-c608-4bed-9a80-0fb99f4f4bf8-1551765684525@3c-app-mailcom-lxa06>

Which commands / situations can result in significant egress usage/bandwidth from a remote repo over ssh to a client other than when intentionally mounting to recover files?

For example, when "borg check" is used, the docs say that the repo is checked on the server/target if using ssh ("If you use a remote repo server via ssh:, the repo check is executed on the repo server without causing significant network traffic."). However, for an archive check, the docs say "If you use a remote repo server via ssh:, the archive check is executed on the client machine (because if encryption is enabled, the checks will require decryption and this is always done client-side, because key access will be required)." Does this mean the borg client will download the ENTIRE archive from the server over SSH to complete a check, resulting in significant bandwidth / egress usage if the archive is large? What if the remote repo/server is NOT encrypted, will it still download the entire archive to run a check?

Also, if I delete my local cache on the client, does the entire repo have to be downloaded from the server over ssh in order for that client to rebuild the cache or is it rebuilt on the server side and sent to the client?

I plan on using multi-terabyte archives and a remote backup host, so want to know which commands will cost me say $20 in egress fees every time I run them, not to mention a significant amount of time and potentially blowing my ISP bandwidth cap unintentionally. Maybe the answer is to do a local borg repo and rsync it to a remote host that can do checksums on both ends instead to avoid bandwidth issues with borg?

From MS at TechDesignPsych.com  Thu Mar  7 11:57:16 2019
From: MS at TechDesignPsych.com (Michael Siepmann)
Date: Thu, 7 Mar 2019 09:57:16 -0700
Subject: [Borgbackup] Data integrity error: Invalid segment entry size 0 -
 too small
Message-ID: <569a3fad-8576-40dd-d076-c0f6f0c4a685@TechDesignPsych.com>

I've been using borgbackup for a while without problems but now have an issue I'm unsure how to fix despite looking through the documentation. I'd very much appreciate any help anyone can offer. The two symptoms of the problem are that the backup used to take only a few minutes but now takes over an hour, and that the log shows a data integrity error. 

Below is an abbreviated copy of the log, noting where I've edited out large numbers of similar lines:

Many thanks in advance,

Michael Siepmann

---

Creating archive at "/mnt/synology/Qubes-personal::{hostname}-{now:%Y-%m-%d_T%H:%M}"
Synchronizing chunks cache...
Archives: 1026, w/ cached Idx: 0, w/ outdated Idx: 0, w/o cached Idx: 1026.
Fetching and building archive index for personal-2017-09-30_T16:00 ...
Merging into master chunks index ...
Fetching and building archive index for personal-2017-10-31_T22:00 ...

  <editing out many similar instances...>

Merging into master chunks index ...
Fetching and building archive index for personal-2019-03-01_T07:52 ...
Merging into master chunks index ...
Done.
------------------------------------------------------------------------------
Archive name: personal-2019-03-06_T07:28
Archive fingerprint: 76673c3396fc9ee3f74f927b75e0c7ce0c8d76c7b953469a09c160d2d827fc12
Time (start): Wed, 2019-03-06 07:28:18
Time (end):   Wed, 2019-03-06 08:15:13
Duration: 46 minutes 55.46 seconds
Number of files: 169474
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
		       Original size	  Compressed size    Deduplicated size
This archive:		   109.52 GB		 91.18 GB	       3.15 GB
All archives:		   108.63 TB		 90.24 TB	     335.80 GB

		       Unique chunks	     Total chunks
Chunk index:		      510924		211576111
------------------------------------------------------------------------------
Keeping archive: personal-2019-03-06_T07:28	      Wed, 2019-03-06 07:28:18 [76673c3396fc9ee3f74f927b75e0c7ce0c8d76c7b953469a09c160d2d827fc12]
 
   <editing out several similar instances...>

Keeping archive: personal-2019-02-14_T12:00	      Thu, 2019-02-14 12:00:10 [720753a98c0b33243dd8f1407ea54159833af9691ae4324f1f7ad6149205c1aa]
Pruning archive: personal-2019-02-14_T10:00	      Thu, 2019-02-14 10:00:13 [08cf1a86db65c55cfeeecaab1c6977f7f2baf37e8c6c9803ca3334bd26b9adfc] (1/981)

   <editing out many similar instances - looks like there are 981, but after it gets to 954 there's an error...>

Pruning archive: personal-2018-07-20_T15:00	      Fri, 2018-07-20 15:00:16 [7f3e8e3477faf30de92fefec41baf2243045c085b8a957b9d764a63202ad16a8] (954/981)
Data integrity error: Invalid segment entry size 0 - too small [segment 253305, offset 2331522]
Traceback (most recent call last):
  File "/usr/lib64/python3.7/site-packages/borg/archiver.py", line 4436, in main
    exit_code = archiver.run(args)
  File "/usr/lib64/python3.7/site-packages/borg/archiver.py", line 4368, in run
    return set_ec(func(args))
  File "/usr/lib64/python3.7/site-packages/borg/archiver.py", line 152, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/lib64/python3.7/site-packages/borg/archiver.py", line 1590, in do_prune
    Archive(repository, key, manifest, archive.name, cache).delete(stats, forced=args.forced)
  File "/usr/lib64/python3.7/site-packages/borg/archive.py", line 809, in delete
    chunk_decref(chunk_id, stats)
  File "/usr/lib64/python3.7/site-packages/borg/archive.py", line 786, in chunk_decref
    self.cache.chunk_decref(id, stats, wait=False)
  File "/usr/lib64/python3.7/site-packages/borg/cache.py", line 925, in chunk_decref
    self.repository.delete(id, wait=wait)
  File "/usr/lib64/python3.7/site-packages/borg/repository.py", line 1113, in delete
    size = self.io.read(segment, offset, id, read_data=False)
  File "/usr/lib64/python3.7/site-packages/borg/repository.py", line 1392, in read
    size, tag, key, data = self._read(fd, self.put_header_fmt, header, segment, offset, (TAG_PUT, ), read_data)
  File "/usr/lib64/python3.7/site-packages/borg/repository.py", line 1419, in _read
    size, segment, offset))
borg.helpers.IntegrityError: Data integrity error: Invalid segment entry size 0 - too small [segment 253305, offset 2331522]

Platform: Linux personal 4.14.74-1.pvops.qubes.x86_64 #1 SMP Mon Oct 8 17:14:24 UTC 2018 x86_64
Linux: Fedora 29 Twenty Nine
Borg: 1.1.8  Python: CPython 3.7.2
PID: 12045  CWD: /home/user/Apps/ScriptsByMMS
sys.argv: ['/usr/bin/borg', 'prune', '-v', '--list', '/mnt/synology/Qubes-personal', '--prefix', '{hostname}-', '--keep-hourly=24', '--keep-daily=7', '--keep-weekly=6', '--keep-monthly=9']
SSH_ORIGINAL_COMMAND: None


From tw at waldmann-edv.de  Thu Mar  7 12:27:44 2019
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Thu, 7 Mar 2019 18:27:44 +0100
Subject: [Borgbackup] Data integrity error: Invalid segment entry size 0
 - too small
In-Reply-To: <569a3fad-8576-40dd-d076-c0f6f0c4a685@TechDesignPsych.com>
References: <569a3fad-8576-40dd-d076-c0f6f0c4a685@TechDesignPsych.com>
Message-ID: <e7409d7a-3b57-2c50-1813-81eca93bff5d@waldmann-edv.de>

> Data integrity error: Invalid segment entry size 0 - too small [segment 253305, offset 2331522]

Maybe have a look at repo_dir/data/.../253305 file.

How long is it? Can it be read w/o problems?

> borg.helpers.IntegrityError: Data integrity error: Invalid segment entry size 0 - too small [segment 253305, offset 2331522]

Possible explanations:

- data file in repo corrupt
- repo index corrupt (pointing to a bad segment/offset location for some 
object)
- in-memory corruption happening

Corruption can be caused by: bad RAM, hard disk issues, cabling, 
filesystem and other OS issues.

Before trying to borg check --repair the repo, you should rule out 
hardware issues (or it might get worse), so:
- run a full run of memtest86+ on repo server and borg client.
- also run smartctl -t long on the repo disk and check the results 
afterwards with smartctl -a.

> Platform: Linux personal 4.14.74-1.pvops.qubes.x86_64 #1 SMP Mon Oct 8 17:14:24 UTC 2018 x86_64

Yay, Qubes \o/

> Linux: Fedora 29 Twenty Nine
> Borg: 1.1.8  Python: CPython 3.7.2

There is 1.1.9 (just saying, not related to your problem).


-- 

GPG Fingerprint: 6D5B EF9A DD20 7580 5747  B70F 9F88 FB52 FAF7 B393
Encrypted E-Mail is preferred / Verschluesselte E-Mail wird bevorzugt.

From MS at TechDesignPsych.com  Fri Mar  8 10:37:59 2019
From: MS at TechDesignPsych.com (Michael Siepmann)
Date: Fri, 8 Mar 2019 08:37:59 -0700
Subject: [Borgbackup] Data integrity error: Invalid segment entry size 0
 - too small
In-Reply-To: <e7409d7a-3b57-2c50-1813-81eca93bff5d@waldmann-edv.de>
References: <569a3fad-8576-40dd-d076-c0f6f0c4a685@TechDesignPsych.com>
 <e7409d7a-3b57-2c50-1813-81eca93bff5d@waldmann-edv.de>
Message-ID: <bf06358e-e360-98d1-e9b8-3300bc30055b@TechDesignPsych.com>


On 3/7/19 10:27 AM, Thomas Waldmann wrote:
>> Data integrity error: Invalid segment entry size 0 - too small
>> [segment 253305, offset 2331522]
>
> Maybe have a look at repo_dir/data/.../253305 file.
>
> How long is it? Can it be read w/o problems?

ls -la 253305 shows this:

-rw------- 1 M-Laptop users 6743797 Jul 20? 2018 253305

When I do "cat 253305" it seems to work without an error, and copying it
to another location with cp also seemed to work fine. Is there a better
way to test whether it can be read without problems?

>
>> borg.helpers.IntegrityError: Data integrity error: Invalid segment
>> entry size 0 - too small [segment 253305, offset 2331522]
>
> Possible explanations:
>
> - data file in repo corrupt
> - repo index corrupt (pointing to a bad segment/offset location for
> some object)
> - in-memory corruption happening
>
> Corruption can be caused by: bad RAM, hard disk issues, cabling,
> filesystem and other OS issues.
>
> Before trying to borg check --repair the repo, you should rule out
> hardware issues (or it might get worse), so:
> - run a full run of memtest86+ on repo server and borg client.
> - also run smartctl -t long on the repo disk and check the results
> afterwards with smartctl -a.

I did those, and found no problems. I couldn't use memtest86+ on the
server since it's a Synology Diskstation, but it has its own memtest
option and it passed that.

I tried "borg -v check --repository-only $REPOSITORY" before, without
the repair option, and it did nothing beyond saying "Starting repository
check" for many hours. I'm running that again now (without the repair
option for now). How long might I need to wait to let it complete?


>
>> Platform: Linux personal 4.14.74-1.pvops.qubes.x86_64 #1 SMP Mon Oct
>> 8 17:14:24 UTC 2018 x86_64
>
> Yay, Qubes \o/

:-)

>
>> Linux: Fedora 29 Twenty Nine
>> Borg: 1.1.8? Python: CPython 3.7.2
>
> There is 1.1.9 (just saying, not related to your problem).
>
Good to know. I guess the Fedora 29 repo hasn't upgraded to that yet.

Thanks!


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20190308/c4d377fd/attachment.sig>

From imperator at jedimail.de  Fri Mar  8 11:30:44 2019
From: imperator at jedimail.de (Sascha Ternes)
Date: Fri, 8 Mar 2019 17:30:44 +0100
Subject: [Borgbackup] Data integrity error: Invalid segment entry size 0
 - too small
In-Reply-To: <bf06358e-e360-98d1-e9b8-3300bc30055b@TechDesignPsych.com>
References: <569a3fad-8576-40dd-d076-c0f6f0c4a685@TechDesignPsych.com>
 <e7409d7a-3b57-2c50-1813-81eca93bff5d@waldmann-edv.de>
 <bf06358e-e360-98d1-e9b8-3300bc30055b@TechDesignPsych.com>
Message-ID: <21030228-da4a-055d-09c7-2dfbebed3efc@jedimail.de>

Am 08.03.19 um 16:37 schrieb Michael Siepmann:
> I tried "borg -v check --repository-only $REPOSITORY" before, without
> the repair option, and it did nothing beyond saying "Starting repository
> check" for many hours. I'm running that again now (without the repair
> option for now). How long might I need to wait to let it complete?

Try borg check with -p / --progress which will give you a percentage
indicator.

From MS at TechDesignPsych.com  Fri Mar  8 14:17:58 2019
From: MS at TechDesignPsych.com (Michael Siepmann)
Date: Fri, 8 Mar 2019 12:17:58 -0700
Subject: [Borgbackup] Data integrity error: Invalid segment entry size 0
 - too small
In-Reply-To: <21030228-da4a-055d-09c7-2dfbebed3efc@jedimail.de>
References: <569a3fad-8576-40dd-d076-c0f6f0c4a685@TechDesignPsych.com>
 <e7409d7a-3b57-2c50-1813-81eca93bff5d@waldmann-edv.de>
 <bf06358e-e360-98d1-e9b8-3300bc30055b@TechDesignPsych.com>
 <21030228-da4a-055d-09c7-2dfbebed3efc@jedimail.de>
Message-ID: <0e868a91-7051-a315-522a-9b967831728e@TechDesignPsych.com>


On 3/8/19 9:30 AM, Sascha Ternes wrote:
> Am 08.03.19 um 16:37 schrieb Michael Siepmann:
>> I tried "borg -v check --repository-only $REPOSITORY" before, without
>> the repair option, and it did nothing beyond saying "Starting repository
>> check" for many hours. I'm running that again now (without the repair
>> option for now). How long might I need to wait to let it complete?
> Try borg check with -p / --progress which will give you a percentage
> indicator.

Thank you. That's very helpful. It's taken 130 min to get to "Checking
segments 8.1%" so I can infer it will probably take almost 27 hours in
total to finish checking the segments.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20190308/2cbc21cc/attachment.sig>

From imperator at jedimail.de  Fri Mar  8 15:04:32 2019
From: imperator at jedimail.de (Sascha Ternes)
Date: Fri, 8 Mar 2019 21:04:32 +0100
Subject: [Borgbackup] Data integrity error: Invalid segment entry size 0
 - too small
In-Reply-To: <0e868a91-7051-a315-522a-9b967831728e@TechDesignPsych.com>
References: <569a3fad-8576-40dd-d076-c0f6f0c4a685@TechDesignPsych.com>
 <e7409d7a-3b57-2c50-1813-81eca93bff5d@waldmann-edv.de>
 <bf06358e-e360-98d1-e9b8-3300bc30055b@TechDesignPsych.com>
 <21030228-da4a-055d-09c7-2dfbebed3efc@jedimail.de>
 <0e868a91-7051-a315-522a-9b967831728e@TechDesignPsych.com>
Message-ID: <e8fd13c9-8bdb-14df-b711-20067ce19656@jedimail.de>

Am 08.03.19 um 20:17 schrieb Michael Siepmann:
> On 3/8/19 9:30 AM, Sascha Ternes wrote:
>> Am 08.03.19 um 16:37 schrieb Michael Siepmann:
>>> I tried "borg -v check --repository-only $REPOSITORY" before, without
>>> the repair option, and it did nothing beyond saying "Starting repository
>>> check" for many hours. I'm running that again now (without the repair
>>> option for now). How long might I need to wait to let it complete?
>> Try borg check with -p / --progress which will give you a percentage
>> indicator.
> 
> Thank you. That's very helpful. It's taken 130 min to get to "Checking
> segments 8.1%" so I can infer it will probably take almost 27 hours in
> total to finish checking the segments.

No. This is a file counter. In a repository with much deduplication you
will have a lot of very small data files ("segments"). These will be
checked very fast, compared to the big segments of your very first
archive in this repository. I guess it will take much less than 27 hours.

From MS at TechDesignPsych.com  Sun Mar 10 10:34:41 2019
From: MS at TechDesignPsych.com (Michael Siepmann)
Date: Sun, 10 Mar 2019 08:34:41 -0600
Subject: [Borgbackup] Data integrity error: Invalid segment entry size 0
 - too small
In-Reply-To: <e8fd13c9-8bdb-14df-b711-20067ce19656@jedimail.de>
References: <569a3fad-8576-40dd-d076-c0f6f0c4a685@TechDesignPsych.com>
 <e7409d7a-3b57-2c50-1813-81eca93bff5d@waldmann-edv.de>
 <bf06358e-e360-98d1-e9b8-3300bc30055b@TechDesignPsych.com>
 <21030228-da4a-055d-09c7-2dfbebed3efc@jedimail.de>
 <0e868a91-7051-a315-522a-9b967831728e@TechDesignPsych.com>
 <e8fd13c9-8bdb-14df-b711-20067ce19656@jedimail.de>
Message-ID: <1a6f53b3-a3a2-f98b-a33d-f0c0169fc1d4@TechDesignPsych.com>

On 3/8/19 1:04 PM, Sascha Ternes wrote:
> Am 08.03.19 um 20:17 schrieb Michael Siepmann:
>> On 3/8/19 9:30 AM, Sascha Ternes wrote:
>>> Am 08.03.19 um 16:37 schrieb Michael Siepmann:
>>>> I tried "borg -v check --repository-only $REPOSITORY" before, without
>>>> the repair option, and it did nothing beyond saying "Starting repository
>>>> check" for many hours. I'm running that again now (without the repair
>>>> option for now). How long might I need to wait to let it complete?
>>> Try borg check with -p / --progress which will give you a percentage
>>> indicator.
>> Thank you. That's very helpful. It's taken 130 min to get to "Checking
>> segments 8.1%" so I can infer it will probably take almost 27 hours in
>> total to finish checking the segments.
> No. This is a file counter. In a repository with much deduplication you
> will have a lot of very small data files ("segments"). These will be
> checked very fast, compared to the big segments of your very first
> archive in this repository. I guess it will take much less than 27 hours.

It took about 20 hours and said this:

===

Starting repository check
Data integrity error: Invalid segment entry size 0 - too small [segment
253305, offset 2331522]???????????????????????????????????????????
attempting to recover /mnt/synology/Qubes-personal/data/25/253305
Starting repository index
check???????????????????????????????????????????????????????????????????????????????????????????????????????????

Completed repository check, errors found and repaired.

===

I then tried running my backup script again and this time it ended with
this error:

===

Pruning archive: personal-2018-07-20_T15:00??? ????? Fri, 2018-07-20
15:00:16
[7f3e8e3477faf30de92fefec41baf2243045c085b8a957b9d764a63202ad16a8] (955/982)
Object with key
aaffc27b2efdca19941a533c2d62644e9d1bfb9867f030378eca7df6c9e5df1f not
found in repository /mnt/synology/Qubes-personal.
Traceback (most recent call last):
? File "/usr/lib64/python3.7/site-packages/borg/archiver.py", line 4436,
in main
??? exit_code = archiver.run(args)
? File "/usr/lib64/python3.7/site-packages/borg/archiver.py", line 4368,
in run
??? return set_ec(func(args))
? File "/usr/lib64/python3.7/site-packages/borg/archiver.py", line 152,
in wrapper
??? return method(self, args, repository=repository, **kwargs)
? File "/usr/lib64/python3.7/site-packages/borg/archiver.py", line 1590,
in do_prune
??? Archive(repository, key, manifest, archive.name,
cache).delete(stats, forced=args.forced)
? File "/usr/lib64/python3.7/site-packages/borg/archive.py", line 809,
in delete
??? chunk_decref(chunk_id, stats)
? File "/usr/lib64/python3.7/site-packages/borg/archive.py", line 786,
in chunk_decref
??? self.cache.chunk_decref(id, stats, wait=False)
? File "/usr/lib64/python3.7/site-packages/borg/cache.py", line 925, in
chunk_decref
??? self.repository.delete(id, wait=wait)
? File "/usr/lib64/python3.7/site-packages/borg/repository.py", line
1110, in delete
??? raise self.ObjectNotFound(id, self.path) from None
borg.repository.Repository.ObjectNotFound: Object with key
aaffc27b2efdca19941a533c2d62644e9d1bfb9867f030378eca7df6c9e5df1f not
found in repository /mnt/synology/Qubes-personal.

Platform: Linux personal 4.14.74-1.pvops.qubes.x86_64 #1 SMP Mon Oct 8
17:14:24 UTC 2018 x86_64
Linux: Fedora 29 Twenty Nine
Borg: 1.1.8? Python: CPython 3.7.2
PID: 17824? CWD: /home/user/Apps/ScriptsByMMS
sys.argv: ['/usr/bin/borg', 'prune', '-v', '--list',
'/mnt/synology/Qubes-personal', '--prefix', '{hostname}-',
'--keep-hourly=24', '--keep-daily=7', '--keep-weekly=6', '--keep-monthly=9']
SSH_ORIGINAL_COMMAND: None

===

Any suggestions anyone has on what to do next would be much appreciated.
Thanks!


From tw at waldmann-edv.de  Sun Mar 10 10:55:53 2019
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Sun, 10 Mar 2019 15:55:53 +0100
Subject: [Borgbackup] Data integrity error: Invalid segment entry size 0
 - too small
In-Reply-To: <1a6f53b3-a3a2-f98b-a33d-f0c0169fc1d4@TechDesignPsych.com>
References: <569a3fad-8576-40dd-d076-c0f6f0c4a685@TechDesignPsych.com>
 <e7409d7a-3b57-2c50-1813-81eca93bff5d@waldmann-edv.de>
 <bf06358e-e360-98d1-e9b8-3300bc30055b@TechDesignPsych.com>
 <21030228-da4a-055d-09c7-2dfbebed3efc@jedimail.de>
 <0e868a91-7051-a315-522a-9b967831728e@TechDesignPsych.com>
 <e8fd13c9-8bdb-14df-b711-20067ce19656@jedimail.de>
 <1a6f53b3-a3a2-f98b-a33d-f0c0169fc1d4@TechDesignPsych.com>
Message-ID: <b7167890-344f-5354-d72b-f20e8c53383e@waldmann-edv.de>

> Starting repository check
> Data integrity error: Invalid segment entry size 0 - too small [segment
> 253305, offset 2331522]???????????????????????????????????????????
> attempting to recover /mnt/synology/Qubes-personal/data/25/253305

Here, it likely discarded one object ("segment entry") from the repo.

Because you checked with --repository-only, it did not do an archives
check / consistency fix, so it might well be the case that some archive
is still pointing to the object it just deleted (because it was corrupt)
and this is likely why you get:

> Pruning archive: personal-2018-07-20_T15:00??? ????? Fri, 2018-07-20
> 15:00:16
> [7f3e8e3477faf30de92fefec41baf2243045c085b8a957b9d764a63202ad16a8] (955/982)
> Object with key
> aaffc27b2efdca19941a533c2d62644e9d1bfb9867f030378eca7df6c9e5df1f not
> found in repository /mnt/synology/Qubes-personal.

So, what you can do is a full borg check --repair.

If you are still confident in the repo being ok now and no hardware
corrupting anything, you could also try with --archives-only to only do
what you did not already do in the first check/repair run (and not
repeat the repository part).

It will then try to make the archives consistent again. If a content
data chunk object is missing, it will replace it by an all-zero
replacement chunk of same length.

After that, do backup(s) of your data (if you are lucky, the lost
content chunk will still be produced by borg create).

Then do another borg check --repair. If you are lucky, it can heal the
archives, replacing the all-zero chunk by the correct chunk.

TODO: do we have this ^^^^ in the docs / faq?


-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393

From MS at TechDesignPsych.com  Tue Mar 12 10:47:39 2019
From: MS at TechDesignPsych.com (Michael Siepmann)
Date: Tue, 12 Mar 2019 08:47:39 -0600
Subject: [Borgbackup] Data integrity error: Invalid segment entry size 0
 - too small
In-Reply-To: <b7167890-344f-5354-d72b-f20e8c53383e@waldmann-edv.de>
References: <569a3fad-8576-40dd-d076-c0f6f0c4a685@TechDesignPsych.com>
 <e7409d7a-3b57-2c50-1813-81eca93bff5d@waldmann-edv.de>
 <bf06358e-e360-98d1-e9b8-3300bc30055b@TechDesignPsych.com>
 <21030228-da4a-055d-09c7-2dfbebed3efc@jedimail.de>
 <0e868a91-7051-a315-522a-9b967831728e@TechDesignPsych.com>
 <e8fd13c9-8bdb-14df-b711-20067ce19656@jedimail.de>
 <1a6f53b3-a3a2-f98b-a33d-f0c0169fc1d4@TechDesignPsych.com>
 <b7167890-344f-5354-d72b-f20e8c53383e@waldmann-edv.de>
Message-ID: <7969e393-4a3f-60ae-85e1-08f81cc6d22d@TechDesignPsych.com>

?

On 3/10/19 8:55 AM, Thomas Waldmann wrote:
>> Starting repository check
>> Data integrity error: Invalid segment entry size 0 - too small [segment
>> 253305, offset 2331522]???????????????????????????????????????????
>> attempting to recover /mnt/synology/Qubes-personal/data/25/253305
> Here, it likely discarded one object ("segment entry") from the repo.
>
> Because you checked with --repository-only, it did not do an archives
> check / consistency fix, so it might well be the case that some archive
> is still pointing to the object it just deleted (because it was corrupt)
> and this is likely why you get:
>
>> Pruning archive: personal-2018-07-20_T15:00??? ????? Fri, 2018-07-20
>> 15:00:16
>> [7f3e8e3477faf30de92fefec41baf2243045c085b8a957b9d764a63202ad16a8] (955/982)
>> Object with key
>> aaffc27b2efdca19941a533c2d62644e9d1bfb9867f030378eca7df6c9e5df1f not
>> found in repository /mnt/synology/Qubes-personal.
> So, what you can do is a full borg check --repair.
>
> If you are still confident in the repo being ok now and no hardware
> corrupting anything, you could also try with --archives-only to only do
> what you did not already do in the first check/repair run (and not
> repeat the repository part).
>
> It will then try to make the archives consistent again. If a content
> data chunk object is missing, it will replace it by an all-zero
> replacement chunk of same length.
>
> After that, do backup(s) of your data (if you are lucky, the lost
> content chunk will still be produced by borg create).
>
> Then do another borg check --repair. If you are lucky, it can heal the
> archives, replacing the all-zero chunk by the correct chunk.

It worked! Thanks so much for your help!

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20190312/1a276082/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20190312/1a276082/attachment.sig>

From tw at waldmann-edv.de  Thu Mar 21 17:01:56 2019
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Thu, 21 Mar 2019 22:01:56 +0100
Subject: [Borgbackup] borgbackup 1.2.0a5 alpha release
Message-ID: <8ed6ef55-ade6-8669-5040-bcad0aff5e30@waldmann-edv.de>


released a new alpha test version of borgbackup.


this one is especially interesting for package and platform maintainers
due to major changes in the build system (using pkgconfig now), see
changelog and setup.py.


https://github.com/borgbackup/borg/releases/tag/1.2.0a5


-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393

From tw at waldmann-edv.de  Thu Mar 21 17:42:36 2019
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Thu, 21 Mar 2019 22:42:36 +0100
Subject: [Borgbackup] for package / platform maintainers: about 1.2.0a5
In-Reply-To: <8ed6ef55-ade6-8669-5040-bcad0aff5e30@waldmann-edv.de>
References: <8ed6ef55-ade6-8669-5040-bcad0aff5e30@waldmann-edv.de>
Message-ID: <52ac6c1e-bf62-aa3e-df3d-a2566be006f4@waldmann-edv.de>

If you maintain a borgbackup package for some (linux, bsd, ...)
distribution or you are interested in porting to new platforms (Haiku,
anyone?), I have some notes for you and you can test it with 1.2.0a5.

pkg-config support
==================

I did a big refactoring of the setup*.py code, introducing pkg-config
support (via the pkgconfig python package, which must be installed
before invoking setup.py) and also simplifying the code a lot.

Some notes how borg 1.2.0a5 finds headers/libraries or falls back to
bundled code are there:

https://github.com/borgbackup/borg/blob/1.2.0a5/setup.py#L27

required libraries
==================

As the easiest way to find libs is just having the lib support
pkg-config, it would be good if you all could push for better pkg-config
support for the libs borg requires:

- openssl (and compatible libs) - for example, it seems that freebsd 12
has no pkg-config support for openssl, this is why I needed to point to
is via BORG_OPENSSL_PREFIX=/usr .

- libb2 (blake2 hash / mac) - until very recently, this lib had no
pkg-config support, but Ronny Pfannschmidt added it (thanks!). After
this, there was a new release 0.98.1 with these changes, so borg could
find libb2 now IF >=0.98.1 is installed. Earlier libb2 versions also had
a serious bug for some AMD cpus. so you really want to make sure you
have >= 0.98.1.

- liblz4 and libzstd: just check if they are recent enough (they also
should have pkg-config support).

- pkgconfig python package: there recently was some work to improve it
and I hope there will be a release of that soon (e.g. it had troubles to
parse the openssl version number, which I fixed). Keep an eye on it,
make sure you have the latest and greatest release.

advantages
==========

In general, by supporting pkg-config for the libs, you can help to have
a clean separation and keep 3rd party code in separately maintainable
packages (instead of borg falling back to bundled code).

Some distributions which have libs and headers at unusual places also
will be supported without an issue, IF there is appropriate pkg-config
support.

One day, if everybody has these libs, we maybe could even remove the
bundled code again.


-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393

From tve at voneicken.com  Fri Mar 22 18:36:40 2019
From: tve at voneicken.com (Thorsten von Eicken)
Date: Fri, 22 Mar 2019 22:36:40 +0000
Subject: [Borgbackup] merging repositories
Message-ID: <01000169a78c8f9b-f35c1c99-3896-4020-b8f4-d411e9783d6e-000000@email.amazonses.com>

Is there a way to consolidate archives from multiple repositories into a 
new repository?

I've been backing up multiple systems each into its own repository for 
several years and I'd like to create a consolidated repository for 
archival purposes. The reason for consolidating is deduplication because 
it turns out that I've moved large directory trees from one system to 
another over time. I would continue adding archives to this consolidated 
repository, but at a slow scale, e.g. monthly, so I'm OK with the cache 
synchronization overhead and with ensuring that only one system creates 
an archive at a time.

I have the feeling that right now the best I can do is borg mount an old 
archive and then run a borg create against the mount point. Is that 
correct? It would be cool if I could pipe borg export-tar into borg 
create, but that doesn't seem supported. Is there another option I'm 
missing?

Thanks!

Thorsten

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20190322/24903ae7/attachment.html>

From mszpak at wp.pl  Sun Mar 24 17:34:22 2019
From: mszpak at wp.pl (=?UTF-8?Q?Marcin_Zaj=c4=85czkowski?=)
Date: Sun, 24 Mar 2019 22:34:22 +0100
Subject: [Borgbackup] Reasons of remove --compression-from
Message-ID: <019c216b-2f56-7ffa-79ec-937e06b5224f@wp.pl>

Hi

I was looking for information how Borg deals with compression of
different kind of files and I noticed that in the auto mode it probes if
the file "can be compressed" (with a hardcoded percent value to
determine that) . It's nice, but for some file extensions it's a waste
of time/CPU (I have a drive with a lot of photos, together with some
other kind of files) and "--compression-from" would be a nice complement.

Unfortunately it was removed in August 2017:
https://github.com/borgbackup/borg/commit/69fb9bd40371813d85cad112599eac70f8c881eb

I wonder what was the reason and if Borg can by any chance do it in some
other way as of 1.1/1.2?

Marcin

-- 
https://blog.solidsoft.info/ - Working code is not enough

From tw at waldmann-edv.de  Sun Mar 24 18:08:04 2019
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Sun, 24 Mar 2019 23:08:04 +0100
Subject: [Borgbackup] Reasons of remove --compression-from
In-Reply-To: <019c216b-2f56-7ffa-79ec-937e06b5224f@wp.pl>
References: <019c216b-2f56-7ffa-79ec-937e06b5224f@wp.pl>
Message-ID: <5c03f39d-4586-b0ff-387a-e8e929ef1aa5@waldmann-edv.de>

> I wonder what was the reason and if Borg can by any chance do it in some
> other way as of 1.1/1.2?

The reason was that "auto" is way easier from a user perspective and not
expensive on the CPU.

The previously planned feature would have required to maintain a file
with a list of file formats and advised compression algorithms, reliably
detect file format (which is an non-trivial issue by itself) and then
still having an issue with file formats that might be compressed or not
internally.

You could find the discussion on borgbackup's github issue tracker.


-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393

From mszpak at wp.pl  Sun Mar 24 19:21:29 2019
From: mszpak at wp.pl (=?UTF-8?Q?Marcin_Zaj=c4=85czkowski?=)
Date: Mon, 25 Mar 2019 00:21:29 +0100
Subject: [Borgbackup] Reasons of remove --compression-from
In-Reply-To: <5c03f39d-4586-b0ff-387a-e8e929ef1aa5@waldmann-edv.de>
References: <019c216b-2f56-7ffa-79ec-937e06b5224f@wp.pl>
 <5c03f39d-4586-b0ff-387a-e8e929ef1aa5@waldmann-edv.de>
Message-ID: <24ae67b6-664c-b448-346a-4743b93962bd@wp.pl>

Thanks for your reply.

On 2019-03-24 23:08, Thomas Waldmann wrote:
>> I wonder what was the reason and if Borg can by any chance do it in some
>> other way as of 1.1/1.2?
> 
> The reason was that "auto" is way easier from a user perspective and not
> expensive on the CPU.
> 
> The previously planned feature would have required to maintain a file
> with a list of file formats and advised compression algorithms, reliably
> detect file format (which is an non-trivial issue by itself) and then
> still having an issue with file formats that might be compressed or not
> internally.

In fact, auto is much easier to use, however, it is much less flexible.
Especially the ratio determining if file should be compressed seems to
be hardcoded in code to 0.97:
https://github.com/borgbackup/borg/blob/2b16fc9039660abba0ce6f5a25ae9c0f31ad48f5/src/borg/compress.pyx#L317

It is not the best value in some situations. My test case - 760MB of JPG
photos. Two different empty repos on /tmp/ (in memory) with AES encryption.

Without compression:
> time -p borg create --progress --compression none --list --filter=AME
--stats ...
> ...
> Duration: 15.18 seconds
> Number of files: 128
> Utilization of max. archive size: 0%
> ------------------------------------------------------------------------------
>                        Original size      Compressed size    Deduplicated size
> This archive:              761.66 MB            761.68 MB            761.68 MB
> All archives:              761.66 MB            761.68 MB            761.68 MB
> 
>                        Unique chunks         Total chunks
> Chunk index:                     411                  411
> ------------------------------------------------------------------------------
> real 16.20
> user 10.43
> sys 0.89


With auto,lmza:

> $ time -p borg create --progress --compression auto,lzma --list --filter=AME --stats ...
> ...
> Duration: 1 minutes 53.94 seconds
> Number of files: 128
> Utilization of max. archive size: 0%
> ------------------------------------------------------------------------------
>                        Original size      Compressed size    Deduplicated size
> This archive:              761.66 MB            757.23 MB            757.23 MB
> All archives:              761.66 MB            757.23 MB            757.23 MB
> 
>                        Unique chunks         Total chunks
> Chunk index:                     421                  421
> ------------------------------------------------------------------------------
> real 115.02
> user 102.09
> sys 3.36

16 vs 115 seconds is noticeable. Especially that I have GBs of photos.

Why lzma? Over time, it is better for me to have smaller size of the
backup (to keep more snapshots) over backup duration (it can be done "in
background").

As a workaround, I could try to divide my data into more groups, but it
is much less convenient to manage, especially that for example 7z
archives can be placed in the same directory structure as "normal" files.

I suspect the ratio is to high in my case. I would like to have an
ability to change it from a command line. However, even better would be
an ability to define, at least, a list of extensions that should be
ignored from compression (as a lighter version of the removed mechanism).

What do you think about that?

Marcin

-- 
https://blog.solidsoft.info/ - Working code is not enough

From tw at waldmann-edv.de  Mon Mar 25 09:07:52 2019
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Mon, 25 Mar 2019 14:07:52 +0100
Subject: [Borgbackup] Reasons of remove --compression-from
In-Reply-To: <24ae67b6-664c-b448-346a-4743b93962bd@wp.pl>
References: <019c216b-2f56-7ffa-79ec-937e06b5224f@wp.pl>
 <5c03f39d-4586-b0ff-387a-e8e929ef1aa5@waldmann-edv.de>
 <24ae67b6-664c-b448-346a-4743b93962bd@wp.pl>
Message-ID: <d035c36d-826d-91c7-0e47-e8eb6907f958@waldmann-edv.de>

> In fact, auto is much easier to use, however, it is much less flexible.
> Especially the ratio determining if file should be compressed seems to
> be hardcoded in code to 0.97:
> https://github.com/borgbackup/borg/blob/2b16fc9039660abba0ce6f5a25ae9c0f31ad48f5/src/borg/compress.pyx#L317

That could be:
- adjusted
- if no generally good value can be determined, it can be made an option

> It is not the best value in some situations. My test case - 760MB of JPG
> photos. Two different empty repos on /tmp/ (in memory) with AES encryption.
> 
> Without compression:
>> time -p borg create --progress --compression none --list --filter=AME
> --stats ...
>> ...
>> Duration: 15.18 seconds
>> Number of files: 128
>> Utilization of max. archive size: 0%
>> ------------------------------------------------------------------------------
>>                        Original size      Compressed size    Deduplicated size
>> This archive:              761.66 MB            761.68 MB            761.68 MB
>> All archives:              761.66 MB            761.68 MB            761.68 MB
>>
>>                        Unique chunks         Total chunks
>> Chunk index:                     411                  411
>> ------------------------------------------------------------------------------
>> real 16.20
>> user 10.43
>> sys 0.89

On unmodified code, could you also run just lzma compression mode
without auto, to see if it takes about 1:53 for your data, which would
indicate the current threshold takes the wrong decision for your data?

After that, can you adjust the value in the borg src and find the
highest one that gives close-to-uncompressed performance with auto,lzma?

Then, maybe verify doing the same with a bigger amount of photos.

Not sure if --debug option gives some more clues, but just try it to see
if it gives useful output for your case.

> 16 vs 115 seconds is noticeable. Especially that I have GBs of photos.

Lots of photos are a quite widespread use case, so we can adjust for that.

> Why lzma? Over time, it is better for me to have smaller size of the
> backup (to keep more snapshots) over backup duration (it can be done "in
> background").

zstd might be a more modern option.

> I suspect the ratio is to high in my case. I would like to have an
> ability to change it from a command line.

We can do that if no good value can be determined.
We should avoid adding lots of commandline options though as it makes
using borg harder and the documentation you need to read bigger.

> However, even better would be
> an ability to define, at least, a list of extensions that should be
> ignored from compression (as a lighter version of the removed mechanism).

Similar for that plus the effort of having to maintain yet another list.



-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393

From melkor.lord at gmail.com  Mon Mar 25 13:51:31 2019
From: melkor.lord at gmail.com (Melkor Lord)
Date: Mon, 25 Mar 2019 18:51:31 +0100
Subject: [Borgbackup] borgbackup 1.2.0a5 alpha release
In-Reply-To: <8ed6ef55-ade6-8669-5040-bcad0aff5e30@waldmann-edv.de>
References: <8ed6ef55-ade6-8669-5040-bcad0aff5e30@waldmann-edv.de>
Message-ID: <aac7d81e-1346-484b-2dac-8854d1cb794e@gmail.com>

On 21/03/2019 22:01, Thomas Waldmann wrote:

> released a new alpha test version of borgbackup.

Any chance to see a "pull mode" one day?

I'm having a hard time understanding why this is not yet implemented in 
Borg (nor Restic for that matter).

 From an architecture stand point, having the backup host contacting the 
client hosts to backup their data is, by far, the strongest security 
model in all cases! There's nothing a client can do to compromise the 
backup server in any way, only data flows back to the backup server. 
With appropriate firewall rules, there's no way for any client to be 
able to contact the backup server. This is a very secure design.

Having the client push data to the backup server is always less secure 
and from tools like Borg/Restic which focus on security/integrity, I 
really can not understand why the "PUSH" model is the favored one and 
worse, the only available model from the start.

I really hope a PULL mode will be implemented soon, in a simple way 
where the borg backup host executes a borg "minion" through SSH on the 
client which does all the work and flows data back to be simply stored 
on the backup server, securely and out of reach to anyone with bad 
intentions.

-- 
Unix _IS_ user friendly, it's just selective about who its friends are.

From mszpak at wp.pl  Mon Mar 25 19:07:11 2019
From: mszpak at wp.pl (=?UTF-8?Q?Marcin_Zaj=c4=85czkowski?=)
Date: Tue, 26 Mar 2019 00:07:11 +0100
Subject: [Borgbackup] Reasons of remove --compression-from
In-Reply-To: <d035c36d-826d-91c7-0e47-e8eb6907f958@waldmann-edv.de>
References: <019c216b-2f56-7ffa-79ec-937e06b5224f@wp.pl>
 <5c03f39d-4586-b0ff-387a-e8e929ef1aa5@waldmann-edv.de>
 <24ae67b6-664c-b448-346a-4743b93962bd@wp.pl>
 <d035c36d-826d-91c7-0e47-e8eb6907f958@waldmann-edv.de>
Message-ID: <7cdb7294-2e88-dab7-9d97-35fafee995c7@wp.pl>

On 2019-03-25 14:07, Thomas Waldmann wrote:
>> In fact, auto is much easier to use, however, it is much less flexible.
>> Especially the ratio determining if file should be compressed seems to
>> be hardcoded in code to 0.97:
>> https://github.com/borgbackup/borg/blob/2b16fc9039660abba0ce6f5a25ae9c0f31ad48f5/src/borg/compress.pyx#L317
> 
> That could be:
> - adjusted
> - if no generally good value can be determined, it can be made an option
> 
>> It is not the best value in some situations. My test case - 760MB of JPG
>> photos. Two different empty repos on /tmp/ (in memory) with AES encryption.
>>
>> Without compression:
>>> time -p borg create --progress --compression none --list --filter=AME
>> --stats ...
>>> ...
>>> Duration: 15.18 seconds
>>> Number of files: 128
>>> Utilization of max. archive size: 0%
>>> ------------------------------------------------------------------------------
>>>                        Original size      Compressed size    Deduplicated size
>>> This archive:              761.66 MB            761.68 MB            761.68 MB
>>> All archives:              761.66 MB            761.68 MB            761.68 MB
>>>
>>>                        Unique chunks         Total chunks
>>> Chunk index:                     411                  411
>>> ------------------------------------------------------------------------------
>>> real 16.20
>>> user 10.43
>>> sys 0.89
> 
> On unmodified code, could you also run just lzma compression mode
> without auto, to see if it takes about 1:53 for your data, which would
> indicate the current threshold takes the wrong decision for your data?

That's the case:
> Duration: 1 minutes 46.45 seconds
(enforced lzma)

> After that, can you adjust the value in the borg src and find the
> highest one that gives close-to-uncompressed performance with auto,lzma?

In Fedora Borg is compiled with cpython
(compress.cpython-37m-x86_64-linux-gnu.so), so I can't just edit it in
the .py file. I have to handle how to rebuild it quickly from source and
just replace that file.

> Then, maybe verify doing the same with a bigger amount of photos.

Possibly with other file formats (such as music or video files).

> Not sure if --debug option gives some more clues, but just try it to see
> if it gives useful output for your case.
> 
>> 16 vs 115 seconds is noticeable. Especially that I have GBs of photos.
> 
> Lots of photos are a quite widespread use case, so we can adjust for that.

I wonder if compression is not widely used or just the speed is not a
top priority for most of the users and because that it remained
undetected for time being.

>> Why lzma? Over time, it is better for me to have smaller size of the
>> backup (to keep more snapshots) over backup duration (it can be done "in
>> background").
> 
> zstd might be a more modern option.

Hmm, thanks for a tip. zstd was much faster with a similar ratio for the
same files (both auto and always):

> Duration: 16.37 seconds
> Number of files: 128
> Utilization of max. archive size: 0%
> ------------------------------------------------------------------------------
>                        Original size      Compressed size    Deduplicated size
> This archive:              761.66 MB            757.42 MB            757.42 MB
> All archives:              761.66 MB            757.42 MB            757.42 MB
> 
>                        Unique chunks         Total chunks
> Chunk index:                     390                  390
> ------------------------------------------------------------------------------
> real 17.10
> user 11.64
> sys 0.92

I definitely want to give it a try.


>> I suspect the ratio is to high in my case. I would like to have an
>> ability to change it from a command line.
> 
> We can do that if no good value can be determined.
> We should avoid adding lots of commandline options though as it makes
> using borg harder and the documentation you need to read bigger.

I know, I ran through it recently :).
On the other hang, aware users might always want to tune that parameter
for their specific data sets.

>> However, even better would be
>> an ability to define, at least, a list of extensions that should be
>> ignored from compression (as a lighter version of the removed mechanism).
> 
> Similar for that plus the effort of having to maintain yet another list.

For me it would be enough to have an ability to specify those extensions
in the command line directly (as I run Borg from scripts) - to just have
a workaround, but an ability to refer to a file is more convenient. Borg
could provide a sample list (in the project resources), but (interested)
people could adjust it to their needs (e.g. by adding some strange
extensions to VM images).

Marcin

-- 
https://blog.solidsoft.info/ - Working code is not enough

From ngoonee.talk at gmail.com  Tue Mar 26 03:58:56 2019
From: ngoonee.talk at gmail.com (Oon-Ee Ng)
Date: Tue, 26 Mar 2019 15:58:56 +0800
Subject: [Borgbackup] borgbackup 1.2.0a5 alpha release
In-Reply-To: <aac7d81e-1346-484b-2dac-8854d1cb794e@gmail.com>
References: <8ed6ef55-ade6-8669-5040-bcad0aff5e30@waldmann-edv.de>
 <aac7d81e-1346-484b-2dac-8854d1cb794e@gmail.com>
Message-ID: <CAGQ70euPOaOpfOK1aM=i2jZjkwqOBNUG0J5v_yXfURQNFHXn0A@mail.gmail.com>

On Tue, Mar 26, 2019 at 1:51 AM Melkor Lord <melkor.lord at gmail.com> wrote:

> I'm having a hard time understanding why this is not yet implemented in
> Borg (nor Restic for that matter).
>
>  From an architecture stand point, having the backup host contacting the
> client hosts to backup their data is, by far, the strongest security
> model in all cases! There's nothing a client can do to compromise the
> backup server in any way, only data flows back to the backup server.
> With appropriate firewall rules, there's no way for any client to be
> able to contact the backup server. This is a very secure design.
>

The assumption that the client is the risk and not the server is not shared
by all. In particular when the server is off-site or on a third-party
server, it makes sense that it be treated as untrusted (this is what borg
assumes based on the implementation and documentation).

 Besides, a 'sort-of' pull mode (with a capable server) can already be done
by remote mounting client directories on the backup server and then running
borgbackup from that.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20190326/2335f6fb/attachment.html>

From matej.kovacic at telefoncek.si  Tue Mar 26 04:57:50 2019
From: matej.kovacic at telefoncek.si (=?UTF-8?B?TWF0ZWogS292YcSNacSN?=)
Date: Tue, 26 Mar 2019 09:57:50 +0100
Subject: [Borgbackup] GUI?
Message-ID: <b32cac96-2147-f3fa-92c1-d693d0cce21f@telefoncek.si>

Hi,

is there any development for BorgBackup GUI?

I have found several project (two projects developing web interface,
http://www.borgbackupserver.com/, some mentions of Qt GUI
(https://github.com/borgbackup/borg/issues/2960),
https://bitbucket.org/tuomov/borgend, https://demo.burp-ui.org/ ... is
anything that is already "production ready"?


Regards,

Matej

-- 
PGP Fingerprint: CAB3 88B5 69F0 226C 7A5A  8C16 535C 4A5A 666F 1CCE
PGP Key: https://keyserver.ubuntu.com/pks/lookup?search=0x535C4A5A666F1CCE&fingerprint=on&op=vindex
Personal blog: https://telefoncek.si



From tkpapp at gmail.com  Tue Mar 26 05:21:29 2019
From: tkpapp at gmail.com (Tamas Papp)
Date: Tue, 26 Mar 2019 10:21:29 +0100
Subject: [Borgbackup] GUI?
In-Reply-To: <b32cac96-2147-f3fa-92c1-d693d0cce21f@telefoncek.si>
References: <b32cac96-2147-f3fa-92c1-d693d0cce21f@telefoncek.si>
Message-ID: <875zs6f0ti.fsf@gmail.com>

I wonder what the use case is --- most robust debugging setups I have
seen are fully automated (may check for being on a fast/home/company
network, and just warn the user if a backup did not happen for a while,
usually by e-mail).

Involving a GUI is usually just annoying ("You have not backed up for a
while --- do you want to do it now? Since you said no for the last 72
days this may take some time.") or fragile ("No, do it later".)

Best,

Tamas

On Tue, Mar 26 2019, Matej Kova?i? wrote:

> Hi,
>
> is there any development for BorgBackup GUI?
>
> I have found several project (two projects developing web interface,
> http://www.borgbackupserver.com/, some mentions of Qt GUI
> (https://github.com/borgbackup/borg/issues/2960),
> https://bitbucket.org/tuomov/borgend, https://demo.burp-ui.org/ ... is
> anything that is already "production ready"?
>
>
> Regards,
>
> Matej

From tw at waldmann-edv.de  Tue Mar 26 05:34:22 2019
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Tue, 26 Mar 2019 10:34:22 +0100
Subject: [Borgbackup] GUI?
In-Reply-To: <b32cac96-2147-f3fa-92c1-d693d0cce21f@telefoncek.si>
References: <b32cac96-2147-f3fa-92c1-d693d0cce21f@telefoncek.si>
Message-ID: <f982d9f6-ee36-9915-ce46-002bb16c6de4@waldmann-edv.de>

> is there any development for BorgBackup GUI?

Try vorta.

It's linked from there btw.:

https://github.com/borgbackup/community

-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393

From tw at waldmann-edv.de  Tue Mar 26 05:36:23 2019
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Tue, 26 Mar 2019 10:36:23 +0100
Subject: [Borgbackup] borg pull mode (was: Re: borgbackup 1.2.0a5 alpha
 release)
In-Reply-To: <CAGQ70euPOaOpfOK1aM=i2jZjkwqOBNUG0J5v_yXfURQNFHXn0A@mail.gmail.com>
References: <8ed6ef55-ade6-8669-5040-bcad0aff5e30@waldmann-edv.de>
 <aac7d81e-1346-484b-2dac-8854d1cb794e@gmail.com>
 <CAGQ70euPOaOpfOK1aM=i2jZjkwqOBNUG0J5v_yXfURQNFHXn0A@mail.gmail.com>
Message-ID: <644dcb66-5017-67b1-4c3c-9b5c11861256@waldmann-edv.de>

On 3/26/19 8:58 AM, Oon-Ee Ng wrote:
> On Tue, Mar 26, 2019 at 1:51 AM Melkor Lord <melkor.lord at gmail.com
> <mailto:melkor.lord at gmail.com>> wrote:
> 
>     I'm having a hard time understanding why this is not yet implemented in
>     Borg (nor Restic for that matter).

This has already been discussed previously, check issue tracker and ML.

-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393

From melkor.lord at gmail.com  Tue Mar 26 11:36:26 2019
From: melkor.lord at gmail.com (Melkor Lord)
Date: Tue, 26 Mar 2019 16:36:26 +0100
Subject: [Borgbackup] borgbackup 1.2.0a5 alpha release
In-Reply-To: <CAGQ70euPOaOpfOK1aM=i2jZjkwqOBNUG0J5v_yXfURQNFHXn0A@mail.gmail.com>
References: <8ed6ef55-ade6-8669-5040-bcad0aff5e30@waldmann-edv.de>
 <aac7d81e-1346-484b-2dac-8854d1cb794e@gmail.com>
 <CAGQ70euPOaOpfOK1aM=i2jZjkwqOBNUG0J5v_yXfURQNFHXn0A@mail.gmail.com>
Message-ID: <c032b6c0-e667-0c2b-80a8-9a445ce623d4@gmail.com>

On 26/03/2019 08:58, Oon-Ee Ng wrote:

> The assumption that the client is the risk and not the server is not shared
> by all. In particular when the server is off-site or on a third-party
> server, it makes sense that it be treated as untrusted (this is what borg
> assumes based on the implementation and documentation).

If the client is the risk, then having the server contacting the client 
is secure by design. There's nothing the client can do to the server to 
compromise it, especially if there's no entry point to the server (ssh 
or other with appropriate firewall rules)

If the server is treated as untrusted, I wouldn't put my backup data 
there in the first place :-)

>   Besides, a 'sort-of' pull mode (with a capable server) can already be done
> by remote mounting client directories on the backup server and then running
> borgbackup from that.

This is cumbersome and fragile at best! It works for the most basic 
setups but you can forget it for more serious scenarios. For example 
when using FS ACLs, there's no remote mounting solution supporting that 
properly.

I guess I'm stuck with no proper backup solution for a big server farm 
then, at least not using Borg nor Restic for the time being.

Having all the servers (clients) banging/hammering the backup server at 
the same time it definitely not a nice solution. Trying to add some 
delay to the cron jobs on the clients to avoid that is an inelegant 
kludge in my opinion. Instead, having the backup server decide which 
servers to contact and efficiently use the I/O and network bandwidth is 
way better.

Is there anyone here using Borg in a datacenter infrastructure? With a 
good deal of servers pushing their data to a "central" backup server? I 
wonder how you manage it efficiently.

I've read here some stories about huge repos but no story yet about lots 
of clients pushing to a backup server. I'd like some feedback on that if 
you're willing to share your experience.

-- 
Unix _IS_ user friendly, it's just selective about who its friends are.

From dave at gasaway.org  Tue Mar 26 11:51:11 2019
From: dave at gasaway.org (David Gasaway)
Date: Tue, 26 Mar 2019 08:51:11 -0700
Subject: [Borgbackup] borgbackup 1.2.0a5 alpha release
In-Reply-To: <aac7d81e-1346-484b-2dac-8854d1cb794e@gmail.com>
References: <8ed6ef55-ade6-8669-5040-bcad0aff5e30@waldmann-edv.de>
 <aac7d81e-1346-484b-2dac-8854d1cb794e@gmail.com>
Message-ID: <CAGsJ5AWfBZt+fEjDrByi_K_J4iiYr+Qfy-HspHhJTT6TvfM0fQ@mail.gmail.com>

On Mon, Mar 25, 2019 at 10:52 AM Melkor Lord <melkor.lord at gmail.com> wrote:

> On 21/03/2019 22:01, Thomas Waldmann wrote:
>
>  From an architecture stand point, having the backup host contacting the
> client hosts to backup their data is, by far, the strongest security
> model in all cases! There's nothing a client can do to compromise the
> backup server in any way, only data flows back to the backup server.
> With appropriate firewall rules, there's no way for any client to be
> able to contact the backup server. This is a very secure design.
>

On the other hand, the server would require remote admin access on all the
client systems.  In a push configuration, no remote admin access is
necessary.

-- 
-:-:- David K. Gasaway
-:-:- Email: dave at gasaway.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20190326/3795e1e6/attachment.html>

From melkor.lord at gmail.com  Tue Mar 26 11:52:35 2019
From: melkor.lord at gmail.com (Melkor Lord)
Date: Tue, 26 Mar 2019 16:52:35 +0100
Subject: [Borgbackup] borg pull mode
In-Reply-To: <644dcb66-5017-67b1-4c3c-9b5c11861256@waldmann-edv.de>
References: <8ed6ef55-ade6-8669-5040-bcad0aff5e30@waldmann-edv.de>
 <aac7d81e-1346-484b-2dac-8854d1cb794e@gmail.com>
 <CAGQ70euPOaOpfOK1aM=i2jZjkwqOBNUG0J5v_yXfURQNFHXn0A@mail.gmail.com>
 <644dcb66-5017-67b1-4c3c-9b5c11861256@waldmann-edv.de>
Message-ID: <4b56d6cf-115f-326f-7933-bdd3e52a9ff0@gmail.com>

On 26/03/2019 10:36, Thomas Waldmann wrote:

>>      I'm having a hard time understanding why this is not yet implemented in
>>      Borg (nor Restic for that matter).
> 
> This has already been discussed previously, check issue tracker and ML.

I've checked! It dates back to 2016 so I guess nobody in Borg's dev team 
is willing to work on that which is sad.

I was kinda hoping to raise a bit of interest again.

For my PoV, it does not seem hard to implement as quite everything is 
already there in Borg. If I'm not mistaken, it would consist of a new 
command for Borg ("pipe"?) that would listen to orders given by the 
master caller and just flow data back through the pipe to be stored.

- HostA >> (ssh) >> HostB (borg pipe)
   1/ Backup /this /that /etc.
   2/ Data is sent back to HostA through the pipe
   3/ HostA stores the backup data to the repo

A very simple language would be used to communicate back and forth the 
orders and their result.

Some checksum of the blocs on HostB is sent back to HostA to see if the 
same bloc already exists, so do not transmit, or is new/unknown, so flow 
it back to be stored. This is bandwidth efficient and does not require 
huge amounts of new development unless I'm sadly mistaken.

With the same idea, the repo keys are stored only on the backup server, 
transmitted only through the pipe when needed to do the actual backup. 
As already mentioned, the client does not know the repo key and can not 
contact the backup server in any way.

I really hope this gains a bit of interest and lands on the roadmap :-)

-- 
Unix _IS_ user friendly, it's just selective about who its friends are.

From fabio.pedretti at unibs.it  Tue Mar 26 12:05:15 2019
From: fabio.pedretti at unibs.it (Fabio Pedretti)
Date: Tue, 26 Mar 2019 17:05:15 +0100
Subject: [Borgbackup] borgbackup 1.2.0a5 alpha release
In-Reply-To: <c032b6c0-e667-0c2b-80a8-9a445ce623d4@gmail.com>
References: <8ed6ef55-ade6-8669-5040-bcad0aff5e30@waldmann-edv.de>
 <aac7d81e-1346-484b-2dac-8854d1cb794e@gmail.com>
 <CAGQ70euPOaOpfOK1aM=i2jZjkwqOBNUG0J5v_yXfURQNFHXn0A@mail.gmail.com>
 <c032b6c0-e667-0c2b-80a8-9a445ce623d4@gmail.com>
Message-ID: <CAGpfsQ1xusg67mTW4AZbdaWq35G-WcR0pk8a-ABHuxju+Pwv8Q@mail.gmail.com>

Il giorno mar 26 mar 2019 alle ore 16:38 Melkor Lord <melkor.lord at gmail.com>
ha scritto:

> Is there anyone here using Borg in a datacenter infrastructure? With a
> good deal of servers pushing their data to a "central" backup server? I
> wonder how you manage it efficiently.
>
> I've read here some stories about huge repos but no story yet about lots
> of clients pushing to a backup server. I'd like some feedback on that if
> you're willing to share your experience.
>

I am using a central borg server  backupping 20 remote clients (which are
server themself).
The borg server mount with NFS automount the needed directory of the
remotes (exported with read only mode).
Then it start a sequential backup of the 20 servers, using a single
repository. This way deduplication is improved, also I found it a bit more
simpler and cleaner.
Just make sure to set the BORG_FILES_CACHE_TTL variable to a number bigger
than the number of servers, to avoid emptying the cache (I set it to 4x the
servers number).
Using that since some years, working beautifully.
1.2 should also hopefully improve this scenario, since I could do a manual
compaction just 1 single time after the 20 daily backups, rather than being
forced for all the 20 servers. Probably saving around 3minutes * 20 servers
~ 1 hour every day (just a guess).

-- 
ing. Fabio Pedretti
Responsabile U.O.C. "Reti, Sistemi e Sicurezza Informatica"
https://www.unibs.it/node/4305
Universit? degli Studi di Brescia
Via Valotti, 9 - 25121 Brescia
E-mail: fabio.pedretti at unibs.it

-- 


Informativa sulla Privacy: http://www.unibs.it/node/8155 
<http://www.unibs.it/node/8155>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20190326/ea0f9498/attachment.html>

From melkor.lord at gmail.com  Tue Mar 26 12:08:52 2019
From: melkor.lord at gmail.com (Melkor Lord)
Date: Tue, 26 Mar 2019 17:08:52 +0100
Subject: [Borgbackup] borgbackup 1.2.0a5 alpha release
In-Reply-To: <CAGsJ5AWfBZt+fEjDrByi_K_J4iiYr+Qfy-HspHhJTT6TvfM0fQ@mail.gmail.com>
References: <8ed6ef55-ade6-8669-5040-bcad0aff5e30@waldmann-edv.de>
 <aac7d81e-1346-484b-2dac-8854d1cb794e@gmail.com>
 <CAGsJ5AWfBZt+fEjDrByi_K_J4iiYr+Qfy-HspHhJTT6TvfM0fQ@mail.gmail.com>
Message-ID: <55ca74a9-c5e8-172c-d882-18ba51474f16@gmail.com>

On 26/03/2019 16:51, David Gasaway wrote:

> On the other hand, the server would require remote admin access on all the
> client systems.  In a push configuration, no remote admin access is
> necessary.

All the "professional/enterprise" backup solutions I've dealt with from 
the '90s to now use the same model : A backup server + agents deployed 
on clients (machines to backup).

I've used Arkeia, ArcServe, BackupExec and some others. I've quoted 
"professional/enterprise" because despite using a good model, they all 
have drawbacks, pitfalls and bad design issues!

The backup server was a physical console with no way to be contacted 
from the outside world. The local network or internet was enough to 
contact the agents on the client machines.

Nowadays, I don't have a physical console anymore but I still use a 
central backup server where the security is awfully tight. The model 
hasn't changed because it's battle tested. This is way easier to control 
than having it wide open so any random client can happily contact it 
like on the "push data" model.

In our "put everything on the cloud" time, new products/software are 
written to focus on that model exclusively which is a bit annoying I 
think. "The cloud is someone else's computer" which I don't want to use, 
especially on critical data for some (rightfully) picky clients.

-- 
Unix _IS_ user friendly, it's just selective about who its friends are.

From dave at gasaway.org  Tue Mar 26 12:29:29 2019
From: dave at gasaway.org (David Gasaway)
Date: Tue, 26 Mar 2019 09:29:29 -0700
Subject: [Borgbackup] borgbackup 1.2.0a5 alpha release
In-Reply-To: <55ca74a9-c5e8-172c-d882-18ba51474f16@gmail.com>
References: <8ed6ef55-ade6-8669-5040-bcad0aff5e30@waldmann-edv.de>
 <aac7d81e-1346-484b-2dac-8854d1cb794e@gmail.com>
 <CAGsJ5AWfBZt+fEjDrByi_K_J4iiYr+Qfy-HspHhJTT6TvfM0fQ@mail.gmail.com>
 <55ca74a9-c5e8-172c-d882-18ba51474f16@gmail.com>
Message-ID: <CAGsJ5AVg8eguCnbBHAa_ABHzVb68iAVUH9NuOKvwxMaRw-L44g@mail.gmail.com>

On Tue, Mar 26, 2019 at 9:09 AM Melkor Lord <melkor.lord at gmail.com> wrote:

>
> All the "professional/enterprise" backup solutions I've dealt with from
> the '90s to now use the same model : A backup server + agents deployed
> on clients (machines to backup).
>

I was simply pointing out that there are security implications to either
model.  I'm no expert, so I'm not going to debate which is better.

-- 
-:-:- David K. Gasaway
-:-:- Email: dave at gasaway.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20190326/c6ab7ddc/attachment.html>

From melkor.lord at gmail.com  Tue Mar 26 12:57:47 2019
From: melkor.lord at gmail.com (Melkor Lord)
Date: Tue, 26 Mar 2019 17:57:47 +0100
Subject: [Borgbackup] borgbackup 1.2.0a5 alpha release
In-Reply-To: <CAGpfsQ1xusg67mTW4AZbdaWq35G-WcR0pk8a-ABHuxju+Pwv8Q@mail.gmail.com>
References: <8ed6ef55-ade6-8669-5040-bcad0aff5e30@waldmann-edv.de>
 <aac7d81e-1346-484b-2dac-8854d1cb794e@gmail.com>
 <CAGQ70euPOaOpfOK1aM=i2jZjkwqOBNUG0J5v_yXfURQNFHXn0A@mail.gmail.com>
 <c032b6c0-e667-0c2b-80a8-9a445ce623d4@gmail.com>
 <CAGpfsQ1xusg67mTW4AZbdaWq35G-WcR0pk8a-ABHuxju+Pwv8Q@mail.gmail.com>
Message-ID: <50a290d6-26d9-f153-ecc7-908c5fbde412@gmail.com>

On 26/03/2019 17:05, Fabio Pedretti wrote:

> I am using a central borg server  backupping 20 remote clients (which are
> server themself).
> The borg server mount with NFS automount the needed directory of the
> remotes (exported with read only mode).
> Then it start a sequential backup of the 20 servers, using a single
> repository. This way deduplication is improved, also I found it a bit more
> simpler and cleaner.

If I'm reading you correctly, you bended a bit the system to essentially 
do as I want since the beginning :-)

Your backup server controls everything and gives orders to the clients 
so you end up with a nice sequential backup instead of having 20 clients 
hammering your backup server at the same time.

But I want to hear about people using the default Borg model : A backup 
repo hammered by lots of clients, unaware of each other, at the same time.

> Using that since some years, working beautifully.

Yep, the good model : The backup server controls everything.

-- 
Unix _IS_ user friendly, it's just selective about who its friends are.

From melkor.lord at gmail.com  Tue Mar 26 13:03:36 2019
From: melkor.lord at gmail.com (Melkor Lord)
Date: Tue, 26 Mar 2019 18:03:36 +0100
Subject: [Borgbackup] borgbackup 1.2.0a5 alpha release
In-Reply-To: <CAGsJ5AVg8eguCnbBHAa_ABHzVb68iAVUH9NuOKvwxMaRw-L44g@mail.gmail.com>
References: <8ed6ef55-ade6-8669-5040-bcad0aff5e30@waldmann-edv.de>
 <aac7d81e-1346-484b-2dac-8854d1cb794e@gmail.com>
 <CAGsJ5AWfBZt+fEjDrByi_K_J4iiYr+Qfy-HspHhJTT6TvfM0fQ@mail.gmail.com>
 <55ca74a9-c5e8-172c-d882-18ba51474f16@gmail.com>
 <CAGsJ5AVg8eguCnbBHAa_ABHzVb68iAVUH9NuOKvwxMaRw-L44g@mail.gmail.com>
Message-ID: <6ec892bc-3626-8550-438a-c7affe626aac@gmail.com>

On 26/03/2019 17:29, David Gasaway wrote:

> I was simply pointing out that there are security implications to either
> model.  I'm no expert, so I'm not going to debate which is better.

I don't want to debate on which is better either :-) Each person chooses 
whatever is best for their use, I don't have a problem with that.

I would like _both_ models to exist within Borg/Restic. What bugs me is 
that only the "push" model is favored and, even though the "pull" model 
was mentioned as far back as 2016, it seems there's no will to even try 
implementing it. I find it sad :-(

-- 
Unix _IS_ user friendly, it's just selective about who its friends are.

From kbass at kenbass.com  Tue Mar 26 13:34:58 2019
From: kbass at kenbass.com (Ken Bass)
Date: Tue, 26 Mar 2019 13:34:58 -0400
Subject: [Borgbackup] borgbackup 1.2.0a5 alpha release
In-Reply-To: <c032b6c0-e667-0c2b-80a8-9a445ce623d4@gmail.com>
References: <8ed6ef55-ade6-8669-5040-bcad0aff5e30@waldmann-edv.de>
 <aac7d81e-1346-484b-2dac-8854d1cb794e@gmail.com>
 <CAGQ70euPOaOpfOK1aM=i2jZjkwqOBNUG0J5v_yXfURQNFHXn0A@mail.gmail.com>
 <c032b6c0-e667-0c2b-80a8-9a445ce623d4@gmail.com>
Message-ID: <ee7cfd1a-1d02-5c5b-18d8-4b3186d968ba@kenbass.com>

On 3/26/2019 11:36 AM, Melkor Lord wrote:
> On 26/03/2019 08:58, Oon-Ee Ng wrote:
>
>> The assumption that the client is the risk and not the server is not 
>> shared
>> by all. In particular when the server is off-site or on a third-party
>> server, it makes sense that it be treated as untrusted (this is what 
>> borg
>> assumes based on the implementation and documentation).
>
> If the client is the risk, then having the server contacting the 
> client is secure by design. There's nothing the client can do to the 
> server to compromise it, especially if there's no entry point to the 
> server (ssh or other with appropriate firewall rules)
>
> If the server is treated as untrusted, I wouldn't put my backup data 
> there in the first place :-)

The thinking I have heard mentioned is that since you can (optionally) 
encrypt the repo, you do not need to trust the server.

>
>> ? Besides, a 'sort-of' pull mode (with a capable server) can already 
>> be done
>> by remote mounting client directories on the backup server and then 
>> running
>> borgbackup from that.
>
> This is cumbersome and fragile at best! It works for the most basic 
> setups but you can forget it for more serious scenarios. For example 
> when using FS ACLs, there's no remote mounting solution supporting 
> that properly.

Personally, I think mounting client directories is not only slow, 
options open new attack vectors, but isn't a good solution either. It 
can work, but it is a kludge.

> Is there anyone here using Borg in a datacenter infrastructure? With a 
> good deal of servers pushing their data to a "central" backup server? 
> I wonder how you manage it efficiently.

This is a usability problem for me too. I have been using 'safekeep' 
which is a pull model wrapper around rdiff-backup and then use borg to 
backup that backup. Not at all a great solution but it was the only way 
I felt I could coordinate that only a single backup was running at a 
time / limit bandwidth, etc.


From tve at voneicken.com  Tue Mar 26 15:29:38 2019
From: tve at voneicken.com (Thorsten von Eicken)
Date: Tue, 26 Mar 2019 19:29:38 +0000
Subject: [Borgbackup] borgbackup 1.2.0a5 alpha release
In-Reply-To: <ee7cfd1a-1d02-5c5b-18d8-4b3186d968ba@kenbass.com>
References: <8ed6ef55-ade6-8669-5040-bcad0aff5e30@waldmann-edv.de>
 <aac7d81e-1346-484b-2dac-8854d1cb794e@gmail.com>
 <CAGQ70euPOaOpfOK1aM=i2jZjkwqOBNUG0J5v_yXfURQNFHXn0A@mail.gmail.com>
 <c032b6c0-e667-0c2b-80a8-9a445ce623d4@gmail.com>
 <ee7cfd1a-1d02-5c5b-18d8-4b3186d968ba@kenbass.com>
Message-ID: <01000169bb7ac1da-c7d4284c-3e0c-463a-b82a-1572d8666949-000000@email.amazonses.com>

Let me try to understand... Are the following statements correct WRT a 
pull model?

- you want a central schedule on the server, specifically, have the 
server decide when each client should start its backup

- you want the server to hold the credentials to log into the clients, 
not the other way around

- you're OK with installing some form of backup software on each client

If the above is all true, then this should be pretty easy to implement 
(famous last words). Borg already supports a client/server model with 
`borg serve`, the catch is that currently it's one-way: the client 
connects to the server using ssh. But what if you did the following:

- on the server, ssh to each client in turn and run `borg create` on the 
client that way, while you start a `borg serve` on the local end of the 
ssh connection/tunnel for the client to use

- instead of using a remote repository in `borg create`, tell it to talk 
to `borg serve` on the already-open ssh connection, or alternatively, on 
an ssh tunnel connection opened at the same time

- for prototyping purposes, use the `--rsh` option to `borg create` to 
tell it to use the tunnel, there must be some combo of 
cat/nc/netcat/socat that should be able to do the trick.

By using the appropriate `--restrict` option to `borg serve` you can 
limit which repositories each client has access to. What borg doesn't 
support is fine-grained control over a shared repository (perhaps adding 
an option to `borg serve` to force an archive name prefix could help), 
but there is the `--append-only` flag.

Thoughts?

Thorsten

On 3/26/19 10:34 AM, Ken Bass wrote:
> On 3/26/2019 11:36 AM, Melkor Lord wrote:
>> On 26/03/2019 08:58, Oon-Ee Ng wrote:
>>
>>> The assumption that the client is the risk and not the server is not 
>>> shared
>>> by all. In particular when the server is off-site or on a third-party
>>> server, it makes sense that it be treated as untrusted (this is what 
>>> borg
>>> assumes based on the implementation and documentation).
>>
>> If the client is the risk, then having the server contacting the 
>> client is secure by design. There's nothing the client can do to the 
>> server to compromise it, especially if there's no entry point to the 
>> server (ssh or other with appropriate firewall rules)
>>
>> If the server is treated as untrusted, I wouldn't put my backup data 
>> there in the first place :-)
>
> The thinking I have heard mentioned is that since you can (optionally) 
> encrypt the repo, you do not need to trust the server.
>
>>
>>> ? Besides, a 'sort-of' pull mode (with a capable server) can already 
>>> be done
>>> by remote mounting client directories on the backup server and then 
>>> running
>>> borgbackup from that.
>>
>> This is cumbersome and fragile at best! It works for the most basic 
>> setups but you can forget it for more serious scenarios. For example 
>> when using FS ACLs, there's no remote mounting solution supporting 
>> that properly.
>
> Personally, I think mounting client directories is not only slow, 
> options open new attack vectors, but isn't a good solution either. It 
> can work, but it is a kludge.
>
>> Is there anyone here using Borg in a datacenter infrastructure? With 
>> a good deal of servers pushing their data to a "central" backup 
>> server? I wonder how you manage it efficiently.
>
> This is a usability problem for me too. I have been using 'safekeep' 
> which is a pull model wrapper around rdiff-backup and then use borg to 
> backup that backup. Not at all a great solution but it was the only 
> way I felt I could coordinate that only a single backup was running at 
> a time / limit bandwidth, etc.
>
> _______________________________________________
> Borgbackup mailing list
> Borgbackup at python.org
> https://mail.python.org/mailman/listinfo/borgbackup
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20190326/cf03b92f/attachment.html>

From kbass at kenbass.com  Tue Mar 26 16:08:21 2019
From: kbass at kenbass.com (Ken Bass)
Date: Tue, 26 Mar 2019 16:08:21 -0400
Subject: [Borgbackup] borgbackup 1.2.0a5 alpha release
In-Reply-To: <01000169bb7ac1da-c7d4284c-3e0c-463a-b82a-1572d8666949-000000@email.amazonses.com>
References: <8ed6ef55-ade6-8669-5040-bcad0aff5e30@waldmann-edv.de>
 <aac7d81e-1346-484b-2dac-8854d1cb794e@gmail.com>
 <CAGQ70euPOaOpfOK1aM=i2jZjkwqOBNUG0J5v_yXfURQNFHXn0A@mail.gmail.com>
 <c032b6c0-e667-0c2b-80a8-9a445ce623d4@gmail.com>
 <ee7cfd1a-1d02-5c5b-18d8-4b3186d968ba@kenbass.com>
 <01000169bb7ac1da-c7d4284c-3e0c-463a-b82a-1572d8666949-000000@email.amazonses.com>
Message-ID: <37d578bc-4368-d8e7-fe49-35e85593c710@kenbass.com>

On 3/26/2019 3:29 PM, Thorsten von Eicken wrote:
>
> Let me try to understand... Are the following statements correct WRT a 
> pull model?
>
> - you want a central schedule on the server, specifically, have the 
> server decide when each client should start its backup
>
> - you want the server to hold the credentials to log into the clients, 
> not the other way around
>
> - you're OK with installing some form of backup software on each client
>
> If the above is all true, then this should be pretty easy to implement 
> (famous last words). Borg already supports a client/server model with 
> `borg serve`, the catch is that currently it's one-way: the client 
> connects to the server using ssh. But what if you did the following:
>
The above is included in my desired usage, but there is one other item - 
deduplication. I don't know/understand how the above would work wrt the 
cache files that borg maintains so that it knows whether a file is 
identical or not. From this perspective, I think the pull model might 
not be so straightforward. Something in the protocol might need to 
communicate the cache checksum info across the client / server model. I 
had considered just not using borg at all and using a deduplicating 
filesystem but I wanted to keep things simple with ext4, not needing 
huge amounts of RAM on the backup-server to support something like ZFS. 
Centos/RHEL 7.5 introduces a VDO (Virtual Data Optimizer) which supports 
dedup (and compression) - it too has heavier memory requirements, but 
not as severe as ZFS.



From tve at voneicken.com  Tue Mar 26 21:51:27 2019
From: tve at voneicken.com (Thorsten von Eicken)
Date: Wed, 27 Mar 2019 01:51:27 +0000
Subject: [Borgbackup] borgbackup 1.2.0a5 alpha release
In-Reply-To: <37d578bc-4368-d8e7-fe49-35e85593c710@kenbass.com>
References: <8ed6ef55-ade6-8669-5040-bcad0aff5e30@waldmann-edv.de>
 <aac7d81e-1346-484b-2dac-8854d1cb794e@gmail.com>
 <CAGQ70euPOaOpfOK1aM=i2jZjkwqOBNUG0J5v_yXfURQNFHXn0A@mail.gmail.com>
 <c032b6c0-e667-0c2b-80a8-9a445ce623d4@gmail.com>
 <ee7cfd1a-1d02-5c5b-18d8-4b3186d968ba@kenbass.com>
 <01000169bb7ac1da-c7d4284c-3e0c-463a-b82a-1572d8666949-000000@email.amazonses.com>
 <37d578bc-4368-d8e7-fe49-35e85593c710@kenbass.com>
Message-ID: <01000169bcd8543a-bbe8142b-1abb-4000-8b88-bbca5cf09f3d-000000@email.amazonses.com>


On 3/26/19 1:08 PM, Ken Bass wrote:
> The above is included in my desired usage, but there is one other item 
> - deduplication. I don't know/understand how the above would work wrt 
> the cache files that borg maintains so that it knows whether a file is 
> identical or not. From this perspective, I think the pull model might 
> not be so straightforward.

Yup. I believe that what I'm describing is relatively easy to prototype 
and to add more cleanly into borg if successful. If you are using a 
single repo for all clients then there will be cache sync overhead on 
each client. If the cache sync is a show-stopper, then fixing that would 
be a significant change to borg as far as I can tell. At that point 
might as well use the sshfs mount as described in the `borg create` doc 
examples.

Thorsten

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20190327/b74f0f6b/attachment-0001.html>

From tr.ml at gmx.de  Wed Mar 27 05:03:09 2019
From: tr.ml at gmx.de (Rainer Traut)
Date: Wed, 27 Mar 2019 10:03:09 +0100
Subject: [Borgbackup] borgbackup 1.2.0a5 alpha release
In-Reply-To: <CAGpfsQ1xusg67mTW4AZbdaWq35G-WcR0pk8a-ABHuxju+Pwv8Q@mail.gmail.com>
References: <8ed6ef55-ade6-8669-5040-bcad0aff5e30@waldmann-edv.de>
 <aac7d81e-1346-484b-2dac-8854d1cb794e@gmail.com>
 <CAGQ70euPOaOpfOK1aM=i2jZjkwqOBNUG0J5v_yXfURQNFHXn0A@mail.gmail.com>
 <c032b6c0-e667-0c2b-80a8-9a445ce623d4@gmail.com>
 <CAGpfsQ1xusg67mTW4AZbdaWq35G-WcR0pk8a-ABHuxju+Pwv8Q@mail.gmail.com>
Message-ID: <6a492ae3-cd63-89d3-ebcd-5b171eac9ecd@gmx.de>

Hi list,

Am 26.03.19 um 17:05 schrieb Fabio Pedretti:
> Il giorno mar 26 mar 2019 alle ore 16:38 Melkor Lord
> <melkor.lord at gmail.com <mailto:melkor.lord at gmail.com>> ha scritto:
>
>     Is there anyone here using Borg in a datacenter infrastructure? With a
>     good deal of servers pushing their data to a "central" backup server? I
>     wonder how you manage it efficiently.
>
>     I've read here some stories about huge repos but no story yet about
>     lots
>     of clients pushing to a backup server. I'd like some feedback on
>     that if
>     you're willing to share your experience.
>
>
> I am using a central borg server? backupping 20 remote clients (which
> are server themself).
> The borg server mount with NFS automount the needed directory of the
> remotes (exported with read only mode).
> Then it start a sequential backup of the 20 servers, using a single
> repository. This way deduplication is improved, also I found it a bit
> more simpler and cleaner.
> Just make sure to set the BORG_FILES_CACHE_TTL variable to a number
> bigger than the number of servers, to avoid emptying the cache (I set it
> to 4x the servers number).
> Using that since some years, working beautifully.
> 1.2 should also hopefully improve this scenario, since I could do a
> manual compaction just 1 single time after the 20 daily backups, rather
> than being forced for all the 20 servers. Probably saving around
> 3minutes * 20 servers ~ 1 hour every day (just a guess).

Yes, I'm doing the same with sshfs and the borg opts:

borg create --files-cache=ctime,size

so that caching works.

Can I omit that with NFS?

VG
Rainer



From fabio.pedretti at unibs.it  Wed Mar 27 11:23:34 2019
From: fabio.pedretti at unibs.it (Fabio Pedretti)
Date: Wed, 27 Mar 2019 16:23:34 +0100
Subject: [Borgbackup] borgbackup 1.2.0a5 alpha release
In-Reply-To: <6a492ae3-cd63-89d3-ebcd-5b171eac9ecd@gmx.de>
References: <8ed6ef55-ade6-8669-5040-bcad0aff5e30@waldmann-edv.de>
 <aac7d81e-1346-484b-2dac-8854d1cb794e@gmail.com>
 <CAGQ70euPOaOpfOK1aM=i2jZjkwqOBNUG0J5v_yXfURQNFHXn0A@mail.gmail.com>
 <c032b6c0-e667-0c2b-80a8-9a445ce623d4@gmail.com>
 <CAGpfsQ1xusg67mTW4AZbdaWq35G-WcR0pk8a-ABHuxju+Pwv8Q@mail.gmail.com>
 <6a492ae3-cd63-89d3-ebcd-5b171eac9ecd@gmx.de>
Message-ID: <CAGpfsQ0e5VvVYyGR1Zu3vtOVB594jy1DOArNM3wvU0yrB_RTGw@mail.gmail.com>

Il giorno mer 27 mar 2019 alle ore 10:03 Rainer Traut <tr.ml at gmx.de> ha
scritto:

> Hi list,
>
> Am 26.03.19 um 17:05 schrieb Fabio Pedretti:
> > Il giorno mar 26 mar 2019 alle ore 16:38 Melkor Lord
> > <melkor.lord at gmail.com <mailto:melkor.lord at gmail.com>> ha scritto:
> >
> >     Is there anyone here using Borg in a datacenter infrastructure? With
> a
> >     good deal of servers pushing their data to a "central" backup
> server? I
> >     wonder how you manage it efficiently.
> >
> >     I've read here some stories about huge repos but no story yet about
> >     lots
> >     of clients pushing to a backup server. I'd like some feedback on
> >     that if
> >     you're willing to share your experience.
> >
> >
> > I am using a central borg server  backupping 20 remote clients (which
> > are server themself).
> > The borg server mount with NFS automount the needed directory of the
> > remotes (exported with read only mode).
> > Then it start a sequential backup of the 20 servers, using a single
> > repository. This way deduplication is improved, also I found it a bit
> > more simpler and cleaner.
> > Just make sure to set the BORG_FILES_CACHE_TTL variable to a number
> > bigger than the number of servers, to avoid emptying the cache (I set it
> > to 4x the servers number).
> > Using that since some years, working beautifully.
> > 1.2 should also hopefully improve this scenario, since I could do a
> > manual compaction just 1 single time after the 20 daily backups, rather
> > than being forced for all the 20 servers. Probably saving around
> > 3minutes * 20 servers ~ 1 hour every day (just a guess).
>
> Yes, I'm doing the same with sshfs and the borg opts:
>
> borg create --files-cache=ctime,size
>
> so that caching works.
>
> Can I omit that with NFS?
>

I think so, NFS should have stable inodes.


-- 
ing. Fabio Pedretti
Responsabile U.O.C. "Reti, Sistemi e Sicurezza Informatica"
https://www.unibs.it/node/4305
Universit? degli Studi di Brescia
Via Valotti, 9 - 25121 Brescia
E-mail: fabio.pedretti at unibs.it

-- 


Informativa sulla Privacy: http://www.unibs.it/node/8155 
<http://www.unibs.it/node/8155>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20190327/521baefc/attachment.html>

From fw at deneb.enyo.de  Sun Mar 31 16:38:54 2019
From: fw at deneb.enyo.de (Florian Weimer)
Date: Sun, 31 Mar 2019 22:38:54 +0200
Subject: [Borgbackup] borgbackup 1.2.0a5 alpha release
In-Reply-To: <01000169bb7ac1da-c7d4284c-3e0c-463a-b82a-1572d8666949-000000@email.amazonses.com>
 (Thorsten von Eicken's message of "Tue, 26 Mar 2019 19:29:38 +0000")
References: <8ed6ef55-ade6-8669-5040-bcad0aff5e30@waldmann-edv.de>
 <aac7d81e-1346-484b-2dac-8854d1cb794e@gmail.com>
 <CAGQ70euPOaOpfOK1aM=i2jZjkwqOBNUG0J5v_yXfURQNFHXn0A@mail.gmail.com>
 <c032b6c0-e667-0c2b-80a8-9a445ce623d4@gmail.com>
 <ee7cfd1a-1d02-5c5b-18d8-4b3186d968ba@kenbass.com>
 <01000169bb7ac1da-c7d4284c-3e0c-463a-b82a-1572d8666949-000000@email.amazonses.com>
Message-ID: <87zhpa4w4h.fsf@mid.deneb.enyo.de>

* Thorsten von Eicken:

> Let me try to understand... Are the following statements correct WRT a 
> pull model?
>
> - you want a central schedule on the server, specifically, have the 
> server decide when each client should start its backup
>
> - you want the server to hold the credentials to log into the clients, 
> not the other way around
>
> - you're OK with installing some form of backup software on each client

There's also the model where the client initiates all connections, but
the server can decline a backup attempt and tells the client to come
back at some later time.  If I recall correctly (it's been a *really*
long time), TSM used that model.

From ipsens at ripsbusker.no.eu.org  Sun Mar 31 23:12:32 2019
From: ipsens at ripsbusker.no.eu.org (Ipsen S Ripsbusker)
Date: Mon, 01 Apr 2019 03:12:32 +0000
Subject: [Borgbackup] Check
Message-ID: <20190401031235.84A0CE46AB@mailuser.nyi.internal>

Dear Collective,

Suppose I run "borg create ..." on one computer with the repository
on another computer. Then I remove power from the first computer.
How can I inspect the records on the second computer to figure
out what "borg create ..." did before I removed the power?

For context, here is my motivitation for this question. Lately, a few
minutes I start my full system borg backup, the system ceases to respond
to my interactions and begins to print "ahci0: unrecoverable errors
(IS: 10<UFS>), disabling port." and over and over to dmesg. Then I turn
it off, reboot, run fsck, and get errors about "softraid0: sd4: i/o
error 5 @ CRYPTO block". This has also happened at least once when
I wasn't running borg, but it happens reliably when I run borg.
So it seems like something is wrong with my (OpenBSD) sd4 softraid
device. Or maybe the SSD underneath it. Since borg accesses lots
of things on the filesystem and writes stuff to another computer,
it may create records that are useful for debugging this issue.

With great humility,
Ipsen