[Borgbackup] Backup taking very long - sometimes

[Roland]

you can have a look with "stat" utility for file timestamp change of the
affected files

i'm using borg with btrfs with no problems

regards
roland

Am 17.09.20 um 10:32 schrieb Samuel Greiner:
> Thank you for your advices,
>
> I'm now using the most recent version 1.1.13.
>
> Still the problem persists.
>
> With the option "--list --filter="AME"" I see that some backups run very
> fast and only a considerable amount of files is changed or added.
> The backups that have a really long duration backup a whole lot of files
> as modified which should be untouched.
>
> The source is a nextcloud data directory and the data is on a btrfs volume.
>
> Do you have advice how I could investigate this further?
>
> Thank you very much
> Samuel
>
>
>
> Am 10.09.20 um 10:47 schrieb Winfried Schürmann:
>> Just a hint:
>> it's easy to install the binaries of Borg 1.1.13 not waiting for updates
>> of repos:
>> https://borgbackup.readthedocs.io/en/stable/installation.html#pyinstaller-binary
>>
>> Winfried
>>
>>
>> Am 10.09.20 um 09:52 schrieb Samuel:
>>> Okay.
>>>
>>> I'm now using 1.1.11 from buster-backports.
>>>
>>> Next time I will try to shorten the logs to the relevant part.
>>>
>>> You are right, there is a huge amount of files classified as modified,
>>> which should be unchanged (for several years).
>>> Now I'm unsure in how to investigate further.
>>> The files are stored on a btrfs filesystem across 4 HDs as RAID1.
>>>
>>> Unfortunately I didn't find a hint in the FAQ regarding this behaviour.
>>>
>>> I will now wait for the next Backups with the more recent version of
>>> borgbackup and try to investigate the new logs.
>>>
>>>
>>> Thanks a lot for your help!
>>> Samuel
>>>
>>> Am 09.09.20 um 12:34 schrieb Thomas Waldmann:
>>>>> Stats was already enabled and the duration of the backup doesn't seem to
>>>>> depend on the data added (i attached the of the borg backup).
>>>> You should use a more recent release than borg 1.1.9, see the advisory
>>>> at the top of the changelog.
>>>>
>>>>> Logs with stats enabled but without --list --filter="AME" can be found here:
>>>>> https://pastebin.com/raw/RW43kBHi
>>>> Next time you could shorten the logs so they only show the relevant /
>>>> interesting stuff.
>>>>
>>>> Didn't see anything remarkable there (except maybe that your "System"
>>>> backup includes a rather high count of files).
>>>>
>>>>> With --list --filter="AME" enabled the logs since 2020-09-06 exceed
>>>>> 200Mbyte. Therefore i can't send them.
>>>> It wasn't intended that you publish that log, but rather that you check
>>>> yourself whether the A)dded and M)odified files shown in there are
>>>> somehow as expected.
>>>>
>>>> If it shows files as M)odified that had no content modification, that is
>>>> an indication that the borg files cache maybe doesn't work ok for you /
>>>> for that filesystem type or that the files were slightly "touched" by
>>>> something.
>>>>
>>>> You have to find out then what the root cause for this is, see the FAQ
>>>> and the documentation of --files-cache.
>>>>
>>>> Or if it shows a huge amount of E)rrors, then find out if the errors are
>>>> severe or if they can be ignored (sometimes one can just exclude
>>>> problematic stuff if it does not need to be backed up anyway).
>>>>
>>>>
>>>>
>>>>
>>>>> Am 04.09.20 um 14:29 schrieb Thomas Waldmann:
>>>>>>> Borg runs on a regular daily basis and is working stable.
>>>>>>> But the consumed cpu time varies between around 4 hours and 12-13 hours:
>>>>>> Could be because on different days, a different amount of data changes?
>>>>>>
>>>>>> Or (if you do backups to a remote repo server), the connection speed to
>>>>>> / system load of source or target server is different.
>>>>>>
>>>>>> Without more infos, it is hard to say.
>>>>>>
>>>>>> You could start by adding this to borg create:
>>>>>>
>>>>>> --stats --list --filter="AME"
>>>>>>
>>>>>> stats should tell if there was really much data added.
>>>>>> if so, the list output should hint about what it was.
>>> _______________________________________________
>>> Borgbackup mailing list
>>> Borgbackup at python.org
>>> https://mail.python.org/mailman/listinfo/borgbackup
>>>
>> _______________________________________________
>> Borgbackup mailing list
>> Borgbackup at python.org
>> https://mail.python.org/mailman/listinfo/borgbackup
> _______________________________________________
> Borgbackup mailing list
> Borgbackup at python.org
> https://mail.python.org/mailman/listinfo/borgbackup