[Borgbackup] Baffling behavior on low cpu system with borg backup

[Thomas Waldmann]

On 06/29/2016 03:16 AM, tmhikaru at gmail.com wrote:
> 	Been a while. Sorry about this, I injured my wrist and it is hard to
> type one handed.

Wish you a good recovery!

> 	Did tests over the weekend. Despite everything I tried, I could not
> coerce the RPI into syncing properly, so out of frustration, I tried doing
> this to see if it'd do something differently:
> https://github.com/borgbackup/borg/pull/238

Aside from not caching the single-archive chunk indexes on disk, it does
a very similar thing.

> RPI, 32bit armv6 ~480MB of ram:
> -rw------- 1 root root 96336214 Jun 24 21:17 .cache/borg/40f5689bc9c5faac015dd94283a4dfb42dc5361bf0c77fd1402c365235ebd8f9/chunks

OK, so the 96MB of chunk index data will still fit into RAM.
Even twice maybe if it has to grow the hash table a bit (copying the
smaller current table into a bigger new one).

But note that is already half of your memory.

Then, while it is merging the single-archive indexes, one of them is
also in memory, so now we are at 2/3 of your RAM (only for that data).

I still think you are maybe experiencing a memory / paging issue.
And if not now, then maybe in future, if your repo or file count grows a
bit.

> Now, the interesting thing is, this was not a permanent fix! Despite that it
> was able to sync twice in a row, a few days later when I tried to run a full
> backup using my script which is run the same way my test was, it got
> completely jammed

If your chunks count in the repo grew significantly due to that, it
maybe was using much more than 96MB then.

There is also a files cache eating some memory, see the docs for the
formula.

The memory is needed on the machine running "borg create" (not: "borg
serve", there it only needs to hold the repo index in memory).

> and was still trying to sync one of the archives it had
> worked through days before in less than an hour in both previous tests, but
> six hours later when I came home it was just sitting there hammering away at
> the cpu.  I have since given up on trying to use it this way, and am back to
> using sshfs.  sshfs is much faster despite its drawbacks since there is no
> cache to sync, and apparently borg client reading sshfs on the server doing
> a full backup runs ~1 hr faster than it does running on the RPI even not
> counting sync time.

I assume you mean sshfs as a source of backup data.

> I am still utterly clueless as to why this is happening. Ideas?

Besides memory issues, it could be also an instance of the suspected
"hashtable performance breakdown" (see issue tracker) - this might
depend on the specific values stored into the hashtable.


-- 

GPG ID: FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393