[Borgbackup] Lots of files that change rarely and predictably

[Thomas Waldmann]

On 09/11/2017 12:24 PM, Thomas Levine wrote:
> I have a directory of about 670,683 files in a single borg repository.
> Archive creation takes a long time because borg checks all of the files.
> I should be able to make the backup go faster because the files change
> very predictably, and I seek your advice on how to do this.

Well, if these are years of email, maybe creating an archive of the past
years in mbox format would be the easiest performance fix.

> 
> The directory contains emails in mail handler format, with one email per
> file.

This is kind of known/expected to be slow and inefficient.

A filesystem that supports fast handling of such huge directories might
help a bit. See dir_index option and similar.

> Usually, only the files in "drafts" and the target of "old/current"
> change. I want to configure the backups to take advantage of this
> pattern, by having borg ignore the files that I know have not changed.

I am not sure this is doable. You'ld still have to look into the
directory for new files. Borg's files cache lookup only needs to know
mtime, size and inode number to decide which files did not change.

BTW, can you do a few runs with borg 1.1.0rc3 into a new repo to see
whether performance changes when going from 1.0 to 1.1?



-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393