<div dir="ltr">I thought I'd follow up on this. I now have a solution where it's working very quickly on successive backups after the first one. <div><br></div><div>Here's my setup:</div><div><br></div><div>/usr/local/borg --borg repository on local hard disk</div><div>/mnt/efs_borg --NFS (actually Amazon EFS) share with Borg data folder on it.</div><div>/mnt/efs_prod --NFS share with production source data to be backed up.</div><div><br></div><div>symlink /usr/local/borg/data to /mnt/efs_borg/data. That way all data is on the EFS share and the other index-related stuff is on the local drive.</div><div><br></div><div>Backing up /mnt/efs_prod initially took about 45 minutes for 2.5 GB. Second backup took under two minutes.</div><div><br></div><div>I back up /mnt/efs_prod and then I back up everything in /usr/local/borg and exclude /usr/local/borg/data. Then, for easy restores, I tar everything in /usr/local/borg except data and copy that tar file to /mnt/efs_borg/. This last tar file just overwrites the last one, but all I would need it for would be to </div><div><br></div><div>This seems to work great. Any thoughts, concerns, etc, about using it like this with the index files local and the data folder remote? Any concerns for disaster recovery like this? What happens if I lose everything on the local drive? Can I rebuild an index from just the data folder on NFS?</div><div><br></div><div>Thanks,<br>Dave</div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Aug 22, 2017 at 10:53 AM, David Thibault <span dir="ltr"><<a href="mailto:dave@artlauncher.online" target="_blank">dave@artlauncher.online</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thanks for the quick response!<div><br></div></div><div class="m_-2211290398204552935HOEnZb"><div class="m_-2211290398204552935h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Aug 22, 2017 at 10:37 AM, Thomas Waldmann <span dir="ltr"><<a href="mailto:tw@waldmann-edv.de" target="_blank">tw@waldmann-edv.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>> I've got Borg working with all of the files (data, indexes, and<br>
> everything) on an NFS share (actually, it's Amazon EFS). It's actually<br>
> backing up image data from one NFS share (my WP Uploads files) to the<br>
> borg repo on a different NFS share.<br>
><br>
> It works fine, but it takes about 45 minutes to run through less than<br>
> 3GB of image data, even on subsequent runs.<br>
<br>
</span>That's likely because it reads/chunks everything because it thinks the<br>
file has changed. This is when one of mtime, inode nr, size is not<br>
stable (usually the inode nr is not stable on network filesystems).<br>
<br>
If you can trust only mtime and size for the "is the file changed"<br>
detection, you can tell borg to ignore the inode number (see<br>
--ignore-inode or so).<br>
<span><br>
> I'm guessing the process<br>
> would be much faster if I stored the Borg metadata on the local server<br>
> and only the data blocks on NFS. Then, after the main borg backup, I<br>
> could do a cron script that would zip up the remaining borg metadata<br>
> files, then borg could back that zip file as well.<br>
<br>
</span>Not sure how you mean that, but in general local storage is usually<br>
faster than network storage.<br>
<br>
Also have a look at cache / index sizes and maybe increase borg 1.0.x<br>
--checkpoint-interval from 300 (5mins) to 1800 (30mins) to have fewer<br>
checkpoints but also less overhead - esp. if indexes/caches are large<br>
and reading/writing them is slow.<br>
<span><br>
> If you agree, which files would be best kept on the local disk and which<br>
> should be on NFS. Judging by the repo contents, I'm guessing the data<br>
> folder on NFS and the rest on the local disk. Can you confirm?<br>
<br>
</span>Maybe try the --checkpoint-interval first, then you maybe do not have to<br>
do that.<br>
<span class="m_-2211290398204552935m_-1193217580131750963HOEnZb"><font color="#888888"><br>
<br>
--<br>
<br>
GPG ID: 9F88FB52FAF7B393<br>
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393<br>
<br>
______________________________<wbr>_________________<br>
Borgbackup mailing list<br>
<a href="mailto:Borgbackup@python.org" target="_blank">Borgbackup@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/borgbackup" rel="noreferrer" target="_blank">https://mail.python.org/mailma<wbr>n/listinfo/borgbackup</a><br>
</font></span></blockquote></div><br></div>
</div></div></blockquote></div><br></div></div>