From adrian.klaver at aklaver.com Fri Jan 8 11:46:14 2016 From: adrian.klaver at aklaver.com (Adrian Klaver) Date: Fri, 8 Jan 2016 08:46:14 -0800 Subject: [Borgbackup] Collect Backups In-Reply-To: References: Message-ID: <568FE7D6.2040706@aklaver.com> On 12/27/2015 11:38 AM, Matthias Rieche wrote: > Hey, > > i have a Question. > When i have Backuprepositorys > /mnt/borg/::mon > /mnt/borg/::die > /mnt/borg/::mi > /mnt/borg/::do > /mnt/borg/::fr > > Can i collect the 5 Repositories to example /mnt/borg/::week50 ? > > After the Collect borg should have only have the repository ::week50 ? What do want ::week50 to be? In others word do you want it to contain all the various versions in the daily repos or do want it to be the state at the end of the week? If the latter take a look at: http://borgbackup.readthedocs.org/en/stable/usage.html#borg-prune If not, what is the harm of keeping the daily repos as the point of dedup is that they are differences not complete backups. > > I?ve seen someone with an Borg Hoodie at the Congress. > So i find this Project and i think it?s perfect for Backup my Fileserver's ;) > > Gru? aus Hamburg > > Matze > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > -- Adrian Klaver adrian.klaver at aklaver.com From tw at waldmann-edv.de Fri Jan 8 11:53:24 2016 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Fri, 8 Jan 2016 17:53:24 +0100 Subject: [Borgbackup] Collect Backups In-Reply-To: References: Message-ID: <568FE984.7040806@waldmann-edv.de> Hi Matze, > i have a Question. > When i have Backuprepositorys > /mnt/borg/::mon > /mnt/borg/::die > /mnt/borg/::mi > /mnt/borg/::do > /mnt/borg/::fr Maybe leave away the rightmost "/", it is not needed. Also, these mon die mi ... things are called archives (== individual backups), not repositories. The repository is "/mnt/borg". > Can i collect the 5 Repositories to example /mnt/borg/::week50 ? Easiest way is to use an ISO datetime string as archive name and use prune (as seen in the docs). Be careful when configuring / first using prune, always first use dry run to see what it would do. There are some useful hints in the docs. You could also use borg delete and borg rename to deal with the already existing archives as you like. > I?ve seen someone with an Borg Hoodie at the Congress. That was me. :) Glad that I worked as living advertisement. :D Cheers, Thomas -- GPG ID: FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From adrian.klaver at aklaver.com Fri Jan 8 11:54:41 2016 From: adrian.klaver at aklaver.com (Adrian Klaver) Date: Fri, 8 Jan 2016 08:54:41 -0800 Subject: [Borgbackup] Fwd: --exclude-from | --remote-path In-Reply-To: <568FA81F.1080305@gmail.com> References: <568ED1FB.4020704@gmail.com> <568FA81F.1080305@gmail.com> Message-ID: <568FE9D1.3090803@aklaver.com> On 01/08/2016 04:14 AM, Stan Armstrong wrote: > My exclusion list is being ignored. I have attached it. I am trying to > backup to a remote server on my LAN. The size of the backup is much too > large. I finally interrupted it. I assume that my exclusion of /media is > being ignored, and my large data disks are being included. > > Here is the command I have used: > > # borg create --exclude-from /home/stan/backup-exclusions > root at Barbara-office:/media/Backup-S/Borg-S::Thursday /* Wonder if this, /*, is the problem? Have you tried just, / ? > > Is the format of my list wrong? I can't tell if those are double quotes or two single quotes. If it where me I would just use single quotes, i.e '/dev/*' > Do I have it in the wrong place? > > I started borg on the server machine using "borg server" issued from > that machine. > > In future, will I have to do this, or is the purpose of the > "--remote-path PATH" option to make the borg on the server start > processing the archive? If so, what should PATH be? The documentation > says (default: "borg"). I don't understand how to interpret that so as > to end up with the correct option. > > > > > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > -- Adrian Klaver adrian.klaver at aklaver.com From tw at waldmann-edv.de Fri Jan 8 12:06:50 2016 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Fri, 8 Jan 2016 18:06:50 +0100 Subject: [Borgbackup] Fwd: --exclude-from | --remote-path In-Reply-To: <568FA81F.1080305@gmail.com> References: <568ED1FB.4020704@gmail.com> <568FA81F.1080305@gmail.com> Message-ID: <568FECAA.1060908@waldmann-edv.de> Hi Stan, > My exclusion list is being ignored. I have attached it. I am trying to > backup to a remote server on my LAN. The size of the backup is much too > large. I finally interrupted it. I assume that my exclusion of /media is > being ignored, and my large data disks are being included. Always use create --dry-run --verbose when setting such stuff up (same goes for prune policy and restrictions), then you quickly see what it considers for backup and what not. > # borg create --exclude-from /home/stan/backup-exclusions > root at Barbara-office:/media/Backup-S/Borg-S::Thursday /* Maybe rather use an ISO datetimestring as shown in the docs as archive name. Archive names need to be unique and a week later, there will be another Thursday. /* is likely not doing what you want. A unix shell will immediately expand this to all items found in the root directory and give the expanded list to borg. Try just: / > Is the format of my list wrong? Remove all the double quotes and retry. > I started borg on the server machine using "borg server" issued > from that machine. You don't need to manually start "borg serve" on the server. If you invoke borg on the client and give a ssh://... repo url, it will automatically establish a connection to the server, invoke "borg serve" there and then the both borg talk to each other over the ssh connection. > In future, will I have to do this, or is the purpose of the > "--remote-path PATH" option to make the borg on the server start > processing the archive? You usually do not need that option, except if borg is not in the $PATH system search path on the server or isn't named "borg". Cheers, Thomas -- GPG ID: FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From tw at waldmann-edv.de Fri Jan 8 12:11:33 2016 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Fri, 8 Jan 2016 18:11:33 +0100 Subject: [Borgbackup] mailing list configuration - quickly reject non-subscribers Message-ID: <568FEDC5.3090007@waldmann-edv.de> I changed the mailing list configuration to immediately reject mails from addresses that are not subscribed to the list (instead of holding them and requiring me to "moderate" them, what may take a while and in effect the poster still won't be subscribed). Everybody can subscribe to this list via: https://mail.python.org/mailman/listinfo/borgbackup That way, new posters get quicker feedback (then will subscribe and re-post) and we are still a little protected from spammers. -- GPG ID: FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From mfseeker at gmail.com Fri Jan 8 13:20:23 2016 From: mfseeker at gmail.com (Stan Armstrong) Date: Fri, 08 Jan 2016 14:20:23 -0400 Subject: [Borgbackup] Fwd: --exclude-from | --remote-path In-Reply-To: <568FECAA.1060908@waldmann-edv.de> References: <568ED1FB.4020704@gmail.com> <568FA81F.1080305@gmail.com> <568FECAA.1060908@waldmann-edv.de> Message-ID: <568FFDE7.9020906@gmail.com> Thanks Thomas, Now I am getting somewhere. I learned that I needed to run my borg create from root in order to avoid numerous "Permission denied" errors. However, the verbose output is too large and fast to be examined carefully. I tried both "> filename" and "| tee filename" redirects. Both create empty files. Is there some other way to get --verbose output into a file? Stan On 08/01/16 01:06 PM, Thomas Waldmann wrote: > Hi Stan, > >> My exclusion list is being ignored. I have attached it. I am trying to >> backup to a remote server on my LAN. The size of the backup is much too >> large. I finally interrupted it. I assume that my exclusion of /media is >> being ignored, and my large data disks are being included. > Always use create --dry-run --verbose when setting such stuff up (same > goes for prune policy and restrictions), then you quickly see what it > considers for backup and what not. > >> # borg create --exclude-from /home/stan/backup-exclusions >> root at Barbara-office:/media/Backup-S/Borg-S::Thursday /* > Maybe rather use an ISO datetimestring as shown in the docs as archive > name. Archive names need to be unique and a week later, there will be > another Thursday. > > /* is likely not doing what you want. A unix shell will immediately > expand this to all items found in the root directory and give the > expanded list to borg. Try just: / > >> Is the format of my list wrong? > Remove all the double quotes and retry. > >> I started borg on the server machine using "borg server" issued >> from that machine. > You don't need to manually start "borg serve" on the server. > > If you invoke borg on the client and give a ssh://... repo url, it will > automatically establish a connection to the server, invoke "borg serve" > there and then the both borg talk to each other over the ssh connection. > >> In future, will I have to do this, or is the purpose of the >> "--remote-path PATH" option to make the borg on the server start >> processing the archive? > You usually do not need that option, except if borg is not in the $PATH > system search path on the server or isn't named "borg". > > Cheers, Thomas > From tw at waldmann-edv.de Fri Jan 8 13:49:16 2016 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Fri, 8 Jan 2016 19:49:16 +0100 Subject: [Borgbackup] Fwd: --exclude-from | --remote-path In-Reply-To: <568FFDE7.9020906@gmail.com> References: <568ED1FB.4020704@gmail.com> <568FA81F.1080305@gmail.com> <568FECAA.1060908@waldmann-edv.de> <568FFDE7.9020906@gmail.com> Message-ID: <569004AC.60203@waldmann-edv.de> > Now I am getting somewhere. I learned that I needed to run my borg > create from root in order to avoid numerous "Permission denied" errors. Sure, if you have no permissions, you can't read some stuff. Usually one uses backup tools in 2 ways: a) as a normal user to backup that user's data (and nothing else) b) as root to backup all or anything one wants > However, the verbose output is too large and fast to be examined > carefully. I tried both "> filename" and "| tee filename" redirects. > Both create empty files. Is there some other way to get --verbose output > into a file? Use 2> as we emit log or user directed output on stderr. We do not use stdout as sometimes we need stdout to transport actual (binary) backup data. -- GPG ID: FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From tw at waldmann-edv.de Fri Jan 8 13:53:16 2016 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Fri, 8 Jan 2016 19:53:16 +0100 Subject: [Borgbackup] IRC channel Message-ID: <5690059C.8050900@waldmann-edv.de> BTW, for chatting and more interactive discussions, there is also a channel on chat.freenode.net called #borgbackup . You can either use a local IRC client to access that or some of the web-based irc clients available on the web. -- GPG ID: FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From mfseeker at gmail.com Sat Jan 16 10:59:41 2016 From: mfseeker at gmail.com (Stan Armstrong) Date: Sat, 16 Jan 2016 11:59:41 -0400 Subject: [Borgbackup] bprg-prune documentation inconsistent Message-ID: <569A68ED.7070204@gmail.com> http://borgbackup-anarcat.readthedocs.org/en/docs/usage.html In the entry for "borg prune", the examples are of the form: $ borg prune /mnt/backup --dry-run --keep-daily=7 --keep-weekly=4 The target repository is placed /before/ the options. The "usage:" entry has the more usual command-options-REPOSITORY order. I am guessing that in this case the order doesn't matter, but it would be good to tidy things up. Stan -------------- next part -------------- An HTML attachment was scrubbed... URL: From tw at waldmann-edv.de Sat Jan 16 11:05:57 2016 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Sat, 16 Jan 2016 17:05:57 +0100 Subject: [Borgbackup] bprg-prune documentation inconsistent In-Reply-To: <569A68ED.7070204@gmail.com> References: <569A68ED.7070204@gmail.com> Message-ID: <569A6A65.9060201@waldmann-edv.de> On 01/16/2016 04:59 PM, Stan Armstrong wrote: > http://borgbackup-anarcat.readthedocs.org/en/docs/usage.html Btw, that is some readthedocs experiment from anarcat, the official docs site is at http://borgbackup.readthedocs.org/ . "stable" docs refer to the latest release (can be weeks/months old) and "latest" docs reflect what we have in the master branch of the repo. > In the entry for "borg prune", the examples are of the form: > > $ borg prune /mnt/backup --dry-run --keep-daily=7 --keep-weekly=4 > > The target repository is placed /before/ the options. The "usage:" entry > has the more usual command-options-REPOSITORY order. > > I am guessing that in this case the order doesn't matter, but it would > be good to tidy things up. Yes, good point (and also applies to current master branch). https://github.com/borgbackup/borg/issues/560 -- GPG ID: FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From mfseeker at gmail.com Mon Jan 18 09:54:44 2016 From: mfseeker at gmail.com (Stan Armstrong) Date: Mon, 18 Jan 2016 10:54:44 -0400 Subject: [Borgbackup] extract pattern never matched Message-ID: <569CFCB4.2080500@gmail.com> I am unable to extract due to an "Include pattern ... never matched" error.The command that is failing is: borg extract -v /media/4TB-1/SonS::Documents-201601180856 /media/2TB-1/home/stan/Documents/pdf* 2>/home/stan/Desktop/borg-extraction-log The exact error message is: "Include pattern '/media/2TB-1/home/stan/Documents/pdf' never matched." I am running the command from my home directory /home/stan/Desktop with root privileges. The file manager shows that the directory '/media/2TB-1/home/stan/Documents/pdf' exists. I clearly do not understand how to enter the pattern. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tw at waldmann-edv.de Tue Jan 19 02:53:59 2016 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Tue, 19 Jan 2016 08:53:59 +0100 Subject: [Borgbackup] extract pattern never matched In-Reply-To: <569CFCB4.2080500@gmail.com> References: <569CFCB4.2080500@gmail.com> Message-ID: <569DEB97.1050207@waldmann-edv.de> On 01/18/2016 03:54 PM, Stan Armstrong wrote: > I am unable to extract due to an "Include pattern ... never matched" > error. borg --version ? > The command that is failing is: > > borg extract -v /media/4TB-1/SonS::Documents-201601180856 > /media/2TB-1/home/stan/Documents/pdf* > 2>/home/stan/Desktop/borg-extraction-log That's unrelated to borgbackup: in general you need to be careful with wildcard expressions: if you do not quote them, the shell will expand them before giving them to borg. * -> will be expanded by shell '*' -> won't be expanded by shell > The exact error message is: > > "Include pattern '/media/2TB-1/home/stan/Documents/pdf' never matched." You see: no * there. Also, you seem to be ahead of time, we just recently added full pattern support in github master. So maybe rather try extracting the full filename(s) or the base directory (no * at the end). > The file manager shows that the directory > '/media/2TB-1/home/stan/Documents/pdf' exists. For extract you rather have to check with borg list whether a filename inside the archive matches the expression you gave. -- GPG ID: FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From anarcat at debian.org Sat Jan 23 15:05:30 2016 From: anarcat at debian.org (Antoine =?utf-8?Q?Beaupr=C3=A9?=) Date: Sat, 23 Jan 2016 15:05:30 -0500 Subject: [Borgbackup] bprg-prune documentation inconsistent In-Reply-To: <569A6A65.9060201@waldmann-edv.de> References: <569A68ED.7070204@gmail.com> <569A6A65.9060201@waldmann-edv.de> Message-ID: <8760ykqcxh.fsf@marcos.anarc.at> On 2016-01-16 11:05:57, Thomas Waldmann wrote: > On 01/16/2016 04:59 PM, Stan Armstrong wrote: >> http://borgbackup-anarcat.readthedocs.org/en/docs/usage.html > > Btw, that is some readthedocs experiment from anarcat, the official docs > site is at http://borgbackup.readthedocs.org/ . well, that's embarrassing... i have removed the project to avoid further confusion. a. -- No animal has more liberty than the cat; but it buries the mess it makes. The cat is the best anarchist. Until they learn that from the cat I cannot respect them. - For whom the bell tolls, Ernest Hemingway From tw at waldmann-edv.de Sat Jan 23 19:24:18 2016 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Sun, 24 Jan 2016 01:24:18 +0100 Subject: [Borgbackup] borgbackup 0.30.0 released Message-ID: <56A419B2.8060606@waldmann-edv.de> See there: https://github.com/borgbackup/borg/releases/tag/0.30.0 It's also available on PyPi (via pip install). Binaries not there yet, but will come soon. Please help testing, this is the last release before 1.0. Cheers, Thomas -- GPG ID: FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From jungleboogie0 at gmail.com Sat Jan 23 23:06:04 2016 From: jungleboogie0 at gmail.com (jungle Boogie) Date: Sat, 23 Jan 2016 20:06:04 -0800 Subject: [Borgbackup] borgbackup 0.30.0 released In-Reply-To: <56A419B2.8060606@waldmann-edv.de> References: <56A419B2.8060606@waldmann-edv.de> Message-ID: Hi Thomas, On Jan 23, 2016 4:24 PM, "Thomas Waldmann" wrote: > > See there: > > https://github.com/borgbackup/borg/releases/tag/0.30.0 > > It's also available on PyPi (via pip install). > > Binaries not there yet, but will come soon. > > Please help testing, this is the last release before 1.0. Thanks for this release. I've updated to .30 via pip this evening. > > Cheers, > > Thomas > -- > > GPG ID: FAF7B393 > GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From adrian.klaver at aklaver.com Mon Jan 25 10:43:50 2016 From: adrian.klaver at aklaver.com (Adrian Klaver) Date: Mon, 25 Jan 2016 07:43:50 -0800 Subject: [Borgbackup] borgbackup 0.30.0 released In-Reply-To: <56A419B2.8060606@waldmann-edv.de> References: <56A419B2.8060606@waldmann-edv.de> Message-ID: <56A642B6.2080209@aklaver.com> On 01/23/2016 04:24 PM, Thomas Waldmann wrote: > See there: > > https://github.com/borgbackup/borg/releases/tag/0.30.0 Thanks, especially for restoring day of week to output. > > It's also available on PyPi (via pip install). > > Binaries not there yet, but will come soon. > > Please help testing, this is the last release before 1.0. Loaded it on three machines and so far no issues. > > Cheers, > > Thomas > -- Adrian Klaver adrian.klaver at aklaver.com From mfseeker at gmail.com Fri Jan 29 09:35:24 2016 From: mfseeker at gmail.com (Stan Armstrong) Date: Fri, 29 Jan 2016 10:35:24 -0400 Subject: [Borgbackup] Finding exclusive lock files Message-ID: <56AB78AC.2040609@gmail.com> I know that exclusive locks can be found in two places (as per the documentation at http://borgbackup.readthedocs.org/en/0.29.0/internals.html?highlight=locks: the cache locks are at ~/.cache/borg/REPOID/lock.* and the repository locks are in repository/lock.*. I am running a simple home LAN with two machines to be backed up both locally and to the other machine. Even confining my question to local backups on each machine, I have two .cache directories, one for root and one for each local user. Each repository contains several archives; therefore there are several subdirectories in each place, each of which must be manually opened to find whether it contains a lock.exclusive file. Is there a simpler way of determining where exclusive locks might be? ("find" takes too long.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tw at waldmann-edv.de Fri Jan 29 09:44:50 2016 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Fri, 29 Jan 2016 15:44:50 +0100 Subject: [Borgbackup] Finding exclusive lock files In-Reply-To: <56AB78AC.2040609@gmail.com> References: <56AB78AC.2040609@gmail.com> Message-ID: <56AB7AE2.7060807@waldmann-edv.de> On 01/29/2016 03:35 PM, Stan Armstrong wrote: > I know that exclusive locks can be found in two places (as per the > documentation at > http://borgbackup.readthedocs.org/en/0.29.0/internals.html?highlight=locks: > > the cache locks are at ~/.cache/borg/REPOID/lock.* > and the repository locks are in repository/lock.*. > > I am running a simple home LAN with two machines to be backed up both > locally and to the other machine. Even confining my question to local > backups on each machine, I have two .cache directories, one for root and > one for each local user. Each repository contains several archives; > therefore there are several subdirectories in each place, each of which > must be manually opened to find whether it contains a lock.exclusive file. > > Is there a simpler way of determining where exclusive locks might be? > ("find" takes too long.) Well, it is deterministic, so you don't need "find". The repo lock is always in the repo you use, so that should be pretty clear. The cache lock is always in ~/.cache/borg/ - you can find out the repoid in the repo's "config" file (look into the repo directory). ~ refers to the home directory of the user running borg. That might be root or a normal user, depending on how you invoke borg. If you use the "borg break-lock" command, both locks will be automatically broken / removed. Just make sure first that the locks are erroneous and they are not from a still running borg process. -- GPG ID: FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From mfseeker at gmail.com Fri Jan 29 10:12:41 2016 From: mfseeker at gmail.com (Stan Armstrong) Date: Fri, 29 Jan 2016 11:12:41 -0400 Subject: [Borgbackup] Finding exclusive lock files In-Reply-To: <56AB7AE2.7060807@waldmann-edv.de> References: <56AB78AC.2040609@gmail.com> <56AB7AE2.7060807@waldmann-edv.de> Message-ID: <56AB8169.5080706@gmail.com> Yes, I had found those locations, but I was unaware of the borg break-lock command. That's exactly what I was hoping for. It is a great time-saver. Thanks for implementing it. On 29/01/16 10:44 AM, Thomas Waldmann wrote: > On 01/29/2016 03:35 PM, Stan Armstrong wrote: >> I know that exclusive locks can be found in two places (as per the >> documentation at >> http://borgbackup.readthedocs.org/en/0.29.0/internals.html?highlight=locks: >> >> the cache locks are at ~/.cache/borg/REPOID/lock.* >> and the repository locks are in repository/lock.*. >> >> I am running a simple home LAN with two machines to be backed up both >> locally and to the other machine. Even confining my question to local >> backups on each machine, I have two .cache directories, one for root and >> one for each local user. Each repository contains several archives; >> therefore there are several subdirectories in each place, each of which >> must be manually opened to find whether it contains a lock.exclusive file. >> >> Is there a simpler way of determining where exclusive locks might be? >> ("find" takes too long.) > Well, it is deterministic, so you don't need "find". > > The repo lock is always in the repo you use, so that should be pretty clear. > > The cache lock is always in ~/.cache/borg/ - you can find out > the repoid in the repo's "config" file (look into the repo directory). > ~ refers to the home directory of the user running borg. > That might be root or a normal user, depending on how you invoke borg. > > If you use the "borg break-lock" command, both locks will be > automatically broken / removed. Just make sure first that the locks are > erroneous and they are not from a still running borg process. > From mfseeker at gmail.com Fri Jan 29 12:43:14 2016 From: mfseeker at gmail.com (Stan Armstrong) Date: Fri, 29 Jan 2016 13:43:14 -0400 Subject: [Borgbackup] Interpreting the progress bar Message-ID: <56ABA4B2.4090705@gmail.com> Is the progress indicator (processing files) interpreted somewhere? Here is a snapshot of one of mine: processing files 93.93 GB O 93.93 GB C 25.58 GB D 28258 N media/2TB-1/Down...rnity_1.4.0.6.tar.gz "N" appears to be the full pathname of the current file being processed. The first and second entries change quickly, with number of GB remaining almost identical. the second entry increases by 0.01GB a fraction of a second before the first entry does. What does "O" mean? "C" sometimes increments in step with the first two entries, but it doesn't quite keep up. What is it? "D" changes very slowly. What is it? Thanks, Stan From adrian.klaver at aklaver.com Fri Jan 29 13:09:33 2016 From: adrian.klaver at aklaver.com (Adrian Klaver) Date: Fri, 29 Jan 2016 10:09:33 -0800 Subject: [Borgbackup] Interpreting the progress bar In-Reply-To: <56ABA4B2.4090705@gmail.com> References: <56ABA4B2.4090705@gmail.com> Message-ID: <56ABAADD.7070809@aklaver.com> On 01/29/2016 09:43 AM, Stan Armstrong wrote: > Is the progress indicator (processing files) interpreted somewhere? > > Here is a snapshot of one of mine: > > processing files > 93.93 GB O 93.93 GB C 25.58 GB D 28258 N > media/2TB-1/Down...rnity_1.4.0.6.tar.gz > > "N" appears to be the full pathname of the current file being processed. > > The first and second entries change quickly, with number of GB remaining > almost identical. the second entry increases by 0.01GB a fraction of a > second before the first entry does. What does "O" mean? > > "C" sometimes increments in step with the first two entries, but it > doesn't quite keep up. What is it? > > "D" changes very slowly. What is it? http://borgbackup.readthedocs.org/en/stable/usage.html borg create -p, --progress show progress display while creating the archive, showing Original, Compressed and Deduplicated sizes, followed by the Number of files seen and the path being processed, default: False > > Thanks, > > Stan > > > > > > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup -- Adrian Klaver adrian.klaver at aklaver.com From mfseeker at gmail.com Fri Jan 29 13:48:04 2016 From: mfseeker at gmail.com (Stan Armstrong) Date: Fri, 29 Jan 2016 14:48:04 -0400 Subject: [Borgbackup] Interpreting the progress bar In-Reply-To: <56ABAADD.7070809@aklaver.com> References: <56ABA4B2.4090705@gmail.com> <56ABAADD.7070809@aklaver.com> Message-ID: <56ABB3E4.8010507@gmail.com> Thanks, guys. Makes good sense. I don't know how I missed that. I have been reading and rereading the docs for many days. On 29/01/16 02:09 PM, Adrian Klaver wrote: > On 01/29/2016 09:43 AM, Stan Armstrong wrote: >> Is the progress indicator (processing files) interpreted somewhere? >> >> Here is a snapshot of one of mine: >> >> processing files >> 93.93 GB O 93.93 GB C 25.58 GB D 28258 N >> media/2TB-1/Down...rnity_1.4.0.6.tar.gz >> >> "N" appears to be the full pathname of the current file being processed. >> >> The first and second entries change quickly, with number of GB remaining >> almost identical. the second entry increases by 0.01GB a fraction of a >> second before the first entry does. What does "O" mean? >> >> "C" sometimes increments in step with the first two entries, but it >> doesn't quite keep up. What is it? >> >> "D" changes very slowly. What is it? > > http://borgbackup.readthedocs.org/en/stable/usage.html > > borg create > > > -p, --progress show progress display while creating the archive, > showing Original, Compressed and Deduplicated > sizes, followed by the Number of files seen and the path being > processed, default: False >> >> Thanks, >> >> Stan >> >> >> >> >> >> >> _______________________________________________ >> Borgbackup mailing list >> Borgbackup at python.org >> https://mail.python.org/mailman/listinfo/borgbackup > > From anarcat at debian.org Wed Feb 3 13:23:34 2016 From: anarcat at debian.org (Antoine =?utf-8?Q?Beaupr=C3=A9?=) Date: Wed, 03 Feb 2016 13:23:34 -0500 Subject: [Borgbackup] borg internals narrative Message-ID: <87twlpodp5.fsf@angela.anarcat.ath.cx> I wrote this to a friend to clarify how borg internals works. I think it'd be a great addition to the existing internals docs, but I don't know where to start to make a pull request. Hence this mailing list dump. Please do turn this into a patch. At least, I think the sections in the existing internals docs have to be reordered. Wording and language obviously needs tweaking as well. Probably. Not knowing what people know about backups system, i start from the simplest ones and go in more complex examples, with a parallel to versionning systems. In short: * tar is like renaming files * rsync and rdiff-backups are like CVS * bup, attic and borg are like git Tar --- Glues files together, sends them in a stream, compress. Uses timestamps to add new files to tar or not, but the whole file gets sent in. To restore, you need to read the whole tape or you may miss some bits. Tapes. We know those. dump & restore work the same way. Those are "incremental" in that it can backup only the newest file changes if you want. Takes up a lot of space and is terribly slow at restoring, but reasonably fast at backing up. Compared to a versionning system this is a little like shoving your files on a FTP server and renaming ".1, .2, .3" as you go. rsync & rdiff-backup -------------------- Kind of like tar: glues files together, sends them in a stream (which is usually a pipe to another rsync which unglues the file and so on). Trick is: only send the parts that changed on the stream. Use clever tricks with timestamps and optionally whole file checksums to decide which files to send, and then try to send only parts of the file too. No history, unless you start doing crazy things like hardlinks, something that was built into something we pretend is backup software called rsnapshot. rdiff-backup is similar, except you store the binary diffs on disk. and for good measures, add a bunch of metadata metadata files to store ownership and so on. Those here are "incremental" in storage and transfer as well: only the files that are changed are sent, but also the files are stored by diffing with another version. It's annoying that we use the same word for "incremental rsync" and "incremental tar" because it's an important difference. (In the case of rsnapshot, full copies of all versions are kept, really wasteful.) restoring is slow as you need to uncompress the stack of diffs as deep as you need. rdiff stores the latest version in full and works the diffs down, which means every backup is computationally expensive, as you need to recompute the whole diff tree. fail again. To compare this with versionning, this is the coming of RCS or CVS. You have the file, and next to it you got a `,v` file that stores the previous revisions as diffs. When you add a new version, you either add the latest diff on top (CVS) or you recompute the whole tree (rdiff-backup). bup, attic and borg ------------------- so sorry for the long introduction, but i needed something to hook into. i jumbled them all together bup, attic and borg because that's more or less how i discovered their algorithms. i first found out about bup of course, and was *amazed* at how fast it was compared to my previous backup solution (just rsync, aka the "JWZ method"). it was comparable or faster, which was puzzling to me. if tar is renaming files, and rdiff-backup is CVS, then this is git. in fact, bup is based on git, something i can't quite get my head around it. but we know that git has trouble storing large objects right? because when you index a large file in git, it makes this giant checksum on the giant file. if you change only one bit of that file, the checksum changes and you need to rewrite the whole damn thing. well, the above backup software, instead of doing checksums on file, they do checksums on blocks. attic, in particular, uses a clever algorithm called buzhash: https://en.wikipedia.org/wiki/Buzhash It's used in the rsync protocol, so it's nothing new. What is new is instead of treating the backup source as a collection of files, we treat it as a collection of blocks. bup, for example, has a "bup split" command that can take random stdin and split it into checksummed blocks. you can feed it block devices, disk images, tarballs, and it will do the right thing. I wrote the internals documentation of borg based on mailing lists discussion and a bit of code dissecation: http://borgbackup.readthedocs.org/en/stable/internals.html .. I am not sure I understand it all, to be honest. The fact that borg/attic uses msgpack should just be seen as an implementation detail, so I think you can ignore that in the descriptions. But basically, in borg, you have multiple archives (your different backup checkpoints), that each containing a list of chunks, which contains metadata, which points to other chunks. So basically, your files are bundled up together, i guess a bit like tar does, and then it's indexed based on a configurable, rolling chunk size, which turns that data chunk into basically a list of indexes. The clever thing is that all this works across multiple archives; those chunks are shared across files, backups, you can even share them across machines. And since one file can be many chunks, if you change a smaller part of the file, you don't change all the chunks and the changes to the backup are minimal. Here's a nasty ascii art diagram for you +--------+ +--------+ +--------+ |archive0| |archive1| ... |archiveN| +--------+ +--------+ +--+-----+ | | | | | | | +---+ | | | | | | | +------+-------+ | | | | | /chunk\/chunk\/chunk\... /maybe different chunks lists\ +-----------------------------------------------------------------+ |item list | +-----------------------------------------------------------------+ | +-------------------------------------+--------------+ | | | | | | +-------------+ +-------------+ | |item0 | |item1 | | | - owner | | - owner | | | - size | | - size | ... | - ... | | - ... | | - chunks | | - chunks | +----+--------+ +-----+-------+ | | | +-----+----------------------------+-----------------+ | | | | +-o-----o------------+ | | | | | | /chunk0\/chunk1\ ... /chunkN\ /chunk0\/chunk1\ ... /chunkN'\ +-----------------------------+ +------------------------------+ |file0 | |file0' | +-----------------------------+ +------------------------------+ Notice how the metadata list itself is chunked and deduplicated. Chunks storage is a simple key/value storage system (more or less). This allows purging backups: when you remove an archive, if no other archive refers to some chunk, you drop it from the storage. Then there's encryption shoved on top of that: basically, I *think* encryption is done by chunk: each chunk is AES-256 encrypted and, on the result, a HMAC-SHA256 is computed (instead of the regular SHA256 key for unencrypted backups). Then there's all sorts of indexes and caches to make this really fast. Oh, and chunks can be compressed of course. a. -- That's one of the remarkable things about life: it's never so bad that it can't get worse. - Calvin From anarcat at debian.org Wed Feb 3 13:24:59 2016 From: anarcat at debian.org (Antoine =?utf-8?Q?Beaupr=C3=A9?=) Date: Wed, 03 Feb 2016 13:24:59 -0500 Subject: [Borgbackup] borg internals narrative In-Reply-To: <87twlpodp5.fsf@angela.anarcat.ath.cx> References: <87twlpodp5.fsf@angela.anarcat.ath.cx> Message-ID: <87r3gtodms.fsf@angela.anarcat.ath.cx> This diagram was copied to: https://github.com/jborg/attic/issues/216 this wasn't dealt with in borg yet either: https://github.com/jborg/attic/pull/223 ... but probably should. a. -- O gentilshommes, la vie est courte. Si nous vivons, nous vivons pour marcher sur la t?te des rois. - William Shakespeare From tw at waldmann-edv.de Sun Feb 7 08:29:36 2016 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Sun, 7 Feb 2016 14:29:36 +0100 Subject: [Borgbackup] borgbackup 1.0.0rc1 released Message-ID: <56B746C0.8050702@waldmann-edv.de> https://github.com/borgbackup/borg/releases/tag/1.0.0rc1 It's also available on PyPi (via pip install). Please read CHANGES.rst and help testing to find the bugs before 1.0.0 release. Cheers, Thomas --- Waldmann EDV Systeme/Service GbR Gerhard Waldmann, Thomas Waldmann Pleidelsheimer Str. 25 74321 Bietigheim-Bissingen Weitere Infos siehe: http://waldmann-edv.de/ GPG ID: FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From jungleboogie0 at gmail.com Sun Feb 7 18:12:33 2016 From: jungleboogie0 at gmail.com (jungle Boogie) Date: Sun, 7 Feb 2016 15:12:33 -0800 Subject: [Borgbackup] borgbackup 1.0.0rc1 released In-Reply-To: <56B746C0.8050702@waldmann-edv.de> References: <56B746C0.8050702@waldmann-edv.de> Message-ID: Hi Thomas, On 7 February 2016 at 05:29, Thomas Waldmann wrote: > https://github.com/borgbackup/borg/releases/tag/1.0.0rc1 > > It's also available on PyPi (via pip install). > > Please read CHANGES.rst and help testing to find the bugs before 1.0.0 > release. Really happy to see you made this a rc before going to straight 1.0. Are you sure this is in pip, or would the version number not change? % sudo pip3 install --upgrade borgbackup Requirement already up-to-date: borgbackup in /usr/local/lib/python3.4/site-packages Collecting msgpack-python>=0.4.6 (from borgbackup) Downloading msgpack-python-0.4.7.tar.gz (126kB) 100% |????????????????????????????????| 126kB 732kB/s Building wheels for collected packages: msgpack-python Running setup.py bdist_wheel for msgpack-python ... done Stored in directory: /root/.cache/pip/wheels/2e/9d/d0/ab7eadb620bf358177875fa0a25dc83273e341ecb71c61509d Successfully built msgpack-python Installing collected packages: msgpack-python Found existing installation: msgpack-python 0.4.6 Uninstalling msgpack-python-0.4.6: Successfully uninstalled msgpack-python-0.4.6 Successfully installed msgpack-python-0.4.7 % borg -V borg 0.30.0 > > Cheers, > > Thomas > > --- > > Waldmann EDV Systeme/Service GbR > Gerhard Waldmann, Thomas Waldmann > Pleidelsheimer Str. 25 > 74321 Bietigheim-Bissingen > > Weitere Infos siehe: http://waldmann-edv.de/ > > GPG ID: FAF7B393 > GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 > -- ------- inum: 883510009027723 sip: jungleboogie at sip2sip.info xmpp: jungle-boogie at jit.si From tw at waldmann-edv.de Sun Feb 7 19:02:18 2016 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Mon, 8 Feb 2016 01:02:18 +0100 Subject: [Borgbackup] borgbackup 1.0.0rc1 released In-Reply-To: References: <56B746C0.8050702@waldmann-edv.de> Message-ID: <56B7DB0A.6030009@waldmann-edv.de> >> https://github.com/borgbackup/borg/releases/tag/1.0.0rc1 >> >> It's also available on PyPi (via pip install). >> >> Please read CHANGES.rst and help testing to find the bugs before 1.0.0 >> release. > > Really happy to see you made this a rc before going to straight 1.0. > > Are you sure this is in pip, or would the version number not change? Use: pip install "borgbackup==1.0.0rc1". Otherwise you will only get releases. -- GPG ID: FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From tw at waldmann-edv.de Sun Feb 7 19:03:58 2016 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Mon, 8 Feb 2016 01:03:58 +0100 Subject: [Borgbackup] borgbackup 0.30.1 released Message-ID: <56B7DB6E.6050700@waldmann-edv.de> This is a small bugfix release only for people stuck with Python < 3.4. Everybody else please use: pip install "borgbackup==1.0.0rc1" -- GPG ID: FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From mfseeker at gmail.com Wed Feb 10 15:48:10 2016 From: mfseeker at gmail.com (Stan Armstrong) Date: Wed, 10 Feb 2016 16:48:10 -0400 Subject: [Borgbackup] Backup to remote repository format Message-ID: <56BBA20A.1070607@gmail.com> OK, I have happily constructed some simple scripts to backup, check, and purge various local directories from one disk to a local repository on a different disk. They all work as expected, both manually and via crontabs. Today, I returned to the task of making borg backups to a different machine on my LAN. I was able to create a repository on the remote machine by following the example in the documentation; however, I cannot find an example of the proper format for a borg create command that will result in a new archive in the remote repository. (I do have borg installed on the remote machine, as can be seen from the success I had in creating an archive there from my main machine.) When I try "borg create -v barbara at Barbara-office/media/Backup-S/StoB::TV /media/2TB-1/dlna_library/1-TV", I get: "Repository /root/barbara at Barbara-office/media/Backup-S/StoB does not exist." I can't get rid of the initial "/root/". ["ssh barbara at Barbara-office" at my terminal takes me to "$ barbara at Barbara-office" as expected.] The proper way to do this must be described somewhere in the documentation, but I haven't been able to find it. Can you give me an example I can follow? From adrian.klaver at aklaver.com Wed Feb 10 16:17:12 2016 From: adrian.klaver at aklaver.com (Adrian Klaver) Date: Wed, 10 Feb 2016 13:17:12 -0800 Subject: [Borgbackup] Backup to remote repository format In-Reply-To: <56BBA20A.1070607@gmail.com> References: <56BBA20A.1070607@gmail.com> Message-ID: <56BBA8D8.10506@aklaver.com> On 02/10/2016 12:48 PM, Stan Armstrong wrote: > OK, I have happily constructed some simple scripts to backup, check, and > purge various local directories from one disk to a local repository on a > different disk. They all work as expected, both manually and via crontabs. > > Today, I returned to the task of making borg backups to a different > machine on my LAN. I was able to create a repository on the remote > machine by following the example in the documentation; however, I cannot > find an example of the proper format for a borg create command that will > result in a new archive in the remote repository. (I do have borg > installed on the remote machine, as can be seen from the success I had > in creating an archive there from my main machine.) > > When I try "borg create -v > barbara at Barbara-office/media/Backup-S/StoB::TV > /media/2TB-1/dlna_library/1-TV", I get: > > "Repository /root/barbara at Barbara-office/media/Backup-S/StoB does not > exist." I can't get rid of the initial "/root/". Something like this, assuming media/ is in your home directory: borg create -v barbara at Barbara-office:media/Backup-S/StoB::TV /media/2TB-1/dlna_library/1-TV otherwise if it is off / then borg create -v barbara at Barbara-office:/media/Backup-S/StoB::TV /media/2TB-1/dlna_library/1-TV > > ["ssh barbara at Barbara-office" at my terminal takes me to "$ > barbara at Barbara-office" as expected.] > > The proper way to do this must be described somewhere in the > documentation, but I haven't been able to find it. > > Can you give me an example I can follow? > > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup -- Adrian Klaver adrian.klaver at aklaver.com From mfseeker at gmail.com Fri Feb 12 10:26:35 2016 From: mfseeker at gmail.com (Stan Armstrong) Date: Fri, 12 Feb 2016 11:26:35 -0400 Subject: [Borgbackup] Find repository version number Message-ID: <56BDF9AB.5080501@gmail.com> I am running 1.0.0rc. I have several repositories on two different machines. How do I find the version that created a specific repository? From adrian.klaver at aklaver.com Fri Feb 12 10:40:36 2016 From: adrian.klaver at aklaver.com (Adrian Klaver) Date: Fri, 12 Feb 2016 07:40:36 -0800 Subject: [Borgbackup] Find repository version number In-Reply-To: <56BDF9AB.5080501@gmail.com> References: <56BDF9AB.5080501@gmail.com> Message-ID: <56BDFCF4.5060104@aklaver.com> On 02/12/2016 07:26 AM, Stan Armstrong wrote: > I am running 1.0.0rc. I have several repositories on two different > machines. How do I find the version that created a specific repository? Do not know how you find the program version, but if you look in the config file in each repo it will show you the repo version. > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup -- Adrian Klaver adrian.klaver at aklaver.com From mfseeker at gmail.com Fri Feb 12 12:39:35 2016 From: mfseeker at gmail.com (Stan Armstrong) Date: Fri, 12 Feb 2016 13:39:35 -0400 Subject: [Borgbackup] Find repository version number In-Reply-To: <56BDFCF4.5060104@aklaver.com> References: <56BDF9AB.5080501@gmail.com> <56BDFCF4.5060104@aklaver.com> Message-ID: <56BE18D7.6010007@gmail.com> Yes, but the config version (=1 for both a 0.30 and a 1.0.0rc repository) gives no help with the problem. Since there was a change in the default chunk size between program versions, knowing the version of a repository makes a difference in judging whether or not to replace a 0.30 repository with an entirely new, larger chunk size 1.0.0rc repository. I expected the borg info -v command to give me the program version. It doesn't. I thought perhaps the README file in the repository directory might have the program version. It doesn't. On 12/02/16 11:40 AM, Adrian Klaver wrote: > On 02/12/2016 07:26 AM, Stan Armstrong wrote: >> I am running 1.0.0rc. I have several repositories on two different >> machines. How do I find the version that created a specific repository? > > Do not know how you find the program version, but if you look in the > config file in each repo it will show you the repo version. > >> >> _______________________________________________ >> Borgbackup mailing list >> Borgbackup at python.org >> https://mail.python.org/mailman/listinfo/borgbackup > > From adrian.klaver at aklaver.com Fri Feb 12 13:04:54 2016 From: adrian.klaver at aklaver.com (Adrian Klaver) Date: Fri, 12 Feb 2016 10:04:54 -0800 Subject: [Borgbackup] Find repository version number In-Reply-To: <56BE18D7.6010007@gmail.com> References: <56BDF9AB.5080501@gmail.com> <56BDFCF4.5060104@aklaver.com> <56BE18D7.6010007@gmail.com> Message-ID: <56BE1EC6.2090307@aklaver.com> On 02/12/2016 09:39 AM, Stan Armstrong wrote: > Yes, but the config version (=1 for both a 0.30 and a 1.0.0rc > repository) gives no help with the problem. > > Since there was a change in the default chunk size between program > versions, knowing the version of a repository makes a difference in > judging whether or not to replace a 0.30 repository with an entirely > new, larger chunk size 1.0.0rc repository. If I follow the Changelog correctly: https://github.com/borgbackup/borg/blob/1.0.0rc1/docs/changes.rst what you want to do is going to be complicated if you are already using 1.0.0rc against an existing repo. Basically a repo can have multiple chunk size archives within it and that can result from the program version change or user changes to --chunker-params. So knowing what version created a repo is not necessarily an indication of the chunkiness of the archives within. > > I expected the borg info -v command to give me the program version. It > doesn't. I thought perhaps the README file in the repository directory > might have the program version. It doesn't. > > On 12/02/16 11:40 AM, Adrian Klaver wrote: >> On 02/12/2016 07:26 AM, Stan Armstrong wrote: >>> I am running 1.0.0rc. I have several repositories on two different >>> machines. How do I find the version that created a specific repository? >> >> Do not know how you find the program version, but if you look in the >> config file in each repo it will show you the repo version. >> >>> >>> _______________________________________________ >>> Borgbackup mailing list >>> Borgbackup at python.org >>> https://mail.python.org/mailman/listinfo/borgbackup >> >> > -- Adrian Klaver adrian.klaver at aklaver.com From billy at worldofbilly.com Fri Feb 12 13:05:50 2016 From: billy at worldofbilly.com (Billy Charlton) Date: Fri, 12 Feb 2016 10:05:50 -0800 Subject: [Borgbackup] Find repository version number In-Reply-To: <56BE1EC6.2090307@aklaver.com> References: <56BDF9AB.5080501@gmail.com> <56BDFCF4.5060104@aklaver.com> <56BE18D7.6010007@gmail.com> <56BE1EC6.2090307@aklaver.com> Message-ID: Perhaps the borg info command should describe the default chunk parameters for a repository, as well as the specific chunk parameters for an archive. that would be better than inferring chunk parameters from the code version. On Fri, Feb 12, 2016 at 10:04 AM, Adrian Klaver wrote: > On 02/12/2016 09:39 AM, Stan Armstrong wrote: > >> Yes, but the config version (=1 for both a 0.30 and a 1.0.0rc >> repository) gives no help with the problem. >> >> Since there was a change in the default chunk size between program >> versions, knowing the version of a repository makes a difference in >> judging whether or not to replace a 0.30 repository with an entirely >> new, larger chunk size 1.0.0rc repository. >> > > If I follow the Changelog correctly: > > https://github.com/borgbackup/borg/blob/1.0.0rc1/docs/changes.rst > > what you want to do is going to be complicated if you are already using > 1.0.0rc against an existing repo. Basically a repo can have multiple chunk > size archives within it and that can result from the program version change > or user changes to --chunker-params. So knowing what version created a repo > is not necessarily an indication of the chunkiness of the archives within. > > >> I expected the borg info -v command to give me the program version. It >> doesn't. I thought perhaps the README file in the repository directory >> might have the program version. It doesn't. >> >> On 12/02/16 11:40 AM, Adrian Klaver wrote: >> >>> On 02/12/2016 07:26 AM, Stan Armstrong wrote: >>> >>>> I am running 1.0.0rc. I have several repositories on two different >>>> machines. How do I find the version that created a specific repository? >>>> >>> >>> Do not know how you find the program version, but if you look in the >>> config file in each repo it will show you the repo version. >>> >>> >>>> _______________________________________________ >>>> Borgbackup mailing list >>>> Borgbackup at python.org >>>> https://mail.python.org/mailman/listinfo/borgbackup >>>> >>> >>> >>> >> > > -- > Adrian Klaver > adrian.klaver at aklaver.com > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nl at nachtgeist.net Fri Feb 12 17:25:21 2016 From: nl at nachtgeist.net (Daniel Reichelt) Date: Fri, 12 Feb 2016 23:25:21 +0100 Subject: [Borgbackup] Fwd: Re: Bug#814058: borgbackup: FTBFS on jessie: wrong build-deps and failing unittests In-Reply-To: <1522813231.4609931.1455209418370.JavaMail.yahoo@mail.yahoo.com> References: <1522813231.4609931.1455209418370.JavaMail.yahoo@mail.yahoo.com> Message-ID: <56BE5BD1.2070604@nachtgeist.net> Hi *, below you'll find a forwarded message from the Debian bugtracker regarding borgbackup. @ThomasWaldmann I'm not sure if your upstream ML's intended audience is borg users or developers. Anyway, in the sense of collaboration: would you deem it useful to subscribe this list to the ML for the borgbackup package of the Debian Package Tracking System? https://packages.qa.debian.org/b/borgbackup.html (bottom left) Cheers Daniel -------- Forwarded Message -------- Subject: Re: Bug#814058: borgbackup: FTBFS on jessie: wrong build-deps and failing unittests Date: Thu, 11 Feb 2016 16:50:18 +0000 (UTC) From: Gianfranco Costamagna Reply-To: Gianfranco Costamagna To: Daniel Reichelt , 814058 at bugs.debian.org <814058 at bugs.debian.org> Hi, >Hauler if you require manpower to test the packaging. help is always appreciated :) (we might disable the testsuite when the target release is not "sid") >Thomas Waldmann gave his ok to use the upstream ML for coordination "if >they behave" ;-)) they just need to subscribe on PTS the mail list and click on the confirmation link https://packages.qa.debian.org/b/borgbackup.html bottom left >I think in the long run, borg-dev and borg-users lists on l.d.o would be>helpful. sure! thanks Gianfranco From mfseeker at gmail.com Sat Feb 13 13:18:45 2016 From: mfseeker at gmail.com (Stan Armstrong) Date: Sat, 13 Feb 2016 14:18:45 -0400 Subject: [Borgbackup] [SOLVED-almost]Find repository version number In-Reply-To: References: <56BDF9AB.5080501@gmail.com> <56BDFCF4.5060104@aklaver.com> <56BE18D7.6010007@gmail.com> <56BE1EC6.2090307@aklaver.com> Message-ID: <56BF7385.8070102@gmail.com> The current borg version number can be found by intentionally issuing a command that will return a traceback error report. Is there a simple way to do this? On 12/02/16 02:05 PM, Billy Charlton wrote: > Perhaps the borg info command should describe the default chunk > parameters for a repository, as well as the specific chunk parameters > for an archive. that would be better than inferring chunk parameters > from the code version. > > On Fri, Feb 12, 2016 at 10:04 AM, Adrian Klaver > > wrote: > > On 02/12/2016 09:39 AM, Stan Armstrong wrote: > > Yes, but the config version (=1 for both a 0.30 and a 1.0.0rc > repository) gives no help with the problem. > > Since there was a change in the default chunk size between program > versions, knowing the version of a repository makes a > difference in > judging whether or not to replace a 0.30 repository with an > entirely > new, larger chunk size 1.0.0rc repository. > > > If I follow the Changelog correctly: > > https://github.com/borgbackup/borg/blob/1.0.0rc1/docs/changes.rst > > what you want to do is going to be complicated if you are already > using 1.0.0rc against an existing repo. Basically a repo can have > multiple chunk size archives within it and that can result from > the program version change or user changes to --chunker-params. So > knowing what version created a repo is not necessarily an > indication of the chunkiness of the archives within. > > > I expected the borg info -v command to give me the program > version. It > doesn't. I thought perhaps the README file in the repository > directory > might have the program version. It doesn't. > > On 12/02/16 11:40 AM, Adrian Klaver wrote: > > On 02/12/2016 07:26 AM, Stan Armstrong wrote: > > I am running 1.0.0rc. I have several repositories on > two different > machines. How do I find the version that created a > specific repository? > > > Do not know how you find the program version, but if you > look in the > config file in each repo it will show you the repo version. > > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > > > > > > > -- > Adrian Klaver > adrian.klaver at aklaver.com > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From adrian.klaver at aklaver.com Sat Feb 13 13:48:33 2016 From: adrian.klaver at aklaver.com (Adrian Klaver) Date: Sat, 13 Feb 2016 10:48:33 -0800 Subject: [Borgbackup] [SOLVED-almost]Find repository version number In-Reply-To: <56BF7385.8070102@gmail.com> References: <56BDF9AB.5080501@gmail.com> <56BDFCF4.5060104@aklaver.com> <56BE18D7.6010007@gmail.com> <56BE1EC6.2090307@aklaver.com> <56BF7385.8070102@gmail.com> Message-ID: <56BF7A81.4000708@aklaver.com> On 02/13/2016 10:18 AM, Stan Armstrong wrote: > The current borg version number can be found by intentionally issuing a > command that will return a traceback error report. Is there a simple way > to do this? aklaver at panda:~> borg -V borg 0.30.0 -- Adrian Klaver adrian.klaver at aklaver.com From mfseeker at gmail.com Sat Feb 13 14:18:03 2016 From: mfseeker at gmail.com (Stan Armstrong) Date: Sat, 13 Feb 2016 15:18:03 -0400 Subject: [Borgbackup] [SOLVED]Find repository version number In-Reply-To: <56BF7A81.4000708@aklaver.com> References: <56BDF9AB.5080501@gmail.com> <56BDFCF4.5060104@aklaver.com> <56BE18D7.6010007@gmail.com> <56BE1EC6.2090307@aklaver.com> <56BF7385.8070102@gmail.com> <56BF7A81.4000708@aklaver.com> Message-ID: <56BF816B.9070103@gmail.com> Got it! Thanks. On 13/02/16 02:48 PM, Adrian Klaver wrote: > On 02/13/2016 10:18 AM, Stan Armstrong wrote: >> The current borg version number can be found by intentionally issuing a >> command that will return a traceback error report. Is there a simple way >> to do this? > > aklaver at panda:~> borg -V > borg 0.30.0 > > > > From jungleboogie0 at gmail.com Sat Feb 13 14:44:39 2016 From: jungleboogie0 at gmail.com (jungle Boogie) Date: Sat, 13 Feb 2016 11:44:39 -0800 Subject: [Borgbackup] [SOLVED]Find repository version number In-Reply-To: <56BF816B.9070103@gmail.com> References: <56BDF9AB.5080501@gmail.com> <56BDFCF4.5060104@aklaver.com> <56BE18D7.6010007@gmail.com> <56BE1EC6.2090307@aklaver.com> <56BF7385.8070102@gmail.com> <56BF7A81.4000708@aklaver.com> <56BF816B.9070103@gmail.com> Message-ID: On 13 February 2016 at 11:18, Stan Armstrong wrote: > Got it! Thanks. yeah, generally if app -v doesn't work, try variations of: app -V app --version app --Version -- ------- inum: 883510009027723 sip: jungleboogie at sip2sip.info xmpp: jungle-boogie at jit.si From jungleboogie0 at gmail.com Sat Feb 13 14:59:55 2016 From: jungleboogie0 at gmail.com (jungle Boogie) Date: Sat, 13 Feb 2016 11:59:55 -0800 Subject: [Borgbackup] [SOLVED]Find repository version number In-Reply-To: References: <56BDF9AB.5080501@gmail.com> <56BDFCF4.5060104@aklaver.com> <56BE18D7.6010007@gmail.com> <56BE1EC6.2090307@aklaver.com> <56BF7385.8070102@gmail.com> <56BF7A81.4000708@aklaver.com> <56BF816B.9070103@gmail.com> Message-ID: On 13 February 2016 at 11:44, jungle Boogie wrote: > On 13 February 2016 at 11:18, Stan Armstrong wrote: >> Got it! Thanks. > > yeah, generally if app -v doesn't work, try variations of: > app -V > app --version > app --Version Also, typing just borg will print this: % borg usage: borg [-h] [-V] ... Borg - Deduplicated Backups optional arguments: -h, --help show this help message and exit -V, --version show version number and exit required arguments: serve start repository server process init initialize empty repository check verify repository change-passphrase change repository passphrase migrate-to-repokey migrate passphrase-mode repository to repokey create create backup extract extract archive contents rename rename archive delete delete archive list list archive or repository contents mount mount repository info show archive information break-lock break repository and cache locks prune prune archives upgrade upgrade repository format debug-dump-archive-items dump archive items (metadata) (debug) debug-get-obj get object from repository (debug) debug-put-obj put object to repository (debug) debug-delete-obj delete object from repository (debug) -- ------- inum: 883510009027723 sip: jungleboogie at sip2sip.info xmpp: jungle-boogie at jit.si From tw at waldmann-edv.de Sat Feb 13 19:02:20 2016 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Sun, 14 Feb 2016 01:02:20 +0100 Subject: [Borgbackup] Fwd: Re: Bug#814058: borgbackup: FTBFS on jessie: wrong build-deps and failing unittests In-Reply-To: <56BE5BD1.2070604@nachtgeist.net> References: <1522813231.4609931.1455209418370.JavaMail.yahoo@mail.yahoo.com> <56BE5BD1.2070604@nachtgeist.net> Message-ID: <56BFC40C.8020906@waldmann-edv.de> > I'm not sure if your upstream ML's intended audience is borg users or > developers. Well, as there is just 1 mailing list, I guess it's both. Although most development activities rather happen on github or irc. > Anyway, in the sense of collaboration: would you deem it > useful to subscribe this list to the ML for the borgbackup package of > the Debian Package Tracking System? > > https://packages.qa.debian.org/b/borgbackup.html > (bottom left) I don't know that this list is transporting usually, thus I can't say. >> Thomas Waldmann gave his ok to use the upstream ML for coordination "if >> they behave" ;-)) If it's people technically discussing about borgbackup (in debian), it's ok. Rants about how bad python and the world is in general are off-topic. Also, policy and other non-technical discussions are off-topic. -- GPG ID: FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From mh+borgbackup at zugschlus.de Tue Feb 16 07:22:23 2016 From: mh+borgbackup at zugschlus.de (Marc Haber) Date: Tue, 16 Feb 2016 13:22:23 +0100 Subject: [Borgbackup] How to get the old -v behavior back? Message-ID: <20160216122223.GK26042@torres.zugschlus.de> Hi, in borgbackup 0.29 and earlier, borg create -v listed the files being processed on standard output, one per line, scrolling, so that one could scroll back through the log and see what happened. Borgbackup 0.30 does not list the file names any more, since the meaning of the --verbose switch was changed, and --verbose --progess now avoids scrolling by tabbing back on the same line and trying to pretty-print the output, losing the ability to scroll back in the process. I have tried a number of command line switch combinations to get the old behavior back, but have not succeeded. Is there a possibility to get the old behavior back? If not, I will file an issue about that to either add another command line switch for the old behavior or to detect whether standard output is a terminal and if not, do not try fancy games with the terminal. Thanks for helping! Greetings Marc -- ----------------------------------------------------------------------------- Marc Haber | "I don't trust Computers. They | Mailadresse im Header Leimen, Germany | lose things." Winona Ryder | Fon: *49 6224 1600402 Nordisch by Nature | How to make an American Quilt | Fax: *49 6224 1600421 From adrian.klaver at aklaver.com Tue Feb 16 10:11:15 2016 From: adrian.klaver at aklaver.com (Adrian Klaver) Date: Tue, 16 Feb 2016 07:11:15 -0800 Subject: [Borgbackup] How to get the old -v behavior back? In-Reply-To: <20160216122223.GK26042@torres.zugschlus.de> References: <20160216122223.GK26042@torres.zugschlus.de> Message-ID: <56C33C13.9070806@aklaver.com> On 02/16/2016 04:22 AM, Marc Haber wrote: > Hi, > > in borgbackup 0.29 and earlier, borg create -v listed the files being > processed on standard output, one per line, scrolling, so that one > could scroll back through the log and see what happened. > > Borgbackup 0.30 does not list the file names any more, since the > meaning of the --verbose switch was changed, and --verbose --progess > now avoids scrolling by tabbing back on the same line and trying to > pretty-print the output, losing the ability to scroll back in the > process. > > I have tried a number of command line switch combinations to get the > old behavior back, but have not succeeded. > > Is there a possibility to get the old behavior back? Did you try?: https://github.com/borgbackup/borg/blob/0.30.1/docs/changes.rst "For borg create, you need --list (additionally to -v) to see the long file list (was needed so you can have e.g. --stats alone without the long list)" > > If not, I will file an issue about that to either add another command > line switch for the old behavior or to detect whether standard output > is a terminal and if not, do not try fancy games with the terminal. > > Thanks for helping! > > Greetings > Marc > -- Adrian Klaver adrian.klaver at aklaver.com From mh+borgbackup at zugschlus.de Tue Feb 16 10:24:21 2016 From: mh+borgbackup at zugschlus.de (Marc Haber) Date: Tue, 16 Feb 2016 16:24:21 +0100 Subject: [Borgbackup] How to get the old -v behavior back? In-Reply-To: <56C33C13.9070806@aklaver.com> References: <20160216122223.GK26042@torres.zugschlus.de> <56C33C13.9070806@aklaver.com> Message-ID: <20160216152421.GN26042@torres.zugschlus.de> On Tue, Feb 16, 2016 at 07:11:15AM -0800, Adrian Klaver wrote: > Did you try?: > > https://github.com/borgbackup/borg/blob/0.30.1/docs/changes.rst > > "For borg create, you need --list (additionally to -v) to see the long file > list (was needed so you can have e.g. --stats alone without the long list)" For some reason, no. That's what I wanted. Maybe I got misled by the man page, but on second view the docs are clear. Greetings Marc -- ----------------------------------------------------------------------------- Marc Haber | "I don't trust Computers. They | Mailadresse im Header Leimen, Germany | lose things." Winona Ryder | Fon: *49 6224 1600402 Nordisch by Nature | How to make an American Quilt | Fax: *49 6224 1600421 From tw at waldmann-edv.de Sun Feb 28 11:12:03 2016 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Sun, 28 Feb 2016 17:12:03 +0100 Subject: [Borgbackup] borgbackup 1.0.0rc2 released Message-ID: <56D31C53.9000704@waldmann-edv.de> https://github.com/borgbackup/borg/releases/tag/1.0.0rc2 Please help testing to find the bugs before 1.0.0 release. Currently it looks like there are no major problems, so a final release might come in 1 or 2 weeks. Cheers, Thomas --- GPG ID: FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From jungleboogie0 at gmail.com Sun Feb 28 15:03:05 2016 From: jungleboogie0 at gmail.com (jungle Boogie) Date: Sun, 28 Feb 2016 12:03:05 -0800 Subject: [Borgbackup] borgbackup 1.0.0rc2 released In-Reply-To: <56D31C53.9000704@waldmann-edv.de> References: <56D31C53.9000704@waldmann-edv.de> Message-ID: Hi Thomas, On Feb 28, 2016 8:20 AM, "Thomas Waldmann" wrote: > > https://github.com/borgbackup/borg/releases/tag/1.0.0rc2 > > Please help testing to find the bugs before 1.0.0 release. > Does this version contain libfuse for use with the mount command? > Currently it looks like there are no major problems, so a final release might come in 1 or 2 weeks. > > Cheers, > > Thomas > > --- > > GPG ID: FAF7B393 > GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 Sent from my iPhone 7.1 -------------- next part -------------- An HTML attachment was scrubbed... URL: From tw at waldmann-edv.de Sun Feb 28 15:07:43 2016 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Sun, 28 Feb 2016 21:07:43 +0100 Subject: [Borgbackup] borgbackup 1.0.0rc2 released In-Reply-To: References: <56D31C53.9000704@waldmann-edv.de> Message-ID: <56D3538F.4060001@waldmann-edv.de> > Does this version contain libfuse for use with the mount command? Yes, the binaries should work for "borg mount". -- GPG ID: FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From tw at waldmann-edv.de Sat Mar 5 16:57:01 2016 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Sat, 5 Mar 2016 22:57:01 +0100 Subject: [Borgbackup] borgbackup 1.0.0 released Message-ID: <56DB562D.8070005@waldmann-edv.de> https://github.com/borgbackup/borg/releases/tag/1.0.0 Finally, the first major release - with major changes, so please read the changelog before upgrading: https://github.com/borgbackup/borg/blob/1.0.0/docs/changes.rst Cheers, Thomas --- GPG ID: FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From jungleboogie0 at gmail.com Sat Mar 5 19:12:13 2016 From: jungleboogie0 at gmail.com (jungle Boogie) Date: Sat, 5 Mar 2016 16:12:13 -0800 Subject: [Borgbackup] borgbackup 1.0.0 released In-Reply-To: <56DB562D.8070005@waldmann-edv.de> References: <56DB562D.8070005@waldmann-edv.de> Message-ID: On Mar 5, 2016 1:57 PM, "Thomas Waldmann" wrote: > > https://github.com/borgbackup/borg/releases/tag/1.0.0 > > Finally, the first major release - with major changes, so please read the changelog before upgrading: > Thanks for all your hard work put into this. I'll update my systems and report any problems! > https://github.com/borgbackup/borg/blob/1.0.0/docs/changes.rst > > Cheers, > > Thomas > > --- > > GPG ID: FAF7B393 > GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup -------------- next part -------------- An HTML attachment was scrubbed... URL: From david_pearce at wycliffe.org Mon Mar 7 09:17:43 2016 From: david_pearce at wycliffe.org (Pearce, David) Date: Mon, 7 Mar 2016 09:17:43 -0500 Subject: [Borgbackup] Larger setups Message-ID: I've read some about BorgBackup and am very intrigued. The company I work for wants to switch from tape-based backups to file-based backups. We're building a 140 TB NAS/SAN already for this purpose. My question is, I've got about 100 Linux servers to backup. Does BorgBackup scale that large? Does it become a configuration/maintenance nightmare when it gets that big? Can I do concurrent backups of remote clients? Thanks. David Pearce Network Systems Administrator Wycliffe Bible Translators -------------- next part -------------- An HTML attachment was scrubbed... URL: From tw at waldmann-edv.de Mon Mar 7 12:16:48 2016 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Mon, 7 Mar 2016 18:16:48 +0100 Subject: [Borgbackup] Larger setups In-Reply-To: References: Message-ID: <56DDB780.5050600@waldmann-edv.de> > The company I work for wants to switch from tape-based backups to > file-based backups. We're building a 140 TB NAS/SAN already for this > purpose. I am not aware whether borg was tested on that scale yet, maybe some other reader can comment on it. I did personal tests with ~ 7.5TB of data. And I'ld love to get feedback from you after your tests. :) > My question is, I've got about 100 Linux servers to backup. Does > BorgBackup scale that large? I think it should not be a problem, if you do some careful planning and scripting. > Does it become a configuration/maintenance nightmare when it gets that big? Aside from the commandline parameters and some env vars, there is no configuration. > Can I do concurrent backups of remote clients? Not to the same repository at the same time. But considering that amount of servers and data, using multiple repositories is a good idea anyway. Also, assuming the backup data goes over your network, running too many backups in parallel would likely need more bandwidth than your network has available anyway. So, what I'ld do for a test setup is: - read the docs once or twice, there are useful / important hints in there - divide all your servers into smaller sets of servers. servers that share a lot of identical or similar files go into same set. the servers in the same set will share 1 repository and will have to run backups one after each other (you can tweak --lock-wait so one waits until the previous one has finished) - try to achieve sets that have less than 10-20 TB of data. The size of the chunks index is proportional to the amount of unique data chunks stored into the repository (and that is usually proportional to the size of your data, except if you have a lot of duplication inside your files). The index is stored on disk and loaded fully into RAM when borg is running, so having that index stay within a reasonable size (that is below your available RAM) is a good idea. The "internals" part of the docs have a formular to estimate the index size. Be aware that chunks index will need regular rebuilding/resyncing if you backup multiple servers into same repository. That will cost some time and cpu, but if the content is similar, it will save you backup space. If you want to avoid the rebuid/resync, you'll need 1 repository per machine (in that case, there will be no inter-machine deduplication, but still usually a lot of historical dedup). - if a server has a unusually high count of files or high amount of data, use a separate repo just for that server. - if a server has a small count of files and a small amount of data, you can put it into another set of servers. When you run the first backup of the server(s), it will transfer ALL data into the repositories. Depending on your setup, that might take a while. The 2nd+ backup will then work a lot faster and only transfer the changed / added data. To tune the excludes, you may want to use -v --dry-run --list and review the file list it outputs (whether you have all you want, but no unneeded files/dirs). If you need help, just ask. -- GPG ID: FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From sitaramc at gmail.com Sat Mar 12 02:23:59 2016 From: sitaramc at gmail.com (Sitaram Chamarty) Date: Sat, 12 Mar 2016 12:53:59 +0530 Subject: [Borgbackup] setting default compression Message-ID: <56E3C40F.806@gmail.com> Hi, I discovered borg a day or so ago and it seems really great; thanks for a very, very, nice piece of software. Are there any plans to add something like, say, BORG_COMPRESSION, to set default compression? I expect most of my use of borg will be scripted, but the command line is so easy compared to duplicity, that I am luxuriating in the ability to do ad hoc backups of "just this directory" whenever I want to, and it would be nice not to have to remember to add '-C ...' each time. (Yes I could alias the whole command, and I probably will, but thought I'd ask anyway.) regards sitaram From adrian.klaver at aklaver.com Sat Mar 12 09:37:23 2016 From: adrian.klaver at aklaver.com (Adrian Klaver) Date: Sat, 12 Mar 2016 06:37:23 -0800 Subject: [Borgbackup] setting default compression In-Reply-To: <56E3C40F.806@gmail.com> References: <56E3C40F.806@gmail.com> Message-ID: <56E429A3.3010504@aklaver.com> On 03/11/2016 11:23 PM, Sitaram Chamarty wrote: > Hi, > > I discovered borg a day or so ago and it seems really great; thanks for > a very, very, nice piece of software. > > Are there any plans to add something like, say, BORG_COMPRESSION, to set > default compression? http://borgbackup.readthedocs.org/en/stable/faq.html#why-is-my-backup-bigger-than-with-attic-why-doesn-t-project-name-do-compression-by-default > > I expect most of my use of borg will be scripted, but the command line > is so easy compared to duplicity, that I am luxuriating in the ability > to do ad hoc backups of "just this directory" whenever I want to, and it > would be nice not to have to remember to add '-C ...' each time. > > (Yes I could alias the whole command, and I probably will, but thought > I'd ask anyway.) > > regards > sitaram > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > -- Adrian Klaver adrian.klaver at aklaver.com From sitaramc at gmail.com Sat Mar 12 10:57:33 2016 From: sitaramc at gmail.com (Sitaram Chamarty) Date: Sat, 12 Mar 2016 21:27:33 +0530 Subject: [Borgbackup] environment variable to set compression? In-Reply-To: <56E429A3.3010504@aklaver.com> References: <56E3C40F.806@gmail.com> <56E429A3.3010504@aklaver.com> Message-ID: <56E43C6D.2010708@gmail.com> On 12/03/16 20:07, Adrian Klaver wrote: > On 03/11/2016 11:23 PM, Sitaram Chamarty wrote: >> Hi, >> >> I discovered borg a day or so ago and it seems really great; thanks for >> a very, very, nice piece of software. >> >> Are there any plans to add something like, say, BORG_COMPRESSION, to set >> default compression? > > http://borgbackup.readthedocs.org/en/stable/faq.html#why-is-my-backup-bigger-than-with-attic-why-doesn-t-project-name-do-compression-by-default Let me rephrase my question without using the word "default" (I've changed the subject line also). Is there a possibility - in the future - that we can set our compression choice using an environment variable - like, say BORG_COMPRESSION - instead of using the command line flag. (BORG_REPO is an example of an env var that is similar). regards sitaram > >> >> I expect most of my use of borg will be scripted, but the command line >> is so easy compared to duplicity, that I am luxuriating in the ability >> to do ad hoc backups of "just this directory" whenever I want to, and it >> would be nice not to have to remember to add '-C ...' each time. >> >> (Yes I could alias the whole command, and I probably will, but thought >> I'd ask anyway.) >> >> regards >> sitaram >> _______________________________________________ >> Borgbackup mailing list >> Borgbackup at python.org >> https://mail.python.org/mailman/listinfo/borgbackup >> > > From adrian.klaver at aklaver.com Sat Mar 12 13:49:47 2016 From: adrian.klaver at aklaver.com (Adrian Klaver) Date: Sat, 12 Mar 2016 10:49:47 -0800 Subject: [Borgbackup] environment variable to set compression? In-Reply-To: <56E43C6D.2010708@gmail.com> References: <56E3C40F.806@gmail.com> <56E429A3.3010504@aklaver.com> <56E43C6D.2010708@gmail.com> Message-ID: <56E464CB.9000403@aklaver.com> On 03/12/2016 07:57 AM, Sitaram Chamarty wrote: > > On 12/03/16 20:07, Adrian Klaver wrote: >> On 03/11/2016 11:23 PM, Sitaram Chamarty wrote: >>> Hi, >>> >>> I discovered borg a day or so ago and it seems really great; thanks for >>> a very, very, nice piece of software. >>> >>> Are there any plans to add something like, say, BORG_COMPRESSION, to set >>> default compression? >> >> http://borgbackup.readthedocs.org/en/stable/faq.html#why-is-my-backup-bigger-than-with-attic-why-doesn-t-project-name-do-compression-by-default > > Let me rephrase my question without using the word "default" (I've > changed the subject line also). > > Is there a possibility - in the future - that we can set our compression > choice using an environment variable - like, say BORG_COMPRESSION - > instead of using the command line flag. (BORG_REPO is an example of an > env var that is similar). I think it points to a need for configuration files for backup jobs: https://github.com/borgbackup/borg/issues/315 > > regards > sitaram > >> >>> >>> I expect most of my use of borg will be scripted, but the command line >>> is so easy compared to duplicity, that I am luxuriating in the ability >>> to do ad hoc backups of "just this directory" whenever I want to, and it >>> would be nice not to have to remember to add '-C ...' each time. >>> >>> (Yes I could alias the whole command, and I probably will, but thought >>> I'd ask anyway.) >>> >>> regards >>> sitaram >>> _______________________________________________ >>> Borgbackup mailing list >>> Borgbackup at python.org >>> https://mail.python.org/mailman/listinfo/borgbackup >>> >> >> > -- Adrian Klaver adrian.klaver at aklaver.com From tw at waldmann-edv.de Sat Mar 12 15:50:25 2016 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Sat, 12 Mar 2016 21:50:25 +0100 Subject: [Borgbackup] setting default compression In-Reply-To: <56E3C40F.806@gmail.com> References: <56E3C40F.806@gmail.com> Message-ID: <56E48111.9020100@waldmann-edv.de> > Are there any plans to add something like, say, BORG_COMPRESSION, to set > default compression? Maybe, maybe not. A configuration file for the defaults might be another option. Having some defaults stored in the repo are another option. > I expect most of my use of borg will be scripted, but the command line > is so easy compared to duplicity, that I am luxuriating in the ability > to do ad hoc backups of "just this directory" whenever I want to, and it > would be nice not to have to remember to add '-C ...' each time. Until there is another way, you can always write a little wrapper script that adds these options. -- GPG ID: FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From sitaramc at gmail.com Sun Mar 13 01:43:45 2016 From: sitaramc at gmail.com (Sitaram Chamarty) Date: Sun, 13 Mar 2016 12:13:45 +0530 Subject: [Borgbackup] setting default compression In-Reply-To: <56E48111.9020100@waldmann-edv.de> References: <56E3C40F.806@gmail.com> <56E48111.9020100@waldmann-edv.de> Message-ID: <56E50C21.8060500@gmail.com> On 13/03/16 02:20, Thomas Waldmann wrote: >> Are there any plans to add something like, say, BORG_COMPRESSION, to set >> default compression? > > Maybe, maybe not. A configuration file for the defaults might be > another option. Having some defaults stored in the repo are another > option. that's fine too; a global one would be really nice, or a repo-local config could be populated with the chosen algorithm at 'init' time. >> I expect most of my use of borg will be scripted, but the command line >> is so easy compared to duplicity, that I am luxuriating in the ability >> to do ad hoc backups of "just this directory" whenever I want to, and it >> would be nice not to have to remember to add '-C ...' each time. > > Until there is another way, you can always write a little wrapper > script that adds these options. Sure, in the short term that's fine. Thanks once again for an awesome tool! regards sitaram From sitaramc at gmail.com Sun Mar 13 11:51:55 2016 From: sitaramc at gmail.com (Sitaram Chamarty) Date: Sun, 13 Mar 2016 21:21:55 +0530 Subject: [Borgbackup] question about inconsistent use of compression Message-ID: <56E58C9B.6020905@gmail.com> Hi, This is probably a stupid question, but are there any caveats to running some 'create's with and some without '-C'? Or even, with different compressions at different times? Does it affect dedup in any way, for instance? I suspect not but thought I'd ask anyway. regards sitaram From adrian.klaver at aklaver.com Sun Mar 13 11:53:21 2016 From: adrian.klaver at aklaver.com (Adrian Klaver) Date: Sun, 13 Mar 2016 08:53:21 -0700 Subject: [Borgbackup] question about inconsistent use of compression In-Reply-To: <56E58C9B.6020905@gmail.com> References: <56E58C9B.6020905@gmail.com> Message-ID: <56E58CF1.4040709@aklaver.com> On 03/13/2016 08:51 AM, Sitaram Chamarty wrote: > Hi, > > This is probably a stupid question, but are there any caveats to running some 'create's with and some without '-C'? Or even, with different compressions at different times? > > Does it affect dedup in any way, for instance? I suspect not but thought I'd ask anyway. http://borgbackup.readthedocs.org/en/stable/internals.html#compression "Compression is applied after deduplication, thus using different compression methods in one repo does not influence deduplication." > > regards > sitaram > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > -- Adrian Klaver adrian.klaver at aklaver.com From sitaramc at gmail.com Sun Mar 13 12:20:22 2016 From: sitaramc at gmail.com (Sitaram Chamarty) Date: Sun, 13 Mar 2016 21:50:22 +0530 Subject: [Borgbackup] question about inconsistent use of compression In-Reply-To: <56E58CF1.4040709@aklaver.com> References: <56E58C9B.6020905@gmail.com> <56E58CF1.4040709@aklaver.com> Message-ID: <56E59346.4010104@gmail.com> On 13/03/16 21:23, Adrian Klaver wrote: > On 03/13/2016 08:51 AM, Sitaram Chamarty wrote: >> Hi, >> >> This is probably a stupid question, but are there any caveats to running some 'create's with and some without '-C'? Or even, with different compressions at different times? >> >> Does it affect dedup in any way, for instance? I suspect not but thought I'd ask anyway. > > > http://borgbackup.readthedocs.org/en/stable/internals.html#compression > > "Compression is applied after deduplication, thus using different compression methods in one repo does not influence deduplication." Thanks! I seem to have missed the internals doc when I was looking but in hindsight it is obviously the first place I should have looked for such a question. /me feels silly regards sitaram From sitaramc at gmail.com Sun Mar 13 21:24:09 2016 From: sitaramc at gmail.com (Sitaram Chamarty) Date: Mon, 14 Mar 2016 06:54:09 +0530 Subject: [Borgbackup] sending --list output to stdout instead of stderr Message-ID: <56E612B9.6070003@gmail.com> Hello, When using "borg create -v -s -p --list ...", would it be possible to send the output of the "--list" to stdout instead of stderr? I would like to do some post-processing on the list output, while also seeing progress (-v, -s, -p) on the screen as it runs. regards sitaram PS: please let me know if the correct protocol here is to raise a github issue directly instead of asking on the list first. From tech at age.bdeb.qc.ca Mon Mar 14 15:09:00 2016 From: tech at age.bdeb.qc.ca (Technicien Informatique) Date: Mon, 14 Mar 2016 15:09:00 -0400 Subject: [Borgbackup] Migrating from 0.3 to 1.0 with passphrase Message-ID: Hi! I'm currently trying to upgrade two of our repos that were made with the 0.3 version and encrypted using passphrases. As stated in the changelog, I'm using "migrate-to-repokey" to do it, but here's what I get: $ borg migrate-to-repokey backup Enter passphrase for file:///mnt/backup: Enter passphrase for file:///mnt/backup: Enter passphrase for file:///mnt/backup: exceeded the maximum password retries I know my password is right since I'm able to use the "list" option: $ borg list backup/ Enter passphrase for key /mnt/backup: boisdeboulogne-2016-01-31 Sun, 2016-01-31 03:18:12 boisdeboulogne-2016-02-14 Sun, 2016-02-14 03:21:28 boisdeboulogne-2016-02-21 Sun, 2016-02-21 03:20:28 boisdeboulogne-2016-02-23 Tue, 2016-02-23 03:21:41 boisdeboulogne-2016-03-06 Sun, 2016-03-06 03:07:41 boisdeboulogne-2016-03-08 Tue, 2016-03-08 03:08:53 boisdeboulogne-2016-03-09 Wed, 2016-03-09 03:08:39 boisdeboulogne-2016-03-10 Thu, 2016-03-10 03:08:27 boisdeboulogne-2016-03-11 Fri, 2016-03-11 03:16:13 boisdeboulogne-2016-03-12 Sat, 2016-03-12 03:10:10 boisdeboulogne-2016-03-13 Sun, 2016-03-13 03:08:49 boisdeboulogne-2016-03-14 Mon, 2016-03-14 03:08:08 I am doing something wrong? This is the case for two different repos using different password. -- Louis-Philippe V?ronneau - Informaticien AGEBdeB - Association G?n?rale ?tudiante de Bois-de-Boulogne 10555 av. de Bois-de-Boulogne, local S-258 Montr?al, (Qu?bec), H4N 1L4 T?l?phone: (514) 332-3000 #7580 Site web: http://agebdeb.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From tech at age.bdeb.qc.ca Mon Mar 14 15:38:24 2016 From: tech at age.bdeb.qc.ca (Technicien Informatique) Date: Mon, 14 Mar 2016 15:38:24 -0400 Subject: [Borgbackup] Migrating from 0.3 to 1.0 with passphrase In-Reply-To: References: Message-ID: Hmm, I'm sorry for the noise, I just realized I was already using the repokey method. 2016-03-14 15:09 GMT-04:00 Technicien Informatique : > Hi! > > I'm currently trying to upgrade two of our repos that were made with the > 0.3 version and encrypted using passphrases. > > As stated in the changelog, I'm using "migrate-to-repokey" to do it, but > here's what I get: > > $ borg migrate-to-repokey backup > > Enter passphrase for file:///mnt/backup: > Enter passphrase for file:///mnt/backup: > Enter passphrase for file:///mnt/backup: > exceeded the maximum password retries > > I know my password is right since I'm able to use the "list" option: > > $ borg list backup/ > Enter passphrase for key /mnt/backup: > boisdeboulogne-2016-01-31 Sun, 2016-01-31 03:18:12 > boisdeboulogne-2016-02-14 Sun, 2016-02-14 03:21:28 > boisdeboulogne-2016-02-21 Sun, 2016-02-21 03:20:28 > boisdeboulogne-2016-02-23 Tue, 2016-02-23 03:21:41 > boisdeboulogne-2016-03-06 Sun, 2016-03-06 03:07:41 > boisdeboulogne-2016-03-08 Tue, 2016-03-08 03:08:53 > boisdeboulogne-2016-03-09 Wed, 2016-03-09 03:08:39 > boisdeboulogne-2016-03-10 Thu, 2016-03-10 03:08:27 > boisdeboulogne-2016-03-11 Fri, 2016-03-11 03:16:13 > boisdeboulogne-2016-03-12 Sat, 2016-03-12 03:10:10 > boisdeboulogne-2016-03-13 Sun, 2016-03-13 03:08:49 > boisdeboulogne-2016-03-14 Mon, 2016-03-14 03:08:08 > > I am doing something wrong? This is the case for two different repos using > different password. > > > -- > Louis-Philippe V?ronneau - Informaticien > AGEBdeB - Association G?n?rale ?tudiante de Bois-de-Boulogne > 10555 av. de Bois-de-Boulogne, local S-258 > Montr?al, (Qu?bec), H4N 1L4 > T?l?phone: (514) 332-3000 #7580 > Site web: http://agebdeb.org > > > -- Louis-Philippe V?ronneau - Informaticien AGEBdeB - Association G?n?rale ?tudiante de Bois-de-Boulogne 10555 av. de Bois-de-Boulogne, local S-258 Montr?al, (Qu?bec), H4N 1L4 T?l?phone: (514) 332-3000 #7580 Site web: http://agebdeb.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From sitaramc at gmail.com Tue Mar 15 02:24:11 2016 From: sitaramc at gmail.com (Sitaram Chamarty) Date: Tue, 15 Mar 2016 11:54:11 +0530 Subject: [Borgbackup] converting a repo from 'repokey' to 'keyfile' Message-ID: <56E7AA8B.90003@gmail.com> Hi, Is there a way to migrate a repo from repokey to keyfile? regards sitaram From david_pearce at wycliffe.org Tue Mar 15 10:21:49 2016 From: david_pearce at wycliffe.org (Pearce, David) Date: Tue, 15 Mar 2016 10:21:49 -0400 Subject: [Borgbackup] Pulling from a remote server Message-ID: Can I use BorgBackup to pull from a remote server? So I have a box with Borg and the repo on it. I want to backup a remote box. Can I do something like this? borg create /mnt/Mirror/Borg3::rh6-test2 borgbu at rh6-test2:/ David Pearce Network Systems Administrator Wycliffe Bible Translators 407-243-5008 -------------- next part -------------- An HTML attachment was scrubbed... URL: From rumpelsepp at sevenbyte.org Tue Mar 15 16:08:16 2016 From: rumpelsepp at sevenbyte.org (Stefan Tatschner) Date: Tue, 15 Mar 2016 21:08:16 +0100 Subject: [Borgbackup] Pulling from a remote server In-Reply-To: References: Message-ID: <56E86BB0.7080403@sevenbyte.org> On 15.03.2016 15:21, Pearce, David wrote: > Can I use BorgBackup to pull from a remote server? Currently that is not possible: https://github.com/borgbackup/borg/issues/36 Stefan From tw at waldmann-edv.de Tue Mar 15 20:23:26 2016 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Wed, 16 Mar 2016 01:23:26 +0100 Subject: [Borgbackup] sending --list output to stdout instead of stderr In-Reply-To: <56E612B9.6070003@gmail.com> References: <56E612B9.6070003@gmail.com> Message-ID: <56E8A77E.80606@waldmann-edv.de> > When using "borg create -v -s -p --list ...", would it be possible to > send the output of the "--list" to stdout instead of stderr? > > I would like to do some post-processing on the list output, while also > seeing progress (-v, -s, -p) on the screen as it runs. We usually use stderr for log output because stdout might be used for connecting borg to other tools via a pipe (and it will transport file data then), like: borg extract --stdout ... | dd of=/dev/sdx stats output is also logged as I expect many people wanting to have that in their backup logs. So it's currently not possible to write this to different files / channels. I also do not expect that functionality for the future, except if someone comes up with an easy system to configure and control that. Deciding how to handle output is rather difficult (when to use print, when to use print >>stderr, when to use log. There's a little piece about it in the docs, but even with that it is not always easily decided. > PS: please let me know if the correct protocol here is to raise a github > issue directly instead of asking on the list first. There is no rule that requires you to use the mailing list first. It might make sense though if you just have a question or want to discuss something. There is also the IRC channel for realtime / near realtime communication. -- GPG ID: FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From tw at waldmann-edv.de Tue Mar 15 20:26:21 2016 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Wed, 16 Mar 2016 01:26:21 +0100 Subject: [Borgbackup] converting a repo from 'repokey' to 'keyfile' In-Reply-To: <56E7AA8B.90003@gmail.com> References: <56E7AA8B.90003@gmail.com> Message-ID: <56E8A82D.8080409@waldmann-edv.de> > Is there a way to migrate a repo from repokey to keyfile? There is no code for that yet, but you can maybe do it manually (be careful, make backups of files you modify). https://github.com/borgbackup/borg/issues/510 (sitaram: sorry for duplicate, sent the first mail just to you privately, not to the list) -- GPG ID: FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From sitaramc at gmail.com Tue Mar 15 21:17:22 2016 From: sitaramc at gmail.com (Sitaram Chamarty) Date: Wed, 16 Mar 2016 06:47:22 +0530 Subject: [Borgbackup] sending --list output to stdout instead of stderr In-Reply-To: <56E8A77E.80606@waldmann-edv.de> References: <56E612B9.6070003@gmail.com> <56E8A77E.80606@waldmann-edv.de> Message-ID: <56E8B422.1050809@gmail.com> On 16/03/16 05:53, Thomas Waldmann wrote: >> When using "borg create -v -s -p --list ...", would it be possible to >> send the output of the "--list" to stdout instead of stderr? >> >> I would like to do some post-processing on the list output, while also >> seeing progress (-v, -s, -p) on the screen as it runs. > > We usually use stderr for log output because stdout might be used for connecting borg to other tools via a pipe (and it will transport file data then), like: > > borg extract --stdout ... | dd of=/dev/sdx > > stats output is also logged as I expect many people wanting to have that in their backup logs. So it's currently not possible to write this to different files / channels. Problem solved. Using the logging conf fixed things for me. Now I get the "-p" I want to see on screen while still getting the list of files to process later. I don't see the summary stats on screen anymore but I can do a "tail -15" or so of the logfile if I need that. > I also do not expect that functionality for the future, except if someone comes up with an easy system to configure and control that. Naah; you already have it. I just handn't looked at the logging stuff, not being familiar with that config file format at all. regards sitaram From sitaramc at gmail.com Tue Mar 15 21:18:54 2016 From: sitaramc at gmail.com (Sitaram Chamarty) Date: Wed, 16 Mar 2016 06:48:54 +0530 Subject: [Borgbackup] converting a repo from 'repokey' to 'keyfile' In-Reply-To: <56E8A82D.8080409@waldmann-edv.de> References: <56E7AA8B.90003@gmail.com> <56E8A82D.8080409@waldmann-edv.de> Message-ID: <56E8B47E.1070801@gmail.com> On 16/03/16 05:56, Thomas Waldmann wrote: >> Is there a way to migrate a repo from repokey to keyfile? > > There is no code for that yet, but you can maybe do it manually (be careful, make backups of files you modify). > > https://github.com/borgbackup/borg/issues/510 Thanks. I simply took the backups again, but it's nice to know it's on the roadmap for some time. > (sitaram: sorry for duplicate, sent the first mail just to you privately, not to the list) no problem :) regards sitaram From anarcat at debian.org Tue Mar 15 21:32:14 2016 From: anarcat at debian.org (Antoine =?utf-8?Q?Beaupr=C3=A9?=) Date: Tue, 15 Mar 2016 21:32:14 -0400 Subject: [Borgbackup] sending --list output to stdout instead of stderr In-Reply-To: <56E8B422.1050809@gmail.com> References: <56E612B9.6070003@gmail.com> <56E8A77E.80606@waldmann-edv.de> <56E8B422.1050809@gmail.com> Message-ID: <87shzr9pi9.fsf@angela.anarcat.ath.cx> On 2016-03-15 21:17:22, Sitaram Chamarty wrote: > On 16/03/16 05:53, Thomas Waldmann wrote: >>> When using "borg create -v -s -p --list ...", would it be possible to >>> send the output of the "--list" to stdout instead of stderr? >>> >>> I would like to do some post-processing on the list output, while also >>> seeing progress (-v, -s, -p) on the screen as it runs. >> >> We usually use stderr for log output because stdout might be used for connecting borg to other tools via a pipe (and it will transport file data then), like: >> >> borg extract --stdout ... | dd of=/dev/sdx >> >> stats output is also logged as I expect many people wanting to have that in their backup logs. So it's currently not possible to write this to different files / channels. > > Problem solved. Using the logging conf fixed things for me. Now I get > the "-p" I want to see on screen while still getting the list of files > to process later. I don't see the summary stats on screen anymore but I > can do a "tail -15" or so of the logfile if I need that. Can you show us what configuration you used? I'd be great to have samples for that in the documentation... a. -- The steel horse fills a gap in modern life, it is an answer not only to its needs, but also to its aspirations. It's quite certainly here to stay. - Le V?locip?de Illustr?, 1869 From sitaramc at gmail.com Tue Mar 15 22:09:50 2016 From: sitaramc at gmail.com (Sitaram Chamarty) Date: Wed, 16 Mar 2016 07:39:50 +0530 Subject: [Borgbackup] sending --list output to stdout instead of stderr In-Reply-To: <87shzr9pi9.fsf@angela.anarcat.ath.cx> References: <56E612B9.6070003@gmail.com> <56E8A77E.80606@waldmann-edv.de> <56E8B422.1050809@gmail.com> <87shzr9pi9.fsf@angela.anarcat.ath.cx> Message-ID: <56E8C06E.9050408@gmail.com> On 16/03/16 07:02, Antoine Beaupr? wrote: > On 2016-03-15 21:17:22, Sitaram Chamarty wrote: >> On 16/03/16 05:53, Thomas Waldmann wrote: >>>> When using "borg create -v -s -p --list ...", would it be possible to >>>> send the output of the "--list" to stdout instead of stderr? >>>> >>>> I would like to do some post-processing on the list output, while also >>>> seeing progress (-v, -s, -p) on the screen as it runs. >>> >>> We usually use stderr for log output because stdout might be used for connecting borg to other tools via a pipe (and it will transport file data then), like: >>> >>> borg extract --stdout ... | dd of=/dev/sdx >>> >>> stats output is also logged as I expect many people wanting to have that in their backup logs. So it's currently not possible to write this to different files / channels. >> >> Problem solved. Using the logging conf fixed things for me. Now I get >> the "-p" I want to see on screen while still getting the list of files >> to process later. I don't see the summary stats on screen anymore but I >> can do a "tail -15" or so of the logfile if I need that. > > Can you show us what configuration you used? I'd be great to have > samples for that in the documentation... Sure, but it's way too simple, though far too verbose for my taste. Everything is copied from the link in the docs, with no real understanding (on my part). [loggers] keys=root [handlers] keys=hand01 [formatters] keys=form01 [logger_root] level=NOTSET handlers=hand01 [logger_out] level=INFO handlers=hand02 [handler_hand01] class=FileHandler level=DEBUG formatter=form01 args=('borg.log', 'w') [formatter_form01] format=F1 %(asctime)s %(levelname)s %(message)s datefmt= class=logging.Formatter From tom.b.rushworth at gmail.com Wed Mar 16 20:41:31 2016 From: tom.b.rushworth at gmail.com (Tom Rushworth) Date: Wed, 16 Mar 2016 17:41:31 -0700 Subject: [Borgbackup] building borg backup for OS X Snow Leopard? Message-ID: <56E9FD3B.8010602@gmail.com> Hi All, I'm a new user of borgbackup, trying to use it on a variety of different OSes on a small network. It works very nicely on the supported OSes, thank you borg collective for the fine tool :). I'd like to be able to build it for the other OSes I am dealing with, but I run into trouble with libattr on both OS X Snow Leopart (10.6) and FreeBSD 10.1. These are systems that just aren't practical up upgrade. I'm an experienced C, C++, and Python programmer, and I've looked into the extended attribute functions available from the problem systems so I know there is sufficient OS support. What I don't know is how to configure libattr to build on these systems. The usual ./config, make approach fails with: ... features.h: No such file or directory Can anyone provide more detail on how the standalone binaries for OS X 10.10 and FreeBSD 10.2 were built? I understand from the documentation they were packaged by pyinstaller, but my understanding is that you need a working application before you can package it up with pyinstaller. Thanks in advance for any information! Regards, -- Tom Rushworth From tw at waldmann-edv.de Thu Mar 17 12:18:30 2016 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Thu, 17 Mar 2016 17:18:30 +0100 Subject: [Borgbackup] building borg backup for OS X Snow Leopard? In-Reply-To: <56E9FD3B.8010602@gmail.com> References: <56E9FD3B.8010602@gmail.com> Message-ID: <56EAD8D6.3010808@waldmann-edv.de> > I'd like to be able to build it for the other OSes I am dealing with, > but I run into trouble with libattr on both OS X Snow Leopart (10.6) and > FreeBSD 10.1. These are systems that just aren't practical up upgrade. Maybe have a look at the Vagrantfile (and at the developer docs), maybe you can get some hint there about what is missing / what went wrong: https://github.com/borgbackup/borg/blob/master/Vagrantfile http://borgbackup.readthedocs.org/en/stable/development.html > I'm an experienced C, C++, and Python programmer, and I've looked into > the extended attribute functions available from the problem systems so I > know there is sufficient OS support. OK. > What I don't know is how to > configure libattr to build on these systems. The usual ./config, make > approach fails with: > > ... features.h: No such file or directory Looks like some header file is missing. Maybe that?: http://stackoverflow.com/questions/29645238/what-is-the-purpose-of-features-h-header > Can anyone provide more detail on how the standalone binaries for OS X > 10.10 and FreeBSD 10.2 were built? See the Vagrantfile, search for pyinstaller or borg.exe. > I understand from the documentation > they were packaged by pyinstaller, but my understanding is that you need > a working application before you can package it up with pyinstaller. Yes, a working python program is a requirement, so it first needs to compile the C / Cython stuff and have all the libraries / headers installed also. -- GPG ID: FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From baz at irc.msk.ru Mon Mar 21 07:07:31 2016 From: baz at irc.msk.ru (Alexey Bazhin) Date: Mon, 21 Mar 2016 14:07:31 +0300 Subject: [Borgbackup] migrating from attic Message-ID: <20160321140731.8eafd5463f36be7886a4ac2a@irc.msk.ru> Hi! I had attic backup of 110GB source data with passphrase encryption. 700 snpashots was using about 150GB of space on backup storage which was filled 65% of total capacity. I had tried migrating without --inplace option, but ran out of space (with 35% being free before migration), so I had to migrate with --inplace which was successfull. Next I tried to make a new backup with borg (on same almost unchanged data), but ran out of space on backup storage again. Command is: # borg create --stats --compression zlib host:/backup::2016-03-20 /data Why deduplication haven't worked as it was in attic? Also /root/.cache/borg used 42GB space which is IMHO way too much. -- Alexey Bazhin From tw at waldmann-edv.de Mon Mar 21 09:37:43 2016 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Mon, 21 Mar 2016 14:37:43 +0100 Subject: [Borgbackup] migrating from attic In-Reply-To: <20160321140731.8eafd5463f36be7886a4ac2a@irc.msk.ru> References: <20160321140731.8eafd5463f36be7886a4ac2a@irc.msk.ru> Message-ID: <56EFF927.7040108@waldmann-edv.de> > I had tried migrating without --inplace option, but ran out of space > (with 35% being free before migration), Sure, as it copies your repo, space requirement doubles. > so I had to migrate with > --inplace which was successfull. > > Next I tried to make a new backup with borg (on same almost unchanged > data), but ran out of space on backup storage again. Command is: > # borg create --stats --compression zlib host:/backup::2016-03-20 /data > Why deduplication haven't worked as it was in attic? Did you run it as the same user as the attic backup and as the migration? IIRC the migration converts the attic chunks and files cache, so quick skipping of unmodified files and deduplication should work. > Also /root/.cache/borg used 42GB space which is IMHO way too much. If that's too much for you, read the FAQ for a workaround. -- GPG ID: FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From baz at irc.msk.ru Mon Mar 21 09:51:28 2016 From: baz at irc.msk.ru (Alexey Bazhin) Date: Mon, 21 Mar 2016 16:51:28 +0300 Subject: [Borgbackup] migrating from attic In-Reply-To: <56EFF927.7040108@waldmann-edv.de> References: <20160321140731.8eafd5463f36be7886a4ac2a@irc.msk.ru> <56EFF927.7040108@waldmann-edv.de> Message-ID: <20160321165128.1538e1bb0bb365825075d457@irc.msk.ru> On Mon, 21 Mar 2016 14:37:43 +0100 Thomas Waldmann wrote: > > I had tried migrating without --inplace option, but ran out of space > > (with 35% being free before migration), > > Sure, as it copies your repo, space requirement doubles. http://borgbackup.readthedocs.org/en/stable/usage.html#borg-upgrade "Unless --inplace is specified, the upgrade process first creates a backup copy of the repository, in REPOSITORY.upgrade-DATETIME, using hardlinks." I thought that hardlinks part meant it wouldn't use twice the space... > > > so I had to migrate with > > --inplace which was successfull. > > > > Next I tried to make a new backup with borg (on same almost > > unchanged data), but ran out of space on backup storage again. > > Command is: > > # borg create --stats --compression zlib > > # host:/backup::2016-03-20 /data > > Why deduplication haven't worked as it was in attic? > > Did you run it as the same user as the attic backup and as the > migration? I did migration locally, but backups was made as the same user remotely. > > IIRC the migration converts the attic chunks and files cache, so > quick skipping of unmodified files and deduplication should work. Is there a way to fix it? I can't add space to existing backup storage. But I can copy backup repo to some other place, make space consuming operations and copy it back. -- Alexey Bazhin From tw at waldmann-edv.de Mon Mar 21 10:23:57 2016 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Mon, 21 Mar 2016 15:23:57 +0100 Subject: [Borgbackup] migrating from attic In-Reply-To: <20160321165128.1538e1bb0bb365825075d457@irc.msk.ru> References: <20160321140731.8eafd5463f36be7886a4ac2a@irc.msk.ru> <56EFF927.7040108@waldmann-edv.de> <20160321165128.1538e1bb0bb365825075d457@irc.msk.ru> Message-ID: <56F003FD.7050309@waldmann-edv.de> >> Sure, as it copies your repo, space requirement doubles. > http://borgbackup.readthedocs.org/en/stable/usage.html#borg-upgrade > "Unless --inplace is specified, the upgrade process first creates a > backup copy of the repository, in REPOSITORY.upgrade-DATETIME, using > hardlinks." > I thought that hardlinks part meant it wouldn't use twice the space... Well, this seems a case of too much (misleading) details in the docs. Yes, it uses hardlinks at first (to quickly make a pseudo-copy of the repo). This does not need much space. The reason why it does this is via hardlinks is because it is quick. After that, it will iterate over all files, show a progress indication, read, convert and rewrite them (thus breaking the hardlink). And while it does that, space requirements doubles. >> Did you run it as the same user as the attic backup and as the >> migration? > I did migration locally, but backups was made as the same user remotely. > >> >> IIRC the migration converts the attic chunks and files cache, so >> quick skipping of unmodified files and deduplication should work. > Is there a way to fix it? I can't add space to existing backup storage. > But I can copy backup repo to some other place, make space consuming > operations and copy it back. I'm still not sure what went wrong there. The cache is located via the repo-id, so it should match no matter where that repo dir is located. Did you run the repo conversion on the machine that is normally running the borg (client) to backup the data that is in the repo? -- GPG ID: FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From baz at irc.msk.ru Mon Mar 21 10:28:58 2016 From: baz at irc.msk.ru (Alexey Bazhin) Date: Mon, 21 Mar 2016 17:28:58 +0300 Subject: [Borgbackup] migrating from attic In-Reply-To: <56F003FD.7050309@waldmann-edv.de> References: <20160321140731.8eafd5463f36be7886a4ac2a@irc.msk.ru> <56EFF927.7040108@waldmann-edv.de> <20160321165128.1538e1bb0bb365825075d457@irc.msk.ru> <56F003FD.7050309@waldmann-edv.de> Message-ID: <20160321172858.0945752c1b5e6756847080cd@irc.msk.ru> On Mon, 21 Mar 2016 15:23:57 +0100 Thomas Waldmann wrote: > >> Did you run it as the same user as the attic backup and as the > >> migration? > > I did migration locally, but backups was made as the same user > > remotely. > > > >> > >> IIRC the migration converts the attic chunks and files cache, so > >> quick skipping of unmodified files and deduplication should work. > > Is there a way to fix it? I can't add space to existing backup > > storage. But I can copy backup repo to some other place, make space > > consuming operations and copy it back. > > I'm still not sure what went wrong there. The cache is located via > the repo-id, so it should match no matter where that repo dir is > located. > > Did you run the repo conversion on the machine that is normally > running the borg (client) to backup the data that is in the repo? No, I ran repo upgrade on backup storage itself. Backups are made from another box via ssh. -- Alexey Bazhin From tw at waldmann-edv.de Mon Mar 21 10:36:58 2016 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Mon, 21 Mar 2016 15:36:58 +0100 Subject: [Borgbackup] migrating from attic In-Reply-To: <20160321172836.3726f8b46ab52b9dc57913aa@yume.ru> References: <20160321140731.8eafd5463f36be7886a4ac2a@irc.msk.ru> <56EFF927.7040108@waldmann-edv.de> <20160321165128.1538e1bb0bb365825075d457@irc.msk.ru> <56F003FD.7050309@waldmann-edv.de> <20160321172836.3726f8b46ab52b9dc57913aa@yume.ru> Message-ID: <56F0070A.6050603@waldmann-edv.de> >> Did you run the repo conversion on the machine that is normally >> running the borg (client) to backup the data that is in the repo? > No, I ran repo upgrade on backup storage itself. Backups are made from > another box via ssh. OK, then it only converted the repo, not the attic cache on that other box (backup client box). That explains why there is no borg files and chunks cache on that box. You could copy ~/.attic ~/.cache/attic to the storage box and run migration again, then copy ~/.config/borg ~/.cache/borg back to the backup client box. -- GPG ID: FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From baz at irc.msk.ru Mon Mar 21 10:46:54 2016 From: baz at irc.msk.ru (Alexey Bazhin) Date: Mon, 21 Mar 2016 17:46:54 +0300 Subject: [Borgbackup] migrating from attic In-Reply-To: <56F0070A.6050603@waldmann-edv.de> References: <20160321140731.8eafd5463f36be7886a4ac2a@irc.msk.ru> <56EFF927.7040108@waldmann-edv.de> <20160321165128.1538e1bb0bb365825075d457@irc.msk.ru> <56F003FD.7050309@waldmann-edv.de> <20160321172836.3726f8b46ab52b9dc57913aa@yume.ru> <56F0070A.6050603@waldmann-edv.de> Message-ID: <20160321174654.a0c54bafa9b274121d66c033@irc.msk.ru> On Mon, 21 Mar 2016 15:36:58 +0100 Thomas Waldmann wrote: > > >> Did you run the repo conversion on the machine that is normally > >> running the borg (client) to backup the data that is in the repo? > > No, I ran repo upgrade on backup storage itself. Backups are made > > from another box via ssh. > > OK, then it only converted the repo, not the attic cache on that > other box (backup client box). > > That explains why there is no borg files and chunks cache on that box. > > You could copy ~/.attic ~/.cache/attic to the storage box and run > migration again, then copy ~/.config/borg ~/.cache/borg back to the > backup client box. I started migration to borg because attic was crashing with OverflowError while accessing the cache. I thought that "cache" is something redundant and maybe broken and dropped it already. Any other way to recreate right cache? -- Alexey Bazhin From tw at waldmann-edv.de Mon Mar 21 10:54:57 2016 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Mon, 21 Mar 2016 15:54:57 +0100 Subject: [Borgbackup] migrating from attic In-Reply-To: <20160321174654.a0c54bafa9b274121d66c033@irc.msk.ru> References: <20160321140731.8eafd5463f36be7886a4ac2a@irc.msk.ru> <56EFF927.7040108@waldmann-edv.de> <20160321165128.1538e1bb0bb365825075d457@irc.msk.ru> <56F003FD.7050309@waldmann-edv.de> <20160321172836.3726f8b46ab52b9dc57913aa@yume.ru> <56F0070A.6050603@waldmann-edv.de> <20160321174654.a0c54bafa9b274121d66c033@irc.msk.ru> Message-ID: <56F00B41.9070907@waldmann-edv.de> > I started migration to borg because attic was crashing with > OverflowError while accessing the cache. I thought that "cache" is > something redundant and maybe broken and dropped it already. Any other > way to recreate right cache? It will be recreated automatically at your first (completed) backup. But if it operates without files cache, it might be that it first stores all chunks again into the repo and afterwards kills the old chunks. So temporarily it might need (much) more space. You can maybe safe a bit space by first doing some partial backups before you do a full backup. -- GPG ID: FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From baz at irc.msk.ru Tue Mar 22 05:31:19 2016 From: baz at irc.msk.ru (Alexey Bazhin) Date: Tue, 22 Mar 2016 12:31:19 +0300 Subject: [Borgbackup] migrating from attic In-Reply-To: <56F00B41.9070907@waldmann-edv.de> References: <20160321140731.8eafd5463f36be7886a4ac2a@irc.msk.ru> <56EFF927.7040108@waldmann-edv.de> <20160321165128.1538e1bb0bb365825075d457@irc.msk.ru> <56F003FD.7050309@waldmann-edv.de> <20160321172836.3726f8b46ab52b9dc57913aa@yume.ru> <56F0070A.6050603@waldmann-edv.de> <20160321174654.a0c54bafa9b274121d66c033@irc.msk.ru> <56F00B41.9070907@waldmann-edv.de> Message-ID: <20160322123119.e01d785332a22997c59a7b06@irc.msk.ru> Making partial backups didn't helped. I was able to reproduce the problem on a clean repo and small amount of static data: attic init repo attic create --stats repo::backup1 data borg upgrade repo rm -rf ~/.cache/borg borg create -v --stats repo::backup2 data (deduplication doesn't work) rm -rf ~/.cache/borg borg create -v --stats repo::backup3 data (deduplication does work) Also I tried a number of actions with repository to recreate cache (borg prune for example or borg create -n), but it didn't help. On Mon, 21 Mar 2016 15:54:57 +0100 Thomas Waldmann wrote: > > > I started migration to borg because attic was crashing with > > OverflowError while accessing the cache. I thought that "cache" is > > something redundant and maybe broken and dropped it already. Any > > other way to recreate right cache? > > It will be recreated automatically at your first (completed) backup. > But if it operates without files cache, it might be that it first > stores all chunks again into the repo and afterwards kills the old > chunks. So temporarily it might need (much) more space. > > You can maybe safe a bit space by first doing some partial backups > before you do a full backup. > > -- > > GPG ID: FAF7B393 > GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup -- Alexey Bazhin From tw at waldmann-edv.de Tue Mar 22 17:13:54 2016 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Tue, 22 Mar 2016 22:13:54 +0100 Subject: [Borgbackup] migrating from attic In-Reply-To: <20160322123119.e01d785332a22997c59a7b06@irc.msk.ru> References: <20160321140731.8eafd5463f36be7886a4ac2a@irc.msk.ru> <56EFF927.7040108@waldmann-edv.de> <20160321165128.1538e1bb0bb365825075d457@irc.msk.ru> <56F003FD.7050309@waldmann-edv.de> <20160321172836.3726f8b46ab52b9dc57913aa@yume.ru> <56F0070A.6050603@waldmann-edv.de> <20160321174654.a0c54bafa9b274121d66c033@irc.msk.ru> <56F00B41.9070907@waldmann-edv.de> <20160322123119.e01d785332a22997c59a7b06@irc.msk.ru> Message-ID: <56F1B592.9020406@waldmann-edv.de> On 03/22/2016 10:31 AM, Alexey Bazhin wrote: > I was able to reproduce the problem on a clean repo and small amount of > static data: > > attic init repo > attic create --stats repo::backup1 data > borg upgrade repo > rm -rf ~/.cache/borg > borg create -v --stats repo::backup2 data (deduplication doesn't work) > rm -rf ~/.cache/borg > borg create -v --stats repo::backup3 data (deduplication does work) > > Also I tried a number of actions with repository to recreate cache > (borg prune for example or borg create -n), but it didn't help. I've opened a ticket with this: https://github.com/borgbackup/borg/issues/787 -- GPG ID: FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From tw at waldmann-edv.de Tue Mar 22 17:23:59 2016 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Tue, 22 Mar 2016 22:23:59 +0100 Subject: [Borgbackup] migrating from attic In-Reply-To: <56F1B592.9020406@waldmann-edv.de> References: <20160321140731.8eafd5463f36be7886a4ac2a@irc.msk.ru> <56EFF927.7040108@waldmann-edv.de> <20160321165128.1538e1bb0bb365825075d457@irc.msk.ru> <56F003FD.7050309@waldmann-edv.de> <20160321172836.3726f8b46ab52b9dc57913aa@yume.ru> <56F0070A.6050603@waldmann-edv.de> <20160321174654.a0c54bafa9b274121d66c033@irc.msk.ru> <56F00B41.9070907@waldmann-edv.de> <20160322123119.e01d785332a22997c59a7b06@irc.msk.ru> <56F1B592.9020406@waldmann-edv.de> Message-ID: <56F1B7EF.5070406@waldmann-edv.de> > I've opened a ticket with this: > > https://github.com/borgbackup/borg/issues/787 ... and "enkore" already solved it - see there. :) -- GPG ID: FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From baz at irc.msk.ru Wed Mar 23 04:21:48 2016 From: baz at irc.msk.ru (Alexey Bazhin) Date: Wed, 23 Mar 2016 11:21:48 +0300 Subject: [Borgbackup] migrating from attic In-Reply-To: <56F1B7EF.5070406@waldmann-edv.de> References: <20160321140731.8eafd5463f36be7886a4ac2a@irc.msk.ru> <56EFF927.7040108@waldmann-edv.de> <20160321165128.1538e1bb0bb365825075d457@irc.msk.ru> <56F003FD.7050309@waldmann-edv.de> <20160321172836.3726f8b46ab52b9dc57913aa@yume.ru> <56F0070A.6050603@waldmann-edv.de> <20160321174654.a0c54bafa9b274121d66c033@irc.msk.ru> <56F00B41.9070907@waldmann-edv.de> <20160322123119.e01d785332a22997c59a7b06@irc.msk.ru> <56F1B592.9020406@waldmann-edv.de> <56F1B7EF.5070406@waldmann-edv.de> Message-ID: <20160323112148.965a5d649608efdf95e6c8e7@irc.msk.ru> Thank you. On Tue, 22 Mar 2016 22:23:59 +0100 Thomas Waldmann wrote: > > > I've opened a ticket with this: > > > > https://github.com/borgbackup/borg/issues/787 > > ... and "enkore" already solved it - see there. :) > > > > -- > > GPG ID: FAF7B393 > GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup -- Alexey Bazhin