BIT 1.4.1-1: Restore process is not what I expected. Have I misunderstood it ?

I am having difficulty in setting up backintime-qt to function as I expect; I would appreciate advice from experienced users. I’m running BIT 1.4.1-1 under Linux Mint 21.2 Cinnamon (based on Ubuntu Jammy), with Kernel 6.2.0-37. Hardware is 11th Gen Intel Core i5-11600, with 32 GiB. /Root and /Home are on separate NVME SSDs. Back up is to a 1TB 7200 rpm HDD. An nVidia 3060 GPU with 12 GiB of memory is installed. This post is about the process of restore, which seems to operate different to my expectation. That expectation is that a file which exists in the ‘original’ location in the file system at the time of restore will be overwritten by the file of the same name from the ‘full’ backup. This will in turn be overwritten by any later files of that file name in the sequence of incremental backups, from oldest to most recent, no matter how many incremental backups there are. The final state of the file will be identical to the newest incremental backup prior to the restore operation. There will be just a single file ‘A’ in the original location.. This is the process that every backup application, using incremental backups, appears to work, according to my understanding of the application descriptions that I have read on Google and YouTube. In contrast. the Man Page, for BIT, under the heading ‘DESCRIPTION’ says: “When you restore a file ‘A’, if it already exists on the file system it will be renamed to ‘A.backup.currentdate’.” I understand ‘currentdate’ to mean ‘current date and time to a sufficient resolution for file name to be unique’ – probably to the millisecond. The manual page statement then implies that after the restore, my data set will contain a file ‘A’ plus as many versions of the file A.backup.currentdate’ as there are unique versions of File A in the incremental backups. This could be 10’s to hundreds of additional file versions for EVERY file in my data set which has been changed in the period from the initial, or ‘full’ backup, to the last incremental backup. This could result in the original file location being filled to capacity, at which point I assume the restore will be terminated. This is not what I want and, more to the point, doesn't seem to be generally desirable. What have I misunderstood?

Thanks for asking, it's the way to learn :) The documentation of BackInTime could be better, so it's good that you ask. There is one big misunderstanding that is the source of most of your confusion. buhtz has already mentioned this in the other thread: BackInTime does not make "incremental" backups in the way that you understand the word. BackInTime uses hardlinks to "store" unchanged files without using any space. It makes copies of all the files that have changed between one backup and another. If you're not sure what a hardlink is, how to recognize one, and why hardlinks "save space" when used for backups, you should make yourself familiar with them. You can start here: https://www.redhat.com/sysadmin/linking-linux-explained – but most of all, play around with a few small files with unimportant content and try out BackInTime on them. Imagine you have three files, and their content is listed in brackets here. If the file named "foo" contains the letters "ABC", then I'll write foo[ABC]. So you have three files: - foo[ABC] - bar[DEF] - baz[XYZ] You make a BackInTime backup of these files, and they get copied to the backup location. Now you change the content of baz from [XYZ] to [999]. The next time you make a backup with BackInTime, your backup will contain this: - foo[ABC] <-- not a copy, just a hardlink to the same file from the last backup! - bar[DEF] <-- not a copy, just a hardlink to the same file from the last backup! - baz[999] <-- a new file! Your backup is now "incremental" in the sense that there are NO NEW COPIES of foo[ABC] and bar[DEF] on the backup location. They're just hardlinks, because the files are identical to before. BUT: BackInTime's backups are never "incremental" in the sense that the CONTENTS of a file are updated from one backup to another. If you change one byte in a 5GB video file, your backup will contain a totally new copy of that file. BackInTime will use 10GB of space to represent the fact that a 5GB file has changed by one byte. That's the way it's designed. I hope this makes things clearer. If not, feel free to ask again. Also: Play around with BackInTime! Use data that doesn't matter, a few small files or a few large ones, and just see what happens. Check the space usage of the backup location to learn how hardlinks are used. Cheers Michael On 30.11.2023 17:58, LateJunction wrote:

Hi, I trust that by now you have realised that I did not know your (and all others') replies to my appends to the mailing list existed. That discovery came as an embarrassing and regrettable shock to me this morning. Now, I imagine, everybody on this particular mailing list regards me as a useless old duffer, who doesn't read what people have spent their time to write, or who is just plain rude. The only accurate part of that characterisation is the word 'old'. Anyway, thanks for this explanation, which I follow/understand. I had been somewhat misled by reviews, by others, on the internet, that BIT is indeed an implementation of the incremental back-up model - and their claim is wrong. Indeed the implementation of BIT has advantages over both the incremental and differential backup models, so I am quite enthused about it. I suppose that what I would really like is a backup service that continuously keeps my backup up-to-date, mirroring CRUD (Create, Read, Update, Delete) activity on the source into the backup, on the fly a bit like a RAID 1, couple with a journaling file system. This makes restore the simplest it can be. Up to now I have implemented a crude, manual, equivalent to this process using FreeFileSync. It is quite effective, but a huge consumer of my time and has one fatal flaw: updates to the source are lost if a disaster occurs between an update and the next running of FreeFileSync. BIT is of course subject to the same 'hole' in the resiliency of my overall disaster recovery strategy. I just need to be aware of this.

Dear Tony, Am 06.12.2023 13:24 schrieb Tony Hamilton:
regards me as a useless old duffer
Enough with that self insult. :) We have understood it. Heads up and move forwared.
Indeed the implementation of BIT has advantages over both the incremental and differential backup models
To much flowers for BIT. The magic happens with "rsync" in the back.
What you describe with keeping a backup "up-to-date" is IMHO (in my humble opinion) not a backup but a synchronisation. That is different. RAID1 for example is such a sync mechanic. Or nextCloud is something like this. But that is not a backup. I never used FreeFileSync but from what I can read on the website it seems to be that the application do mix the concepts of data synchronisation and backup. That is not bad. I just mention it. Kind Christian

Hi, Thanks for your valuable comments, including this: "What you describe with keeping a backup "up-to-date" is IMHO (in my humble opinion) not a backup but a synchronisation. That is different. " So, is there some additional functionality in 'backup' which is not inherently present as a result of synchronisation? Synchronisation, per se, is not my objective: that has to be a resilient disaster recovery process. Much of my data, especially photos I took in the 1950s onwards, if lost, is unrecoverable. FreeFileSync does make robust copies for me, which I regard as backups, because they are on separate media, on separate computers and in separate locations. It is just a very time consuming process, which BIT addresses. Synchronisation - for example by using FreeFileSync - is just a by-product. I don't really like it because of, for example, Apple's implementation of synchronisation of my low resolution image portfolio: I would like to keep my 'master' collection on one of my desktop PCs, copied, for convenience, to iCloud and a subset of this collection on my iPhone. I can't do that: if I delete an image from my iPhone (to conserve memory) it is gone too from iCloud, and my desktop if I allow synchronisation to happen. That's Apple telling me which images I can have and where. This infuriates me.

On Thu, 2023-12-07 at 17:39 +0000, Tony Hamilton wrote:
So, is there some additional functionality in 'backup' which is not inherently present as a result of synchronisation?
Backup is a one-way "synchronisation", real "synchronisation" goes in both directions, eg.: If you delete a file in your source or target folder doesn't matter, once you sync the folders the file is deleted (lost!) in the source **and** target folder. Another difference is that backups **should** keep old file versions (which Back In time [abrev: BiT]) does with each snapshot using hardlinks.

Thanks for asking, it's the way to learn :) The documentation of BackInTime could be better, so it's good that you ask. There is one big misunderstanding that is the source of most of your confusion. buhtz has already mentioned this in the other thread: BackInTime does not make "incremental" backups in the way that you understand the word. BackInTime uses hardlinks to "store" unchanged files without using any space. It makes copies of all the files that have changed between one backup and another. If you're not sure what a hardlink is, how to recognize one, and why hardlinks "save space" when used for backups, you should make yourself familiar with them. You can start here: https://www.redhat.com/sysadmin/linking-linux-explained – but most of all, play around with a few small files with unimportant content and try out BackInTime on them. Imagine you have three files, and their content is listed in brackets here. If the file named "foo" contains the letters "ABC", then I'll write foo[ABC]. So you have three files: - foo[ABC] - bar[DEF] - baz[XYZ] You make a BackInTime backup of these files, and they get copied to the backup location. Now you change the content of baz from [XYZ] to [999]. The next time you make a backup with BackInTime, your backup will contain this: - foo[ABC] <-- not a copy, just a hardlink to the same file from the last backup! - bar[DEF] <-- not a copy, just a hardlink to the same file from the last backup! - baz[999] <-- a new file! Your backup is now "incremental" in the sense that there are NO NEW COPIES of foo[ABC] and bar[DEF] on the backup location. They're just hardlinks, because the files are identical to before. BUT: BackInTime's backups are never "incremental" in the sense that the CONTENTS of a file are updated from one backup to another. If you change one byte in a 5GB video file, your backup will contain a totally new copy of that file. BackInTime will use 10GB of space to represent the fact that a 5GB file has changed by one byte. That's the way it's designed. I hope this makes things clearer. If not, feel free to ask again. Also: Play around with BackInTime! Use data that doesn't matter, a few small files or a few large ones, and just see what happens. Check the space usage of the backup location to learn how hardlinks are used. Cheers Michael On 30.11.2023 17:58, LateJunction wrote:

Hi, I trust that by now you have realised that I did not know your (and all others') replies to my appends to the mailing list existed. That discovery came as an embarrassing and regrettable shock to me this morning. Now, I imagine, everybody on this particular mailing list regards me as a useless old duffer, who doesn't read what people have spent their time to write, or who is just plain rude. The only accurate part of that characterisation is the word 'old'. Anyway, thanks for this explanation, which I follow/understand. I had been somewhat misled by reviews, by others, on the internet, that BIT is indeed an implementation of the incremental back-up model - and their claim is wrong. Indeed the implementation of BIT has advantages over both the incremental and differential backup models, so I am quite enthused about it. I suppose that what I would really like is a backup service that continuously keeps my backup up-to-date, mirroring CRUD (Create, Read, Update, Delete) activity on the source into the backup, on the fly a bit like a RAID 1, couple with a journaling file system. This makes restore the simplest it can be. Up to now I have implemented a crude, manual, equivalent to this process using FreeFileSync. It is quite effective, but a huge consumer of my time and has one fatal flaw: updates to the source are lost if a disaster occurs between an update and the next running of FreeFileSync. BIT is of course subject to the same 'hole' in the resiliency of my overall disaster recovery strategy. I just need to be aware of this.

Dear Tony, Am 06.12.2023 13:24 schrieb Tony Hamilton:
regards me as a useless old duffer
Enough with that self insult. :) We have understood it. Heads up and move forwared.
Indeed the implementation of BIT has advantages over both the incremental and differential backup models
To much flowers for BIT. The magic happens with "rsync" in the back.
What you describe with keeping a backup "up-to-date" is IMHO (in my humble opinion) not a backup but a synchronisation. That is different. RAID1 for example is such a sync mechanic. Or nextCloud is something like this. But that is not a backup. I never used FreeFileSync but from what I can read on the website it seems to be that the application do mix the concepts of data synchronisation and backup. That is not bad. I just mention it. Kind Christian

Hi, Thanks for your valuable comments, including this: "What you describe with keeping a backup "up-to-date" is IMHO (in my humble opinion) not a backup but a synchronisation. That is different. " So, is there some additional functionality in 'backup' which is not inherently present as a result of synchronisation? Synchronisation, per se, is not my objective: that has to be a resilient disaster recovery process. Much of my data, especially photos I took in the 1950s onwards, if lost, is unrecoverable. FreeFileSync does make robust copies for me, which I regard as backups, because they are on separate media, on separate computers and in separate locations. It is just a very time consuming process, which BIT addresses. Synchronisation - for example by using FreeFileSync - is just a by-product. I don't really like it because of, for example, Apple's implementation of synchronisation of my low resolution image portfolio: I would like to keep my 'master' collection on one of my desktop PCs, copied, for convenience, to iCloud and a subset of this collection on my iPhone. I can't do that: if I delete an image from my iPhone (to conserve memory) it is gone too from iCloud, and my desktop if I allow synchronisation to happen. That's Apple telling me which images I can have and where. This infuriates me.

On Thu, 2023-12-07 at 17:39 +0000, Tony Hamilton wrote:
So, is there some additional functionality in 'backup' which is not inherently present as a result of synchronisation?
Backup is a one-way "synchronisation", real "synchronisation" goes in both directions, eg.: If you delete a file in your source or target folder doesn't matter, once you sync the folders the file is deleted (lost!) in the source **and** target folder. Another difference is that backups **should** keep old file versions (which Back In time [abrev: BiT]) does with each snapshot using hardlinks.
participants (5)
-
BiT dev
-
c.buhtz@posteo.jp
-
LateJunction
-
Michael Büker
-
Tony Hamilton