multi-volume M-Disc Bluray backup, and how to increment it later
by furryspider from LinuxQuestions.org on (#58TR8)
Dear folks,
the background:
Most of you will know this: Over the years, huge amounts of personal data, photos, correspondence, manuscripts, plans for world domination etc. get accumulated. The data contains lots of important and really irreplaceable stuff, and it needs to be safeguarded by good backups.
In my case, the most relevant data sums up to appr. 2 TB, and I keep two separate backups on USB hard drives. Now I'd like to add a third backup, in which I put the data on a set of M-Disc blurays and store them in a different location.
The problem:
As the pile of data is unlikely to stop growing, I will want to add another M-Disc from time to time, containing all the files that have changed or are new. Since I don't want to write out _all_ of the data every time I do this (which is also a cost factor), I need a way to identify the new and changed files.
How do I identify them, when my source is a huge directory structure on a hard drive, and my original backup is a set of bluray discs, only one of which can be mounted at a time?
First I was thinking about using git to manage my source structure and to identify the differences between certain 'branches', which would be dated backup snapshots. But my past experiences with git weren't always happy, and I'd have to duplicate (or more) the amount of my data, which is not feasible at this size.
Next I somehow felt that overlayfs had some potential, basically stitching together the several backup volumes, overlaying it with a difference directory and rsyncing that with the source directory structure. But I can't get over the problem of having only one out of a set of several discs mounted at a time.
Which made me wonder if I can somehow dump the file listings of each backup volume into a text file, accumulate them to produce a listing of all backuped files (including size and date), and then feeding this to rsync to use instead of the real backup volume for comparison with the source directory. But so far I have no clue if this is possible and how to accomplish it. So my second question would be: Is it possible to feed a text file with a directory listing to rsync? I know about the --files-from functionality, but I would have to simulate the _target_, not the source. And my shell skills are negligible.
---
I don't exclude the possibility that I'm being stupid and blind here, so perhaps if anybody here did something like this before, how did they solve it? Are there specialized tools, or clever approaches that I don't see? Am I going about it the wrong way?
All hints are welcome.
the background:
Most of you will know this: Over the years, huge amounts of personal data, photos, correspondence, manuscripts, plans for world domination etc. get accumulated. The data contains lots of important and really irreplaceable stuff, and it needs to be safeguarded by good backups.
In my case, the most relevant data sums up to appr. 2 TB, and I keep two separate backups on USB hard drives. Now I'd like to add a third backup, in which I put the data on a set of M-Disc blurays and store them in a different location.
The problem:
As the pile of data is unlikely to stop growing, I will want to add another M-Disc from time to time, containing all the files that have changed or are new. Since I don't want to write out _all_ of the data every time I do this (which is also a cost factor), I need a way to identify the new and changed files.
How do I identify them, when my source is a huge directory structure on a hard drive, and my original backup is a set of bluray discs, only one of which can be mounted at a time?
First I was thinking about using git to manage my source structure and to identify the differences between certain 'branches', which would be dated backup snapshots. But my past experiences with git weren't always happy, and I'd have to duplicate (or more) the amount of my data, which is not feasible at this size.
Next I somehow felt that overlayfs had some potential, basically stitching together the several backup volumes, overlaying it with a difference directory and rsyncing that with the source directory structure. But I can't get over the problem of having only one out of a set of several discs mounted at a time.
Which made me wonder if I can somehow dump the file listings of each backup volume into a text file, accumulate them to produce a listing of all backuped files (including size and date), and then feeding this to rsync to use instead of the real backup volume for comparison with the source directory. But so far I have no clue if this is possible and how to accomplish it. So my second question would be: Is it possible to feed a text file with a directory listing to rsync? I know about the --files-from functionality, but I would have to simulate the _target_, not the source. And my shell skills are negligible.
---
I don't exclude the possibility that I'm being stupid and blind here, so perhaps if anybody here did something like this before, how did they solve it? Are there specialized tools, or clever approaches that I don't see? Am I going about it the wrong way?
All hints are welcome.