Comment 1PE Re: Rolled my Own

Story

What Is Your Offsite Storage Solution?

Preview

Was tempted... (Score: 1)

by songofthepogo@pipedot.org on 2014-05-16 17:01 (#1NZ)

...to answer "my butt" or "right here in my pants", but bravely managed to resist.

Back on topic, I've been avoiding the offsite backup thing because I never thought I had anything so terribly important that it required that sort of protection. Backups, absolutely yes, but offsite? Nope. Then again, I'm also someone who can't imagine the house burning down or flooding (or what if both??). That said, I have briefly looked into Amazon Glacier. Do I really, really need to insure all those photos of my trips and videos of my cats are protected against acts of god/fsm or theft? Those of you who do do offsite backups, what sort of data are you protecting/preserving?

Rolled my Own (Score: 3, Insightful)

by commonjoe@pipedot.org on 2014-05-16 19:53 (#1P4)

I'm also someone who can't imagine the house burning down or flooding (or what if both??).
Most of my friends and family got hit by Katrina. I happened to live out of state at the time so almost everything I had was ok. (Almost.) Nearly everything they owned was gone: out of print sci-fi games and comic books, priceless heirlooms and wedding albums and family pieces, vehicles and houses.

Back on topic: I back up on hard drive and swap out hard drives between my apartment and a lock box somewhere off site. There is always at least one hard drive off site. I rotate at least once or twice every two weeks.

I've dealt with my personal files and backups of those of a two small businesses. I was so sick of backup programs that failed (and I've broken every single one I tried), I finally designed my own and got as far as a prototype. (I'm a programmer by profession.) Unfortunately, the prototype never made it out of the very early stages. It has enough to keep me happy but not enough to be viable on the Internet. Basically, it syncs some folders and makes a full copy of other folders. Then it double checks them. It makes a note of any failures.
Those of you who do do offsite backups, what sort of data are you protecting/preserving?
I have about 100,000 files / 200 GB that I backup on a regular basis. A lot of it is music and videos, but there are photos of my family, programs I've half written, stories and books I've written, and information I've downloaded from the deep recesses of the Internet (some of which I've had so long that they cannot be found again).

I'd love to write my own backup program. A lot of professional backups claim to do what I want to do now, but the big question is: will it work when disaster strikes? I've seen failures on the most insane things (and sometimes, the restore / backup program wouldn't even tell you about the error; you just had to accidentally encounter it). 256 character limit for file/folder path names. Crapping out on restore when you have over 50,000 files. Taking over an hour to scan your (large) restore drive so you can recover just one file. Telling you a compare failed, but it won't tell you which file has the problem (the bit rot problem). Windows machines that cannot backup files on a Linux shared drive. (Windows doesn't know what to do with two files named "HelloWorld" and "helloworld".

Then there are the other things that I think should be standard: Deduplicating files on the backup and saving space. (Deduplication seems to be more the norm today, but it wasn't when I started this list.) Multiple versions of a file on backup. (I hear Mac is pretty good at this.) Settings that pop up alarms if the number of old copies on your backup drop below a certain number. Copy of the backup / restore program on the backup disk itself. Backup program that isn't locked into a particular operating system. The ability to skip locked files and come back to them later without user intervention or program termination. The ability to perform a compare on every file on a client and test it against the copies on the server without having both online at the same time except for about 60 seconds. (Yes, that is possible.) The ability to detect if the hard drive or CPU is too bogged down so the backup program lightens the load so the user has more control over the computer. Easy to read reports. Easy to use program. Easy to read documentation about the program. Backups that span multiple computers or multiple hard drives. Stopping a backup in the middle and picking up where it left off several hours or days later.

Here's one I have to see: Plug the external hard drive in and only the backup program takes care of everything... Basically, the backup program leaves the user alone unless something like "A folder did not back up without failure at least once every 24 hours" or every 7 days or whatever the setting is. Then it gets in the user's face.

Sorry for the rant. Maybe such a program exist for the small time user, but I haven't seen it yet. The programs in Linux that I've seen aren't very sophiscated and the Windows versions cost a fair bit of money for stuff that I thought should be standard. If I had the time and energy to program it, I would. It's been a dream of mine to get something out there like that, but life gets in the way.

I have 12 pages of ideas. Despite all of the ideas, I think it's still possible to make it stupid simple for the noob while also having the crazy advanced settings for the expert. I just don't see that option very often in most programs!

Re: Rolled my Own (Score: 1)

by songofthepogo@pipedot.org on 2014-05-17 00:02 (#1PA)

Most of my friends and family got hit by Katrina. I happened to live out of state at the time so almost everything I had was ok. (Almost.) Nearly everything they owned was gone: out of print sci-fi games and comic books, priceless heirlooms and wedding albums and family pieces, vehicles and houses.
Man, that's awful. That had to have been simply devastating for them.
I have 12 pages of ideas. Despite all of the ideas, I think it's still possible to make it stupid simple for the noob while also having the crazy advanced settings for the expert. I just don't see that option very often in most programs!
You just might be right. I can think of programs that do some of what you outline, but not everything. If you write such a program, you'd likely have a lot of fans.

Re: Rolled my Own (Score: 2, Interesting)

by commonjoe@pipedot.org on 2014-05-17 03:59 (#1PE)

Man, that's awful. That had to have been simply devastating for them.
Yes. It even affected me 800+ miles away because I suddenly had my folks, my brother, and their dogs living with me for several months. I mention that not for sympathy, but because you said you said:
I've been avoiding the offsite backup thing because I never thought I had anything so terribly important that it required that sort of protection. [Snip] Then again, I'm also someone who can't imagine the house burning down or flooding
Disaster will happen. It's not a question of if, but a question of when, how, and how bad. ("The how" can be really bizarre as it was in the case of friends and family. Interestingly enough, nearly everyone who had a laptop brought their laptop with them before Katrina struck, and that helped them reconnect with friends and resources very quickly.) I had another friend who had a house fire and had no insurance. (I met him several years after the incident.) He had to start over

To me, as a programmer who writes not only programs but short stories and novels and can't remember what he ate for breakfast (even though it's the same thing every morning), the very thought of permanently losing everything digital scares the hell out of me. My life are those programs, stories, and notes / articles I have stored. I'd rather lose all the rest of my earthly possessions than lose everything on my hard drive. I couldn't get in touch with all of my friends again (scattered across the world) if I lost the names, addresses, phone numbers, and email addresses of everyone I know stored only in hard drives and backups.

Most of today's business couldn't live if they lost what was put on storage. Frankly, it amazes me so many companies trust "the cloud" especially when companies or technologies come and go at a whim. I think cloud storage can be great, but I see it as a way to supplement, not replace my hard drive technologies. I wouldn't use the cloud as my only backup because I don't trust their security. It's another reason I'd like to roll my own backup because I'd be in control of the encryption program -- not them. (It's another thing in my design notes.) Also, the cloud cost a lot of money for the sizes I'm talking about. Well, it's a lot of money for me. Also, I don't think my ISP would like me downloading 200+ GB every week for my usual check in addition to my usual surfing.

I know not everyone places the same emphasis as I do on their bytes and bits. I just mention all of it merely as food for thought.

Moderation

Time Reason Points Voter
2014-05-17 18:12 Interesting +1 songofthepogo@pipedot.org
2014-05-18 22:21 Normal 0 kbahey@pipedot.org

Junk Status

Not marked as junk