Twenty years ago, I began saving videos and clips that I downloaded from the Internet. Back then, dial-up modems and crappy video codecs meant that anything you downloaded took days and was a great accomplishment once successfully transferred. Obviously, I wanted to save the downloads, so I started looking for archiving options.
Hard drives back then where far too small, so I started to burn the videos off to CD media. The blank CDRs where cheap and could be bought by the spindle. A few years later, DVD burners became available and were a welcome upgrade (1 DVD could hold about 7 CDs worth of data) All in all, I burned approximately 600 CDs and 400 DVDs. Many of the CDs have started to deteriorate - the thin foil on the top of the disk just flakes off. The DVD media has protective plastic layers on both sides and have endured a little better, but read errors still occur on all of these 10+ year old optical disks.
Spinning 3.5" hard drives are the new best bet for large archives. I could store an entire spindle-worth of CDR disks on a single large capacity hard drive. Of course, loosing one of these hard drives to a failure made relying on a single copy a significant risk. Nearly 100% of all hard drives (around 20 drives) that I purchased with capacities between 80GB and 1TB failed within 5 years. I recovered from nearly all failures by simply storing 2 copies of all the data. Each drive was purchased in pairs - one drive as the master while the other was an offline full copy backup.
I tried creating large arrays of hard drives, but there are significant limitations and costs to doing this in a single system. After experimenting with a number of different setups with large 3ware raid cards as well smaller consumer raid cards, I've come to the conclusion that an ideal system should not include more than about 8 hard drives per computer. Trying to find which "bad disk" is spewing SATA errors to the console in a 16 drive array is just not fun. Scaling storage beyond 8 drives should involve a second computer and a network filesystem over Ethernet.
But what if you don't need all of the data online all the time? Instead of keeping all the disks spinning all the time, why not place the drives into external USB enclosures and only turn them on when needed? This system worked great for me - up to about 20 drives. You see, USB doesn't have enough power to run a 3.5" hard drive. So 20 external USB enclosures + 20 wall wart power adapters + 20 power cables + 20 USB cables + daisy chaining multiple USB hubs and their associated power adapters = one giant rats nest of wires. I tried to salvage this system with a custom made wooden cabinet with 18 front mounted switches, an series of internal USB hubs, and individual cubby-holes for each drive. It worked, but the low speed of USB2 (less than 20MB/s) as well as incompatibility of the external USB enclosures on drives larger than 2TB has caused me to abandon this system.
My latest attempt at archival is simply storing caddy-less hard drives on the shelf. It's so simple, I have to wonder why I bothered with any of the other approaches. Now that hard drives have ditched the IDE ribbon cable for SATA connectors, a new breed of USB enclosures have sprung up. These devices are USB3 (nearly as fast as the drive itself), can easily be upgraded (to support multi-terabyte drives), and simply accept a bare drive into its slot at the top of the unit. Increasing storage capacity is simply a matter of buying another drive or two. I did, however, end up purchasing several dozen hard drive sleeves (just a few dollars each) to protect the drives from dust and minor vibrations while on the shelf.
Hard drives back then where far too small, so I started to burn the videos off to CD media. The blank CDRs where cheap and could be bought by the spindle. A few years later, DVD burners became available and were a welcome upgrade (1 DVD could hold about 7 CDs worth of data) All in all, I burned approximately 600 CDs and 400 DVDs. Many of the CDs have started to deteriorate - the thin foil on the top of the disk just flakes off. The DVD media has protective plastic layers on both sides and have endured a little better, but read errors still occur on all of these 10+ year old optical disks.
Spinning 3.5" hard drives are the new best bet for large archives. I could store an entire spindle-worth of CDR disks on a single large capacity hard drive. Of course, loosing one of these hard drives to a failure made relying on a single copy a significant risk. Nearly 100% of all hard drives (around 20 drives) that I purchased with capacities between 80GB and 1TB failed within 5 years. I recovered from nearly all failures by simply storing 2 copies of all the data. Each drive was purchased in pairs - one drive as the master while the other was an offline full copy backup.
I tried creating large arrays of hard drives, but there are significant limitations and costs to doing this in a single system. After experimenting with a number of different setups with large 3ware raid cards as well smaller consumer raid cards, I've come to the conclusion that an ideal system should not include more than about 8 hard drives per computer. Trying to find which "bad disk" is spewing SATA errors to the console in a 16 drive array is just not fun. Scaling storage beyond 8 drives should involve a second computer and a network filesystem over Ethernet.
But what if you don't need all of the data online all the time? Instead of keeping all the disks spinning all the time, why not place the drives into external USB enclosures and only turn them on when needed? This system worked great for me - up to about 20 drives. You see, USB doesn't have enough power to run a 3.5" hard drive. So 20 external USB enclosures + 20 wall wart power adapters + 20 power cables + 20 USB cables + daisy chaining multiple USB hubs and their associated power adapters = one giant rats nest of wires. I tried to salvage this system with a custom made wooden cabinet with 18 front mounted switches, an series of internal USB hubs, and individual cubby-holes for each drive. It worked, but the low speed of USB2 (less than 20MB/s) as well as incompatibility of the external USB enclosures on drives larger than 2TB has caused me to abandon this system.
My latest attempt at archival is simply storing caddy-less hard drives on the shelf. It's so simple, I have to wonder why I bothered with any of the other approaches. Now that hard drives have ditched the IDE ribbon cable for SATA connectors, a new breed of USB enclosures have sprung up. These devices are USB3 (nearly as fast as the drive itself), can easily be upgraded (to support multi-terabyte drives), and simply accept a bare drive into its slot at the top of the unit. Increasing storage capacity is simply a matter of buying another drive or two. I did, however, end up purchasing several dozen hard drive sleeves (just a few dollars each) to protect the drives from dust and minor vibrations while on the shelf.