Computer problems – RAID and backup

I may make very few posts during the next six weeks or so.

One reason is that I’ve been spending a lot of time rebuilding my computer data storage.  First the backup of my travel images became corrupted so I tried to recreate it.  Then one of the backup drives failed.  Then one of my data drives failed.

My main data drive was a RAID 5 array with four 1TB drives.  This means that three of those drives were for storage, with “striping” or writing across drives for speed, and the remaining drive was for backup.  My backup system is a Drobo S, with five 3TB drives, functioning similarly to RAID 5.   I had a lot of spare space there so when the drive failed it just rebuilt to the remaining four drives but that made it unusably slow for backing up for several days in the meanwhile.

Then, when one of the drives in the RAID 5 data drive failed,  I managed to backup Antarctica, Japan and New York and just two TIFF files from the North Atlantic trip before it died completely.   This was good because I had no backup of the images I restored and for the travel files I didn’t manage to restore, I still had all the RAW files on the laptop.

I now have all backups restored.  I lost about 250 TIFF files from the North Atlantic trip where I worked in Photoshop CC, Vivesa, Silver Efex Pro, SNS HDR or AutoPano Pro.  Those I can always regenerate from the RAW files though.  However, it appears I failed to create a backup for Blues Festivals in 2011 and 2012 and I have lost all those images, about 7,000 in all.  The best 1,000 or so are published to the web on JAlbum (in reduced size) but losing the source files is unfortunate and there’s no redress.

.

The original file for this image is now lost. This is Darren Jack at Thredbo Blues Festival 2012.

.

The conventional wisdom is that you should have three copies of your data, including one in a remote location in case say your house burns down.  I thought I could get away with two because they were both in systems that can usually cope with drive failures.  I almost did get away with that but then the main problem was that I was missing a backup.

I have rebuilt the data drive with four new 2TB drives as a RAID 10 array.  This means that two of those drives store data while the other two are backups (mirroring the data).  The drives are also “striped”, writing between each other for speed.  That can cope with losing up to two drives, provided they are not both drives of a mirrored pair.

I have many spare internal drives and a “disk caddy” to access them.  So I will create a second set of backups on those drives.  Finally, I will backup selected images to the Cloud.  These will probably be those with three stars or more in Lightroom, only essential images to minimise expense.

It’s also important to have a backup of your computer’s image, i.e. your C Drive including all the hidden system files.  You can use that to restore all your functionality and programs if you get a virus or your C Drive crashes and even (with a higher version of Acronis) to a different disk that replaces your C Drive.  This didn’t come to play in this situation but has a couple of times in the past.  I have saved images from both Acronis and Windows.  There’s a link here that shows you how to do it on Windows 8 which also has the benefit that Windows will use this image if you elect to Refresh PC.  Otherwise refreshing the PC just wipes everything.

I lost ten days with all of that.  I had about ten posts prepared in advance but not any more.  I’ve started preparing posts on St Kilda where I took about 400 images and another 420 sailing around the islands.  This will take a while and there’s also a lot to say.  Then this coming weekend I’m off to cover the Thredbo Blues Festival and it will take me at least two weeks to process the images I take there.  After that I’m in India for most of February (more on that soon).  I may not get much opportunity to post from there.  There may be very few posts between now and early March.

9 comments on “Computer problems – RAID and backup

  1. leecleland says:

    How devastating! At least you didn’t loose the lot, but 7000 is a huge amount to lose in any ones language and a lot of time yet to be spent by the sounds of it. Enjoy the Thredbo Blues Festival it is a wonderful weekend isn’t it?

    Like

  2. Murray Foote says:

    It could have been worse. I nearly lost all images from Antarctica, Japan and New York due to the corrupted backup.

    Thredo’s always great. Nice and cool this year too. Maximum temp of only 26degC.

    Like

  3. Russell Hunt says:

    Bugger with a capital B. Despite all our best efforts eventually all the holes in the swiss cheese line up and there’s big trouble. I use Crash Plan online backup because they’ll send you a hard disk of the files you’ve uploaded to their cloud. Remember, Murphy was an optimist.

    Like

  4. Cem says:

    So sorry to read about these issues Murray. If I may say so, it seems that you’ve made matters worse by continuing to work with a failed backup drive. Having had similar issues in the past, I’ve stopped working with raid drives. I now backup every raw file to two separate disks in two separate servers already during ingestion. Followed by a periodical off site backup copy. If there are problems with any drive, I don’t ingest new images until I’ve replaced the faulty drive, which is easy and fast since it is not part of a raid array.

    Never mind my ramblings. I wish you lots of fun in the festival and also in India. I’m looking forward to seeing the pictures.

    Like

    • Murray Foote says:

      I’ll essentially have that when my new setup is completed because I’ll have a spare drive to replace any disk in the RAID10 array that fails immediately, backups to spare disks as well as to the Drobo and essential files backed up to the Cloud.

      The important images I lost were from a backup I thought I’d created but I hadn’t. If I’d had multiple backups and done the same thing for all of them, my backup mechanism would have made no difference. Even there, I still have 71 full-sized jpegs from those files that I had sent to people on request.

      Ah well, we live and learn. All we can do is keep going and find new experiences. Thredbo is always great. India will be mind-blowing, I’m sure.

      Like

  5. aussie greg says:

    HI Murray, I Know excatly how you feel, I Have in the past two years knocked over 3 external drives on the floor, 2-2000 gig and one 1000 gig hd drives. I had 12 years of Documentaries in the bush all over Australia..As well as 125 bands I had filmed/photographed.. All up I estimated I lost 2300 songs, and about 19,000 images and also the 12 years of finished documentaries about 1500 trips/expeditions into the bush..Technology isn’t the be all it promised..

    Like

  6. […] Computer problems – RAID and backup  […]

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s