2006 -- What's your archive plan?

leebase · January 2, 2006

New year -- time to evaluate all things -- and one of them should be "what is my backup plan" for my digital photos.

If you have your photos on your computer hard drive -- and nowhere else -- you NEED to take action. Hard drives fail -- always. It's just a matter of when, not if. Having only one copy of your digital images on a computer hard drive is a sure recipe for losing all your photos at once.

There are levels of "safety" and price tags associated with those levels. You'll want to familiarize yourself with the options and choose your own safety/cost/hassle level.

Topics to consider
1. Protection from "single point of failure".
2. Protection from your house burning down, flood or other natural disaster
3. Protection from your archive media failing
4. Protection from your file format becoming obsolete

My current plan

I present this not as "the optimal plan for everyone" -- but just my own current actions based on my own budget and need for safety.
I have two hard drives, one internal, one external. I use a free utility SyncBack (http://www.2brightsparks.com/) that automatically takes files from my primary photo dirctory and copies any new or changed files to my other hard drive.

This alone is pretty good protection from "single point of failure". Hard drives fail, but the chances of two failing at the same time is pretty rare. A better, but more expensive solution, is to get something like the Terastation for $750. (http://www.amazon.com/gp/product/B0007MHF1I/sr=1-1/qid=1136229737/####/102-9887836-4197752?%5Fencoding=UTF8)
It's 4 250gig hard drives using raid 5. Any one hard drive can fail, and you don't loose data. The advantage is that you don't have to have two copies of your data or sync between the two.

With the two hard drives, I'm protected against a single drive failing -- but my house could still burn down, or my office could flood.
My next level of protection is that I archive all my photos onto DVD. For this I have the wonderful tool called Archive Creator
(www.rawworkflow.com) I make two copies of my archive, and I use two different brands of DVD. I keep one set of my dvd's at a place other than my house. At less than $.50 each, the dvd's are a very economical way of storing my photos.

The "long term safety" of files on a dvd is not known. Most reports have them lasting 5 to 10 years. There are, of course, anectdotes of people who've had discs go bad within 6 months. As such, I do not reccomend DVD's (or CD's) to be one's ONLY copy of data. It's also why I burn 2 different sets and use different brand of media for each.
If you are going to rely on DVD's or CD's, it would be wise to set up a schedule for "re-burning" your archives onto new media. Each person will have to set their own safety vs. hassle level about this. Reburn every year? Every 3 years?

Keep in mind that DVD's and CD's will not have the longevity of film negatives. You can't put them in a shoebox in the basement for 50 to 100 years. Even if the data on them survives, will computers 50 years from now have dvd drives? I don't think so.

Instead, you must plan on moving your data onto new media as technology progresses. Consider the DVD's I burned this weekend (56 dvd's to archive 2 sets of my 2005 photos). Three years from now when it's time to reburn them, I'll probably burn them on Blu-Ray dvd's which will come out later this year, and hold 50gig per double sided disc.

Finally there is "file type". You want your photos archived in a file type that will still be supported in the future. The ubiquity of jpg and it's massive presence in all areas of photography makes it a good "future safe" bet. Of more concern to me is the raw file format. What if the camera company goes belly-up....who will be around to make the software necessary to process the proprietary raw formats?

Right now I have my files in both raw and jpg format. I'm keeping my eye on Adobe's DNG format which attempts to address the "future protection" of raw formats. It's not mature enough, or supported widely enough, that I'm going to bother converting my Canon raw files to DNG just yet -- but I am keeping my eye on the subject.

Ok, folks, let the conversation begin. What's your plan for the new year?

Lee

Chase · January 2, 2006

I upload them to smugmug and store em there and on my computer.

Unless all of California falls into the ocean I should be ok.

gus · January 2, 2006

SmugMug & DVD for me & its a small price to pay for safety.

John Mueller · January 2, 2006

I just removed 100 gigs of raw files.A new year,time to start over:D
Jpegs on hard drive,cd and dvd.

DanielB · January 2, 2006

when i get a smugmug it will be that and DVD's for me:D

DavidTO · January 2, 2006

DanielB wrote:

when i get a smugmug it will be that and DVD's for me:D

When you do, use my coupon code and save $5: CLGbTHzpo7yxc

Art Scott · January 2, 2006

David;

What brand of DVD do you use?

+/_ R or RW?

How many do you buy at a time?

Thanks.

Zanotti · January 2, 2006

Humungus wrote:

SmugMug & DVD for me & its a small price to pay for safety.

I agree and this is my back up solution as well.

What do you think the long term viability of Sumumug is? Would there be enough notice should it go out of business to get copies of you files made? Has anyone used thie back up CD service and can you comment on its viability?

Daniel: Get a smugmug account now - its really a delightful way to handle photos and sharing. You will notice that sharing is much easier to do and you will do it much more often, making the hobby evan more fun when you get it.

(You can ask me for a coupon as well, but I see someone has already offered it to you. Any smugmug member can give you a $5 off coupon and that person gets the same off as well.)

ScottMcLeod · January 2, 2006

My solution is 2x DVD's (I attempt to make each copy on different brands), and then I try to keep the most recent stuff on HDD.

I'm going to have to invest in some kind of RAID system soon though. I wish the external RAID systems would come down in price.

badtz · January 2, 2006

Hrmm, perhaps I show paranoia here, but Id never trust a third party for backups. I currently archive my photos along with any other critical data on a HD in my workstation, on a raid array on a server I have at home, remotely to a server hosted in a complete different location, and occasionally when Im not lazy I put things on DVD as well.

I know it probably seems like overkill, but i've had computers stolen, house fires, and all other kinds of fun things, but I still have all my critical data going back to 2000 thanks to having anything critical offsite. Im also an IT geek, so I might be overdoing it a wee bit.

ScottMcLeod · January 2, 2006

badtz wrote:

Hrmm, perhaps I show paranoia here, but Id never trust a third party for backups. I currently archive my photos along with any other critical data on a HD in my workstation, on a raid array on a server I have at home, remotely to a server hosted in a complete different location, and occasionally when Im not lazy I put things on DVD as well.

I know it probably seems like overkill, but i've had computers stolen, house fires, and all other kinds of fun things, but I still have all my critical data going back to 2000 thanks to having anything critical offsite. Im also an IT geek, so I might be overdoing it a wee bit.

I need a decent offsite hosting solution, but most of them are fairly pricey...

Art Scott · January 2, 2006

Archiving questions
For those of you archiving on HDD....If it is external do you let it run all the time?? How large HDD do you use??

Those of you using DVD....What brands do you use? Are you finding any significant difference in brands (speed excluded)??

I know Mitsui has release a GOLD DVD that is supposed to be great butthe cost was over $1.50 each in bulk in Nov. We used Mitsui gold cd at the University I worked for...hardly any ever errored out not duplicated the master cd's.

Thanks

JamesJWeg · January 2, 2006

using smugmug is out of the question for me, I have about 500GB of photo's at this point, would take too long to upload. I little tech info for you, with RAID 5 you do NOT get to use all of your room, about 20% - 25% of your space is lost, four 250GB HDD's will NOT get you 1.0 TB. Four 250GB HDD's will leave you with about 770GB of usable space when in RAID 5. Also do NOT!!!!!!!! trust RAID 5 as your only means of backup, if more than one drive fails at once you loose it all, I have had this happen several times. Think about it, all 5 disks are exactly the same and are used the same amount, most times you get lucky and they fail one at a time, but I have had more than one drive fail at the same time twice in 2005, that is twice in one year over 40 servers, or about 1 out of every 20 each year.

James.

erich6 · January 2, 2006

First of all I want to thank Lee for such a thoughtful rundown on a complex topic like digital photography backup. I have a smugmug account but I don't rely on it as a backup although it could provide that for those images that I have posted.

I use an external 250 GB USB drive to backup my data. I have a fireproof safe at home where I keep it and other important documents.

I like the idea of DVD's but backing up to them is too slow and inefficient for my busy lifestyle. Maybe the new DVD formats will be better....

I think the question of digital file formats is very important. I don't like the thought of relying on JPG's. I like having my RAW files accessible so when I learn more about post-processing I can recreate prints that are even better. That said, you are relying on the well-being of the company and continued support for that format in future releases of software. You might say "I'll just keep my old version of software" but that software may not work in future systems. That means you would have to preserve your computer as well (if it can last that long). That's why I'm really looking forward to Adobe really establishing the DNG as an open-source standard.

Erich

leebase · January 2, 2006

JamesJWeg wrote:

using smugmug is out of the question for me

It should be out of the question for everyone. I do not think it's fair to use smugmug as your "off site storage facility". I peronally think that people who use smugmug in this fashion are going to hurt the service.

I little tech info for you, with RAID 5 you do NOT get to use all of your room, about 20% - 25% of your space is lost, four 250GB HDD's will NOT get you 1.0 TB.

Still, without using raid, and keeping 2 copies of all your files, means you loose 50% of your storage space. So a raid array is more efficient losing only 25% to provide "single point of failure" protection.

Also do NOT!!!!!!!! trust RAID 5 as your only means of backup, if more than one drive fails at once you loose it all, I have had this happen several times.

You have been very unlucky. If such a thing were common, the whole concept of raid would not be accepted. Still, a raid array is as vulnerable to fire and flood as any other solution so no, it shouldn't be your only backup plan. I'd still one 2 sets of dvd archives -- one for off site storage.

Think about it, all 5 disks are exactly the same and are used the same amount, most times you get lucky and they fail one at a time, but I have had more than one drive fail at the same time twice in 2005, that is twice in one year over 40 servers, or about 1 out of every 20 each year.

That's an incredibly high failure rate. I would imagine that the provider of your hard drives should be reconsidered. Still, without raid, each of those failures would have resulted in data loss -- every time -- not just when "two failed at the same time".

Lee

JamesJWeg · January 2, 2006

leebase wrote:

You have been very unlucky. If such a thing were common, the whole concept of raid would not be accepted. Still, a raid array is as vulnerable to fire and flood as any other solution so no, it shouldn't be your only backup plan. I'd still one 2 sets of dvd archives -- one for off site storage.

That's an incredibly high failure rate. I would imagine that the provider of your hard drives should be reconsidered. Still, without raid, each of those failures would have resulted in data loss -- every time -- not just when "two failed at the same time".

Lee

That is a fairly standard failure rate, one thing I forgot to point out is that those drives run 24/7/365, for home use a drive will last a lot longer. My failure rate per drive at this site is 1/40 per year. 40 servers X 6 drives each = 240 drives, 6 of those 240 drives failed in 2005, those 6 drives went in only 4 failures as TWICE it was two drives that went at the same time, or 50% of the time I had a total data loss. For these numbers I only counted servers using RAID 5. Just ask baldy how often they replace drives. These are top-of-the-line Seagates. My point is that multiple drives failing at the same time does happen and is not that rare. Don't get me wrong, I love RAID 5, but it is not a viable sole backup solution.

James.

W.W. Webster · January 3, 2006

Data security is a personal responsibility

leebase wrote:

What's your plan for the new year?

The same plan as last year!

I do an incremental backup of my total system (presently totalling 95GB including around 45GB of photos) to a second partition on my primary internal hard drive (250GB split into two partitions) no less than daily. This is no hardship as it only takes around 15-20 minutes and runs in the background. In fact, it's running now while I type this.

Then, every few days, instead of the 'daily' internal backup, I run a 'weekly' incremental backup to an external hard drive attached by a firewire interface. Then, every month or so, I run a 'monthly' incremental backup to a second external, firewire hard drive. Both of these external drives are normally stored in my office in my house, which has monitored fire and security detectors. However, before I go away on holiday I store them in my basement where they are less likely to be affected by the devastation of a fire or earthquake.

However, to be absolutely sure my photos are always 'reasonably' secure, I copy these off to CD-ROMs progressively as I have enough files to fill a new volume or before going away on holiday. These CD-ROMs are stored off-site at my son's place on the other side off town.

My normal maintenance routines include running a disk diagnostic and repair program (DiskWarrior) on my internal primary and daily backup partitions and my weekly backup drive every couple of weeks or so, and my monthly backup drive every time before I backup to it, to ensure the integrity of all my hard drives. There's no security in backing up corrupted data to corrupted backup disks. I also always review my CD-ROM backups immediately after I burn them to ensure all the images are readable. I do not, and would not, ever rely on Smugmug as any part of my backup strategy as I have no way of controlling its systems and processes.

Call me paranoid? Guilty as charged! It never ceases to amaze me how many stories you read of people losing important data they couldn't afford to lose because of a defective (or no) backup strategy, and then cursing their bad luck and looking for someone else to blame.

Crispin · January 3, 2006

I was thinking of cutting to DVD and then posting them to myself. Sometimes the postal service is slow slow I'll have a monthly cycle

Seriously though:
I have a drive which solely has my pics on. Nothing else hammers it.
I back that up, after every addition / editing of pics to an external.
I then backup that to DVD. Normally I leave a set of DVD's at a friends.

Flaws with stratagies mentioned so far: (Not trying to tread on toes)

To have a nice shiny raid solution standing next to your machine is pointless. Looks cool - that's all. Virus, Fire, water etc etc will destroy both machine and external storage. Cheaper option is to have two drives internally and RAID 1 (Mirror) them. Most new MB's have built in RAID. Use it - cheaper. (BTW: as mentioned, RAID 5 not perfect and, flawed when recovering from a crash, can cause a second drive to fail while rebuilding data. (Details on request:D )
Of site is the only way: To put your rem's in the basement to protect from fire... Where does water go when the firemen trying to put the fire out? Down. Rather give it to your son. Also, having a partition as a weekly backup drive is not wise. If the physical drive goes, so does your partition. You loose all changes from your last inc. Could be a lot. Invest in another drive.

I would say most people's idea are, within reason, sound. Get your data in two places. It's the only way... Oh, and do it quick.

W.W. Webster, While checking your disk is vital, to do it so often is bad. Disk checkers are rather intensive on the disks. I have crashed many a "barely alive" disk by running a check. Best way is to check the NT event log. Any signs of disk troubles will show up there. See it, remove data ASAP and throw disk against the wall...

I would cry, really cry, if I lost my pics. :cry :cry :cry

Zanotti · January 4, 2006

leebase wrote:

It should be out of the question for everyone. I do not think it's fair to use smugmug as your "off site storage facility". I peronally think that people who use smugmug in this fashion are going to hurt the service.
Lee

Can you explain this a little bit? I thought the whole point of Smugmug was a viable offsite place where photos can be stored and viewed ( and shared)

As a total casual photographer, I doubt I have more than a couple GB of photos on there and they are my primary backup. Why would this in any way damage Smugmug?

Your comment and my use are different - why? How does this hurt Smugmug? I wouldnt want to do that!

binghott · January 4, 2006

i have pictures saved on my laptop, backed up on a 200gb external hard drive, smugmug, and dvd's for important things.

i plan on getting another external hard drive because i don't trust them. i've had one crap out on me already, i was lucky enough to be able to recover the files from it. the second external will be there in case my main one craps out again.

ScottMcLeod · January 4, 2006

Zanotti wrote:

Can you explain this a little bit? I thought the whole point of Smugmug was a viable offsite place where photos can be stored and viewed ( and shared)

As a total casual photographer, I doubt I have more than a couple GB of photos on there and they are my primary backup. Why would this in any way damage Smugmug?

Your comment and my use are different - why? How does this hurt Smugmug? I wouldnt want to do that!

I think he's referring to the photogs that shoot ~100GB/year

ScottMcLeod · January 4, 2006

Crispin wrote:

I would cry, really cry, if I lost my pics. :cry :cry :cry

I did, and I did cry.

I blew out a 120GB drive before i'd backed the photos up from that month, and lost 4.7gb of data, roughly. Luckily the rest was backed up.

My plan:
2x DVD backup
External 120GB
Internal 120GB

Until I fill all of that, in which case I get a bigger drive each year, and store the old drive in a fireproof safe, which I have yet to buy.

JamesJWeg · January 4, 2006

ScottMcLeod wrote:

I think he's referring to the photogs that shoot ~100GB/year

Exactly, I have over 100GB of photos that I do not have on SM, yet I do have 65+GB on SM. I am guessing that SM uses RAID 5, with a 25% loss rate that would be about 206+GB, multiply that by two for a secound server to house a backup, and 412+GB would take them years to recover thier cost from me. Those numbers are for the original only, not counting the resized photos.

James.

CalfeeRider · January 5, 2006

I burn two DVD copies and keep one at home, the other at my office. And as much as I enjoy Smugmug, my understanding is there are some limitations that for me preclude it from being a viable backup strategy. First, I'm pretty sure you can't upload RAW, and as I recall from previous discussions I also think SM converts jpgs to sRGB, so if you prefer to work in Adobe RGB that *may* be an issue to others as well.

Can anyone confirm or correct me?

Thanks,
Jack

mrcoons · January 5, 2006

My plan is very similar to others:
1 DVD backup on a DVD-R disc (RAW, TIFF & JPG)
1 DVD backup on a DVD+R disc (RAW, TIFF & JPG)
(larger projects get sent to a dual layered disc)
Internal 120GB
External 120GB (incremental backup copy)
Smugmug only stores my JPG's at this point and periodically I order a DVD back up from them. (Which I use as my onsite back up.)

leebase · January 6, 2006

Zanotti wrote:

As a total casual photographer, I doubt I have more than a couple GB of photos on there and they are my primary backup. Why would this in any way damage Smugmug?

Your comment and my use are different - why? How does this hurt Smugmug? I wouldnt want to do that!

$100/yr for smugmug does not support endless gigs of storage space. Smugmug is a photohosting site for SELLING photos. I suppose if someone had tens of thousands of photos, and were selling thousands of prints -- that wouldn't be a big deal.

Have you seen ANY "off site archiving" site that offers unlimited storage for $100 per year? Smugmug has not priced itself for "offsite storage"....nor do I remember them marketing themselves as "the place to go for off line storage".

I think they were naive in saying "let's allow unlimited storage" -- not understanding that this turns them into a VERY cheap online storage service.

I do not believe they'll stay in business long if customers REALLY start using them for that purpose. A small percentage taking advantage of the "all you can eat" storage space can probably be absorbed. There will be a limit, though, and that's why I don't think it's fair to use smugmug as one's off site archiving facility.

Lee

leebase · January 6, 2006

JamesJWeg wrote:

Exactly, I have over 100GB of photos that I do not have on SM, yet I do have 65+GB on SM. I am guessing that SM uses RAID 5, with a 25% loss rate that would be about 206+GB, multiply that by two for a secound server to house a backup, and 412+GB would take them years to recover thier cost from me. Those numbers are for the original only, not counting the resized photos.

And keep in mind that the price for storage that we home user's pay is NOTHING like how much it costs for enterprise class storage that smugmug uses.

Lee

ysr612 · January 6, 2006

leebase wrote:

It should be out of the question for everyone. I do not think it's fair to use smugmug as your "off site storage facility". I peronally think that people who use smugmug in this fashion are going to hurt the service.

the way I read the info on the site they encourage people to use it this way. Andy how do you read what Baldy has in mind??

ExposeTheMoment · January 10, 2006

leebase wrote:

$100/yr for smugmug does not support endless gigs of storage space. Smugmug is a photohosting site for SELLING photos. I suppose if someone had tens of thousands of photos, and were selling thousands of prints -- that wouldn't be a big deal.

Have you seen ANY "off site archiving" site that offers unlimited storage for $100 per year? Smugmug has not priced itself for "offsite storage"....nor do I remember them marketing themselves as "the place to go for off line storage".

I think they were naive in saying "let's allow unlimited storage" -- not understanding that this turns them into a VERY cheap online storage service.

I do not believe they'll stay in business long if customers REALLY start using them for that purpose. A small percentage taking advantage of the "all you can eat" storage space can probably be absorbed. There will be a limit, though, and that's why I don't think it's fair to use smugmug as one's off site archiving facility.

Lee

Lee you must not be reading the forums [EMAIL="!@because"]because[/EMAIL] just last week I read here on the forums that SMUGMUG was going to... is going to SPEND 20,000 dollars so one person could upload 2TB worth of photos.

Here is the thread http://www.digitalgrin.com/showthread.php?t=24847&highlight=unlimited+space

and here is a snip it:
onethumb vbmenu_register("postmenu_218132", true);
smugmug CEO & Chief Geek

He says:

Wow, that must be close to 500,000 JPEGs! Quite a collection - I can't wait to see them.

So we've always said "unlimited storage" and we mean what we say. We're happy to take your photos and host them, but we need a little time to prepare.

Just so you know, this is something like a $20,000 first-year commitment for us in terms of disk space, power, cooling, and physical space. You'll basically have two complete RAID arrays to yourself in our datacenter. Of course, you'll only pay your $30, $50, or $100 per year, depending on your account level. Again, we're happy to do it - but I want to be up front here and let you know that we need to order some equipment and get it installed to accept your photos. We're not geared for accepting 2.5TB overnight.

We'll also be buying extra image processing machines just for your batch of photos. Luckily, once yours are done, everyone else at smugmug will get to benefit from them, so I don't consider that a cost to host you.

Does that sound fair? Can we ask you to hold off while we order and install the equipment and power required?

Thanks!

Don

gluwater · January 10, 2006

There are a lot of good ideas being discussed in this thread so far but with all the talk about RAID 5 I'm kind of surprised no one has mentioned RAID 6. Basically it is RAID 5 but with parity data stored across multiple data rows so you could have 2 drives fail. Or you could even do a RAID 5+1 but then you're looking at at least 6 hard drives and can only use half the physical space. RAID 6 also has slow write speeds due to the heavy processing involed in the parity creation. But I guess it really comes down to how much is your data worth to you. Has anyone here actually implemented a RAID 6 array or 5+1? If so did you ever have any problems with it?

ivar · January 10, 2006

I do have a 4*320gb disk RAID 6 dataserver system set up. It is definely slower than a RAID 5. Sofar i have not lost a single bit, but then again, i have not lost a single bit of the RAID 5 system i had either. For speed i would stick to a RAID 5 system, if you're data is important enough to get a fail-operational rather then fail-safe hard-disk system, you could go for a 6

2006 -- What's your archive plan?

Comments