Department of redundancy department / by Andrew Dacey

Today's post is all about redundancy. Today's post is all about redundancy. Okay, I think I've beaten that joke to death. Seriously though, I've already hit on RAID and that is one form of redundancy but when you're running a business and time is money there's still a lot more to think about in terms of redundancy. Big businesses tend to get pretty paranoid about redundancy. At my day job, everything that's in production is supposed to be fully redundant. We install our servers in pairs that are in different physical buildings, which are often quite a distance apart (if not in a nearby town). My understanding is they even go as far as making sure that the fibre optic cabling from the data centres to the ISPs (and yes, they use more than 1) don't have any points in common. The idea here is that 1 of this buildings could completely go offline and we should be able to have things up and running at the other location as quickly as possible. The other piece of this puzzle is to ensure that whatever took down the first site should not impact the other.

Now this may sound like overkill, the company I'm under contract for is a large financial services company in the US, they have literally millions of dollars on the line if there's an outage so they need this level of protection right? You're just a small self-employed photographer so you don't need something that elaborate right? Wrong. While you may not have to get as paranoid about redundant fibre links and such consider the impact of an outage to your business. While it may not number in the millions it doesn't have to in order to bring your business to its knees. You might only lose out on a few thousand dollars from a missed deadline but how easily can you absorb that loss? So what's your plan if something goes south?

Computers

Let's start with the basics, what happens if your computer dies on you? I've already covered backup options previously but this goes a little further than that. Suppose you're under the gun on a big deadline and the power supply on your computer dies. I'll assume here that you've got a killer backup strategy so you've only lost any changes since your last save (you do save frequently don't you?). Okay so your files are safe but that doesn't do you any good if you don't have a computer to pull them up on now does it? If you're a larger studio then you probably have more than 1 computer in the studio so that's a viable solution. Or how about a laptop? Worst case scenario, do you have a home computer that you could press into service (you are keeping your work computer separate from your home computer right?), does it have all the necessary software installed? Can it get the files you need?

Internet Service

It's a couple hours until your deadline, you just need to upload them to the client's server, but your internet connection has just dropped. You call up your ISP and find out they need to send out a technician to investigate the issue and since you're a residential customer that's going to be 3 days from now. Okay, first of all, what the heck are you doing running your business on a residential internet connection? Yes business internet accounts tend to be a lot more expensive but they usually carry with that expense a higher priority when it comes to outages. Even if you have a business account maybe it's still too long to wait to get it fixed. Do you have a 2nd internet connection with another ISP? If you have a studio separate from your home can you use your home broadband connection? If so, is it with the same provider? What do you do if the outage isn't a problem on your end but is due to a backhoe cutting the ISPs main fibre connection, cutting off the entire city's customers? Sound far fetched? I've seen it happen. Okay, so work and home internet connection are out of the question. Well, can you use your cell phone's data plan to upload the files (probably painfully slow but I'm talking desperation time now)? Or how about the coffee shop's wi-fi connection? Again, probably not the fastest option but could work in a pinch.

Physical location

This one is really tough for the small business. If you have a separate studio then the obvious option is to work out of your house. More importantly though, this really hammers home the importance of having at least 1 offsite backup for your files. If a catastrophe like a fire hits you most likely have other immediate concerns on your plate but you've also lost your livelihood if you're a full-time photographer. How long until insurance pays out? How do you keep your cash flow running in the meantime. Getting back to work might be the furthest thing from your mind but it may be a necessity in order to keep the money flowing.

You

I'll wrap up my thoughts with asking what happens if you're incapacitated or worse? Obviously if you're a one person shop then you're future bookings are more than likely off. But, do you have an assistant that can fill in for you? What about your partner or spouse, how much of the books do they know in order to sort things out if you can't? For that matter, are they even authorized to do so? Worst case scenario, if you die how well protected is your family in terms of not being burdened with a huge debt from you and how able will they be to make a living off your legacy of images? I know it's not a fun topic to think about but how would your feel if your family had no way of benefiting from the wealth of the images you've created over your career?

Test, test, test!

My final closing words will be that whatever options you deem necessary, make sure you test them! With my day job we're required to completely power down each of our major data centres on an annual basis. This is a literal powering down of the building. This forces us to make sure that nothing have slipped in that only runs in 1 data centre or can't be easily failed over. Similarly, it forces us to prove that we can keep fully operational while 1 of the data centres is out of commission. Similarly, this policy can even extend to key people in the company, it's not as frequent but many staffers are required to go on mandatory vacation. During the period of mandatory vacation their access is turned off so that they can't log onto any systems. There are some safeguards in place that can allow them to get it back if necessary but the purpose is to demonstrate that they can be gone for a period of time and things won't fall apart without them. I'm not saying you have to go to all of these extremes but make sure you do test out your redundancy plans before you find yourself having to rely on them.