Evacuate! Dealing with natural disasters.

Ah, the joys of working in New Orleans during hurricane season. Today I spent most of my day executing our poorly planned strategy to deal with natural disasters. Beginning with finding out what everyone's plans were.

Are you evacuating? Y/N
Are you planning to work? Y/N
Do you have what you need to work remotely? Y/N

These are questions I wouldn't have normally had to ask in our prior remote-only working environment. But we moved in to an office about a year ago and since then have grown comfortable with on-site servers holding critical data and things like that. We have offsite backups, which is great because we can't lose any data. While that's critical, it overlooks another very important component of dealing with disasters:

Service Continuity.

With a firm of 15+ employees, every day counts when contributing to payroll, and the work we are performing. Our employees time is what directly generates nearly all of our revenue, roughly a $100/hr average per employee. So what happens when 15 people who generate $100/hr for the company suddenly can't work for a day? Math time!

That's right, $12,000 of lost revenue... per day.

While this may not seem like much in the grand scheme of things, cash flow is usually tight for small companies, and natural disasters like hurricanes can always present the possibility of lost income. So what did we do faced with an impending hurricane?

On-site document storage server
We use a NAS (Network Access Storage) to store contracts, scanned checks, company policy documents, training documents, and company processes. In order to make sure these things were still available in the event of a hard disk crash, we bought a team Dropbox account and have it synced with that. That's sort of an "oh-shit" backup solution that isn't great, but could end up saving us one day, like tomorrow for example. We were able to open up the backup dropbox account to our team in order to give them remote access to the NAS, even if power goes down in our office.

On-site Windows development server
The portion of our staff working on Windows development do not work on local machines. They instead remote in to a Terminal Server across our LAN on gigabit ethernet lines. This is neccessary due to the system configuration involved in making a Windows machine play nicely with the variety of software needed to develop C#.NET applications, MS Access, and Silverlight. So what happens when this server goes down? Noone can work any more. Our solution here was to export our VMWare images of the various Windows images we develop on, and export it to an Amazon EC2 AMI. We were able to export this AMI and get it up and running on EC2 in no time at all. Setting up a VPC/VPN was equally trivial, and now we're locked and loaded and everyone is ready to work remotely.

While we were able to get these two critical systems in to the cloud before the hurricane hit, that was only *just barely*. We really should have done this switch sooner. If you work in an office were critical data is stored locally, consider moving things to the cloud (at least as backups) before you are on the receiving end of your own natural disaster.