Evacuate! Dealing with natural disasters.

Ah, the joys of working in New Orleans during hurricane season. Today I spent most of my day executing our poorly planned strategy to deal with natural disasters. Beginning with finding out what everyone's plans were.

Are you evacuating? Y/N
Are you planning to work? Y/N
Do you have what you need to work remotely? Y/N

These are questions I wouldn't have normally had to ask in our prior remote-only working environment. But we moved in to an office about a year ago and since then have grown comfortable with on-site servers holding critical data and things like that. We have offsite backups, which is great because we can't lose any data. While that's critical, it overlooks another very important component of dealing with disasters:

Service Continuity.

With a firm of 15+ employees, every day counts when contributing to payroll, and the work we are performing. Our employees time is what directly generates nearly all of our revenue, roughly a $100/hr average per employee. So what happens when 15 people who generate $100/hr for the company suddenly can't work for a day? Math time!

That's right, $12,000 of lost revenue... per day.

While this may not seem like much in the grand scheme of things, cash flow is usually tight for small companies, and natural disasters like hurricanes can always present the possibility of lost income. So what did we do faced with an impending hurricane?

On-site document storage server
We use a NAS (Network Access Storage) to store contracts, scanned checks, company policy documents, training documents, and company processes. In order to make sure these things were still available in the event of a hard disk crash, we bought a team Dropbox account and have it synced with that. That's sort of an "oh-shit" backup solution that isn't great, but could end up saving us one day, like tomorrow for example. We were able to open up the backup dropbox account to our team in order to give them remote access to the NAS, even if power goes down in our office.

On-site Windows development server
The portion of our staff working on Windows development do not work on local machines. They instead remote in to a Terminal Server across our LAN on gigabit ethernet lines. This is neccessary due to the system configuration involved in making a Windows machine play nicely with the variety of software needed to develop C#.NET applications, MS Access, and Silverlight. So what happens when this server goes down? Noone can work any more. Our solution here was to export our VMWare images of the various Windows images we develop on, and export it to an Amazon EC2 AMI. We were able to export this AMI and get it up and running on EC2 in no time at all. Setting up a VPC/VPN was equally trivial, and now we're locked and loaded and everyone is ready to work remotely.

While we were able to get these two critical systems in to the cloud before the hurricane hit, that was only *just barely*. We really should have done this switch sooner. If you work in an office were critical data is stored locally, consider moving things to the cloud (at least as backups) before you are on the receiving end of your own natural disaster.

What I want to know when hiring developers.

I was thinking the other day about how I try to quickly evaluate someone's developer chops during a phone interview. One of the things I try to do is ask something baked in to the theory of what people learn from school. This is less about trying to see if someone has a formal education on the topic, but more that they are alternatively the type of person to read about these things. I think that's a really strong character trait; the interest and motivation to independently learn the ins and outs of your field. There's no real reason a programmer can't go throughout their entire life never spending much time learning about design patterns, or how the compiler works, or other academic subjects of computer science. They could still very well succeed in their career, and be perfectly happy, but maybe never truly shine as a star. But, when I am evaluating a prospective employee, I tend to care about things like this. I want the guy who is passionate about what he does. I want the person who will help try and drive us towards goals, and not just be clocking in.

Another thing I like to be able to evaluate is how a developer will make technology decisions. They will certainly be using their judgement on things throughout any project, so hopefully their judgement is good. I recently came up with something that I think may show how someone thinks. It's a simple question:

"How would you build a clone of twitter if you could only write 20 lines of code (in any language)"

The answer from the Prima Donna developer may sound something like this:

"I would set up a mongo database with the schema for tweets, and then write a small series of small scripts that can handle the functional requirements in a lisp-like DSL I'm working on right now."

The answer from a new developer, or graphic designer may sound like this:

"I would do a wordpress install and see if we can get some of the social plugins to function like Twitter."

The answer from a developer who is interested in startup culture:

"Before we do that, we should set up a landing page with exclusive invites only to test the demand for this Twitter-like product"

and so on...

There's not specifically a right or wrong answer to this question, but based on the type of ship you are on, these answers can be very revealing. This does a lot to check prospective employees as cultural fits. The question can reveal for example, that the developer has a cost-cutting attitude that may be perfect in a services firm, but horrible in a vc funded startup.