Thursday, June 7, 2012

Datacenter Guys


People call me a “datacenter guy”. Collectively, they call us all “datacenter guys”. Some of the best datacenter guys I’ve known are women, but they’re still “datacenter guys”. I’ve been thinking a lot lately during my travels as to what it means to be a datacenter guy. Here’s what I’ve come up with so far.

First, some background on what I do. Datacenter guys locate, design, build, or operate facilities that house computers and electronic storage. This once niche specialty profession has emerged to become very much in demand in recent years with the increasing popularity of online goods and services. It’s good to be us! We get the rare privilege of straddling the worlds of the internet and mission critical facilities. Truly blessed.

For the uninitiated, some datacenters are small rooms, the size of a closet. Most companies of any size have at least one of these in some form or fashion on premises. Some datacenters are quite large, spanning several hundred thousand square feet in size. These facilities can hold hundreds of thousands of individual computers. So what is so unique about a datacenter that makes it different from any other building? Each one of those computers requires two critical things: 1) power (electricity) to run and 2) a way to deal with the resultant heat that is generated. Put 100,000 computers in one confined space and you can imagine how much power you need to deliver to the systems and how much heat they produce. If you’ve never been in a datacenter, think about hundreds of shelves filled with running hair dryers and you’ll get a pretty accurate picture of a production datacenter environment.

Datacenter guys are odd birds. We are a mix of several professions – internet engineer, network and server architect, real estate, IT, construction, negotiator, soils engineer, political strategist, tax expert, psychologist. Perhaps that’s why it’s so hard to characterize us – why we have our own designation. We’re datacenter guys.

One of the reasons I love being a datacenter guy is because people generally smile when they say it. “He’s a datacenter guy”, followed by the obligatory smirk. It’s kinda like being a gynecologist. “What do you do for a living?” “I’m a gynecologist”. Similar reaction, I’d think. Sometimes I wonder what it means. It could mean – poor bastard, he’s a datacenter guy. Imagine what he does every day. Or it could mean – geez I wish I were a datacenter guy. In any case it’s nice to get the smiles. It sure beats saying “I’m a coroner”. They probably get a different reaction, I’m guessing.

Datacenter guys travel a lot. We have to. There’s not many datacenters being built in our neighborhood. If there were, we’d want to live somewhere else. Datacenters are built in faraway places in the neighborhoods with all the auto body shops and distribution warehouses. Some of the best ones are built in the middle of nowhere, in alfalfa fields or abandoned aluminum smelters or in Iceland. And to complicate things, the best datacenter guys are notoriously cheap because we know all travel costs are going against the project. So we end up doing wacky things to save money, like staying inside the unfinished datacenter to save on hotel costs. Or the local Convent for $60/night (yes, seriously).

Needless to say, we have to leave our families to go to where the action is, and this makes it tough on the loved ones that we leave behind. We’re far from the only profession that has to log significant flight miles, but in the realm of the internet when most things can be done via cell/txt/skype/email/irc we stand out as dinosaurs who still have to go onsite to get things done.  The most fortunate of us have spouses and kids that understand why we aren’t home. It’s because we’re datacenter guys.

Datacenter guys know there’s something infectious, almost spiritual, in bringing a datacenter out of the ground. I suppose it’s the same with any kind of major construction project, like a skyscraper or hospital. I call it “the drug”. It’s the buzz, the energy around a project when you design a datacenter from the ground up and you watch it become a living, breathing hunk of reality before your very eyes. That thrill of “topping out” steel framing with an evergreen tree if you’ve not killed anyone so far in the process. You can tell when someone is on “the drug”. They will leave family and friends, forego food and water, they will even leave their current job, to go join a company that is designing and building. They’re on the drug, poor bastards. They’re addicted.

People who have built their share of datacenters are unique in the internet disciplines in that they have probably experienced something firsthand that the vast majority of their other colleagues typically don’t see – they have had someone on their project team maimed or killed. I was at an internet industry dinner a few weeks ago when someone asked a question of everyone at the table.

“Tell me about the most memorable operational incident you had to handle at work.”

The answers all had some variation on reactions to service outages.

First guy – “I remember our site cratering just minutes before we were to stream the very first online Victoria’s Secret fashion show.” Agreed, this sounded truly tragic.

Another – “I had 3 team members resign right in the middle of a crucial maintenance window.” Admittedly stressful.

Me – “I know of a father of two who fell off a steel structure to his death, and several guys who went home after their shift with fewer fingers or toes than when they started.”

Silence.

Get any semi-seasoned datacenter guy to talking about the topic of danger on the job and we’ll regale you with tales of horrendous incidents. There’s the story of the guy who incinerated himself by leaning over a battery string with an adjustable wrench in his pocket. I had another guy inches from sure death when a clutch exploded on a generator. Another fell off some scaffolding almost 2 stories tall and fractured his back and pelvis. Needless to say, it’s a dangerous job building and operating these facilities. Next time you’re touring a production quality datacenter, notice that lifeguard’s crook that is hanging innocuously in the electrical room. Strange, there’s no pool nearby. It’s called the “meat hook”, and it’s used to pry someone off the switchgear as they’re being electrocuted without actually having to touch them and becoming yet another organic conductor.

Another unique thing good datacenter guys do is to find that elusive perfect location for their next datacenter. In the industry we call it “site selection”. An integral part of any good site selection process is the negotiatation of incentive packages with states, counties, and local municipalities. I’m convinced that this is an art form that is born, not bred into some of us. It involves meeting with everyone from Mary Jane at the local Chamber of Commerce to the Governor to the Senator. In order to get the very best incentive packages, you need to please everybody. Mary Jane wants you to sponsor the German Shepherd for the town’s first police K-9 unit. And you can’t imagine how expensive they are to acquire and train. The Governor wants you to commit to a certain number of jobs and capital spend that you’ll bring to his or her state. And oh, by the way please site your datacenter over at the Technology Park that I pushed through the legislature because it’s gone nowhere since. And of course the Senator demands to be the first speaker at your groundbreaking so that he’ll be able to take credit for all of the aforementioned.  Site selection is a three ring circus but it is one of my favorite aspects of a new datacenter project.

All in all, I think those of us in the datacenter industry consider ourselves very fortunate. Our professions require us to build grand things, dabble in a wide variety of technologies, meet all kinds of interesting characters, and to see the world. Nothing could be better.

2 comments:

  1. Great Post Kevin! The one thing you forgot to mention is that the entire "industry" is pretty small. We all generally know each other and we are all truly "unique".

    ReplyDelete
  2. Just found this blog, nice post! We are in the middle of building a data center and found this post a breath of fresh air!

    ReplyDelete