GratITude: Keeping Customers Online In the Face of Disaster

Today, we’re kicking off our GratITude Series, highlighting the hard-working members of our data center and IT teams in essential roles powering a connected world. Their dedication enables our thousands of global customers to successfully operate their businesses in good times and in times of immense challenge. None of us could do what we do without them, and here we’ll share their stories.

When Texas was hit with a winter storm this February, nobody imagined it would lead to a regional disaster. But as the power grids failed, pipes froze and burst and gas deliveries ground to a halt, thousands were left scrambling to fulfill their most basic needs. In the aftermath, officials are looking to figure out why the situation escalated to a crisis so rapidly, but for those on the ground during the storm and its aftereffects, what mattered in the moment was finding the way forward. In our Dallas flagship data center, the INAP team put in the hours to keep our customers online. What did it take to get through this crisis without one failure?

The Dallas Data Center Team

INAP data centers offer full redundancy and reliable network connectively, among other benefits, to guarantee uptime to our customers. To meet this guarantee, Dallas Data Center Operations Supervisor Billy Boland and Regional Infrastructure Engineer David Thornton were on-site 24/7 for several days through the storm and the aftermath to keep the Dallas flagship data center up and running. Both men credit the tenure of the majority of the Dallas team and years of cross training as two factors that saw them through the disaster without dropping a single customer.

“I’m the only facility guy here in Dallas,” said Thornton. “Billy and his team, they’re the data center engineers. They work primarily with the customer, but they support me. Then on top of that, I’ve been cross training them on multiple things for years.”

That cross training would prove key in getting through the long hours it took to weather the storm. Cross training, and years of experience that gave Boland and Thornton the problem-solving skills to get through an unthinkable crisis.

Boland has been with INAP since 2011 and will be celebrating his 10-year-anniversary in September this year. He’s been working at the Dallas facility since it opened, starting out as a data center engineer and progressing to his current position as the data center operations supervisor. Thornton has been with INAP nearly as long, joining the Dallas team in 2012. Although he self-effacingly calls himself a “glorified maintenance man,” in his role as regional infrastructure engineer the facility is his responsibility—anything to do with the building itself, including the power, cooling, plumbing, paint, carpet, light bulbs and more.

Prior to joining the INAP team, both men picked up expertise via differing avenues. Boland started out in telecommunications installation, which could have taken him down any number of routes in his career. As luck would have it for INAP, he went the data center engineer route and has been able stick around and move up in the company.

Thornton began his career as an electrician in the local Dallas market before going overseas with the military. In Iraq, he learned all about critical infrastructure—how to maintain bases, how to keep equipment running and how to “MacGyver” it when needed. After traveling the world with the military for 10 years, moving through various disciplines into construction management, he settled back down in the states at the behest of his wife and started his position with INAP.

“I wanted a nice, simple job,” he said with a laugh.

Of course, the job became anything but simple in February.

The Disaster, Problem Solving and All the Right Decisions

Texas has seen its fair share of winter weather over the years. And like any data center, the Dallas team has a backup plan in place for failures, utility outages and cold weather. What was different in this case was that the power grid began to collapse at the outset of the storm. Rolling outages were the biggest issue for Thornton and Boland. Typically, they could run backup generators that function on diesel fuel, but the power outages also affected the refineries and fuel vendors. And because the cold was so unexpected, the fuel vendors didn’t have the antifreeze additive in their fuel.

The issues compounded with gas stations running out of fuel, hotels filling up because homes didn’t have power (and then half of the hotels not having power) and food and water shortages. Once the storm passed, the region was still dealing with the aftermath for days.

Fortunately, INAP leadership had the foresight to have Thornton and Boland book hotel rooms ahead of the storm, before space filled up. This allowed them a respite from the data center to rest, dry off and warm up from working outside in the snow and cold.

“The hotel didn’t lose power,” Boland said. “Insanely, it didn’t lose power.”

“Having the hotel paid off,” added Thornton. “When the winter storm hit, it was 24/7 for us for two or three days. We were dealing with one issue after the other. The power going on and off didn’t have a major impact to our critical systems because our facility is designed for that, but little things were occurring that we had to deal with.”

Both their experience and the cross-training Thornton has done with the Dallas team also helped them overcome the difficulties they were facing. Not only was Boland able to back Thornton up in the 24/7 facility work, but they were able to make the right decisions in the moment to keep the data center running. In a rolling blackout scenario, the logical choice would be to switch to the generator full-time to keep the power flow consistent. However, with the fuel vendor issues, this was not a viable option. They realized this early on and worked to conserve the fuel they did have, rather than go on an extended generator run. Several data centers in the area did make the choice to do an extended run, however, and ended up dropping customers.

“If we had done an extended run, we wouldn’t have made it,” Thornton said.

Their decision kept the data center up and running. “We never dropped anybody,” Boland added.

It’s easy to quip that “teamwork makes the dream work,” but in this case, when the going got really rough, the team was able to see things through.

“The team being what they were, we were able to lean on each other for different things,” said Thornton, “And we were able to identify upcoming issues and adapt.”

INAP is endlessly grateful for the Dallas team members and their resilience and problem solving in the face of this unforeseeable situation. They are an asset to the company and our customers who rely on our services to power their infrastructures.