How Our Dallas PoP Stayed Online During the Costliest Disaster in the History of Texas
Disaster recovery (DR) planning is good practice for just about every business – but for data center operators, it is critically essential. Even a single second of power loss can cause major issues for the thousands of customers residing in its racks, and the critical services they provide.
NetActuate’s deployment in Dallas, Texas, is the second largest of the 30+ locations we provide service from. In February this year, Texas experienced a power crisis of epic proportions. Three severe winter storms swept across Texas over the course of 10 days, with subfreezing temperatures and large quantities of snow – a huge departure from a “typical” Texas winter. Improperly winterized natural gas equipment across the state began to fail. Importing power from nearby states was impossible, due to the isolation of Texas’ power grid. Over 4 million people were left without power in a winter storm for several days. The only way to save the entire power grid was partial shutdowns until the temperatures got warmer. It turned out to be the costliest disaster in the state’s history.
During the entire outage and disaster, NetActuate’s services in Texas never went down. Servers remained powered, and services stayed up. This was all due to the meticulous planning and proactive investment in backup systems by the staff at Carrier-1, the operator of the Dallas, Texas data center our racks are located in.
“When we choose an operator to partner with, we have an extensive process we go through to ensure they are able to support our uptime requirements,” said Mark Mahle, CEO and Principal Technology Architect. “Carrier-1 has been a great partner for us for many years, and their handling of this winter storm disaster reminds us why we choose to work with operators who take disaster recovery and systems redundancy seriously.”
“For the entirety of the winter storm, the facility did not experience a power outage,” said Julia Morgan, President of Carrier-1. “When utility power was cut several times by ERCOT during the rotating black-outs, redundant power systems were utilized as planned.”
Unfortunately, some data center facilities in Texas didn’t fare nearly as well. Some had generators that were undersized, or couldn’t refuel because they didn’t have fuel contractor agreements in place ahead of time. When it was time to failover, other data centers discovered that their batteries were not charged, or were missing a block heater required for the extremely low temperatures.
As part of their DR planning, Carrier-1 proactively built redundant systems and put them in place to ensure an ample amount of backup power, cooling, network capacity, and personnel would be ready when the time came. The building has eight megawatts of power available, which is oversized for the current usage. They also invested in multiple generators, transformers, UPS systems, ATSs, PDUs, and fuel reserves.
“But you can’t stop there,” said Morgan. “You need a procedure in place so that everyone knows how to test and respond. We consistently test our systems to make sure that they will function as designed in the event of an outage.”
Beyond testing, Carrier-1 also developed a monitoring system in-house that was designed specifically for their environmental controls. Alerts from the monitoring sensors allow them to proactively repair or replace a part in a system if needed, ensuring automatic failover in the event of an outage disaster.
“At the end of the day, our people make all the difference,” said Morgan. “We could not be more proud of our technical staff, facilities crew and electrical team for their dedication, talent, and sacrifices to see our customers through the storm.”