The ISP had redundant electrical grid connections for reliability, but the two connections were not isolated at the electrical utility level. A failure in one substation cascaded to the other substation. The operation of one electrical feed depended on the operation of the other, so they were effectively only a single feed.
But I don’t understand why them being connected makes one dependent on the other, unless half of the supply alone can’t support the workload. What is the “electrical utility level”
The “utility level” is Madison Gas & Electric’s infrastructure. Our ISP had two independent electrical service connections based on the idea that if one went down, they’d still get power sufficient to run their data center from the other. That would be the case if each connection reached all the way to the generating station completely independently. However, the two substations to which the ISP was connected were linked in such a way that a catastrophic failure of one caused failure of the other, so it got no electrical power.
Can you please elaborate on the technical details of the failures? What was the hidden dependency?
The ISP had redundant electrical grid connections for reliability, but the two connections were not isolated at the electrical utility level. A failure in one substation cascaded to the other substation. The operation of one electrical feed depended on the operation of the other, so they were effectively only a single feed.
But I don’t understand why them being connected makes one dependent on the other, unless half of the supply alone can’t support the workload. What is the “electrical utility level”
The “utility level” is Madison Gas & Electric’s infrastructure. Our ISP had two independent electrical service connections based on the idea that if one went down, they’d still get power sufficient to run their data center from the other. That would be the case if each connection reached all the way to the generating station completely independently. However, the two substations to which the ISP was connected were linked in such a way that a catastrophic failure of one caused failure of the other, so it got no electrical power.