“I have to buy three of everything. It doesn’t make any sense, but I have to. I’m worried I might lose it, and if I lose it, then I have a backup and then I have a backup to my backup.” — Huda Kattan
I was travelling this week and while at the airport, I overheard a lot of discussion about the airport shutdowns in the northeast on Monday, September 21, 2026. People were aghast because not only did the primary communications systems fail, but the backup failed.
As a process safety engineer who gives a lot of thought to redundancy as a means to reduce the likelihood of incidents, my mind immediately started thinking about possible reasons for a redundant system to fail.
Why Redundant Systems Fail
Everything fails. It is not a matter of “if” but a matter of “when”. Because of that, we provide critical systems with backups. But everything fails, including backup systems.
Three reasons came to mind to explain why a backup system would fail to protect against the failure of the primary system. That is, assuming that the backup system was capable of providing that protection when it was working as intended.
The first reason for a backup system to fail to protect against the failure of the primary system is a common cause failure. Whatever caused the primary system to fail also caused the backup system to fail.
The second reason for a backup system to fail to protect against the failure of the primary system is because the backup system was already in an unrecognized failed state when the primary system failed.
The third reason—the most improbable reason—for a backup system to fail to protect against the failure of the primary system is because both systems independently failed for unrelated reasons. Unlikely, but not impossible.
Common Cause Failure
There are occasions when the cause of a primary system failure is simultaneously the cause of the backup system failure. When the two systems depend on a shared component, the failure of that component will take both systems down, which is why we really want the backup system to be completely independent of the primary system. The more shared components, the greater the probability of a common cause failure.
Common operating principles can also lead to common cause failure. The two systems can be physically independent, but when they both work the same way, the same external factor can lead to both systems failing. This is why we want to see diversity in our systems as well as independence.
Unrecognized Failed State
There are occasions when the backup system fails while the primary system continues to work. Because we are relying on the primary system, however, there is no recognition of the failed state in the backup system. After all, when the primary system is working, we don’t need the backup system. Then, when the primary system fails, the already-failed backup system doesn’t protect against the failure. It is only when we need it to work that the failure of the backup system reveals itself.
For this reason, periodic proof tests on backup systems are vital. Before the proof test, we don’t know if the backup system is working. After the proof test, we know that it is. Either because the proof test showed that the backup system was working, or because the proof test showed that the backup system was not working and we fixed it. The more often we conduct proof tests, the greater the probability is that the backup system will work when we need it.
The problem with proof tests is that they interrupt operations and take resources. In many cases, the backup system is disabled during the proof tests. So, there is tradeoff between proof testing backup systems too frequently and too infrequently. Optimizing that tradeoff is an essential part of designing the proof test of a backup system, and designing the proof test of a backup system is in turn an essential part of designing the backup system itself.
Independent Failure
Finally, there is the impact of random failures, which effect everything because, yes, everything fails. Sometimes, despite independence and diversity, despite rigorous proof testing, a backup system suffers a random failure just when we need it. This is less likely than most other causes, but it can happen.
What Happened at the FAA?
Undoubtedly, the Federal Aviation Administration (FAA) is still investigating the primary failure and backup system failures that impacted the northeast on September 21. However, a summary of press accounts indicate that the primary failure was of a communications circuit at the Philadelphia Terminal Radar Approach Control (TRACON) facility. Then, when they went to switch over at TRACON, they discovered that construction crews had accidentally cut the backup fiber-optic line. Whether cutting the fiber-optic line happened prior to the primary system failure or during the switch over is not clear from the various press accounts.
As a result of the communication system failure, air traffic controllers lost communication and radar links, and so called for ground stops across major hubs: Newark, Philadelphia, New York (JFK), and New York (LGA). Additionally, there were cascading impacts on other airports throughout the northeast. While there were thousands of flights cancelled or delayed, inconveniencing hundreds of thousands of passengers, fortunately no one was injured or killed as a result of the incident.
Improving the Reliability of Redundant Systems
A backup system that does not work when needed is of little use. While random failures can disable a backup system, the more likely reasons that backup systems to fail to protect against the failure of their primary system are common cause failures and inadequate proof testing. For those of us working in the field of process safety, the September 21 incident is a good reminder to take another look at our own redundant systems. Are they sufficiently independent and diverse? If not, are we adequately accommodating common causes in our failure rate estimations? Are there proof tests for the redundant systems? Are we conducting at the frequency we should?
If we are not addressing these concerns now, while primary systems are working, then we are going to be bitterly disappointed when (not if) our primary systems fail and our backup system fails to protect us.
