In today’s fast-paced world of technology, it’s not a matter of if something will go wrong, but when. Telecommunications outages, in particular, serve as a stark reminder of this reality. After every major outage, voices from all corners of the tech world emerge, questioning, "How could this happen?" The truth, however, is that significant incidents are inevitable. Technology isn't perfect, and neither are the systems that power it. The trick lies not in preventing SHIT from happening, but in minimizing its impact when it does.

Embracing Murphy’s Law & Heinrich’s Incident Pyramid

Murphy’s Law—“If anything can go wrong, it will”—is a cornerstone of technology. It reminds us that no system, no matter how well-designed, is immune to failure. The layers of complexity in telecommunication networks make outages not just possible, but expected. This aligns with Heinrich’s Incident Pyramid, which shows that small incidents often precede larger ones. A few packet losses today could escalate into a full-blown outage tomorrow.

Yet, despite these warnings, the goal isn’t to eliminate incidents entirely but to reduce the occurrence of significant havoc in technology (SHIT). Proactive steps, better processes, and more resilient systems can decrease these events from daily nuisances to rare disruptions.

The Maturity Gap in Technology Incident Management

One glaring issue in the industry is the lack of maturity in managing incidents. Too often, technology teams focus solely on prevention, failing to develop comprehensive incident management strategies. This “ostrich approach,” where heads are buried in the sand until a disaster strikes, leaves companies ill-prepared for when SHIT hits the fan.

More mature industries recognize that incidents are a part of life, and they prepare accordingly. Take, for example, Herbert William Heinrich’s 1-30-300 rule, where for every major incident, there are 30 minor incidents and 300 near-misses. This pyramid highlights the need to take minor incidents seriously because they often precede the major ones.

Going Beyond Scapegoating | Understanding Incident Causes

Heinrich’s theories of incident causation stress that multiple factors contribute to an incident. Focusing on a single failure—be it human error, a faulty device, or bad timing—misses the point. Instead, true root cause analysis requires understanding the chain of events, each piece of the puzzle contributing to the bigger picture. For every major outage, there’s typically a trail of minor issues that were ignored, dismissed, or poorly handled.

In telecommunications, for instance, a network failure often stems from multiple factors. Minor packet losses could be compounded by poor routing decisions, outdated equipment, or even physical infrastructure issues like cable cuts. By the time the system goes down, it's no longer about that one failure, but the culmination of many.

SD-WAN | Automating Incident Management and Recovery

Now, let’s talk solutions. SD-WAN is becoming a crucial player in dealing with telecommunications incidents, allowing for a level of automation that streamlines recovery. Instead of relying on manual troubleshooting, SD-WAN enables real-time adjustments based on network conditions.

When SHIT happens, SD-WAN can reroute traffic through healthier paths, ensuring that disruptions are minimal. This is especially valuable for businesses as they can avoid costly downtimes and dedicate fewer resources to reactive incident management. Fusion's SD-WAN, for example, excels in creating this automation, making telecommunication incidents smoother to manage and quicker to resolve.

Wrap | SHIT Happens, But We Can Manage It

While significant havoc in technology (SHIT) is inevitable, we can take measures to mitigate its impact. Embracing principles like Murphy’s Law and Heinrich’s Pyramid helps us prepare for these incidents rather than be caught off guard. And with solutions like SD-WAN, we’re in a better position to handle the inevitable outages that occur. The future isn't about avoiding SHIT altogether, but about learning to deal with it when it comes—and learning how to emerge stronger on the other side.

---

Ronald Bartels ensures that Internet inhabiting things are connected reliably online at Fusion Broadband South Africa - the leading specialized SD-WAN provider in South Africa. Learn more about the best SD-WAN provider in the world! 👉 Contact Fusion

---