Reflecting on over three decades in networking, there are recurring themes and lessons that have stuck with me. Technology may have advanced at lightning speed, but these foundational insights remain relevant.
1. Ostrich Syndrome
Just because you can’t see it doesn’t mean it’s not happening.
In networking, simply ignoring a problem doesn’t make it disappear. This “Ostrich Syndrome” – burying one’s head in the sand and hoping the issue will resolve itself – is common, especially among ISPs. ISPs frequently test only from their core, not from the actual user edge where customers experience issues. Their tools may not detect what the user is facing, and while they may rely on data showing that "everything looks fine," end-users know better. The key takeaway here? Don’t assume that because you’re not seeing an issue, it doesn’t exist. Real visibility into customer experience requires testing from all perspectives, and better tools are often the difference between discovering and ignoring hidden problems.
An example of Ostrich Syndrome is the way many techies approach the use of spanning tree:
2. Eating the Elephant One Bite at a Time
Effective maintenance is proactive, not reactive.
Networks are complex, and it’s tempting to ignore maintenance until there’s a forced outage. But most networks exhibit signs of degradation long before they fail entirely. Proactive maintenance is essential – by addressing the top interfaces with errors each day, you make incremental progress toward a healthier network, effectively “eating the elephant one bite at a time.” This approach requires a robust Network Management System (NMS) to identify and prioritize these problem areas, avoiding the “crisis-only” maintenance that results in bigger, preventable outages.
3. Causation
Workarounds are temporary; addressing root causes is vital.
The focus should always be on addressing the root causes rather than merely putting out fires. A common trap in networking is the quick-fix mentality: a workaround that temporarily resolves an issue but doesn’t address the underlying problem. This approach may seem efficient, but it leads to chronic recurrence, higher costs, and increased complexity. A mature organization takes time to investigate the root causes, knowing that the cost of fire-fighting far outweighs the effort of tackling the underlying issues once and for all. The bottom line? Invest in solving root causes and save time and resources in the long run.
4. Normalization
Complexity is a risk, not an asset.
When it comes to firewall rules and network architecture, complexity is the enemy of reliability. You might have a million firewall rules, but if only four would suffice, then you’re complicating your network unnecessarily. Normalization means simplifying your network to its core essentials, eliminating redundant components, and ensuring that what remains is easy to maintain and troubleshoot. Complexity often arises from incremental, organic growth, where layer upon layer of fixes and features build up until the network becomes unmanageable. A well-normalized network is efficient, easier to troubleshoot, and more resilient. Cut the excess and streamline.
5. Time
Time is a fixed, limited asset in network management.
Time is an ever-present factor in networking, impacting everything from detection to recovery. Whether it’s the actual time versus detected time, restoration time, or escalation time, every second counts. Networks demand quick thinking and rapid response times; often, you won’t have the luxury of unlimited time to troubleshoot. Time must be respected and managed effectively, knowing that the faster you can respond, the less impact on your users and business. From workarounds to incident reporting, mastering time management within network operations is a skill every professional must develop.
6. Visualization
Seeing the big picture reveals the hidden details.
While command-line interfaces (CLI) are invaluable, sometimes a picture is worth more than a thousand command outputs. Visualizations – in the form of network diagrams, metrics, and trend graphs – allow you to see the bigger picture and quickly identify problem areas. Without visualization, you’re often stumbling through dense data trying to spot patterns. With it, you can grasp overall health, spot trends, and, importantly, identify issues before they escalate. Visualization is about making the invisible visible, turning data into actionable insight, and helping you cut through the noise to focus on what matters.
7. The Spawn of Satan
Technology from the dark side
Over the years, certain technologies have felt like they were designed more to frustrate than facilitate. Printers, wireless extenders, and certain manufacturers (I’m looking at you, Huawei) have consistently failed to live up to expectations. These “spawn of Satan” technologies test your patience and resilience, proving that not all tech is created equal. Choose your tech carefully and know which tools work best for your environment. A lesson for the ages: sometimes, it’s best to avoid problematic tech altogether rather than deal with the headache it brings.
8. Captain Jean-Luc Picard
“Make it So!”
In the world of Star Trek, Captain Picard could issue a command, and his crew would immediately execute it. Unfortunately, in networking, you can’t just say “Make it so” and expect results. Technology operates on its own logic, and there’s no way to command it into compliance. Understanding that systems function on their own terms, not ours, is crucial. You have to respect the constraints and rules of each technology, knowing that patience, troubleshooting, and methodical analysis are the real pathways to success, not just commanding it to “work.”
9. Rabbit Holes
Overthinking instead of solving
Networking issues can lead you down some deep, time-consuming rabbit holes. It’s easy to overthink a problem, losing hours on something that turns out to have a simple solution staring you in the face. Sometimes the answer comes by stepping back, clearing your mind, or consulting a colleague for a fresh perspective. That “lightbulb moment” often arrives when you stop overcomplicating things. Networking requires persistence, but it’s essential to recognize when to pull back and re-evaluate rather than getting stuck in endless troubleshooting cycles.
10. When All Else Fails, Wireshark
Packets don’t lie
When every other tool and technique fails, there’s always Wireshark. Packet analysis is the ultimate truth in networking: packets don’t lie. Learning to read and interpret packets is a vital skill that every network engineer should master, as it provides an unfiltered view of network traffic and issues. Wireshark allows you to get to the heart of the matter, tracking down problems that can elude even the best diagnostic tools. In networking, Wireshark is the final arbiter – if you haven’t learned to read packets yet, it’s time to start.
Bonus
Documentation | Don’t Leave It for Later
Good documentation can save hours in crisis
Documentation is often neglected until something goes wrong, but it’s one of the most powerful tools for network engineers. It ensures that knowledge isn’t lost, facilitates faster troubleshooting, and can expedite onboarding for new team members. Networks are complex environments with a multitude of interconnected devices, protocols, and configurations; without accurate and up-to-date documentation, troubleshooting is like wandering in the dark.
Committing to robust documentation saves time and money by reducing dependence on memory, simplifying maintenance tasks, and allowing smoother handovers when personnel change.
Redundancy | Avoiding Single Points of Failure
Failure will happen; redundancy keeps services running
Networks are expected to be up 24/7, but hardware, software, and connections fail. That’s why redundancy is essential. When every critical component has a backup, you have a safety net to catch unexpected failures. Redundant links, power sources, and failover configurations can mean the difference between seamless operation and extended downtime.
Redundancy isn’t just about duplicating infrastructure; it’s about creating resilience in the system design. However, it’s also critical to avoid “over-redunancy,” where too many backup systems become cost-inefficient and unnecessarily complex.
Each of these ten lessons (as well as the bonus ones) encapsulates a piece of hard-earned wisdom from years in the field. They’re reminders of what it takes to succeed in networking: clarity, simplicity, attention to causation, respect for time, and above all, the willingness to dive deep into the packet flow when all else fails. Networking has evolved, but these lessons remain as relevant today as they were in the nineties.
