In the fast-paced world of business IT, security is undeniably critical. However, when information security (Infosec) teams operate without coordination or consideration for the broader IT landscape, they can become a significant liability rather than an asset. This article explores a real-world example where uncoordinated vulnerability scanning by Infosec led to severe disruptions in business operations, highlighting the importance of balancing security with operational stability.
The Incident | A Daytime Disaster
The incident began innocuously enough—Infosec decided to conduct vulnerability scans on devices across the network. While this might sound like a routine security measure, the timing and execution were anything but routine. These scans were initiated in the middle of the business day, with no prior notice to the IT team or any consideration for the critical operations taking place.
One of the primary targets of these scans was the company’s Cisco Catalyst switches. Unbeknownst to Infosec, these switches had a known bug that could cause them to reboot if certain SNMP (Simple Network Management Protocol) requests were triggered—a vulnerability that the Infosec team inadvertently exploited. The result? Switches rebooted across the network, causing widespread disruption.
The Fallout | A Catastrophic Impact on Business Operations
The timing of this uncoordinated scanning could not have been worse. Among the most affected areas was the company’s dealing room, where critical trading operations were taking place. The unexpected network disruption caused by the rebooting switches led to significant downtime, with traders unable to execute transactions. In the world of trading, where seconds can mean millions, this was nothing short of a disaster.
As the network administrator responsible for the stability of IT operations, I was immediately alerted to the chaos unfolding. Upon investigating, I discovered that the source of the problem was the Infosec team’s vulnerability scanner. Without hesitation, I disabled the switch port connected to their scanner, effectively halting their scans and preventing further damage.
The Aftermath | A Clash of Priorities
Following the incident, I reported the Infosec team for violating change management protocols. Their response was dismissive—they claimed that they didn’t need change requests to conduct vulnerability scans and that I should be thankful they were taking the initiative to secure the network. This cavalier attitude was alarming, particularly given the damage their actions had caused.
In response, I made it clear that the company had a board-approved policy requiring changes to be coordinated and approved through the proper channels. If they wanted to resume their scans, they would need to obtain a board resolution to bypass this policy. Unsurprisingly, this marked the beginning of a strained relationship between IT and Infosec.
The Bigger Picture | The Importance of Coordination
This incident underscores a critical lesson: security cannot operate in isolation. While vulnerability scanning is essential for identifying and mitigating risks, it must be done in a way that does not compromise the availability or integrity of the network. The Infosec team’s failure to coordinate with IT and consider the broader impact of their actions resulted in a preventable business disruption.
A key principle of cybersecurity is the CIA triad—Confidentiality, Integrity, and Availability. However, many Infosec professionals become so focused on confidentiality and integrity that they overlook availability, which is just as crucial to the business. A secure system is of little value if it is unavailable when needed, as was painfully demonstrated in this case.
Lessons Learned | The Need for a Holistic Approach
This experience highlighted several key points that are often overlooked in business environments:
Change Management Matters: Even security teams must adhere to change management protocols. This ensures that all stakeholders are aware of potential impacts and that risks are mitigated appropriately.
Timing Is Critical: Vulnerability scans and other security measures should be scheduled during off-peak hours to minimize the impact on business operations. This is particularly important in environments where uptime is critical, such as trading floors or financial services.
Collaboration Is Key: Infosec cannot operate in a vacuum. Security initiatives must be coordinated with IT and other relevant departments to ensure that they support, rather than hinder, business operations.
Security Is Not Just About Protecting Data: While protecting data is crucial, ensuring the availability of systems and services is equally important. Infosec teams must consider the full scope of the CIA triad in their efforts.
Wrap | Avoiding the Next "Clownstrike"
The recent "Clownstrike" incident, where a single cybersecurity vendor’s failure led to widespread IT outages, serves as a stark reminder of the risks of uncoordinated security efforts. Just as relying too heavily on a single vendor can be disastrous, so too can allowing Infosec teams to operate independently without considering the broader IT landscape.
Wrapping, while security is a critical aspect of any business, it should not come at the cost of operational stability. A holistic, coordinated approach that balances security with availability is essential for ensuring that the business remains both secure and functional. As this incident demonstrates, failure to do so can have severe consequences for the business.
Ronald Bartels ensures that Internet inhabiting things are connected reliably online at Nepean Networks - the leading specialized SD-WAN provider in South Africa. Learn more about the best SD-WAN provider in the world! 👉 Contact Nepean
