An anonymous Western United States utility became the first to report a malicious “cyber event” that disrupted US grid operations. In a report highlighting the “lesson learned” from a past incident, NERC (North American Electric Reliability Corporation) said hackers repeatedly rebooted firewalls causing communication outages in multiple remote generation sites in the western United States, on March 5, 2019. This impacted utility belongs to Western Electricity Coordinating Council (WECC), the NERC affiliate that monitors grid reliability and security across western North America. This is the first reported malicious “cyber event” that disrupted United States grid operations. Unlike 2015 Ukraine cyber security attack, this attack didn’t cause any power disruptions, but only caused brief communication outages between field devices at sites and between the sites and the control center. In Ukraine 2015 cyber security attack, hackers were capable of disrupting power for about a quarter-million people for at least a few hours.
This Denial-Of-Service (DOS) cyber attack impacted firewalls deployed at a Low-impact control center and multiple remote low-impact generation sites in the western United States. Hackers exploited a vulnerability in the web interface of the firewall, which allowed an unauthenticated attacker to cause unexpected reboots of firewalls. NERC report shows that these firewall reboots occurred over a 10-hour time period with each firewall showing offline status for less than five minutes. The firewall served as a communication filter for data flowing between generation sites and the utility’s control center, so operators lost visibility to those parts of the utility’s supervisory control and data acquisition (SCADA) system each time the firewall rebooted. These communication outages at multiple sites raised suspicion and led to a more in-depth investigation.
The biggest problem in this attack is the fact that hackers were able to successfully take advantage of a known flaw in the firewall’s user interface. Even though the firewall vendor released a firmware update to address the exploited firewall vulnerability prior to this cyber incident, impacted utilities failed to apply those firewall security updates due to lack of a proper firmware review and update process. Timely firmware updates, vulnerability management and patch management are crucial to a proper cyber security posture in critical systems. Vendors are responsible for providing security updates and firmware updates to mitigate the risk of exploiting vulnerabilities in their devices. These security fixes are triggered by public vulnerability disclosures, internal vulnerability discovery and vulnerability reports (reported by customers and research institutes). It is vendor’s responsibility to clearly communicate these security updates to their customers in a timely manner. But the responsibility of continuously monitoring vendor releases and updating the firmware on devices lie with power utilities. It is important to stay up to date with security updates for the install base, and vet those security updates in a test environment (eg: replica system) before being deployed in the productions system.
A firmware update on an Operational Technology (OT) equipment could be challenging, as it depends on many factors such as hardware compatibility, End Of Life Cycle (EOL) notices, system architecture, device/system interoperability, firmware vetting process and lengthy acceptance test procedures. Firmware updates of assets in power utilities should be carefully managed to prevent any adverse effects on asset’s functionality and verified the impact of an update on the system through acceptance test procedures. To mitigate these operational risks of an update, first utilities should apply security updates or firmware updates on a test environment that would not impact operational assets and monitor the changes for any adverse effects. Only after ensuring that there are no adverse effects with the update, utilities can then deploy the firmware/security update at a non-critical operational site. After closely monitoring the operations and traffic in that non-critical production environment for a period of time, utilities can deploy the firmware/security update to all remaining BES (Bulk Energy System) assets that had common hardware with the firmware vulnerability.
The NERC “lesson learned” report calls on utilities to add layered defenses beyond firewalls to prevent future malicious cyber attacks on power utilities. With layered defense, it is harder to penetrate a screening router, a virtual private network terminator, and a firewall in series than just a firewall. And firewalls should be deployed in high availability/redundant pair configurations, and firewall rules should be configured to restrict traffic to the minimum only allowing traffic required to operate the assets. Further, power utilities can utilize Network Intrusion Detection Systems (NIDS) and Security Information & Event Management (SIEM) systems to add another layer of defense to protect their BES from cyber attacks.
This cyber security attack showed us that, it does not take a sophisticated attack to damage critical electrical infrastructure, and that’s scary. Critical power utilities are required to maintain awareness of vulnerabilities in their environments, check for and apply security fixes to sensitive grid assets and configuration software. If utilities could reduce their attack surface and only have few access points in their networks, they could enforce stronger security controls at those limited access points and monitor security of those access points continuously.
