Why Power Generators Can’t Ignore the Ukraine Cyberattack

Although the December attack on Ukraine's power infrastructure mostly affected the distribution grid, generators are just as vulnerable to cyberattack, in part because they tend to rely more on outside contractors working remotely. Here's the latest on the attackers' path and methods, areas in generation that are potentially vulnerable, and recommendations to address the vulnerabilities.

December 23, 2015: Kyivoblenergo Control Room, Kiev Region, Ukraine. It's late afternoon, and a grid operator is sitting at his workstation, monitoring the flow of power in the local distribution grid of the Kiev Oblast (Ukrainian province). Around 3:30, the operator notices the mouse cursor is moving. Grabbing the mouse, he tries to move it out of the way, but the cursor continues to move on its own. The cursor's destination is a set of 35kV and 100kV substations that provide power to almost 80,000 customers (Figure 1).

1. Notice of hack.This web-translated announcement from Kyivoblenergo announces its control system was hacked, but the post was deleted a few days later. Courtesy: Chris Sistrunk

With deliberate action, a remote individual had taken control of the operator workstation and was systematically opening breakers at 30 substations. Opening breakers doesn't take much time; it was all over in 5 minutes. For the people disconnected, and the utility personnel deployed, recovery would take considerably longer.

Kyivoblenergo wasn't the only utility hacked remotely; two more control centers suffered similar compromises, blacking out a total of 225,000 customers for over 6 hours in cold weather. Prior to opening the breakers, attackers reconfigured battery backup systems, disabling the automatic transfer functionality. Once the breakers opened, those backups failed to keep systems online, and placed operators in the same darkness. And while some details differ, a software component called KillDisk was also used to fully wipe the hard drives of corporate and control systems, requiring time-consuming reinstallation of the operating system and other important software.

Operators discovered vital devices at the individual substations had been reprogrammed with invalid firmware, denying remote control capability. The Ukrainian distribution control centers had been completely blinded and were relying on voice radio and cellphones for communication, using pen and paper to manage their local distribution grids.

The Ukrainians, according to trustworthy reports, had a reasonable security infrastructure in place. They had a perimeter, limited access for both data and personnel, and good logging of ingress and egress from the control system. However, their remote access infrastructure used only a single means of authentication: a password.

The root cause teams concluded the original infection was via malicious Microsoft Word and Excel documents sent to employees of the utility via public email. When opened on corporate systems, the Office documents would install malware that would spy on users and report that activity to attackers on the Internet. That malware captured usernames and passwords from the corporate desktops of remote personnel, and attackers then used these stolen credentials to access the control system. Once on the control system, the attacker had full access-and used that access to great effect.

Power Generators Have Similar Vulnerabilities

The Ukraine cyberattack co-opted the authorized remote access already in place for remote maintenance and used this access to operate substation breakers using the human-machine interface. The utilities hit by the attackers used reasonable perimeter controls, but the attackers spent enough time compromising the corporate infrastructure to effectively gain access with the usernames and passwords of authorized control system personnel. The basic process followed by the attackers is shown in Figure 2.

2. Remote access attack.The Ukraine grid attack was made possible by a simple first step. Courtesy: Michael Toecker

This attack was similar to monitoring the service entrance of an important building and watching employees enter their passcode for access to the building. Eventually, the attacker will see the PIN code and will be able to enter the building.

Generation personnel have become familiar with remote access to the plant via the Internet, because many generation facilities are very reliant on having remote personnel service and troubleshoot issues. In an era of increasingly small profit margins for many generators, 24/7, always available, remote access by critical personnel can mean the difference between being in the red and being in the black.

Additionally, there are now service providers that offer remote services to support generators from afar. These services range from very mundane predictive maintenance to full engineering and control system work, all done via the Internet, all using some form of interactive remote access. Service providers in this area are even more interesting because they have the potential to access many different generation facilities remotely, and, presumably, use the passwords for that remote access from their own corporate systems.

Remote Access Vulnerability in Power Generation

When vulnerability assessments are conducted on generation systems, it's the North American Electric Reliability Corp. Critical Infrastructure Protection (NERC CIP)-regulated entities that are the most resilient to this sort of attack. NERC requires two-factor authentication, a method stronger than just a password. The two-factor approach is very resistant to password stealing, as it requires a password plus access to a physical token (Figure 3). An attacker can steal the password by infecting your desktop system, but the physical token cannot be captured as easily. There are other means of providing two-factor authentication, but this is the standard NERC implementation used by many utilities and owners of power infrastructure.

3. A two-factor authentication token. Courtesy: Wikipedia Contributor Kharitonov

The physical token generates a unique string of numbers every 30 to 60 seconds. The unique numbers are the output of a time-based function, which is also stored and computed on your security systems. When users want remote access, they enter their username, password, and unique numbers. If all three are correct, then they are allowed into the system.

For generation facilities that are non-NERC, of which there are many, the vulnerability assessment is not as promising. Without the NERC requirements for two-factor authentication, many generation facilities have extremely poor security for their remote access, generally one of the methods shown in Figure 4.

4. Risk of various remote access methods.Source: Michael Toecker

Because the remote access infrastructure isn't a direct part of the revenue stream, generators treat remote access with a very casual attitude. On the one hand, they will often consider remote access to be a vital business need, paramount to maintaining production by allowing personnel to troubleshoot from anywhere, at any time. Then, often in the same breath, generation owners will refuse to invest in a reasonable and standards-based method of ensuring that their vital remote access is reliable and secure.

If engineers bought and installed protective relays with the same attitude used when buying infrastructure for remote control via the Internet, there would be more broken and unreliable generation facilities. A poor protection relay might work pretty well under normal conditions, but it would likely fail to protect equipment in a real-world event, where conditions are less predictable. The same goes for poor cybersecurity infrastructure. The industry has been lucky that motivated attackers are spending more time on credit cards and personal data than on compromising control systems.

The Vendor Challenge

Remote service providers/vendors actually present a larger risk than those existing at an individual power plant. Vendors who remotely access generation facilities usually have contracts with multiple facilities and routinely install their own methods for remote access to the site. Methods of access provided by remote vendors are often very poor. Vendor-installed remote access gains this distinction by a combination of:

a- Poor proposal requirements for security.

a- A vendor's need for fast, simple, and cheap-where security isn't welcome.

a- A budget need to award the support contract to the cheapest vendor.

Here's one example (some details have been changed to obscure the vendor and implementation). Years ago, I was conducting a vulnerability assessment at a facility where the remote access router had been installed for use by the facility's distributed control system (DCS) vendor. It used a strong virtual private network (VPN), the equivalent of a deadbolt on a good steel door, and used a physical key to enable and disable remote access. While it was not two-factor authentication, this was a reasonable method, on paper, for remotely accessing the facility. Vendor personnel would call in and request access, and an operator would "turn the key" in order to allow the vendor to access the systems. At the end of the need, the vendor would call back in and let the operator know to disable the remote access by literally turning the key to the lock position.

Problems with this method became apparent quickly:

a- Facility personnel routinely forgot to turn the remote access off, so it would stay enabled for months at a time, directly connected to the vendor's internal network via that same VPN connection.

a- The vendor would often forget to call the operators when the work was complete.

a- There were also no procedures specifying time limits for the remote access, nor any regular checks to ensure it was turned off when not in use.

a- The equipment was not open to inspection by the generator-it was part of a secret sauce in the remote support market.

On paper, the physical key method looked good. But when performing a walkdown, I noticed a serial cable coming out of the key box that led to a dial-up modem. This cable allowed direct access to the internals of the remote access router, which could completely reconfigure the device to permit access. This created a glaring hole, considering that a phone call to the remote access router could enable remote access just the same as an operator turning a key, or could completely disable the VPN to put the router directly on the Internet.

What's more, discussions with the vendor on reviewing the configuration of the device were unproductive. The device, the VPN connection, and the dial-up modems were all considered vendor property and were off limits. This particular vendor has come a long way recently in terms of remote access, but the early days and conversations were particularly difficult. Persuading them that a malicious individual could take advantage of these weaknesses was a long process.

Situations like this still exist today. Not more than nine months ago, I reviewed another site's remote access system from a different vendor. In this case, I was able to conduct a full review of the vendor "remote access solution" and was completely blown away. In the course of installing the remote access, a technician had completely disabled most of the safeguards while trying to get it working and had left it that way. Originally, the remote access solution was decent, but now it was completely open to compromise. Needless to say, that remote access systemwas disconnected swiftly.

An Action Plan for Secure Remote Access

From my experience working at several generators on multiple DCS and programmable logic controller (PLC) systems, once an attacker is within the perimeter, the fight is over. It doesn't matter if you're fully compliant with NERC regulations either, because the current state of security capability in control systems is woefully poor. The perimeter is the only effective means of protecting these networks, and many generators allow remote personnel to come in from anywhere, at any time, for most any reason.

Taking care of the following tasks will put your plant in a more secure position.

Task 1: Get a Remote Access Policy in Place. If remote access is critical to your generating business, treat it as such. Place limits on when remote access is allowed, who has the capability, and by what method access is obtained. If the generator does not set the policy, individuals and vendors will set their own policies, using their own individual risk evaluations, which can leave a plant vulnerable.

Task 2: Invest in a Reasonable Infrastructure. There are many resources for developing a good perimeter, and the NERC CIP Version 5 standards require a perimeter for all generation assets, regardless of whether they are Low or Medium:

a- NERC: Guidance for Secure Interactive Remote Access (bit.ly/1WUiQVd)

a- National Institute of Standards and Technology: Guide to Industrial Control Systems Security (1.usa.gov/1TgbSef)

a- SANS Industrial Control System: Defense Use Case #5 (bit.ly/1WUjoub)

a- NERC: NERC-CIP-003 Version 6, See Requirement 1.2 for Low Impact facilities (bit.ly/1VRkQPJ)

Task 3: Use Two-Factor Authentication. Properly implemented, two-factor is the best method of ensuring remote access to the control system so that passwords cannot be stolen via simple methods. Ideally, this should be related to Task 4 by keeping the two-factor token in the hands of operations personnel and not giving it to the remote access individual. Additionally, a remote access timeout should be used; no personnel should be able to stay continually connected to a remote control system.

Task 4: Require Authorization by Operations Personnel. When personnel with remote access are performing work on the system, operations personnel should not only be aware, but should also be actively responsible for authorizing each remote access. This could be done by operations holding the tokens used for authentication and only giving PIN codes when the requirements are met, or via some other technical control.

Task 5: Configure and Respond to Remote Access Alerts. Logs of remote access attempts, both successful and denied, should be reconciled against the remote access authorized by operations personnel on a regular basis. Discrepancies should be promptly investigated as potential incidents.

Don't Turn a Tool into a Weapon

Generation facilities are important to the stability of the grid, and they have a duty to both the public and stakeholders to take reasonable measures to protect themselves from malicious actors. While remote access is important, and often vital, to the bottom line, it is also a means that can reduce profitability and increase risk.

Do it right, or don't allow remote access at all. a-

- Michael Toecker, PE (toecker@ context-is.com) specializes in the security of industrial control systems, particularly those in power generation and transmission.

The post Why Power Generators Can't Ignore the Ukraine Cyberattack appeared first on POWER Magazine.