Richard Clark is a Technical Specialist and Controls Engineer at InduSoft concentrating on cybersecurity, 3rd party product integration, specialized application development, and product marketing.
Mr. Clark has been in Automation, Process System, and Control System design and implementation for more than 25 years and was employed by Wonderware, where he developed a non-proprietary means of using IP-Sec for securing current and legacy Automation, SCADA, and Process Control Systems, and developed non-proprietary IT security techniques. He is an industry expert by peer review and spokesperson on IT security; consultant, analyst and voting member of ISA- SP99. He is a contributor to PCSF Vendor Forum, consultant to NIST and other government labs and NSA during the development of NIST Special Publication 800-82. He has published engineering white papers, manuals, and instruction documents, developed and given classes and lectures on the topic of ICS/SCADA Security.
Q: Can you tell us more about your background and how you got interested in industrial automation? How did you get interested in SCADA security?
In the mid 1970’s I worked for robotics company and did a lot of CNC and Step-to Step programming, and installations of massive equipment, then learned 6502 Assembler, Cal Comp Assembler, and 8086 Assembler, ANSI C and did a custom project that that involved developing and building a color HMI for a repurposed giant X-Y plotter, along with developing the PC interface to its mini that ran the analog electronics. It eventually cut glass and painted Suncatchers, those round things that people used to hang in their windows made out of colored glass. By this time in the industry, the Modicon PLC had become firmly embedded in production systems all over the world and most of you know how industrial automation progressed from there.
Security for automation caught us by surprise. And while I used to hack assembly code, back when hackers were the “good guys”, the medium of transport for the sharing of information was only then getting really mature in the early 1980’s. What was then known as “The Internet” was really designed by the U.S. government to share info between academics and universities, and helped, among other things, to spur on the Space Race. But that’s when it all went weird.
The internet exploded and technology could not keep up with the use vectors that people were finding for the medium… including eventually interconnecting automated industrial machinery using many protocols, trying to break out of the “proprietary” box. Until Wonderware came along, really, almost all HMIs were using some sort of a text-based video card and iffy drivers. Windows 3.0 could barely be called a graphical user interface and was unstable, and indeed the first version of InTouch was very primitive. Programming any machine to machine communications and data collection at that time was very proprietary and tricky.
Why would you possibly need security with systems as arcane as this? What good was the control data to anyone? Well, somewhere in 1998 or 1999, I visited a milk formula canning facility in Northern Denmark, right on the North Sea, and fortunately, only the canning/labeling section of the plant was automated and interconnected with some HMIs that were also visible on the corporate Ethernet. The poor plant tech was pulling his hair out trying to stay ahead of the HMIs that were crashing, and couldn’t figure out what was wrong. The HMIs of course, were PC based, and with the advent of PCs in the previous 10 years, people were now using them at home, (usually without virus protection, which we called “barefoot”) and became quite familiar with them. Finding an identical machine at work, many employees started surfing the internet, or playing games on them, mostly at night, unknowingly finding all manner of viruses and Trojans, which became embedded in every machine in the business and production facilities. The only solution was to pull the data plug to the outside, reformat everything, disconnect from the corpnet, and lock down the HMIs. The poor Plant Tech there and many others that I subsequently met were not trained to understand the needs of this interconnectivity and cybersecurity and so, in the late 1990’s, this is when IT departments reluctantly got thrown into the mix of machinery vs security. This was also the moment that got me interested in Industrial Control System Security.
Q: Traditionally, there was complete separation between the operational technology (OT) network(device and control processes) and the information technology (IT) network (business and desktop applications). Today, companies are bridging the air gap to get benefits like remote monitoring and administration. However, how does this impact security?
The research and lab work that I did with an IT security guy in the early 2000’s and the meetings with old ISA-SP99 and early CERT working groups confirmed that there never was anything that could be called an “air-gap” in industrial control system security, except in the very early proprietary days of Industrial Control Systems. There was this prevailing belief that industrial control systems containing PC based HMIs and servers were air gapped. Ultimately this always proved to be false, because the infection vectors were always far greater than anyone anticipated. These problems were actually exacerbated when IT departments started assuming that HMI PC boxes, Server boxes, and networks were their domain, and unknowingly in many cases, started consolidating them with the business networks without proper training or understanding of what was going on inside the ICS.
The main thing that was not understood was that the enterprise contained within industrial control systems were actually an “end point” or a single machine, and needed to be secured from that standpoint. This involved an attempt at “air gapping” but ultimately it was always foiled by live USB ports, CD ROMs that were active, live telephone connections to internal modems that integrators had left connected or used for telemetry from tank or pumping farms, etc. These particular entry points could be violated by a number of methods; anything from the introduction of viruses and Trojans to social engineering to forcing the modems into a maintenance mode to give command line access to the machine containing it.
As the Windows platform evolved through the 2000s the security profiles changed on the individual machines, that were heretofore peer connected, and needed to become user authenticated through a domain controller or other means of access control. The definition of what user authorization actually means boils down to 3 factors: Something you have, something you do, something you know. This also applies to a machine or virtual users. Additionally, and most recently due to the benefits that you mentioned, authentication factors augment these authorization factors to include location like GPS or a physical machine screen spot-generated bar code, and Time of Day. There are more, but these are the ones that are mostly used right now.
Q: According to Dell’s 2015 Security Annual Threat Report, the threats against SCADA systems more than doubled from 2013 -2014. What do you think SCADA system should do to protect themselves?
The cyber-incidents that occurred in 2015 not only doubled, but were more invasive and damaging. We saw massive breaches into government systems and databases, health care systems, customer data and credit card info was stolen from retailers and, on the industrial side of things, a steel factory in Germany was damaged. The trend is very disturbing.
Most IT departments still believe that they can manage these breaches, but it is clear that the ability to fend off cyber-attacks by company IT departments is widely uneven. Part of the issue is that IT departments still do not understand that control system protocols need to be evaluated and protected. Most virus/malware protection simply ignores these protocols.
There are many external and Plug-and-play appliances available that actively work within industrial control systems that can detect surreptitious activity, block ports, sandbox questionable devices, etc. Additionally, a properly developed SCADA system can serve as a “gatekeeper” of the data and processes, allowing only proxied data out to parties or machines that need to see it.
Q: One thing I found interesting is that Buffer overflow vulnerabilities were the primary method used to attach SCADA systems, accounting for 25% of the attacks. How can someone protect themselves from Buffer Overflow vulnerabilities?
This is an old problem that doesn’t seem to go away. The issue is that in virtually every case of a buffer overrun cybersecurity breach, the PLC or machine had not been patched to the latest software. Older machinery, on the whole, is vulnerable to buffer overflows because no one ever thought at the time it was built that overflow data would ever be input, so tests for this use case were never performed. Properly written software nowadays will not allow buffer overflow failures, and if there is any doubt when you are developing your own SCADA system, you should perform this use case. Again, many of these plug-and-play security appliances that can be inserted into the industrial control system enterprise protect vulnerable equipment, especially legacy equipment, that could be affected by this.
Q: One point that people don’t bring up, but seems like an important issue in SCADA security is the aging industrial machinery infrastructure. How do you think we can make people aware of the need to upgrade their systems?
There’s this old adage about beating a dead horse, and it is my hope that this problem would have eventually gone away through attrition or example and concern for self-preservation. Sadly, many systems will not get updated until the equipment either fails and there is no recourse but upgrade. Sadly, most people will not upgrade until they are forced to or they suffer massive losses related to not doing it earlier.
The good news is that because there is so much aging industrial machinery out there, we know that the market will open up at some point when it becomes entirely impractical or too expensive to maintain the old machinery, or that it simply is not compatible with newer and more efficient ways or doing the same thing.
Q: What are the top-three threat vectors you are most concerned with?
The top three threat vectors are:
1) Contractors. Contractors are usually not as motivated to care about a customer’s infrastructure as they could be. In virtually every case, the cybersecurity and/or data handling of the vast majority of cybersecurity incidents last year was due to the fact that the contractors had a back door that was exploited, they left something undone that opened a vulnerability, or something similar. It is important to vet your contractors thoroughly and also make sure that they are bonded and insured to the amount that could possibly be lost by a mistake or oversight being made on their part.
2) Low Tech. This is often the most underestimated cybersecurity breach that occurs, and many people never even run use cases against such events. Things like managers with administrator credentials that allow unknowledgeable maintenance people into systems that they are not trained for. Safety systems that are overridden for convenience sake. Back doors that are left in software. Persistent passwords or the use of passwords that are easy to guess are just a few things that come to mind.
3) Believing that your systems are safe and have not already been breached. An out-of-sight-out-of-mind attitude. This is by far one of the most persistent threat vectors in cybersecurity. Virtually every major cyber event in 2015 that has been reported could likely have been detected if such an attitude were adopted.
Because these threat vectors are not really “technological” per se, they are easy to incorporate into a cybersecurity culture that needs to be adopted by everyone. There are many tools available that can help anyone assess their systems and enterprise, and determine their cyber-readiness and provide assistance in closing or mitigating the gaps that are discovered. We have 2 eBooks available on Smashwords, and are also discussed in our corporate blogs. Each of them discussed the NIST framework, the CSET tool, using gap analyses, risk management, and more. The book “Framework for SCADA Cybersecurity” co-authored by Professor Stephen Miller of ENMU is used as a textbook for his Cybersecurity Certification courses, and this is a good place to start for anyone who wants to know more about the field. Go to www.enmu.edu and search for cybersecurity.
Q: Some of the ways that I have heard about protecting the security of your plant is to restrict USB ports, ensuring Bluetooth is disabled, making sure all software and systems are up to date and ensuring that the network only allows connections with approved IPs. What else can someone do to protect their network?
Again, the suggestions that I’ve already given are the best place to start. Since I didn’t already mention it, I want to plug our other Cybersecurity eBook also available for free from Smashwords, “InduSoft Application Design and SCADA Deployment Recommendations for Industrial Control System Security ” which has a compilation of InduSoft Security guidance that has been given in talks, webinars, whitepapers, interviews, and best practices.
InduSoft Application Design and SCADA Deployment Recommendations for Industrial Control System Security – This eBook provides guidance when building and implementing HMI and SCADA systems, and describes best practices to secure them against cyber-attacks and known vulnerabilities.
Framework for SCADA Cybersecurity – This eBook will provide Critical Infrastructure customers and academic students an understanding of the NIST Cybersecurity Critical Infrastructure Framework and how to apply the framework to new and existing SCADA applications and implementations.
Q: There are so many standards out there when it comes to SCADA security. Which ones should we be watching?
The main standards are the ISA/IEC 62443 Security for Industrial Automation and Control Systems All other standards will closely follow that. Additionally NIST and ICS-CERT have excellent guidance and standards that can assist.
Q: With the advent of Internet of Things, how can you limit vulnerability through effective technology?
This has to be done with the technologies I mentioned previously, authentication and authorization, including localization and other similar means.
The days of “security by obscurity” are gone, and it is importance to implement security programs specifically tailored for industrial systems and the operational technologies they utilize. What are some of the recommendations that you would give companies as they put together this security program?
The NIST Cybersecurity framework and the CSET tool available through ICS-CERT are great places to start. This is covered in detail in the eBooks.
Q: What do you see happening in SCADA security in 2016?
Much of the emphasis will be put onto the Internet of Things and being able to secure it. We may be using security methods that we don’t fully understand or aren’t aware of at this point. There are so many things that are going to happen in the next few years that it’s hard to predict how the industry will grow.
We have unbelievable technology that’s going to explode in the next 5-10 years. I think the biggest thing we need to watch out for is ourselves.
Q: How do you predict industrial automation will change in ten years?
I think the greatest changes will be the personalization of automation and the ability to manufacture things on the fly.
Q: What do you think customers will expect from future HMI/SCADA software?
Well, I am a bit biased, but I think InduSoft has really nailed it. I think they’re going to look for ease of use, value, accessibility, and basically everything that InduSoft has been developing. I’ve heard from so many people, including people from other industrial software companies, about how easy to use and flexible InduSoft Web Studio is, and this is one of the reasons that I love working here. Plus, everyone here is great.
Q: What advice do you have for someone entering the industry? What advice would you give to yourself 10 year ago?
I’d say, learn as much as possible. Things are changing so rapidly that it’s incredibly easy to become specialized, and if you do that the technology may disappear in a short period of time, and then you’ll have to move on to something completely new.
Broaden yourself as much as possible and learn as much as possible about everything you’re interested in.