Aug 10, 2018 - go to an OT focused conference where Moxa and N-Tron may be presenting new products or offering training. Therefore, the IT / OT separation ...
Implications of Implementing Software Defined Networking to Improve Cybersecurity for Operational Technology Networks
by Charles R. Tommey
A Capstone Project Submitted to the Faculty of Utica College
August 2018
in Partial Fulfillment of the Requirements for the Degree of
Master of Science in Cybersecurity
© Copyright 2018 by Charles R. Tommey All Rights Reserved
ii
Abstract Real-time business practices require copious amounts of data directly from the production assets. This new thirst for accurate and timely data has forced the convergence of the traditionally business-focused information technology (IT) environment with the productionfocused operational technology (OT). This research project explores the application of softwaredefined networking (SDN) within OT networks. The research found numerous differences between IT and OT network requirements as well as significant differences in priorities and approach to cybersecurity. The SDN methodologies benefit OT networks with enhanced situational awareness, centralized configuration, deny-by-default forwarding rules, and increased performance. Arguably, SDN brings the negative possibilities of an enlarged attack surface, increased complexity, and reduced reliability. While these negative implications were found to have partial or full mitigation strategies, some proponents of SDN contend that properly implemented SDN in an OT environment presents a reduced attack surface, decreased complexity, and enhanced reliability. The overall effects on OT personnel were found to be a net positive based on better situational awareness and ease of network design, implementation, and maintenance. This research concluded that management must find a way to align the IT and OT teams in support of cybersecurity for the entire organization. Implementation of SDN presents an opportunity to meld the IT / OT cultures while enhancing overall cyber-resiliency. While more research and testing of SDN technologies in OT environments is needed, the preponderance of existing research indicates that SDN is a powerful tool for enhancing OT cybersecurity. Keywords: Cybersecurity, Dr. Michael Sanchez, critical infrastructure, industrial control systems, ICS, zero-trust. iii
Acknowledgements Many people have contributed to the efforts that produced this capstone project and I want to acknowledge fully that this accomplishment is in no way mine alone. I owe many nights out and weekends away to my understanding and longsuffering wife Theresa. Honey, I am officially back from my two-and-a-half-year exile and intend to redeem you from your capstone widowhood. My mother-in-law, Linda, also helped tremendously by reading through my overexpressive and longwinded prose which originally contained more than a few run-ons. I also wish to thank several Utica College professors. Dr. Leslie Corbo for being an incredibly supportive student advocate and very welcoming professor for my first class in the Utica Cybersecurity program. Professor Vernon McCandlish for his detailed explanations, rich storytelling, and his ability to make dry subjects entertaining and memorable. Professor Chet Hosmer for providing an enormous amount of information and including many practical labs on data hiding which elicited much learning. Many thanks to my second reader, Roger Hill of Veracity Industrial Networks, who took a good deal of his very valuable time to review and comment on this paper. Conversations with Roger several years ago planted the early seeds of the research topics for this capstone project. To my friends and family, I also thank you all for understanding my lack of availability for the last several years. To my employer, A&E Engineering, the owner/founder Wright Sullivan, and my direct supervisor Joshua Ruppe, I want to express my gratitude for the flexibility to work around the workload of this master’s program. And finally, a big thanks to the U.S. government for providing the G.I. Bill that paid for 50 percent of this program.
iv
Table of Contents List of Illustrative Materials .............................................................................................. vi Introduction ......................................................................................................................... 1 Background .................................................................................................................. 8 Statement of the Problem .................................................................................................. 12 Expanding Need for Data from OT Networks ........................................................... 12 The Cyber-Threats are Increasing ............................................................................. 13 Current State of Cybersecurity for OT Networks ...................................................... 14 Electric Power Grid is a Current Focus ..................................................................... 14 Uniqueness of OT Networking Environment ............................................................ 15 Standards and Regulations ......................................................................................... 15 Shortage of Qualified Personnel ................................................................................ 16 Summary of the Problem ........................................................................................... 17 Purpose of the Study .................................................................................................. 17 Research Questions .................................................................................................... 18 Literature Review ............................................................................................................. 19 Available Literature ................................................................................................... 20 Determining the Unique Cybersecurity Characteristics of OT Networks ................. 20 Industry Standards and Government Regulators ....................................................... 22 Cybersecurity Vulnerabilities of OT Networks ......................................................... 25 How Does SDN Technology Address Cybersecurity in an OT Network? ................ 35 Effects of SDN Workloads on OT Network Personnel ............................................. 62 Summary .................................................................................................................... 64 Discussion of the Findings ................................................................................................ 66 Unique Attributes of OT Networks Affecting Cybersecurity.................................... 66 How SDN Enhances the Cybersecurity of OT Networks .......................................... 69 How SDN detracts from the cybersecurity of OT networks ...................................... 73 Effects of SDN on the Personnel Responsible for OT Networks .............................. 75 Summary of Findings ................................................................................................ 77 Recommendations ............................................................................................................. 80 The IT and OT Teams Must Work Together ............................................................. 80 Develop a Comprehensive Cybersecurity Program ................................................... 80 Implement an SDN Test Lab with an Eye Toward a Production Trial ..................... 81 Support Additional OT-Focused SDN Research ....................................................... 81 Encourage Standards and Regulatory Bodies to Investigate SDN for Cybersecurity 82 Summary of Recommendations ................................................................................. 83 Conclusion ........................................................................................................................ 84 IT and OT Networks are Significantly Different ....................................................... 84 SDN Effects on OT Network Cybersecurity Appear Largely Positive ..................... 85 SDN Effects on OT Support Personnel are Generally Positive ................................. 86 Final Thoughts ........................................................................................................... 86 References ......................................................................................................................... 88
v
List of Illustrative Materials Figure 1: The Purdue Reference Model as adapted to ISA-95 ..................................................... 23 Figure 2: SDN data flow paths from the switching and routing hardware. .................................. 38
vi
Introduction In 1832, on the floor of the United States Senate, Senator William Marcy coined the phrase; to the victor belong the spoils (The Editors of Encyclopaedia Britannica, 2018). Today, corporations have incorporated this maxim into their business practices. Efficient and highquality production, prompt deliveries, and quicker time to market with new products are all methods to defeat a competitor in business (Littlefield, 2015). Wouter Aghina, is an Organizational Design Practice leader with McKinsey & Company, where he helps companies master change, manage human resources better, and develop management structures to extract lasting value from corporate restructurings (McKinsey & Company, n.d.). In a 2015 interview, Aghina said, “Governance, for us, is about decision making. We need speed in decision making…” (Murarka, 2015, para. 24). In 2016, Gartner, a world-class research and consulting company, released a report that stated, “Real-time analytics enable faster, more precise and more effective decisions than conventional decisions made with stale data or no data” (Pettey, 2016). Pettey (2016) further emphasized how important it was that data be immediately available and that the speed of data acquisition was often the limiting factor for quick decision-making. This quest for immediate decision-making leads directly to a requirement for nearly instantaneous and highly accurate data (Littlefield, 2015). The proliferation of electronic data systems able to deliver accurate, real-time data finally makes real-time business management practical, allowing companies to react to changing customer demands, material and utility prices, or production conditions nearly instantaneously (Oxford Economics, 2011). Managers attempting to implement real-time business management methodologies depend on the rapid and accurate flow of information, both from and to as many parts of the business as possible (Leukert, 2017; Oxford Economics, 2011). In a manufacturing firm, one of
1
the main sources and destinations for information flows is the manufacturing process itself. Data residing in manufacturing systems include inventory levels, process line utilization, machinery health, quality reject rates, production counts, maintenance downtime, and current usage rates of utilities, such as electricity, water, and natural gas (Hughes & Littlefield, 2018). All this information and much more are critical to business managers and can change on an hourly or even minute-by-minute basis. Therefore, management must have the data and have it promptly and accurately (Littlefield, 2015). Businesses incorporate these disparate pieces of data into key performance indicators and reinforce their importance by displaying them on large screens in the production areas or as dashboards on managers’ computers (Wayne W. Eckerson, 2011). The revamping of manufacturing plants to automatically provide real-time data and accept real-time adjustments is known in various parts of the world or in different industries by several competing terms; this paper will use the term digitalization when referring to the overlapping concepts and technologies of Digitalization, Industry 4.0, Smart Factories, or the Fourth Industrial Revolution (Leukert, 2017; McKinsey Digital, 2015; Schwab, 2016). The digitalization campaign and associated requirements for actionable information drive the adoption of technologies that generate, transfer, store, analyze, and visualize data. All these technologies either rely on or can most easily be implemented using Ethernet based networks (Bayda, 2018). Ethernet standards, originally developed in the 1970s, are now the networking foundation for the Internet and nearly every corporate or home network installed in the last 30 years (Spurgeon, 2000). The computer-controlled automation, which revolutionized manufacturing starting in the 1970s, is rapidly adopting standard commercial computers and Ethernet based networks to reduce costs and requirements for specialized knowledge (Jacinto, 2010).
2
Industrial Control Systems (ICSs) are the electronic brains of the automated infrastructure on which our 21st century society depends. Engineers deploy ICSs anywhere automation can improve safety, quality, or production rates; this includes factories, chemical process facilities, pipelines, transportation systems, and energy production plants. A modern ICS includes one or more Ethernet networks that allow fast and efficient communication between various sub-systems (Bayda, 2018). The Ethernet networks directly connecting parts of an ICS are referred to as Operational Technology (OT) networks (BAE Systems Applied Intelligence, 2014). These OT networks quickly transport large flows of data and are essential to the digitalization of all industrial sectors. While OT networks incorporate standard Ethernet hardware to speed data transfer and keep costs low, they carry special limitations, requirements, and challenges related to their use in hazardous areas, human safety systems, and tough reliability requirements (Cisco Systems, Inc., 2008). Unfortunately, these networks bring all the same cybersecurity risks, complexities, and specialized staffing requirements of businessfocused Information Technology (IT) networks (Gregory-Brown & Harp, 2016). Regulatory and standards bodies employ cybersecurity maturity models to help companies gauge their current compliance with standards and encourage those companies to take small steps toward increasing their OT cybersecurity readiness (DOE, 2014). The DHS Control Systems Security Program (CSSP) asserts in their 2009 primer that increasing cybersecurity readiness leads directly to a reduction in overall risk to control systems. Historically, most industry sectors, especially those not tightly regulated, have shown poor-levels of cybersecurity maturity. These low-levels of maturity are manifested in a lack of management sponsorship for comprehensive and continuous cybersecurity improvement programs, lack of qualified personnel, and lack organizational goals based on measurable metrics. According to a 2018
3
report sponsored by anti-malware software company, Kaspersky Labs, three-fourths of respondents rank IT/OT cybersecurity as a major priority (Schwab & Poujol, 2018). Yet, only 23 percent of the 320 surveyed OT/ICS cybersecurity professionals claim to be compliant with the basic cybersecurity requirements and regulations for their industries (Schwab & Poujol, 2018). Encouragingly, 56 percent of the respondents expect cybersecurity budgets to increase over the next two years while another 35 percent expect their budgets to stay level (Schwab & Poujol, 2018). However, the current pitiful state of readiness leaves many OT networks and their ICS’s vulnerable to attack with malevolent actors and their methods growing more sophisticated, daily. In June of 2010, cybersecurity researchers discovered the first known, physically destructive, cyber weapon targeted at an industrial process (Zetter, 2014). Christened the Stuxnet worm, this complex and sophisticated malware forever revealed the vulnerabilities of modern control systems to the world (Farwell & Rohozinski, 2011). Stuxnet specifically targeted the Siemens SIMATIC S7 Programmable Logic Controllers (PLCs) and WinCC Supervisory Control and Data Acquisition (SCADA) systems responsible for controlling and monitoring the high-speed centrifuges essential to the uranium enrichment process at a government facility in Natanz, Iran (Farwell & Rohozinski, 2011). The Natanz S7 PLCs were responsible for controlling the valves feeding uranium hexafluoride to the centrifuges at the correct rates and pressures; additionally, the S7/WinCC control system precisely controls the rotational speed of the centrifuges (Langner, 2013). If the centrifuges spin too fast, the bearings can fail or too much internal gas pressure can be created; if they spin too slowly the enrichment will take much longer or not happen at all (Langner, 2013; Zetter, 2014). Langer (2013) also notes that high-speed centrifuges have resonate rotational
4
speed ranges that may cause damage to the rotors when accelerating and decelerating through those ranges on startups and shutdowns. Part of the Stuxnet attack code caused the centrifuges to slow to 120 rpm and then reaccelerate to full speed of 63,000 rpm; while another section of Stuxnet code caused centrifuges to accelerate from their normal 63,000 rpm to 84,600 rpm; these maneuvers stressed the centrifuges in ways that caused random failures of different types which masked the control system as the root cause (Langner, 2013). The Iranian PCS 7 network was carefully architected and isolated from the outside world with no known connection to the Internet (Langner, 2013). This is described as an air-gapped network security configuration (Farwell & Rohozinski, 2011). In an amazing display of technical sophistication and bold spy-craft, Stuxnet was crafted to breach the air-gap via infected USB flash drives that were carried into the enrichment plant by Iranian ICS programmers (Langner, 2013; Zetter, 2014). The audacious attack leveraged stolen digital certificates and an undisclosed Windows bug (known as a zero-day vulnerability) which tricked the target computer into accepting the USB drive as a human interface device and allowing automatic code execution. Contractors used the USB drives to transfer updated control system applications from the contractors’ development computers to the running PCS 7 control system computers (Zetter, 2011). However, since these USB drives were compromised with the Stuxnet code when they had been plugged into the development computers belonging to Iranian ICS contractors, the Stuxnet worm was also transferred to the PCS 7 computers (Zetter, 2011). Computers at four separate ICS contractors were targeted by Stuxnet’s developers (Zetter, 2011). Zetter (2014) claims all four contractors were carefully selected due to their Siemens experience and previous known work at Iranian industrial sites. Stuxnet eventually infiltrated the SIMATIC S7/WinCC based ICS controlling thousands of the delicate, high-speed,
5
enrichment centrifuges (Zetter, 2014). Langer (2013) details how the Stuxnet malware then wreaked havoc by changing application programs and even rewriting the firmware on the controller for the centrifuges. By taking control at such a low level, the malware was able to open and close valves, command the centrifuges to speed up and slow down, while communicating information back to its designers via the same USB flash drives and laptops (Langner, 2013; Zetter, 2014). Part of the genius of Stuxnet was the incorporation of the ability to maintain normal readings on the operator displays, while mayhem was playing out in the centrifuge halls. This ability was the result of a man-in-the-middle attack executed by wrapping the legitimate communication code with code that could control and modify the data going to and from the PLCs and SCADA computers. These features both extended the length of time Stuxnet was able to stay effective, undetected and caused Iran to delay introduction of thousands of new centrifuges while they tried to understand why the existing centrifuges were failing (Langner, 2013; Zetter, 2011, 2014). Ultimately, Stuxnet was able to cause the destruction of 2000 or more centrifuges while delaying the start of enrichment in thousands of additional centrifuges (Zetter, 2011). This elaborate plan might have gone on longer if the creators of Stuxnet had not made the worm too good at spreading. Over a 12 to 18 month after its initial release, Stuxnet programmers upgraded the worm several times, eventually endowing it with four separate methods to propagate itself automatically (Zetter, 2011). After those updates, Stuxnet escaped from the original four target companies and began infecting computers around the world. Eventually, Sergey Ulasen, a researcher in Belarus at the little-known cybersecurity firm, VirusBlokAda, discovered the Stuxnet malware (Zetter, 2011). A client of VirusBlokAda, thought to be completely unrelated to the Iranian uranium enrichment effort, e-mailed Ulasen
6
with a description of a computer caught in a reboot loop, shutting down and then booting up again, continuously (Zetter, 2011). Zetter (2011) explains that as Ulasen’s team dissected the files on the malfunctioning computer’s hard drive, they found a large, sophisticated piece of malware. After notifying Microsoft that Stuxnet was using a previously unknown bug to propagate, the team posted their initial findings in a cybersecurity forum (Zetter, 2011). Brian Krebs, a well know cybersecurity researcher with a large following, found the forum entry and posted details about the Stuxnet worm on his blog (Zetter, 2011). Krebs’ post initiated a worldwide race among cybersecurity research firms to unravel the secrets of the fascinating malware (Zetter, 2011). The facts came quickly, but it took a while to understand that all the technical sophistication and detailed targeting were aimed squarely at a specific SIMATIC S7/WinCC control system. Zetter (2011) exposed how clues buried in the Stuxnet code pointed toward a system isolated from the Internet, with a distinct set of Siemens controllers and Finish and Iranian manufactured variable speed drives, which were controlling valves and high-speed motors arranged in sets of 164 with a standard operating speed of 1064 Hz. Over several months, it became clear that the target had been the Iranian centrifuges at the Natanz enrichment plant (Zetter, 2011). While neither the United States (US) nor Israel admitted involvement, cybersecurity researchers claimed this sophisticated worm was created by the two countries working together to frustrate Iran’s uranium enrichment program (Alexander, 2016; Nakashima & Warrick, 2012). Cybersecurity researchers quickly realized that Stuxnet had inadvertently shown what was possible and that ICS’s were now fair game for all malevolent cyber-actors (Farwell & Rohozinski, 2011). The proverbial genie was out of the bottle, and it was never going back.
7
Background Cybersecurity is an exceedingly broad term that attempts to capture all the efforts deployed to prevent computing systems from being attacked and exploited. Cybersecurity experts define exploitation in three categories: theft of data, unauthorized alteration of data, and interference with the intended use of the computing or networking resources (Bodungen, Singer, Shbeeb, Hilt, & Wilhot, 2017). Cybersecurity practitioners label these aspects respectively as Confidentiality, Integrity, and Availability, which is also known as the CIA triad (Bodungen et al., 2017). Comprehensive and effective cybersecurity must adequately address all three areas. Cybersecurity in an OT network. The primary goals of network administrators working in OT environments are providing a safe working environment while maximizing manufacturing up time known as availability. Administrators’ secondary goal is ensuring the integrity of the stored data and finally protecting the confidentiality of the stored data. On the other hand, IT network administrators are more interested in maintaining the confidentiality of the data, and then the integrity of the data and finally their focus is maintaining a high degree of availability (Smith, Kipp, Gammel, & Watkins, 2016). Due to these differing priorities, IT and OT cybersecurity professionals often arrive at opposing opinions as to the best methods and resources required to secure a network. These differing priorities affect the basic approach to cybersecurity in OT networks and can cause friction when corporate IT cybersecurity practitioners attempt to help their OT counterparts (Lobo, 2018). These differences in priorities represent a threat to the manufacturing process. Multiple government agencies and standards bodies define best practices for cybersecurity within the OT environment to help educate cybersecurity professionals on the different architectures, unique priorities, specific limitations,
8
and best practices for OT networks (CSSP, 2009; DHS Cyber Security Division, 2016; ISA, 2007; Stouffer, Falco, & Scarfone, 2011; NCCIC, 2015; Toth, 2017). Special considerations in OT networks. Regulatory and standards organizations that concentrate on OT networks, point to the distinct requirements, limitations, and challenges that OT networks present to existing cybersecurity methodologies, plans, and tools (Stouffer et al., 2011). These unique circumstances require many modifications to the cybersecurity solutions that have been implemented in IT networks for many years. A simple example is an Anti-Virus program that runs on every server or client on a corporate IT network, may use too many local computing resources (processor cycles or memory) when running a scan and can cause a process to shut down or momentarily blind the operators of a manufacturing process (Falco, Hurd, & Teumim, 2006). Another unfortunate issue in OT networks is the presence of many older and unsupported operating systems; a 2017 survey found 62 percent of companies are using unsupported legacy systems (0patch, 2017). These operating systems still exist because older control system applications were built on older versions and either will not run or are unsupported on a newer version (BitSight Insights, 2017). Additionally, most control system vendors warn against upgrading operating systems or applying security patches until the vendor completes testing. This can delay patching for weeks while known threats are circulating in the wild (INCIBE, 2015). The OT networks and their dependent ICSs harbor many challenges to traditional IT network cybersecurity best practices and methodologies; new approaches are needed to address the challenges (BitSight Insights, 2017; DarkTrace Industrial, 2017). Software-defined networks. Software Defined Networking (SDN) is the culmination of multiple research efforts including the Active Networking research sponsored by the Defense Advanced Research Projects Agency (DARPA) in the 1990s and 2000s (Bakhshi, 2017). Other
9
projects included Separating the Control and Data Planes in network switches, Creation of the OpenFlow API for switch configuration, and Network Virtualization (Feamster, Rexford, & Zegura, 2014). Network administrators are gradually accepting SDN as a flexible method of managing networks. The flexible management capabilities of SDN allow quick and programmatic configuration of network switching assets in response to changes in loading, availability of nodes, or any other detectable conditions (Anwer, Motiwala, Tariq, & Feamster, 2010). Most existing switches and routers operate based on configuration scripts, which must be individually configured to instruct the hardware how to transport information across the network (Bobba et al., 2014). The centrally managed SDN data flow definitions abstract the detailed rule scripts by describing the high-level network functions without necessitating intimate knowledge of the arcane configuration scripting languages requisite by each device (Open Networking Foundation, 2017). Benefits of SDN. The essential insight leading to SDN is that the rules governing the flow of data through a network are easier to define, understand, and monitor if they are abstracted from individual switches and routers (Open Networking Foundation, 2015). These centrally managed data flow rules are easier to understand and manage for OT network users since they view the data paths from a high-level, network-wide perspective (Hadley, Nicol, & Smith, 2017). Administrators configure SDN enabled networks by describing logical data flow paths, which may include the connected ports, allowed traffic types, encryption requirements, and direction of data flow (Hill & Smith, 2017). The network administrator no longer has to login to multiple switches and change each configuration manually by typing commands in a command line interface (Bobba et al., 2014). This high-level data flow definition approach is self-documenting and promotes better situational awareness than traditional switch programming
10
methods (Raja, 2017). The SDN approach can have the effect that the whole switching infrastructure appears to the network administrators as one big switch with many widely distributed ports (Kang, Liu, Rexford, & Walker, 2013). Of course, not all aspects of SDN are considered net positives for the ICS environment; researchers have voiced concerns about the centralized configuration server, the communication between the application, control, and data planes, and the reliability of allowing programmatic changes to a live network (Bobba et al., 2014). In summary, the threats to OT networks are increasing as the pool of experienced cybersecurity personnel is shrinking. The differences between OT and IT priorities create risks that reduce the benefits of using IT cybersecurity personnel and standard IT cybersecurity tools in an OT environment. Therefore, new technologies or approaches are required to shore up the gaps in OT cybersecurity.
11
Statement of the Problem New management philosophies and modern technological capabilities are creating a dire situation for the critical infrastructure of modern societies around the world. While management is clamoring for more and fresher data to drive higher operating efficiencies, cybersecurity threats to OT networks are proliferating at a faster pace than most infrastructure companies can track, with little means to counter (Houlden, 2018). Malicious hackers from multiple countries with diverse motivations are increasingly targeting OT networks and specific control system equipment attached to those networks, which are instrumental to the reliable functioning of all sixteen Critical Infrastructure Sectors (CISs) identified by the US Department of Homeland Security (DHS) (Chalfant, 2018a, 2018b; Cisco Systems, Inc., 2018; US-CERT, 2018a; Zuckerman, 2017). Hackers affect the function of CIS processes either by attacking ICS components (PLCs, SCADAs, Human Machine Interfaces (HMIs), databases, or other Ethernet connected devices) that directly control a critical process or by attacking those same ICS components when found in support systems. Many of the same ICS components such as PLCs, HMIs, and databases that control critical processes are found in the heating and cooling systems, security systems, elevators, energy management, and other systems which directly support the critical infrastructure functions of the organization (Ademolake, 2017; Trend Micro, 2017). Expanding Need for Data from OT Networks Data are now recognized as critical to the efficient management of any corporation, and even more so for highly competitive infrastructure companies (Marr, 2016). Management requires more data, as accurately and quickly as possible, to realize their Real-Time Business (RTB) goals (Oxford Economics, 2011). Thirty percent of a group of 525 C-level executives surveyed worldwide in 2011, said their firms were already enjoying considerable benefit from
12
RTB techniques while another 45 percent of those surveyed planned to implement RTB concepts in the following five years (Oxford Economics, 2011). These requirements lead to connecting the business function IT networks to the production focused OT networks, which exposes the OT networks to more types of threats originating from more possible vectors (Higgins, 2018; Raja, 2017). The Cyber-Threats are Increasing Whether the hackers are sponsored by nation states, terrorist cells, or criminal cabals, they have many, sometimes competing, sometimes aligned, reasons to infiltrate, explore, map, and exploit these OT networks (US-CERT, 2018a; US-CERT, 2018b). Hacker motivations range from international and domestic politics, to economic competition, terrorism, insider grudges, espionage, criminal extortion, inquisitive research, hacktivism, misguided attempts at fun, and many possible combinations of these individual motivations (“7 types of hacker motivations,” 2011; Kovacs, 2015). While primarily concentrated on the power grid and defense industrial base, hackers are targeting all industries, whether for money, intellectual property, curiosity, or just pure mischief. Hackers behind ransomware are indiscriminate and cast wide nets looking for anyone that would pay quickly to regain control of their computers. The recent NotPetya ransomware attacks shutdown production at multiple Merck & Co. facilities and left the pharmaceutical maker struggling to restart manufacturing and fulfill orders for critical medicines a full month after the original attack (Erman & Finkle, 2017). Downtime for utilities and manufacturing plants is expensive and the ransom is likely much less than the cost of lost production. Many Russian and Chinese hacking teams are thought to be infiltrating critical infrastructure plants for three primary reasons: the stealing of intellectual property, mapping for future strategic use, and especially in the Russian case, funding current and future operations
13
(Sulmeyer, 2017). All this activity by state-sponsored and protected hackers leads to rapid advances in technology and sophistication of attack vectors, which exposes average manufacturing plants to highly sophisticated cyber-attacks with little or no protection. Current State of Cybersecurity for OT Networks The dismal state of cybersecurity for many of these networks makes it easier for the hackers and much harder for the defenders; this, of course, pre-supposes that there are actual resources assigned as OT network defenders in the plants or at the corporate level. In 2014, a Ponemon Institute survey found that 55 percent of critical infrastructure organizations had only one person dedicated to cybersecurity while 25 percent had no dedicated personnel. Original Equipment Manufacturers (OEMs) have upgraded and hardened individual Ethernet components such as PLCs, switches, routers; however, the OEMs have yet to develop a comprehensive, integrated approach to securing an OT network as a whole system especially when that system contains a heterogenous mix of equipment from different suppliers (Bisale & Kohl, 2015). Electric Power Grid is a Current Focus To date, most outright-malicious cyber-attacks on critical infrastructure have been observed in the power sector. Specifically, the Ukrainian power grid has been attacked and shut down twice, once in 2015 and again in 2016 (US-CERT, 2016; ICS-CERT, 2017). American electrical grids have been infiltrated and explored by several entities, including various hacker groups thought to be backed by Chinese, Russian, Iranian, North Korean, and Islamic State jihadi interests (Burke & Fahey, 2015; Crawford, 2014; Glenn, Sterbentz, & Wright, 2016; US-CERT, 2018). Cyber-war theorists point to the electrical power grid as the bedrock foundation of contemporary society (Schmidthaler & Reichl, 2016). If an enemy can shut down a target country’s power grid for substantial periods, they can significantly affect the general social
14
order, military defense, and economic viability of that country (Matthewman & Byrd, 2014; Swanson, 2016). These pressures can cause political turmoil in addition to the physical and emotional suffering (Schmidthaler & Reichl, 2016). Uniqueness of OT Networking Environment The cyber-defense of manufacturing systems is more complex and different from cybersecurity in standard IT circumstances. This is because cybersecurity tools were developed with the IT networking environment in mind. Thus, most cyber-defense tools are tailored toward IT networks and the IT cybersecurity version of the CIA triad that prioritizes confidentiality, not availability. The IT focused tools and best practices do not adequately comply with the unique requirements and restrictions of OT networks (CSSP, 2011). Many common IT cybersecurity tools can be counter-productive when used in an OT network environment (Coffey, Smith, Maglaras, & Janicke, 2018). The combination of older operating system versions, implementations of Ethernet interfaces running on field devices with minimal computational resources, and proprietary serial protocols adapted to run on top of Ethernet make for a challenging set of network applications. Coupling these challenges with a production environment and minimal or no ability to absorb downtime scanning demonstrate that the OT environment uniquely fragile. This fragility requires different tools or at least a deep understanding of OT requirements and a careful approach to using the existing tools in new ways (Wedgbury & Jones, 2015). Standards and Regulations Encouragingly, there are many standards and regulatory organizations proactively creating OT-specific, broad-based, cybersecurity standards and vociferously encouraging the implementation of those standards across all industry segments. Some of the standards
15
applicable to ICS cybersecurity include ISO/IEC 27001 & 27002, ISA/IEC 62443, National Institute of Standards and Technology (NIST) SP 800, NERC CIP, and IEEE 1686-2007. The plethora of standards, guidelines, and frameworks define many different cybersecurity concepts; however, the regulatory bodies for several CISs do not require or provide comprehensive methods of implementation. In 2018, many regulatory agencies are still only suggesting the use of cybersecurity capability maturity models like NIST’s Cybersecurity Framework or the National Cybersecurity and Communications Integration Center’s (NCCIC) ICS Cyber Security Evaluation Tool (CSET). However, a few regulatory agencies are moving from merely encouraging standards compliance to requiring compliance with specific standards in the future (NIST, 2018; Van Erp, 2018). The dearth of regulatory pressure leaves many plants with no detailed prescriptions for implementing the recommended concepts, no auditing of the installed network (with or without cybersecurity measures), and no local, qualified cybersecurity support resources with which to begin planning and implementation, or for continuing support (Van Erp, 2018). Shortage of Qualified Personnel A severe shortage of qualified and experienced cybersecurity engineers and technicians makes it hard to find and hold on to good cybersecurity talent. In early and mid-2017, various companies estimated a current shortfall of approximately 350,000 qualified cybersecurity professionals in the US and projected 1.8 to 3.5 million unfilled positions worldwide by 2021 or 2022 depending on the study (Frost & Sullivan, 2017; Morgan, 2017). Salaries for experienced cybersecurity personnel are climbing rapidly according to Robert Walters PLC (2018), a global staffing agency, which predicts a seven percent gain for cybersecurity salaries in 2018. Starting with a small pool of experienced talent and adding rapidly rising compensation makes it difficult
16
to hire and retain good talent while simultaneously meeting the cost reduction targets most manufacturing and industrial companies are setting. Lean and smart production concepts often equate to decreases in technical staff, not increases, yet the escalating number and types of threats would seem to require more resources to protect all the vulnerable networks (Bose & Sinha, 2012). Summary of the Problem This perfect storm of a new emphasis on real-time data, increasing cyber-threats, unique requirements, poor current cyber-defenses, and a lack of qualified personnel created the current crisis in OT cybersecurity around the world (Van Erp, 2018). Management’s demands for more detailed, accurate, and timely data is not going away, which requires increased external access to OT networks while the cybersecurity threats to those networks are rapidly increasing from multiple fronts (Pettey, 2016). Threats to OT networks are proliferating, including criminals; nation-state supported hacking teams, increasing numbers of hobbyist hackers, and even the unintended consequences of insiders (Kaspersky Lab ICS CERT, 2017). Another major issue is the current state of the regulatory environment. Additionally, many current IT cybersecurity tools and technologies are not compatible with the unique OT network requirements (Cardenas et al., 2009). Finally, there is a lack of experienced cybersecurity candidates, especially with an understanding of the unique challenges of an OT network and the increased salary/benefit demands from those who are available (Kaspersky Lab ICS CERT, 2017). All these issues combine to create an untenable situation that needs further study and new ideas. Purpose of the Study The purpose of this study is to identify the challenges that are unique to the application of cybersecurity principles in OT networks and then to assess the characteristics of SDN
17
technologies which may provide solutions to the unique cybersecurity challenges of OT networks. This research project will first review the available literature concerning IT / OT network convergence, which will set the stage with the challenges unique to implementing cybersecurity in OT networks. Subsequently, the literature will be examined to explore objectively the possible advantages and shortcomings of SDN technologies when deployed in an OT networking environment. Finally, this paper will investigate how SDN technologies may affect positively or negatively the personnel charged with developing and maintaining OT networks. This portion of the research will review the literature focused on how SDN technologies affect staffing workload and training requirements in standard IT environments and project those findings into the ICS environment of OT networks. Research Questions This research will investigate the following three research questions: Q1.
What are the unique limitations and special considerations regarding the implementation of cybersecurity in an OT network?
Q2.
How efficacious is SDN technology in mitigating cyber-threats specific to OT networks?
Q3.
What effects on personnel workload and training are expected with the implementation of SDN technology in an OT networking environment?
18
Literature Review The current competitive atmosphere of global business competition is driving the adoption of data-driven management. As management realizes the benefits of automated data collection and the near instantaneous availability of that data, they want more (Bayda, 2018; Marr, 2017). Industry must understand the unique challenges of establishing robust cybersecurity within an OT network environment and determine how to accomplish the hardening of the OT networks while still allowing the free flow of data in and out of the ICS (Gregory-Brown & Harp, 2016). Due to severe shortages in knowledgeable and experienced personnel, increased levels of cybersecurity must be managed with minimal or no increase in staffing (Schwab & Poujol, 2018). Recent software-defined networking technology deployments by several large telecommunications and IT services firms have caught the attention of ICS cybersecurity practitioners. Some cybersecurity researchers believe that SDN holds the promise of alleviating many OT specific cybersecurity issues (Bobba et al., 2014; Dong, Lin, Tan, Iyer, & Kalbarczyk, 2015; Fraile, Flores, Poler, & Saiz, 2018; Hadley et al., 2017; Hill & Smith, 2017). This paper will identify the unique challenges of implementing cybersecurity in an OT network while maintaining the desired data flows. Then the researcher will catalog the SDN characteristics that positively address those challenges and enumerate other aspects of SDN software that may negatively impact cybersecurity. Finally, the researcher will explore the effects of implementing SDN technologies on the personnel responsible for OT networks within the critical infrastructure industries.
19
Available Literature Researchers in industry and academe have been investigating the cybersecurity weaknesses of ICS’s and OT networks with great intensity for more than two decades; this research has created a large body of literature in peer reviewed journals, published conference papers, and the general ICS industry press. Meanwhile, SDN technologies are relatively new within the communication carrier and IT networking environments (Doyle, 2018). In early 2018, telecommunication service providers (e.g. AT&T, Verizon, CenturyLink, and others), and large cloud and web services providers (e.g. Microsoft, Amazon, Google, and Facebook) had only recently deployed large-scale SDN networks after several years of test deployments starting in 2013 (Doyle, 2018; DukeNet Communications, 2013). Few high quality, peer-reviewed primary sources exist, which have published research specifically on SDN technologies used in OT networks. The researcher utilized multiple search engines including Utica College ProQuest, Utica College E-Journal Portal, Networked Digital Library of Theses and Dissertations (NDLTD), and Google Scholar to find scholarly articles and alternatively relied on the standard Google search engine for more recent conference papers and industry journal articles. Determining the Unique Cybersecurity Characteristics of OT Networks This research project focuses on the cybersecurity aspects of OT networks and their inherent differences from IT networks. The researcher will first review the literature concerning the growing use of standard IT style networking equipment and architectures within the ICS or OT network environment. The history becomes important when explaining why OT networks use the same physical equipment with different priorities and goals. Convergence of OT and IT equipment. In their paper IT/OT Convergence - Bridging the Divide, Harp and Gregory-Brown (2015) discuss how OT networks and their dependent ICSs
20
are increasingly incorporating standard IT networking gear. The authors state that the convergence is being driven by lack of skilled labor, regulatory pressures, and basic business efficiency considerations. As commercial-off-the-shelf networking gear became capable of supporting the specialized needs of ICS’s, more and more control system engineers are adopting the standard commercial hardware and software to implement their OT networks (Biswas & Karunakaran, 2015). Today, some OT networks are still more proprietary and restrictive, while others appear almost identical to a standard IT network. Gregory-Brown and Harp (2016) detail that while IT managers typically reserve five to ten percent of their total budgets for cybersecurity costs; the OT budget has traditionally allocated nothing to cybersecurity due to the lack of a perceived threat. Unfortunately, this is still often the case even though today the threat is much more palpable, while many more OT networks utilize the same switch and router hardware, and are architected in the same manner as a standard IT network. Leading companies from the mainstream commercial IT equipment market are cooperating, merging, or competing with industrial OEMs to capture more of the OT market share. Cisco is the preeminent supplier of IT networking hardware in the world (Gold, 2017) while Siemens, Belden, and Moxa lead Cisco in OT sales. In 2007 Cisco formed a strategic alliance with Rockwell Automation, which allows Rockwell to sell Rockwell-branded switches, routers, and security appliances containing Cisco designed hardware and software (Rockwell Automation, 2010). Rockwell Automation was the fifth largest ICS equipment manufacturer in the world and the second largest in the US in 2015 (O’Brien & Avery, 2016). Siemens, the worldwide leader in automation equipment, sells Siemens-designed switches, routers, wireless equipment, security appliances, and network management software to both the industrial OT and the commercial IT markets. In 2012, Siemens accelerated the convergence by buying a well-
21
respected network equipment provider specializing in the critical infrastructure space, RuggedCom (Mintchell, 2012). Researchers, engineers, and OEMs from OT focused firms recognized this infusion of commercial IT equipment and general-purpose Ethernet networking gear into OT networks. Manufacturing based OT networking has a long history of being designed and installed by the same engineering focused firms and personnel that had been providing the complex ICSs. This engineering focus generated calls to the traditional ICS focused standards organizations for a common taxonomy and understanding of the typical components of an OT network. Industry Standards and Government Regulators Industry standards setting bodies and government regulatory agencies coordinate among themselves to ensure the most critical sectors are strengthening their cybersecurity postures. The sectors currently in focus are the power generation, transmission, and distribution sector and the financial services sector. With regard to the implementation of cybersecurity for an ICS, the primary standards setting bodies are (ICS-CERT, n.d.; Stouffer et al., 2011): 1. The International Society of Automation (ISA) 2. The National Institute of Standards and Technology (NIST) 3. The North American Electric Reliability Corporation (NERC) 4. The International Standards Organization (ISO) 5. The International Electrotechnical Commission (IEC) These organizations have all existed for many decades. Their ICS safety and quality related standards setting activities significantly pre-date the need for OT network cybersecurity standards. However, due to the innate link from an ICS’s control of the process to the safety of workers and quality of the products, these organizations quickly realized the need for
22
standardization concerning the flow of data in and out of an ICS via the OT network (Assante & Conway, 2014). The Purdue Model and ISA-95. In the 1980s and 1990s, researchers at Purdue University created an ICS network reference architecture model titled, A Reference Model for Computer Integrated Manufacturing (CIM), hereafter referred to as the Purdue Model (CIM Reference Model Committee, 1989). Then, in the year 2000, the International Society of Automation (ISA) introduced the ISA-95: Enterprise-Control System Integration (2010) family
Figure 1: The Purdue Reference Model as adapted to ISA-95 (ISA, 2010).
of standards. In 2003, the ISA-95 standard was adopted by the American National Standards Institute and renamed as ANSI/ISA-95 and their international equivalents, the ISO and the IEC, as ISO/IEC-62264. For brevity, this paper will refer to the standard as ISA-95. The ISA-95 23
standard co-opted the Purdue Model as the basis for developing standard interfaces between business functions at the enterprise level and manufacturing functions at the control system level (ISA, 2010). The ISA-95 standard defines seven levels, levels labeled “Level 0” through “Level 5” and the De-Militarized Zone (DMZ) as depicted by the yellow box between Level 4 and Level 3 in Figure 2 below (ISA, 2010). Each level refers to a specific class of device and a set of functions. Level 0 devices are the sensors and actuators that bring real-world measurements into the ICS including temperatures, pressures, valve states, and motor speeds while enabling the system to affect the real-world by controlling pumps, valves, and motors (ISA, 2010). The DMZ is a segment of the network reserved to separate the parts of the ICS that are needed to control the process from those parts that manage and report about the process (ISA, 2010). The DMZ is primarily a cybersecurity concept that makes it more difficult for any malware or hackers to access the critical production-oriented parts of the network even if they have established persistent access to the business network (ISA, 2010). The ISA-95 model was instrumental in standardizing data transfer into and out of ICS’s via OT networks. By providing a standard model and propagating the idea that accessing the production data was affordable and manageable, the ISA-95 standard accelerated the adoption of data-driven management and realtime business management methodologies (ISA, 2010). ISA-99 adds security. Unfortunately, when ISA-95 was initially published in the year 2000, cybersecurity was not at the forefront of manufacturing industry concerns (ISA, 2007). It was not until 2010 when the Stuxnet worm was discovered, that cybersecurity began rapidly rising in the consciousness of industry in general, and control system designers and builders specifically. Fortunately, by that time, ISA and others had become concerned with the general
24
use of Ethernet-based networks and standard commercial grade computers within OT networks. ISA-99: Security for Industrial Automation and Control Systems, published in late 2007, addressed the security implications of ICS’s relying on networks now vulnerable to viruses, worms, DDoS attacks, and all manner of IT network scourges (ISA, 2007). The national and international standards bodies have also adopted ISA-99 and renamed it to ANSI/ISA-99 and ISO/IEC-62443, again this researcher will continue to use ISA-99 for brevity and clarity (ISA, 2007). The ISA-99 committee took up where the ISA-95 committee stopped, with a renewed look at the Purdue Model focused on the cybersecurity vulnerabilities (ISA, 2007). While ISA95 had concentrated on making it easier to get data in and out of control systems, ISA-99 concentrated on making ICS’s more secure from cyber-threats while maintaining the free exchange of data (ISA, 2007). Cybersecurity Vulnerabilities of OT Networks The ISA-99 committee absorbed many ideas from their IT counterparts who, by the late 2000s, had been fighting on the frontlines of cybersecurity for the better part of three decades. Due to their widespread adoption of standard, commercial IT networking equipment, OT networks inherited the same threats and vulnerabilities experienced by IT networks. NIST’s Smart Grid Cybersecurity Committee (2014) and DHS’s Comprehensive National Cybersecurity Initiative (CNCI, 2009) discuss some of the multitude of threats to ICSs, some of these common threats include: •
Standard computer viruses and worms
•
Interconnected business networks
•
Denial of Service (DoS) and Distributed Denial of Service (DDoS) attacks
•
Trojan malware delivered by phishing messages
25
•
Botnets
•
Ransomware
•
Advanced Persistent Threats (APTs)
•
Man-in-the-Middle (MITM) attacks
•
Spyware and key loggers
All these threats, which are common to IT networks, pose as serious if not disastrous problems for ICSs (Ericsson, 2010; Glenn et al., 2016). A compromised ICS may begin acting unpredictably and become slow or unresponsive which are conditions counter to their purpose of steady and deterministic control. Unique aspects of OT networks. In addition to threats common to IT networks, the OT networks are subject to a myriad of cyber-threats due to their unique operating conditions, past isolation from other networks, and stringent availability requirements. In their book Hacking Exposed Industrial Control Systems: ICS and SCADA Security Secrets & Solutions the authors list safety requirements, the impact of cybersecurity controls on process performance and availability, and the numerous operational differences between OT and IT as categories of unique characteristics of ICS equipment (Bodungen et al., 2017). A primary difference in OT and IT operating procedures mentioned by multiple authors concerns cyber-hygiene, which entails the day-to-day effort to maintain cybersecurity health, including patching, making backups, penetration testing, performing audits, and maintaining access controls (physical, network connections, and user credentials) (Aydell, 2015; Bodungen et al., 2017). These differences can be demonstrated in the opposite prioritization of cybersecurity goals of Confidentiality, Integrity, and Availability to be more fully explored later in this paper. Other researchers mention the infusion of IT equipment and highly connected architectures with little
26
understanding of the concomitant risks (Luiijf & Paske, 2015). Further research is explored below which reveals many other differences between IT and OT networks that inform the different approaches and priorities. Safety first. The highest priority of any ICS is safety, much like a doctor’s Hippocratic Oath to: first do no harm. Human safety is one of the primary reasons for installing an automated control system and always a critical design requirement (Bing, 2018; Chalfant, 2018b). The computer controllers in an ICS can monitor thousands of process values and equipment statuses nearly simultaneously and then make decisions in milliseconds to initiate safe shutdowns before conditions escalate out of control (Bing, 2018). Any interruption of safety functions is a serious failure. This extremely high emphasis on availability required for safety functions simultaneously makes it easier for an attacker to cause downtime and increases the difficulty in maintaining good cyber-hygiene (Aydell, 2015). If an attacker can affect the OT network in a way that triggers a safety shutdown, it may be much easier than reverse engineering a PLC program and reprogramming the PLC. Inversion of the CIA triad. A great majority of sources on cybersecurity in OT environments mention the difference in priorities between IT and OT administrators (Aydell, 2015; Bodungen et al., 2017; Pauna, Moulinos, Lakka, May, & Tryfonas, 2013). These differences are said to culminate with innate differences in how the two groups prioritize the basic categories of cybersecurity; IT practitioners hold that Confidentiality is most important, followed by Integrity, then Availability. This hierarchy is represented by the CIA triad and is a central tenet of IT cybersecurity (Bodungen et al., 2017). However, OT practitioners flip the priority triad completely upside down and focus on Availability first and Confidentiality last. They often even add Safety to the top of the list to emphasize its importance above all else
27
(Brocklehurst, 2017). In his System Administration, Networking, and Security (SANS) Institute supported survey Securing Industrial Control Systems - 2017 Gregory-Brown (2017) found that ensuring reliability, availability, and human safety combined for the top concern for 42 percent of respondents with the next closest category at eight percent. Glenn Aydell (2015) in his paper The Perfect ICS Storm discusses both the inverted CIA triad and the vastly different physical consequences of an ICS cybersecurity failure. Aydell (2015) remarks that often patches, backups, and other maintenance activities cannot be performed while the ICS controlled process is running. This requires special scheduling and coordination or more expensive system design to include redundancy of key components allowing one machine to be updated while the other stays online controlling the process (Aydell, 2015). Aydell (2015) gives two examples of the difference in how downtime affects IT systems versus OT systems. In the case of IT downtime, loss of access to an e-mail server for two hours every month or a laptop taking an extra 15 minutes to shut down after a software update is annoying, but not costly or dangerous in the IT world. However, two hours of downtime for a control system server could translate into tens or hundreds of thousands of dollars of lost production, and if an operator’s workstation rebooted during a critical phase of a process it would leave the operator with no visibility and no way to control the process (Aydell, 2015). These situations highlight the different consequences of downtime between IT and OT environments and illuminate the reason for the OT practitioner’s view that the CIA triad is upside-down and missing an emphasis on safety. Organizational challenges. The opposite views of the CIA triad are one symptom of a bigger divide, which is the almost universal separation of management structures between OT and IT organizations (Bindseil, 2003). Often the first place in the corporate organizational chart
28
where the two structures meet is just below the Chief Executive Officer (CEO) where the Chief Information Officer (CIO) or Chief Information Security Officer (CISO) sits at the same conference table as the Chief Operations Officer (COO). A 2017 Automation World survey revealed that less than eight percent of industrial companies have combined OT and IT operations in the same organizational structure (Thilmany, 2017). The COO and CIO/CISO organizations are often structured completely differently with the COO being primarily concerned with the efficient operation of the business, part of which in a manufacturing firm includes the timely production of high-quality products in a safe environment that maximizes revenue for the company (Ernst & Young Global Limited, 2014b). While conversely, the CIO/CISO organizations are primarily service and support organizations structured as cost centers, which add to overhead costs (Ernst & Young Global Limited, 2014a). In a manufacturing company, the COO organization is often full of career engineers with little or no training in IT, only picking up what IT networking skills they needed to make the plant run. With little exposure to IT priorities and a history of running production processes in a heretoforeisolated plant, incompatible with Ethernet and the Internet, the operations staff has had no need to understand and implement good cybersecurity practices (Glenn et al., 2016). The CIO/CISO organization is staffed with computer science graduates that may have never worked with a PLC or safety system; either of which can injure or kill someone with only an innocent disruption of the carefully engineered environment. The same Automation World survey indicated that in 73 percent of the respondents plants no cross training between IT and OT departments (Thilmany, 2017). It is no wonder that these two cultures misunderstand each other and fail to value what each brings to the company (Luiijf & Paske, 2015).
29
Corporate pressures for advanced manufacturing technologies. The broad term Advanced Manufacturing Technologies (AMTs) can imply changes to a wide swath of OT including software, hardware, design principles, manufacturing processes, data analytics, business systems integration, and more. All AMTs are generally aimed at reducing costs while increasing safety, quality, or production rates (Sirkin, Zinser, & Rose, 2015). Some examples of AMTs are autonomous robots, integrated computational materials engineering, industrial internet, flexible manufacturing, advanced data analytics, predictive maintenance, and predictive quality (Sirkin et al., 2015). The introduction of any AMT typically includes the adoption of more IT style equipment networking and computing equipment to allow more data transfers and more processing of the data. These constant pressures to implement change in the OT infrastructures rapidly leave the door open for mistakes and oversights in design and implementation. This rapid change could be an acceptable risk if a formal cyber-risk assessment and management program were in place, but the more likely situation in OT environments is that risk management still applies only to the production process machinery and not the automation equipment controlling it (Kure, Islam, & Razzaque, 2018). Lack of integrated OT cybersecurity program. A general lack of extant or wellintegrated and maintained cybersecurity defenses can often be traced to the lack of a C-level sponsor for a comprehensive corporate cybersecurity program (Luiijf & Paske, 2015). A comprehensive cybersecurity program would require participation and constant attention from all management levels including both IT and OT organizations. A management principle often used in lean manufacturing or quality management classes is You can’t manage something if you don’t measure it. While perhaps too absolute to be a golden rule of manufacturing, the use of the correct metrics and KPIs has proven central to current management systems (Mauboussin, 2012).
30
Collier et al. argue that the lack of clearly defined metrics is one of the greatest barriers to success in implementing ICS security (Collier, Panwar, Ganin, Kott, & Linkov, 2016). Luiijf and Paske (2015) urge that an ICS cybersecurity program should cut a broad path across several areas of executive oversight including human resources for hiring and training, supply chain management, and risk management in addition to the expected OT and IT hierarchies. Luiijf and Paske (2015) further recommend that the cybersecurity management be fully integrated into the corporate-level risk management framework along with active consultation of other companies in their industry and with government agencies. Insecure by design. Dale Peterson (2011) popularized the phrase insecure by design in his August 2, 2011, Digital Bond blog entry. The point of the article was that securing the workstations and servers on an OT network had reached a point of diminishing returns since the other devices on the same network, such as PLCs or other Ethernet devices, were wholly unprotected, and presented much easier targets (Peterson, 2011). Additionally, a properly functioning PLC or machine will take a properly formatted, clear text, command string, and execute it, no questions; there is no built-in authentication or encryption in most legacy devices and very few modern devices have options for authentication or encryption (Peterson, 2011). The inherently insecure equipment presents two problems. First, is the wide-open nature of many older ICS components that require significant additional cybersecurity design considerations and most likely will require additional cybersecurity hardware or software to secure. Second, is that if it is deemed critical to the process and too expensive to upgrade the insecurity could be there for another twenty, thirty or even fifty years. Kobara (2016) highlights the difficult realities of extended lifetimes stretching 15-25 years or more with no budget to replace any earlier. These lifetimes are extended due to high initial costs, asynchronous
31
lifecycles, proprietary interfaces, and lack of current operating system support (Luiijf & Paske, 2015; Pauna et al., 2013). Inherent physical differences. Differences between IT and OT networks can sometimes be seen and felt as they are physical in nature and often due to the differing design requirements based on environmental conditions and usage patterns (Harcharan, Houmb, & Engum, 2018). The IT network is typically found in clean, air-conditioned, office environments while OT networks are often in dirty, hot, humid environments whether on the plant-floor or outdoors (Maddison, 2018). Somewhat counterintuitively, the IT network equipment is often expected to last a mere five to seven years before needing a significant upgrade to keep up with the rapidly changing technologies, while conversely, the OT network hardware is expected to be installed and last until the process control equipment is refreshed perhaps fifteen or twenty years in the future (Maddison, 2018). This indefinite deployment period also introduces numerous support and security update issues (0patch, 2017; Glenn et al., 2016). The typical IT network is designed to safeguard access to data from outsiders and unauthorized insiders at all costs and then to ensure the data is accurate, safe from changes either accidental or intentional (Fulton, 2018). This drives high levels of access control, chokepoints to ensure data is inspected and only going where it should be going, and redundancy at the storage and computing layers (Fulton, 2018). The OT network is built for reliability and deterministic speed with few controls on local area peer-to-peer access because that can slow data transmission and cause issues when new systems need to be integrated with existing systems (Gregory-Brown & Harp, 2016). These environmental and robustness requirements drive purchasing decisions to different sets of equipment suppliers.
32
The differences between IT and OT priorities and implementation requirements are also reflected and reinforced by the partitioning of network equipment suppliers, with IT and OT being supported by different sets of OEMs. Cisco, Dell, Hewlett-Packard, 3Com, and Juniper are companies that mainly target the commercial IT world. While, typical industrial focused OT network providers are Siemens, Belden (with acquired brands GarrettCom and Hirshman), Moxa, Rockwell Automation, RuggedCom (now owned by Siemens), N-Tron (now owned by Red Lion), Schweitzer Engineering Laboratories, and PhoenixContact. Each of these suppliers have their own switch operating systems with its own command language, each hold their own user group meetings, and each provides training or support within their own partner organizations. This separation on the networking equipment and software level surreptitiously reinforces separation between IT and OT organizations. An IT network engineer heavily experienced in Juniper switches and routers is not going to be immediately effective implementing or troubleshooting Siemens switches nor is that engineer likely to get funding to go to an OT focused conference where Moxa and N-Tron may be presenting new products or offering training. Therefore, the IT / OT separation is again reinforced on multiple levels. Large and complex attack surface. Manufacturing plant and chemical processing sites typically encompass large, spread out, physical facilities with many points of access to the ICS and its OT networks. The plant environment may seem very secure due to the physical protection of a guardhouse, fence, and gate; and, indeed Carney (2011) asserts a cybersecurity program must rely on a solid physical-security program. However, the electronic nature of the networks and unseen connections to the outside world present a large, varied cyber-attack surface. That attack surface can include internet connections, wireless networks, cell phone modems, dial-up modems, USB ports, CD/DVD drives, unmonitored Ethernet network ports,
33
and Virtual Private Network (VPN) connections (Tripwire, 2014). While many OT network administrators feel comfortable with a firewall segmenting the ICS network from the business IT network, John Pirc (2009) attempts to disabuse OT practitioners of this dangerous misconception. An often-overlooked attack surface is the instrumentation that is part of the ICS itself. Joe Weiss (2018) warns about concentrating too much on the controller to HMI and HMI to other system portions of the OT networks while ignoring the Level 0 instrumentation and actuators that are also communicating over serial and Ethernet. Several old serial protocols have been adapted to communicate over the new Ethernet based networks (Red Tiger Security, 2011). The problem is that the protocols were originally written with no thought given to security, and when the protocols were upgraded. Most were adapted merely to run the same unsecure protocol on top of the new faster Ethernet infrastructure (Weiss, 2018). No security upgrades were included and thus they are still insecure protocols not well understood by IT cybersecurity experts and often incompatible with IT cybersecurity software (Red Tiger Security, 2011). While IT networks are also vast and complex the Requirements for remote access. The outsourcing phenomenon has infiltrated most manufacturing and process plants. Various equipment vendors offer to lease machinery or maintain it without having dedicated on-site personnel. The vendors can provide these services by sending technicians on-site for regular inspections or more cost effectively by monitoring the assets remotely and only sending technicians when required. This requires a remote connection to their equipment, which is not a difficult technical issue (Wallace & Carter, 2010). However, now that the process owners want more data about any and everything on their site that contributes to making a product, the equipment is being connected to the plant OT networks for data collection and rudimentary supervisory control (ISA, 2010). Often these connections are
34
made hastily with poor cybersecurity design including leaving the vendor modems, which bypass firewalls. If the modems are removed, then the vendor still needs a way to monitor the equipment and now firewall rules are often disabled or bypassed to get the remote access working again (Kolaks, 2003). Another form of outsourcing is attributable to lean engineering staffs at manufacturing companies and a general shortage of ICS engineers and programmers (Futrell, 2016). The three trends of low graduation rates from engineering schools, high numbers of retirements corresponding with the baby-boomer retirement wave, and the resurgence of manufacturing in the US leave companies with few staffing options (Bersin, 2018; Wright, 2014). Now in the interest of remote technical support, remote access is being granted to the crown jewels, the engineering stations, which allow configuration and programming of the entire ICS (Kolaks, 2003). There are methods to fortify and protect outside access to the OT networks, however, they are complex and expensive to install and maintain and sometimes not well understood by the OT staff who are primarily tasked with their installation and maintenance (Davidson & Wright, 2008). How Does SDN Technology Address Cybersecurity in an OT Network? There is a need for an OT cybersecurity solution that addresses as many unique vulnerabilities of OT networks as possible while minimizing any negative consequences. The available literature is lacking in specific research on the effects of SDN technologies on the cybersecurity of industrial or OT networks with the notable exceptions of papers published in connection with three DOE sponsored projects: Project Watchdog, SDN Project, and Chess Master, which are all reviewed and discussed herein. However, additional research is leveraged by reviewing the research available on the cybersecurity vulnerabilities of OT networks and
35
ICSs. That general research is then compared to the research available on the cybersecurity implications of SDN technologies, which allows the cybersecurity benefits and detriments of SDN technology to be applied to the cybersecurity challenges of OT networks. This section of the literature review will concentrate on the innate cybersecurity related characteristics of SDN technology and how those characteristics affect general, non-OT specific cybersecurity issues. The following section of the literature review will focus of research involving the deployment of SDN in ICS environments. Review of SDN technology. Software Defined Networking technology is a new way of thinking about networks and the methods of configuring the switches, routers, and firewalls which do the actual work of moving data. Network administrators use SDN software to configure networks using the simple, abstract concept of a data flow. A data flow definition can completely characterize a connection using only the two endpoints or ports (Fulton, 2018). This type of abstract definition, with logic determining the actual configurations to be downloaded to the data moving devices, generally referred to as packet brokers or packet forwarding devices, allows for additional automation within the system (Banks, 2014) In a standard IT style network all the switches, routers, and firewalls are individually configured. Each device only knows about itself and transfers data from one port to one or more other ports based on its own configuration. The device configuration is normally created by a network administrator who types text-based commands in a Command Line Interface (CLI) much like the disk operating system (DOS) on old personal computers (Lawson, 2013). Consequently, each device may have a separate set of CLI-style commands and use different syntaxes that the administrator must learn and understand (Goransson, Black, & Culver, 2017). The configurations can be exported and stored elsewhere for backups in case of device failure.
36
In a network that spans multiple large buildings, there can be hundreds to thousands of devices each with its own configuration file to be managed (Goransson et al., 2017). Separation of logical functions into planes. Wang (2016) describes the network functions of an SDN network as being separated into three logical layers or planes. The top layer is the application plane, where the data flow definitions are created and stored (Open Networking Foundation, 2015). The next layer is the control plane, which processes the data flow definitions and creates low-level switch configuration command sets that are downloaded to the third layer or data plane, which is the physical switch hardware and sometimes called the infrastructure plane (Open Networking Foundation, 2015). This separation allows simpler, edge-to-edge, high-level network configuration from a central controller while relieving the forwarding plane devices of complex packet analysis and peer-to-peer coordination (Sharma, 2015). Concentrating the configuration and programmatic control in the central controller allows the physical network devices in the data plane to be simplified which reduces costs and increases performance (Hadley et al., 2017). Communication between planes via APIs. The applications on the application plane communicate with the middle-layer control plane via the northbound application-programming interface (API) (Ahmad, Namal, Ylianttila, & Gurtov, 2015). The northbound API allows the applications to monitor the condition of the network and then request changes. There are several competing northbound APIs with no clear leader. The control plane communicates to the individual devices in the data or infrastructure plane via the southbound API (Ahmad et al., 2015). The southbound API allows a controller to download new configurations and monitor performance and diagnostic metrics at each device. While there are also several competing southbound APIs, the most prevalent is the OpenFlow API (Bakhshi, 2017). Figure 2 depicts the
37
three layers of the SDN model network with the northbound and southbound APIs (Ahmad et al., 2015).
Figure 2: SDN separates the logical definition of data flow paths from the switching and routing hardware (Ahmad et al., 2015).
Potential uses for applications. The applications on the application plane work with the control plane through the northbound API to enable autonomous programmatic responses to events or conditions on the network (Jammal, Singh, Shami, Asal, & Li, 2014). These programmatic responses could include: 1. Determining if unexpected traffic should be forwarded or dropped. 2. Implementing a backup data flow path if the primary path fails. 3. Managing load sharing when a flow path becomes saturated. 4. Provisioning new virtual resources on demand
38
All manual configuration is accomplished through an application and then passed to the control plane where it can be validated against policies and design rules before being compiled and downloaded to the data plane devices (Goransson et al., 2017). SDN allows high-level design and visibility. The SDN technology advances network architecture by allowing high-level definition of data flows across the network while negating the need for a highly skilled network engineer to configure individual pieces of network gear (Hadley et al., 2017). Because the applications and controllers are purely implemented in software there is tremendous flexibility in how an SDN network is deployed (Cranford, 2017). The application plane applications can be hosted on the same server as the control plane software in a small network or they can be spread across multiple servers to balance load and reduce latency in a larger implementation (Dixit, Hao, Mukherjee, Lakshman, & Kompella, 2013; Taylor, 2017). The servers can be standalone physical servers situated locally or they could be virtual machines running in a datacenter or on a cloud infrastructure (Patil, Gokhale, & Hakiri, 2015). The centralized, software-based management of SDN creates a homogenous network design environment, which also allows extensive simulation and scenario testing, complete network configuration backups, automated configuration validation, and faster disaster recovery (Dolezilek, 2018). One of the primary benefits of SDN is that the network administrator can design the network and configure it from a network-wide perspective (Du & Herlich, 2016). Instead of thinking about the configuration for each device and manually entering commands on a CLI interface, the SDN software implements data flow definitions. These data flow definitions are high-level abstracted configuration definitions, which define an entire data path from one starting port to another ending port (Bobba et al., 2014). The system then determines the best route
39
between ports without requiring manual selection and detailed configuration of all the individual ports on the intervening switches, routers, firewalls. If the network designer wants to specify specific parts of the route or a backup route, they can do so, but not required. When the network is fully designed, the software compiles the individual configuration rule sets, checks them for completeness for all the affected devices, and when authorized, downloads the new rules. The compiled forwarding rules are transferred to the individual forwarding plane switches and routers via one an SDN switch configuration protocol, one of the most well-known is OpenFlow from the Open Networking Foundation (Duffy, 2011; Mitchell, 2015). Automation through programmability. Another benefit of SDN is the built-in programmability of the system; because the data flow definitions are abstracted, it makes writing programs for automated responses to network conditions or unexpected data easier (Fraile et al., 2018). This is a major benefit to the data centers and commercial wide-area networks for which SDN was initial developed. Programmability allows the automated deployment of new virtualized servers and the networking infrastructure to support them in minutes instead of the hours or days it requires with manual configuration (Fulton, 2018). It also allows SDN systems to programmatically evaluate unexpected data and either automatically determine where it should be sent or flag it for human classification (Banks, 2014). An OT specific use case for programmability involves temporal filtering where specific flow rules are only enabled when an authorized user requests a connection to specific resource; this could be used to drastically reduce the attack surface by controlling access to PLCs from engineering stations for program changes or further securing VPN connections for remote troubleshooting (Tsuchiya, Fraile, Koshijima, Ortiz, & Poler, 2018). The centralized management and programmatic functions of an SDN system reside in a server commonly referred to as the flow controller or Orchestration
40
Engine when specifically referring to the programmatic capabilities (Vilalta, Mayoral, Casellas, Martínez, & Muñoz, n.d.). While vital to commercial environments, SDN in an OT environment would likely not use the programmable features to avoid the possibility of creating an unstable network or would use them sparingly to directly reduce the attack surface as in the previously discussed temporal filtering use case (Hill & Smith, 2017; Tsuchiya et al., 2018). A typical OT network is static, and any structural changes are usually planned, verified, and performed during downtime associated with physical changes to the process and/or the control system (Bobba et al., 2014). Cybersecurity implications of SDN technologies. Cybersecurity is directly tied to the flow of information in a network. The SDN technologies allow fine-grained control and monitoring of the network with a deny-by-default or zero-trust philosophy. This project will next explore the literature discussing the generally beneficial aspects of SDN. These potential benefits come with some drawbacks, and those will be explored in a separate sub-section following the benefits. The majority of the literature discussed in this section will be general and not directly related to OT networks or ICSs. However, most of the benefits and detriments are similarly applicable in IT and OT networks alike. Where appropriate, differences in application between IT and OT networks will be noted. Potential SDN benefits to cybersecurity. The advantages of SDN are many, and most of them have application in enhancing the cybersecurity of the network. Accurate documentation of the network, including architectural layout and connections, and individual switch configuration has always been tough to maintain; now it is built-in (Hill & Smith, 2017). The deny-all-by-default default condition of SDN is exactly what the cybersecurity practitioners preach (Hadley et al., 2017). Centralized management of the entire network is an additional
41
positive for cybersecurity as it allows a full view of the network and the ability to weigh alternatives (Du & Herlich, 2016). Accurate network diagrams. One advantage of SDN for cybersecurity is the fact that the SDN configuration completely defines the network. No data flows through the network without a specific data flow definition from source to destination (Hill & Smith, 2017). This allows multiple additional positive benefits. The architecture and specific equipment relationships and types of hardware must be completely defined in the SDN controller (Dolezilek, 2018). The SDN controller cannot create forwarding plane device configurations without knowing the network’s architectural connections and specific device types, which forces a current and accurate network architecture to be generated prior to provisioning of any data flow paths (Gregory-Brown, 2017). Current and fully accurate network diagrams have been the bane of network administrators since the first two multiport switches were connected (Lloyd, 2016). Good cybersecurity depends on accurate information especially regarding the architecture and configuration of the network (Gregory-Brown, 2017; National Cybersecurity and Communications Integration Center, 2016; Wedgbury & Jones, 2015). Hill and Smith’s (2017) work through Veracity and SEL on the Chess Master project have advanced the ability for an SDN system to learn the current composition and layout of an attached network via link layer discovery protocol (LLDP) frames during a learning phase. Whitelisting the network. The detailed and specific data flow definitions of SDN can be thought of as whitelisting the network (Hadley et al., 2017). Data are only allowed to flow through the network when specific data flow definitions exist and have been compiled into individual forwarding plane equipment configurations (Dolezilek, 2018). If data are received on a port with no data flow definition or data are received which do not match the defined data flow
42
type and destination, the data are not forwarded (Sharma, 2015). Any unexpected data packets may be forwarded to an SDN controller for inspection, and then a determination can be made programmatically of where to send the data or if it should be ignored (Hill & Smith, 2017). This deny-all-unless-specifically-allowed behavior is much more secure than current network equipment default behavior and parallels the least functionality best practices espoused by cybersecurity experts and codified in various standards including NIST 800-53 CM-7, NIST 800-171 Section 3.13.6, and ISA-99 (ISA, 2007; NIST, 2015; Ross, Viscuso, Guissanie, Dempsey, & Riddle, 2018). Unprecedented visibility leads to realistic modeling. The detailed and clear visibility SDN provides into networks is another strength that allows detailed simulation of multiple network states and failure modes (Myer, 2018). With a complete and accurate model of the network architecture and all the switching device configuration rule sets available in a central repository, it now becomes straightforward to leverage that wealth of information for highfidelity simulations (Jin & Ning, 2014). While it was technically possible to simulate legacy networks, it was always difficult to determine if the model was completely accurate; even if it was accurate at the start of a project there was a high probability something would change before the project was finished. The SDN framework requires accurate architectures and can immediately validate the accuracy of a network model against the actual running network (Dolezilek, 2018). Now network administrators can be confident that the simulations they run are accurate and then can use those results to build detailed and validated disaster recovery plans. From a cybersecurity viewpoint, this allows time for better cybersecurity validation prior to implementation of new network architectures or minor expansions to existing networks.
43
Enhanced network performance. Hadley et al. (2017) document seven separate network performance metrics they tested including network healing, network hop latency, packet delivery, network size and scaling, priority services, baselining, and change control. For six out of seven categories SDN was either favored or the clear performance leader; network hop latency was the single category where standard networking was favored (Hadley et al., 2017). Network healing time was the category where SDN had the biggest advantage; SDN was two orders of magnitude faster at 100 microseconds versus 10 milliseconds for a standard Rapid Spanning Tree Algorithm (RSTA) algorithm-based switch network (Hadley et al., 2017). Reduced costs over time. While redesigning a large network with updated technology will not save money in the short term. Some cost savings from an SDN managed network will eventually include saved management time, saved hardware replacement costs, saved time-todeployment for network service consumers, and reduced cost for increased cybersecurity (Cisco Systems, Inc., 2017). Of these costs, the savings derived from using a commodity or whitebox hardware instead of highly specialized name-brand switches and routers should be significant; (Patrizio, 2018). Recently, the rumor that Amazon might start selling some of their internally developed SDN networking switching and routing devices caused the stock prices of Cisco, Juniper, Arista Networks, F5 Networks, and Broadcom to plunge by as much as 5 percent (Owens, 2018). Some media outlets estimated the generic Amazon equipment could undercut Cisco switch prices by 70 to 80 percent (Patrizio, 2018). The multiple beneficial cybersecurity principles inherent in SDNs including accurate documentation, increased situational awareness, zero-trust packet forwarding, and full-time network monitoring improve the basic cyberresilience of an SDN network. These aspects lead to spending less on installing and managing additional security solutions; for example, the intrinsic data flow matching at each forwarding
44
plane device act like deep-packet inspection firewalls providing fine-grained monitoring and control (Tsuchiya et al., 2018). These characteristics reduce the total cost of ownership of an SDN across the broad areas of hardware, software, and network management labor. Centralized network management. The centralization of network management allows the accurate mapping, verification, modeling, and simulation capabilities discussed above (Bobba et al., 2014). Additionally, central management is a force multiplier that allows fewer personnel to manage more network resources (Hardy, 2017). Provisioning new network resources now is a matter of getting the hardware installed and then downloading a simulation-tested configuration that has been vetted by the cybersecurity team. In many ways, it is one of the best features of SDN because it enables so many of the other beneficial features (Hardy, 2017). One SDN orchestration software provider estimates reductions of 50 to 80 percent in time to deploy, 60 to 85 percent in deployed hardware costs, and 85 to 94 percent in post deployment management effort (Fruehe, 2016). The central management sever also facilitates the hugely increased situational awareness by concentrating the network monitoring into a single location that also stores the full architecture and software configuration for the network (Hadley et al., 2017). However, centralized management can be a two-edged sword, presenting several potential areas of concern that must be considered and designed around which will be discussed below. Potential SDN detriments to cybersecurity. The preceding literature has centered on SDN’s beneficial aspects regarding cybersecurity. Multiple researchers, as detailed in the next section, still have opposing views as to whether SDN is a net positive or negative cyber-risk. Now the discussion will switch to the research covering potentially negative aspects. Where research or information is available, potential mitigation avenues will be reviewed.
45
Enlarged yet concentrated attack surface. The split architecture and centralized management of an SDN system is mentioned by several researchers as presenting a larger attack surface than individual switches and routers (Chesla, 2013; Dong et al., 2015; Kreutz, Yu, Esteves-Verissimo, Magalhaes, & Ramos, 2017). Other researchers, including Bobba et al. (2014), Hadley et al. (2017), and Hill & Smith (2017), describe the overall effect of employing SDN networking in an ICS environment as greatly reducing the overall attack surface of the network; this more positive analysis comes with the caveat of ensuring encrypted and authenticated communications from the control plane to the forwarding plane within the OpenFlow protocol. Notwithstanding the narrow application space of ICSs with enhanced communication protections, the increased attack surface area is perhaps the most consistent negative cybersecurity aspect of SDN mentioned in the literature and requires thoughtful consideration and engineering to mitigate the attendant threats. Centralized management increases attack surface. One example of the increased attack surface comes from the mere presence of the centralized management and control (Chesla, 2013; Open Networking Foundation, 2015). This single server concentrates all the SDN network architecture knowledge, flow monitoring, and control in one server; if this server is compromised, at that time control over the whole network and a large part of its cyber-defenses go to the attacker. Careful cyber-aware design can mitigate the risks of central management. The single-point-of-failure can be mitigated by choosing orchestration software capable of running on redundant servers or on a high availability server cluster. Additionally, the orchestration server can be defended with extra layers in the overall network defense-in-depth strategy.
46
Management communication itself is a vulnerability. Another increased opportunity for service interruption involves two threats related to the split between the control plane and the forwarding plane (Open Networking Foundation, 2015). The split requires that the control plane will need to communicate new configurations to the forwarding plane periodically. This communication is now subject to either interception via a man-in-the-middle style attack or from a crude denial-of-service attack (Chesla, 2013). The man-in-the-middle attack can now be mitigated with the use of encrypted and authenticated configuration downloads, while the denialof-service attack should be partially mitigated by the SDN system itself due to the zero-trust posture of the forwarding plane (Hadley et al., 2017). Hill and Smith (2017) claim OT networks can be completely protected by ensuring all unauthorized packets are immediately dropped at the edge of the network. More testing and development will be required to determine if these threats are credible and if the mitigations are effective. Possible timing issues with distribution of configuration messages. The centralized management causes an additional possible problem. When a new configuration is compiled and needs to be sent to the forwarding plane there is no guaranteed delivery on the first try or deterministic timing to ensure all the messages arrive and are implemented simultaneously (Bobba et al., 2014). This allows for some unknown time-frame during which the switching fabric would not be synchronized and some messages transiting the network could be lost or delayed (Little, 2013). Although this temporary unknown state is potentially a real problem, Bobba et al. (2014) point out that a standard network has the same problem, but to a larger degree. In an SDN network, the unknown state might last several milliseconds causing a few data packets to be dropped; while in a standard network, human implemented configuration changes could require a much larger time window, on the order of tens of minutes to hours, for
47
all the configuration changes to be entered manually across a large network. Mitigation could be a transmission protocol with verification of message receipt and future time of implementation that waits for a confirmation. More research is required to determine if time synchronized configuration changes can be reliable and avoid indeterminate states within the forwarding plane of the network. Programmability introduces uncertainty. The centralized controller allows a programmatic configuration changes in response to conditions within the network or specific traffic on the network (Open Networking Foundation, 2015). This type of flexibility could lead to unstable configurations during operation due to programming errors. Additionally, a denialof-service attack could render the controller incapable of sending programmatic responses to the forwarding plane (Chesla, 2013). Possible remediation includes simulation of the programming prior to enabling in a live system and fail-safes that revert to previous or default configurations in certain situations (Bobba et al., 2014). For ICS implementations, the use of programmatic configuration changes would likely be disabled or used only in well-understood conditions. Here, OT SDN researchers argue that the simulation abilities available to SDN networks via Open vSwitch and Mininet simulation tools allow specific network configurations to be preverified making them safe for programmatic implementation (Dolezilek, 2018). OT networks should be stable networks and the need for programmable configuration changes in response to unforeseen network conditions or unexpected data traffic should be non-existent. SDN specifically implemented for cybersecurity in OT networks. There are few instances of peer-reviewed research into SDN implemented specifically in ICS or OT environments. The earliest relevant research appears to be from the International Journal of Critical Infrastructure Protection in the 2013 paper Flow Whitelisting in SCADA Networks
48
(Barbosa, Sadre, & Pras, 2013). While not a full SDN implementation, flow whitelisting is a central component of SDN. Then, in 2014, Software-Defined Networking Addresses Control System Requirements directly addresses SDN in a critical infrastructure OT environment (Bobba et al., 2014). Dong et al. (2015) submitted a position paper on the use of SDN for improving the resilience of the smart grid communications. This paper was accepted for the first workshop of the Association of Computing Machinery on Cyber-Physical System Security, a term roughly equivalent to ICS cybersecurity. End-to-end SDN Orchestration of IoT Services Using an SDN/NFV-enabled Edge Node advances the idea of using SDN technologies to integrate the differing network layers tightly and service requirements found in typical OT environments (Vilalta et al., 2016). Hadley et al. (2017) present detailed research, which was performed as part of the ongoing US Department of Energy (DOE) Chess Master smart grid network resiliency improvement project in Software-Defined Networking Redefines Performance for Ethernet Control Systems. A Spanish research team presents cybersecurity solutions based on SDN to protect the data networks required for their proposed Virtual Factory Open Operating System (Fraile et al., 2018). Finally, Fraile, and Poler join with another research team to detail the merits of an SDN-based Firewall in securing OT networks (Tsuchiya et al., 2018). While not a large number of papers, they cover a significant range of SDN benefits and detriments in an OT setting while reinforcing each other and noting no significant disagreements. Due to the relative paucity of journal articles, additional industry reports and whitepapers will be reviewed to reinforce the scholarly papers in the literature review. In 2016, a group of networking and virtualization focused companies held the Future: Net conference after which the MIT Technology Review staff produced the Pioneers and Giants: Insights on the Future of Networking report (MIT Technology Review Insights, 2016). The DOE Chess Master project,
49
initiated in 2015, has not yet concluded and thus a peer-reviewed discussing the full project results does not yet exist; however, a detailed whitepaper does exist and is reviewed below (Hill & Smith, 2017). The 2016 Recommended Practice: Improving ICS Cybersecurity with Defensein-Depth Strategies report from the ICS-CERT team did not mention SDN technologies directly; however, it discusses in detail the best practices for OT network cybersecurity (ICS-CERT, 2016). Network Virtualization: The Bridge to Digital Transformation is another MIT Technology Review report from August of 2017 that discusses how software-based network management strategies provide security and flexibility required for the continued digitalization of multiple industries (MIT Technology Review Insights, 2017). Early research on flow whitelisting in OT networks. A core concept of SDN is the use of high-level data flow definitions on the control plane to abstract the individual device configurations on the forwarding plane. This construct relieves the SDN network administrator from maintaining the complex, difficult to implement device level configurations. While Barbosa et al. (2013) do not directly mention SDN in their Flow Whitelisting in SCADA Networks paper, they comprehensively argue the merits of whitelisting in general, and specifically the whitelisting of data flows in OT environments. This paper states that flow whitelisting effectively reduces the number of cybersecurity attacks to which a SCADA system is exposed (Barbosa et al., 2013). The authors acknowledge they encountered many falsepositive, unknown data-flow alarms due to their inadequate attempt to automatically learn the appropriate data flows. They remark that they intend to improve the machine learning algorithm as part of their future research. Various implementations of SDN methodologies discuss both a self-learning approach for existing networks and a fully pre-designed, simulated, and validated approach for existing and new networks (Dolezilek, 2018; Hill & Smith, 2017).
50
DOE sponsored research on SDN networks. The US government formally recognized the need for intense, broad-based, and sustained research into security for information networks as early as 1997 with the development of the Hard Problem List (CNCI, 2009). The HPL was created by a group of more than 15 US government agencies and departments collectively known as the INFOSEC (Information Security) Research Council (IRC). The IRC also regularly consults with Canadian and British government agencies. The HPL was formally published in 1999 and then significantly updated in 2005 when it included eight technical problem areas (CNCI, 2009). Then, in 2009 the Department for Homeland Security published A Roadmap for Cybersecurity Research in response to the 2008 National Security Presidential Directive 54/Homeland Security Presidential Directive 23 (CNCI, 2009). These joint presidential directives created the Comprehensive National Cybersecurity Initiative (CNCI) to lead the US government’s efforts to transform the United States cyber-infrastructure (CNCI, 2009). CNCI research efforts. The CNCI effort is focused on protecting critical US national interests from catastrophic damage while allowing society to continue adoption of advanced technology. The roadmap for cybersecurity research included the following research topic areas: 1. Scalable trustworthy systems (including system architectures and requisite development methodology) 2. Enterprise-level metrics (including measures of overall system trustworthiness) 3. System evaluation life cycle (including approaches for sufficient assurance) 4. Combatting insider threats 5. Combatting malware and botnets 6. Global-scale identity management 7. Survivability of time-critical systems
51
8. Situational understanding and attack attribution 9. Provenance (relating to information, systems, and hardware) 10. Privacy-aware security 11. Usable security This list adopted the eight IRC HPL areas and added three additional areas (i.e. 3, 5, and 11) which had become more relevant in the four years since the update of the HPL (CNCI, 2009). The DOE, an original member of the IRC and the CNCI, fully realized the needs for advanced cybersecurity in OT networks. Early smart grid initiatives highlighted the already large and increasing data transmission volume and security requirements (Gao, Xiao, Liu, Liang, & Chen, 2012). The smart grid presented equally pressing needs for highly-secure, deterministic, high-speed, low-latency networks to fully realize complex new power coordination schemes that increase power reliability for health care, fire and rescue, financial, and other critical sectors (Gao et al., 2012; He, Liu, Ding, Li, & Zhang, 2017). DOE specific cybersecurity roadmap. These pressures led to the creation of the DOE Roadmap to Achieve Energy Delivery Systems Cybersecurity (Energy Sector Control Systems Working Group [ESCSWG], 2011). The DOE specific cybersecurity roadmap included the follow strategies: 1. Build a Culture of Security 2. Assess and Monitor Risk 3. Develop and Implement New Protective Measures to Reduce Risk 4. Manage Incidents 5. Sustain Security Improvements
52
The third strategy has been manifested with three separate public/private/academic partnership projects; Project Watchdog, SDN Project, and Chess Master. The last two projects have produced several journal articles and whitepapers, which will be discussed below. Additionally, these DOE sponsored research projects have resulting in the commercial release of OT targeted hardware and software including an SDN switch and controller appliance from SEL and a network management application from Veracity. Bobba et al. (2014) produced a paper titled Software-defined networking addresses control system requirements, which directly and comprehensively addresses the use of SDN to improve network performance while increasing cybersecurity in an OT network environment. This paper is based on a DOE sponsored project called the SDN Project and highlights the advantages of SDN in energy-sector specific ICS networks for improving reliability and cybersecurity. Bobba et al. (2014) identify five categories of challenges in electrical infrastructure OT networking not met by traditional commercial IT equipment and software including: 1. Planning, design, and testing to prove system performance before installation 2. Programmatic and time-certain change control that minimizes system disruptions 3. Engineering tools allowing performance guaranteed design with pre-designated backup and recovery schemes and post-implementation verification monitoring 4. Continuous and granular monitoring and visualization of the network 5. Cybersecurity with defense-in-depth and fully whitelisted data flows Each of these categories is addressed and discussed in detail. Planning, design, and testing. Bobba et al. (2014) state that the engineering of data circuits should be similar to electrical circuit design and allow precise definitions of the data
53
paths. Traditional IT networking implements the RSTA which, automatically determines the least-cost routing for packets from source to destination with little ability to specifically design the initial and/or backup paths in case of saturation or equipment failure (Dolezilek, 2018). In the case of SDN, these flow definitions are fundamental to the design process and even if automated can be modeled and verified in advance (Bakhshi, 2017; Dolezilek, 2018). The detailed network documentation of the physical layout and data configurations of SDN deployed in OT networks allows detailed simulation and extensive testing of normal and failure modes (Bobba et al., 2014; Dolezilek, 2018). Programmatic and time-certain configuration management. Configuration management or change control involves tracking both changes of the hardware and firmware underlying the network and the device configurations, which affect data routing (Banks, 2014; Slattery, 2013). The SDN methodology improves on both fronts. Change control of the hardware is positively enforced in that no physical changes can be implemented without updating the network definitions and then recompiling the configurations for the forwarding plane devices (Open Networking Foundation, 2015). Additionally, with a much smaller and simplified code base required on the forwarding plane devices due to the separation of the control and forwarding planes, fewer firmware patches are expected. Furthermore, these patches are centrally managed in the orchestration engine software. Similarly, any data flow definition changes are selfdocumenting in the orchestration engine, can be pre-validated with high-fidelity simulation, and then rolled out in a coordinated manner (Bobba et al., 2014). Engineering tools allowing performance guaranteed design. Existing network configuration and monitoring tools from some large hardware providers allow centralized management of their network devices (Bobba et al., 2014). However, these tools are still
54
focused on the low-level device configurations of individual devices and do not allow edge-toedge data flow definitions with sophisticated simulation and verification (Dolezilek, 2018). Additionally, these vendor solutions typically work best on homogeneous networks of the specific vendor’s equipment; this leaves many existing OT networks with largely varied installed bases stuck with transitioning to expensive new switching equipment (Bobba et al., 2014). On the other hand, SDN is focused on open systems with simplified and less expensive hardware. The data flow-based configuration and operation of SDN brings an often little appreciated, but tremendous advantage over traditional IT networking. The RSTA is an excellent solution for traditional IT networks that require only best effort delivery of packets and can easily tolerate individual data packet loss and retransmission (Dolezilek, 2018). However, power grid systems now rely on OT networks for safety and reliability (Bobba et al., 2014). This means data loss and retransmission is not just a nuisance, but also rather a life safety issue. For proper arc flash remediation, response times in the low, single millisecond range are required (Hadley et al., 2017). The pre-engineered backup flow paths allowed in an SDN environment allow network heal times in the 100s of microseconds; this is two orders of magnitude better than the fastest RSTA heal times (Hadley et al., 2017). These heal times come with the added benefit of pre-engineering the exact backup path, which allows pre-verification of the path latency and loading (Bobba et al., 2014). Conversely, RSTA network healing paths are discovered by algorithms running on each individual switch at the time of network failure and are highly dependent on network architecture (Bobba et al., 2014). This sometimes forces extensive network redesign to obtain acceptable heal times in critical networks, often requiring additional equipment be purchased at the last minute. The interaction between multiple devices running the RST protocol is very difficult to correctly model and simulate; on large, complex networks this
55
often requires waiting for real-world testing and then expensive redesign and retesting iterations to obtain specified heal times. Continuous and granular monitoring and visualization. Critical infrastructure systems, like the power grid, demand continuous detailed monitoring and fast fault responses. Detailed monitoring of IT networks requires pre-engineering and additional hardware and software, while SDN comes with low-level, highly detailed monitoring that is data flow based, not just port or switch based (Open Networking Foundation, 2015). The flow-based monitoring allows much better operator situational awareness because the metrics can be tied to critical ICS functions (Hadley et al., 2017). Instead of diagnosing which ICS functions are affected by a saturated switch-to-switch trunk line or a malfunctioning port, the impacted ICS function itself can be identified immediately by the data flow name. If multiple data flows are impacted by a single fault, they can all be identified and prioritized, leading to fast, priority-based remediation and reverification of the healed network. Cybersecurity with defense-in-depth and fully whitelisted data flows. Maintaining a highdegree of cybersecurity while providing these advanced services is challenging but made easier with the deployment of SDN technologies (Bobba et al., 2014). The very nature of SDN with configurations based on explicit data flow definitions makes SDN a zero-trust architecture, which is a basic tenant of cybersecurity best practices. Fraile et al. (2018) further describe the SDN configured network as having individual firewalls at each port due to the packet matching and forwarding rules, which are applied at each port. The flow-based architecture also forces complete and always up-to-date, detailed, and accurate network documentation with real-time verification feedback, another foundational cybersecurity best practice (ISA, 2007). Data flow definitions allow complete deep packet inspection and protocol verification at every device
56
interface in the network (Fraile et al., 2018). This ensures the correct source/destination pairings with correct protocol, and that the protocol payload is properly constructed. This enables a complete whitelisted lockdown of all data traffic in the network denying-by-default any data that has not been specifically pre-authorized (Hadley et al., 2017). Possible SDN specific cybersecurity weaknesses. All five categories defined by Bobba et al. (2014) were determined to have positive benefits to electric power-related OT networking. This includes significant advantages in the cybersecurity readiness and resiliency category. In fact, the first four categories all reinforce aspects that are important to comprehensive cybersecurity. Notwithstanding all the positive benefits, there are also several possible cybersecurity weaknesses introduced to an SDN managed network (Bobba et al., 2014). These weaknesses are largely present in non-SDN networks but may be amplified in the SDN environment. Most of the additional cybersecurity weaknesses discussed by Bobba et al. concern the central management and programmatic control of the SDN forwarding plane by the control plane and possible programs running on the application plane. An attacker that gained control of the orchestration engine server would have full control of the network. Full network control would allow the attacker to shut the network down completely by erasing all forwarding plane device configurations, to monitor all communications surreptitiously, to modify, insert, or delete data at will, or all the above (Dong et al., 2015). Additionally, the communication between the centralized orchestration engine and the network forwarding plane devices can be attacked with man-in-the-middle and denial-of-service attacks. Attacks on the central management server or the communications from or to it are consistently mentioned in the literature (Dong et al., 2015; Hadley et al., 2017; Hill & Smith, 2017). This additional risk can be mitigated with proper
57
cybersecurity controls for high-value assets. These controls include physical access control, network micro-segmentation, multi-factor authentication, encryption of configuration data in motion and at rest, dual-authorization, audit trail monitoring, remote non-networked monitoring, and more (US DHS, 2017). DOE supporting ongoing research on SDN for OT networks. The US Department of Energy (DOE) is currently sponsoring research investigating the use of SDN based OT networks in the energy delivery and protection industry. One project named the Engineered Cybersecurity Chess Master Project, is expected to complete September 2019 and is part of the Cybersecurity for Energy Delivery Systems (CEDS) program (CEDS, 2017; Hill & Smith, 2017). The goals of the research are to develop, test, and commercialize SDN technologies that enhance the cybersecurity and network performance of ICSs (Shockley, 2017). The DOE’s partners on the project are Schweitzer Engineering Laboratories (SEL), Veracity Industrial Networks (Veracity), Ameren Corporation, and Sempra Energy (Santo, 2017). SEL provided the SDN compatible switches and the flow controller while Veracity is providing the Industrial Security Orchestrator software (CEDS, 2017). Ameren is an electrical power company and Sempra is an electric and gas energy company; together they provide real-world testbeds for the new technology. The Chess Master project. The Chess Master project has two phases, each with specific goals. Phase One concentrates on developing the software and hardware required to deploy SDN in a real ICS environment in the energy sector (Hill & Smith, 2017). Phase Two entails the actual deployment of the equipment and software in a test environment and development of bestpractice guides (Hill & Smith, 2017). Specifically, the Chess Master research deliverables include research, development, testing, and release of the following (Hill & Smith, 2017): 1. A security policy enforcement software engine which runs on the flow controller
58
2. A small OpenFlow compatible Ethernet switch for field mounting 3. An extension to the OpenFlow standard that allows for automated cryptographic key distribution 4. Visualization tools to increase operator situational awareness 5. Software to automatically apply pre-configured security postures in response to network conditions and traffic 6. Best practices documentation for system architectures and administrative procedures 7. Techniques to evaluate pre-configured contingency plans and suggest improvements Phase One of the Chess Master project is complete. The deliverables listed above were turned over to the DOE in mid-2017 (Santo, 2017). The project is now in the real-world demonstration phase with results expected in mid-2019. The Hadley et al. (2017) paper directly addresses SDN used to define OT networks and enhance the performance of critical ICS networks. This research includes test results from work performed as part of the US DOE Chess Master program and presented at IEEE conferences. Hadley et al. (2017) found that SDN significantly improves cybersecurity. The team cited cybersecurity improvements from increasing situational awareness, to removing two primary attack vectors present in non-SDN networks, eliminating the need for signature-based security software, and reducing the opportunity for human error. Additionally, the Hadley team determined that SDN could improve safety and cybersecurity by increasing packet delivery performance, enhancing overall network reliability, and allowing fast, pre-planned failure recovery (Hadley et al., 2017). The SDN methodologies map well to ICS environments due to the pre-engineered nature of OT networks, all devices and communication flows should be known as part of the design process (Hadley et al., 2017).
59
The Hadley et al. (2017) team reviewed SDN cybersecurity impacts by evaluating the effects of known vulnerabilities of IT networks on an SDN network and by performing a sideby-side threat modeling exercise. Two vulnerabilities known to be difficult to defend in nonSDN IT networks are the Bridge Priority Data Unit (BPDU) spoofing and Media Access Control (MAC) source address spoofing and additionally known as MAC table poisoning (Hadley et al., 2017). The MAC address spoofing attempts to steal data destined for the device, which legitimately owns the spoofed MAC address; it can be defending by deploying a complicated server-based MAC address authentication service. The end goal of BPDU spoofing is for the attacker to attain the role of root switch from which they can control routing of all packets (Hadley et al., 2017). Unfortunately, there are few ways to guarantee protection from BPDU spoofing and they all include manually turning off the functionality on a port-by-port basis or adding filtering to a system that could complicate future changes to the network topology. An SDN network defeats both issues with no additional configuration. By default, the BPDU attack cannot succeed since it requires the RSTA functionality, which is not present in an SDN based network (Hill & Smith, 2017). Similarly, the MAC address spoofing attack is defeated due to the lack of MAC tables in SDN-style forwarding plane switches, thus there is nothing to poison. Hadley et al. (2017) emphasize that all SDN configuration messages are encrypted and authenticated further securing against attempts to gain control of individual devices on the forwarding plane. The built-in data flow whitelisting of SDN provides additional protections by identifying legitimate packets at multiple levels and dropping packets, which are not identified as part of a known and authorized data flow (Hadley et al., 2017). The Hadley et al. (2017) team also identified the improved overall performance of SDN networks as a benefit to cybersecurity. The fact that the network is more streamlined and
60
performs better allows for additional cybersecurity safeguards without negatively impacting performance which would likely discourage the addition of cyber-safeguards in a standard ITstyle network supporting the deterministic needs of an ICS. The researchers also commented on the improved detail of flow-based monitoring leading to improved situational awareness as being a benefit to cybersecurity (Hadley et al., 2017). Additional use cases where SDN increases resilience. Dong et al. (2015) propose additional uses of the programmatic capabilities of SDN to further cybersecurity and overall resilience of OT networks. They propose dynamically reconfiguring the network to allow only commands to remote controllers (e.g. PLCs, RTUs, Variable Frequency Drives, etc.) during very short windows of time thereby limiting the opportunity for rouge elements to issue unauthorized commands. This Temporal Filtering technique is promoted by Tsuchiya et al. (2018) in their paper describing an SDN based firewall. Tsuchiya et al. (2018) also describe a Spatial Filtering technique that dynamically varies the SDN forwarding rules based on open connections within the ICS. Dong et al. (2015) envision a monitoring system like an Intrusion Prevention System (IPS) with the capability to reset switch configurations automatically if a compromised or misbehaving switch is detected. Finally, Dong et al. (2015) propose a disaster recovery fallback where the communications would be temporarily routed over the public Internet if the internal network were degraded to the point of not being capable of supporting the intended process. For long distance links, this could be a serviceable backup option with proper encryption and other controls (Dong et al., 2015). All these additional techniques rely on active programmatic data forwarding rule generation and individual downloads to the forwarding plane infrastructure. This requires highly reliable and verifiable rule generation programs with regular updates from the flow orchestration engine to the forwarding plane and would therefore increase
61
overall system reliance on the orchestration server and communication between the server and the individual forwarding devices. Effects of SDN Workloads on OT Network Personnel Network administrators supporting business function IT networks are often full-time dedicated personnel while production OT networks tend to be supported by local engineering or technician level resources as one of their many responsibilities. According to Lobo (2018), most OT teams are still learning basic IT networking skills and require significant training in cybersecurity. This creates a dependency that causes friction between IT and OT staffs when IT tries to help but does not fully understand the OT priorities and operating requirements (Lobo, 2018). Further friction results from the delays induced by the dependencies on IT resources, when OT personnel need to make a change to start or restore production they may be tempted to go outside previously negotiated policies and procedures. This get-it-done attitude is rewarded in OT environments, but may cause additional work for the IT personnel tasked with investigating and resolving the resulting IT policy violations. The instinct among OT responsible personnel to resist reliance on outside resources results in the avoidance of implementation of any equipment or software that would require help from those external resources (Lobo, 2018). Unfortunately, the lack of cybersecurity training and experience in OT support personnel leads directly to significant resistance around the installation and maintenance of critical cybersecurity controls (Bruijn & Janssen, 2017). Very little help available. Additional strain is caused by the difficulty in finding and hiring additional personnel to dedicate to the cyber-improvement work. The NIST cyberworkforce report delivered to the US Whitehouse on November 17, 2017 claims 299,000 active job openings for cybersecurity-related positions in the United States as of August 2017. The
62
same report sites global projections for a shortage of 1.8 million cyber-workers by 2022 (NIST, 2017). The U.S. Presidential Executive Order 13,800 (2017) makes the development, continued expansion, and sustainment of a skilled cyber-workforce a national goal. A 2017 survey of 1,900 LinkedIn Information Security Community professionals revealed that 45 percent of respondents viewed the lack of skilled employees and the lack of budget as the biggest impediments to improving cybersecurity (Crowd Research Partners & ISC on LinkedIn, 2017). In the same survey, 39 percent of firms would consider outsourcing some cybersecurity to a managed service provided due to the lack of qualified employees (Crowd Research Partners & ISC on LinkedIn, 2017). Reduction in overall training requirements. The incorporation of SDN technologies can significantly reduce the training requirements and time required to maintain and implement changes in a complex OT network (Mahankali & Rungta, 2014). There is a need for additional training on the SDN orchestration software, especially for network device layout definition, data flow definitions, and new troubleshooting methods; however, the improved situational awareness and the removal of need to learn multiple CLI-style languages helps OT resources quickly adapt to SDN provisioned networks (Mahankali & Rungta, 2014). The OT personnel also appreciate the engineering-oriented capabilities of SDN, which allow new definitions to be simulated and verified before deployment in the live production environment. Additionally, engineers and technicians maintaining OT networks benefit from the data collected in the central management server of an SDN installation. The network telemetry inherent to SDN-style networks reports network-wide, fine-grained metrics and diagnostic data to the central server. This wealth of data gives a more complete view of the network status and eliminates the need to physically visit or login to multiple switches for troubleshooting purposes. The central management sever
63
simplifies making changes to the network in the same way, eliminating the need to visit or login to multiple devices. The SDN central management functions also reduce the specialized skills and time required to recover from hardware failure, removing dependencies on human configuration and verification after replacement of field devices. (Hadley et al., 2017) Summary The literature demonstrates that current corporate management techniques and business goals require the integration and convergence of IT and OT networks (McKinsey Digital, 2015; Oxford Economics, 2011). However, the two disciplines of IT and OT have differing priorities and constraints, which have not been reconciled in most companies (Brocklehurst, 2017; Harp & Gregory-Brown, 2015; Lobo, 2018). These differences include the ever-present safety-first mantra of manufacturing which leads to the inversion of the original IT cybersecurity CIA triad of priorities into the OT cybersecurity order of priorities, Safety, Availability, Integrity, and Confidentiality (SAIC) (Bodungen et al., 2017; Luiijf & Paske, 2015). At the same time, OT network administrators are typically multi-taskers with a myriad of other responsibilities making training and obtaining experience with cybersecurity tools challenging. The lack of deep cybersecurity skills in the OT network administration resource pool leads to unwanted and uncomfortable dependencies by OT personnel on IT networking and cybersecurity design and configuration resources (Lobo, 2018). These dependencies can lead to friction when IT personnel do not respond to OT network and cybersecurity configuration requests promptly or with full understanding of the IT / OT differences (Lobo, 2018). Many of the problems introduced by the IT / OT convergence and subsequent dependencies can be addressed with SDN technologies (Anwer et al., 2010; Open Networking Foundation, 2017). By its very nature, SDN improves the cybersecurity resilience of most
64
networks by forcing accurate and up-to-date network documentation, zero-trust data flow definitions for the network, and greatly enhanced situational awareness due to granular, understandable, flow-based monitoring (Bobba et al., 2014; Hadley et al., 2017; Hardy, 2017). Additionally, SDN configuration techniques closely map to standard process engineering steps of detailed design, iterated simulation, and testing, with pre-installation verification (Bobba et al., 2014). The training requirements of SDN also lend themselves to engineers and technicians on the factory floor by not requiring the learning of multiple, individual switch and router configuration CLI languages, but rather allowing centralized edge-to-edge data flow definitions (Lobo, 2018). The increased attack surface area of SDN afforded by the centralized configuration management in the orchestration server rightly concerns IT professionals (Dong et al., 2015). The centralized configuration server concerns on two levels. First, multiple sources point out that having the configuration and control in one place makes the orchestration server itself a tempting target for hackers (Bobba et al., 2014; Dong et al., 2015). Second, the possibility of an attacker disrupting the orchestration server’s communication to the forwarding plane devices with a MITM or DOS attack must be remediated effectively (Hadley et al., 2017). Lastly, SDN represents a significant change in design and operations, and requires rethinking the design, implementation, and support of OT and IT networks (Lobo, 2018).
65
Discussion of the Findings The business-focused IT networking environment and production-oriented OT networking environments are inexorably converging (BAE Systems Applied Intelligence, 2014; Gregory-Brown & Harp, 2016). Business requirements for accurate and timely data are constantly reinforcing the need for high-volume and high-speed connections between IT networks and OT networks (Oxford Economics, 2011). The desire for lower costs and standardization are driving the adoption of Ethernet at all levels within the OT space (Littlefield, 2015; Pettey, 2016). In turn, both trends are forcing the necessity for increased cybersecurity in the OT space while respecting the differences in priorities, CIA versus SAIC, and the differences in background and skill sets between IT and OT personnel (Lobo, 2018; Thilmany, 2017). This research project set out to determine the unique properties and requirements of OT networks that are not well understood by most IT personnel. These misunderstandings are important because they lead to friction between IT and OT administrators and ultimately to poor implementation of IT developed cybersecurity tools and best practices in OT networks. Furthermore, this research project intends to discover the possible cybersecurity benefits and detriments of SDN implementations in OT networks. Finally, the research focused on how SDN deployments impact OT network support personnel. Unique Attributes of OT Networks Affecting Cybersecurity Research question Q1 seeks to identify unique requirements or special considerations for OT networks in the context of implementing cybersecurity. The research reviewed clearly identifies fundamental differences between IT and OT practitioners’ approach to and understanding of the order of priorities cybersecurity should provide (Harp & Gregory-Brown, 2015). Aside from significant cultural differences, there are additional technological differences
66
that must be understood and accounted for when attempting to apply the IT-centric cybersecurity standards and best practices (Brocklehurst, 2017). However, a solution that reinforces overall OT cybersecurity without hindering the root purposes and requirements of the OT network can and should be embraced by both philosophical groups (Lobo, 2018). Dissimilar priorities. Aydell (2015), Bodungen et al. (2017), Brocklehurst (2017), Lobo (2018), and many other researchers and commentators discuss the differences in cybersecurityrelated priorities between IT and OT network administrators and engineers. This is most easily demonstrated with the inversion of the IT focused CIA triad of cybersecurity priorities into the OT focused SAIC priority progression (Weiss, 2012; Maddison, 2018). The clear takeaway is that the OT emphasis on safety and availability above data confidentiality and integrity fundamentally separates the OT networking administrator’s mindset from that of an IT networking administrator (Chalfant, 2018b; Harcharan et al., 2018). This basic difference in approach to design and operation of networks leads to misunderstandings and a lack of trust between OT and IT network personnel (Brocklehurst, 2017; Lobo, 2018). A lack of trust between these key groups leads to friction and an inability to drive change and take the actions required to increase cybersecurity in the OT environment (Lobo, 2018). Separate reporting structures and cultures. The misunderstandings at the network design and administration level are echoed and reinforced by the organizational structures of most companies. The IT and OT lines of control do not usually combine until the C-Suite where the CIO and the COO sit at the same table. This creates a whole hierarchy of competition for personnel and budget resources. Luiijf & Paske (2015) and Bindseil (2003) argue for strong CLevel sponsorship of increased cybersecurity using a risk-based approach. These ideas mesh with Collier’s et al. (2016) calls for well-defined metrics to guide the continuous improvement
67
approach to changing the culture and increasing cybersecurity readiness. Several researchers and standards bodies, including NIST, advocate for the cross-training or combining of the IT and OT teams to foster understanding, communication, cooperation, and hopefully an appreciation for the difficult tasks both teams face (Lobo, 2018; Stouffer et al., 2015; Thilmany, 2017). Physical differences due to design requirements. The IT and OT networking environments are drastically different due to their different design criteria (Harcharan et al., 2018). It is not hard to understand why the two types of networks are significantly different. The differences are apparent when one considers the different priorities expressed in the CIA versus SAIC acronyms and the dissimilar physical environments and life cycles (Brocklehurst, 2017; Luiijf & Paske, 2015). Similarly, it is easy to contrast IT’s generally clean, airconditioned environment and short lifecycles with OT’s often dirty, hot, humid, electrically noisy environment and long life-cycles (Harp & Gregory-Brown, 2015; Maddison, 2017). These differences are reflected in the companies that sell equipment into the IT and OT spaces. There is a separation of IT and OT even at the equipment supplier level. This separation is further expressed in the separation of engineering and technician talent due to training and experience with one set of equipment tending to keep those engineers and technicians working in the area with which they are familiar (Welander, 2013). Attack surface distinctiveness. When IT professionals review an OT network they see many of the same vulnerabilities that exist in IT networks, including unpatched OSs and applications, remote access backdoors, USB flash drive hygiene, anti-virus update requirements, lax or out-of-date firewall rules, and many others (CSSP, 2011). All these vulnerabilities do exist and are important in an OT network; however, the distinctive features between IT and OT networks must be respected in how these vulnerabilities are addressed (Kobara, 2016). One
68
essential difference is that many ICS applications on the OT network are static with no updates available (BitSight Insights, 2017). The static condition may be due to lack of budget for the update maintenance contract or the applications may not be supported by the developer anymore. This requires careful research and testing to determine if the OS patches and other application updates are compatible with the older applications. Dale Peterson (2011) has long been cautioning about a vulnerability not seen in today’s IT networks which is the existence of insecure-by-design equipment. Many of the PLCs and proprietary machine controllers are designed with no authentication or encryption for their Ethernet interfaces. Additionally, Joe Weiss (2018) warns that the Level 0 instrumentation connected to the controllers presents additional attack surface, which is mostly ignored in cybersecurity audits. Weiss (2018) says that if the innate vulnerabilities of the equipment connected to the network are not addressed then much of the rest of the cybersecurity protections are a waste of time and effort. Another unique attack vector for OT networks is the continued use of old serial protocols, which have been adapted to communicate over Ethernet, but have not been upgraded with security provisions. Finally, Wallace & Carter (2010) remind that many industrial processes and support equipment are leased or bought with maintenance contracts that require twenty-four-hour access via the internet for monitoring and technical support. How SDN Enhances the Cybersecurity of OT Networks Research question Q2 focuses on how well SDN technologies address the issues of cybersecurity within an OT network. This section will discuss the ways that SDN technologies benefit cybersecurity of OT networks. A following section will discuss possible detriments to cybersecurity and their associated mitigation strategies.
69
The application of SDN technology in an OT network contributes many benefits to the cybersecurity and overall resiliency of the network. The research reviewed has shown SDN leads to greater rigor in the documentation, better situational awareness, and a higher level of detail in the design of a network (Dolezilek, 2018; Hill & Smith, 2017). The SDN basic methodology of whitelisting data flows or zero-trust fits well with the static and fully characterized nature of an ICS network (Bobba et al., 2014; Hadley et al., 2017). The SDN benefits work together to lower overall workload on the personnel responsible for OT network implementation, maintenance, and support. Admittedly, there are a few SDN characteristics that cause additional cyber security concerns including centralized management, programmability, and personnel requirements (Chesla, 2013; Dong et al., 2015; Kreutz et al., 2017). These general categories are discussed in detail below. Comprehensive documentation via data flow definitions. An SDN configured network inherently provides the complete, up-to-date, and detailed documentation that is often mentioned as the first step toward securing a network (Gregory-Brown, 2017; Weiss, 2012). The first part of designing an SDN network is to provide a complete list of equipment present in the forwarding plane and the connections available between the individual devices (Gregory-Brown, 2017). Then every data flow within the system is explicitly defined so the controller can determine the best routes and download the flow forwarding configurations to the individual forwarding devices (Dolezilek, 2018). At this point, the network is comprehensively documented to a level rarely, if ever, seen in a standard network. Additionally, there is immediate feedback if the documentation is wrong because the configuration download will fail, or data will not flow between the designated devices. Data flow definitions are much easier to formulate at design time than tracing data transmission paths through multiple switches and
70
routers and building individual configuration scripts for each device. The data flow definitions are even easier to interpret when audit personnel are reviewing the network or when new engineers, maintainers, and operators are learning a network (Fruehe, 2016). A well-documented network is much easier to understand and defend than a network with multiple unknowns, whether they are rouge-networking devices in the field, undocumented data flows, or connections that have been added or moved. Enhanced situational awareness. The full, intrinsic documentation of an SDN network, with constant and detailed monitoring at a granular level, produces unparalleled situational awareness (Bobba et al., 2014). The SDN data flow definitions contribute here again. These definitions provide more context to the data than simply creating a series of port assignments, routing statements, and firewall rules across multiple switches and routers. The data flow definitions encapsulate the purpose for the data transiting a switch or router and allow the proper amount of specific routing and provisioning that is required to complete the transmission (Hadley et al., 2017). An SDN network also provides intrinsic monitoring of the network with multiple diagnostic counters implemented in each forwarding device (Open Networking Foundation, 2015). The various counters exist for network health monitoring and programmability, but they also provide unparalleled situational awareness (Open Networking Foundation, 2015). Network administrators can easily monitor the individual device-based and port-based statistics that are familiar and common to non-SDN networks. However, they also have access to data flow path statistics, which give much more fine-grained detailed information about where data are coming from and going to and how to reroute individual flows to relieve bandwidth capacity issues (Hadley et al., 2017). Network administrators can also provide short, high-speed backup paths to high-priority data flows while shunting lower-priority data on longer,
71
slower paths (Bobba et al., 2014). This added awareness and fine control makes it easier for an administrator to diagnose suspect data flows and recover from abnormal conditions, thereby increasing cybersecurity and overall resiliency of the network. Full whitelist control of the data in the network. Data only enter the network when it matches the expected source, type, and destination (Dolezilek, 2017). The network-forwarding plane is configured by default to drop any data traffic that does not explicitly match one of the data flow definitions associated with that port including the data type, source, and destination (Hill & Smith, 2017). This will defeat most viruses, worms, and ransomwares by not allowing lateral movement within the network and blocking any attempt to communicate with a command and control server. Another benefit is that SDN does not require constant updating of signatures like anti-virus and other blacklisting methods. The lack of needing constant updates reduces the overall attack surface of the network because no connection is required to an outside server, and it eliminates the inherent delays of a signature-based system imposed by the time required to find, analyze, fingerprint, and upload signatures for new threats (Hadley et al., 2017). A whitelisted system is proactive and able to protect against new unknown cyber-threats instead of being only capable of reacting to old, previously identified threats. Lower management effort and costs. Centralized management of SDN allows for fully documented, managed, and auditable change control (Fujitsu Network Communications Inc., 2014). The central management server also facilitates nearly instantaneous changes synchronized across the whole network. Changes to a non-SDN network require a technician or several technicians logging into multiple switches and making configuration changes that might temporarily put the network into a non-functioning state until all the changes are completed (Goransson et al., 2017). An SDN orchestration server can download multiple changes to
72
multiple devices with a pre-defined time to take effect, which greatly minimizes the timewindow for the network to be in an undesirable configuration. This means one engineer or technician can make complex, network-wide changes requiring hundreds of individual configuration modifications. The engineer can then simulate those changes to pre-validate them, and then schedule those changes to be effective at a certain date and time. The full documentation, detailed monitoring, and simulation capabilities of SDN all facilitate quicker and easier learning by new or transitioning personnel, allowing them to become effective team members more quickly. All these benefits of SDN contribute to reduced management overhead and engineering and technician time for support of the network, which directly translate into more time for planning and testing network enhancements including additional cybersecurity measures. How SDN detracts from the cybersecurity of OT networks Research reviewed during this project correspondingly revealed some concerns about how SDN technologies might negatively affect cybersecurity for any network. The concern most often mentioned focused on the use of a central server for configuration and control of the network. Additionally, the ability to respond to network conditions autonomously via preprogrammed responses created concern about whether those changes could create instabilities in the network. Finally, the lack of network engineers with training and experience in both SDN technologies and OT networks supporting complex ICSs was mentioned as an increased risk. These possibly detrimental aspects of SDN are discussed below along with some potential mitigation strategies. Increased risk from centralized management. The research revealed consistent concerns with the centralized management server as a potential single-point-of-failure and a
73
high-value target for attackers. With all network control emanating from one orchestration server, if that server were compromised, an attacker would have complete control over the network. That control would allow the attacker to re-configure the entire network and either shut it down or give them the means to cover their tracks while accessing the entire network. Even without control of the orchestration server, an attacker could still target the communication from the server to the forwarding plane devices with a MITM or DOS attack. In these cases, the SDN network itself is a mitigating factor, by providing increased network security on multiple levels. The SDN deny-by-default, zero-trust network provides significantly increased protection against external penetration of the network and eventual access to the orchestration server. Additional mitigations can be employed to secure the orchestration server further. These mitigations might include physical access control, multi-factor authentication, application control (a.k.a. executable whitelisting), and any other protection afforded a high-value network asset. The single-point-offailure concern can be easily mitigated with stand-by servers or at least virtual machine backups ready to be restored quickly. While these risk scenarios are concerning, the dramatically increased cybersecurity capabilities, and other benefits afforded by the SDN technology paired with the available mitigation techniques seem to outweigh the increased risk. Unknown risk of instability from programmed responses. The research identified another possible risk in the SDN feature that allows programmatic changes to the network configuration. These changes can be made in response to network status, unexpected data, equipment failure, or even new service requests. The concern is that any program can contain bugs or can react in unintended ways to unexpected inputs. This could allow a case where automated responses change the network configuration in a way that creates an unstable or nonfunctional network. In the case of an ICS where network changes are seldom made on the fly
74
and should never be made without advance engineering, testing, and implementation planning, this is a moot point. Most ICS network administrators would probably restrict any programmatic configuration changes to pre-tested configurations with the programmatic implementation being well tested off-line. Additionally, the engineering leadership and management might decide not to allow any programmatic changes as ICS networks are typically static in nature. Again, there appear to be adequate mitigation strategies and controls that would negate the increased risks. Lack of trained and experienced personnel. The literature additionally points to the lack of qualified personnel as a potential risk point in a move to SDN controlled OT networks. Most companies are already having difficulty finding enough trained and experienced engineers and technicians to maintain existing IT and OT networks. The added difficulty of finding personnel conversant with OT requirements, cybersecurity principles, and experienced with SDN technology may push timelines and increase costs. While these requirements might seem daunting, creativity might allow hiring an SDN expert and pairing them with an internal IT cybersecurity expert and an OT network specialist. Additionally, there are a few companies beginning to offer products and services aimed at improving the cybersecurity of ICS networks with SDN technologies. Effects of SDN on the Personnel Responsible for OT Networks Research question Q3 concerns the effects of SDN implementation on personnel supporting OT networks and has been the most difficult for which to find quality research literature. It seems that most human resource focused research literature concerning manufacturing personnel centers on worker health, job satisfaction, change management, lean manufacturing, and the effects of various management techniques. There is some mention of personnel issues in the literature on SDN and IT / OT convergence, which mainly focuses on the
75
availability of trained and experienced personnel (NIST, 2017). These issues have already been addressed in this research project from a cyber-risk point-of-view. This part of the discussion will focus on the direct effects from the workers’ points-of-view and will rely on logical inference based on research reviewed in the previous two sections. Reduced overall load on personnel. Like any major technological change, a transition to an SDN based network will require a significant amount of work, to include research, learning new tools, partnering, design exercises, simulation, budgeting, scheduling, implementation planning, and some physical work. However, the SDN ecosystem helps transform what seems like a huge workload into a manageable exercise with most of these tasks available as outsourceable services (Morgan, 2017). The first big payback from an SDN transition is that many of the tasks required to prepare for SDN have cybersecurity related benefits that network administrators should be working toward already and will improve the maintainability and possibly the performance of the current network. These steps include fully understanding and documenting the existing network including hardware devices with models and firmware versions, physical connections between hardware devices, and logical connections via data flow descriptions. While these steps can take a significant amount of time, they can also provide immediate benefits to the existing network by discovering unauthorized devices, identifying throughput bottlenecks, and highlighting outdated equipment. Learning the basics of SDN architecture and how to re-characterize an existing network with data flow definitions will create a temporary increase in workload for OT responsible personnel. Additionally, the discovery and documentation phase will most often require a large amount of effort as many OT networks have poor documentation if any at all. This phase will determine which switches, routers, and firewall devices are compatible with the selected
76
southbound API and then which new devices and software are required to build out the infrastructure, controller, and application planes of the new network. Once the transition to SDN is made, the increase in situational awareness and the ability of one person to configure, simulate, validate, and roll out network-wide changes should significantly reduce the time required to monitor the existing network, troubleshoot issues, and deploy changes. This extra time allows for proactive what-if thought exercises with the opportunity to simulate and validate expected outcomes. Any additional testing before making changes helps reduce risk and lower pressure on the OT network resources. Increased staff training requirements. Anytime a new technology is introduced, there is a transition period and additional training is required. One possible mitigating factor here may be that less overall training might be required for the SDN style OT network than would be required to train engineers and technicians on standard network design principles and technologies. Training on SDN may eliminate or reduce the need for training on multiple switch and router configuration languages, cybersecurity-centric network design guidelines, network monitoring techniques, and low-level, device-based, network troubleshooting techniques. Summary of Findings This research project identified multiple unique properties and requirements of OT networks that are not well understood by the IT community. At the most fundamental level, the basic goals of OT network personnel are to support a safe and reliable manufacturing facility. This means they architect a network to provide safety first, then high-availability, and then data integrity and confidentiality (Luiijf & Paske, 2015). This is the opposite approach to priorities taken by IT personnel who value data confidentiality above integrity and availability and do not normally need to consider human or machine safety (Bodungen et al., 2017). These differing
77
cultures can lead to misunderstandings, which often lead to mistakes and mistrust and those can result in little or no cooperation when IT is asked to help OT get up to speed on cybersecurity (Lobo, 2018). Other major differences are found in the physical deployment environments, which directly influences the design of products sold and who the sellers are of that equipment; they are generally two separate groups of OEMs, traditionally with very little overlap (Maddison, 2018). This important insight illuminates why there is little cross learning between the two groups, they are insulated from each other and do not often attend the same tradeshows or training events. The research also found that the differing priorities combined with the different cultures and styles of network architecture all contribute to a network environment with many cyber-vulnerabilities unique to OT networks while retaining all of the vulnerabilities found in standard IT networks. With all these unique issues it is interesting to note that even though SDN was developed for IT’s large datacenters and geographically dispersed wide-area LANs, SDN may provide more benefits when deployed in an OT network environment. Furthermore, several innate properties of SDN networks were found to have beneficial effects on the cyber-resilience of OT networks. The cybersecurity benefits of SDN include the detailed documentation and monitoring, centralized control, whitelisted network traffic philosophy, reduced switching hardware cost, and improved overall network performance. While the detriments revealed include enlarged and centralized attack surface, possible nonfunctional network states during reconfiguration, and possible instability of the network due to automated programmatic changes to the network configuration. Finally, the research focused on how SDN deployments impact OT network support personnel. Possible negative consequences were found to be the general massive shift from one network management style to the completely new SDN style, addition of cybersecurity
78
responsibilities, and the training associated with these changes. The positive effects are increased situational awareness, reduction in amount of work required to deploy new network resources or reconfigure existing resources, centralized monitoring that helps with both troubleshooting of configuration and hardware issues, and the intrinsic application of several cybersecurity principles.
79
Recommendations This capstone project validated the existence of many significant cybersecurity issues in OT environments. This research enumerates many reasons for the existence and persistence of these issues. The research reveals several paths to solutions for these problems. The IT and OT Teams Must Work Together It is undeniable that the different worlds of IT and OT are converging and, in one way or another, these two cultures must learn to work together to implement a wholistic security strategy. As a first step, management should look for ways to cross-train IT and OT personnel and align the management structures, so the teams have joint responsibilities at the lowest possible level in the organization. Full assimilation of OT into IT may not be the best solution for every company; however, strict separation of duties using the firewall separating the IT and OT networks as the line of demarcation has proven highly problematic. The end goal should be joint understanding and respect for the reasons IT and OT teams approach the implementation of cybersecurity differently. More research is needed to determine whether cross training IT and OT teams or combining them completely facilitates better cybersecurity outcomes. Develop a Comprehensive Cybersecurity Program A comprehensive cybersecurity program is a vital tool for improving understanding and cooperation between IT and OT teams. To drive convergence of IT and OT to a common purpose, it is imperative the program encompasses all IT and OT personnel and assets. This program must be sponsored at the C-Level with active executive leadership and board-level oversight, with the day-to-day responsibilities flowing down through every level of the organization to embrace the front-line personnel and support associates. Planning and efforts toward improving cybersecurity should be the result of continuous and thorough evaluations of
80
an organization’s current cybersecurity capability maturity level and the setting of goals based on agreed metrics. This will require choosing good metrics to determine whether outcomes are demonstrably improved. More research is required into the appropriate metrics for determining how to measure success in increasing cybersecurity readiness and other cyber related outcomes. Implement an SDN Test Lab with an Eye Toward a Production Trial Because SDN field-testing in OT networks is just beginning and SDN’s published OTbased research has been confined to small power grid use cases, many more experiments and studies are required to determine SDN’s overall cybersecurity efficacy. It is likely that companies will want to test SDN technologies in OT networks representative of their own industry. This requires setting up individual SDN labs or leveraging industry consortiums to perform the specific testing. This is another area where it will be important to develop businessoutcome-focused metrics, to gauge probable improvements accurately and to assess overall efficacy. Important questions remain as to whether SDN networks should be corporately wholistic or if there is added value in maintaining a logical separation between business-focused IT functions and production-based OT functions. Either way, the testing of SDN technologies by both the IT and OT responsible teams is an opportunity for cross training and learning of underlying requirements and goals of the OT environment. Organizations should begin positioning themselves now for the time when SDN evaluation, testing, and standardization is mature enough for deployment in production environments. Support Additional OT-Focused SDN Research There are still relatively few OT-focused SDN options with most SDN deployments in the SD-WAN space of large commercial communication carrier companies or large datacenters. The good news is that the current OT-based SDN trials like Chess Master are taking place in the
81
electrical grid environment and face some of the most stringent requirements and the closest regulatory scrutiny of any OT application. Electrical grid monitoring and control messaging present many technically challenging requirements, including fast network healing speed and minimal reconfiguration downtime. These challenging technical network requirements are presented in an equally challenging outdoor, long-distance physical environment all while having to simultaneously support high reliability and safety criteria. It is also fitting to address the electrical grid’s needs in initial testing since many of the current concerns for OT cybersecurity are in the electrical grid arena. Additional OT-focused research into SDN usage in all types of OT networks is required. Concurrently, significant effort should be directed toward penetration testing against SDN-style networks. Red team / blue team war game exercises should be sponsored and closely evaluated. Red team attacks with full access to the network design and operational details will be most effective at finding weak links and holes in the security of SDN-based OT networks. This is another area where cooperation between IT and OT teams is vital, with most penetration testing experience residing in IT teams. Encourage Standards and Regulatory Bodies to Investigate SDN for Cybersecurity Organizations such as NIST, NERC, ICS-CERT, ISA, IEC, and ISO are already active in the OT network cybersecurity space. With NIST 800-82 and ISA/IEC-62443 making specific network security recommendations for OT networks, these bodies should be deeply involved with testing OT-focused SDN software and hardware. They should also play an important role in advising the SDN standards organizations like ONF on how to extend the current standards to encompass the needs of OT networks. The ICS experts in ICS-CERT’s Red Team penetration
82
testing group would be invaluable for testing SDN-based OT networks and should be engaged by hardware and software vendors early in the development process. Summary of Recommendations To summarize, individual corporations and organization should prioritize the development of a comprehensive cybersecurity program. The program must have senior executive sponsorship and work toward establishing deep IT and OT cooperation based on real understanding of each other’s priorities and goals. More research and real-world testing is required to determine objective cybersecurity efficacy in OT network environments. Standards and regulatory bodies should step up efforts to characterize where SDN fits in future versions of cybersecurity-related standards and regulations. The ICS community should hasten investments in continued hardware and software development, testing, standards development, and regulatory direction.
83
Conclusion Operational technology (OT) networks are the blood vessels carrying the lifeblood of data on which our modern manufacturing plants, processing facilities, and utility systems depend. Without the dependable, secure, and consistent movement of data within these networks, our current society would quickly begin to decay. These OT networks are increasingly under attack from many directions and in many cases are poorly defended. This capstone project reinforced the perceived existence of many significant cybersecurity issues in OT environments. However, the research shows SDN-based networks to be a potential solution with many benefits and few potential detriments. Additionally, the effects of SDN implementation on personnel responsible for OT networks were found to be mostly positive. IT and OT Networks are Significantly Different The pursuit of answers to research question Q1 identified several limitations and special considerations of OT networks, which the IT community does not appreciate. Personnel administering OT networks base their network performance goals on safety first and reliable availability next followed by data integrity and confidentiality. This means they design a network with exactly the opposite approach to priorities taken by IT personnel who value data confidentiality above integrity and availability and do not normally need to consider human or machine safety. Their opposing mindsets often cause mutual misunderstandings, which lead to mistrust and little or no cooperation. No matter how it comes to fruition, the two cultures of IT and OT must merge into a single culture of security; they must understand and respect each other’s basic goals while coming together to reinforce the overall cybersecurity of the organization.
84
SDN Effects on OT Network Cybersecurity Appear Largely Positive The second research question called for investigating the potential of SDN technologies and methodologies to mitigate cybersecurity threats within OT network environments. Networks based on SDN technologies were found to have mostly beneficial effects on cyber-resilience with the potentially negative effects having practical mitigation schemes. The potential cybersecurity benefits of SDN map well to OT networks including detailed self-documentation, enhanced situational awareness, centralized control, high-level conceptual configuration, whitelisted data definitions, reduced hardware costs, and improved overall network performance. The potential detriments were found to include an enlarged attack surface concentrated in the central server, potential unknown states during reconfiguration, and potential lack of stability of the network if programmatic changes to the network configuration are allowed. These potential negative aspects were found to have reasonable mitigation tactics. The added protections of SDN’s whitelisted data access controls, OpenFlow’s encrypted and authenticated control messaging, and the addition of high-value asset security overlays combine to secure the central server and its communication channels. Time-synchronized configuration changes can significantly reduce the possibility of the network being in an undefined state and greatly reduce the total time in such a state. Finally, restrictions on allowing programmatic changes to an active OT network can remove the questions of reliability of programmatic configuration changes. There is not enough published research available or testing in enough different OT regimes to call conclusively for immediate incorporation of SDN technologies in all OT networks. However, the general experience with SDN technology in other network environments and the limited testing in specific OT networks is extremely positive. The ICS community is just beginning to investigate the implementation of SDN in OT networks.
85
SDN Effects on OT Support Personnel are Generally Positive Finally, the research question Q3 focused on how SDN deployments impact the personnel implementing and maintaining OT networks. Several characteristics of SDN-style networks were found to be beneficial to the operating personnel. These beneficial aspects consist of enhanced situational awareness, reduced effort for network resource deployment, centralized configuration and monitoring that simplifies troubleshooting, and the inherent application of the strong cybersecurity principle of whitelisting. Possible detrimental characteristics include the need to learn a new network management system, the subtle addition of cybersecurity responsibilities, and the additional training time required. The benefits of SDN, notwithstanding the potential detriments, add up to an overall reduction in OT network personnel workload with the added benefit of enhanced cybersecurity, which is likely to reduce crisis management events in the future. Final Thoughts Differences in IT and OT requirements and priorities will continue for the foreseeable future, dictated by the different environments in which IT and OT networks exist. The important goal is to develop a mutual culture around a comprehensive cybersecurity program that embraces the differences and employs the best technologies and methodologies for each specific use case. Much as the internal safety programs matured in the 1980s and 90s, it is essential, that executive management provides consistent leadership and funding to create and maintain a constant cybersecurity effort, which flows through every level of the organization and is managed with meaningful metrics. The SDN technologies researched in this capstone project fit well with these general conclusions. The applicability of the cybersecurity and network management aspects of SDN in
86
both IT and OT environments provide a potential bridge to help meld the two cultures. Moreover, the increased breadth and granularity of SDN’s integrated data flow metrics provide enhanced situational awareness for operations and management alike. Additionally, centralized management promises quicker time to deployment while reducing the overall workload on already stretched personnel. Furthermore, the inherent, detailed, data-flow whitelisting of SDN provides superior cybersecurity on the network level, which severely limits the ability of malware to infiltrate a network; if a malevolent beachhead is established, the same whitelisting makes peer-to-peer transmission difficult while simultaneously detecting undefined data transmission attempts. All these characteristics validate SDN as a worthy current research topic and a potentially potent cybersecurity tool for OT networks in the near future.
87
References 0patch. (2017). Security patching is hard: Survey results 2017. Retrieved from https://0patch.com/files/SecurityPatchingIsHard_2017.pdf 7 types of hacker motivations. (2011, March 16). Retrieved August 10, 2018, from https://securingtomorrow.mcafee.com/consumer/family-safety/7-types-of-hackermotivations/ Ademolake, W. (2017, October 11). Building management systems / environmental monitoring systems. Retrieved July 30, 2018, from https://www.linkedin.com/pulse/buildingmanagement-systems-environmental-monitoring-wale-ademolake/ Ahmad, I., Namal, S., Ylianttila, M., & Gurtov, A. (2015). Security in software defined networks: A survey. IEEE Communications Surveys & Tutorials, 17(4), 2317–2346. https://doi.org/10.1109/COMST.2015.2474118 Alexander, N. (2016, July 15). Did the Israeli-American Stuxnet virus launch a cyber world war? Retrieved June 18, 2018, from https://www.haaretz.com/israelnews/.premium.MAGAZINE-did-stuxnet-launch-a-cyber-world-war-1.5410099 Anwer, M. B., Motiwala, M., Tariq, M. bin, & Feamster, N. (2010). SwitchBlade: A platform for rapid deployment of network protocols on programmable hardware. SIGCOMM 2010, 12. Assante, M. J. (2017). Digital ghost: Turning the tables. SANS Institute Reading Room. Aydell, G. (2015). The perfect ICS storm. SANS Institute Reading Room. BAE Systems Applied Intelligence. (2014). The convergence of information technology and operational technology: A new industrial revolution. BAE Systems plc. Bakhshi, T. (2017). State of the art and recent research advances in software defined networking. Wireless Communications and Mobile Computing, 2017. https://doi.org/10.1155/2017/7191647 Banks, E. (2014a). SDN essentials: Why network orchestration and virtualization? Retrieved July 24, 2018, from https://searchsdn.techtarget.com/tip/SDN-essentials-Why-networkorchestration-and-virtualization Banks, E. (2014b, September 9). Incremental SDN: Automating network device configuration. Retrieved July 29, 2018, from https://www.networkworld.com/article/2603534/softwaredefined-networking/incremental-sdn-automating-network-device-configuration.html Bayda, M. (2018, May 8). Control, monitor, process: The role of industrial Ethernet in manufacturing. Retrieved June 28, 2018, from 88
https://www.automationworld.com/article/topics/industrial-internet-things/controlmonitor-process-role-industrial-ethernet Bersin, J. (2018, July 3). The ugly side to today’s low unemployment rate. Retrieved July 27, 2018, from https://www.forbes.com/sites/joshbersin/2018/07/03/the-ugly-side-to-todayslow-unemployment-rate/ Bindseil, J. L. (2003, July 1). C-level sponsors: Factoring in the business bottom line. SC Media. Retrieved from https://www.scmagazine.com/c-level-sponsors-factoring-in-the-businessbottom-line/article/549061/ Bing, C. (2018, January 16). Trisis has the security world spooked, stumped and searching for answers. Retrieved January 24, 2018, from https://www.cyberscoop.com/trisis-icsmalware-saudi-arabia/ Bisale, C., & Kohl, A. (2015, August). Consistent cyber security in energy automation for critical infrastructures. Siemens AG, Nuremberg. Retrieved from https://www.downloads.siemens.com/download-center/download?DLA20_38 Biswas, A., & Karunakaran, S. (2015). Cybernetic modeling of industrial control systems: Towards threat analysis of critical infrastructure. Eprint ArXiv:1510.01861. BitSight Insights. (2017). A growing risk ignored - Critical updates. BitSight Insights. Retrieved from https://cdn2.hubspot.net/hubfs/277648/Insights/BitSight%20Insights%20%20A%20Growing%20Risk%20Ignored%20%20Critical%20Updates.pdf?t=1532375725345&utm_campaign=Q217%20BitSight%20 Insights&utm_source=hs_automation&utm_medium=email&utm_content=52515743&_ hsenc=p2ANqtz-Y7iLCFRBDF4e0kNilKdKJtSa21Q3mUCyyIqsMHsNgGI4iKvAu9A0KMw0oMMNAYDBDL4BYC3xTNMw--wGrbb3kRp3JA&_hsmi=52515743 Bobba, R., Borries, D. R., Hilburn, R., Sanders, J., Hadley, M., & Smith, R. (2014). Softwaredefined networking addresses control system requirements. Bodungen, C. E., Singer, B. L., Shbeeb, A., Hilt, S., & Wilhot, K. (2017). Hacking exposed industrial control systems: ICS and SCADA security secrets and solutions. McGraw Hill Education. Bose, A. J. C., & Sinha, S. (2012). Human side of lean production: Aren’t we on a slippery slope? International Journal of Lean Thinking, 3(2), 102–116. Brocklehurst, K. (2017, May 3). IT-OT convergence and conflict: Who owns ICS security? Retrieved July 1, 2018, from https://www.belden.com/blog/industrial-security/it-otconvergence-and-conflict-who-owns-ics-security
89
Bruijn, H. de, & Janssen, M. (2017). Building cybersecurity awareness: The need for evidencebased framing strategies. Government Information Quarterly. https://doi.org/10.1016/j.giq.2017.02.007 Burke, G., & Fahey, J. (2015, December 22). Iranian hackers breached US power grid to engineer blackouts. Retrieved June 29, 2018, from https://www.timesofisrael.com/iranian-hackers-breached-us-power-grid-to-engineerblackouts/ Cardenas, A. A., Amin, S., Sinopoli, B., Giani, A., Perrig, A., & Sastry, S. (2009). Challenges for securing cyber physical systems. Carney, J. (2011). Why integrate physical and logical security? (p. 8). Cisco Systems, Inc. Retrieved from https://www.cisco.com/c/dam/en_us/solutions/industries/docs/gov/plsecurity.pdf CEDS. (2017, May). SEL Chess Master fact sheet. Cybersecurity for Energy Delivery Systems (CEDS). Retrieved from https://www.energy.gov/sites/prod/files/2017/05/f34/SEL_ChessMaster_FactSheet.pdf Chalfant, M. (2018a, March 18). Five things to know about Russian attacks on the energy grid. Retrieved June 16, 2018, from http://thehill.com/policy/cybersecurity/378869-fivethings-to-know-about-russian-attacks-on-the-energy-grid Chalfant, M. (2018b, May 31). Deadly attacks feared as hackers target industrial sites. Retrieved June 16, 2018, from http://thehill.com/policy/cybersecurity/389983-deadly-attacksfeared-as-hackers-target-industrial-sites Chesla, A. (2013, January 23). Software defined networking - A new network weakness? Retrieved December 11, 2017, from http://www.securityweek.com/software-definednetworking-new-network-weakness CIM Reference Model Committee. (1989). A reference model for computer integrated manufacturing (CIM). Purdue Research Foundation. Retrieved from http://www.pera.net/Pera/PurdueReferenceModel/ReferenceModel.pdf Cisco Systems, Inc. (2008). 1.2 Design and implementation guide. In Ethernet-to-the-Factory. Cisco Systems, Inc. Retrieved from https://www.cisco.com/c/en/us/td/docs/solutions/Verticals/EttF/EttFDIG/ch1_EttF.html Cisco Systems, Inc. (2017). IT/OT convergence: Moving digital manufacturing forward. CNCI. (2009). A roadmap for cybersecurity research (p. 126). U.S. DHS Comprehensive National Cybersecurity Initiative (CNCI). Retrieved from https://www.dhs.gov/sites/default/files/publications/CSD-DHS-CybersecurityRoadmap.pdf 90
Coffey, K., Smith, R., Maglaras, L., & Janicke, H. (2018). Vulnerability analysis of network scanning on SCADA systems. Security and Communication Networks, Volume 2018(Article ID 3794603), 21. Collier, Z. A., Panwar, M., Ganin, A. A., Kott, A., & Linkov, I. (2016). Security metrics in industrial control systems. In E. J. M. Colbert & A. Kott (Eds.), Cyber-security of SCADA and Other Industrial Control Systems (Vol. 66, pp. 167–185). Cham: Springer International Publishing. https://doi.org/10.1007/978-3-319-32125-7_9 Cranford, N. (2017, June 22). Three examples of SDN deployments. Retrieved July 30, 2018, from https://www.rcrwireless.com/20170621/wireless/three-examples-of-sdndeployments-tag99-tag27 Crawford, J. (2014, November 21). Govt: China could take down U.S. power grid. Retrieved June 29, 2018, from https://www.cnn.com/2014/11/20/politics/nsa-china-powergrid/index.html Crowd Research Partners & ISC on LinkedIn. (2017). Cybersecurity trends - 2017 spotlight report. Retrieved May 27, 2018, from https://www.herjavecgroup.com/wpcontent/uploads/2017/06/Cybersecurity-trends-2017-survey-report.pdf CSSP. (2009). Primer control systems cyber security framework and technical metrics. Control Systems Security Program. Retrieved from https://ics-cert.uscert.gov/sites/default/files/documents/Metrics_Primer_7-13-09_FINAL.pdf CSSP. (2011). Common cybersecurity vulnerabilities in industrial control systems. Control Systems Security Program. Retrieved from https://ics-cert.uscert.gov/sites/default/files/recommended_practices/DHS_Common_Cybersecurity_Vulne rabilities_ICS_2010.pdf DarkTrace Industrial. (2017). Cyber security for industrial control systems: A new approach. Darktrace, Ltd. Retrieved from https://www.aquion.com.au/wpcontent/uploads/2017/11/wp-ics.pdf Davidson, J., & Wright, J. (2008). Configuring and managing remote access for industrial control systems (p. 66). U.S. Department of Homeland Security. Retrieved from https://ics-cert.uscert.gov/sites/default/files/recommended_practices/RP_SecuringModems_S508C.pdf DHS Cyber Security Division. (2016). Transition to practice - Technology guide (p. 68). Department of Homeland Security - Cyber Security Division. Dixit, A., Hao, F., Mukherjee, S., Lakshman, T. V., & Kompella, R. (2013). Towards an elastic distributed SDN controller, 6.
91
DOE. (2014). Cybersecurity capability maturity model (C2M2) version 1.1 (p. 76). U.S. Department of Energy. Retrieved from https://www.energy.gov/sites/prod/files/2014/03/f13/C2M2-v1-1_cor.pdf Dolezilek, D. J. (2018). Using Software-defined network technology to precisely and reliably transport process bus Ethernet messages (p. 7). Presented at the 14th International Conference on Developments in Power System Protection, Belfast, United Kingdom. Dong, X., Lin, H., Tan, R., Iyer, R. K., & Kalbarczyk, Z. (2015). Software-defined networking for smart grid resilience: Opportunities and challenges. In Proceedings of the 1st ACM Workshop on Cyber-Physical System Security (pp. 61–68). Singapore: ACM Press. https://doi.org/10.1145/2732198.2732203 Doyle, L. (2018, March 11). What’s the status of SDN deployments in the enterprise? Retrieved March 11, 2018, from http://searchsdn.techtarget.com/answer/Whats-the-status-of-SDNdeployments-in-the-enterprise Du, J. L., & Herlich, M. (2016). Software-defined networking for real-time Ethernet (pp. 584– 589). Presented at the 13th International Conference on Informatics in Control, Automation and Robotics (ICINCO), SCITEPRESS - Science and and Technology Publications. https://doi.org/10.5220/0005996605840589 Duffy, J. (2011, April 14). FAQ: What is OpenFlow and why is it needed? Retrieved July 4, 2018, from https://www.networkworld.com/article/2202144/data-center/faq--what-isopenflow-and-why-is-it-needed-.html DukeNet Communications. (2013, December 19). DukeNet presents live demonstration of SDN orchestration across the wan and data center [Financial News]. Retrieved June 24, 2018, from https://www.marketwatch.com/story/dukenet-presents-live-demonstration-of-sdnorchestration-across-the-wan-and-data-center-2013-12-19 Eckerson, W. W. (2011). How to create effective performance metrics (2nd ed.). John Wiley & Sons, Inc. Energy Sector Control Systems Working Group [ESCSWG]. (2011). Roadmap to achieve energy delivery systems cybersecurity (p. 80). Energy Sector Control Systems Working Group. Ericsson, G. N. (2010). Cyber security and power system communication—Essential parts of a smart grid infrastructure. IEEE Transactions on Power Delivery, 25(3), 1501–1507. https://doi.org/10.1109/TPWRD.2010.2046654 Erman, M., & Finkle, J. (2017, July 28). Merck says cyber attack halted production, will hurt profits [News]. Retrieved June 3, 2018, from https://www.reuters.com/article/us-merckco-results/merck-says-cyber-attack-halted-production-will-hurt-profitsidUSKBN1AD1AO
92
Ernst & Young Global Limited. (2014a). The DNA of the CIO: Opening the door to the C-suite (p. 44). Ernst & Young Global Limited. Ernst & Young Global Limited. (2014b). The DNA of the COO: Time to claim the spotlight (p. 44). Ernst & Young Global Limited. Falco, J., Hurd, S., & Teumim, D. (2006). Using host-based antivirus software on industrial control systems: integration guidance and a test methodology for assessing performance impacts. version 1.0 (No. NIST SP 1058). Gaithersburg, MD: National Institute of Standards and Technology. https://doi.org/10.6028/NIST.SP.1058 Farwell, J. P., & Rohozinski, R. (2011). Stuxnet and the future of cyber war. Survival, 53(1), 23– 40. https://doi.org/10.1080/00396338.2011.555586 Feamster, N., Rexford, J., & Zegura, E. (2014). The road to SDN - An intellectual history of programmable networks. ACM SIGCOMM Computer Communication Review, Volume 44(Issue 2), 87–98. Fraile, F., Flores, J. L., Poler, R., & Saiz, E. (2018). Software defined networking to improve cybersecurity in manufacturing orientated interoperability ecosystems. Presented at the 9th INTERNATIONAL CONFERENCE on INTEROPERABILITY FOR ENTERPRISE SYSTEMS AND APPLICATIONS, Berlin, Germany. Frost & Sullivan. (2017). 2017 global information security workforce study. Fruehe, J. (2016). Software-defined network orchestration arrives (p. 9). Moor Insights & Strategy. Retrieved from http://www.moorinsightsstrategy.com/wpcontent/uploads/2016/07/Software-Defined-Network-Orchestration-Arrives-by-MoorInsights-and-Strategy.pdf Fujitsu Network Communications Inc. (2014). Software-defined networking for the utilities and energy sector. Whitepaper. Retrieved from https://www.fujitsu.com/us/Images/SDN-forUtilities.pdf Fulton, S., III. (2018, May 22). What is SDN? How software-defined networking changed everything. Retrieved July 20, 2018, from https://www.zdnet.com/article/softwaredefined-networking-101-what-sdn-is-and-where-its-going/ Futrell, P. (2016, June 13). Risks vs. rewards of ICS remote access. Retrieved July 2, 2018, from http://www.remotemagazine.com/main/articles/risks-vs-rewards-of-ics-remote-access/ Gao, J., Xiao, Y., Liu, J., Liang, W., & Chen, C. L. P. (2012). A survey of communication/networking in smart grids. Future Generation Computer Systems, 28(2), 391–404. https://doi.org/10.1016/j.future.2011.04.014
93
Glenn, C., Sterbentz, D., & Wright, A. (2016). Cyber Threat and Vulnerability Analysis of the U.S. Electric Sector (No. INL/EXT--16-40692, 1337873). https://doi.org/10.2172/1337873 Gold, J. (2017, August 2). The 10 most powerful companies in enterprise networking. Retrieved March 11, 2018, from https://www.networkworld.com/article/3211410/lan-wan/the-10most-powerful-companies-in-enterprise-networking.html Goransson, P., Black, C., & Culver, T. (2017). Software Defined Networks: A Comprehensive Approach (2nd ed.). Morgan Kaufmann. Gregory-Brown, B. (2017). Securing industrial control systems-2017. SANS Institute Reading Room, 34. Gregory-Brown, B., & Harp, D. (2016). Security in a converging IT/OT world. Whitepaper. Retrieved from https://www.sans.org/reading-room/whitepapers/analyst/securityconverging-it-ot-world-37382 Hadley, M., Nicol, D., & Smith, R. (2017). Software-defined networking redefines performance for Ethernet control systems (p. 10). Presented at the Power and Energy Automation Conference. Harcharan, D., Houmb, S. H., & Engum, E. A. (2018). How to safeguard sophisticated operational technology for targeted, highly dangerous cyber threats. Hardy, B. (2017, September 25). Advantages of software defined networking (SDN). Retrieved July 1, 2018, from https://www.fidelus.com/software-defined-networking-advantages/ Harp, D. R., & Gregory-Brown, B. (2015). IT/OT convergence - Bridging the divide. nexdefense. Retrieved from https://ics.sans.org/media/IT-OT-Convergence-NexDefenseWhitepaper.pdf He, J., Liu, L., Ding, F., Li, C., & Zhang, D. (2017). A new coordinated backup protection scheme for distribution network containing distributed generation. Protection and Control of Modern Power Systems, 2(1), 10. https://doi.org/10.1186/s41601-017-0043-3 Higgins, K. J. (2018, January 11). Vulnerable mobile apps: The next ICS/SCADA cyber threat. Retrieved June 16, 2018, from https://www.darkreading.com/endpoint/privacy/vulnerable-mobile-apps-the-next-icsscada-cyber-threat/d/d-id/1330801 Hill, R., & Smith, R. (2017). Purpose-engineered, active-defense cybersecurity for industrial control systems, 16.
94
Houlden, N. (2018, February 19). What are the fastest-growing cyber threats to your business? The Telegraph. Retrieved from https://www.telegraph.co.uk/business/cybersecurity/what-are-the-fastest-growing-cyber-threats-to-your-business/ Hughes, A., & Littlefield, M. (2018). Reinvent lean today with digital technology (Improving Continuous Improvement) (p. 29). LNS Research. Retrieved from http://www.lnsresearch.com/docs/default-source/manufacturing-operationsmanagement/2018_reinventleantoday_full-edition.pdf?sfvrsn=0 ICS-CERT. (2016). Recommended practice: Improving industrial control system cybersecurity with defense-in-depth strategies. National Cybersecurity and Communications Integration Center. Retrieved from https://ics-cert.uscert.gov/sites/default/files/recommended_practices/NCCIC_ICSCERT_Defense_in_Depth_2016_S508C.pdf ICS-CERT. (2017, June 13). Indicators associated with WannaCry ransomware (Update I). Retrieved June 3, 2018, from https://ics-cert.us-cert.gov/alerts/ICS-ALERT-17-135-01I ICS-CERT. (n.d.). Standards and references. Retrieved July 24, 2018, from https://ics-cert.uscert.gov/Standards-and-References INCIBE. (2015, June 11). Antivirus issues in industrial environments. Retrieved June 21, 2018, from https://www.certsi.es/en/blog/antivirus-issues-ics ISA. (2007). ANSI/ISA-99: Security for Industrial Automation and Control Systems: Concepts, Terminology and Models (ANSI/ISA-99.01.01-2007). Raleigh, NC, USA: International Society of Automation. ISA. (2010). ANSI/ISA-95: Enterprise-Control System Integration (ANSI/ISA-95.00.01-2000). Raleigh, NC, USA: International Institute of Automation. Jacinto, J. (2010, February 24). Cutting cost with COTS: Leveraging commercial hardware. Retrieved July 14, 2018, from https://www.totallyintegratedautomation.com/2010/02/cutting-cost-with-cots-leveragingcommercial-hardware/ Jammal, M., Singh, T., Shami, A., Asal, R., & Li, Y. (2014). Software defined networking: State of the art and research challenges. Computer Networks, 72, 74–98. https://doi.org/10.1016/j.comnet.2014.07.004 Jin, D., & Ning, Y. (2014). Securing industrial control systems with a simulation-based verification system. In SIGSIM-PADS ’14 (pp. 165–166). Denver, CO, USA. https://doi.org/10.1145/2601381.2601411 Kang, N., Liu, Z., Rexford, J., & Walker, D. (2013). Optimizing the “one big switch” abstraction in software-defined networks. In Proceedings of the ninth ACM conference on Emerging 95
networking experiments and technologies - CoNEXT ’13 (pp. 13–24). Santa Barbara, California, USA: ACM Press. https://doi.org/10.1145/2535372.2535373 Kaspersky Lab ICS CERT. (2017). Threat landscape for industrial automation systems in the second half of 2016. Retrieved June 22, 2018, from https://ics-cert.kaspersky.com/wpcontent/uploads/sites/6/2017/03/KL-ICS-CERT_H2-2016_report_FINAL_EN.pdf Kobara, K. (2016). Cyber physical security for industrial control systems and IoT. IEICE Transactions on Information and Systems, E99.D(4), 787–795. https://doi.org/10.1587/transinf.2015ICI0001 Kolaks, M. S. (2003). Securing out-of-band device management. SANS Institute, 21. Kovacs, E. (2015, September 17). Russian hackers target industrial control systems: US intel chief. Retrieved August 10, 2018, from https://www.securityweek.com/russian-hackerstarget-industrial-control-systems-us-intel-chief Kreutz, D., Yu, J., Esteves-Verissimo, P., Magalhaes, C., & Ramos, F. M. V. (2017). The KISS principle in software-defined networking: An architecture for keeping it simple and secure. ArXiv:1702.04294 [Cs]. Retrieved from http://arxiv.org/abs/1702.04294 Kure, H., Islam, S., & Razzaque, M. (2018). An integrated cyber security risk management approach for a cyber-physical system. Applied Sciences, 8(6), 898. https://doi.org/10.3390/app8060898 Langner, R. (2013). A technical analysis of what Stuxnet’s creators tried to achieve (p. 37). The Langner Group. Retrieved from https://www.langner.com/wpcontent/uploads/2017/03/to-kill-a-centrifuge.pdf Lawson, S. (2013, August 30). Will software-defined networking kill network engineers’ beloved CLI? Retrieved July 24, 2018, from https://www.computerworld.com/article/2484358/it-careers/will-software-definednetworking-kill-network-engineers--beloved-cli-.html Leukert, B. (2017, September 18). Digitalization, Industry 4.0, and the future of industrial production. Retrieved March 9, 2018, from http://www.digitalistmag.com/digitaleconomy/2017/09/28/digitalization-industry-4-0-future-of-industrial-production05386125 Little, R. G. (2013, June 26). Using SDN switches to flexibly scale data center networks. Retrieved June 24, 2018, from https://searchsdn.techtarget.com/news/2240186921/UsingSDN-switches-to-flexibly-scale-data-center-networks Littlefield, M. (2015). Capturing the business value of the industrial IoT (Smart Connected Operations). LNS Research. Retrieved from http://www.lnsresearch.com/docs/defaultsource/test/lns_smartconnectedoperations.pdf?sfvrsn=0 96
Lloyd, W. (2016, April 5). You think your network diagram’s right? Retrieved July 2, 2018, from https://www.redseal.net/think-network-diagrams-right/ Lobo, R. (2018, April 23). OT intent-based security policies. Retrieved July 7, 2018, from https://blogs.cisco.com/digital/ot-intent-based-security-policies Luiijf, E., & Paske, B. J. T. (2015). Cyber security of industrial control systems. Presented at the Global Conference on Cyber Space 2015, The Netherlands: TNO. https://doi.org/10.13140/rg.2.1.3797.4566 Maddison, J. (2018, June 21). Resolving the challenges of IT-OT convergence. Retrieved June 22, 2018, from https://www.csoonline.com/article/3283238/security/resolving-thechallenges-of-it-ot-convergence.html Mahankali, S., & Rungta, S. (2014). Adopting software-defined networking in the enterprise, 8. Marr, B. (2016, June 14). Data-driven decision making: 10 simple steps for any business. Retrieved July 31, 2018, from https://www.forbes.com/sites/bernardmarr/2016/06/14/data-driven-decision-making-10simple-steps-for-any-business/ Marr, B. (2017). Why every business needs a data and analytics strategy. Retrieved February 25, 2018, from https://www.bernardmarr.com/default.asp?contentID=768 Matthewman, S., & Byrd, H. (2014). Blackouts: a sociology of electrical power failure, 26. McKinsey & Company. (n.d.). About this practice: Organization. Retrieved June 28, 2018, from https://www.mckinsey.com/business-functions/organization/how-we-help-clients/aboutthis-practice McKinsey Digital. (2015). Industry 4.0: How to navigate digitization of the manufacturing sector. McKinsey & Company. Retrieved from https://www.mckinsey.de/files/mck_industry_40_report.pdf Mintchell, G. (2012, January 31). Siemens acquires RuggedCom to expand ethernet presence in manufacturing. Retrieved March 11, 2018, from https://www.automationworld.com/article/topics/industry-news/siemens-acquiresruggedcom-expand-ethernet-presence-manufacturing MIT Technology Review Custom. (2016). Pioneers and giants: Insights on the future of networking (Future: Net) (p. 7). MIT Technology Review. Retrieved from https://s3.amazonaws.com/files.technologyreview.com/whitepapers/VMware_futurenet_Tech_Insights.pdf MIT Technology Review Insights. (2017, August 30). Network virtualization: The bridge to digital transformation. MIT Technology Review. Retrieved from 97
https://www.technologyreview.com/s/608773/network-virtualization-the-bridge-todigital-transformation/ Mitchell, S. (2015, December 16). What is OpenFlow and why should you care? Retrieved July 4, 2018, from https://itbrief.co.nz/story/what-openflow-and-why-should-you-care/ Morgan, S. (2017, May 30). Cybersecurity jobs report 2018-2021. Retrieved June 22, 2018, from https://cybersecurityventures.com/jobs/ Murarka, M. (2015, December). The keys to organizational agility. Retrieved June 3, 2018, from https://www.mckinsey.com/business-functions/organization/our-insights/the-keys-toorganizational-agility Myer, P. (2018, February 7). Three Ways SDN Helps to Solve Industrial IoT’s Undiscovered Security Problems. Retrieved July 7, 2018, from http://iiotworld.com/uncategorized/three-ways-sdn-helps-to-solve-industrial-iots-undiscoveredsecurity-problems/ Nakashima, E., & Warrick, J. (2012, June 2). Stuxnet was work of U.S. and Israeli experts, officials say [Newspaper]. Retrieved June 18, 2018, from https://www.washingtonpost.com/world/national-security/stuxnet-was-work-of-us-andisraeli-experts-officialssay/2012/06/01/gJQAlnEy6U_story.html?utm_term=.bd897bd6bc80 National Cybersecurity and Communications Integration Center (NCCIC). (2015). Seven steps to effectively defend industrial control systems. Retrieved from https://ics-cert.uscert.gov/sites/default/files/documents/Seven%20Steps%20to%20Effectively%20Defend %20Industrial%20Control%20Systems_S508C.pdf NIST. (2015). NIST SP 800-53 Revision 4: Security and privacy controls for information systems and organizations (p. 494). Gaithersburg, MD, USA: National Institute of Standards and Technology. Retrieved from https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-53r4.pdf NIST. (2017). Supporting the growth and sustainment of the nation’s cybersecurity workforce: Building the foundation for a more secure American future. National Institute of Standards and Technology. Retrieved from https://www.nist.gov/sites/default/files/documents/2018/07/24/eo_wf_report_to_potus.pd f NIST. (2018). Framework for improving critical infrastructure cybersecurity, Version 1.1 (No. NIST Cybersecurity White Paper) (p. 55). Gaithersburg, MD: National Institute of Standards and Technology. https://doi.org/10.6028/NIST.CSWP.02122014
98
Obama & United States. (2013, February 12). PPD-21 critical infrastructure and resilience. Retrieved June 22, 2018, from https://www.dhs.gov/sites/default/files/publications/PPD21-Critical-Infrastructure-and-Resilience-508.pdf O’Brien, L., & Avery, A. (2016, October 17). Positioned for recovery: Top 50 automation companies of 2015. Retrieved March 11, 2018, from https://www.controlglobal.com/articles/2016/positioned-for-recovery-top-50-automationcompanies-of-2015/ Open Networking Foundation. (2015, January). Principles and practices for securing softwaredefined networks. Open Networking Foundation. Retrieved from https://www.opennetworking.org/images/stories/downloads/sdn-resources/technicalreports/Principles_and_Practices_for_Securing_SoftwareDefined_Networks_applied_to_OFv1.3.4_V1.0.pdf Open Networking Foundation. (2017). Software-defined networking (SDN) definition. Retrieved December 11, 2017, from https://www.opennetworking.org/sdn-definition/ Owens, J. C. (2018, July 13). Cisco and other networking stocks plunge amid Amazon competition fears. Retrieved July 28, 2018, from https://www.marketwatch.com/story/cisco-and-other-networking-stocks-plunge-amidamazon-competition-fears-2018-07-13 Oxford Economics. (2011, October). Real-time Business: Playing to win in the new global marketplace. Retrieved March 9, 2018, from https://www.oxfordeconomics.com/Media/Default/economic-impact/marketappraisals/real-time-business.pdf Patil, P., Gokhale, A., & Hakiri, A. (2015). Bootstrapping Software Defined Network for flexible and dynamic control plane management (pp. 1–5). IEEE. https://doi.org/10.1109/NETSOFT.2015.7116132 Patrizio, A. (2018, July 17). Amazon rumored to be entering the networking market. Retrieved July 28, 2018, from https://www.networkworld.com/article/3290423/networkswitch/amazon-rumored-to-be-entering-the-networking-market.html Pauna, A., Moulinos, K., Lakka, M., May, J., & Tryfonas, T. (2013). Can we learn from SCADA security incidents?, 10. Peterson, D. G. (2011, August 2). PLC’s: Insecure by design v. vulnerabilities. Retrieved July 1, 2018, from http://www.digitalbond.com/blog/2011/08/02/plcs-insecure-by-design-vvulnerabilities/ Pettey, C. (2016, March 15). Six best practices for real-time analytics [Web Magazine]. Retrieved June 3, 2018, from https://www.gartner.com/smarterwithgartner/six-bestpractices-for-real-time-analytics/ 99
Pirc, J. (2009). Common network security misconceptions: Firewalls exposed. SANS Institute. Retrieved from https://www.sans.edu/cyber-research/security-laboratory/article/pircjohn-firewalls Raja, S. (2017, March 16). A better approach to securing IoT systems in ICS environments. Retrieved November 14, 2017, from http://www.lumeta.com/resources/blog/betterapproach-securing-iot-systems-ics-environments/ Red Tiger Security. (2011). Securing the move to IP-based SCADA/PLC networks. Center for the Protection of National Infrastructure (CPNI). Retrieved from https://www.ncsc.gov.uk/content/files/protected_files/document_files/The%20Move%20t o%20IP%20Based%20SCADA%20Networks%20151111.pdf Robert Walters PLC. (2018). Salaries for cyber security professionals set to rise in 2018. Retrieved June 22, 2018, from https://www.robertwalters.co.uk/career-advice/salariesfor-cyber-security-professionals-set-to-rise-in-2018.html Rockwell Automation. (2010). Cisco and Rockwell Automation partnership: Fact sheet. Retrieved from http://www.apac.rockwellautomation.com/ethernetip/ko/docs/RockwellAutomationandCi scoQandA.pdf Ross, R., Viscuso, P., Guissanie, G., Dempsey, K., & Riddle, M. (2018). NIST SP 800-171 Revision 1: Protecting Controlled Unclassified Information in Nonfederal Systems and Organizations (p. 125). Gaithersburg, MD, USA: National Institute of Standards and Technology. Retrieved from https://nvlpubs.nist.gov/nistpubs/specialpublications/nist.sp.800-171r1.pdf Santo, B. (2017, July 20). DOE receives first installment of anti-cyberwarfare networking technology. Retrieved July 2, 2018, from https://www.fiercetelecom.com/telecom/doereceives-first-installment-anti-cyberwarfare-networking-tech Schmidthaler, M., & Reichl, J. (2016). Assessing the socio-economic effects of power outages ad hoc. Computer Science - Research and Development, 31(3), 157–161. https://doi.org/10.1007/s00450-014-0281-9 Schwab, K. (2016, January 14). The Fourth Industrial Revolution: What it means and how to respond. Retrieved March 9, 2018, from https://www.weforum.org/agenda/2016/01/thefourth-industrial-revolution-what-it-means-and-how-to-respond/ Schwab, W., & Poujol, M. (2018). The state of industrial cybersecurity 2018. Sharma, N. (2015, January 20). Eight big benefits of software-defined networking. Retrieved July 2, 2018, from https://www.serverwatch.com/server-tutorials/eight-big-benefits-ofsoftware-defined-networking.html
100
Shockley, E. (2017, July 20). Veracity Industrial Networks delivers new SDN-based network infrastructure to the U.S. Department of Energy. Retrieved December 15, 2017, from http://www.marketwired.com/press-release/veracity-industrial-networks-delivers-newsdn-based-network-infrastructure-u-s-department-2226894.htm Sirkin, H. L., Zinser, M., & Rose, J. (2015, January 30). Why advanced manufacturing will boost productivity. Retrieved July 2, 2018, from https://www.bcg.com/publications/2015/leanand-manufacturing-production-why-advanced-manufacturing-boost-productivity.aspx Slattery, T. (2013, September 4). Will SDN be the future of network change management? Retrieved July 29, 2018, from https://www.nojitter.com/post/240160806/will-sdn-be-thefuture-of-network-change-management Smith, J., Kipp, N., Gammel, D., & Watkins, T. (2016). Defense-in-depth security for industrial control systems. Control Systems, 9. Spurgeon, C. E. (2000). Ethernet: The definitive guide. O’Reilly. Retrieved from https://www.safaribooksonline.com/library/view/ethernet-thedefinitive/1565926609/ch01.html Stouffer, K., Falco, J., & Scarfone, K. (2011, June). NIST Special Publication 800-82 - Guide to industrial control systems (ICS) security. NIST - U.S. Department of Commerce. Retrieved from http://csrc.nist.gov/publications/nistpubs/800-82/SP800-82-final.pdf Sulmeyer, M. (2017, May 12). What the rise of Russian hackers means for your business. Retrieved June 22, 2018, from https://hbr.org/2017/05/what-the-rise-of-russian-hackersmeans-for-your-business Swanson, I. (2016, May 30). Why a power grid attack is a nightmare scenario [Text]. Retrieved July 31, 2018, from http://thehill.com/policy/cybersecurity/281494-why-a-power-gridattack-is-a-nightmare-scenario Taylor, C. R. (2017). Software-defined networking: Improving security for enterprise and home networks (Degree of Doctor of Philosophy). WORCESTER POLYTECHNIC INSTITUTE. The Editors of Encyclopaedia Britannica. (2018). William L. Marcy. In Encyclopaedia Britannica (Internet). Retrieved from https://www.britannica.com/biography/William-LMarcy The Smart Grid Interoperability Panel–Smart Grid Cybersecurity Committee. (2014). NISTIR 7628 Revision 1 Guidelines for smart grid cybersecurity (No. NIST IR 7628r1). National Institute of Standards and Technology. https://doi.org/10.6028/NIST.IR.7628r1
101
Thilmany, J. (2017, October 10). Bridging the IT and OT Divide. Retrieved July 2, 2018, from https://www.automationworld.com/article/topics/industrial-internet-things/bridging-itand-ot-divide Toth, P. (2017). NIST MEP cybersecurity self-assessment handbook for assessing NIST SP 800171 security requirements in response to DFARS cybersecurity requirements (Handbook No. NIST HB 162) (p. 170). Gaithersburg, MD: National Institute of Standards and Technology. https://doi.org/10.6028/NIST.HB.162 Trend Micro. (2017, January 26). Why do attackers target industrial control systems? Retrieved July 30, 2018, from https://www.trendmicro.com/vinfo/us/security/news/cyberattacks/why-do-attackers-target-industrial-control-systems Tripwire. (2014). Understanding your attack surface - The first step in risk-based security intelligence. Whitepaper. Tsuchiya, A., Fraile, F., Koshijima, I., Ortiz, A., & Poler, R. (2018). Software defined networking firewall for industry 4.0 manufacturing systems. Journal of Industrial Engineering and Management, 11(2), 318. https://doi.org/10.3926/jiem.2534 US DHS. (2017). High value asset control overlay (p. 42). Department of Homeland Security Federal Network Resilience Division. Retrieved from https://www.dhs.gov/sites/default/files/publications/HVA%20Control%20Overlay%20v1 .0.pdf US-CERT. (2016). Alert (IR-ALERT-H-16-056-01) - Cyber-attack against Ukrainian critical infrastructure (Alert No. IR-ALERT-H-16-056-01). ICS-CERT. Retrieved from https://ics-cert.us-cert.gov/alerts/IR-ALERT-H-16-056-01 US-CERT. (2017). Alert (TA17-163A) - CrashOverride malware (Alert No. TA17- 163A). USCERT. Retrieved from https://www.us-cert.gov/ncas/alerts/TA17-163A US-CERT. (2018a, March 15). Alert (TA17-293A) - Advanced persistent threat activity targeting energy and other critical infrastructure sectors [U.S. Government Agency DHS]. Retrieved May 12, 2018, from https://www.us-cert.gov/ncas/alerts/TA17-293A US-CERT. (2018b, March 16). Alert (TA18-074A) - Russian government cyber activity targeting energy and other critical infrastructure sectors [U.S. Government Agency DHS]. Retrieved May 12, 2018, from https://www.us-cert.gov/ncas/alerts/TA18-074A Van Erp, G. (2018, June 26). Future cyber security regulations: Are you ready for what’s coming? Retrieved July 31, 2018, from https://applied-risk.com/blog/future-cybersecurity-regulations-are-you-ready-whats-coming
102
Vilalta, R., Mayoral, A., Casellas, R., Martínez, R., & Muñoz, R. (2017). SDN/NFV orchestration of multi-technology and multi-domain networks in cloud/fog architectures for 5G services, 3. Wallace, D. I., & Carter, R. W. (2010, June). Reduce compressor maintenance costs with remote monitoring. Retrieved July 2, 2018, from https://pgjonline.com/magazine/2010/june2010-vol-237-no-6/features/reduce-compressor-maintenance-costs-with-remotemonitoring Wang, T. (2016). Benefits and the security risk of software-defined networking. ISACA Journal, 4. Retrieved from https://www.isaca.org/Journal/archives/2016/volume-4/Pages/benefitsand-the-security-risk-of-software-defined-networking.aspx Wedgbury, A., & Jones, K. (2015). Automated asset discovery in industrial control systems Exploring the problem (pp. 73–83). https://doi.org/10.14236/ewic/ICS2015.8 Weiss, J. (2012). Ensuring the cybersecurity of plant industrial control systems, 11. Weiss, J. (2018, January). Cyber security of industrial control and safety systems. Retrieved from https://www.controlglobal.com/assets/wp_downloads/pdf/2018-01-INSIDER-JoeWeiss.pdf Welander, P. (2013, August 16). IT vs. OT: Bridging the divide [Online Magazine]. Retrieved July 23, 2018, from https://www.controleng.com/single-article/it-vs-ot-bridging-thedivide Wright, J. (2014, September 12). The most in-demand (and aging) engineering jobs. Retrieved July 27, 2018, from https://www.forbes.com/sites/emsi/2014/09/12/the-most-in-demandand-oldest-engineering-jobs/ Zetter, K. (2011, July 11). How digital detectives deciphered Stuxnet, the most menacing malware in history. Retrieved June 28, 2018, from https://www.wired.com/2011/07/howdigital-detectives-deciphered-stuxnet/ Zetter, K. (2014, November 3). An unprecedented look at Stuxnet, the world’s first digital weapon. Retrieved June 18, 2018, from https://www.wired.com/2014/11/countdown-tozero-day-stuxnet/ Zuckerman, M. (2017, August). Anatomy of an attack - Industrial control systems under siege. TrapX Security, inc.
103
