Testing Security Mechanisms with Real-life IT/OT Scenarios

www.otorio.com | [email protected]

2

Leveraging Caldera for

Industrial Cybersecurity

Testing Security Mechanisms with Real-life IT/OT Scenarios

Authors Matan Dobrushin Head of Research, OTORIO Idan Helzer Cyber Security Analyst, OTORIO


3

INTRO

v

Testing IT security systems is a mature, well-defined field. There are best

practices, procedures and a plethora of tips and information that can help

IT teams inexperienced in testing security systems to hit the ground

running. Testing industrial networks, AKA OT (operational technology), is

a new frontier with very little information and past data to learn from.

Moreover, the lessons learned from testing IT networks cannot be

transferred to the operations floor. IT systems are constantly updated,

while industrial networks are often running decades-old tools. IT systems

can be shut-down for testing and updates; the operations floor runs

24/7/365. Finally, the stakes in operations networks are much higher as

they control physical systems that require stable values such as

temperatures. In this whitepaper, OTORIO lays out the process for testing

operational networks, the available tools, the tool we chose to work with,

how it can be used to test real-life scenarios and how to assess

operational network testing results.


4

Table of Contents 1 SECURITY SYSTEMS TESTING CRITERIA ..................................................................................................... 6

1.1 HOW TO DEFINE TESTING CRITERIA FOR SECURITY SYSTEMS ............................................................................................. 6

1.2 OT LAB SETUP ........................................................................................................................................................ 7

1.3 ATTACK SCENARIO TOOLS ......................................................................................................................................... 9

2 OT CALDERA ........................................................................................................................................... 11

2.1 CALDERA TERMINOLOGY ......................................................................................................................................... 12

2.2 CALDERA IS BASED ON MITRE ATT&CK ................................................................................................................... 12

2.3 OT CALDERA - A DISRUPTIVE OT SECURITY PLATFORM ................................................................................................ 13

2.4 THREE OT CALDERA SCENARIOS ............................................................................................................................... 14

3 ASSESSING RESULTS .................................................................................................................................. 18

3.1 ASSESSING ATTACK SCENARIO RESULTS .................................................................................................................... 18

4 SUMMARY ............................................................................................................................................... 21

5 APPENDIX ................................................................................................................................................ 22


5

SECURITY SYSTEMS

TESTING CRITERIA

1


6

1 SECURITY SYSTEMS TESTING CRITERIA Before buying a security product, service, or system, there are success criteria and standards that companies set in order to ensure that newly acquired tools meet company standards. Post-purchase, security systems should be periodically tested to verify that they continue to meet success criteria. The proof of value (POV) success criteria for security systems differ on a per company and per product category level. Each industry has distinct inherent security strengths and weaknesses. Each security product category (antivirus, firewall, intrusion detection system) has different objectives and limitations. This document will offer best practices for testing security systems in operation technology (OT) settings and scenarios.

1.1 How to define testing criteria for security systems • Planning Phase - Gather information needed for assessment execution, such as the

assets to be assessed, the threats of interest against the assets, and the security controls to be used to mitigate those threats.

• Execution Phase - Primary goals for the execution phase are to identify vulnerabilities and validate them when appropriate. This phase should address activities associated with the intended assessment method and technique.

• Assessment Phase - The assessment phase focuses on analyzing identified vulnerabilities to determine root causes, establish mitigation recommendations, and develop a final report.

Testing security systems in OT is complex. For one, OT environments comprise a multitude of devices including endpoints, controllers, servers, and other network devices. Additionally, OT networks differ from one another. You therefore need a testbed that resembles your environment, which is not a trivial matter. While there are a multitude of known IT security testing tools and implementations, OT's multigenerational and software-hardware-machinery architecture have resulted in a lack of standard tools for testing cybersecurity in real-life environments.

___________________________________________________ 1: https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-115.pdf

1


7

1.2 OT Lab Setup Testing an OT network necessitates defining custom scenarios and testing them on the network. Running live attacks on an OT network is a non-starter because of the above-mentioned ramifications of ceasing operations. Therefore, an OT lab setup is the recommended way to simulate live attacks. There are many options - from low budget open source simulators to fully functional OT physical labs running real processes. The below image illustrates the spectrum of OT lab setups.

Image 1 - OT lab setup


8

OTORIO chose to design its testing lab based on the high end model, so it is flexible and capable of testing various type of OT environment. As shown in Image 2, perched on top of the OTORIO security system testing lab are the security servers that are currently being tested, with virtual endpoints and servers. All of the security servers logs (alerts) are forwarded to an ELK (Elasticsearch, Logstash, and Kibana) stack for later investigation.

Image 2 - The OTORIO security system testing lab


9

Multiple segments allow to simulate lateral movement within the network. On the bottom of the above image are physical labs with OT infrastructure which are monitored by two industrial firewalls and IDSs (intrusion detection systems). Attacks take place from any location, to any location on the network. After running an attack scenario, the first step is to investigate the ELK logs and learn how each system reacted to the scenario.

1.3 Attack Scenario Tools The next step is to dive into attack scenario tools. The tools are divided into two categories - blue team and red team. Both blue and red team tools can assist in testing security systems, networks, and products.

Table 1 - A comparison of popular attack scenario tools


10

2.4 Denial of Service

2 OT CALDERA


11

2 OT CALDERA

Caldera stands for Cyber Adversary Language and Decision Engine for Red Team Automation. Caldera is interface based, which is key for OT as not every security tool operator is a cyber security expert. Caldera can be run with little or no technical background. In industrial environments, many operational professionals "wear different hats" and perform operations, business, OT, and cybersecurity tasks during their workday. Caldera is an open source adversary emulation platform developed by MITRE. Caldera was originally designed to easily run autonomous security exercises for red or blue teams. We adapted it to automate testing of security systems. Table 2 below sums up the main benefits of utilizing Caldera:

Table 2 - Caldera benefits

___________________________________________________ 2: Red teams are penetration testers while blue teams defend against attacks. In simple terms, red teams are offense and blue teams are defense. Both red and blue teams hunt for vulnerabilities and assess network security. When attack scenarios are played out, both teams are required and a post-attack attempt assessment involves red and blue teams.


12

2.1 Caldera Terminology

• Agent - a program which connects to Caldera in order to get instructions and is deployed on endpoints

• Group - a collection of agents • Ability - a specific ATT&CK technique implementation • Adversary - a collection of abilities • Operation - an execution of an Adversary on a group

2.2 Caldera is Based on MITRE ATT&CK Caldera is built on the MITRE ATT&CK framework, a globally-accessible knowledge base of adversary tactics and techniques based on real-world observations. The ATT&CK knowledge base is used as a foundation for the development of specific threat models and methodologies in the private sector, in government, and in the cybersecurity product and service community. Leveraging MITRE ATT&CK in Caldera facilitates the process of choosing attack abilities and scenario creation. The user chooses the desired ability, which is a specific implementation of a technique that is a part of ATT&CK’s tactics. ATT&CK was designed with IT networks in mind. ATT&CK for Industrial Control Systems (ICS) is the OT version of MITRE ATT&CK. It introduces many new abilities, such as project file injection, ladder logic modification, and more. ATT&CK for ICS is a good starting point to assemble all known techniques, tactics, and procedures (TTPs). As previously stated, Caldera is an IT security system testing tool. ATT&CK is also IT-centric while ATT&CK for ICS is OT focused. Until recently, there was no Caldera for the industry. OTORIO has developed the world's first industrial version of Caldera, aptly named OT Caldera.

__________________________________________ 3: MITRE is an American non-profit organization which manages federally funded research & development centers, supporting several US government agencies. MITRE maintains the Common Vulnerabilities and Exposures (CVE) system and the Common Weakness Enumeration (CWE) project, which are in common use among cybersecurity professionals 4: Read more about MITRE ATT&CK here - https://collaborate.MITRE.org/attackics/index.php/Main_Page


13

2.3 OT Caldera - a Disruptive OT Security Platform

Image 3 - The capabilities in OT Caldera

OTORIO has developed 18 new abilities into OT Caldera. There is no other security testing platform that features this level of depth in implemented ICS abilities. OT Caldera's capabilities are divided into three logical categories: • Discovery and identification of OT devices using industrial protocols • OT impact and attacks (disrupting industrial processes) • Industrial data collection OTORIO uses these capabilities in order to create OT security testing scenarios - or "adversaries" as they are referred to in Caldera.


14

2.3 Three OT Caldera Scenarios OTORIO has created dozens of security system testing scenarios via OT Caldera. In this section, three are depicted. First is OT Discovery, a loud reconnaissance adversary. OT Discovery begins by performing an industrial port scan in order to discover which hosts are listening to industrial ports. Then it uses industrial protocols to run a deeper scan in order to identify the devices. The scenario finishes with a default credentials search on Scalance devices from Siemens.

Image 4 – OT Discovery


15

The second scenario is Mini Stuxnet, a gesture to the legendary Stuxnet. Mini Stuxnet was designed to run in a PCS 7- like environment. (The PCS 7 is a DCS from Siemens.) Its architecture contains an SMB share in which there are industrial project files. Mini Stuxnet starts by finding these shares and copying the project files from them. In addition, it finds S7 PLCs, and eventually tries to stop these PLCs.

Image 5 – Mini Stuxnet

_________________________________ 5: SMB (Server Message Block) is a network communication protocol for providing shared access to files, printers, and serial ports between nodes on a network. 6: PLCs are programmable logic controllers


16

The third adversary is ModbuStroyer, an Industroyer-like scenario. The original Industroyer used energy protocols, like IEC 61850. OTORIO used some of Industroyer TTPs, but implemented them using the Modbus protocol which is more common within the manufacturing sector.

Image 6 - Modbus Attack Scenario ________________________ 7: TTPs - Tactics, techniques and procedures


17

3 ASSESSING RESULTS


18

3 ASSESSING RESULTS

3.1 Assessing Attack Scenario Results The new scenarios, or adversaries, extend OT Caldera's capabilities as they introduce the first OT-generation attacks platform. The same scenarios can be used to test other types of OT networks, each with distinct testing challenges and goals. The next step after running attack scenarios with OT Caldera is to see how (or if) our OT security tools detected the attacks. When assessing OT attack scenarios, begin with viewing the logs from the product interfaces. Since the three attack scenarios involve many products, it is recommended to use any log aggregating system. ELK is one option (and is implemented in the OTORIO lab) to aggregate the logs from all of the security products.

Image 7 – ELK Testing


19

ELK allows operators to filter certain types of attacks, such as a port scan. If an attack is executed and no alert is issued, check your configurations and make necessary changes. The key is to receive the important data from each alert. Is your security product providing enough relevant information in order to maintain high security standards? When working in an OT environment, it is essential to find ICS-aware security products.


The above example is from a leading OT IDS vendor. This IDS knows how to parse OT protocols and alert when an Industrial command is issued in the network. Readers can view a recorded demo of the OT Caldera tool using this link: https://youtu.be/Ixifg9_85ZQ?t=797



20

4

Summary


21

4 SUMMARY OT environments are almost always unique, therefore, creating a testbed for them requires customization which is not always available when using the tools available on the market today. When designing a testbed for your network, it is important to define the techniques and attack scenarios that are most relevant to you. Whatever you do, do not rely solely on past or known TTPs. The OTORIO Research team has created 18 abilities dedicated to OT, using Caldera. These abilities can be used to test security systems in the ICS environment. Because of the sensitivity of creating attack tools, OT Caldera is not open sourced. However, researchers from the ICS community are encouraged to reach out to our team for additional information and knowledge sharing. Feel free to reach out to OTORIO at [email protected].

OTORIO designs and markets the next generation of OT security and digital risk management solutions. The company combines the experience of top nation-state cybersecurity experts with cutting edge digital risk management technologies to provide the highest level of protection for the manufacturing industry.

ABOUT OTORIO

Testing Security Mechanisms with Real-life IT/OT Scenarios

Documents

Transcript of Testing Security Mechanisms with Real-life IT/OT Scenarios