Hit ‘em Where it Hurts:A Live Security Exercise on

Cyber Situational Awareness

Adam Doupé, Manuel Egele, Benjamin Caillat, Gianluca Stringhini, Gorkem Yakin, Ali Zand, Ludovico

Cavedon, and Giovanni Vigna

University of California, Santa Barbara

ACSAC 2011 – 7/12/11

What Are Live Security Competitions?

• AKA Hacking Competitions• Useful educational tool for teaching computer security• Born as a way to showcase security skills

– DefCon’s CTF

• Various forms– Challenge set (DefCon quals, iCTF challenges, CMU’s

competition, DIMVA competition, RuCTF)– Capture the flag (DefCon, iCTF 2003-2007, CIPHER)– Other designs

• Attack-only (e.g., iCTF 2008)• Defense-only (e.g., Cyber Defense eXercise)

Doupé - 7/12/11

Why Live Security Competitions?

• Real-time factor enhances understanding• Forces teams to:

– Analyze unknown services/binaries– Defend systems from attack– Utilize different security skills– Work as a team– Create novel tools

Doupé - 7/12/11

Key Insight

• Security competitions can be designed to generate datasets for research

• In the 2010 international Capture The Flag (iCTF), we structured the competition to create a Cyber Situational Awareness dataset

Doupé - 7/12/11

Situational Awareness

• By putting perceived events into the context of the currently executing mission, one can improve decision making

• Mission– Series of tasks that an organization wishes to carry

out

• Task– Discrete step that is carried out using a service

• Service– Provided to users to accomplish a task

Doupé - 7/12/11

Cyber Situational Awareness

• Situational awareness extended to the cyber domain

• Large organizations constantly under attack– Which attacks are important?– Which assets are important?

• “What if” scenarios

Doupé - 7/12/11

Overview

• Live Security Competitions• Situational Awareness• Design of the 2010 iCTF• Cyber Situational Awareness Metrics• Lessons Learned• Conclusion

The 2010 iCTF: A Cyber SA Competition

• Introduced the concept of cyber-mission• “Not all attacks are created equal”• Participants must be aware of cyber-

missions and cyber-assets• Attackers must time their attacks to cause

the maximum amount of damage

The Setting

• Teams are part of a coalition to bring down the rogue nation of Litya

• LityaLeaks site used to leak description of Litya’s cyber-missions

• Litya’s network protected by a firewall and an IDS– If an attack is detected, nation’s access is shut off– Nations can bribe network administrator

• Litya has a botnet in each nation, stealing their money– If botnet is disabled, nation’s access shut off

• Money made by solving side challenges.

CARGODSTR-TQ-1442

COMSAT-WK-1127

SEDAFER-GOT-BKT-8217

DRIVEBY-DEPLOY-QFK-9751

Doupé - 7/12/11

Petri-net Representation of Mission

Service 1 Service 2 … Service 10

.. .

.

Service 1 Service 2 … Service 10

The Bank

ScoreBot

.. .

.

InternalNetwork

…

VPN server

Botnet C&C

The Bank

Service 1 Service 2 … Service 10ScoreBot

.. .

.

InternalNetwork

…

VPN server

Firewall/IDS

Botnet C&C

The Bank

Briber

Flag Submission

Service 1 Service 2 … Service 10ScoreBot

.. .

.

InternalNetwork

…

VPN server

Firewall/IDS

Botnet C&C

The Bank

Briber

Flag Submission

Service 1 Service 2 … Service 10ScoreBot

.. .

.

Challenges

ScoreBoard

LityaLeaks

Service 1

InternalNetwork

…

VPN server

Firewall/IDS

Service 2 … Service 10 Botnet C&C

The Bank

ScoreBot

Briber

Flag Submission

.. .

.

Challenges

ScoreBoard

LityaLeaks

Doupé - 7/12/11

Competition Overview

• December 3rd 2010 ~8 hours• 72 teams• ~900 participants (largest at the time) • 7 of 10 services compromised• 39 teams submitted 872 flags• 69 of 72 teams solved at least 1 challenge• 37 GB of traffic

Doupé - 7/12/11

Analysis of iCTF Data

• Use the data to validate models and theories

• We introduce two Situational Awareness metrics:– Toxicity

• Capture the amount of damage an attacker has caused

– Effectiveness• Capture how effective the attacker was at causing

damage

Analysis – CAD - Criticality

• C(s, t): service criticality [0,1]– Expresses the criticality of service s at time t– Function can have any shape

• iCTF: 1 when service active, 0 otherwise

Service: MostWanted

Analysis – CAD - Attacker

• A(a, s, t): attacker activity [0, 1]– Represent the attacker’s activity with respect

to a service– Can have any shape

• iCTF: 1 when team attacked a service, 0 if no attack

Team: PPP Service: MostWanted

Analysis – CAD - Damage

• D(s, t): Damage to the attacker [0, 1] – Represents the penalty for performing an

attack against service s at time t– Can have any shape

• iCTF: 1 when service is inactive, 0 when active

Service: MostWanted

Analysis – Toxicity

Analysis – Effectiveness

Analysis – Toxicity of PPP

Team: PPP Service: OvertCovert

Analysis – Toxicity and Effectiveness

Doupé - 7/12/11

Overview

• Live Security Competitions• Situational Awareness• Design of the 2010 iCTF• Cyber Situational Awareness Metrics• Lessons Learned• Conclusion

Doupé - 7/12/11

Lessons Learned

• The Good– Pre-competition information prepared teams who

took advantage– Winning team automatically qualified for DefCon

• The Bad– Structure of the competition was complex and was

understood by a subset of the teams– Services too hard

• The Ugly– Intentionally put a root backdoor into bot– Losing points sucks

Conclusions

• Live security exercises great for learning and security education

• They can be designed to create a research dataset

• Designed the 2010 iCTF to produce the first publically available dataset on CSA

• Presented SA metrics: toxicity and effectiveness

Doupé - 7/12/11

Questions?

Data: http://ictf.cs.ucsb.edu/data/ictf2010/

Email: adoupe@cs.ucsb.eduTwitter: @adamdoupe

Service Exploitation