Gray, the New BlackGray-Box Web Vulnerability Testing

Brian ChessFounder / Chief Scientist

Fortify Software, an HP CompanyJune 22, 2011

Todo

• Define gray-box testing• Why black-box is insufficient• What we built• Examples• Haters club

Definitions

• Black-box testing• System-level tests• No assumptions about implementation

Definitions

• White-box testing• Examine implementation• Test components in isolation

Definitions

• Gray-box testing• System-level tests (like black-box)• Examine implementation (like white-box)

The Software Security Game

• Objective• Rules vs. Strategy• Playing Field

OBJECTIVE:Protect everything

OBJECTIVE:Exploit one vulnerability

Rules for the Defender

1. Don’t attack the attacker

Rules vs. Strategy

Rules• Don’t attack the attacker

Strategy• Emulate attacker’s techniques

Who wins?

• Technology• Expertise

Who wins?

• Time

• Technology• Expertise

Who wins?

• Technology• Expertise• Time

Changing the odds

The Defender’s Advantage

• Time

• InsideAccess

• Technology• Expertise

Prior Art

• 2005: Concolic testing: Sen, University of Illinois

• 2008: Microsoft SAGE: Godefroid, MSR

• 2008: Test Gen for Web Apps: Shay et al, U. Washington

• 2008: Accunetix: Accusensor

Access to the Software

Allows for ‘Hybrid’ analysis

Dynamic

Analysis

Black-box Approach

Static Analysis

White-box Approach

‘Hybrid’ Analysis

Dynamic Analysis

Static Analysis

Mostly Broken

Correlation Engine

The ‘Real-Time Hybrid’ Approach

Dynamic Analysis

Static Analysis

Good Results

Real-Time Analysis

Correlation Engine

Evolving to Integrated Analysis

Dynamic Analysis

Application

Real-Time Analysis

Real-time link

• Find More• Fix Faster

Find More

• Reduce false negatives• Automatic attack surface identification• Understand effects of attacks

• Detect new types of vulnerabilities• Privacy violation, Log Forging

Attack surface identification

/login.jsp

/pages/account.jsp

/pages/balance.jsp

/admin/admin.jsp

• File system• Configuration-driven• Programmatic

Understand effects of attacks

/admin/admin.jsp✗

Command Injection

sysadmin$./sh

✔

Fix Faster

• Reduce False Positives• Confirm vulnerabilities

• Provide Actionable Details• Stack trace• Line of code

• Collapse Duplicate Issues• Tie to root cause

Reduce False Positives

/admin/admin.jsp

SQLi?✔

Actionable Details

/login.jsp

Collapse Duplicate Issues

/login.jsp

/pages/account.jsp

/pages/balance.jsp

1 Cross-Site Scripting 2 3 1

JavaBB – Case Study

• Open Source Bulletin Board

• Additional Vulnerabilities• Finds18 SQL Injection results

• Root cause analysis• 18 SQL injection results have 1 root cause

Vulnerability Diagnosis

Confirmed SQL Injection

Actionable Details

Line of Code

Parameters

Stack Trace

Yazd – Case Study

• Open Source Forum

• Additional Attack Surface• Discovers hidden ‘admin’ area• 3 Additional Cross-Site Scripting results

• Root cause analysis• Collapses 34 XSS into 24 root-cause vulnerabilities

Attack surface identification

Hidden ‘admin’ area

Collapse Duplicate Issues

One More Case Study

Future

• Automated anti-anti automation

The Case Against “Hybrid”

• Hard to find attack surface with static analysis• Static/dynamic correlation doesn’t work• Doesn’t help with false positives / false negatives• Nobody will run a software monitor (cheating!)

The Case for Gray-Box Testing

• Black-box is a losing game• Find more

• Attack surface• Vulnerability diagnosis

• Fix faster• Root cause analysis• Collapse duplicates

Gray, the New BlackGray-Box Web Vulnerability Testing

Brian ChessFounder / Chief Scientist

Fortify Software, an HP CompanyJune 22, 2011