Application Communities Phase IITechnical Progress, Instrumentation, System Design, Plans

March 10, 2009

2March 10, 2009 Application Communities Phase 2

Instrumentation

3March 10, 2009 Application Communities Phase 2

Instrumentation

• Firefox is run under PIN tool

• PIN allows us to insert call-outs into running executable

• Allows us to execute our code before or after executable’s code

• Can be targeted to specific system calls• CreateFileW

• Can be targeted to general events• Function calls

• Function returns

4March 10, 2009 Application Communities Phase 2

Call Stack Instrumentation

• Determine stack at particular program points

• Gather valid stack traces during normal operation

• Detect invalid stack traces during attack

5March 10, 2009 Application Communities Phase 2

Backtrace and Shadow Stack

• Shadow Stack• Each function call and return is monitored

• Tricky bits: (Setjmp, longjmp)

• Maintains mirror of stack during execution

• Overhead associated with each call and return

• Stack accuracy is ensured

• Backtrace• When system call is reached, walk back up stack to determine stack

information

• Only system calls are instrumented so overhead is less

• Compiler optimizations can hide stack information

6March 10, 2009 Application Communities Phase 2

Stack Trace Examplefunccall:[0]<CreateFileW>c:\AC2\firefox_bin\searchplugins\google.gif

54: CreateFileW[0x10760] C:\WINDOWS\system32\kernel32.dll [S]53: CreateFileA[0x1a24] C:\WINDOWS\system32\kernel32.dll [S]52: __PR_MD_OPEN[0x25920] c:\AC2\firefox_bin\nspr4.dll [S]51: _PR_Open[0x1c8d0] c:\AC2\firefox_bin\nspr4.dll [S]50: nsLocalFile::OpenNSPRFileDesc[0x39c90] c:\AC2\firefox_bin\xpcom.dll [S]49: nsFileInputStream::Open[0xa2f0] c:\AC2\firefox_bin\components\necko.dll [S]48: nsFileInputStream::Init[0xa440] c:\AC2\firefox_bin\components\necko.dll [S]47: NS_NewLocalFileInputStream[0x7b750] c:\AC2\firefox_bin\components\necko.dll [S]46: nsFileChannel::EnsureStream[0x7fe20] c:\AC2\firefox_bin\components\necko.dll [S]45: nsFileChannel::AsyncOpen[0x80bb0] c:\AC2\firefox_bin\components\necko.dll [S]44: imgLoader::LoadImage[0x5520] c:\AC2\firefox_bin\components\imglib2.dll [S]43: nsContentUtils::LoadImage[0x138600] c:\AC2\firefox_bin\components\gklayout.dll [S]42: nsImageBoxFrame::UpdateImage[0x10cdc0] c:\AC2\firefox_bin\components\gklayout.dll [S]40: nsImageBoxFrame::AttributeChanged[0x10c790] c:\AC2\firefox_bin\components\gklayout.dll [S]39: nsCSSFrameConstructor::AttributeChanged[0x8fcb0] c:\AC2\firefox_bin\components\gklayout.dll [S]38: PresShell::AttributeChanged[0x334e0] c:\AC2\firefox_bin\components\gklayout.dll [S]37: nsXULDocument::AttributeChanged[0x2c0f60] c:\AC2\firefox_bin\components\gklayout.dll [S]36: nsXULElement::SetAttrAndNotify[0x3587d0] c:\AC2\firefox_bin\components\gklayout.dll [S]35: nsXULElement::SetAttr[0x358500] c:\AC2\firefox_bin\components\gklayout.dll [S]34: nsIContent::SetAttr[0x33230] c:\AC2\firefox_bin\components\gklayout.dll [S]33: nsXBLPrototypeBinding::AttributeChanged[0x2ab860] c:\AC2\firefox_bin\components\gklayout.dll [S]32: nsXBLBinding::AttributeChanged[0x2a8e10] c:\AC2\firefox_bin\components\gklayout.dll [S]31: nsXULElement::SetAttrAndNotify[0x3587d0] c:\AC2\firefox_bin\components\gklayout.dll [S]30: nsXULElement::SetAttr[0x358500] c:\AC2\firefox_bin\components\gklayout.dll [S]29: nsIContent::SetAttr[0x33230] c:\AC2\firefox_bin\components\gklayout.dll [S]28: nsXBLPrototypeBinding::AttributeChanged[0x2ab860] c:\AC2\firefox_bin\components\gklayout.dll [S]27: nsXBLBinding::AttributeChanged[0x2a8e10] c:\AC2\firefox_bin\components\gklayout.dll [S]26: nsXULElement::SetAttrAndNotify[0x3587d0] c:\AC2\firefox_bin\components\gklayout.dll [S]24: nsXULElement::SetAttr[0x3550f0] c:\AC2\firefox_bin\components\gklayout.dll [S]23: nsXULElement::SetAttribute[0x354fd0] c:\AC2\firefox_bin\components\gklayout.dll [S]22: XPTC_InvokeByIndex[0x8e2b0] c:\AC2\firefox_bin\xpcom.dll [S]9: nsJSContext::CallEventHandler[0x325830] c:\AC2\firefox_bin\components\gklayout.dll [S]8: GlobalWindowImpl::RunTimeout[0x314310] c:\AC2\firefox_bin\components\gklayout.dll [S]7: GlobalWindowImpl::TimerCallback[0x315180] c:\AC2\firefox_bin\components\gklayout.dll [S]6: nsTimerImpl::Fire[0x6d050] c:\AC2\firefox_bin\xpcom.dll [S]5: nsTimerManager::FireNextIdleTimer[0x6dad0] c:\AC2\firefox_bin\xpcom.dll [S]4: nsAppShell::Run[0x14cd0] c:\AC2\firefox_bin\components\gkwidget.dll [S]2: xre_main[0x2f30] c:\AC2\firefox_bin\firefox.exe [S]0: mainCRTStartup[0x16d20] c:\AC2\firefox_bin\firefox.exe [S]

7March 10, 2009 Application Communities Phase 2

Detect Exploit with Stack Traces

• Gather set of call stacks for a valid execution

• Confirm subsequent valid executions do not produce new stacks

• Gather set of call stacks when exploit is run

• Compare two sets to determine if exploit can be detected with stack traces only

• Exploit produced 8 different call stacks

• In this example, exploit writes file “c:\booom.bat”

8March 10, 2009 Application Communities Phase 2

Exploit is Detected!

funccall:[0]<CreateFileW>c:\booom.bat--------------------------------------

43: CreateFileW[0x10760] C:\WINDOWS\system32\kernel32.dll [S]42: CreateFileA[0x1a24] C:\WINDOWS\system32\kernel32.dll [S]41: __PR_MD_OPEN[0x25920] c:\AC2\firefox_bin\nspr4.dll [S]40: _PR_Open[0x1c8d0] c:\AC2\firefox_bin\nspr4.dll [S]39: nsLocalFile::Create[0x39e20] c:\AC2\firefox_bin\xpcom.dll [S]38: nsLocalFile::CreateUnique[0x42100] c:\AC2\firefox_bin\xpcom.dll [S]37: XPTC_InvokeByIndex[0x8e2b0] c:\AC2\firefox_bin\xpcom.dll [S]30: nsJSContext::EvaluateString[0x324b50] c:\AC2\firefox_bin\components\gklayout.dll [S]29: nsJSThunk::EvaluateScript[0x3366c0] c:\AC2\firefox_bin\components\gklayout.dll [S]28: nsJSChannel::InternalOpen[0x337ed0] c:\AC2\firefox_bin\components\gklayout.dll [S]27: nsJSChannel::AsyncOpen[0x337eb0] c:\AC2\firefox_bin\components\gklayout.dll [S]26: nsDocumentOpenInfo::Open[0x35cf0] c:\AC2\firefox_bin\components\docshell.dll [S]25: nsURILoader::OpenURI[0x37d40] c:\AC2\firefox_bin\components\docshell.dll [S]24: nsDocShell::DoChannelLoad[0x23740] c:\AC2\firefox_bin\components\docshell.dll [S]23: nsDocShell::DoURILoad[0x226a0] c:\AC2\firefox_bin\components\docshell.dll [S]22: nsDocShell::InternalLoad[0x21360] c:\AC2\firefox_bin\components\docshell.dll [S]21: nsDocShell::LoadURI[0x12370] c:\AC2\firefox_bin\components\docshell.dll [S]20: nsWindowWatcher::OpenWindowJS[0x2af0] c:\AC2\firefox_bin\components\embedcomponents.dll [S]19: GlobalWindowImpl::OpenInternal[0x312b50] c:\AC2\firefox_bin\components\gklayout.dll [S]18: GlobalWindowImpl::OpenDialog[0x30f080] c:\AC2\firefox_bin\components\gklayout.dll [S]17: XPTC_InvokeByIndex[0x8e2b0] c:\AC2\firefox_bin\xpcom.dll [S]9: nsJSContext::CallEventHandler[0x325830] c:\AC2\firefox_bin\components\gklayout.dll [S]8: GlobalWindowImpl::RunTimeout[0x314310] c:\AC2\firefox_bin\components\gklayout.dll [S]7: GlobalWindowImpl::TimerCallback[0x315180] c:\AC2\firefox_bin\components\gklayout.dll [S]6: nsTimerImpl::Fire[0x6d050] c:\AC2\firefox_bin\xpcom.dll [S]5: nsTimerManager::FireNextIdleTimer[0x6dad0] c:\AC2\firefox_bin\xpcom.dll [S]4: nsAppShell::Run[0x14cd0] c:\AC2\firefox_bin\components\gkwidget.dll [S]3: .text[0x2739] c:\AC2\firefox_bin\components\appshell.dll [U]2: xre_main[0x2f30] c:\AC2\firefox_bin\firefox.exe [S]1: .text[0x1005] c:\AC2\firefox_bin\firefox.exe [U]0: mainCRTStartup[0x16d20] c:\AC2\firefox_bin\firefox.exe [S]

9March 10, 2009 Application Communities Phase 2

Callstack Tree

10March 10, 2009 Application Communities Phase 2

Conclusions: Instrumentation

• Able to gather stack traces by instrumenting executables

• Stack traces can detect exploits

11March 10, 2009 Application Communities Phase 2

System Design

12March 10, 2009 Application Communities Phase 2

AC2 Deployment approach

ACServer

RMI Registry

Linux server

InvariantsAttacks

App RecordsPatches

ACClientService

ACClientService

Windows Host N

Windows Host 1

1. bind 2. Lookup(find server)

Socket

messages3. registerService

(give self as callback)

RMISocket Call

ACServer and ACClientService are implemented in Java

Firefox

instrumentation

Firefox

instrumentation

KeystoreTruststore Keystore

Truststore

SSL handshake

13March 10, 2009 Application Communities Phase 2

Design features

• Java implementation of ACServer and ACClientService for cross-platform deployment

• RMI is a high performance, secure middleware technology for a Java-based distributed application• Point-to-point rather than message based

• Easy to deploy and debug

• High throughput

• Socket-based interface to instrumented applications for generality• Standards-based (POSIX), platform neutral

14March 10, 2009 Application Communities Phase 2

Security considerations

• Use Secure Sockets Layer (SSL)/Transport Layer Security (TLS) protocols (supported by Java 1.5) instead of Java Remote Method Protocol (JRMP) to secure communication between client and server• javax.rmi.ssl.SslRMIClientSocketFactory and • javax.rmi.ssl.SslRMIServerSocketFactory

• Will use keytool to generate key and trust stores• Will pre-deploy security information

• ACClientService will be configured to accept communications only from the local host • An outside attacker will not be able to hijack ACClientService unless the

host is irretrievably compromised

• For connection between applications and ACClientService we will consider• TCP which is completely portable and securable• Named pipes, more efficient and easier to secure

15March 10, 2009 Application Communities Phase 2

ACServer and ACClientService Interfacesclass AC2SystemClasses

«interface»ACServ er

+ registerService(ACClientService) : void+ serviceShutdown(ACClientService) : void+ applicationStartup(ApplicationInstance) : void+ applicationShutdown(ApplicationInstance) : void+ applicationFailure(ApplicationInstance, Context) : void+ invariantViolation(InvariantChecker, Context, ApplicationInstance) : void+ patchExecuted(Patch, Context, ApplicationInstance) : void

class AC2SystemClasses

«interface»ACClientServ ice

+ applyPatches(ApplicationInstance, Patch []) : void+ getName() : String

RMI

RMI

16March 10, 2009 Application Communities Phase 2

Application Startup Sequence Diagramsd ACOv erv iewSequence

anApplication

«implementationClass»

:ClientServiceImpl

«interface»

:AppInstanceHandler

«interface»

:ACServer

:CommunityListenerCreate()

write(startup_msg(application, pid))

create()

appStartup()

applicationStartup(ApplicationInstance)

applyPatches(ApplicationInstance, InvariantChecker[])

applyPatches(List<InvariantChecker>)

write(apply_patches_msg(patches))

17March 10, 2009 Application Communities Phase 2

RMI Transport Object Structure (1/2)class AC2SystemClasses

Application

+ getName() : String+ getVersion() : String

ApplicationInstance

+ getProcessId() : int+ getApplication() : Application+ getClientService() : ACClientService

Patc h

+ fromBytes(byte[]) : MessageElement+ getPatchUID() : long+ toBytes() : byte[ ]

RepairInv ariantChecker

PatchEx ecution

We need an OCL constraint here:if a patch is applied to an application instance, it must be in the set of applicablePatches.

Code«interface»MessageElement

+ fromBytes(byte[]) : MessageElement+ toBytes() : byte[ ]

0..*

executingInstanceOf

1

+patchedInstances 0..*

+appliedPatches 0..*

+forApplication 1

+applicablePatches0..*

18March 10, 2009 Application Communities Phase 2

RMI Transport Object Structure (2/2)class AC2SystemClasses

Patc h

+ fromBytes(byte[]) : MessageElement+ getPatchUID() : long+ toBytes() : byte[ ]

Inv ariantChecker

«C-struct»Contex t

+ fromBytes(byte[]) : MessageElement+ getStack() : Stack+ getRegisterValues() : RegisterValues+ toBytes() : byte[ ]+ getMemoryMap() : MemoryMap

«C-struct»Stac k

«C-struct»RegisterValues

«C-struct»MemoryMap

Attack

Instantiated as a result of an invariantViolation call

«interface»MessageElement

+ fromBytes(byte[]) : MessageElement+ toBytes() : byte[ ]

0..*

0..1

19March 10, 2009 Application Communities Phase 2

Progress Summary

And Plans

20March 10, 2009 Application Communities Phase 2

• Instrumentation• Initial shadow stack on Unix and Windows using Pin

• Implemented using callouts • Could inline for improved performance

• Initial backtrace on Unix and Windows using Pin• Faster, but requires re-building the binary

• Compared the two approaches for testing

• Stack comparison tools for detecting exploits• Visual tree representation of stack sets

• Evaluated Apache, insufficient number of in-scope exploits• Reproduced several …

• Evaluated Firefox, many in-scope, explored details of many• Investigated GUI-replay to recreate traces for learning

• Because many exploits require some user interaction• Advanced Key and Mouse Recorder ($60/user, 5 users)• http://mouse-recorder.macro-expert.com/recorder-features.htm

Progress to date – Instrumentation & Exploits

21March 10, 2009 Application Communities Phase 2

Progress to date – Implementation and Infrastructure

• AC2 designed as a distributed system• ACServer interacts with AClientService nodes, deployed different

workstations

• Each ACClientService node interacts with applications running on local host

• Designed the AC2 Environment so that it can be tested in isolation by running mock application instances that execute configurable attack scenarios

• Status: Scenario-driven simulated application can send messages to ACClientService, which passes messages to ACServer, receives responses from the server and transmits the responses to the simulated application.

22March 10, 2009 Application Communities Phase 2

Plans – Infrastructure/Implementation• Implement socket-based messages between applications and

ACClientService• Complete the design and implementation of the RMI transportable

objects between ACServer and ACClientService• Incrementally add functionality to ACServer• Add detailed debug scenarios to thoroughly test the infrastructure with a

covering set of use cases• Integrate with instrumented Firefox• Flesh out learning infrastructure – design & implementation.• Create demonstration for next program review

• Attack on Firefox

• Response based on default repair to abnormal stack trace or context

• Prevent a system call and return error code

• Elaborate this demo by reporting invariant violation and receiving repairs