Controlling System Calls and Protecting Application Data

in Virtual Machines

Koichi Onoue* Yoshihiro Oyama** Akinori Yonezawa*

* The University of Tokyo** The University of Electro-Communications

Protection for Applications

• Security systems has been widely applied to provide secure computing environments– Sandboxing systems

– Intrusion detecion/prevension systems (IDSes/IPSes)

– Anti-virus tools

OS kernel

Security system

Applicationcontrol

Confidential data

Security Systems Can Also Be Compromised！

• Security systems and the other applications are running in the same execution space

Application 1

OS kernel

Security system

Application 2

Confidential data

Application 1 was compromised !

control✗

Advantages of Virtual Machine Monitor(VMM) in Terms of Security

• VMM provides strong isolation between VMs– VMM prevents a

compromised VM from attacking the other VMs

• VMM can control access to physical resources such as physical memory and a disk– VMM is running at the

higher privileged level than VMs Hardware

Virtual machine monitor (VMM)

Virtual machine(VM)

OS （Guest OS）

Virtual machine(VM)

OS（Guest OS）

Application Application

Our Goal

• Enhancing application security by a system running outside of VMs– In cooperation with VMM, the security system

controls behaviors of application and protects application data

VMM

Control VM Target VM

ApplicationSecuritysystem

Our Approach

• Our system consists of program in VMM and program in control VM– They run outside of target VMs

• It controls system calls invoked by application process• It controls memory and file operations related to target

applications

Our system controls only the target applications specified by users

We extend a para-virtualization version of Xen

Controlling System Calls fromOutside of target VMs

Comparison between “w/o VMM” and “w/ VMM”

Securitysystems

(“w/o VMM”)

Security system in cooperation with VMM

(“w/ VMM”)

Attack against security systems

✗Not hard

○Hard

Execution states obtained by

security systems

○OS-level

✗Hardware-level

Securitysystems

Security systems in cooperation with VMM

Attack against security systems

✗Not hard

○Hard

Execution states obtained by

security systems

○OS-level

✗Hardware-level

Goal for Controlling System Calls

Our goal

Semantic gap

Approach to Controlling System Calls

• Controlling system calls from outside of VMs– Using information on target OSes

created in kernel build

– Conforming to security policies

VMM

Control VM Target VM

application

Securitysystem

Security policy System call is

invoked

control

Bridging the Semantic Gap

• What a VMM can observe– Events ：Privileged instructions, interrupts, …

– Execution States：Registers, memory pages, …

• What security systems require– Events ：System calls, …

– Execution states： Process ID, system call number, …

Semantic gap

Security Policy

• Specifies controlled system calls withpattern matching

...open default: allow

fileEq(“/etc/passwd”)or filePrefixEq(“/etc/cron.d”)

deny(EPERM)...

Controlling Memory and File Operations Related to Application Data

Goal for Protecting Application Data

• Prevent compromised programs from leaking target data and tampering with them– We assume attackers read/write application data

with ptrace system call and kernel modules, etc.

VM

Target

OS kernel Application

Confidential data

Compromised program

Leaking

Tampering

Memory

Target

Virtual disk

Approach to Protecting Application Data

• Hiding “real” application data on memory and a virtual disk from compromised programs– Compromised programs include target OS kernel

• Application data on memory– Code region, data region, stack region, etc.➔VMM multiplexes physical pages

• Overshadow[Chen et al., 2008]• [Rosenblum et al.,2008]

• Application data on a virtual disk– Executables, configuration files, etc.➔ Control VM manages them

OS Memory Management

Virtual address space

Physical address space

Application

Application

OS addresstranslation

VMM Memory Management

Target VM virtual address space

（gVA）

Target VM physical address

space（gPA）

Application

application

gVA → hPA

VMM address translation

gPA → hPA

VMM physical address space

（hPA）

Application

17

Protecting Memory (1/2)

• According to the operational mode, a VMM switches accessible physical pages

Target VM virtual address space

（gVA）

Target VMphysical address

space（gPA）

VMM physical address space

（hPA）

ApplicationApplication

“Real” data

Dummy data

Accessible page at user-mode

Accessible page at kernel-mode

Protecting Memory (2/2)

• VMM switches page tableswhen the operational mode is changed– Exception/Interrupt handling

– System call handling

Approach to Protecting Application File Data (1/5)

• Control VM manages “real” target files– Executables, configuration and data base files, etc.

– Security policy specifies target files

• Target VM manages “dummy” target files

VMM

Control VM Target VM

Security system

Security policy

“Real” configuration file

Application

“Real” executable

Dummy configuration

fileDummy

executable

Approach to Protecting Application File Data (2/5)

VMM

Control VM Target VM

Security system

Configuration file

Application

Memory

Application readsa configuration file

Approach to Protecting Application File Data (3/5)

VMM

Control VM Target VM

Security system

Configuration file

Application

Memory

VMM intercepts “read” system call

Approach to Protecting Application File Data (4/5)

VMM

Control VM Target VM

Security system

Configuration file

Application

Security system emulates “read”

Memory

Approach to Protecting Application File Data (5/5)

VMM

Control VM Target VM

Security system

Configuration file

Application

Memory

VMM notifies application of a result of “read”

Conclusion

• We have proposed a system that enhances application security inside target VMs– Controlling of application behaviors

• Controlling of system calls from outside of target VMs

– Protecting application data on memory anda virtual disk

• Application memory data: VMM multiplexes target physical pages

• Application file data: Control VM manages them

Thank you for your attention