SECURITY IN LIVE VIRTUAL MACHINE MIGRATION

A Thesis by

Shah Payal Hemchand

B.E., Government College of Engineering, North Maharashtra University, Jalgaon, 2008

Submitted to Department of Electrical and Computer Engineering

and faculty of the Graduate School of

Wichita State University

in partial fulfillment of

the requirements for the degree of

Master of Science

December 2011

ii

© Copyright 2011 by Shah Payal Hemchand

All Rights Reserved

iii

SECURITY IN LIVE VIRTUAL MACHINE MIGRATION

The following faculty members have examined the final copy of this thesis for form and content,

and recommend that it be accepted in partial fulfillment of the requirement for the degree of

Master of Science with a major in Computer Networking.

_________________________________

Ravi Pendse, Committee Chair

_________________________________

Abu Asaduzzaman, Committee Member

_________________________________

Linda Kliment, Committee Member

iv

DEDICATION

To the Almighty, my family, for their continuing support and patience; to my WSU friends

for their significant advice and time throughout the completion of my thesis.

v

ACKNOWLEDGEMENTS

I sincerely thank my thesis advisor, Dr. Ravindra Pendse for his devoted motivation and

supervision throughout my career at Wichita State University. His guidance helped me complete

my thesis successfully. By working as a Graduate Research Assistant for him I gained

knowledge, and professional work ethics.

I take this opportunity to thank Amarnath Jasti for his constant support and guidance

throughout my thesis. His suggestion and advice helped me understand the technology and gain

more knowledge. His opinion towards my academic and career were valuable. I would like to

thank members of the committee for their effort and time.

I would like to extend my gratitude towards to Yonatan Assefa and all those who directly

or indirectly helped motivate me with my research.

vi

ABSTRACT

Virtualization has become an essential technology for organizations. With on-demand

services from vendors, a substantial rise in the use of virtualization has been noticed. Today there

are various kinds of virtualization techniques offering different advantages. One of the important

features of virtualization is live virtual machine (VM) migration. In live VM migration, the

controls of a VM are migrated from one physical host to another. Workload balancing, and

server maintenance becomes easy by migrating the VM. The ability to reboot or shut down the

physical server without affecting running applications is greatly beneficial to an organization.

With this indispensable feature of live VM migration, the security factor is still

unanswered. Very little research has been done in exploring the security concerns inherent while

data moves between the two physical machines. This thesis looks at this poorly explored area

and attempts to propose a proper solution, and thereby maintain security. Man–in–the–middle

attack could be created by sniffing data between the hypervisors and confidentiality is lost. Data

in transit could be read and then tampered with or misused, and can create havoc in the network

and bring it down completely.

The research shows how a malicious attacker can sniff the data while performing live

VM migration over Xen hypervisor and exploit information. Using this experiment the author

proposes strategies that can be used to have a secure live migration process.

.

vii

TABLE OF CONTENTS

Chapter Page

1. INTRODUCTION 1

1.1 Overview 1

1.2 Storage Area Network 2

1.3 Security 4

1.4 Problem Identification 6

1.5 Organization of Thesis 6

2. VIRTUALIZATION 7

2.1 Definition 7

2.2 Origin of Virtualization 7

2.3 Need of Virtualization 8

2.4 Types of Virtualization 9

3. LITERATURE SURVEY 13

3.1 Security in Virtualization 13

3.2 Related Work 15

4. XEN OVERVIEW 19

4.1 Xen Hypervisor 19

4.1.1 Why Xen? 20

4.1.2 Other Features of Xen 20

4.1.3 Working of Xen 21

4.2 Requirements for Live VM Migration 22

viii

TABLE OF CONTENTS (cont…)

Chapter Page

5. TEST BED AND RESULTS 24

5.1 Hardware 24

5.2 Software

5.2.1openSUSE 24

5.3 Test Bed 24

5.4 Results 26

5.5 Proposed Solution 29

5.5.1 IPSec Tunnel 29

5.5.2 Hypervisor Encryption 31

6. FUTURE WORK AND CONCLUSION 32

6.1 Future Work 32

6.2 Conclusion 32

REFERENCES 34

ix

LIST OF FIGURES

Figures Page

1 Basic SAN Environment 3

2 Key elements of Data Center 4

3 OS Virtualization 10

4 Paravirtualization 11

5 Full Virtualization 12

6 Test Bed 25

7 Three-way handshake between host A and host B 26

8 Three-way handshake capture 27

9 Wireshark capture showing ascii values 28

10 Live VM migration through IPSec tunnel 30

11 Encrypted data after VM migration through IPSec tunnel 30

x

LIST OF ABBREVIATIONS

AoE ATA over Ethernet

DoS Denial – of – Service

IDS Intrusion Detection System

IPS Intrusion Prevention System

iSCSI Internet Small Computer System Interface

NFS Network File System

OS Operating System

SAN Storage Area Network

VM Virtual Machine

VMM Virtual Machine Monitor

1

Chapter 1

Introduction

1.1 Overview

In an IT environment, aggregation of computing technology and storage resources

enables an enterprise to reduce the operational cost, and improve efficiency and flexibility in the

hardware utilization. Organizations have huge servers, costly data centers, consume a lot of

power, require and space and generates large amount of heat. In earlier times, it was cumbersome

to maintain the larger mainframe computers. However all the work was done by the mainframe

and workstations were the dumb terminals [1]. Organizations had to invest a considerable

amount of time and money into the mainframe technology to keep up with tasks. Mainframe

Computers were quite expensive, occupied more space for hardware, and required more human

attention and more resource consumption. Organizations required more hardware for different

projects, and to manage more data and software; huge data centers were required. To meet

business requirements, exponentially growing data, organizations sought effective and maximal

storage capabilities. All this led to the rise of virtualization and Storage Area Network (SAN) [2]

to gain higher network efficiency. Virtualization came into play to consolidate the individual

servers and today more and more servers are opting for SAN storage, thereby making the local

hard drives insignificant.

With the advent of virtualization [3], hardware is consolidated in data centers to a single

modern server that runs several virtual servers, thereby maintaining high performance and

efficiency. Less hardware utilization leads to less consumption of electricity, less space used and

reduced number of electrical components; leading to Green Computing [4]. Virtualization has

gained extensive attention in the growing world of technology. Hardware miniaturization and the

capability of installing more software require an organization to use multi-tenant services. Due to

2

rapid growth and popularity, virtualization is said to be, “The Fundamental block for today‟s

technological world”. Virtualization is further explained in detail in Chapter 2.

1.2 Storage Area Network (SAN)

Every small organization has a strategy to build a central storage so servers, new software

or larger applications which need more memory are constantly developed. Storing data on a local

hard disk is becoming archaic. So the idea of having a central storage that constitutes a SAN

could be as simple as just plugging in the terminal in the mainframe world.

SAN is a collection of hubs, fabric, and software. It is a high-speed network of storage

elements, such as shared storage arrays, and clusters servers, where one or more interfaces create

complete connectivity. A basic SAN environment is as shown in Figure 1. In a typical SAN

environment, uninterrupted availability of data is very important for the survival of an

organization. It is essential to have a reliable infrastructure to ensure that data is available at all

times. The key characteristics of a typical data center are capacity, security; availability,

performance, data integrity and scalability [5]. If all are met appropriately then an organization

can have unremitting services, as shown in Figure 2. Following is the brief description of key

characteristics of a data center:

3

Figure 1 Basic SAN Environment

a) Capacity: As per the growing needs and user demands, capacity requirements should

be increased to provide uninterrupted services. Data center operations must have

adequate resources to store a large amount of data. If more resources are required,

then the data should be reallocated rather than adding new resources.

b) Security: Proper integration and authentication should be maintained in the network

to ensure enough security. Clients should have complete access to allocated resources

on the specific storage arrays.

c) Availability: In an organization, data should be available all the time without any

inconvenience.

4

Figure 2 Key elements of Data Center

d) Performance: The system should have the ability to provide optimal performance by

offering services at high speed.

e) Data Integrity: Data integrity should be maintained by mechanisms such as error

detection and correction.

f) Scalability: The storage should be able to grow with user demands. The system

should be able to allocate additional processing capabilities for storage on demand

without interrupting the other services.

1.3 Security

With the upcoming needs of organizations, more attention is required to provide security

in a network [6]. Storage and virtualization are the evolving technologies, and so are the threats

5

and vulnerabilities. Features in such technologies can lead to unwanted and unexpected security

threats. Often technology is implemented without being aware of the security consequences.

Eavesdropping, data theft, and hacking are the basic security concerns within an organization

and over the Internet. Security could be broken accidentally or intentionally. If the critical

infrastructure is compromised, the consequences would be severe and lead to data loss. SAN is

considered wealthy since it has valuable information and sensitive data. If SAN fabric is

configured using conventional methods without any security controls, then compromising the

network not be difficult.

Organizations should think carefully about their data storage needs for a secure storage

and an integrated solution for threats and vulnerabilities. Security should be implemented layer-

by-layer, so that if one layer is compromised, the assets of the other layers are under protection.

Proper security measures, tools, and encryption techniques should be implemented to have a

secure organization. The technology should be understood to the core and eventually security

should be maintained by taking preventive measures across the network. A virtual environment

is more susceptible to security threats. Possible ways to break into virtualization are to

compromise a Virtual Machine (VM), hack the hypervisor, insert a rootkit, and sniffing the

traffic. All these possible ways need attention in an organization to have fail-proof security.

With virtualization, live VM migration [7] has gained more popularity. Live VM

migration attracted the users because of its transparent nature, and clean separation between

hardware and software. Administrators can consolidate the system load, improve power

efficiency, and improve infrastructure maintenance and flexible relocation of the sources [8].

Security is a bigger concern in Live VM migration as it poses threats to VM as well as network.

Data is in flight as well as at rest, which could be sniffed and security can collapse.

6

1.4 Problem Identification

An indispensable feature in the world of virtualization is live migration of a Virtual

Machine (VM). Live VM migration means migration of an entire OS and its associated

applications from one physical machine to another. An administrator can ensure complete

utilization of the available resources on any one machine or another. But with some added

advantage comes a threat. Security is the biggest concern while doing migration and has not

received its due attention. The VMs are migrated “live”, without disrupting the applications and

the network with the least minimal downtime. During the migration, an attacker in the middle

can gain control over the VM or capture the traffic and try to recover the data. The traffic

between the hypervisors can be sniffed and is more vulnerable to man-in-the-middle attacks or

Denial-of-Service (DoS). Confidentiality, data integrity, and essential credentials are lost easily.

Similarly, this could cause killing of the available resources for that particular VM. After losing

vital data, the host machines might have to shut the VM‟s to protect themselvess from an attack.

1.5 Organization of Thesis

This thesis brings a new vision and direction to less focused areas of securing the live

VM migration. Chapter 1 provides a brief overview of security in SAN and virtualization.

Chapter 2 explains virtualization, needs and types of virtualization in depth. Chapter 3 provides a

literature survey of how VM is migrated with minimal downtime and securing live VM

migration. Chapter 4 gives a detailed explanation how live VM migration could cause a big

security threat and about Xen hypervisor. Chapter 5 consists of the experiment and the results.

Chapter 6 concludes the thesis and recommendations are made for future work for securing live

VM migration.

7

Chapter 2

Virtualization

2.1 Definition

Virtualization is the emulation of hardware in the software platform. Virtualization in

simple words is defined as creating a Virtual Machine (VM) or a virtual copy of a device or a

resource, such as a storage device, network or an Operating System (OS). The VM fetches

resources from the physical resources (such as servers, and OS) and makes them look like a

multiple logical resource and vice-versa. Each application needs one server, which increases

cost. Virtualization offers the ability to run multiple applications and OS on the same machines.

Thus a small number of servers in the data center can perform better and accomplish more work

than before

2.2 Origin of Virtualization

Christopher Strachey, the first Professor of Computation at Oxford University and leader

of the programming research group, brought the term “Virtualization” to life, in his paper „Time

Sharing in Large Fast Computers‟. The term Virtual Machine Monitor (VMM) came into being

in 1960 [9] as a software abstraction layer partitioning a network platform into one or more

VMs. Bob Muglia, Senior Vice-President for server and tools business at Microsoft Cooperation

says “Virtualization is an approach to deploying computer resource that isolates different layers

– hardware, software, data, network and storage from each other” [10]. IBM wanted to logically

partition the mainframe computers into VM [18, 19 and 20]. IBM pioneered virtualization in

1970 [9, 11 and 12], but it was eventually perfected by others. Hardware level VMs disappeared

during the 1980‟s and 1990‟s [13, 14]. The dilemma of utilizing efficient hardware was faced by

the IT departments where eventually virtualization was applied to x86 architecture in order to

8

improve the issues such as high maintenance, infrastructure costs, management costs and disaster

protection. In 1990, Java VM [15] was developed apart from IBM‟s mainframe computers.

VMware Workstations were first utilized in 1999 [16] which allowed running virtual machines.

VMM became popular in 2005 and 2006 in academia and industry.

In 2005-2006, Xen researchers introduced the term hypervisor [17]. Since then Xen

became the de facto virtualization architecture as open source virtualization. Intel, AMD, Sun

Microsystems and IBM are building different virtualization strategies which would give the

technical world a new plethora.

2.3 Need for Virtualization

The demand for virtualization is growing because of the following advantages [21]:

Underutilized hardware: Many data centers have machines that utilize only 10-15% of

total processing capacity and the remaining 85-90% is unused.

Smaller footprint: Over the last two decades, technology has made a big jump and has

reached a platform where everything is a click away. The rise of internet accelerated this

transformation. Real-time communication and video-conferences are possible because of

the growing internet. This has led to the need for huge servers causing a real estate

problem for organizations. Space is required for such data centers. This is one of the

important reasons to adapt virtualization. Virtualization helps in reducing the cost of

building more data center space.

More system administration: Due to more hardware, every machine needs constant

monitoring, regular updates, OS, application and software installations. This increases the

amount of labor. With the ease of virtualization, the system administration‟s job is

reduced.

9

Ease of testing and development: Testing becomes easy with the deployment of

different OS environments and isolating the applications per VM.

Mobility: Virtual Machines can be migrated from one physical host to another, giving

better system mobility. This also improves the resource utilization.

2.4 Types of Virtualization:

There are different types of virtualization techniques as per the application or services.

Following are the types of virtualization [22]:

2.4.1 Server Virtualization

In Sever Virtualization, the resources of servers are hidden from its users. The users don‟t

need to understand the management of resources. Server Virtualization is done to increase the

utilization and sharing of resources. There are different ways to do Server Virtualization as:

2.4.1.a Operating System (OS) Virtualization

OS virtualization means running an OS on top of an existing one. The main reason for

OS virtualization is to provide the set of libraries that the application might interact with. The

application in hosted OS can‟t see any other applications in another virtual OS. OS virtualization

is offered by Sun and SW Soft [23] with the commercial product Virtuozzo [24]. OpenVZ [25] is

another open source OS virtualization. A typical OS Virtualization is shown in Figure 3.

10

Physical Hardware

Host Operating System

Guest Operating

System

Guest Operating

System Guest Operating

System

Figure 3 OS Virtualization

2.4.1.b Hardware Emulation

A hardware emulated environment (usually hypervisor) is presented to the guest OS in

the hosted environment. This is also referred to as the Virtual Machine Monitor (VMM). The

guest OS and the VMM form a complete consistent package and that can be completely migrated

from one physical host to another. A hypervisor can virtualize the physical host resources such as

CPU and memory, creating a virtual environment. The VMs running in a virtual environment

have an illusion of being in non-virtualized ones. A hypervisor is responsible for allocating

resources to each VM on demand.

2.4.1.c Paravirtualization

The guest OS is modified to run in paravirtualization. It is a subset of Server

Virtualization; it has a thin software interface between the hardware and the VM, not identical to

that of the underlying hardware. This is shown in Figure 4. The time spent in performing the

operations is reduced because of the modified interface. The guest VMs are aware that they are

11

running in a virtualized environment. The virtual devices rely on physical device drivers of the

underlying host. The device interaction is similar to that of full virtualization [26].

Physical Hardware

Hypervisor

Guest Operating

System

Dom 0

Guest Operating

System 1

Dom U

Guest Operating

System 2

Dom U

User Application User ApplicationUser Application

HypervisorDom 0 Interface

Figure 4 Paravirtualization

The abstraction formed with paravirtualization means that OS performs much better than

full virtualization. But the flexibility and security is lost. Since the OS needs to be modified,

flexibility is compromised. The readily available OS can‟t be used for paravirtualization. The

guest OS is close to the hardware and has more control. This creates risk of impacting the lower

hardware level and compromising the entire network.

2.4.2 Storage Virtualization

The amount of data being sent and received is growing enormously. With so many VMs,

the space required to store data is decreasing daily. Hence, there arises Storage Virtualization. If

data is stored on one machine, a bottleneck will be created if the single machine stops operating.

Organizations use specialized hardware and software along with array controllers, and disk

drivers to provide reliable storage for data processing. The storage is virtually centralized and

could be delivered over a Fiber channel, iSCSI, NFS or other storage protocols.

12

2.4.3 Full Virtualization

The guest OS remains unmodified in Full Virtualization. It provides a virtual

environment, namely a complete simulation of the underlying hardware as depicted in Figure 5.

Every feature of the hardware, such as interrupts, I/O operations, memory access, and full

instruction set which runs on the bare machine, should be reflected on the VM. The guest OS is

not aware that it is running in virtual world.

Physical Hardware

Hypervisor

Guest Operating

System

Guest Operating

System

Guest Operating

System

User Application User ApplicationUser Application

Figure 5 Full Virtualization

Full Virtualization helps isolate the users, and emulating the hardware to achieve security

and reliability, improve the efficiency, simplifies migration and portability. The hypervisor [27]

is used to allow the I/O devices to go to the guest OS machines by imitating the physical devices

in the VMM. It does not need any actual hardware to support virtualization which adds the

overhead performance [28]. Full virtualization is slower than any other virtualization techniques.

13

Chapter 3

Literature Survey

Senior Director of Product Management for VMware, Patrick Lin says, “Virtualization

plays a role as an opportunity and as well as a threat” [29]. Security is the most important aspect

every organization would strive for. A detailed discussion on different security threats and

measures are discussed here.

3.1 Security in Virtualization

Organizations have taken a risk with the technology and getting accustomed to compete

in the world considering the aspects of security. It is difficult to maintain a secure environment,

where frequent access to critical business applications is necessary. Going global requires many

data centers, and innovative business practices. Security experts [30] know that detecting threats

and information leaks is an order of magnitude more difficult than stopping malware. To protect

against this kind of threat very few measures have been taken to secure the crucial and sensitive

data.

As per Citrix whitepaper [31], workforce is mobile, and distributed for people working

remotely from home, offices or mobile devices having internet access. Security becomes

necessary and even more elusive because it brings the associated vulnerability. Protecting the

cooperate resources and assets is not easy. A single security breach can cost millions of dollars,

thereby compromising the organizations sensitive information. With iron clad security and

safeguards in place, a failed system should not bring an entire organization to its knees. A proper

strategy should be followed. Data should reside on servers and within data centers and not on the

14

client devices [32]. If only limited access is available over the network, data is secure even if a

laptop or a computer is compromised.

Virtualization is causing a transformation in the traditional way of resource utilizations,

speeding IT response to a new, well-structured business environment [33]. Any computer is

prone to attacks and threats. However, the virtualized computer is more likely to be exposed to

attacks since security is quite weak due to more holes to patch, and too many interconnections.

Common defensive steps to limit user access are in place, such as the Intrusion Detection System

(IDS)/ Intrusion Prevention System (IPS), and firewalls. The attacks possible in virtualization are

compromising the hypervisor or the kernel, a host module attack, VM escape, VM hopping, VM

monitoring from host, Hyperjacking, and VM sprawl [34], etc. Attackers and hackers [35, 36]

have an eye on one weak entry and can insert malware to detect a virtual environment and

change itself accordingly. One of the major attacks is the man-in-the-middle attack during live

VM migration. During live VM migration, data is in clear text, allowing man-in-the-middle

attack on a VM‟s hypervisor [37]. Virtualization and live migration enable new features

benefitting the end user. This combination gives rise to novel security threats. The hypervisor

implementing live migration feature can possibly expose both the OS and the guest VM to

attacks, resulting in integrity loss.

A VMM encapsulates the vivid and volatile states of a VM. When a VM is suspended, its

state is mapped to files in the local system of the host. Transition of a VM from physical host to

another is called live migration. Live migration provides workload balancing, and resource

sharing. Following are the advantages of live VM migration [38, 39]:

15

- Maintenance and Upgrade: When a physical host needs to be restarted, all the VMs

running on that machine are migrated over to another identical physical host without

causing interruptions in the services.

- Load Balancing: Depending on availability of resources VMs can be migrated

depending on the workload.

- Power Efficiency: If a physical host has only a couple of VMs and the workload is

low, they could be migrated over to other devices. This would reduce the power and

resources used.

3.2 Related Work

To perform live VM migration, state of the art uses a pre-copy approach [40]. This

technique transfers the memory pages first and then copies pages modified later Pre-copy is most

common for live VM migration as well as for wide area migration [41]. VM migrations have

been discovered as a tool providing mobility to users working at different times on different

machines. Thus they set an example by transferring an OS instance from one machine to another.

This collective project [42] aims to optimize the links, and reduce service downtime. Other

projects [43 and 44] have concentrated over long time span by pausing, stopping and then

migrating the memory pages during live VM migration.

Process migration moves a process from one physical host to another. Zap [45] uses

partial OS virtualization for migration of process domains (pods) with the help of a Linux kernel.

VMware added VMotion as an advantage for OS migration to their VirtualCenter management

software [46]. Process migration had been another area of interest for some time and more

research was done in the 1980‟s [47, 48, 49, 50, 51, 52, 53, 54, and 55]. However, process

16

migration didn‟t gain popularity because of the residual dependency limitation and mobility. A

better explanation and survey on process migration technique has been done in [56].

In contrast to process migration, OS migration handles all the limitations of the process

migration and still does the VM migration efficiently. The difference between OS migration and

process migration is that OS migration overcomes the problem of residual dependencies with the

help of a narrow interface between a virtualized OS and the hypervisor. The administrator need

not be concerned with what is running within the VM; instead they can migrate the OS and its

associated processes as one unit.

VM migration really means to migrate the control and memory of a VM from one

physical host to another, without causing any service disruption. Extensive research has been

done on how the memory pages are migrated over during live VM migration. In general,

memory migration [57] can be described as:

a) Push phase: During migration, the source VM is not suspended but is running while a

few pages are pushed to the destination machine. To make sure there is consistency

between the memory pages, the pages that are modified during the migration must be

re-sent.

b) Stop-and-copy phase: The VM to be migrated is stopped. The VM is only started

after the memory pages are transferred across the destination host.

c) Pull phase: If the destination host happens to access a page which is not yet

transferred from the source host, those pages are faulted in (“pulled”) across the

network from the source VM.

17

The best example of memory migration is stop-and-copy which involves starting a new

VM only after stopping the original VM and transferring all the pages. The downtime [58] and

total migration time are proportional to the amount of physical allocated memory of the VM, and

this gets affected since the migration of memory pages takes quite a long time. To have the least

amount of downtime, pure demand migration was adopted which uses a short stop-and-copy

phase. The essential kernel data structures are copied to the destination host. The new VM is

started after the complete migration of memory pages and the other pages are fetched across the

network at first use. This leads to less downtime but increased total migration time. Performance

degradation also occurs due to a considerable set of pages being faulted across.

The best option for VM migration is the pre-copy migration [59]. The pre-copy approach

provides a balanced way of migration by combining the bounded iterative push phase along with

a short stop – and – copy phase. The pre-copying of memory pages occurs in rounds hence the

term “iterative”. The pages which are modified during the first round are copied during the next

round. There will be a small set of pages for each VM which are updated often. Thess turn out to

be poor pages for pre-copy migration. The Writable Working Set was designed to calculate the

number of iterations for typical workloads.

Wide research has been carried out on the topic of making live VM migration efficient

and with minimal downtime. The live VM migration feature came into existence quite some time

ago. It was performed from one server to another server located in the same room or from one

rack to another rack which has physical security, and has minimal chances for loss of data. As

the demand grew for virtualization, live VM migration was performed in a LAN, from one data

center to another data center located in different places. Now the question arises: “How secure is

the live VM migration”? Due to wide spread locations, physical security is not possible, invites

18

data sniffing and poses a security threat. It is observed that during live VM migration, the data

can be sniffed and compromised. Any third party can gain access to the network, sniff the

ongoing traffic and visualize the data. Three different threats were explored during live VM

migration [60]. The traffic going over the data plane is sensitive and is not secured. The transit

path is open to any outside attacker to read the data during the transfer causing a man-in-the-

middle attack. This area of security requires more research since it is open for any attack. Using

the tool Xensploit, several ways to attack the live VM migration were evaluated. This includes

control plane, data plane and migration module. This thesis looks at how security can be

provided during live VM migration.

19

Chapter 4

Xen Overview

Xen provides an exceptional feature called live VM migration. To perform live VM

migration certain requirements must be fulfilled. During the migration, an attacker in the middle

can gain control over the VM or capture the traffic since it is easy to decode the sniffed traffic

and recover the data. The traffic between the hypervisors could be intercepted and is more

vulnerable to man-in-the-middle attack or Denial-of-Service (DoS). Confidentiality, data

integrity, and essential credentials are lost easily. Similarly, this could cause killing of the

available resources for that particular VM. After losing vital data, the host machines might have

to shut the VMs preventing them from being compromised. Not much research has been done in

securing the live VM migration. As mentioned in chapter 3, threats during live VM migration are

discussed [60] but no research has been done on how the traffic goes between the hypervisors.

This thesis concentrates on how the traffic flows between the hypervisors. A small experiment

was conducted to show that during live VM migration data goes in clear text and is vulnerable to

attack. The following section gives an overview of how Xen works and supports live VM

migration.

4.1 Xen Hypervisor

Xen is open source industry software for virtualization. The virtual world has been

gaining popularity with the ease of the Xen hypervisor. Xen offers an efficient, strong and secure

feature set for virtualization. The OS supported by Xen are Windows, Linux, Solaris, so it is

neutral. Being independent, Xen allows Domain 0 to be the unique VM and it has control over

all the other VMs.

20

4.1.1 Why Xen?

Xen is popular because it has features which dominate in the virtualization world and

hence is in demand. Xen doesn‟t contain any device drivers [62]. The VMs installed are isolated

from each other. They are not aware of any other VM running on the same physical OS.

4.1.2 Other Features of Xen

Xen is chosen to be implemented because of its following salient features:

- Privileged Access: Domain 0, also called Dom0 has access to communicate with the

hardware and other Guest VMs.

- Small Base Code: The Xen hypervisor layer is thin, and has a tiny code, restricting the

areas for attacks.

- OS Separation: The Xen hypervisor is separated from the physical hardware. There is no

way to attack the actual OS from the hypervisor.

Xen allows paravirtualization. Paravirtualization allows the guest OS to communicate

with the hypervisor in order to improve the memory, CPU, I/O and other resources. Since a VM

is aware of run in a virtual world, varieties of tasks are achieved in accordance with the

hypervisor. The overall maximum performance is achieved since all the VMs are isolated from

each other. The task given to each VM is processed during the normal operation without any OS

overhead.

Xen version 4.0 was used for live VM migration. Xen 4.0 is the fastest and most secure

virtualization software today [63]. The delivery of Xen 4.0 with state of the art of features and

hardware support highlight the strength and commitment of the open source Xen.org community

[63].

21

4.1.3 Working of Xen

The Xen hypervisor is an abstraction layer of software located on the hardware below any

OS. The hypervisor plays an important role in the execution of a VM. CPU Scheduling, memory

allocation, and disk sharing are all controlled by the hypervisor. Xen consists of two domains,

Domain 0 and Domain U [64]. Domain 0, known as Dom0, is modified Linux kernel. Dom0 is

the only VM having access to the physical I/O resources. Dom0 also communicates with the

other VM (Domain U, known as Dom U) running on the physical host. Every machine running

Xen should have Dom0 running before any other VM starts running. Dom0 has two drivers; the

Network Backend Driver and the Block Backend Driver. The Network Backend Driver

processes all the requests coming from the Dom U. The Block Backend Driver checks with the

local storage disk to process the read or write request of the other VMs.

Domain U paravirtualized guests do not have direct access to the OS and are often

referred as having unprivileged access. All paravirtualized guests are known as DomU PV Guest.

The Dom U PV Guests are aware of having no access to the physical hardware and are running

in a virtual world. Fully virtualized VMs running on Xen are known as Dom U HVM Guests. A

Domain U PV Guest consists of two drivers; PV Network Driver and PV Block Driver. PV

Network Driver is not located within a Domain U HVM Guest and instead requires another

special daemon Qemu-dm which is initialized for each VM in Dom0. This supports access to the

disk. Any request made by the Dom U PV Guests must communicate with the Xen hypervisor

via Dom 0. There is a direct link between Dom0 and Dom U PV Guests known as an event

channel running through the Xen hypervisor.

22

4.2 Requirements for Live VM Migration

To perform live VM migration, there are certain requirements for live VM migration. The

following precautions should be taken for a successful VM migration:

- The two hosts should be configured with the same version of Xen

- Both hosts should be in the same layer 2 network and IP subnet.

- The disk image and the configuration file should be stored on the shared storage. Both

hosts should have access to it.

- The processor on which Xen is installed should be the same. It can cause problems for

migration if the hardware is different. Migration works for the Intel Xeon processor with

four cores to an Intel Core2Duo processor with two cores. If the hardware is entirely

different, migration is not supported.

- The Xen configuration for the Xen daemon is stored in the file “xend-conFiguresxp”. It

can be found at /etc.xen/xend-conFiguresxp. This file needs to be modified on both

hosts.

The xend-config file consists of the following lines:

(xend-relocation-server no)

(xend-relocation-port 8002)

(xend-relocation-address “ ”)

(xend-relocation-hosts-allow “ ”)

The above lines need to be modified to

(xend-relocation-server yes)

(xend-relocation-port 8002)

(xend-relocation-address “ ”)

(xend-relocation-hosts-allow “ ”)

23

The xend-relocation-server is set to“yes” so that the receiving host can accept the

relocated guests. The second line specifies that xend should listen on the default port 8002. The

third line specifies which IP address the Xen daemon should listen on for migration requests. If

this field is left blank, then it allows listening on all addresses and interfaces. The last line limits

the number of hosts to contact the server for migration. Once the above requirements are done,

live VM migration should be done easily. The different shared storages available are iSCSI,

NFS, ATA over Ethernet (AoE) [65]. We have used NFS (Network File System) as the shared

storage.

24

Chapter 5

Test Bed and Results

Goals and Limitations:

The research focuses on the following main goals:

Tracing the session establishment details

Running any application that leads to data flow in clear text

Loss and misuse of essential credentials

5.1 Hardware

Two Dell Optiplex 780 (x86_64) servers were used for installing openSUSE OS. The processor

used was Intel Core 2 quad cpu q4800@2.66GHz consisting of 4GB Memory. A Cisco Layer 3

Switch 3550 was used for routing between the two host machines.

5.2 Software

5.2.1 openSUSE

To perform live VM migration, a Xen hypervisor was chosen. Xen was installed on

openSUSE 11.3. Novell sponsors Linux based OS openSUSE which includes a command line

interface and a graphical user interface (GUI). It offers various different GUI as such as KDE,

SC, and GNOME. It has the updated Linux kernel 2.6.34. Installation of openSUSE is simple

and it is user friendly [64]. The setup for Xen is simple and creating VMs is fairly easy. [61]

5.3 Test Bed

The physical hosts A and B run Xen hypervisor. A virtual machine VM 1 was created on

host A. The hosts A and B were connected to layer 3 switch in the same vlan 1 as shown in

Figure 6. A Dynamic Host Configuration Protocol (DHCP) was configured on the switch to get

25

the ip addresses automatically. Connectivity between the hosts and the VM was checked with a

ping test. Traffic was sniffed using Wireshark which is a tool used to capture and analyze the

traffic going through the network. Usually Ethernet network is supported on wireshark. To see

the migration data, a Switch Port Analyzer (SPAN) session was configured on the switch. After

the VM was createda a small script was written executing a small known pattern continuously

15,000 times. The pattern was written to a text file, so the script was doing read and write at the

same time. A continuous ping from the host A to VM 1 was started.

Figure 6 Test bed

To start with the migration, host A sent a SYN (synchronize) packet request to initiate a

connection with host B. Host B replied with an ACK (acknowledge) to host A‟s SYN packet

agreeing, upon building a connection and sends its own SYN packet (both messages are

combined to form a SYN + ACK). Host A then sends an ACK to host B‟s SYN thereby agreeing

to establish the connection. Figure 7 shows the pictorial representation of the three way

26

handshake between host A and host B. To perform the capture, wireshark was turned on using a

laptop connected to the switch in the same vlan. The script was executed and at the same time

VM1 was migrated with the command “xm migrate – l 172.16.0.3”. The migration was

completed. The VM1 was now residing on host B and the script was still being executed on the

host B. The script continued on the host B from the point it stopped on the host A. Even though

the VM migrated, connectivity to the VM was not lost and the service was not disrupted. This is

shown by the continuation of the script and no ping packets being lost.

Figure 7 Three-way handshake between host A and host B

5.4 Results

For testing purposes, only the script was running and no other application was running.

This proves that, during the live VM migration, the service will not be lost or it will be lost for a

minimal time. In the wirsehark capture, the ascii value of the string being printed in the script

was seen. It was easy for an attacker to decode the ascii value and see the data being transferred,

hence data is not secure during live VM migration. If any other applications are running or any

important text files containing confidential data are open, then that data is not secure. The below

27

Figure 8 show the wireshark capture where the three way handshake is established between the

source and destination hosts.

Figure 8 Three-way handshake capture

All three packets (SYN, SYN+ACK, ACK) are clearly visible indicating that the

connection has been established between the hosts A and B. Figure 9 shows capture of the ascii

value for the respective string being printed in the script. The string being printed was a known

data pattern of xyxyxyxyxyyxxxxxxyyyyyyyzzzzzzzzz. In the wireshark captures, the ascii value

for the above pattern was seen. Hence it‟s quite easy for anybody to decrypt these values and

recover the data being transmitted. The ascii value for x is 78, y is 79 and z is 7a which is clearly

visible in the wireshark capture. The ascii value is highlighted in red in the wireshark capture.

28

Along with the data, we can also see the source port, and destination port being used for

migration. The destination could also be compromised if too many requests are being sent on the

same destination port from some other host and the migration can never complete.

Figure 9 Wireshark Capture showing the ascii values

The above test shows that live VM migration is not secure and is the biggest security

threat in virtualization. The continuous execution of the string during live VM migration shows

29

that the string was not interrupted when the VM migrated. The wireshark capture shows the

exact ascii value for the string being tested which makes it easy to sniff the data and create man-

in-the-middle attack. This simple experiment proves that data is in clear text and is open for

attacks.

5.5 Proposed Solution

Security measures to protect the data during live VM migration should be designed in

such a way that data integrity, and confidentiality is maintained. There is always a cost to be paid

in order to make a gain some extra benefit. Following are a few ways for a secure live VM which

migration could be implemented without losing data.

5.5.1 IPSec tunnel

One way of securing secure live VM migration is through building an IPSec tunnel. If

the live VM migration is done across the Internet Protocol Security (IPSec) tunnel, then the cost

to be paid would be the downtime of the VM. Since the migration would be done through the

tunnel, extra overhead and processing would be done and the downtime of the VM might

increase. This would cause disruption in the service for a longer duration, but the data would be

secure since it will be encrypted. IPSec is a protocol for securing the Internet Protocol (IP)

traffic. Authentication and encryption of each IP packet in the communication session is done

when passing through the tunnel. In [66], the author explains operation, authentication and

encryption techniques in detail. IPSec tunnel could be used in protecting the data flow at server-

to-server levels or from edge router-to-edge router. When live VM migration is performed

through the IPSec tunnel, data would be encrypted and difficult to trace. Figure 10 shows that an

IPSec tunnel has been created. Live VM migration is done through the tunnel to ensure data is

encrypted.

30

Figure 10 Live VM migration through IPSec tunnel

Figure 11 shows how data would be encrypted and consists of some random characters

which are not easy to decrypt. The red highlighted portion shows that data is encrypted.

Figure 11 Encrypted data after VM migration through IPSec tunnel

While building the tunnel, various parameters have to be considered like Network

Address Translation (NAT), ip route, and application level inspection. All these network

31

parameters add more headers and cause more delay in processing the packet at each stage. In a

network, there could be many devices which process the packet, add headers, and check the route

before forwarding the packet. During this checking, there could be a possibility of losing the

essential packets. If the core network has a low bandwidth, then there can be the following risks:

Loss of ACK packets: If the ACK is lost, the connection could be dropped and VM

migration is not initiated.

Loss of regular packets: There could be intermittent delay or loss of other essential

packets and VM migration can take more time to complete. Also there could be a

chance of the VM crashing and migration failinh. IPSec is an end-to-end protection

scheme at the Internet Layer which can give security by encrypting the packets but

can cause other problems which are not affordable in an IT organization.

5.5.2 Hypervisor Encryption

Another approach is to do encryption at the hypervisor level. If the Xen hypervisor does

the encryption and sends the data during live VM migration, chances of hacking decrease

considerably. Strong encryption keys should be used. Decryption should be done by the Xen

hypervisor itself to maintain the data integrity and confidentiality. If encryption is done at the

hypervisor level, there would be less overhead, less downtime, less migration time. There would

not be a requirement for application inspection through the tunnel. If a host builds a SSH

connection or any secure connection, encryption would be done by the hypervisor. The

hypervisor should maintain the encryption and decryption keys to ensure that the data is secure.

32

Chapter 6

Conclusion and Future Work

6.1 Conclusion

This research addresses the security concerns in live VM migration and proposes

mitigation techniques. With the advent of virtualization and its new features, there are many

security holes. This research conducts a simple experiment to show how data is visible during

live VM migration. If there is an application running on a VM and if that VM is live migrated to

another physical host, then the data being exchanged in that specific application is in clear text

and could be compromised using a tool wireshark. Using wireshark, a man-in-the-middle attack

can be created by sniffing the traffic flowing between the two hypervisors. To have a secure live

VM migration, migration should be done through an IPSec tunnel or using proper encryption

techniques with strong private and public keys.

6.2 Future Work

This thesis brings to attention the less secure feature of virtualization. Performing live

VM migration is beneficial in many ways if done securely. Live VM migration has a limitation

of migrating in the same subnet. To overcome this limitation, a virtual switch should be

configured to note VM‟s new location and routes the packets as per the new default gateway. To

overcome the security issue, IPSec keeps data secure but has the disadvantage of delaying the

migration. This is due to more downtime of the application on the VM, and extra processing of

the packets. One should concentrate on encryption at the hypervisor level. Embedded hypervisor

[67] is a type 1 hypervisor that supports multiple VMs. A very important feature on the

33

embedded hypervisor is secure encapsulation for any subsystem. Embedded hypervisor gives

less exposure to malicious hackers and protects the complete network from being attacked.

34

REFERNCES

35

REFERENCES

[1] Walter Tichy, Technology Review of Mainframe Computer Systems and their

Alternatives

[2] SAN Basics, A Technical White Paper from MetaStor Storage Solutions, 1999

[3] Kara Nance, Brian Hay, Matt Bishop, Virtual Machine Introspection Observation or

Interference, IEEE Computer and Society, 2008

[4] Server Virtualization: A Step Toward Cost Efficiency and Business Agility, Avanade

Perspective, 2009.

[5] G. Somasundaram, Alok Shrivastava, Information Storage and Management, EMC

Education Services, 2009.

[6] SAN Security: A Best Practices Guide. Incorporating SAN security into the enterprise

with the Brocade Secure Fabric OS, Brocade Communications Systems, Inc.

[7] Christopher Clark, Keir Fraser, Steven Hand, Jacob Gorm Hanseny , Eric July,

Limpach C., Pratt I., Wareld A., Live Migration of Virtual Machines

[8] Ma F., Liu F., Liu Z., “Live Virtual Machine Migration based on Improved Pre-copy

Approach,” Network Management Research Centre, Beijing Jiaotong University

Engineering Research Center of Network Management Technology for High Speed

Railway, Ministry of Education, 2010

[9] Rosenblum M., Garfinkel T., "Virtual Machine Monitor: Current Technology and Future

Trends", Published by the IEEE Computer Society, 2005

[10] Al-Rabayah O., Virtualization Concept and History

http://www.remoteitservices.com/content/virtualization-concept-and-history, 2010

[11] Wahlig E., hardware Based Virtualization Technologies

http://www.redhat.com/f/summitfiles/presentation/June2/ECO/Red%20Hat%20Summit%

20Breakout%20A%20Elsie.pdf, 2006

[12] Fleury E., "Virtualization (Wake up Neo, The Matrix got you)",

http://www.labri.fr/perso/fleury/courses/SS07/download/lectures/01-Virtualization.pdf,

2007.

[13] Rosenblum M., "The Reincarnation of Virtual Machines",

http://www.datatrend.com/trendseletter/Issue_09_Articles/ReincarnationoftheVirtualMac

hine.pdf, 2004

[14] Mijat R., Nightingale A., "Virtualization is Coming to a Platform Near You", White

Paper ARM, 2010

36

[15] Veners Bill. 2000, Inside the Java Virtual Machine

http://www.artima.com/insidejvm/ed2/jvm.html

[16] George K. Thiruvathakula, Konstantin Laufer, Konrad Hinsen, and Joe Kaylor,

"Virtualization for Computational Scientists". Computing in Science &

Engineering, Copublished by the IEEE CS and the AIP, 2010

[17] What is Xen hypervisor? , Xen, http://www.xen.org/files/Marketing/WhatisXen.pdf ,

2004

[18] IBM Virtualization, Virtualization on System z, 2009

[19] IBM Virtualization. Virtualization for Linux, 2009

[20] Adair, R.J., R.U. Bayles, L.W. Comeau, and R.J. Creasy,A Virtual Machine System for

the 360/40, Report 320-2007,May 1966, IBM Cambridge Scientific

Center: Cambridge,MA.

[21] Thomas Burger, The advantages of using Virtualization Technology in the Enterprise,

http://software.intel.com/en-us/articles/the-advantages-of-using-

virtualization-technology-in-the-enterprise/, Aug 2010.

[22] Bernard Golden, Clark Scheffy, Virtualization for dummies, Sun and AMD special

Edition, 2009

[23] Virtuozzo,http://www.swsoft.co.uk/, 2001

[24] OS Virtualization, http://www.parallels.com/products/pvc46/info/virtualization/, 2001

[25] Kirill Kolyshkin , Virtualization in Linux, 2006

[26] A. Mann. The pros and cons of virtualization. BTQ, 2007.

http://www.btquarterly.com/?mc=pros-cons-virtualization\&page=virt-view%research

[27] J. Kirch. Virtual machine security guidelines. The center for Internet Security,

September 2007.

http://www.cisecurity.org/tools2/vm/CIS_VM_Benchmark_v1.0.pdf.

[28] Oliver Garraux, Information Technology Fundamentals, How Virtualization Works and

its Effects on IT, Oct 2007.

[29] K. J. Higgins. VM‟s create potential risks. Technical report, darkreading, 2007.

http://www.darkreading.com/document.asp?doc_id=117908.

[30] Roberts Paul, Secure your enterprise data. InfoWorld IT Strategy Guides at

http://www.infoworld.com/ad/sponsored_resources.html, 2007.

37

[31] Citrix , The End of Application Deployment: Virtualized Applications Streamline, Secure

and Manage Your Business. www.citrix.com. 2008

[32] Gruman, Galen ABC: An Introduction to Mobile Security – CIO, March 08, 2007 From:

http://www.cio.com

[33] Dell Virtualize at the Speed of Your Business, 2009.

[34] Doug Hyde, A Survey On the Security of Virtual Machines, April 2009.

[35] Porter, M. E., Competitive strategy: Techniques for analyzing industries and competitors

New York : The Free Press, 1980

[36] Soror, A.A. Aboulnaga, A. Salem, K., Database Virtualization: A New Frontier for

Database Tuning and Physical Design. Data Engineering Workshop,

2007 IEEE 23rd International Conference (April, 2007)

[37] Jim Carr, Two vulnerabilities found in VMware virtualization products,

http://www.scmagazineus.com, February 2008.

[38] A. Ganguly, A. Agrawal. P.O. boykin, and R. Figueiredo, “WOW: Self-Organizaing

Wide Area Overlay Networks of Virtual Work-stations;” Proceedings of the 15th IEEE

International Symposium on High Perfomance Distributed Computing (HPDC), pages

30-41, 2006.

[39] P.Ruth, J.Rhee, D. Xu, R. Kennell and S. Goasguen, “Autonomic Live Adaption of

Virtual COmputational Environment in a Multi-Domain Infrastructure.” IEEE

International COnference on Autonomic Computing (ICAC'06), 2006.

[40] NELSON, M., LIM, B.-H., AND HUTCHINS, G., “Fast transparent migration for virtual

machines.” In Usenix, Anaheim, CA (2005), pp. 25–25.

[41] Bradford, R., Kotsovinos, E., Feldmann, A., And Schi Oberg, H, “Live wide-area

migration of virtual machines including local persistent state,” Proceedings of the

International Conference on Virtual Execution Environments (2007), pp. 169–179.

[42] C. P. Sapuntzakis, R. Chandra, B. Pfaff, J. Chow, M. S. Lam, and M.Rosenblum.

“Optimizing the migration of virtual computers;” Proceedings of the 5th Symposium on

Operating Systems Design and Implementation (OSDI-02), December 2002.

[43] M. Kozuch and M. Satyanarayanan. Internet suspend/ resume. Proceedings of the IEEE

Workshop on Mobile Computing Systems and Applications, 2002.

38

[44] Whitaker, A., Richard S. Cox, Marianne Shaw, and Steven D. Gribble. “ Constructing

services with interposable virtual hardware,” Proceedings of the First Symposium on

Networked Systems Design and Implementation (NSDI '04), 2004.

[45] S. Osman, D. Subhraveti, G. Su, and J. Nieh. “The design and implementation of zap: A

system for migrating computing environments,” Proc. 5th USENIX Symposium on

Operating Systems Design and Implementation (OSDI-02), pages 361-376, December

2002.

[46] VMWare, Inc. VMWare VirtualCenter Version 1.2 User's Manual. 2004.

[47] Michael L. Powell and Barton P. Miller., “Process migration in DEMOS/MP,”

Proceedings of the ninth ACM Symposium on Operating System Principles, pages 110-

119. ACM Press, 1983.

[48] Marvin M. Theimer, Keith A. Lantz, and David R. Cheriton, “Preemptable remote

execution facilities for the V-system.” Proceedings of the tenth ACM Symposium on

Operating System Principles, pages 2 - 12. ACM Press, 1985.

[49] Eric Jul, Henry Levy, Norman Hutchinson, and Andrew Black, “Fine-grained mobility in

the emerald system. ACM Trans,” Comput. Syst., 6(1):109-133, 1988.

[50] Douglis, F., and John K. Ousterhout, “Transparent process migration: Design alternatives

and the Sprite implementation,” Software - Practice and Experience, 21(8):757-785,

1991.

[51] A. Barak and O. La'adan. “The MOSIX multicomputer operating system for high

performance cluster computing,” Journal of Future Generation Computer Systems, 13(4-

5):361-372, March 1998.

[52] D. Milojicic, F. Douglis, Y. Paindaveine, R. Wheeler, and S. Zhou. Process migration.

ACM Computing Surveys, 32(3):241-299, 2000.

[53] Douglis, F., “Transparent process migration in the Sprite operating system,‟ Tech. rep.,

University of California at Berkeley, Berkeley, CA, USA, 1990.

[54] KERRIGHED. http://www.kerrighed.org, 1998.

[55] Thain, D., Tannenbaum, T., And Livny, M., “Distributed computing in practice: the

condor experience,” Concurr. Comput. : Pract. Exper. 17 (2005), 323–356.

[56] Milojicic, D., Douglis, F., Paindaveine, Y., Wheeler, R., And Zhou, S., “Process

migration survey,” ACM Computing Surveys 32(3) (Sep. 2000), 241–299.

[57] Venkatesha S., Sadhu S., Kintali S.Department of Computer Science, University of

California, Santa Barbara, “Survey of Virtual Machine Migration Techniques,” 2009.

39

[58] Nelson M., Beng-Hong L., and Hutchins G., “Fast Transparent Migration for Virtual

Machines,” VMware, Inc. In 2005 USENIX Annual Technical Conference, 2005.

[59] Marvin M. Theimer, Keith A. Lantz, and David R. Cheriton, “Preemptable remote

execution facilities for the V-system,” Proceedings of the tenth ACM Symposium on

Operating System Principles, pages 2 - 12. ACM Press, 1985.

[60] openSUSE 11.1 Start-up,

http://www.novell.com/documentation/opensuse111/pdfdoc/opensuse111_startup/opensu

se111_startup.pdf, March 2009

[61] SUSE Linux Enterprise Server 11SP1, Virtualization with

Xen,http://www.novell.com/documentation/sles11/pdfdoc/book_xen/book_xen.pdf , Aug

2010

[62] Spector ,S., and Xen.org Community, Why Xen?, 2004.

[63] Xen 4.0 Data Sheet, http://www.xen.org/files/Xen_4_0_Datasheet.pdf, 2004.

[64] XEN, How does XEN Work? , http://www.XEN.org, December 2004.

[65] Brantley C., Coraid Inc, ATA Over Ethernet, 2005.

[66] IPSec Tunnel Creation, SANS InfoSec Reading Room, 2003

[67] Robert D., LynuxWorks San Jose, CA Virtualization and hypervisors aid embedded

design

http://www2.electronicproducts.com/Virtualization_and_hypervisors_aid_embedded_des

ign-article-FAJH05_LynuxWorks-Apr2008-html.aspx, 2008