UNIVERSITI PUTRA MALAYSIA

AMIRREZA ZARRABI

FK 2012 90

AUTOMATIC GENERIC PROCESS MIGRATION SYSTEM IN LINUX

AUTOMATIC GENERIC PROCESS MIGRATION SYSTEM IN LINUX

By

AMIRREZA ZARRABI

Thesis Submitted to the School of Graduate Studies, Universiti Putra Malaysia

in Ful�lment of the Requirements for the Degree of Master of Science

December 2012

© COPYRIG

HT UPM

DEDICATIONS

This thesis is dedicated to my lovely parents who gave me bestsupports by their hearts.

ii

© COPYRIG

HT UPM

Abstract of thesis presented to the Senate of Universiti Putra Malaysia in ful�lment ofthe requirement for the degree of Master of Science

AUTOMATIC GENERIC PROCESS MIGRATION SYSTEM IN LINUX

By

AMIRREZA ZARRABI

December 2012

Chairman : Khairulmizam Samsudin, PhD

Faculty : Engineering

Process migration refers to the act of transferring a process in the middle of its execution

from one machine to another in a network. It introduces an intuitively appealing approach

which facilitates dynamic load distribution, fault resilience, easy system administration,

and data access locality. Process migration has not reached widespread use in Linux

Operating System (OS) which was originally designed to run standalone. However, several

packages exist which extend Linux with migration functionality. They generally depend

on speci�c features in kernel or target some well behaved applications. These packages

are speci�cally tailored for speci�c goals, result in prede�ned operation which contradicts

with the concept of generic tools.

In this work, a process migration system architecture for Linux is proposed. It does not

impose any requirement on the existence of speci�c infrastructure in the kernel. It is a

multilayer architecture to con�ne every functionality independent section of the system

in separate layer. This architecture is capable of supporting diverse applications due to

iii

© COPYRIG

HT UPM

dynamic structure that can be modi�ed according to demands. A �exible interface to the

underlying checkpoint/restart subsystem is designed which permits users to specify the

migration mechanism according to process constraints.

A migration algorithm is designed which attempts to exploit the unique features of the

basic migration algorithms to form a generic algorithm. Furthermore, to enhance the

process migration system performance a number of optimizations are enforced in each

layer, including: bu�er aggregation and interleaving, operation parallelism, and early

event request. The entire system is absolutely implemented in multiple kernel modules

which results in ease of use and eliminates the burden of complex system administration.

The proposed architecture is evaluated qualitatively and compared with the proprietary

products to reveal its signi�cance. However, the quantitative experiments are conducted

to assess the implementation performance. The early event request is deployed to speedup

the migration event by reducing the process migration latency time by almost 27% to

77%. Using the migration event concurrency, the freeze time for the migration events

would practically be similar irrespective to the number of processes involved in the event.

Exploiting the concurrency in the proposed migration system and introducing the notion

of parallelism for transferring of the shared resources, the state relocation time is reduced

by almost 20% for nine processes sharing 1MiB of their address space.

iv

© COPYRIG

HT UPM

Abstrak thesis yang dikemukakan kepada Senat Universiti Putra Malaysia sebagaimemenuhi keperluan untuk ijazah Master Sains

AUTOMATIC GENERIK PROSES MIGRASI SISTEM DALAM LINUX

Oleh

AMIRREZA ZARRABI

Disember 2012

Pengerusi : Khairulmizam Samsudin, PhD

Fakulti : Kejuruteraan

Proses migrasi merujuk kepada pendekatan memindahkan proses daripada satu mesin

kepada mesin yang lain. Perpindahan sebegini berlaku semasa proses sedang dijalankan.

Kaedah intuitif ini memudahkan pengagihan beban secara dinamik, daya tahan bersalah,

sistem pentadbiran mudah dan lokaliti akses data. Tanpa mengambil kira matlamat ini

dan usaha penyelidikan yang dijalankan, proses migrasi masih tidak mencapai penggunaan

yang meluas. Satu penjelasan adalah kerumitan menambah ciri migrasi proses yang telus

kepada sistem operasi (OS) yang pada asalnya direka untuk operasi bersendirian. Walau

bagaimana pun, pembangunan sistem operasi baru dengan keupayaan proses migrasi dari

peringkat awal merupakan pendekatan yang tidak realistik.

Dalam kerja ini, kami mencadangkan proses migrasi novel seni bina untuk Linux OS,

yang tidak memerlukan sebarang keperluan kewujudan kemudahan baru dalam kernel dan

boleh melakukan migrasi proses multithreaded tanpa mengira tingkah laku secara telus.

la adalah satu struktur berlapis untuk menghadkan fungsi setiap lapis secara tersendiri

v

© COPYRIG

HT UPM

dalam lapisan yang berasingan. Senibina ini mampu menyokong pelbagai jenis aplikasi

disebabkan antaramuka ruang pengguna yang generik dan strukturk dinamik yang da-

pat diubah berdasarkan permintaan. Selain itu, untuk meningkatkan keupayaan proses

migrasi, beberapa pengoptimuman dikuatkuasakan pada setiap lapisan, termasuk: pen-

gagregatan penampan dan interleaving, penselarian operasi dan permintaan acara awal.

Satu algoritma migrasi telah direka bagi mengeksploitasi ciri-ciri unik algoritma migrasi

asas bagi membentuk algoritma yang generik. Keseluruhan sistem telah dibangunkan

dalam beberapa modul kernel yang memberikan kemudahan penggunaan dan mengha-

puskan beban pentadbiran sistem yang kompleks.

Arkitektur yang dicadangkan telah dinilai secara kualitatif dan dibandingkan dengan

produk-produk proprietari bagi mendedahkan kesigni�kanannya. Walau bagaimanapun,

eksperimen kuantitatif telah dijalankan juga bagi menilai prestasi pelaksanaan. Hasil awal

yang diperoleh daripada ujian menunjukkan kesan saiz ruang alamat proses terhadap masa

pendaman migarasi proses. Oleh itu sebarang algoritma migrasi yang bertujuan untuk

mengurangkan overhed pemindahan ruang alamat akan mengurangkan masa pendaman

dengan banyak. Dalam sistem migrasi yang dicadangkan, permintaan peristiwa awal di-

gunakan bagi mempercepatkan peristiwa migrasi dengan mengurangkan masa pendaman

migrasi proses antara 27% hingga 77%. Hasil kajian juga menunjukkan dengan menggu-

nakan keserempakan peristiwa migrasi, masa beku untuk peristiwa migrasi adalah sama

tanpa mengira bilangan proses yang terlibat dalam peristiwa. Dengan mengeksploitasi

keserempakan dalam sistem migrasi yang dicadangkan dan memperkenalkan tanggapan

keselarian pemindahan sumber-sumber yang dikongsi, tempoh penempatan semula diku-

vi

© COPYRIG

HT UPM

rangkan hampir 20% bagi sembilan perkongsian proses 1MiB ruang alamat.

vii

© COPYRIG

HT UPM

ACKNOWLEDGMENTS

From the formative stages to the �nal draft of my thesis, I owe an immense debt of grat-

itude to my supervisor, Dr. Khairulmizam Samsudin, Senior Lecturer of the Department

of Computer and Communication Systems Engineering, Faculty of Engineering, Univer-

siti Putra Malaysia. I also would like to say thanks to Dr. Wan Azizun Wan Adnan, the

Senior Lecturer of Department of Computer and Communication Systems Engineering,

my supervisory committee member who supports with her valuable advises.

I wish to extend my warmest thanks to all those who have helped me with my work in

the Department of Computer and Communication Systems Engineering. I am indebted

for the help and unconditional love provided to me by my lovely parents. My special

gratitude is due to my dear friends who assisting me with this work.

viii

© COPYRIG

HT UPM

© COPYRIG

HT UPM

I certify that a Thesis Examination Committee has met on 28 December 2012 to conductthe �nal examination of Amirreza Zarrabi on his thesis entitled �Automatic GenericProcess Migration System in Linux� in accordance with the Universities and UniversityCollege Act 1971 and the Constitution of the Universiti Putra Malaysia [P.U.(A) 106] 15March 1998. The Committee recommends that the student be awarded the Master ofScience.

Members of the Thesis Examination Committee were as follows:

Syed Abd Rahman Al-Haddad bin Syed Mohamed, PhDAssociate ProfessorFaculty of EngineeringUniversiti Putra Malaysia(Chairman)

Sharifah Mumtazah Syed Ahmad Abdul Rahman, PhDSenior LecturerFaculty of EngineeringUniversiti Putra Malaysia(Internal Examiner)

Abd. Rahman bin Ramli, PhDAssociate ProfessorFaculty of EngineeringUniversiti Putra Malaysia(Internal Examiner)

Por Lip Yee @ Por Khoon Sun, PhDSenior LecturerUniversity of MalayaMalaysia(External Examiner)

SEOW HENG FONG, PhDProfessor and Deputy DeanSchool of Graduate StudiesUniversiti Putra Malaysia

Date: 26 February 2013

ix

© COPYRIG

HT UPM

This thesis submitted to the Senate of Universiti Putra Malaysia and has been acceptedas ful�lment of the requirement for the degree of Master of Science. The members of theSupervisory Committee were as follows:

Khairulmizam Samsudin, PhDSenior LecturerFaculty of EngineeringUniversiti Putra Malaysia(Chairman)

Wan Azizun Wan Adnan, PhDSenior LecturerFaculty of EngineeringUniversiti Putra Malaysia(Member)

BUJANG BIN KIM HUAT, PhDProfessor and DeanSchool of Graduate StudiesUniversiti Putra Malaysia

Date: 5 April 2013

x

© COPYRIG

HT UPM

DECLARATION

I declare that the thesis is my original work except for quotations and citations whichhave been duly acknowledged. I also declare that it has not been previously, and is notconcurrently, submitted for any other degree at Universiti Putra Malaysia or at any otherinstitution.

© COPYRIG

HT UPM

© COPYRIG

HT UPM

TABLE OF CONTENTS

Page

DEDICATIONS ii

ABSTRACT iii

ABSTRAK v

ACKNOWLEDGMENTS viii

APPROVAL ix

DECLARATION xi

LIST OF TABLES xiv

LIST OF FIGURES xv

LIST OF ABBREVIATIONS xvii

CHAPTER

1 INTRODUCTION 1

1.1 Overview 1

1.2 Problem Statement 4

1.3 Aims and Objectives 5

1.4 Scope and Relevance 7

1.5 Thesis Outline 8

2 LITERATURE REVIEW 9

2.1 Introduction 9

2.2 System with Multiple Processors 9

2.3 Operating System Processes 11

2.4 Object File Format 12

2.4.1 Executable and Linking Format 13

2.4.2 Loading and Execution Cycle in Linux 15

2.5 Characteristics of Process Migration Systems 19

2.6 Process Migration Algorithms 20

2.6.1 Total Copy Algorithm 22

2.6.2 Pre-copy Algorithm 22

2.6.3 Lazy Copy Algorithm 23

2.6.4 Flushing Algorithm 24

2.6.5 Summary of Process Migration Algorithms 24

2.7 Process Migration Mechanism 25

2.8 Development of Process Migration Systems 26

xii

© COPYRIG

HT UPM

2.8.1 Kerrighed 31

2.8.2 OpenMOSIX 33

2.9 Summary 34

3 METHODOLOGY 37

3.1 Introduction 37

3.2 Generic Software Architecture 37

3.2.1 Checkpoint/Restart Subsystem 38

3.2.2 Migration Coordinator/Daemon 39

3.3 System Design 40

3.3.1 Kernel Level Infrastructure 40

3.3.2 User Level Tools 46

3.4 System Implementation 46

3.4.1 Linux Kernel Symbol Table 47

3.4.2 Proposed Migration Algorithm 48

3.4.3 Synchronization and System Consistency 50

3.4.4 Kernel Subsystem Speci�c Manager 54

3.4.5 Multithreaded Process Manager 55

3.4.6 Process Migration Synchronization 60

3.4.7 Memory Subsystem Module 62

3.4.8 Network Transfer Medium Module 69

3.4.9 User Space Interface 74

3.4.10 Multithreaded Process State Consistency 76

3.4.11 Process Migration Event Strategies 77

3.4.12 Error Handling 77

3.4.13 Transfer Protocol 79

3.5 Experimental Platform 80

3.6 Process Migration Performance 83

4 RESULTS AND DISCUSSIONS 85

4.1 Introduction 85

4.2 Analysis of Proposed Migration Algorithm 85

4.3 Analysis of Proposed Migration System Architecture 88

4.4 Performance Assessment 92

4.4.1 Migration Event Behavior 92

4.4.2 Process Migration Latency Time 95

4.4.3 Process Freeze Time 98

4.4.4 Process State Relocation Time 100

xiii

© COPYRIG

HT UPM

5 CONCLUSION AND RECOMMENDATIONS 103

5.1 Conclusion 103

5.2 Contribution 104

5.3 Future Work 105

REFERENCES 107

APPENDICES 114

BIODATA OF STUDENT 135

LIST OF PUBLICATIONS 137

xiv

© COPYRIG

HT UPM