Design and Implementation of the Recursive Virtual Address Space Model for Small Scale...

Design and Implementation of the Recursive Virtual Address Space Model

for Small Scale Multiprocessor Systems

Diploma Thesis

Marcus VölpSupervisor: Volkmar Uhlig

Universität Karlsruhe

September 2002 Marcus Völp Universität Karlsruhe

2

recursive virtual address space model

microkernel

Policy S1 Policy S2

Policy S0

Policy S1a Policy S1bPolicy S2a Policy S2b


3


map


4


unmap


5


grant


6

Policy S1 Policy S2

Policy S0

Policy S1a Policy S1bPolicy S2a Policy S2b


microkernel

Region Mapper

Pager

Pager Pager

Pager

Pager Pager

Pager


7


mapping databasekeep track of mappings

to implement unmap

No long interrupt latencies

No unbounded priority inversion

No helping

0

Hazelnut + Pistachio unmap with interrupts disabled


8



to implement unmap



No helping

0

unbounded priority inversion

Hazelnut + Pistachio (if interrupts enabled)

Fiasco (if helping disabled)


9



to implement unmap



No helping

0

Helping problematic for SMP- xcpu time donation- migration to helper cpu


10

Outline

Motivation Problem analysis

Unbouned priority inversion inherent to

pre-order tree traversal (if not helped out)

Unbounded priority inversion inherent tonon-post-order tree traversal

Mapping database Data Structures Locking Scheme Restart Point Tracking

Performance Conclusion


11

Problem Analysis

Pre-Order tree traversal b in b unmap ( b ) remove node c preemption a unmap ( a )

Priority: a > b

Solutions roll forward unmap (b) help b

keep node c

(last thread removes it)

a

D

b

c


12

Problem Analysis

Unbound priority inversion is inherent to non post-order tree traversal

General traversal algorithm: i child nodes before parent

k child nodes after parent

Post order traversal: k = 0 for all nodes

D

b

… …1 i 1 k

a


13

Conclusion from problem analysis

Post order tree traversal

Find leaf node to start with

Collapse tree from leaf to root of subtree


14

Mapping Database

Datastructure fast insertion (map)

find start-leafnode

post order traversal

Synchronization SMP safe

avoid Priority inversion

avoid Starvation

=> Restart Point Tracking


15

Data Structure

LL

Number of Pointers: 3 + depthFind first leaf: O (N)

a

b c

e

d

gf

h i k

0

1

2

3

Post order sorted doubly linked list

(page reference info+ overmapping)


16

Data Structure

Treea

b c

e

d

gf

h i k

(left child, right sibling)


17

Data Structure

LL Tree

Number of Pointers: 5Find first leaf: O (D)

a

b c

e

d

gf

h i k


18

Data Structure

LL O1

Number of Pointers: 6Find first leaf: O (1) best

O (D) worst

a

b c

e

d

gf

h i k


19

Data Structure

LL O1

Number of Pointers: 6Find first leaf: O (1) best

O (D) worst

a

b c

e

d

gf

h i k


20

Data Structure

Representation Number of Pointers

Find first leaf

LL 3 + depth O (N)

LL Tree 5 O (D)

LL O1 6 O (1) best

O (D)


21

Synchonization

Datastructure consistency Atomic modifiaction

No Priority Inversion

=> Scheduler Conscious Locks

Scalability Maximal possible parallelism

=> Frame granular locking


22

Synchronization

acquire Lock release Lock

unpreemptable

preemption pending? => yield

Roll forward to Preemption point

backoff code


23

Synchronization

Frame granular locking (the real world)

a

b c

d e

4MB

fg

i j

h

k l

1MB

…


24

Restart Point Tracking

(rw)

d(r)

a

b c1

(rw)

(rw)(rw)

d(r)

e(rw)

a

b c1

(rw)

(rw)

Why Restart Point Tracking ?

Required Properties Low overhead on

map / unmap

Guarentee fwd progress to avoid starvation


25



Required Properties Low overhead on

map / unmap

Guarentee fwd progress to avoid starvation

(rw)

d(r)

a

b c1

(rw)

(rw)

a

b c1

(rw)

(rw)


26


Update restart point Find restart point information Update restart point information

On Wakeup Find node to proceed ! Guarantee progress of find !


27


4 2 33 2

Token Based Preempted Thread List


28



4 2 3

1 2


29



4 2 3

1 2


30


Rootnode overrun detection

4 2 3

1 2

check overrun flag in mdb node


31

Summary Mapping Database

Post order traversal Data Structures

LL LL-Tree LL-O1

Synchronization Scheduler Conscious Locks Roll forward to preemption point

Token based preempted thread list Guarantee progress of preempted operations Root overrun detection

size search time for first leaf


32

Performance Evaluation2683

2616

200

2693

264

2860

333

2739

2470

2693

0500

1000

1500

2000

25003000

Ex

ec

uti

on

tim

e

(cy

cle

s)

Pis

tach

io

LL

pu

re M

DB

LL

-Tre

e

pu

re M

DB

LL

-O1

pu

re M

DB

Fia

sco

LL

-op

t

LL

-O1

-op

t

Page fault resolution

Best case performance 5% Quantile

Comparable performance

for map systemcall

(prepare IPC, map page,

resume faulting instruction)

Repre-sentation

# pointer Search time leaf node

Mapping node size

LL 3 + depth O(N) 32 byte

(24 byte)

LL-Tree 5 O(D) 32 byte

LL-O1 6 O(1) best

O(D) worst

40 byte

(32 byte)

on PIII 500 MHz, 256 KB L2 Cache


33

Performance Evaluation

Unmap wide mapping tree

Performance of unmap systemcall

Performance unmapping a wide subtree (4KB pages)

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

11000

12000

13000

14000

15000

16000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Width

Exe

cuti

on

tim

e o

f u

nm

ap (

in

cycl

es)

Pistachio LL LL-Tree LL-O1Fiasco LL-opt LL-O1-opt

…

1 2 n


34

Overhead to Interrupt Latencies

Map: insert mapping update PTE

Unmap: 3 Preemption Points

1 step:search start point

remove mappingupdate PTE

finish unmap

2709 30

79

2524

6613

2948

2077

6522

2934

0

1000

2000

3000

4000

5000

6000

7000

time

(cyc

les)

1

Worst case overhead added to interrupt handling

Map 4K

Map 4M

Unmap 4K: start->PP(ff l)

Unmap 4K: backoff->PP(tr-up)

Unmap 4K: backoff->end

Unmap 4M: start->PP(ff l)

Unmap 4M: backoff->PP(tr-up)

Unmap 4M: backoff->end

Mapping Database responsible for 13ms interrupt latency(largest distance between preemption points)


35

Conclusion

Search time optimization for leaf node does not pay

Good preemptability without helping

No unbounded priority inversion No need for prevention !!!

approx. 12.2% - 26% loss of performance for unmap

Open: explaination of Fiasco performance

bounded tree-size N for real-time unmap


36

End of slide show, click to exit


37

Page Reference Information / Overmapping

[rw]

[r]

[r] [r] [r]

[rw]

Increase rights omitting prior flush: • same source address space• same virtual source address• same virtual destination address• same physical page

parent link


38

Page Reference Information /Overmapping

0

a

cb

d

pa [rw]

pb [rw]

pd [w]

pc [w]

<R>

<W>

0

a

cb

d

pa [rw]

pb [r]

pd [r]

0

a

cb

d

pa [rw]

pb [r]

{R}

{RW}


39

THE REAL END


40

Motivation

Problems in Existing Implementations Long Interrupt Latencies

Orangepip, Hazelnut

Unbounded Priority Inversion Calypso, Hazelnut (two phase lock)

Complex Helping or Timeslice Donation Fiasco, L4 Alpha


41

Guaranteeing Forward Progress

Why Restart Point Tracking ?a

b c

e

d

gf

h i k


42



a

b c1

(rw)

(rw)(rw)

d(r)

e(rw)

a

b c1

(rw)

(rw)(rw)

d(r)

a

b c1

(rw)

(rw)


43




44




45




46


4 2 33 2

Restart Point Tracking Token Based Preempted Thread List


47

Experimental Results

Performance: Map 4KB to handle Pagefault

Unmap Deep + Wide subtree

Interrupt Latencies Time OPs are rolled forward

Intel Pentium III 500 MHz , 512 KB L2, 2x 16 KB L1, 64 / 32 entry 4KB TLB (data / code), 8 / 2 entry 4MB TLB


48


2683

2616

200

2693

264

2860

333

2739

2470

2693

0500

1000

1500

2000

25003000

Ex

ec

uti

on

tim

e

(cy

cle

s)

Pis

tach

io

LL

pu

re M

DB

LL

-Tre

e

pu

re M

DB

LL

-O1

pu

re M

DB

Fia

sco

LL

-op

t

LL

-O1

-op

t

Page fault resolution

Best case performance 5% Quantile


49


Performance unmapping a deep subtree (4KB pages)

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

11000

12000

13000

14000

15000

16000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Depth

Exe

cuti

on

tim

e o

f u

nm

ap (

in

cycl

es)

Pistachio LL LL-Tree LL-O1

LL-opt LL-O1-opt

Performance unmapping a wide subtree (4KB pages)

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

11000

12000

13000

14000

15000

16000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Width

Exe

cuti

on

tim

e o

f u

nm

ap (

in

cycl

es)

Pistachio LL LL-Tree LL-O1Fiasco LL-opt LL-O1-opt


50


2709 30

79

2524

6613

2948

2077

6522

2934

0

1000

2000

3000

4000

5000

6000

7000

tim

e (c

ycle

s)

1


Map 4K

Map 4M








51


Worst case interrupt latency

find first leaf


52


2709 30

79

2524

6613

2948

2077

6522

2934

0

1000

2000

3000

4000

5000

6000

7000

tim

e (c

ycle

s)

1


Map 4K

Map 4M








53

Interrim Measurements

0

100

200

300

400

500

600

Performance (cycles)

LL-Min LL-Tree-Min

LL-O1-Min

Representation

Interrim Measures

set entry

contents

next map link

insert into ds

alloc mdb node

grant

overmap check


54

Interrim Measurements

0

100

200

300

400

500

600

Performance (cycles)

LL -Min LL-Tree-Min

LL-O1-Min

Representation

Interrim Measures unmap

root overrun

traverse up

check end

remove node

flush entry

ref info

flip lock

traverse down

find first leaf

preparation

Design and Implementation of the Recursive Virtual Address Space Model for Small Scale...

Documents

Transcript of Design and Implementation of the Recursive Virtual Address Space Model for Small Scale...