Kernel Synchronization

37
Kernel Synchronization Examples From the Linux Kernel Michael E. Locasto

description

Kernel Synchronization. Examples From the Linux Kernel Michael E. Locasto. kernel control flow is a complicated, asynchronous interleaving. Big Picture: How can the kernel correctly service requests?. Main Ideas / Concepts. Atomic operations in x86 - PowerPoint PPT Presentation

Transcript of Kernel Synchronization

Page 1: Kernel Synchronization

Kernel Synchronization

Examples From the Linux Kernel

Michael E. Locasto

Page 2: Kernel Synchronization

BIG PICTURE: HOW CAN THE KERNEL CORRECTLY SERVICE REQUESTS?

kernel control flow is a complicated, asynchronous interleaving

Page 3: Kernel Synchronization

Main Ideas / Concepts

Atomic operations in x86

Kernel locking / synchronization primitives

Kernel preemption

Read-Copy-Update

The “big kernel lock”

Page 4: Kernel Synchronization

Kernel Preemption

Kernel preemption is a concept in which the kernel can preempt other running kernel control paths (be they on behalf of a user or another kernel thread)

Acquiring a spinlock automatically disables kernel preemption (as we will see in the code)

Page 5: Kernel Synchronization

Synchronization Primitives

Atomic operationsDisable interrupts (cli/sti modify IF of eflags)Lock memory bus (x86 lock prefix)Spin locksSemaphoresSequence LocksRead-copy-update (RCU) (lock free)

Page 6: Kernel Synchronization

Barriers

Barriers are serializing operations; they “gather” and make operations sequential.

Memory barrier:x86 in/out on I/O portsx86 lock prefixx86 writes to CReg, SReg/eflags, DReg

x86 instr meaning

lfence read barrier

sfence write barrier

mfence r/w barrier

Page 7: Kernel Synchronization

Barrier Implementation

Page 8: Kernel Synchronization

Motivating Example: Using Semaphores in the Kernel

what are: down_read, up_read, and mmap_sem

Page 9: Kernel Synchronization

START WITH THE DATA STRUCTURE: MM->MMAP_SEM

Let’s start with the data structure and see where that leads…

Page 10: Kernel Synchronization

current->mm->mmap_sem

struct mm_struct: include/linux/mm_types.h

Page 11: Kernel Synchronization

PRIMITIVE ONE: ATOMIC TYPE AND OPERATIONS

Page 12: Kernel Synchronization

On x86, these operations are atomic

simple asm instructions that involve 0 or 1 aligned memory access

read-modify-update in 1 clock cycle (e.g., inc, dec)

anything prefixed by the IA-32 ‘lock’ prefix

Page 13: Kernel Synchronization

atomic_t: include/linux/types.h

Page 14: Kernel Synchronization

Example: Reference Counters

Refcounts: atomic_t; associated with resources, but keeps count of kernel control paths accessing the resource

Page 15: Kernel Synchronization

PRIMITIVE TWO: SPINLOCKS

Page 16: Kernel Synchronization

/include/linux/spinlock_types.h

typedef struct spinlock{struct raw_spinlock rlock;

} spinlock_t;

typedef struct raw_spinlock{arch_spinlock_t raw_lock;

} raw_spinlock_t;

Page 17: Kernel Synchronization

arch/x86/include/asm/spinlock_types.h#L10

slock=1 (unlocked), slock=0 (locked)

Page 18: Kernel Synchronization

spinlock API (partial)

/include/linux/spinlock.h

/kernel/spinlock.c

Page 19: Kernel Synchronization

include/linux/spinlock_api_smp.h

Page 20: Kernel Synchronization
Page 21: Kernel Synchronization
Page 22: Kernel Synchronization

Linux Tracks Lock Dependencies @ Runtime

Page 23: Kernel Synchronization

PRIMITIVE THREE: SEMAPHORESHere we mainly consider Read/Write Semaphores

Page 24: Kernel Synchronization

Important Caveats about Kernel Semaphores

Semaphores are *not* like spinlocks in the sense that the invoking process is put to sleep rather than busy waits.

As a result, kernel semaphores should only be used by functions that can safely sleep (i.e., not interrupt handlers)

Page 25: Kernel Synchronization
Page 26: Kernel Synchronization

might_sleep() leads (eventually) to:

Page 27: Kernel Synchronization
Page 28: Kernel Synchronization
Page 29: Kernel Synchronization

rwsem_wake

Page 30: Kernel Synchronization

__rwsem_do_wake

On our way out, allow a writer at the front of the waiting queue to proceed.

Then allow unlimited numbers of readers to access the critical region.

Page 31: Kernel Synchronization

Advanced Techniques

Sequence LocksA solution to the multiple

readers-writer problem in that a writer is permitted to advance even if readers are in the critical section.

Readers must check both an entry and exit flag to see if data has been modified underneath them.

Read-Copy-Update (RCU)Designed to protect data structures

accessed by multiple CPUs; allows many readers and writers.

Basic idea is simple (and in the name). Readers access data structure via a pointer; writers initially act as readers & create a copy to modify. “Writing” is just a matter of updating the pointer.

Page 32: Kernel Synchronization

RCUOnly for kernel control paths; disables preemption.

Used to protect data structures accessed through a pointer

by adding a layer of indirection, we can reduce wholesale writes/updates to a single atomic write/update

Heavy restrictions: RCU tasks cannot sleep

readers do little work

writers act as readers, make a copy, then update copy. Finally, they rewrite the pointer.

cleanup is correspondingly complicated.

Page 33: Kernel Synchronization

RCU EXAMPLE: GETPPID(2)http://lxr.linux.no/#linux+v2.6.35.14/kernel/timer.c#L1354

Page 34: Kernel Synchronization
Page 35: Kernel Synchronization
Page 36: Kernel Synchronization

EXERCISE: TIME PERFORMANCE COST OF SYNCHRONIZATION

Does synchronization impose a significant cost?(test at user level)

Page 37: Kernel Synchronization

CODE: AUTOMATICALLY DRAWING RESOURCE GRAPHS