Talk 160920 @ Cat System Workshop

Dynamically Hacking the Kernel with Containers

CAT System Workshop2016.09.20

Quey-Liang KaoNTHUCS

About Myself● PhD Student

– Adviser: Che-Rung Lee● Research topics

➢ HPC (numerical), Heterogeneous computing➢ High-end hardware virtulization (InfiniBand, GPGPU)➢ Container technology

About Myself - the other side● Archoholic

– Archwiki – AUR packager

● runc-git, openscap● kpatch

● GNOME

● BICIB– https://github.com/NonerKao/BICIB

What is this?

Hacking the Kernel ...

Hack...Computers.

● to modify (a computer program or electronic device) or write (a program) in a skillful or clever way.

● to circumvent security and break into (a network, computer, file, etc.), usually with malicious intent: Criminals hacked the bank's servers yesterday.

http://www.dictionary.com/browse/hacking

Hacking the Kernel ... [ OK ] Dynamically ...

3 ways to modify the kernelspace

Modules,Live patches,

and Kernel detouring

Kernel Module: Loading

Kernel

ext.ko

Kernel Space

User Space

Process:

insmod ext.ko

syscall: init_module

Kernel Module: Using

some_ext.ko

Kernel Space

User Space

Process:

cat /dev/some_cdev

syscall: read

Kernel

Kernel

Live Patching: Building

fix.ko

Kernel Space

User Space

Process:

build fix.patch


File:fix.patch

Current kernel source

Bug()

Bug-fixed()

Kernel

Live Patching: Applying

fix.ko

Kernel Space

User Space

Process:

insmod fix.ko


Bug()

Bug-fixed()

Kernel

Live Patching: Applying

fix.ko

Kernel Space

User Space

Bug() Bug-fixed()

ftrace

NormalProcess

Kernel

Kernel Detouring

detour.ko

Kernel Space

User Space

Normal Process

Special Process

func() func()

Hacking the Kernel ... [ OK ] Dynamically ... [ OK ] with Containers ...

Virtual Machines ( KVM )

Kernel

User Space

Kernel Space

kvm.ko

VM 2VM 1Proc 1

Libs / Conf

Guest Kernel

Proc 3Libs / Conf

Guest Kernel

Proc 2

hypercall

Processes

Kernel

Process 1

User Space

Kernel Space

Libraries / Configuration

Process 2

Process 3

Container 2Container 1

Containers

Kernel

Process 1

User Space

Kernel Space

Libs / Conf

Process 2

Process 3

Libs / Conf

Namespaces

Kernel

Kernel Detouring

detour.ko(namespace-

aware)

Kernel Space

User Space

NormalProcess Container

func() func()

Hacking the Kernel ... [ OK ] Dynamically ... [ OK ] with Containers ... [ OK ]

The short story

-- Just for fun!

The long story:1. Impact

http://alaskarobotics.com/2009/11/30/gallery-walk/

+

=

The long story:2. Trend

SaaS

PaaS

IaaS

CaaS?

VMProcessLibs / Conf

Guest Kernel

Container

The long story:3. Towards higher isolation

Isolation++

Performance++

Process

Libs / Conf

kvm

K-Container

Process

Libs / Conf

Detour

VMProcessLibs / Conf

Guest Kernel

Container

Side Note:For better performance

Isolation++

Performance++

Process

Libs / Conf

kvm

HyperVContainer

ProcessLibs / Conf

WindowsServer 10

ThinKernel

Where?

Possible Use Cases● Experimental module/patch test bed● Environment for other OSes● Educational purpose

FreeBSD binary on Linux

01000110011100100110010101100101010000100101001101000100

Why it is not possible: Scene 1

Kernel

mov 0x1, %raxsyscall(translate: I want to exit)


Kernel

...you want to write?nothing specialsys_write()


Kernel

(Eventually)SIGSEGV

End of life but no exit

...what’s wrong with this process?

Kernel

Recall: Kernel Detouring

detour.ko(namespace-

aware)

Kernel Space

User Space

NormalProcess Container

func() func()

Why it IS possible now: Scene 1

Kernel

mov 0x1, %raxsyscall(translate: I want to exit)

Container

Container


Kernel detour.ko(namespace-

aware)

system call tablesystem call table

for FreeBSD

#1 is sys_write, so …...wait, this is a FreeBSD container!

Container


Kernel detour.ko(namespace-

aware)sys_exit()

system call tablefor FreeBSD

Specific Challenges ( FreeBSD )● Corresponding system calls

– Flag numbers are not portable– different calling/exiting conventions

● Unique system calls– Re-implementation

General Challenges● Insufficient isolation● Limitation of development

– live patching should only be a temp. solution

Other Binary Compatibility Work● Wine

– Special loader for PEs/DLLs● FreeBSD, Windows 10

– Kernel built-in compatibility layer for Linux binary– System call remapping/re-implementation

Step 0: Setup● Environment ( x86_64 machine )

– Linux 4.6.2– FreeBSD 10.2

● Tools– kpatch: A tool for kernel livepatch– docker

Step 1: From LivePatching to Detouringkernel/livepatch/core.c.orig:klp_ftrace_handler

klp_arch_set_pc(regs, (unsigned long)func->new_func);

kernel/livepatch/core.c:klp_ftrace_handler

klp_arch_set_pc(regs, (is_freebsd_container()) ?

(unsigned long)func->new_func :

(unsigned long)ip+5);

Ftrace in LivePatching old_func()0x1230 call __fentry__0x1235 push %rbp0x1236 mov %rsp, %rbp0x1239 push %r15...

new_func()0x2230 call __fentry__0x2235 push %rbp0x2236 mov %rsp, %rbp0x2239 push %r14...

ftracehandler

IP

func->new_func

Ftrace in Detouring old_func()0x1230 call __fentry__0x1235 push %rbp0x1236 mov %rsp, %rbp0x1239 push %r15...

new_func()0x2230 call __fentry__0x2235 push %rbp0x2236 mov %rsp, %rbp0x2239 push %r14...

ftracehandler

IP

IP+5

func->new_func

Step 2: Detour-able Entry Pointarch/x86/entry/entry_64.S.orig:

ENTRY(entry_SYSCALL_64)

…

call *sys_call_table(, %rax, 8)

void detour_entry()

{

…

asm("jmp *sys_call_table(, %rax, 8)");

}

arch/x86/entry/entry_64.S:

ENTRY(entry_SYSCALL_64)

…

call detour_entry

Assembly file is NOT detour-able!

In some C source file

Step 3: Detoured Entry Pointvoid detour_entry()

...


void detour_entry()

...

asm("mov fbd_table(, %rax, 8), %rax");


asmlinkage const long long fbd_table[] = {0,60,57,...};

EXPORT_SYMBOL(fbd_table);

remapping syscall#

The work flow

1. Launch a normal container2. Run a init script

1. which enables the specific detour modules

3. A FreeBSD environment in the container4. On exit

1. disable detour modules

Conclusion● The kernel detouring demo attempts to

indicate the possibility of the development of OS containers– as a proof-of-concept– kpatch as a temp. solution

● Future direction– Make more fun

Behind the scenes● How to implement the is_freebsd_container()

function?● How was the ContainerCon Japan?

Talk 160920 @ Cat System Workshop

Engineering

Transcript of Talk 160920 @ Cat System Workshop