Manage custom kernel buildsMarian HackMan Marinov

CEO of 1H Ltd.CTO of GetClouder Ltd.

<mm @ 1h .com >ICQ: 7556201

Jabber: hackman@jabber.orgIRC: irc.freenode.net HackMan

Some definitions

3.X.Y

Some definitions

3.X.Y

➢ Major release (changes to this are rear)

Some definitions

3.X.Y

➢ Major release

➢ Minor release (the new Major :))

Some definitions

3.X.Y

➢ Major release

➢ Minor release

➢ Patch level (the new minor)

Some definitions

➢Mainline kernel - http://kernel.org➢ stable

➢ release candidate (-rc)

➢ next

➢ long term support (LTS)

➢Distribution kernel➢ Debian

➢ RedHat

➢ SuSE

➢Project kernel➢ OpenVZ

Why do we need custom kernels?

➢Better performance

➢Faster boot times

➢More secure kernels(smaller attack serfice)

➢Monolithic kernels (does not work on machines with SW RAID or LVM)

➢Support custom drivers

➢Support features that are not included in the mainline

➢New features that are not supported by the distribution kernels

Support custom drivers

➢network interface drivers

➢storage drivers

➢vendor supplied drivers➢ sometimes require specific kernel version

➢ they should be either back or forward ported

➢ require changes in the kernel in order to build/load properly

Note: this does not happen often

Support custom features

➢Kpatch (live kernel patching)

➢kGraft (live kernel patching)

➢AUFS (Docker)

➢BFQ (I/O scheduling)

➢GRsecurity

Problems with the custom features

➢Irregular releases

➢Required specific kernel versions

➢No release for the kernel you have chosen

How to choose a kernel to build upon

➢TEST

How to choose a kernel to build upon

➢TEST, TEST

How to choose a kernel to build upon

➢TEST, TEST, TEST

How to choose a kernel to build upon

➢TEST, TEST, TEST

➢Build every major release

How to choose a kernel to build upon

➢TEST, TEST, TEST

➢Build every major release

➢Try to experiment with x.y.1 or x.y.2 but not x.y.0 :)

How to choose a kernel to build upon

➢TEST, TEST, TEST

➢Build every major release

➢Try to experiment with x.y.1 or x.y.2 but not x.y.0 :)

➢The above is not a hard rule

How to choose a kernel to build upon

➢TEST, TEST, TEST

➢Build every major release

➢Try to experiment with x.y.1 or x.y.2 but not x.y.0 :)

➢The above is not a hard rule

➢Try LTS versions, but don't limit your self to these kernels

How to choose a kernel to build upon

➢TEST, TEST, TEST

➢Build every major release

➢Try to experiment with x.y.1 or x.y.2 but not x.y.0 :)

➢The above is not a hard rule

➢Try LTS versions, but don't limit your self to these kernels

➢Test the performance

Test the new Kernel performance

➢Look at your own load

➢Try to break the kernel➢ try to replay your normal load 100 or 1000 times

➢ try to break the kernel with community benchmarks

➢Do not compare 32 and 64 bit kernels

➢Test the performance between two kernels in a row

➢If you compare the performance between different major releases, do not include the new features

Test the new Kernel performance

➢I usually check if the following stay relatively the same➢ /proc/loadavg

➢ /proc/schedstat

➢ is the memory allocation keeps similar values

➢ I/O performance

➢Network performance (use tcpreplay)

➢hardware functionality

➢ software compatibility

Feature automation

Initially

➢Create a branch with a name that should include the name of the kernel version you are using

example kernel: 3.17.2

example branch name: grsec-3.17.2

Feature automation

Second time

1. Check if you have a branch with the name of your feature (grsec-3.17.2)

2. Check if you have a branch that includes the name of the future and the version of your current kernel

3. Check if there are newer versions of the kernel

4. If there are no new versions(tags)... don't do anything

5. If there are new versions(tags)

Feature automation

6. Checkout the current branch into a new one named with the new version

example: grsec-3.17.3

7. Rebase the new branch over the last released kernel version

1. If there are errors, send me an e-mail

2. If no errors are detected, commit

3. Rename the branch to reflect the new kernel version

9. Do some cleanup from time to time :)

- You may move your archive branches into a separate repository

Feature automation

Master

featureX-3.17.3

v3.18

v3.17.4

v3.17.2

v3.17.4

v3.17.3

Feature automation

Master

TAGfeatureX-3.17.3

v3.18

v3.17.4

v3.17.2

v3.17.4

v3.17.3

Feature automation

Master

TAGfeatureX-3.17.3

v3.18

v3.17.4

v3.17.2

v3.17.4

v3.17.3

featureX-3.17.4

Feature automation

Master

TAGfeatureX-3.17.3

v3.18

v3.17.4

v3.17.2

v3.17.4

v3.17.3

TAGfeatureX-3.17.4

Feature automation

Master

TAGfeatureX-3.17.3

v3.18

v3.17.4

v3.17.2

v3.17.4

v3.17.3

TAGfeatureX-3.17.4

featureX-3.18

The actual management

➢ this works ONLY for your own patches

➢ keep a list with all your patches

➢ clone the Linus repo

The actual management

➢ git pull➢ if there are new tags, create a test branch with the tag's name:

v3.17.2 -> test-kernel-3.17.2

➢ for each patch branch that you keep

➢ checkout the current patch branch

➢ branch into a new name based on the new kernel version

➢ rebase

➢ checkout that branch

➢ copy my current kernel .config

➢ make olddefconfig (if it breaks, send me an e-mail)

Build

➢ Pull once a day or a little bit more often :)

➢ Build once a week or little bit more often :)

➢ If you are planning on upgrading this kernel, fix found conflicts at least every two weeks

What is next?

➢ Handling CVEs➢ Kernel patch and build

➢ Kpatch or kGraft patch module generation

➢ Load the kpatch or kgraft

➢ Monitor the Linux Kernel Mailing List(LKML) for all subsystems of interest to you

➢ Do not try to monitor the LKML for everything...

➢ Backport the performance fixes made in new versions

What not to patch?

➢ Assembly code

➢ Crypto stuff

➢ Limit the architectures you write patches for

➢ Be extremely careful with:➢ cpu schedule

➢ vm scheduler

➢ i/o schedule

What not to patch?

Questions?

<mm @ 1h .com >ICQ: 7556201

Jabber: hackman@jabber.orgIRC: irc.freenode.net HackMan

CEO of 1H Ltd.CTO of GetClouder Ltd.