Linux field-update-2015

Updating Embedded Linux devicesin the field

Updating Embedded Linux devices in the field 1 Copyright © 2011-2015, 2net Ltd

License

These slides are available under a Creative Commons Attribution-ShareAlike 3.0license. You can read the full text of the license herehttp://creativecommons.org/licenses/by-sa/3.0/legalcode

You are free to

• copy, distribute, display, and perform the work

• make derivative works

• make commercial use of the work

Under the following conditions

• Attribution: you must give the original author credit

• Share Alike: if you alter, transform, or build upon this work, you may distributethe resulting work only under a license identical to this one (i.e. include thispage exactly as it is)

• For any reuse or distribution, you must make clear to others the license terms ofthis work


http://creativecommons.org/licenses/by-sa/3.0/legalcode

About Chris Simmonds

• Consultant and trainer• Working with embedded Linux since 1999• Android since 2009• Speaker at many conferences and

workshops

"Looking after the Inner Penguin" blog at http://2net.co.uk/

https://uk.linkedin.com/in/chrisdsimmonds/

https://google.com/+chrissimmonds


http://2net.co.uk/

https://uk.linkedin.com/in/chrisdsimmonds/

https://google.com/+chrissimmonds

Overview

• The need for software update of embedded devices

• Update by file or system image

• Update agent: bootloader or application

• An Linux update agent: SWupdate

• Update strategies

• Failsafe


The problem

• Embedded software is non-trivial (=> has bugs!)

• Devices are often connected to the Internet

• Allowing hackers to exploit the bugs

• Result: problems with the IoT

• We need an update mechanism to fix those bugs inthe field

• (Not to mention the desire to deploy new features,better performance, etc.)


Components of embedded Linux

• Bootloader

• Linux Kernel (+ Device Tree and kernel modules)

• Root file system (basic operating system libraries andutilities, usually read-only)

• System apps (often a separate partition, but may becombined with root file system)

• User data (configuration settings, logs, user-supplieddata)


Frequency of updateFr

equency

Ease of update

Bootloader

Kernel

Root file system

System applications


Requirements of an updater

• Robust: must not render the device unusable

• Atomic: must not apply half an update

• Fail-safe: have a fall-back mode if all else fails

• Remote, unattended update (as an option)

• Audit trail: so you can tell what updates have beenapplied

• Preserve user data


A simple approach• Majority of updates are to root filesystem and system

apps, which are "just files"

• Many projects have an updater based on tarballs andsome shell scripts

• They all fail, because:

• Lack atomicity

• Can’t update kernel in raw flash memory

• (Usually) no audit trail

• Not fail-safe

• Remember: if it can go wrong, it will go wrong(eventually) 1

1Murphy’s lawUpdating Embedded Linux devices in the field 9 Copyright © 2011-2015, 2net Ltd

Atomic file updates

• Update is either complete or not done at all

• "Easy" to achieve for individual files, e.g. usingPOSIX rename (2) and careful checkpointing withfsync (2)

• More difficult to keep relationships between filesconsistent

• Example: application and dependant libraries

• (Can be done if you pay attention to library versioning)

• Mainstream Linux package managers (RPM anddpkg) solve a lot of the problems but are still notbulletproof


Atomic system image updates

• Instead of a large number of individual files, considera system as a smaller number of system images

• Kernel, root filesystem, system apps, etc

• Embedded devices are typically put together like this

• Need a method to update safely

• Examples:

• Dual copies of OS and ping-pong between them

• Have separate recovery OS just for updating


Two copies of the OS

Bootloader Userdata

Bootflag

OS Copy 1

OS Copy 2

• Update OS copy 2 from OS copy 1; set boot flagwhen done

• "ping-pong" between OS copies

• But, requires double the flash storage


Recovery OS

Bootloader

Main OS

Recovery OS

Userdata

Bootflag

• To perform update, set boot flag and reboot intorecovery OS

• Then update main OS; clear boot flag when done

• Reboot to start updated main OS


Bootloader as update agent

• Seems an attractive option: simple environment withno OS running

• But, bootloaders have limited support for peripherals,file system formats and network protocols

• Incorporating the necessary software would make thebootloader more buggy

• Remember that updating the bootloader itself isdifficult, often a single point of failure

• Conclusion: bootloader is not a good update agent!


Linux application as update agent

• Much better environment

• Access to all peripherals

• Read-write access to file systems

• All network protocols

• Maybe a display for user interaction

• Few open source examples

• One is SWupdate


SWupdate

• Written by Stefano Babic

• Source: https://github.com/sbabic/swupdate

• Doc: https://github.com/sbabic/swupdate/blob/master/doc/source/swupdate.rst

• Yocto meta layer available:https://github.com/sbabic/meta-swupdate

• Can use pre- and post- install scripts to set/reset theboot flag in the bootloader (U-Boot, for example)


https://github.com/sbabic/swupdate

https://github.com/sbabic/swupdate/blob/master/doc/source/swupdate.rst

https://github.com/sbabic/swupdate/blob/master/doc/source/swupdate.rst

https://github.com/sbabic/meta-swupdate

Local update

Main OSimage

Recovery OSSD

SWupdate

• Update supplied on local removable media, initiatedby user or maintenance staff


Remote update 1/2

Main OSimage

Recovery OS

SD

SWupdate

Downloader

Updater

• Download update from network to local storage:allows update to be verified and installed at aconvenient time

• May be initiated by user or automatically

• Requires spare local storage


Remote update 2/2

Main OSimage

Recovery OS

SWupdate

Downloader

Updater

• Update directly from network stream: no localstorage needed

• Device is unavailable while updating, may lead tosignificant down time if link slow or broken

• (Streaming not implemented in SWupadte yet)


Failsafe

• Robust software update is not the whole answer

• Can’t protect against installing (correctly) a buggyupdate

• Last line of defence:

• Watchdog -> reboots if device is not functioning

• Bootloader counts boots triggered by watchdog

• Boot into recovery mode when exceed threshold andattempt to re-install


Further reading

• "Building Murphy-compatible embedded Linuxsystems" by Gilad Ben-Yossef https://www.kernel.org/doc/ols/2005/ols2005v1-pages-21-36.pdf

• "Updating an embedded system :swupdatedocumentation"http://sbabic.github.io/swupdate/

• The rename (2) function:http://pubs.opengroup.org/onlinepubs/

009695399/functions/rename.html

• The fsync (2) function: http://pubs.opengroup.org/onlinepubs/009695399/functions/fsync.html


https://www.kernel.org/doc/ols/2005/ols2005v1-pages-21-36.pdf

https://www.kernel.org/doc/ols/2005/ols2005v1-pages-21-36.pdf

http://sbabic.github.io/swupdate/

http://pubs.opengroup.org/onlinepubs/009695399/functions/rename.html

http://pubs.opengroup.org/onlinepubs/009695399/functions/rename.html

http://pubs.opengroup.org/onlinepubs/009695399/functions/fsync.html

http://pubs.opengroup.org/onlinepubs/009695399/functions/fsync.html

• Questions?

Slides on Slide Share: http://www.slideshare.net/chrissimmonds/linux-field-update-2015


http://www.slideshare.net/chrissimmonds/linux-field-update-2015

http://www.slideshare.net/chrissimmonds/linux-field-update-2015

Linux field-update-2015

Documents

Transcript of Linux field-update-2015