Post on 30-Dec-2020
讓 Linux 核心更安全 – 檢測並修補安全漏洞透過軟體開發生命週期管理核心安全漏洞
SZ Lin (林上智)
12th August, 2020
CYBERSEC 2020
Software R&D Engineer, Software Development Dept.
About Me - 林上智 (SZ LIN)
178F 8338 B314 01E3 04FC
44BA A959 B38A 9561 F3F9
Software Engineer, (In-house) Consultant
Embedded Linux Design and Development
- IIoT platform developer
- Civil Infrastructure Platform – Linux Foundation Project• Former Kernel Team Chair
• Technical Steering Committee Member
Open Source Development and Governance
- Debian Developer (pkg-security-team)
- OpenChain Project Governing Board Member
Cybersecurity
- CISSP – ISSAP, CSSLP
- ISA/ IEC 62443 Cybersecurity Expert
- Security Workgroup member in CIP project
> 80 % > 75 % 100 %> 95 %
img src: https://kernel.org
src: https://www.linuxfoundation.org/about/
of the top one
million domains
run with Linux
of cloud-enabled
enterprises report
using Linux as
their primary cloud
platform
of new
smartphones sold
run Android, which
is based on the
Linux kernel
of the top 500
supercomputers in
the world run on
Linux
Before Using Linux KernelSomething you should know
5
CopyrightCopyright PatentPatent
A patent gives its owner the right to
exclude others from making, using,
selling, and importing an invention
for a limited period of time, usually
twenty years.
src: https://en.wikipedia.org/wiki/Patent
Copyright is a legal right, that grants
the creator of an original work
exclusive rights to determine whether,
and under what conditions, this
original work may be used by others
src: https://en.wikipedia.org/wiki/Copyright
6
CopyrightCopyright PatentPatent
A patent gives its owner the right to
exclude others from making, using,
selling, and importing an invention
for a limited period of time, usually
twenty years.
src: https://en.wikipedia.org/wiki/Patent
Copyright is a legal right, that grants
the creator of an original work
exclusive rights to determine whether,
and under what conditions, this
original work may be used by others
src: https://en.wikipedia.org/wiki/Copyright
Context
7
1400+Members From
41 Countries
80%of Fortune 100
Tech & Telecom
35,000+Developers
Contributing Code
170+Open Source
Projects
$16BShared
Value
Linux Foundation
8
The OpenChain Project defines the key requirements
of a quality open source compliance program [1].
src:https://www.iso.org/standard/81039.html
12
User ApplicationsUser Applications
GNU/ Linux
GNU C libraryGNU C library
Init system
UserSpace
KernelSpace
Hardware and peripheral devices
Architecture-dependent firmwareArchitecture-dependent firmware
BootloaderBootloader
KernelKernelSystem call interfaceSystem call interface
ToolchainToolchain
Root filesystem
More info: Using open source software
to build an industrial-grade embedded
Linux platform from scratch
Open Source Summit Japan, 2019 [57]
End of LTS
13
Linux Kernel Releases
Mainline
Stable
(linux-stable-4.4)
v4.4
Stable
(linux-stable-4.19)
6+? years
v4.5 v4.19 v5.x
EOL
v4.4.x v4.4.y v4.4.z
v4.19.a v4.19.bimg src: https://en.wikipedia.org/wiki/Linux_kernel_version_history
End of LTS
6+? years
27.8 60-90 Day 66,492 3,386,34721,074
Mainline Kernel
Release CycleMillion Lines Files Lines of New Codes
in 2019Different Authors
14
src: https://www.phoronix.com/scan.php?page=news_item&px=Linux-Git-Stats-EOY2019
img src: https://kernel.org
Supply Chain Risk Management
Practices for Federal Information
Systems and Organizations
Special Publication 800-161 [4]
SM-9: Security requirements for
externally provided components
ISA/ IEC 62443-4-1 [5] NERCCIP-010-2 [6]
Configuration Change Management
and Vulnerability Assessments
img src: https://pixabay.com/illustrations/policies-standards-compliance-4720824/
15
src: https://www.ithome.com.tw/news/138633
2020-07-07發表
16
How to Manage
Vulnerabilities in
Linux Kernel?
17
Costs to Fix Software Defects at Different
Stages of SDLC [7]
1x5x
10x
15x
30x
0
5
10
15
20
25
30
35
RequirementsGathering and
Analysis/ ArchitecturalDesign
Coding/Unit Test Integration andComponent/RAISE
System Test
Early CustomerFeedback/Beta Test
Programs
Post-productRelease
18
X is a normalized unit of cost and can be expressed terms of person-hours, dollars, etc.
SDLCSoftware
Development Life Cycle
Requirement Analysis
Design
ImplementationTesting
Maintenance / Evolution
19
SDLCSoftware
Development Life Cycle
Requirement Analysis
Design
ImplementationTesting
Maintenance / Evolution
20
Scope
ScheduleResources
Good enough
principleKISS principle
Core technology
identification
Requirements Analysis
It’s imperative to collect, analyze, identify requirements for Linux
kernel and its configuration, it also reduces the unnecessary
maintenance effort related to security. Moreover, it provides
information for us to choose proper kernel source to fulfill our
requirements.
21
Requirements for the Civil Infrastructure Systems [8]
Industrial Grade
• Reliability
• Functional Safety
• Security
• Real-time capabilities
Sustainability
Security
• Security & vunerability managment
• Firmware updates
• Minimize risk of regressions
This has to be achieve with …
Development time
Shorter development times for more complex
systems
Maintenance costs
Low maintenance costs for commonly uses
software components
Low commissioning and update costs
Development costs
Don‘t re-invent the wheel
• Product life-cycles
of 10 – 60 years
22
SDLCSoftware
Development Life Cycle
Requirement Analysis
Design
ImplementationTesting
Maintenance / Evolution
23
Choose Proper Linux Kernel
only from trusted sites
24
CategoryLatest
versionTarget Application Maintainer
Linux kernel 5.8• Performance
• Resource Limited [9] [10]Kernel.org
Preempt RT
kernel5.6
• Real-time
• Functional safety
• Resource Limited
Real Time Linux
collaborative project
*Real-time application [11][12]
25
*Grsecurity [13]
SoC Board Support Package Kernel
• Kernel version depends on SoC vendors
– Well made but not well maintained
• Contain lots of in-house patches
– Errata patches
– Specific feature patches
– …
• Different SoC might use different versions of kernel
• The lifetime is unsure
26
LTS: Long Term Stable Kernel [3]
Extend software uptime for stable kernel
• Only accept bug fixes and security fixes
img: https://www.kernel.org/category/releases.html
Retrieved 7th August
27
LTSI: Long Term Support Initiative [14]
• Linux Foundation collaborative project – Based on LTS
– Add another chance to include further patches on top of LTS
– Auto Test framework
– Same lifetime with LTS (yearly release and 2 years life time)
28
CIP (Civil Infrastructure Platform) [16]
• Linux Foundation collaborative project – Support kernel and core package
– Auto Test framework
– Maintenance period• 10 years and more (10-20 years)
29
More info: CIP Kernel Team Activities to Accomplish Super Long Term Support
Embedded Linux Conference, 2020 [17]
CIP SLTS Kernel Releases
Mainline
Stable (linux-stable-4.4)
4.4
CIP SLTS 4.4 (linux-4.4.y-cip)
CIP SLTS 4.19 (linux-4.19.y-cip)
End of LTS
Stable (linux-stable-4.19)
Maintained by CIP
Maintained by
CIP Kernel
Maintainers
4.19
10 years
6 years
4 years
End of CIP SLTS
5.x
10 years
6 years
4 years
30
Speed and Efficiency : focus on differentiating parts
31
Linux Kernel Source Comparison Table
Version
Maintenance
Period
(years)
FeaturesLatest
Version
Supported
Real-time
kernel
Maintainer
SoC
BSP kernel? Bug fixes ? N SoC vendor kernel team
LTS
kernel2 ~ ?
• Bug fixes
• Security fixes 5.4 N Kernel.org
LTSI kernel 2 ~ ?
• Bug fixes
• Security fixes
• Specific features
• New features
4.14 NLTSI
(Linux Foundation Projects)
CIP
kernel10 +
• Bug fixes
• Security fixes
• Specific features
• New features
4.19 YCIP
(Linux Foundation Projects)
32
ELISA: Safety-Critical Systems [17]
• Linux Foundation collaborative project
– Build and certify Linux-based safety-critical applications
– Define and maintain a common set of tools and processes
• SIL2LinuxMP [18] project and the Linux Foundation’s Real-Time Linux project
– IEC 61508
33
Year 2038 Problem [19][20]
• The time_t datatype is a data type in the ISO C library and kernel structure defined for storing system time values.
• 32-bit system can represent dates from Dec 13 1901
Jan 19th 2038
• It causes integer overflowing on – 03:14:08 UTC 19 January 2038
34
Don’t choose rolling version
unless necessary
v4.4.1
Security fixesSecurity fixes
Bug fixesBug fixes
Upstream
rolling version
35
v4.4.2 v4.4.3
SDLCSoftware
Development Life Cycle
Requirement Analysis
Design
ImplementationTesting
Maintenance / Evolution
36
Upstream First
37
Kernel inside the organization
Upstream
38
v4.4.1
Kernel inside the organizationIn-house security or
bug patches
In-house security or
bug patches
Upstream
39
v4.4.1
v4.4.1
Security fixesSecurity fixes
Kernel inside the organization
Bug fixesBug fixes
Upstream
40
v4.4.1 v4.4.2
v4.4.1
In-house security or
bug patches
In-house security or
bug patches
Security fixesSecurity fixes
Kernel inside the organizationIn-house security or
bug patches
In-house security or
bug patches
Bug fixesBug fixes
Upstream
41
v4.4.1 v4.4.2
v4.4.1 v4.4.2
Security fixesSecurity fixes
Kernel inside the organizationIn-house security or
bug patches
In-house security or
bug patches
Bug fixesBug fixes
Upstream
42
v4.4.1 v4.4.2
Security fixesSecurity fixes
v4.4.3
v4.4.1 v4.4.2 v4.4.3
Security fixesSecurity fixes
Kernel inside the organizationIn-house security or
bug patches
In-house security or
bug patches
Bug fixesBug fixes
Upstream
43
v4.4.1 v4.4.2
Security fixesSecurity fixes
v4.4.3
v4.4.1 v4.4.2 v4.4.3
Security fixesSecurity fixes
Kernel inside the organizationIn-house security or
bug patches
In-house security or
bug patches
Bug fixesBug fixes
Upstream
44
v4.4.1 v4.4.2
Security fixesSecurity fixes
v4.4.3
v4.4.1 v4.4.2 v4.4.3
• The project shares its results with the upstream
• The project fulfills longer time maintenance and
security fixes
• The project develops their code very quickly
• The project faces difficulties to backport upstream
patches due to conflicts as time goes by
45
Kernel Hardening –
Configuration OptimizationSecure the system by reducing its attack surface
46
47
48
49
50
51
SDLCSoftware
Development Life Cycle
Requirement Analysis
Design
ImplementationTesting
Maintenance / Evolution
52
For Stable Kernel Maintenance
• Automated Linux Kernel Testing [22][23]
– Detect, bisect, report and fix regressions on upstream Kernel trees before release
– Short tests on many configurations
53
img src: https://kernelci.org/
img src: https://kernelci.org/
54
55
Reproducible Builds [25]
• Create an independently-verifiable path from source to binary
– Ensure builds have identical results
– Act as part of a chain of trust
– Prove the source code has not been tampered/modified
56
Continuous Integration • Jenkins [26]
• Jenkins X [27]
Continuous Delivery/ Deployment • LAVA 2 [28]
Distributed compiler service • icecc [29]
• GOMA [30][31]
• distcc [32]
Test Case Management • Jenkins
• LAVA 2
Version Control • Git with gitlab [33]
Static Program Analysis • checkpatch.pl [34]
• sparse [35][36]
• smatch [37]
Dynamic Program Analysis • Profiling tools [38]
Vulnerability Scanning • OpenVAS [39]
• Vuls [40]
Fuzzing Testing • Syzkaller [41]
• Trinity [42]
• perf_fuzzer [43]
More info:
Building, Deploying and Testing an
Industrial Linux Platform
Open Source Summit Japan 2017 [44]
57
SDLCSoftware
Development Life Cycle
Requirement Analysis
Design
ImplementationTesting
Maintenance / Evolution
58
0
400
800
1200v5.4
v4.19
v4.14v4.9
v4.4
Commit Counts per Month
Note: If a patch has an original patch, the date of the patch is that of the original one.
59
v4.19
v4.4
60
v4.9
v4.14
• cve-search [45]
• nvdtools [46]
• Distribution CVE tracker
• National vulnerability database [47]
• Upstream issue tracker or forum
Vulnerability Scanning – Component Level
61
Vulnerability Scanning – System Level
Security
Quick response in
resolving CVE/
vulnerabilities and
attacks in platform
Daily test for CVE
…
Daily test for CVE
…
62
Vulnerability Management Framework
Dependency-Track [49]
SW360 [48]
63
Vulnerability Scanning – Source Code Level
64
• This project tracks the status of security issues, identified by CVE
ID, in mainline, stable, and other configured branches.
Introduction to "cip-kernel-sec”
65
Issue Format - YAML
66
Show via Web I/F
Mainline/LTS
cip-kernel-sec
Webview Command line view
Gather CVE Information for Kernel
Show via Command Line
67
cip-kernel-sec Web View
6868
Linux Kernel Vulnerabilities = Bugs != CVEs
69
src: https://kernel-recipes.org/en/2019/talks/cves-are-dead-long-live-the-cve/
70
71
src: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-19076
72
src: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-19076
Community
Collaboration
Different approach for
multiple target applicationsPreparedness Planning
Testing and
well-maintenance
Conclusion
© Moxa Inc. All rights reserved.
Thank You
[1] https://www.openchainproject.org/
[2] https://www.iso.org/standard/81039.html
[3] https://www.kernel.org/
[4] https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-161.pdf
[5] https://webstore.iec.ch/preview/info_iec62443-4-1%7Bed1.0%7Den.pdf
[6] https://www.nerc.com/pa/Stand/Reliability%20Standards/CIP-010-2.pdf
[7] https://www.nist.gov/system/files/documents/director/planning/report02-
3.pdf
[8] Industrial-grade Open Source Base Layer Development, Yoshitake
Kobayashi, Urs Gleim.
Referneces
[9] https://tiny.wiki.kernel.org/start
[10] https://bootlin.com/pub/conferences/2017/jdll/opdenacker-embedded-
linux-in-less-than-4mb-of-ram/opdenacker-embedded-linux-in-less-than-
4mb-of-ram.pdf
[11] https://xenomai.org/
[12] https://www.rtai.org/
[13] https://grsecurity.net/
[14] https://ltsi.linuxfoundation.org/
[15] https://events.linuxfoundation.org/wp-content/uploads/2017/11/Using-
Linux-for-Long-Term-Community-Status-and-the-Way-We-Go-OSS-
Tsugikazu-Shibata.pdf
Referneces
[16] https://www.cip-project.org/
[17] https://static.sched.com/hosted_files/ossna2020/d0/OSSNA2020-CIPKernelTeam-2.pdf
[17] https://elisa.tech/
[18] http://www.osadl.org/SIL2LinuxMP.sil2-linux-project.0.html
[19] http://elinux.org/images/6/6e/End_of_Time_--_Embedded_Linux_Conference_2015.pdf
[20] https://en.wikipedia.org/wiki/Year_2038_problem
[21] www.cvedetails.com/vulnerability-list.php?vendor_id=33&product_id=47&version_id=261041&page=1&hasexp=0&opdos=0&opec=0&opov=0&opcsrf=0&opgpriv=0&opsqli=0&opxss=0&opdirt=0&opmemc=0&ophttprs=0&opbyp=0&opfileinc=0&opginf=0&cvssscoremin=0&cvssscoremax=0&year=2019&month=0&cweid=0&order=3&trc=72&sha=53735ab937bcf3686d34f3999d8e47f304466007
Referneces
[22] https://kernelci.org/
[23] https://fosdem.org/2019/schedule/event/kernelci_a_new_dawn/attachments/slides/3300/export/events/attachments/kernelci_a_new_dawn/slides/3300/gtucker_kernelci_fosdem_2019_v2_3_1024x768.pdf
[24] https://kernelci.org/build/stable/branch/linux-4.19.y/kernel/v4.19.138/
[25] https://reproducible-builds.org/
[26] https://jenkins.io
[27] https://jenkins.io/projects/jenkins-x/
[28] https://validation.linaro.org/static/docs/v2/#
[29] https://github.com/icecc
[30] https://chromium.googlesource.com/infra/goma/server/
[31] https://chromium.googlesource.com/infra/goma/client
[32] https://github.com/distcc/distcc
[33] https://about.gitlab.com/
Referneces
[34] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/scripts/checkpatch.pl
[35] http://sparse.wiki.kernel.org/
[36] https://git.kernel.org/pub/scm/devel/sparse/sparse.git
[37] http://smatch.sourceforge.net/
[38] https://perf.wiki.kernel.org/index.php/Main_Page
[39] http://www.openvas.org/
[40] https://vuls.io/
[41] https://github.com/google/syzkaller
[42] http://codemonkey.org.uk/projects/trinity/
[43] http://web.eece.maine.edu/~vweaver/projects/perf_events/fuzzer/
Referneces
[44] http://events.linuxfoundation.org/sites/events/files/slides/Build
ing%2C%20Deploying%20and%20Testing%20an%20Industrial%20Linux%
20Platform.pdf
[45] https://github.com/cve-search/cve-search
[46] https://github.com/facebookincubator/nvdtools
[47] https://nvd.nist.gov/
[48] https://www.eclipse.org/sw360/
[49] https://dependencytrack.org/
[50] https://www.cvedetails.com/version/261041/Linux-Linux-Kernel-4.19.html
[51] https://www.cvedetails.com/version/230587/Linux-Linux-Kernel-4.14.html
[52] https://www.cvedetails.com/version/205966/Linux-Linux-Kernel-4.9.html
[53] https://www.cvedetails.com/version/190796/Linux-Linux-Kernel-4.4.html
[54] https://gitlab.com/cip-project/cip-kernel/cip-kernel-sec
Referneces
[55] https://icss20.sched.com/event/ZjMw/managing-vulnerabilities-in-open-
source-components-in-ics
[56]
https://lore.kernel.org/lkml/20191204103955.63c4d9af@cakuba.netronome
.com/
[57] https://ossalsjp19.sched.com/event/OVsf/using-open-source-software-
to-build-an-industrial-grade-embedded-linux-platform-from-scratch-sz-lin-
moxa
Referneces