PRESENTATION BASED ON

NFVSEC(16)000104 PRACTICAL IMPLEMENTATION RESULTS:ATTESTATION, REMOTE ATTESTATION, CONFINEMENT AND ENHANCED PACKET PROCESSING TECHNOLOGIESMichael Lazarmlazar@dataart.com

European Telecommunications Standards Institute (ETSI) Industry Standard Group (ISG)Network Function Virtualization (NFV)

DataArtdesignsandbuildscustomsoftwaresystems.

Wepartnerwithclientstocreateandsupportinnovativesolutionsthathelpbusinessesbecomealastingsuccessinthemarketplace.

in operation

19 years

Professional Consultants & Developers

2000+

Staffturnover

< 7%

Return clients

95%

Billable hours

8+ million

Successfully completed projects

1600+

17 Global Locations

DISCLAIMER

The information in this presentation is provided "as is" and no guarantee or warranty is given that this information is suitable for any particular purpose. The user thereof uses the information at their own risk.

Security in general is use-case driven, this presentation is focused on Telcos, Sensitive Environments and National / Critical Infrastructure.

All trademarks and registered trademarks are the property of their respective owners.

ABOUT THIS PRESENTATION

§ DataArt is a Industry Specification Group Member of ETSI working in the NFV–SEC Group

§ DataArt ‘volunteered’ to build a secure NFV platform following available best practices

§ Used COTS equipment and Open Source Software to put together a ‘real world’ NFV implementation

§ System consisted of OpenStack / OPNFV / OpenDaylight

§ Intel architecture was utilized

§ The implementation utilized several security technologies

§ No specific NFV-SEC use case was used.

§ Presentation is for educational and informative purposes only

TERMINOLOGY / DEFINITIONS

TPM – Trusted Platform Module is a tamper resistant processor that provides cryptographic functions.

Measurement – SHA-1 hash value calculated by the TPM

PCR - Platform Configuration Registers, the TPM provides secure storage and reporting of security measurements. The TPM contains several registers

CRTM – Core Root of Trust for Measurement. Together with the BIOS, the TPM forms a Core Root of Trust for Measurement.

Memory Sharing - A memory-saving de-duplication feature, that merges anonymous (private) pages. Frequently used to “over provision” memory in virtual environments

Enhanced Packet Processing – Various techniques and technologies used to enhance network performance. The Intel Data Plane Development Kit (DPDK) is one example

Confinement Technologies - Provide an enhanced mechanism to enforce the separation of information based on confidentiality and integrity requirements, which allows threats of tampering and bypassing of application security mechanisms to be addressed and enables the confinement of damage that can be caused by malicious or flawed applications. Common examples are SELinux / sVirt / AppArmor.

Attestation / Trusted System – A system and potentially a systems software that has undergone verification to ensure it has not be tampered with

SECURITY

§ Security is and always will be a cat-and-mouse game

§ Security is typically not a primary goal in a product

§ Adding security late in the design or implementation cycle may create unintended consequences

§ Tradeoffs between performance and security may need to made but the impact should be understood

§ Low level security provides a foundation to build on

§ Virtualization brings unique security issues that may not be obvious on first review

§ Taking the time and putting in the effort will benefit everyone in the long run

ETSI NFV REFERENCE ARCHITECTURE

Execution Reference Points Other Reference Points Main NFV Reference Points

CHAIN OF TRUST – ATTESTATION IS DESIGNED TO PRODUCE A SECURE ROOT OF TRUST

Consider that entity A launches entity B, then B launches C.

A measures B then passes control to BB measures C and passes control to C

The question now becomes "who measures A?”

The Core Root of Trust for Measurement (CRTM) is the BIOS boot block code. This piece of code is considered trustworthy. It does not change during the life of the system.

When a TPM is present and enabled (TPM_INIT) the system will evaluate components and extend values in the PCRs. These can then be verified to be valid.

*BIOS is being used as a generic term.

Is the platform trusted?

REMOTE ATTESTATION ARCHITECTURE –OVERVIEW

Remote Attestation is a means by which a trusted computer assures a remote computer of its trustworthy status.

VIRTUALIZATION –THE ‘ROOT’ OF THE ISSUE

The (vast) majority of todays commercial physical compute resources and operating systems fundamentally work off of a implicit trust model. To be more explicit, there is trust between the hardware subsystems and kernel operations. Even when zero trust models are implemented in user space, todays kernels (and kernel variants) rely on implicit trust to function.

Virtualization attack vectors have become more sophisticated focusing on virtual machine attacks (break out), hypervisor attacks (blue pill), side channel and compromised hardware (malicious hardware). These are not hypothetical attacks

Over the last years several hardware and software technologies have been made available, including VT-d, Authenticated boot, Trusted Platform Modules (TPM), Trusted boot (tboot), SELinux, sVirt, AppArmor, OAT SDK (remote attestation toolkit) and Trusted Execution Technology (TXT) to make platforms more secure.

Additional technologies are available or emerging including TrustZone (ARM/AMD) and Software Guard Extensions (Intel SGX).

SOME ATTACK VECTORS –POTENTIAL REMEDIATION

VECTORModified Hardware (malicious)Foreign Hardware (unauthorized/ malicious)Side channel attacks (breakout of confinement)Hypervisor modification (malicious code)

-- Intel terminology used (not meant to be exclusive)

Authenticated boot / Static measurement w/TPMIntel VT-d (AMD IOMMU) prevents raw PCI accessSELinux, sVirt, AppArmor, Seccomp2Trusted Execution Technology (TXT) / tboot / Remote attestation

Several hypervisors are available (OpenStack supports many variants) not all are equally secure

Potential RemediationSRTM - Static Root of Trust for Measurements (authenticated boot ) with TPM is a mature technology. Hardware may be attested at boot time, however, SRTM requires keeping measurements of the entire platform boot sequence including BIOS config, 3rd party boot ROMs (e.g. network cards). Any change to the environment requires new measurements (which are disruptive and complex to maintain).

DRTM - Dynamic Root of Trust for Measurements (tboot) utilizes TXT and TPM. It can verify the hardware and software (hypervisor) have not been tampered with on boot and can take direct action (halt). While it can replace SRTM they may also be used together allowing SRTM to provide a key measurements (PCR0/1) while DRTM provides the remainder of the measurements without ‘loading’ down the attestation process.

Utilizing TXT and TPM the OpenAttestationToolkit (OAT) provides Remote Attestation and “Trusted Compute Pools” (implementation as a Security Console)

CPUTPM / TXT

1 2a

2b

3a

3b

SIMPLIFIED VIEW OF TRUSTED COMPUTE POOLSPUTTING IT ALL TOGETHER CAN GIVE YOU A MORE SECURE ENVIRONMENT

SO WE HAVE “TRUSTED COMPUTE POOLS” EVERYTHING IS GOOD – RIGHT?

§ Implementation in OpenStack/OPFNV is set by trusted_filter flag in the abstract scheduler for a VM.§ Change the flag in memory and the VM is no longer required to run in a trusted environment.§ Alter the scheduler to ignore the flag, while reporting everything is ‘OK’.§ OAT (SDK) uses certificates, if they are compromised…

§ Binary trust model – hosts / hypervisors are either trusted or not. Currently there is no implementation of a hierarchal trust model and its doubtful one can be implemented into the current architecture.

§ VMs with the trusted flag set can only run on trusted hosts (attested), however untrusted VMs (no flag set) can also run on trusted hosts.

§ Only Compute Nodes (NOVA) support attestation. Management, storage nodes do not support attestation natively

§ OpenStack Neutron (networking service) depends on Open vSwitch and MySQL/MariaDB. If you modify records it will change network settings and so on. Network traffic is not subject to trust rules.

§ Similar to hardware trust model issues, the virtualization administrators have nearly unlimited power.

SO WE HAVE TRUSTED COMPUTE POOLS –EVERYTHING IS GOOD – RIGHT?

§ Revoking trust on a running host leaves trusted VMs operational (they continue to run)

§ Compute pools are managed by a “security station”, which now becomes the target. OAT for OpenStack requires SELinuxto run in permissive mode by default.

§ In general SELinux / sVirt / AppArmor rules are deficient. Either missing or set so loose as to reduce the value of isolation

§ Shared memory allows for leakage or exposure of sensitive data (encryption keys, etc). The CAIN (Cross-VM ASL INtrospection) ‘attack’ shows Windows and Linux VMs are vulnerable (there is not a solution other than turning shared memory off)

§ Most implementations of enhanced packet processing (e.g. DPDK) require confinement technologies to be disabled

§ Hypervisor choice matters:

§ Baremetal (Ironic)- still early in development lifecycle, under/over cloud implementation does not support SELinux/sVirt out of the box. PXE+TFTP loading not secure. Ironman plugin available

§ KVM - under/over cloud implementation does not support SELinux/sVirt out of the box. § QEMU – hardening guide specifically mentions as a risk, however high performance solutions continue to

use/require it§ LXD/LXC - containers will always (by design) share the same kernel as the host. Therefore, any vulnerabilities

in the kernel interface, unless the container is forbidden the use of that interface (i.e. using seccomp2) can be exploited by the container to harm the host.

SOME THOUGHTS

Virtual environments create new challenges for security, it is not the same old world just running on a hypervisor.

Several technologies now exist to help create more secure virtual environments that start with the basic concept of ‘hardware root of trust’, through booting trusted hypervisors and virtual machines (containers, etc). However due to implementation issues these are not as secure as they seem at first review.

Vendors need to be encouraged to use available technologies as “default”. (random sampling shows TXT is not enabled and in several cases VT-d was disabled)

The release of the OAT as an SDK was done to enable commercial development (BSD license). It does not appear that this has been embraced, instead the absolute minimum has been done to ‘check the box’.

The single trust zone is a severe issue that needs to be addressed at a fundamental level.

Need to work with Open Source communities to encourage better security models using available technology. Some of these issues can be addressed relatively easily.

RESOURCES – WORKS CONSULTED

§ IBM Trusted Computing for Linux http://www.research.ibm.com/gsal/tcpa/TCFL-TPM_intro.pdf

§ Intel TXT overviewhttp://www.intel.com/content/dam/www/public/us/en/documents/white-papers/trusted-execution-technology-security-paper.pdf

§ Attacking TXT via SNIT - (exploits are old but the detailed explanation is valuable)http://invisiblethingslab.com/resources/2011/Attacking_Intel_TXT_via_SINIT_hijacking.pdf

§ Security Enhanced Linux (NSA)https://www.nsa.gov/research/selinux/

§ sVirt – SELinux mandatory access controls with the virtualization componentshttp://namei.org/presentations/svirt-lca-2009.pdf

§ CAIN: Silently Breaking ASLR in the Cloud https://www.usenix.org/conference/woot15/workshop-program/presentation/barresi

• Hardening the virtualization layerhttp://docs.openstack.org/security-guide/compute/hardening-the-virtualization-layers.html

• Building the infrastructure for Cloud Security (entire book is open access)http://link.springer.com/book/10.1007/978-1-4302-6146-9

• Open Attestation Toolkit (SDK) (Used in Trusted Compute Pools / Remote Attestation)https://01.org/openattestation

• Intel Software Guard Extensionshttp://www.pdl.cmu.edu/SDI/2013/slides/rozas-SGX.pdf

• ARM TrustZone (have partnership with AMD)http://www.arm.com/products/processors/technologies/trustzone/index.php

Q&A Session

Thank You !

Additional Materials

TRUSTED PLATFORM MODULE (TPM)(V1.2 SHOWN)

“Secure Hardware” utilized for cryptographic operations. One use is for trust measurement, the device itself does not understand “trust”

Power On

Static / Dynamic Measurement

Physical System Verified

Trusted Boot Loader (e.g. tboot)

Kernel Loading

Hypervisor Enablement

Data Partitions

Monitoring

Verify Workload Integrity

TEE

Clear TPM PCR

Confinement Technologies (e.g. SELinux)

Confinement Technologies (e.g. sVirt)

Measurement Attestation

EXAMPLE OF SIMPLIFIED BOOT SCHEME DIAGRAM USING A TRUSTED PLATFORM MODULE (TPM)

REMOTE ATTESTATION PROTOCOL OVERVIEW (TPM V1.2)

TCG BASED – CRTM / TPM EXAMPLE BOOT FLOW

Diagram is based on Figure 3 (page 25) - TCG PC Client Specific Implementation Specification for Conventional BIOSSpecification Version 1.21 Errata Revision 1.00 February 24th, 2012 For TPM Family 1.2; Level 2

https://www.trustedcomputinggroup.org/wp-content/uploads/TCG_PCClientImplementation_1-21_1_00.pdf

Step 1 – The device is reset (e.g. power off / on)

Step 2 – The CRTM self verifies (no unauthorized changes)

Step 2A – The TPM is passed control of the system and begins to measure system components and compare the results of the securely stored PCR values

Step 3 / 3A / 3B – After the initial verification has been completed ‘successfully’ the TPM passes some operational controls back to the BIOS to continue the booting and measurement process. As memory is generally not available at this point, some steps performed by the TPM “B” and when permitted by the BIOS or CPU registers ”A”

Steps 4-6 – Comprise of measuring and validating various PCRs. In the diagram certain core measurement functions are performed by the TPM “B” and when permitted by the BIOS or CPU registers ”A”

Step 7 – 8 – After the platform has passed the static measurement process the boot process continues. In the following diagram the static measurement process has been designated as “Measurement Phase I”

If dynamic measurement is being used (e.g. Intel Trusted Execution Technology) the measurement and verification process continues through the operating system load to verify that the operating system itself has not been altered. The dynamic process is shown in slightly more detail in the next slide, designated “Measurement Phase II”

TCG BASED – CRTM / TPM EXAMPLE BOOT FLOW

TPM – PLATFORM CONFIGURATION REGISTERS

UCo

deVa

lidates,

MeasuresBIOSAC

M

ACM

Validates,

MeasuresBIOSUnit

Code

uCode evals BIOS ACM

BIOS ACM (evals BIOS init code)

BIOS

BIOS Options ROMs

Measurement Phase 1(H/W + BIOS)

Measurement Phase 2(TBOOT, OS, Hypervisor..)

UnitT

XT&Mem

Lead

SMM

Measure

SMM

&othe

rTrustedCo

de

Lock

TXT&Mem

ory

Confi

g

Non

-CriticalCo

de

Lead

SINIT&OScode

uCod

eVa

lidates

SINIT

SINITMeasuresTB

OOT

SINITMeasures

OSKe

rnelInitrd+

+

TBOOT-XM

Measures

Hypervisor(s)

Laun

chOS

PCRO PCRO

System Power ON ENTERACCS: LockConfig SENTER

X

PCRO+ PCR17 PCR18 PCR19 PCR19+

Boot loader

uCode (evals SINIT ACM)

SINIT ACM (measures OS Kernel, initrd)

TBOOT-XM measures Hypervisor, other components

BIOS OS