От Treble к MainlineСвященная борьба Google за быстрые апдейты

October 2019

2

Background

Project Mainline

Modular Updates

3

Agenda

• Why Google needed it. The Updates problem.

• What Project Treble was about?

• Project Treble results and why they needed Mainline.

• What is Project Mainline exactly?

• Where it all seems to be going…

4

The Reason

5

Pie chart without Pie

• As of October 26, 2018 only 21,5% of Android devices were running Oreo

• As of October 11, iOS 12 is running on 53% active devices

• Less than 0.1% of active devices run Android Pie

Android Versions

Oreo - 21,5

Nougat- 28,2

Marshmallow- 21,3

Lollipop - 17,9

KitKat - 7,6

Jellybean - 3

Older versions - 0,5

6

The Reason

• All these companies work on the same Android code

• That is good because you get a lot of people who can contribute

• That is bad because you get a lot of people who can break stuff

7

Why Google needed Treble?

• Step 1 to Step 5 used to take 6-12 months (That if device manufacturers bothered with updates at all)

• That’s how much time it takes to SoC and OEM vendors to do their part

8

Project Treble

9

Changed Architecture

• The main idea of Project Treble was to create a better separation between areas of responsibility

• This red line on the right didn’t actually exist before Treble

OEM Vendor

10

Treble components

• New HAL types • Hardware Interface Definition Language (HIDL)• New Partitions• ConfigStore HAL• Device Tree Overlays• Vendor NDK• Vendor Interface Object• Vendor Test Suite (VTS)

11

Four steps of HALs

Go

ogl

eSo

Cve

nd

or

12

Vendor Test Suite

• VTS is essentially the same thing as CTS, but few layers deeper into the system

• VTS checks HALs APIs, Kernel version, C libraries, etc.

13

Before and After

Before Treble:

• HALs were libraries with C-header API

• HALs were included in OTA package

• HALs were mixed with System Services in /system

• Only application API is tested on certification

After Treble:

• HALs are services with HIDL API

• HAL versions are checked in OTA

• HALs have their own /vendor partition

• Vendor interface is also tested on certification

14

Treble results

• As of May 7, 2019 Pie has 10,4% share

• These numbers are moderately better than before Treble

Pie - 10,4

Oreo - 28,3

Nougat - 19,2

Marshmallow - 16,9

Lollipop - 14,5

KitKat - 6,9

Jelly Bean - 3,2

Older versions - 0,6

15

Treble lessons

• After Treble SoC and OEM vendors cold work on updates in parallel. This helped a bit, but ultimately didn’t solve the problem

• Google still was not able to push updates to user phones directly

• Two main ideas from Treble that worked: Divide code into areas of responsibility Create stable versioned APIs between these areas

16

Poject Mainline

17

Summary

• «Project Mainline enables us to update core OS components in a way that's similar to the way we update apps: through Google Play.»

• «End-user devices can receive updates to these modularized system components from the Google Play Store infrastructure or through a partner-provided over-the-air (OTA) mechanism.»

18

Module Interactions

• Module updates don't introduce new APIs. They use only the SDK and System APIs guaranteed by the Compatibility Test Suite (CTS), communicate only with each other, and use only stable C API or stable AIDL interfaces.

• Stable AIDL Interfaces is a new thing in Android 10. They are similar to traditional interfaces but they are stable and they have versions for backward compartibility.

19

Android Pony EXpress

20

APEX: Android Pony Express

• Android Pony EXpress (APEX) is a container format introduced in Android 10 that is used in the install flow for lower-level system modules.

• Some example components are native services and libraries, hardware abstraction layers (HALs), runtime (ART), and class libraries.

• APK was not good enough for 2 reasons:APK-based modules can't be used early in the boot sequence. The APK format (particularly the manifest) is designed for Android apps and

system modules aren't always a good fit.

21

APEX format

22

APEX Manager

• The APEX manager (apexd) is a standalone native process responsible for verifying, installing, and uninstalling APEX files.

• APEX files are normally pre-installed on the device under /system/apex

• New versions of APEX files will be downloaded to /data/apex

• When all APEX files listed in the internal database are mounted, the APEX manager provides a binder service for other system components to query information about the installed APEX files.

23

Modular Components

24

Android 10 Modular System Cmponents

Android 10 modularizes some Android system components and enables them to be updated outside of the normal Android release cycle:

• Android Runtime• Conscrypt• DNS Resolver• DocumentsUI• ExtServices• Media ( Codecs, Media Extractor, MediaPlayer2)• ModuleMetadata• Networking• PermissionController• Time Zone Data

• APEX• APK

25

Три цели

26

Security

27

Conscrypt

• Conscrypt is a Java Security Provider (JSP) that implements parts of the Java Cryptography Extension (JCE) and Java Secure Socket Extension (JSSE). It uses BoringSSL to provide cryptographic primitives and Transport Layer Security (TLS) for Java applications on Android and OpenJDK.

• Conscrypt is available as an open source library, though it has some specializations when included in the Android platform.

• The Conscrypt module is an APEX file that includes: Conscrypt Java code Conscrypt native library that dynamically links to Android NDK libraries The native library also includes a copy of BoringSSL

28

DNS Resolver

• The DNS Resolver module provides user protection for DNS interception and configuration update attacks and improved network performance for DNS resolutions.

• In Android 9 DNS resolver code was spread across Bionic and netd.

• Android 10 moves the DNS resolver code to system/netd/resolv, converts it to C++, then modernizes and refactors the code.

• The DNS Resolver module is delivered as an APEX file and is dynamically linked by netd.

29

Media

• APEX module that makes 3 types of media components updatable: Media Extractors - .so libraries to extract data from different media files. Media Codecs - .so libraries to extract data for encoding/decoding media. MediaSession2 - Part of android.media.* API.

• OEM vendors can add their own Extractors and Codecs and even update them with APEX. But they are forbidden to modify MediaSession2 Interface.

30

Privacy

31

Permission Controller

• com.android.permissioncontroller APK that handles: Runtime permission granting (including granting to system apps) Runtime permission management (including grouping of permissions) Runtime permission usage tracking (new in Android 10) Roles (new in Android 10)

• In Android 9, such permissions were part of com.android.packageinstaller

32

ExtService

• com.android.ext.services APK that includes following services: AutofillFieldClassificationService NotificationAssistantService ResolverRankerService CacheQuotaService

• “Together with DocumentsUI and PermissionController, this module ensures that end users get a consistent permissions UI while enabling iteration in response to the changing privacy landscape and government regulations.”

33

Documents UI

• com.android.documentsui APK controls access to specific files for components that handle document permissions

• Android partners are allowed to customize the features and theming of the app through runtime resource overlays (RROs).

34

Consitency

35

Runtime

• APEX module that includes: ART (Android VM) Bionic (libc, libm, libdl, linker) Managed Core Library (libcore, apache-xml, boringssl, bouncycastle,

conscrypt, expat, fdlibm, icu, okhttp, ziparchive, and zlib) ICU libraries (Unicode and other internationalization support) libnativebridge (Native library support in different process architecture) libnativehelper (JNI calls support) libnativeloader (Creates linker namespaces for Java class loaders)

• Biggest change by far and was not announced on Google I/O!

36

Metadata

• The ModuleMetadata module contains metadata about the list of modules on a given device.

• The ModuleMetadata APK contains ModuleInfoProvider, which implements the getModuleInfo and getInstalledModules methods from the PackageManager API.

• This is about Stable AIDL interface support for the Modules.

37

Networking

• The Networking Components module ensures that Android can adapt to evolving network standards and allows for interoperability with new implementations.

• The Networking Components module is delivered as three APKs: IP services. (IP Client, DHCP client, Network Monitor, some utils)Captive portal login. App in charge of managing login on captive portals.Network Stack Permission Config. Defines a permission intended for use by the

Network Stack and Captive Portal Login modules.

38

Time Zone Data

• The Time Zone Data module updates daylight saving time (DST) and time zones on Android devices, standardizing both the data (which can change frequently in response to religious, political, and geopolitical reasons) and the update mechanism across the ecosystem.

• Android 10 deprecates the APK-based time zone data update mechanism (used in Android 8.1 and Android 9) and replaces it with an APEX-based module update mechanism.

39

Conсlusion

40

What was moved into modules?

• Things that needs to be updated fast: Security-related Permissions-related Politics-related

• Things that Google wants to control directly: Because they are too basic for OEM vendors Because they are too important

• Things that OEM may want to update frequently (e.g. Media Codecs)

41

How is Mainline related to Treble?

• Relatively minor technical dependencies on Treble functionality - VTS allows to enforce APEX Kernel requirements and library versions

• Overall ideological continuation of the same trend - Treble separates SoC and OEM areas of Responsibility:

• Treble separates SoC and OEM areas of Responsibility with Vendor Interface

• Treble creates stable and versioned HAL interfaces with HIDL

• Mainline limits some modules to Google-only modifications

• Mainline introduces Stable AIDL interfaces for modules

42

What should we expect next?

• It looks like Google has found a solution to the Update problem – Separate Areas of responsibility.

• Eventually most of Android OS code can become modular. That could mean a couple of things: Android OS could get regular and gradual updates. They will be smaller but

more frequent. Some modules could be developed by 3rd party developers and distributed on

Play Store. Android API can also be split into modules potentially. Each module with it’s

own version.

43

Q&A

Fedor TcymbalAndroid Solutions Architect at MeraNizhny Novgorod, RussiaE-mail: tsymbal@mera.ruSkype: ftsymbalTel: +79200267490