Skim down AGL Application Framework to bridge …...Skim down AGL Application Framework to bridge...
Transcript of Skim down AGL Application Framework to bridge …...Skim down AGL Application Framework to bridge...
Skim down AGL Application Framework to bridge AGL with hard realtime subsystems
Dresden AGL/AMM Oct/2018Linux Realtime Technical Lead
Oct-2019Micro Binder Architecture 2
Who are we?
Oct-2019Micro Binder Architecture 3
Where ?
LORIENT
vannes
● https://iot.bzh/en/● http://github.com/iotbzh
Oct-2019Micro Binder Architecture 4
Oct-2019Micro Binder Architecture 5
Bridging AGL Application Framework with hard realtime subsystems
Introduction
Oct-2019Micro Binder Architecture 6
How to broaden AGL Connectivity
● Up to the Cloud– Keeping critical services in realtime contexts
● Down to:– (very) smaller, hard realtime systems
You’re at the right place !
Feb-2018 7
Bridging AGL Application Framework with hard realtime subsystems
Oct-2019Micro Binder Architecture 8
Plan
● The RT Quest– What is RealTime in an OS ?
– Soft vs Hard Realtime in Linux
– Which Automotive Apps need RT ?
– Impacts on Linux RT Applications
– Turn “ON” Linux RealTime● Porting to lighter OS
– Binder & bindings
– Impacts on binder
– Defining a boundary to portability
– Inter SOC communication● Focus on the can-low-level binding● To infinity (→smaller) and beyond !● Suggested roadmap
Oct-2019Micro Binder Architecture 9
Plan
● The RT Quest– What is RealTime in an OS ?
– Soft vs Hard Realtime in Linux
– Which Automotive Apps need RT ?
– Impacts on Linux RT Applications
– Turn “ON” Linux RealTime● Porting to lighter OS
– Binder & bindings
– Impacts on binder
– Defining a boundary to portability
– Inter SOC communication● Focus on the can-low-level binding● To infinity (→smaller) and beyond !● Suggested roadmap
Oct-2019Micro Binder Architecture 10
The RT Quest
● Path to RealTime is not straightforward !
Oct-2019Micro Binder Architecture 11
Some urban legends about RealTime
● “I wont’t get RealTime issues, and won’t have to deal with thread priorities, as long as my CPU load keeps low”
Hey buddy, have you ever heard about:
Interrrupts ?
Ping flood ?
Unpredictive (because not tested) time spent in code branches ?
Device latencies with poor driver implementations (eg, SD card) ?
Freezes due to tight loops ?
Feb-2018 12
Some urban legends about RealTime
● “I do not care memory allocations, I have got 4 Go of available RAM !”
Feb-2018 13
What is RealTime in an OS?
● RealTime means “on time” and not “faster”– Realtime is about predictability
● Typically Realtime address following concerns– 80% => Could my execution code be disturbed ?
● If then:– For how much time ?– How often ?– By who ?
– 20% => How big is my latency● Time lost within an external event, and the time my code can
start to handle it ?● Maximum latency fluctuation ?
Feb-2018 14
Linux & Soft/Hard Realtime
● Soft realtime● Periodical tasks/events of several milliseconds● Some acceptable unpredictable delays (10/100 ms)● Often implemented by resource controls● Latency of few ms with exceptionnal unpredictable
fluctuation +-10ms
● Hard realtime● Total Latency magnitude of 10/100 us● Predictable and short delays (< 250ms)
● Current vanilla Linux kernel is Soft RealTime
Feb-2018 15
Which Automotive Apps need RT ?
● Soft Realtime● Data Acquisition● Audio/Video
● Hard Realtime● Cluster● Emergency/Safety signal● Dead reckoning● LIDAR acquisition● Trajectory control● ...
Driv erle ss c ars
Feb-2018 16
RT Options inside AGL
Cluster
Carte handling
Localistion management
POI
CAN GPS
Geopositioning Virtual Signal
Multi ECU & Cloud Aware Architecture
Entertainement
CAN-BUS Virtual Signal
Gyro, AcelerometerCAN-BUS
LIN-BUS
Engine-CAN-BUS
ABS
Transport & ACL
Head Unix
Direction Indication
Cloud
Log Analytics
No-SQL Engine
Statistics & Analytics
Transport & ACL
My Car Portal
Paiement
Subcriptions
Preference
Preferences &
Custumisation
MongoDB Engine
Paiement Service
Cluster Virtual Signal
Transport & ACL
Navigation Service
Maintenance Portal
Know Bugs
Maintenances
Service Packs
Soft RealTimeHard RealTime
Feb-2018 17
Linux RT Application Impact
● Standard Linux: a simple “Ping Flood“ will lag applications.● Linux network IRQs preempt applications too often and for too long, which
significatly increase the latency.
● PREEMPT_RT reduces scheduling latency● Replaces most spinlock by mutexes● Support threaded IRQs● Supports hight resolution timers
Feb-2018 18
Turn “ON” Linux RealTime.
● Objectives● Decrease Application Latencies● Garanty that high priority tasks will not be bothered by
lower priority ones.● Make sure interrupts cannot lag your critical apps.
● Soft Realtime (Standard Kernel)● Container, Ggroups, ...
● Hard Realtime (Kernel must be patched)● PREEMPT-RT● IPIPE+Xenomai
Feb-2018 19
Preempt_RT vs Xenomai
● Xenomai● Xenomai supports to legacy RT non-POSIX applications (eg: VxWorks, PSOS) through
skins● Dual Kernel solution brings more performances when no more than 4 cores run RT
threads● More confidence on the whole RT application (eg: /proc/xenomai statistics)● Integrated debug features● Misses some critical Unix develoment tools (eg: Valgrind, LTTng)
● Preempt_RT● Almost Vanilla Linux (no API/ABI changes)● Continuous testing in OSADL QA farm● No need for extra userspace libraries● Less confidence in app, harder to debug, needs extra code for RT monitoring
Feb-2018 20
Xenomai Dual Kernel Mode
Being replaced by Dovetail
Feb-2018 21
Preempt_RT Latency
Source: http://www.emlid.com/raspberry-pi-real-time-kernel Latency (us)
Oct-2019Micro Binder Architecture 22
Xenomai Latency
Feb-2018 23
Xenomai & Prempt_RT convergence
● Xenomai 3.x offer dual kernel and PREEMPT_RT option (with the same high level API and skins)
● Dual kernel latency remains significantly better● Some options are not available in both
solutions (eg: RtNet only runs on dual kernel)
Feb-2018 24
RT Kernel is only a start (1/4)PREEMPT_RT kernel tuning
● Realtime requires more kernel tuning and clean behaviours on the application side.
● Enable CONFIG_PREEMPT_RT_FULL & CONFIG_HIGHRES_TIMERS to get <1ms precision
● Disable CONFIG_CPU_FREQ !● Might conflict with power management● … and disable THERMAL framework
Feb-2018 25
RT Kernel is only a start (2/4)Impact on Applications
● There are strict rules to follow and actions to take in the application:● Stack pre-faulting● Virtual Memory locking● Fine tuning of threads priorities● malloc() and friends chasing, to avoid page faults (can be difficult with
some C++ libraries, eg BOOST)● Forbidden usage of system(); popen(); execve() … in runtime.● Monitoring run-away threads (ie, tight loops in RT contexts) to prevent
system hanging (and to allow debugging).● clock_nanosleep is your friend, for writing periodic tasks● Carefull initialization parameters of pthread_mutex, default ones do not
have PTHREAD_PRIO_INHERIT !● Fancy some LTTng sessions ? (does not work with IPIPE)
Feb-2018 26
RT Kernel is only a start (3/4)Choose the right clock !
● CLOCK_REALTIME is NOT RealTime ! – Dayligt savings
– NTP adjustements (even THE big jump, when enabled)● CLOCK_MONOTONIC (affected in speed by NTP)
or● CLOCK_MONOTONIC_RAW will only vary depending on the quality
of the quartz and voltage and temperature compensation quality
Feb-2018 27
RT Kernel is only a start (4/4)Impact on Applications :
Last but not least
● Not everything can be RT● Providing high priority to some task means
than the other will inherit of low priority● Base you flow on lock (semaphore) and not on
thread priority● Get rid of any spin lock
Feb-2018 28
Impact on Applications : C specific issues (& workarounds)
● malloc()/realloc() do not lead to pagefault always, (through sys_brk() or sys_mmap_pgoff() because of internal memory pool of the glibc
● Thus, a RT ‘leak’ may be hard to reproduce● Using GDB with a breakpoint on malloc() is usually
sufficient● Another technique, less intrusive, is to use Memory
Allocation Hooks of the glibc.● Some companies allow malloc() for initializations, and
always forbid the free() !
Feb-2018 29
C++ specific issues
● In C++, dynamic allocations are not always explicit
● Example: std::vector growing● In some extra libraries (eg, boost), memory
allocations may be completly out of control (in addition of alien-only-friendly backtraces)
Feb-2018 30
Plan
● The RT Quest– What is RealTime in an OS ?
– Soft vs Hard Realtime in Linux
– Which Automotive Apps need RT ?
– Impacts on Linux RT Applications
– Turn “ON” Linux RealTime● Porting to lighter OS
– Binder & bindings
– Impacts on binder
– Defining a boundary to portability
– Inter SOC communication● Focus on the can-low-level binding● To infinity (→smaller) and beyond !● Suggested roadmap
Feb-2018 31
Bridging AGL Application Framework with hard realtime subsystems
● Cars are made of several systems with different connectivity or responsiveness constraints
● Communication between cluster and subsystems involves several protocol stacks
● Bringing access security to embedded control systems
● Low-level filtering of devices events is leaner than in high level application(s)
Oct-2019Micro Binder Architecture 32
What is the AGL binder
● Bound to a systemd service● Dynamic loader of applications (bindings)● Embeds:
– a tiny http server
– websockets based on Unix named sockets● Provides an event loop and timers API
Feb-2018 33
Use case of the binder
cloudAPPLI
BINDERA
BINDER
BAPPLI
BINDERC
BINDING
BINDERD
BINDING
Connected carMaster
ECU
HTTPS+WSS/TCP HTTPS+WSS/TCP
WS/UDS
WSS/TCP
WS: WebSocketWSS: WS SecuredUDS: Unix Domain
SocketECU: Engine Control
Unit
BINDERBINDING
Oct-2019Micro Binder Architecture 34
Binder/Binding model
●Binder–Container process–Transport–Security–Standardized Sync/Async API–Expose API through HTTP or WebSocket
● Protected by token●Binding
–One or more API published through binder–Provided as:
● A native library weekly coupled (threading allowed)● A proxy to a remote service
APPLICATION
BINDERafb-daemon
BINDING
BINDING
BINDING
SECURITYCONTEXT
http ws
BINDERafb-daemon
BINDING
SECURITYCONTEXT
A
B
Oct-2019Micro Binder Architecture 35
Some existing binders & bindings in AGL
● agl-service-audio-4a– Highlevel API
– Softmixer
– 4a-HALs
– AlsaCore● agl-service-can-low-level● agl-service-iiodevices
Oct-2019Micro Binder Architecture 36
Porting the Binder to a micro-OS
● Leave the Unix world– No unix sockets
– Inter-binding messages should be lighter, less footprint, no big json data.
– Binary json could be an option
– Avoid copying data when possible● Break the existing dependency to libsystemd
– But keep the timers and event loop API● Need to have an OS abstraction layer for non POSIX systems● Add new communication transport with other system chips
? or shared memory
Oct-2019Micro Binder Architecture 37
Inter SOC communication
● Depends on the subsystem connectivity– Subsystem on same PCB → shared memory or
dedicated hw channel (i2c …)
– Remote subsystem → specific bus, or IP in the best case
● Access token can given by an external authentification server
● Can use the encryption as well
Oct-2019Micro Binder Architecture 38
Defining a boundary
● POSIX may be a too strong prerequisite (not available for all the embedded systems, or only partial support
● Assuming Glibc (or µCLibc) seems fair enough● Incomplete libc implies to have more OS
abstraction
or
µClibc
Glibc
Feb-2018 39
Plan
● The RT Quest– What is RealTime in an OS ?
– Soft vs Hard Realtime in Linux
– Which Automotive Apps need RT ?
– Impacts on Linux RT Applications
– Turn “ON” Linux RealTime● Porting to lighter OS
– Binder & bindings
– Impacts on binder
– Defining a boundary to portability
– Inter SOC communication
– Security● Focus on the can-low-level binding● To infinity (→small) and beyond !● Suggested roadmap
Oct-2019Micro Binder Architecture 40
Focus on the can-low-levelbinding
41
CAN agent base on AGL framework
● Clear Isolation ● Low level CAN operation only depends on equipment● High level business logic dedicated to applications
● Security Built In● Navigation APP may access GeoLocation but not Telephony● Implement statistic/counter to monitor unexpected behavior
● Leverage AGL framework● API transparency for client applications● Reuse existing technology (faster, cheaper, safer)
42
AGL CAN Mapping
Binder
CAN Low Level Binding(s)
● Decoding / Encoding● Authentication / Crypto / Firewalling● Transaction (set… ack ...)● Stats & Maths● Caching (low freq. Signals, get() call)● Debug
CAN High Level Binding(s)
● Logic● Aggregation (« vehicle.doors.any.open »)● Advanced Ops
CAN BUS
CAN frames - 011010010
Signals - « vehicle.doors.left.open »(Binder Events)
UI
Publish Subscribe
43
Low Level Binding
● Low Level Binding● Binary encoding decoding● Generate Application Friendly Signal Name & Values
● Close to Automatic Code Generation● OpenXC CAN Vector definition in JSON● Other CAN analysers formats, as CANoe XML (TBD)
● Include Basic Filtering & Statistic● GT/GE LT/LE● Timer, Cycle, Timestamps● Counter: last value, average, invalid ID, …
● Can be ship to developer as binary only
44
From OpenXC to AGL CAN Binding
CAN Low Level Binding(Shared Library)
OPENXC Signal
Description(JSON)
Low LevelBinding
Static Code(AGL)
CANDecoding/EncodingC Code(vendor)
OptionalMessageHandlers(vendor)
AGLCode
Generator
C/C++Compiler
OPENXC Signal
Description(JSON)
OPENXC Signal
Description(JSON)
45
Current Binding Capabilities
● Asynchronous signalization● Basic Subscribe/Unsubscribe event model● Transport onto WebSocket/DBus
● AGL App-Framework event signaling API● handle = afb_daemon_make_event (bindif, name)● afb_req_subscribe(request, handle)● afb_req_unsubscribe(request, handle)● afb_event_push(handle, json_object_get(object));
46
Event Subscription
Waiting verb call
[Subscribe/Unsubscribe]
Find corresponding signals name
Create event handler
Make subscription or unsubscription
[No name provided]
[Name provided]
For each signal
Add recurring request to the diagnostic manager
[Diagnostic message]
[Regular CAN message]
Create Diagnostic handle
[No more signals to process]
Take all signals
Vehicle.*.doors.*50<Vehicle.speed<100
Freq=10Hz
47
OpenXC Specification● Messages
● bus - name of initialized CAN buses where this message can be found● name - The name of the CAN message● handler – Function name applied to entire raw CAN message● max_frequency – slow down high level signal frequency (ex: 1/10)● signals - A list of CAN signal objects that are in this message
● Signals● generic_name - The name of the associated generic signal name● bit_position - Staring bit position of this signal within the message● bit_size - The width in bits of the signal● factor - The signal value is multiplied by this if set● offset - This is added to the signal value if set● decoder - type & function name applied to signal bitfield to decode it● encoder - Idem to encode a value in bitfield to send over CAN bus● states - Mapping between descriptive states and values from CAN● max_frequency – slow down high level signal frequency (ex: 1/10)● send_same – if false only send signal when value change● ignore, enabled - Control parameters about activation level for a signal● writable – signal is writable on CAN
48
OpenXC Sample Definition
"0x620": { "bus": "can0",
“name”: “doors”, "signals": { "doors.driver.open": { "generic_name": "doors.driver.open", "bit_position": 78, "bit_size": 1, "factor": 0, "offset": 0, "decoder": "decoder_t::booleanDecoder"}, "doors.passenger.open": { "generic_name": "doors.passenger.open", "bit_position": 79, "bit_size": 1, "factor": 0, "offset": 0, "decoder": "decoder_t::booleanDecoder"},[...]
Oct-2019Micro Binder Architecture 49
Plan
● The RT Quest– What is RealTime in an OS ?
– Soft vs Hard Realtime in Linux
– Which Automotive Apps need RT ?
– Impacts on Linux RT Applications
– Turn “ON” Linux RealTime● RT Options inside AGL● Porting to lighter OS
– Binder & bindings
– Impacts on binder
– Defining a boundary to portability
– Inter SOC communication
– Security● Focus on the can-low-level binding● To infinity (→ small) and beyond !● Suggested Roadmap
Oct-2019Micro Binder Architecture 50
And next ?
● Going to very smaller systems● Scalability and OS-agnosticism will be the
keys● Would eventually imply new transport
implementations (serial, i2c, specific usb ...)
Oct-2019Micro Binder Architecture 51
Plan
● The RT Quest– What is RealTime in an OS ?
– Soft vs Hard Realtime in Linux
– Which Automotive Apps need RT ?
– Impacts on Linux RT Applications
– Turn “ON” Linux RealTime● RT Options inside AGL● Porting to lighter OS
– Binder & bindings
– Impacts on binder
– Defining a boundary to portability
– Inter SOC communication
– Security● Focus on the can-low-level binding● To infinity (→ small) and beyond !● Suggested Roadmap
Oct-2019Micro Binder Architecture 52
Suggested Roadmap
AGLBinder
POSIXBinder
Zephyr / STM32 POC- map POSIX calls to Zephyr syscalls
PREEMPT_RT POC
- Self-monitoring service- Priorities policies
MapSocketcan
Calls toZephyr
CAN driver
Low-levelCAN
on ZephyrBinder on
ZephyrBinder on
Zephyr
RT Binder
RT Low-level
CAN
PortableRT Binder
Lean Memory
allocations
Memfriendly
Remove theDependencyto systemd
pre-RT
Smartmessaging
Xenomai POC
- Priorities policies
RT Binder
RT Low-level
CAN
Oct-2019Micro Binder Architecture 53
Questions ?