From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS...
Transcript of From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS...
![Page 1: From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5661afd680a51054103ca/html5/thumbnails/1.jpg)
From AGL Supervisor to platform global data collection
Sébastien DouheretTechnical Director at IoT.bzh<[email protected]>
Automotive Linux SubmitJuly 17 - 19, 2019
Tokyo, Japan
![Page 2: From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5661afd680a51054103ca/html5/thumbnails/2.jpg)
July 2019From AGL Supervisor to Platform global data collection
2
Who are we ?
● 25 Engineers dedicated to AGL● Location: Lorient/Brittany/France/Europe● Team : strong visibility in Open Source Community● People background : Telecom, Automotive or Marine
domains, Embedded developments (Intel, WindRiver, ...)
https://iot.bzh/en/ http://github.com/iotbzh
LORIENT
![Page 3: From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5661afd680a51054103ca/html5/thumbnails/3.jpg)
July 2019From AGL Supervisor to Platform global data collection
3
IoT.bzh New Office in Lorient
Large space (700m2) including Hacking Car Lab
![Page 4: From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5661afd680a51054103ca/html5/thumbnails/4.jpg)
July 2019From AGL Supervisor to Platform global data collection
4
Working on ...● Main contributions on AGL :
– Cybersecurity (Secure Application Framework)
– Development tools for AGL
(SDK, X(cross) Development System)
– Integration and board support (Renesas)
– Community Support & Documentation
– 1st AGL technical contributor since 2016
● Connected Boat project
– Consortium with 4 other partners
– Targeting work boats
– Goals : increase Security, predictive maintenance and automation of administrative documents filling
● Conferences / White papers (https://iot.bzh/en/publications)
References: Walt Miner presentationsAGL AMM 2017 https://sched.co/9KB6AGL AMM 2018 https://sched.co/DRAXAGL AMM 2019 https://sched.co/L8W3
![Page 5: From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5661afd680a51054103ca/html5/thumbnails/5.jpg)
July 2019From AGL Supervisor to Platform global data collection
5
Agenda
XDS : Apps and Bindings development (Recap XDS - Code / X-Build / Run / Debug)
Monitor a single binding
XDS Supervision (Needs, Architecture and a Demo)
Challenges & Roadmap
![Page 6: From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5661afd680a51054103ca/html5/thumbnails/6.jpg)
July 2019From AGL Supervisor to Platform global data collection
6
Agenda
XDS : Apps and Bindings development (Recap XDS - Code / X-Build / Run / Debug)
Monitor a single binding
XDS Supervision (Needs, Architecture and a Demo)
Challenges & Roadmap
![Page 7: From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5661afd680a51054103ca/html5/thumbnails/7.jpg)
July 2019From AGL Supervisor to Platform global data collection
7
Build
Develop
Deploy Debug
MonitorXDS
XDS – Quick OverviewX(cross) Development System
● Dedicated to Applications Developers
– enables development without Yocto skills
● Cross-platform build using AGL SDK
– leverages AGL/Yocto toolchain
● Deploy on development boards orQemu
● Remote debugging from IDE
● Documentation on AGL docs website:https://docs.automotivelinux.org/docs/en/master/devguides/#xcross-development-system-user's-guide
![Page 8: From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5661afd680a51054103ca/html5/thumbnails/8.jpg)
July 2019From AGL Supervisor to Platform global data collection
8
XDS – Reducing complexity
● Easy target access ● console, SSH, …
● Developer uses common IDEs● VS Code, Netbeans, Eclipse...
● Dashboard Web App to manage configurations and trigger actions
● XDS API for CI workflows & specific environments● Command line tool also available (xds-cli)
![Page 9: From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5661afd680a51054103ca/html5/thumbnails/9.jpg)
July 2019From AGL Supervisor to Platform global data collection
9
XDS – Key Features● Multi-platform : no dependencies on developer host
(Linux / Windows / MacOS)
● Easy to setupNear-zero install, no admin privileges required
● Application sources remain localCompatibility with existing IT policies (e.g. corporate backup, git, ...)
● Cross toolchain & tools embedded in a container :
● Local : run locally (local subsystem, virtual machine, docker container ...)
● On-premises : run on a local build server● Cloud : SaaS
● Leverage specific OS capabilities where applicable(e.g. Linux for Windows Subsystem, docker)
![Page 10: From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5661afd680a51054103ca/html5/thumbnails/10.jpg)
July 2019From AGL Supervisor to Platform global data collection
10
1) Install xds-server on a build server machine
2) Install xds-agent and start-it on your machine
3) Browse XDS dashboard: http://localhost:8800
4) Add a new project
5) Cross build/compile your project using either :
1) Dashboard build page
2) IDE using xds-cli
6) Deploy app
7) Source debug from IDE (xds-gdb)
XDS – Operations
![Page 11: From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5661afd680a51054103ca/html5/thumbnails/11.jpg)
July 2019From AGL Supervisor to Platform global data collection
11
XDS – Web Dashboard and CLI
Web dashboardavailable on
http://localhost:8800
Command Line Interface
![Page 12: From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5661afd680a51054103ca/html5/thumbnails/12.jpg)
July 2019From AGL Supervisor to Platform global data collection
12
XDS – Dashboard (cont.)
![Page 13: From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5661afd680a51054103ca/html5/thumbnails/13.jpg)
July 2019From AGL Supervisor to Platform global data collection
13
Agenda
XDS : Apps and Bindings development (Recap XDS - Code / X-Build / Run / Debug)
Monitor a single binding
XDS Supervision (Needs, Architecture and a Demo)
Challenges & Roadmap
![Page 14: From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5661afd680a51054103ca/html5/thumbnails/14.jpg)
July 2019From AGL Supervisor to Platform global data collection
14
Monitor : a simple built in tool
● Monitoring API is built in the binder
● Monitor API is enabled by AppFW in any binder(--monitoring option)
● Monitor only ONE binding● UI is available easily:
http://<host>:<port>/monitoring/
NOTE THE TRAILING SLASH !
![Page 15: From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5661afd680a51054103ca/html5/thumbnails/15.jpg)
July 2019From AGL Supervisor to Platform global data collection
15
Build-in Monitor – UI Overview
![Page 16: From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5661afd680a51054103ca/html5/thumbnails/16.jpg)
July 2019From AGL Supervisor to Platform global data collection
16
Monitor – outcome● Trace verbs calls
– API calls with args
– Results with data● Trace events content● View internal log calls (AFB_NOTICE et al.)● Filtering available to reduce “noise”:
– Filter per API
– Filter per trace type
![Page 17: From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5661afd680a51054103ca/html5/thumbnails/17.jpg)
July 2019From AGL Supervisor to Platform global data collection
17
Agenda
XDS : Apps and Bindings development (Recap XDS - Code / X-Build / Run / Debug)
Monitor a single binding
XDS Supervision (Needs, Architecture and a Demo)
Challenges & Roadmap
![Page 18: From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5661afd680a51054103ca/html5/thumbnails/18.jpg)
July 2019From AGL Supervisor to Platform global data collection
18
AGL Micro-Services architecture
![Page 19: From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5661afd680a51054103ca/html5/thumbnails/19.jpg)
July 2019From AGL Supervisor to Platform global data collection
19
Bindings topology agl-demo-platform HH
How to detect abnormal binding exchanges ?
Topology alre
ady
complex
for a si
mple demo !
![Page 20: From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5661afd680a51054103ca/html5/thumbnails/20.jpg)
July 2019From AGL Supervisor to Platform global data collection
20
Why Supervision is needed ?
Debugging (bug analysing) and Tunning
Telematic : Black / Save-box logger
2 Main Needs
Systems based on AGL are becoming more & more complex
Simple or traditionnal debug methods no longer adapted
Constructors will need to push some metrics to cloud
1 SolutionXDS Supervision
![Page 21: From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5661afd680a51054103ca/html5/thumbnails/21.jpg)
July 2019From AGL Supervisor to Platform global data collection
21
Supervision & Vehicle to Cloud● Data collection for debugging
● Collect system data (cpu, memory usage, …) and AGL binding exchanges (supervision / monitoring)
● Only in development mode
● Tuning● Car testing campaign (offline storage)
● Telematics● Based on acquisition CAN, vehicle signals, GPS, ... ● Externalized vehicle data analysis (failure detection)● Black box, Save box● Push useful data to the could (aggregation, retention policies)● Development and Production mode
Store caching: data saved on-board or in a cloud Time-Serie database
![Page 22: From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5661afd680a51054103ca/html5/thumbnails/22.jpg)
July 2019From AGL Supervisor to Platform global data collection
22
Supervision – ArchitectureAPPLICATION
BINDERafb-daemon
BINDING
BINDING
BINDING
http ws
AGL App 1
BINDERafb-daemon
BINDING
BINDING
BINDING
AGL Service 2
APPLICATION
BINDERafb-daemon
BINDING
BINDING
BINDING
http ws
AGL App 2
afs-supervisor
BINDERafb-daemon
BINDING
BINDING
BINDING
AGL Service 1BINDERafb-daemon
BINDING
BINDING
BINDING
AGL Service 3
BINDERafb-daemon
XDS
XDS Supervision
BINDERafb-daemon
HARVESTER
Harvester Service
TimeSeries DB
BINDERafb-daemon
collectd
Low Collector
TARGET HOST
![Page 23: From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5661afd680a51054103ca/html5/thumbnails/23.jpg)
July 2019From AGL Supervisor to Platform global data collection
23
Supervision – Target side● afs-supervisor daemon and xds-supervision
service
xds-supervision control afs-supervisor daemon to gather AGL data (events, resquests, ...)
● Low-collector service
collect system wide metrics (CPU, memory load, temperature, …)
● Harvester service
Received and format metrics to storethem in a local or remote Time Serie Database (TSDB)
![Page 24: From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5661afd680a51054103ca/html5/thumbnails/24.jpg)
July 2019From AGL Supervisor to Platform global data collection
24
Supervision – Host side
● Time Serie Database POC tested with influxDB
● Graph composer and query processing tools
Grafana draws reports based on influxdb records
● XDS Supervision dashboard : control panel (config / start / stop)and virtualization (Grafana dashboards integration)
![Page 25: From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5661afd680a51054103ca/html5/thumbnails/25.jpg)
July 2019From AGL Supervisor to Platform global data collection
25
Low Collector - collectdDeveloped in parternship with a University
● Developed by Marc-Antoine Riou (intership)
collectd integration as an AGL service● reuse collectd source code plugins (≈ 130 plugins available)● run plungins in the Low Collector binding context
Architecture :
![Page 26: From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5661afd680a51054103ca/html5/thumbnails/26.jpg)
July 2019From AGL Supervisor to Platform global data collection
26
Low-Collector - collectd : rendering
![Page 27: From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5661afd680a51054103ca/html5/thumbnails/27.jpg)
July 2019From AGL Supervisor to Platform global data collection
27
Supervision – Visualize in XDS
![Page 28: From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5661afd680a51054103ca/html5/thumbnails/28.jpg)
July 2019From AGL Supervisor to Platform global data collection
28
What about logs? Loki + promtail
● What is promtail ?● responsible for gathering logs and sending them to Loki
● What is Loki ?● Scalable & highly-available log aggregator system
(≈ Prometheus)● main server, responsible for storing logs and processing
queries
● Compatible with Grafana for rendering logs
![Page 29: From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5661afd680a51054103ca/html5/thumbnails/29.jpg)
July 2019From AGL Supervisor to Platform global data collection
29
Demo scenario● Example: 2 apps simulate requests to 2 low
level bindings and receive responses and events
Sources : https://github.com/iotbzh/agl-services-mockup
![Page 30: From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5661afd680a51054103ca/html5/thumbnails/30.jpg)
July 2019From AGL Supervisor to Platform global data collection
30
Demo...
https://iot.bzh/en/publications/xds-supervision-als-2019.htmlhttps://vimeo.com/iotbzh
![Page 31: From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5661afd680a51054103ca/html5/thumbnails/31.jpg)
July 2019From AGL Supervisor to Platform global data collection
31
Agenda
XDS : Apps and Bindings development (Recap XDS - Code / X-Build / Run / Debug)
Monitor a single binding
XDS Supervision (Needs, Architecture and a Demo)
Challenges & Roadmap
![Page 32: From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5661afd680a51054103ca/html5/thumbnails/32.jpg)
July 2019From AGL Supervisor to Platform global data collection
32
Technical challenges
● Loki, promtail powerfull and promising tools but quite recent
● Promtail written in go (go support almost done in IoT.bzh recipe)
● Time Serie Database not initially designed for embedded world (only a challenge for offline mode)
● Local processing in Car / on board (NAND writes)● Collecting too many data = intrusiveness● Tag data (ownership, priority, lifetime)● Security privileges (API V3 and multi-users)
![Page 33: From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5661afd680a51054103ca/html5/thumbnails/33.jpg)
July 2019From AGL Supervisor to Platform global data collection
33
Roadmap● Done : XDS Supervision POC (running on HH)
● Beta version (not production ready) need more time and resources to fully integrate supervision – Increase integration/validation of collectd plugins (today only 6
plugins)– reduce again intrusiveness– Improve smooth integration with AGL security– Log analyser specifically for Journalctl (systemd logs)
more investigations about Loki & promtail integration
● Other improvements: – on-board TSDB with edge load balancing algorithm– offline data collecting for Key Performance Indicator analysis
![Page 34: From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5661afd680a51054103ca/html5/thumbnails/34.jpg)
July 2019From AGL Supervisor to Platform global data collection
34
3 Things to remember
● AGL System is becoming more & more complexSupervision and log analysis mandatory to debug problem on those systems
● Don’t re-invent the wheel Integration of existing tools and reuse data format to benefit of existing UI viewer or log analyser
● Need more time and ressources Integration, Optimization, make tool robust are time consuming
![Page 35: From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5661afd680a51054103ca/html5/thumbnails/35.jpg)
July 2019From AGL Supervisor to Platform global data collection
35
Visiting us in Lorient
![Page 36: From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5661afd680a51054103ca/html5/thumbnails/36.jpg)
July 2019From AGL Supervisor to Platform global data collection
36
Q&A
Gulf of Morbihan, south of Brittany, France
![Page 37: From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5661afd680a51054103ca/html5/thumbnails/37.jpg)
July 2019From AGL Supervisor to Platform global data collection
37
References● Proof of concept sources
● https://git.automotivelinux.org/apps/agl-service-harvester● https://git.automotivelinux.org/apps/agl-service-xds-monitoring● https://github.com/iotbzh/agl-service-low-collector*● https://git.automotivelinux.org/src/xds/xds-agent
● TSDB and Logger and Rendering Tools● https://fosdem.org/2019/schedule/event/loki_prometheus_for_logs/
● https://www.influxdata.com/● https://grafana.com/● https://d3js.org/
* Not latest version, need Univesity acceptance
![Page 38: From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5661afd680a51054103ca/html5/thumbnails/38.jpg)
July 2019From AGL Supervisor to Platform global data collection
38
More References...
● IoT.bzh:
– Website: https://iot.bzh/
– Publications: https://iot.bzh/en/publications
– Github: https://github.com/iotbzh● AGL:
– Website: https://www.automotivelinux.org/
– Documentation: http://docs.automotivelinux.org/
– Sources: https://git.automotivelinux.org/