Sensor fusion between car and smartphone

Gabor Pallergaborpaller@gmail.com

Sfonge Ltd.http://www.sfonge.com

Thanks for the memory

Car offers

● Power (no need to worry about battery drain)● Fixed position (if in car kit)● Sensors

● More sensors → more exciting applications!● Sensor fusion: the use of multiple sensors so

that they compensate each other's weaknesses

Relevant phone sensors

● GPS – if the phone is placed in the car so that it “sees” the satellites

● Accelerometer – shaking, vibration, violent movements like aggressive braking or turns

● Gyroscope – turn detection, support sensor for the accelerometer

● Compass (magnetometer) – direction (not sensitive to drift), magnetic signature

Relevant car sensors

● Speed – driving style, distance (if integrated by time)

● RPM – driving style● Built-in GPS – not subject to incorrect

placement within the car● Large number of car diagnostic indicators –

remote diagnostic → not the subject of this presentation

Obtaining car sensor data

● OBDII (like OBD-2, On-board diagnostic)● Widely deployed (mandatory in the EU since

2001/2004)● Developed for diagnostic, not for obtaining

traffic data● Nevertheless, used for a variety of applications

like fleet management

(MirrorLink)

● Emerging standard for car infotainment system-phone connection

● Developed by Car Connectivity Consortium (CCC), http://www.mirrorlink.com/

● Supported by some recent phones and cars● Even the specification costs 150 USD● Not used in the experiments in this presentation

ELM327

● ELM327 is a popular microcontroller for OBD2 access. Developed by ELM Electronics cloned by many

● On the car side, it talks OBD2 to the car's on-board computer● On the terminal side, it offers a friendly serial interface with AT

commands reminiscent of the Hayes modem command set● Actual devices often offer USB or Bluetooth interface (anything goes

that has serial line support)● Bluetooth is ideal for Android device connectivity

● Basic Bluetooth example programs like BluetoothChat can be used to communicate with the device almost without changes

● Example program: http://mylifewithandroid.blogspot.com/2013/03/data-capture-application-for-car-speed.html

ELM327 sequence

● ATD OK● ATZ ELM327 v1.4● ATE0 OK● ATM0 OK● ATL0 OK● ATH0 OK● ATS0 OK● ATSP0 OK● 0100 SEARCHING..

4100BE3EB811● 010D 410D00

● Set all to defaults● Reset all● Switch off echo● Don't store discovered protocols● Don't send LF after CR● Don't show OBD headers● Don't insert spaces into hex digits● Discover OBD protocol automatically● Query current data 0x00 (register support)

● Query current data 0x0D (speed, 0 km/h)

Cutting edge technology!

OBDII dongle

Precise fixing is crucial

First use case

● Dead reckoning-type navigation when GPS is not available

● E.g. in tunnels, parking houses, dense urban settings (urban valleys) ...

Sensors used

● Base coordinate p0 comes from the GPS or from earlier measurement

● Absolute value of velocity vector (v) comes from the speed sensor of the car

● Direction of velocity vector● Gyroscope● Compass

Gyroscope recap

● Measures rotation around 3 axes● More exactly: measures rotation speed (angular

velocity) around the axes

v x=Δ φ

Δ tΔ φ=v x Δ t

φ '=φ+ Δ φ

The problem of devices

● Newer phones are not always better● Particularly not their sensors

● It happens that a new sensor type – like gyroscope – is introduced in good quality ● Then the cost-cutting starts in later, “more

advanced” device types

Nexus S gyro

Car making 90 degree turns, device horizontal.Red: angular velocity, blue: aggregated rotation degree

Galaxy Nexus gyro

Car making 90 degree turns, device horizontal.Red: angular velocity, blue: aggregated rotation degree

Origin of the spikes: unknown

Only Nexus S measurements will be used in this presentation

The problem of drift

Compass

● Measures the device orientation wrt. the magnetic vector of the Earth

● This vector points toward the magnetic center of the Earth

● It has a component that points to the magnetic North pole – that's what we use for orientation

● Beware of the z component! (also called magnetic inclination). If the device is not held horizontally, the downward vector element influences the measurement

Car's magnetic fieldTurning around, out of car

Phone is inside the car, ignition on, car makes full circle

Find the offset

● Identify 3 points on the circle (use the gyroscope to figure out whether there was enough rotation)

● Calculate the center of the circle

● Subtract the offset

Move the circle to the centerCircle moved to the zero point of the coordinate system

(on the x-y plane)

Initial circle

Compass vs. gyroCompass Gyro

Compass is harder to calibrate and less exact but it has no drift

Tunnels

Tunnels

Effect of external magnetic fields

Not called Chain Bridge by chance ...

Parking houses

Parking houses

Urban valleys

Urban valleys

No GPS readingin the circle

Results

● Dead reckoning using speed (from car) and direction (from smartphone) provides reasonable results when GPS is not available

● Compass and gyroscope need to support each other to obtain sufficiently precise direction vector

● Compass is an extremely problematic sensor in car/urban environment that needs constant recalibration

Second use case: vibrations

● Goal: use the phone's accelerometer for something useful (car does not have any similar sensor)

● Separating motion acceleration: already discussed here: http://mylifewithandroid.blogspot.com/2012/05/better-motion-control-using.html

(can be used to classify driving habits)

● Vibration:

● Damaged road

● Vehicle faults (not discussed here)

bumpy_road

Raw vibration

Bumpy road section

After high-pass filter

Effects of slow acceleration changes (turns) are removed

Power

Bumpy road section

Luckily we have exact position ...

Speed bumps

Drivers slow down ...

Red: speed, blue: acceleration absolute value

Speed drops &vibration spikeat the same time:speed bump

Or they don't ...

Red: speed, blue: acceleration absolute value power

Not possible to detect

Results

● Localized road damages cannot always be recognized

● Damages that span longer distances may be detected more reliably

● In any case, with a large number of measurements, road damages can be mapped efficiently

Idea of a platform

● Available for Android applications● Discovers car connectivity options and sensors● Notifies applications about the connectivity

status and available sensors● Allows single or repeated sampling of sensor

data● Controlling car's systems?

Questions?