28-03-2017 Masterclass Mechatronics 4.0 - Indoor and outdoor localisation and positioning -...

PUBLIC

BENCHMARKING OF DIFFERENT LOCALIZATION APPROACHES

PROF. ELI DE POORTER (UGENT – IDLAB – IMEC)

27 MARCH 2017

OVERVIEW

PART 1 – Real life results

PART 2 – Objective evaluation

PART 3 – Towards scalable and accurate localization

3

PART 1: REAL-LIFE RESULTS

Let’s look at the results of the Microsoft indoor localization competition 2014

Evaluation of 22 localization solutions

Both scientific and commercial solutions

Mainly Wi-Fi based

20 evaluation points

Typical meeting rooms

4

29/03/2017

IPSN COMPETITION

Best

man

n et

al.

Kle

pal

et

al.

Lao

udia

s et

al.

Adl

er

et

al.

Li

et

al.

Lazi

k et

al.

Zou

et

al.

Ash

ok

et

al.

Nik

ode

m

et

al.

Den

tam

aro

et

al.

Ferr

az

et

al.

Li

et

al.

Mar

cale

tti

et

al.

Xia

o et

al.

Abr

uda

n et

al.

Sark

et

al.

Pirk

l et

al.

Sch

mid

et

al.

Zha

ng

et

al.

Gh

ose

et

al.

Jian

g et

al.

Sela

vo

et

al.

Series1 0.7 1.6 2 2 2 2.1 2.2 2.4 2.6 2.7 2.8 3.2 3.5 3.7 3.8 3.9 4 4 4.9 5.2 8.9 10

0123456789

1011

Lo

calizati

on

err

or

Microsoft 2014 IPSN indoor localization

competition

BEST LOCALIZATION SOLUTION

6

“D. Lymberopoulos, J. Liu, X. Yang, R.R. Choudhury, V. Handziski, S. Sen, F. Lemic, J. Büsch et al.: A

Realistic Evaluation and Comparison of Indoor Location Technologies: Experiences and Lessons

Learned, IPSN 2015, 13‐17 April, 2015, Seattle, USA”

Typical realistic

attainable accuracy = 2 m

Best

man

n et

al.

Kle

pal

et

al.

Lao

udia

s et

al.

Adl

er

et

al.

Li

et

al.

Lazi

k et

al.

Zou

et

al.

Ash

ok

et

al.

Nik

ode

m

et

al.

Den

tam

aro

et

al.

Ferr

az

et

al.

Li

et

al.

Mar

cale

tti

et

al.

Xia

o et

al.

Abr

uda

n et

al.

Sark

et

al.

Pirk

l et

al.

Sch

mid

et

al.

Zha

ng

et

al.

Gh

ose

et

al.

Jian

g et

al.

Sela

vo

et

al.

Series1 0.7 1.6 2 2 2 2.1 2.2 2.4 2.6 2.7 2.8 3.2 3.5 3.7 3.8 3.9 4 4 4.9 5.2 8.9 10

0123456789

1011

Lo

calizati

on

err

or


competition

BEST LOCALIZATION SOLUTION

7




Custom hardware: 2.4GHz Phase Offset

Accuracy = 0.72m

Typical realistic

attainable accuracy = 2 m

Best

man

n et

al.

Kle

pal

et

al.

Lao

udia

s et

al.

Adl

er

et

al.

Li

et

al.

Lazi

k et

al.

Zou

et

al.

Ash

ok

et

al.

Nik

ode

m

et

al.

Den

tam

aro

et

al.

Ferr

az

et

al.

Li

et

al.

Mar

cale

tti

et

al.

Xia

o et

al.

Abr

uda

n et

al.

Sark

et

al.

Pirk

l et

al.

Sch

mid

et

al.

Zha

ng

et

al.

Gh

ose

et

al.

Jian

g et

al.

Sela

vo

et

al.

Series1 0.7 1.6 2 2 2 2.1 2.2 2.4 2.6 2.7 2.8 3.2 3.5 3.7 3.8 3.9 4 4 4.9 5.2 8.9 10

0123456789

1011

Lo

calizati

on

err

or


competition

CLEAR TECHNOLOGY WINNER?

8





Off-the-shelf Wi-Fi

Best

man

n et

al.

Kle

pal

et

al.

Lao

udia

s et

al.

Adl

er

et

al.

Li

et

al.

Lazi

k et

al.

Zou

et

al.

Ash

ok

et

al.

Nik

ode

m

et

al.

Den

tam

aro

et

al.

Ferr

az

et

al.

Li

et

al.

Mar

cale

tti

et

al.

Xia

o et

al.

Abr

uda

n et

al.

Sark

et

al.

Pirk

l et

al.

Sch

mid

et

al.

Zha

ng

et

al.

Gh

ose

et

al.

Jian

g et

al.

Sela

vo

et

al.

Series1 0.7 1.6 2 2 2 2.1 2.2 2.4 2.6 2.7 2.8 3.2 3.5 3.7 3.8 3.9 4 4 4.9 5.2 8.9 10

0123456789

1011

Lo

calizati

on

err

or


competition

CLEAR TECHNOLOGY WINNER? NO…

9





Best

man

n et

al.

Kle

pal

et

al.

Lao

udia

s et

al.

Adl

er

et

al.

Li

et

al.

Lazi

k et

al.

Zou

et

al.

Ash

ok

et

al.

Nik

ode

m

et

al.

Den

tam

aro

et

al.

Ferr

az

et

al.

Li

et

al.

Mar

cale

tti

et

al.

Xia

o et

al.

Abr

uda

n et

al.

Sark

et

al.

Pirk

l et

al.

Sch

mid

et

al.

Zha

ng

et

al.

Gh

ose

et

al.

Jian

g et

al.

Sela

vo

et

al.

Series1 0.7 1.6 2 2 2 2.1 2.2 2.4 2.6 2.7 2.8 3.2 3.5 3.7 3.8 3.9 4 4 4.9 5.2 8.9 10

0123456789

1011

Lo

calizati

on

err

or


competition

BETTER TO INSTALL YOUR OWN INFRASTRUCTURE?

10




Existing infrastructure

Installation of additional anchor points

Best

man

n et

al.

Kle

pal

et

al.

Lao

udia

s et

al.

Adl

er

et

al.

Li

et

al.

Lazi

k et

al.

Zou

et

al.

Ash

ok

et

al.

Nik

ode

m

et

al.

Den

tam

aro

et

al.

Ferr

az

et

al.

Li

et

al.

Mar

cale

tti

et

al.

Xia

o et

al.

Abr

uda

n et

al.

Sark

et

al.

Pirk

l et

al.

Sch

mid

et

al.

Zha

ng

et

al.

Gh

ose

et

al.

Jian

g et

al.

Sela

vo

et

al.

Series1 0.7 1.6 2 2 2 2.1 2.2 2.4 2.6 2.7 2.8 3.2 3.5 3.7 3.8 3.9 4 4 4.9 5.2 8.9 10

0123456789

1011

Lo

calizati

on

err

or


competition

MORE TECHNOLOGY IS BETTER?

11




WiFi+IMU Fingerprinting

WiFi Fingerprinting

(same infrastructure)

IPIN 2014 MAIN CONCLUSIONS

No clear technology winners

Accuracy depends on a good combination of technology, algorithms and localization

technique (e.g. ToA vs RSSI)

Not clear how repeatable these results are, and if they also apply to other application

domains

12

IPSN 2016 (TWO YEAR LATER)

13

3D

localization

with custom

infrastructure

2D

localization

with existing

infrastructure

OPEN QUESTION

How objective is the evaluation process?

Different technologies

Differences in cost, range, form factor, etc.

Limited set of evaluation metrics

Response time, room accuracy, energy consumption, scalability, etc.

Taking into account calibration efforts

Number of calibration points for model and/or fingerprint creation

Sensitivity to new environments

19

OVERVIEW



Standardized evaluation metrics

Standardized evaluation scenarios

Standardized data traces


20

STANDARDIZED EVALUATION METRICS

Accuracy does not tell the whole storyUnfortunately, it is often all you will get from many solution providers…

What about

Cost, scalability, energy consumption, response time (real-time nature), deployment complexity, recalibration needs, interference robustness, compactness, etc.

Next: some example observed trade-offs from commercial and/or research localization solutions, as measured in our industrial localization testbed

21

WHY ARE ADDITIONAL METRICS NEEDED? (1)


Accuracy (how close is the reading to the ground truth?) vs precision (how consistent

are the readings?)

Example evaluation of a IEEE 802.15.4 solution

22

WHY ARE ADDITIONAL METRICS NEEDED? (1I)

50 % percentile

90% percentile

3.5 meter 8 meter

T. Van Haute, E. De Poorter, I. Moerman, F. Lemić, V. Handziski, A.

Wolisz, N. Wirström, T. Voigt: Comparability of RF-based Indoor

Localization Solutions in Heterogeneous Environments: An

Experimental Study, International Journal of Ad Hoc and Ubiquitous

Computing, SI on Localization and Positioning for Healthcare

Applications ‐ IJAHUC 2015


Accuracy does not tell the whole storyAccuracy vs response time & energy trade-offs

Example evaluation of a IEEE 802.15.4 solution

23

WHY ARE ADDITIONAL METRICS NEEDED? (III)







Ave

rage

err

or

(m)

# measurements


24

WHY ARE ADDITIONAL METRICS NEEDED? (IV)

Larger preamble settings are better for

obtaining higher accuracies

Larger preamble settings result in

longer ranging durations

Accuracy does not tell the whole storyAccuracy vs response time & energy trade-offs

Example evaluation of an UWB solution (DW1000)


Scalability is impacted by system design choices

Impacted by e.g. system settings, localization approach, MAC protocol, etc.

Example below: scalability of UWB localization using the DW1000

25

WHY ARE ADDITIONAL METRICS NEEDED? (V)

Matteo Ridolfi, Samuel Van de Velde,

Heidi Steendam, Eli De Poorter

“Analysis of the scalability of UWB

indoor localization solutions for high

user densities”, under review


Common approach needed to evaluate localization approaches and to ensure results are

comparable

EVARILOS (http://www.evarilos.eu/)

Evaluation handbook

Evaluation tools

Data repositories

26

EU EVARILOS PROJECT (2012-2015)

http://www.evarilos.eu/


EVARILOS benchmarking handbook

Generic evaluation methodology

Metric definition

Point & room accuracy, response time, energy consumption, interference robustness, scalability, etc.

Evaluation guidelines

Evaluation point selection, metric calculation, etc.

Definition of reference scenarios

27



Joint ISO/IEC 18305 standard on “Test and evaluation of localization and tracking

systems"

ISO (the International Organization for Standardization)

IEC (the International Electrotechnical Commission)

Responsible committee

ISO/IEC JTC1/SC31/WG5

28

INTERNATIONAL STANDARDS


Joint ISO/IEC 18305 standard on “Test and evaluation of localization and tracking systems"

2 types of standards

Single technology equipment validations

Mainly to validate if the performance of a radio is sufficient for localization purposes

Overall system performance evaluation

Wide range of evaluation scenarios

Including e.g. crawling scenarios

Multiple metrics

Aligned with the EVARILOS metrics

Multiple technologies

Not only radio technologies

http://www.iso.org/iso/catalogue_detail.htm?csnumber=62090

29

INTERNATIONAL STANDARDS (II)

http://www.iso.org/iso/catalogue_detail.htm?csnumber=62090


Joint ISO/IEC 18305 standard on “Test and evaluation of localization and tracking

systems"

Example metrics

Floor detection probability, zone detection probability, means of various errors, covariance

matrix of the error vector, variances of the magnitudes of various errors, RMS values of

various errors, absolute mean of the error vector, circular error 95%, vertical error, spherical

error, latency, set-up time, …

30

INTERNATIONAL STANDARDS (III)


EVARILOS

Contains industry relevant criteria (interference robustness, scalability, deployment complexity, etc.).

The defined metrics are not always specified in sufficient detail

ISO/IEC

Describes metrics in great (mathematical) detail

Lists many varieties of similar (accuracy) metric (e.g. bounded accuracy, accuracy rings, directional

accuracy, etc.)

Too many?

Lacks many industry relevant metrics (cost, scalability, interference robustness, etc.)

Both are still not frequently used….

31

COMPARISON

OVERVIEW







32

STANDARDIZED EVALUATION SCENARIOS

Locations near walls / corners consistently perform worse

33

WHY ARE SCENARIOS NEEDED? (I)


Accuracy heatmap

Example from WiFi fingerprinting in an industrial environment

34

WHY ARE SCENARIOS NEEDED? (II)


How many evaluation points to use?

Example evaluation of the accuracy of Wi-Fi localization in Sint-Jozefs kliniek, Izegem

35

WHY ARE SCENARIOS NEEDED? (III)

35

Num

ber

of eva

luat

ion p

oin

ts

Median accuracy2 meter 3 meter

Depending on the # and location of the evaluation

points, the median accuracy can differ by more than

1 meter.

Even when using 30 or more evaluation points!



30 evaluation pointsEli De Poorter, Tom Van Haute, Eric

Laermans, Ingrid Moerman, “Benchmarking of

Localization Solutions: Guidelines for the

Selection of Evaluation Points”, IEEE

Transactions on Vehicular Technology,

January 2017.


Impact of tag placement?

Example evaluation of tag placement for UWB localization (DW1000)

WHY ARE SCENARIOS NEEDED? (IV)


Impact of activity types?

Example evaluation of expected accuracies for different activities for UWB localization (DW1000)

WHY ARE SCENARIOS NEEDED? (V)


Environment strongly impacts the choice of optimal algorithm

Example: evaluation of IEEE 802.15.4 fingerprinting and ToA in different buildings

38


▪ Example

▪ Same algorithms

▪ Different environments

2 2.75

8.13

4.357.16 6.4

Brick office Plywood

office

Industrial

Fingerprinting Time of arrival








EVARILOS project

Mostly focused on evaluation of point accuracy, by providing evaluation point selection guidelines

Focus on repeatability and comparability

ISO/IEC

Mostly focused on track & tracing

Several realistic mobility scenarios that include activities (jumping, crawling, etc.)

Firefighters, office traffic patterns, etc.

EVAAL

Mostly focused on track & tracing

Designed for ambient assisted living scenarios

SCENARIO SOURCES

OVERVIEW







40

STANDARDIZED DATA TRACES

Database with pre-collected wireless traces

Benefits

Reduced complexity for designing new solutions

Objective comparisons between algorithms by giving them the same input traces

41

WHY ARE DATA TRACES NEEDED?

T. Van Haute, E. De Poorter, I. Moerman, F. Lemić, V.

Handziski, A. Wolisz, N. Wirström, T. Voigt Performance

Analysis of Multiple Localization Solutions in a Healthcare

Environment, International Journal of Health Geographics,

January 2016

▪ Example

▪ Same input

▪ Different algorithms


Database with pre-collected wireless traces

From multiple environments

Brick office, ply-wooden wall office, industrial, railway station,

hospital, underground mine

From multiple technologies

Wi-Fi, Zigbee, Bluetooth

From different configurations

# anchor points, tx power, frequency, with or without interference

Suitable for RSSI & ToA localization algorithms

Java & matlab API available

42



WiFi AP + IMU data from 4 buildings

Most suited for fingerprinting

43

EVAAL / IPIN 2015/2016


National Institute of Standards and Technology (NIST)

Automatic evaluation of commercial & research localization solutions using pre-collected

wireless traces & ISO/IEC scenarios

Traces from 5 buildings totaling 30000 m2

Factory, shops, office, underground, tower

Also includes sensor data (IMU)

Accelerometer, gyroscope, etc.

But no time-of-arrival data

Automatic evaluation platform available mid-2017

In collaboration with imec-IDLab

44

NIST


EVARILOS

Multiple environment types (including industrial-like)

Multiple technologies surveyed using the same measurement locations

No IMU data (only information from discrete locations)

EVAAL

Very large scale data collection session (WiFi AP + IMU)

Already outdated

NIST

Very large scale data collection (including IMU data!)

Only for WiFi fingerprinting

Ongoing competition (https://perfloc.nist.gov/) aligned with ISO/IEC

45

COMPARISON

https://perfloc.nist.gov/

OVERVIEW




46

INDOOR POSITIONING provides

accurate knowledge

about the current shopping location

of each customer,

allowing the retailer to provide:

relevant information

@ the right time

@ the right place

EXAMPLE: LUNAR ICON PROJECTINDOOR POSITIONING IN THE RETAIL SECTOR

Add value through functionality

and

user-experience

1. Enhance customer loyalty

2. Create up-selling and cross-selling

opportunities based on real life

behavior

3. Real-life store insights

Customer Retailer

“Reward me”

“Know my history”

“Be relevant”

“Engage me”

“Understand my needs”

“Make it easy”

WIN-WIN

LUNAR ICON PROJECT

Scalable300 persons in 2000 m2 through scalable UWB

localization and networking algorithms

AccurateUp to 30 cm through UWB technology, antenna

design and information fusion

User-centered localizationMock-up based co-creation, ethnographic

observation, tipping point determination, ….

In-situ PoC demonstrations • Colruyt large-scale replica supermarket

• Decathlon flagship innovation store

Innovation outcomesLUNAR ICON PROJECT

LUNAR ICON PROJECT

Improving UWB localization accuracy, scalability & costs

Trade-off analysis based on user centric research

Taking into account

Cost

Scalability

Device size

Battery

DIGCOM

LUNAR ICON PROJECT

54

DERIVING USER & BUSINESS REQUIREMENTS / CONFLICTS

LUNAR ICON PROJECT

55

CUSTOMER EYE TRACKING DEVICES

LUNAR ICON PROJECT

56

IMPACT OF HARDWARE ON LOCALIZATION SOLUTIONS

LUNAR ICON PROJECT

To cover large buildings, multiple cells need to be supported

Need for

Distributed TDMA MAC protocols

Distributed slot allocation algorithms

Cell roaming procedures

COVERAGE

t

Beacon Beacon

Contention Access Period

distributed over

Contention Free Period

Superframe

Contention Access Period

distributed over

Contention Free Period

Custom multi-hop TDMA MAC protocol allowing

roaming and joining of new devices

LUNAR ICON PROJECTANTENNA DESIGN

Two-element ESIW cavity-backed slot antenna Reflection coefficient of the current omnidirectional

tag antenna system as a function of frequency.

LUNAR ICON PROJECT

Inclusion of inertial measurement unit (IMU) sensors

Combination of accelerometers, gyroscope,

magnetometer, barometer, etc.

Sensor fusion is used to combine this information with

UWB position estimates

Anchor node selection algorithms

LOS vs NLOS

Inclusion of map information

Tracking likely paths through e.g. particle filters

ACCURACY IMPROVEMENTS

OVERVIEW




61

CONCLUSIONS

Localization performanceIs influenced by algorithm, technology, environment, # of anchor points, ...

Should include accuracy, but also precision, response time, energy consumption, etc.

Should be expressed using standardized metrics (EVARILOS, ISO/IEC)

Performance analysis in multiple conditions & environments is time-consumingBut crucial!

Efficient analysis is possible, even in a single day!Using automated benchmarking tools (IMEC-IDLab)

Using pre-collected data traces (IMEC-IDLab, NIST)

Nowadays: strong focus on localization solutions, not just algorithmsLocalization-communication MAC co-design

User centric system design

Focus on trade-offs between scalability, accuracy, cost, range, coverage, etc.

62

PUBLIC63

28-03-2017 Masterclass Mechatronics 4.0 - Indoor and outdoor localisation and positioning -...

Technology

Transcript of 28-03-2017 Masterclass Mechatronics 4.0 - Indoor and outdoor localisation and positioning -...