2015 Microsoft Indoor Localization Competition Dimitrios Lymberopoulos (Microsoft Research) Jie Liu...

9
2015 Microsoft Indoor Localization Competition Dimitrios Lymberopoulos (Microsoft Research) Jie Liu (Microsoft Research) Xue Yang (Intel) Niki Trigoni (University of Oxford) Anthony Rowe (CMU) Ayman Naguib (Qualcomm)

Transcript of 2015 Microsoft Indoor Localization Competition Dimitrios Lymberopoulos (Microsoft Research) Jie Liu...

Page 1: 2015 Microsoft Indoor Localization Competition Dimitrios Lymberopoulos (Microsoft Research) Jie Liu (Microsoft Research) Xue Yang (Intel) Niki Trigoni.

2015 Microsoft Indoor Localization Competition

Dimitrios Lymberopoulos (Microsoft Research)Jie Liu (Microsoft Research)Xue Yang (Intel)Niki Trigoni (University of Oxford)Anthony Rowe (CMU)Ayman Naguib (Qualcomm)

Page 2: 2015 Microsoft Indoor Localization Competition Dimitrios Lymberopoulos (Microsoft Research) Jie Liu (Microsoft Research) Xue Yang (Intel) Niki Trigoni.

Background

• 48 systems from 44 teams submitted abstracts• 28 systems showed up• 23 systems actually evaluated• Academia, Industry

• Two categories• Infrastructure-free (7 teams)

• No hardware deployment• Conference venue WiFi and phone sensors

• Infrastructure-based( 16 teams)• Custom hardware deployment

Page 3: 2015 Microsoft Indoor Localization Competition Dimitrios Lymberopoulos (Microsoft Research) Jie Liu (Microsoft Research) Xue Yang (Intel) Niki Trigoni.

Evaluation Process

• 2-day event• Day 1: Monday

• Teams were given 7 hours to setup and calibrate their systems

• Day 2: Tuesday• Each team was asked to provide the

coordinates of 20 test points • Test points’ coordinates were manually

measured using laser range finders

• Evaluation Metric• Average localization error across the 20

test points

Page 4: 2015 Microsoft Indoor Localization Competition Dimitrios Lymberopoulos (Microsoft Research) Jie Liu (Microsoft Research) Xue Yang (Intel) Niki Trigoni.

2000m2 Evaluation Area

Origin Point(0,0)

Y Ax

is

X Axis

Page 5: 2015 Microsoft Indoor Localization Competition Dimitrios Lymberopoulos (Microsoft Research) Jie Liu (Microsoft Research) Xue Yang (Intel) Niki Trigoni.

Fun Times…

Page 6: 2015 Microsoft Indoor Localization Competition Dimitrios Lymberopoulos (Microsoft Research) Jie Liu (Microsoft Research) Xue Yang (Intel) Niki Trigoni.

Towards a standardized evaluation of indoor location technologies

Multi-institutional effort NIST, TU Berlin, and iMinds at University of Ghent

Goal: create a large repository of labeled indoor location data traces 5 buildings on the NIST campus in Gaithersburg, MD 1800 test points More than 30000m2

Data Collection September – December 2015 Available through a web service

Ideas/Feedback/Suggestions Nader Moayeri ([email protected])

Page 7: 2015 Microsoft Indoor Localization Competition Dimitrios Lymberopoulos (Microsoft Research) Jie Liu (Microsoft Research) Xue Yang (Intel) Niki Trigoni.

Official Results

Sanchez e

t al.

Lazik

et al.

Time Domain

Campbell et a

l.

Quantitec G

mbH Intra

nav

Simon et a

l.

Klipp et a

l.

Chen et al.

Zengen et al.

SPIRIT Navigation

Elias e

t al.

Symington et a

l.

Zou et al.

Pirkl e

t al.

Wu et a

l.

Bata et al.

Qiu et al.

Navix

Nikodem et al.

Deora et al.

Kuo et al.

Mirs

hekari et a

l.Tata

0

5

10

15

20

25

0.2 0.31 0.39 0.6 0.73 0.739 0.78 1.37 1.63 1.9 2.63.5 3.65 3.93 4.22 4.31 4.67 4.92 5.6

7.398.24

11.24

22.99

Team

Loca

lizati

on E

rror

(m)

infrastructure-free industry

Page 8: 2015 Microsoft Indoor Localization Competition Dimitrios Lymberopoulos (Microsoft Research) Jie Liu (Microsoft Research) Xue Yang (Intel) Niki Trigoni.

1st

2nd

3rd

Infrastructure Free Infrastructure Based

$1000

$600

$400

STeAM: Sensor Tracking and MappingSanchez et al.(Institute for Transuranium Elements) 0.2m

error

ALPS: An Ultrasonic Localization SystemLazik et al.(CMU) 0.31m

error

SPIRIT NavigationBerkovich et al.(SPIRIT Navigation) 1.9m

error

Precise Indoor LocationD. Elias et al. (Fraunhofer Portugal Research Center)

2.6m error

Time DomainDewberry et al.(Time Domain) 0.39m

error

Exploring State of the Art Commercial LocalizationCampbell et al.(Univ. of Michigan)

0.6m error

Page 9: 2015 Microsoft Indoor Localization Competition Dimitrios Lymberopoulos (Microsoft Research) Jie Liu (Microsoft Research) Xue Yang (Intel) Niki Trigoni.

Point-Level View

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 200

1

2

3

4

5

6

Sanchez et al. SPIRIT Navigation Elias et al.

Evaluation Point

Loca

lizati

on E

rror

(m)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 200123456789

10

Lazik et al. Time Domain Campbell et al.

Evaluation Point

Loca

lizati

on E

rror

(m)