Trust as a Proxy Measure for the Quality of VGI in the Case of OSM
-
Upload
carsten-kessler -
Category
Technology
-
view
244 -
download
1
description
Transcript of Trust as a Proxy Measure for the Quality of VGI in the Case of OSM
![Page 1: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/1.jpg)
Carsten Keßler a,b and René de Groot a a Institute for Geoinformatics, University of Münster | b soon: Hunter College, CUNY
http://carsten.io | @carstenkessler
Trust as a Proxy Measure for the Quality of VGI in the Case of OSM
![Page 2: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/2.jpg)
The Idea
‣ Develop a measure to assess the degree to which a data consumer can trust the quality of a feature
![Page 3: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/3.jpg)
The Idea
‣ Develop a measure to assess the degree to which a data consumer can trust the quality of a feature
‣ Trust measure is based on a feature’s editing history
![Page 4: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/4.jpg)
The Idea
‣ Develop a measure to assess the degree to which a data consumer can trust the quality of a feature
‣ Trust measure is based on a feature’s editing history
‣ Benefits‣ Works at feature level‣ Filter features by quality‣ Spot problematic features
![Page 5: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/5.jpg)
Does this work?
Can we reliably assess the quality of a feature in OpenStreetMap based on its editing history?
![Page 6: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/6.jpg)
Does this work?
Can we reliably assess the quality of a feature in OpenStreetMap based on its editing history?
amenity = universityname = Institute for Geoinformatics
v1
![Page 7: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/7.jpg)
Does this work?
Can we reliably assess the quality of a feature in OpenStreetMap based on its editing history?
amenity = universityname = Institute for Geoinformatics
amenity = universitybuilding = yesname = Institute for Geoinformatics
v1 v2
![Page 8: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/8.jpg)
Does this work?
Can we reliably assess the quality of a feature in OpenStreetMap based on its editing history?
amenity = universityname = Institute for Geoinformatics
amenity = universitybuilding = yesname = Institute for Geoinformatics
addr:city = Münsteraddr:country = DEaddr:housenumber = 253addr:street = Weseler Straßebuilding = yeswheelchair = limited
v1 v2 v3 …
![Page 9: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/9.jpg)
OSM Heatmap Kudos: Johannes Trame
![Page 10: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/10.jpg)
OSM Provenance Ontology
http://carsten.io/osm/osm-provenance.rdf
prv:Tag
includesEdit
Changeset prv:CreationGuideline
Edit
prv:createdBy
prv:precededBy
prv:usedData
NodeState
WayState
prv:DataCreation User
prv:performedBy
changesGeometryaddsTag
removesTag
changesValueOfKey
rdfs:Literal
prv:DataItem
prv:HumanActor
subClassOfhasTagFeatureState
![Page 11: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/11.jpg)
Does this work?
‣ Get a first idea whether this is a viable approach‣ Compare results of
‣ a simple trust measure and‣ observed feature quality
‣ Is there a correlation between the two?
![Page 12: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/12.jpg)
Study area:Münster’s old town
![Page 13: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/13.jpg)
Feature Selection
![Page 14: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/14.jpg)
Feature Selection
‣ Re-mapping the whole district was not feasible
![Page 15: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/15.jpg)
Feature Selection
‣ Re-mapping the whole district was not feasible‣ Up to 100 features were manageable
![Page 16: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/16.jpg)
Feature Selection
‣ Re-mapping the whole district was not feasible‣ Up to 100 features were manageable ‣ Selection based on minimum number of versions
![Page 17: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/17.jpg)
Feature Selection
‣ Re-mapping the whole district was not feasible‣ Up to 100 features were manageable ‣ Selection based on minimum number of versions‣ 74 features with 6+ versions
![Page 18: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/18.jpg)
74 features selected
![Page 19: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/19.jpg)
Trust measure
![Page 20: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/20.jpg)
Trust measure
‣ Positive factors:‣ Versions ‣ Users ‣ Indirect confirmations =
edits in the direct vicinity (50m)
![Page 21: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/21.jpg)
Trust measure
‣ Positive factors:‣ Versions ‣ Users ‣ Indirect confirmations =
edits in the direct vicinity (50m)
‣ Negative factors:‣ Tag corrections‣ Rollbacks
![Page 22: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/22.jpg)
Trust measure (contd.)
‣ Classification for each factor: 5 equal classes‣ Combined into one classification‣ Equal weights
![Page 23: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/23.jpg)
Trust measure
![Page 24: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/24.jpg)
Field Survey
‣ Thematic accuracy 4 classes:
1. Main tag wrong
2. Other tags wrong
3. Thematic ambiguities
4. Thematically correct
![Page 25: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/25.jpg)
Field Survey
‣ Thematic accuracy 4 classes:
1. Main tag wrong
2. Other tags wrong
3. Thematic ambiguities
4. Thematically correct
‣ Results:
‣ 6 features (~8%) ‣ 2 features (~3%)‣ 9 features (~12%) ‣ 57 features (~77%)
![Page 26: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/26.jpg)
Field Survey (contd.)
‣ Topological consistency
![Page 27: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/27.jpg)
Field Survey (contd.)
‣ Topological consistency‣ Is the feature correctly
positioned relative to the surrounding features?
![Page 28: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/28.jpg)
Field Survey (contd.)
‣ Topological consistency‣ Is the feature correctly
positioned relative to the surrounding features?
‣ Results:‣ 73 out of 74 features (~99%)
![Page 29: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/29.jpg)
Field Survey (contd.)
‣ Topological consistency‣ Is the feature correctly
positioned relative to the surrounding features?
‣ Results:‣ 73 out of 74 features (~99%)
‣ Information completeness‣ TF-IDF measure to identify
relevant tags per main tag
![Page 30: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/30.jpg)
Field Survey (contd.)
‣ Topological consistency‣ Is the feature correctly
positioned relative to the surrounding features?
‣ Results:‣ 73 out of 74 features (~99%)
‣ Information completeness‣ TF-IDF measure to identify
relevant tags per main tag
‣ ~37% tags missing (avg.)
![Page 31: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/31.jpg)
Observed quality: combined results
![Page 32: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/32.jpg)
Trust measure
![Page 33: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/33.jpg)
![Page 34: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/34.jpg)
mean quality class: ~4.2
mean trust class: ~2.8
![Page 35: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/35.jpg)
Do we get the trend right?
![Page 36: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/36.jpg)
Do we get the trend right?
‣ Removed outliers‣ Kendall’s τ: 0.52 ‣ Moderate, but significant
positive correlation
![Page 37: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/37.jpg)
Conclusions
![Page 38: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/38.jpg)
Conclusions
‣ Initial study
![Page 39: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/39.jpg)
Conclusions
‣ Initial study‣ A feature’s history can determine its trustworthiness
![Page 40: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/40.jpg)
Conclusions
‣ Initial study‣ A feature’s history can determine its trustworthiness‣ Trust values correlate with observed quality
![Page 41: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/41.jpg)
Conclusions
‣ Initial study‣ A feature’s history can determine its trustworthiness‣ Trust values correlate with observed quality‣ Even with a very simple model
![Page 42: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/42.jpg)
Conclusions
‣ Initial study‣ A feature’s history can determine its trustworthiness‣ Trust values correlate with observed quality‣ Even with a very simple model‣ Outliers cannot be explained yet
![Page 43: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/43.jpg)
Tons of Future Work
![Page 44: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/44.jpg)
Tons of Future Work
‣ Extend and refine the trust model:Classification, weighting, positive vs negative aspects, …
![Page 45: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/45.jpg)
Tons of Future Work
‣ Extend and refine the trust model:Classification, weighting, positive vs negative aspects, …
‣ Social aspects: Who has edited a feature?
![Page 46: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/46.jpg)
Tons of Future Work
‣ Extend and refine the trust model:Classification, weighting, positive vs negative aspects, …
‣ Social aspects: Who has edited a feature?‣ Repeat study without spatial focus
![Page 47: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/47.jpg)
Tons of Future Work
‣ Extend and refine the trust model:Classification, weighting, positive vs negative aspects, …
‣ Social aspects: Who has edited a feature?‣ Repeat study without spatial focus ‣ How to scale the data collection?
![Page 48: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/48.jpg)
Tons of Future Work
‣ Extend and refine the trust model:Classification, weighting, positive vs negative aspects, …
‣ Social aspects: Who has edited a feature?‣ Repeat study without spatial focus ‣ How to scale the data collection?‣ Learn the trust model from the data
![Page 49: Trust as a Proxy Measure for the Quality of VGI in the Case of OSM](https://reader034.fdocuments.in/reader034/viewer/2022042606/5474a5feb4af9fb90a8b5822/html5/thumbnails/49.jpg)
Thank you!
All data used in this research © OpenStreetMap contributors.
[email protected] | http://carsten.io | @carstenkessler
Carsten Keßler | René de Groot