Sidewalk Extraction Using Aerial and Street View Images

Xinyue Ye, Ph.D., Associate Professor

Department of Landscape Architecture and Urban Planning

& Department of Geography

Texas A&M University, Email: xinyue.ye@tamu.edu

Editor-in-Chief, Computational Urban Science

Urban Data Science Lab, TAMU

Acknowledgement

• Ning, H., Ye, X*., Chen, Z., Liu, T., & Cao, T. (2021) Sidewalk Extraction Using Aerial and Street View Images. Environment and Planning B. (in press)

• Ye, X. and Lin, H. (eds.) (2020) Spatial Synthesis: Computational Social Science and Humanities, Springer.

• Jamonnak, S., Zhao, Y., AL-Dohuki, S., Curtis, A., Ye, X., Kamw, F., & Yang J. (2020) GeoVisuals: a visual analytics approach to leverage the potential of spatial videos and associated geonarratives. International Journal of Geographical Information Science. doi: 10.1080/13658816.2020.1737700

• Ye, X. and Liu, X. (eds.) (2019) Cities as Social and Spatial Networks, Springer.

Urban Data Science Lab, TAMU

Outline

• Introduction

• Related Work

• Methodology

• Results

• Discussion

Urban Data Science Lab, TAMU

Outline

• Introduction

• Related Work

• Methodology

• Results

• Discussion

Urban Data Science Lab, TAMU

Walking

Urban Data Science Lab, TAMU

Which information is missed?

Urban Data Science Lab, TAMU

Big Image Data

Urban Data Science Lab, TAMU

http://shadeparadenashville.blogspot.com/2013/12/sidewalk-project-3-walking-from.html

Outline

• Introduction

• Related Work

• Methodology

• Results

• Discussion

Urban Data Science Lab, TAMU

Semantic segmentation based on deep learning

• FCN (Long et al., 2015)

• SegNet (Badrinarayanan et al., 2017)

• U-Net (Ronneberger et al., 2015)

• PSP-Net (Zhao et al., 2016)

• LaU model (He et al., 2019)

• OCR model (Yuan et al., 2019)

Urban Data Science Lab, TAMU

Road extraction from aerial image

• ResUnet on the Massachusetts roads dataset (Mnih and Hinton, 2010)

• SegNet to segment roads in the Massachusetts dataset and the Thailand Earth Observation System (THEOS) dataset (Panboonyuen et al. 2017)

Urban Data Science Lab, TAMU

Sidewalk extraction from street view image

• Senlet and Elgammal (2012): a pixel-wise and a path-wise method to predict and connect the occluded sidewalks covering a university campus.

• Smith et al. (2013): random forest to identify sidewalk after extracting local and global features.

• Mattyus et al. (2016): fully connected deep structured networks and Markov random field.

• Wang et al. (2019):convert street view image into top-view representation.

Urban Data Science Lab, TAMU

Outline

• Introduction

• Related Work

• Methodology

• Results

• Discussion

Urban Data Science Lab, TAMU

sidewalk datasets

• field survey

• deriving from existing land cover data

• extracting from images

Urban Data Science Lab, TAMU

workflow

Urban Data Science Lab, TAMU

Study area (Delaware Valley Regional in Pennsylvania State)

Urban Data Science Lab, TAMU

Training set and test set

Urban Data Science Lab, TAMU

https://medium.com/@abhigoku10/cv2019-papersummary-yolact-real-time-instance-segmentation-e62fa721957f

A sample of the sidewalk training set. The DVRPC sidewalks dataset uses a short line to represent the

conjunction point with crosswalks, which is preserved in the training dataset.

Urban Data Science Lab, TAMU

Missing sidewalks in aerial images between dangles

Urban Data Science Lab, TAMU

Segmentation of street view images by pre-train PSPNet

Urban Data Science Lab, TAMU

A panorama and its depthmap in Equirectangularprojection from Google Street View

Urban Data Science Lab, TAMU

Reproject the segmentation to landcover map

Urban Data Science Lab, TAMU

Extracting sidewalk centerlines from street view images. The centerlines were extracted merely from

landcover map for visual representation

Urban Data Science Lab, TAMU

Outline

• Introduction

• Related Work

• Methodology

• Results

• Discussion

Urban Data Science Lab, TAMU

Sidewalk detection results of YOLACT

Urban Data Science Lab, TAMU

Precision and recall of segmentation results

Urban Data Science Lab, TAMU

An occluded segment of sidewalks was restored using street view images

Urban Data Science Lab, TAMU

A missing sidewalk segment of 85 meters was restored using street view images

Urban Data Science Lab, TAMU

Outline

• Introduction

• Related Work

• Methodology

• Results

• Discussion

Urban Data Science Lab, TAMU

https://www.nwnewsnetwork.org/post/washington-lawmakers-

ponder-rules-sidewalk-driving-delivery-robots

https://www.scientificamerican.com/article/out-of-the-way-human-delivery-robots-want-a-share-of-your-sidewalk

Reality?

Urban Data Science Lab, TAMU

https://www.latimes.com/local/lanow/la-me-ln-fix-broken-sidewalks-20140818-story.html

Urban Data Science Lab, TAMU

https://www.dailynews.com/2018/02/07/la-city-leaders-say-streets-sidewalks-are-broken-and-may-not-be-ready-in-time-

for-olympics/

Urban Data Science Lab, TAMU

Data Challenging Communities

33

Urban Memory

34

query, navigation, and comparison of video, photo, audio, and text

over regions and locations

Urban Memory

35

36

Procedure

37Procedure

GeoVideo

● Location-based video capturer

● Without Internet connection

● Official website

○ http://vis.cs.kent.edu/GeoVisuals/mobile.html

● Appstore○ https://apps.apple.com/us/app/geovideo/id1492419929

● Google Play○ https://play.google.com/store/apps/details?id=com.nlapps.geovideo&hl=en

Apps

39

Human in Cities

40

MethodologicalConceptual

Transformative Outcome

Human-Centered Built Envr. Science

Urban Computing

• Energy Research

• Hazard Reduction & Recovery

• Housing & Land Use

• Smart & Connected Cities/Health

• Spatial Decision Support System

• Transportation & Travel Behavior

Workflow Automation

Urban Systems and Communities in the 21st Century

Text as Data

Emerging AI, ICT, and location-aware technologies

Visual Analytics

Data Fusion

Human Dynamics

Image and Video Analytics

Real-time SystemSpatial Econometrics

Space-time Analysis

Collaborative Research Plan

Turn space to place

from concept to deliverables