1

The Citizen Engineer: Urban Infrastructure Monitoring Via Crowd-Sourced

Data Analytics

Devin K. Harris1, Mohamad Alipour1, Scott T. Acton2, Lisa R. Messeri3, Andrea

Vaccari2, Laura E. Barnes4

1Department of Civil and Environmental Engineering, University of Virginia,

2C. L. Brown Dept. of Electrical and Computer Engineering, University of Virginia,

3

Department of Engineering and Society, University of Virginia, 4

Department of Systems and Information Engineering, University of Virginia

In the US, the challenges of an aging infrastructure network, coupled with

requirements for maintaining continuous functionality of this network, highlights

the need for innovative, and multi-disciplinary solutions aimed at the timely

detection and remediation of defects and deterioration before serious failure

situations materialize. Considering the constant and widespread interactions of

citizens with urban infrastructure systems, and the increasing ubiquity of mobile and

personal electronic devices equipped with onboard sensing capabilities (e.g. camera,

accelerometers, GPS, etc.), the concept of leveraging crowd-sourcing provides a

promising data-driven solution for urban infrastructure monitoring. In this approach,

the vision of the “citizen engineer” is introduced by empowering citizens to become

“active human sensors” at the source of defect detection and data collection, thus

extending the role of citizens from passive infrastructure users to active infrastructure

monitors.

In the proposed method, volunteers are motivated and instructed to use

mobile devices to capture and send geo-tagged images of defects (e.g. cracks,

corrosion, trip/slip hazards, potholes, etc.) that they observe in an urban

infrastructure environment, including a short description and severity rating. The

collected photos and descriptions are processed using object recognition techniques.

Defects are identified and extracted from the photos and quantified, whereas

additional information about the defects and their perceived severity are obtained

from the description field. Beyond the local condition measures, the aggregate data

provides responsible authorities with a quantitative analysis of the detected defects

as well as a measure of importance and severity (i.e. heat maps), as perceived by the

citizen, that can be used to inform maintenance decisions. While the challenges of

this framework are discussed in detail, it is expected to be a highly promising

departure from the traditional top-down infrastructure monitoring approaches.

KEYWORDS: Crowd-Sourcing, Urban Infrastructure, Citizen Engineer, Structural

Health Monitoring, Machine Learning, Damage Detection

2

INTRODUCTION

State of Infrastructure in the US

Members of every community, ranging from individual families to global

corporations, rely on the stability of the national and local infrastructure system to

provide the critical support needed to guarantee quality of life. Although the

criticality of this infrastructure network has prompted system designs that are robust

and able to withstand and recover from a diverse source of events, threats and

incidents (Kavinoky 2007, GAO 2008), the overall performance remains correlated

with the integrity of its individual components. This necessitates early detection of

deterioration and damage resulting from aging and increased usage, before the

mechanisms become failure-critical.

The American Society of Civil Engineers’ latest Infrastructure Report Card rated the

nation’s infrastructure at D+ with a staggering estimated 3.6 trillion-dollar investment

needed to bring the rating up to an acceptable standard by 2020 (ASCE 2013).

The challenges associated with aging and deterioration in the US infrastructure, were

also highlighted by one of the National Academy of Engineering Grand Challenges,

“Restoring and Improving the Urban Infrastructure” (Perry 2008), which further

illustrates the criticality of the current predicament and emphasizes the need for

innovative solutions.

The generic framework of structural health monitoring (SHM), or the more

generalized infrastructure health monitoring (IHM) (Aktan et al. 2000), offers a

rational approach, capable of characterizing the behavior of in-service infrastructure

elements while providing indications of damage and even forewarning of impending

failure. For existing components of the urban infrastructure systems, current health

monitoring strategies have gained some traction amongst the engineering community;

however, despite recent advances in technologies to monitor and detect various

deterioration mechanisms, current IHM practices are tuned toward the simple

detection of existing deterioration conditions, rather than their quantification,

especially at incipient stages of damage (Chang et al 2003). As a result, most current

IHM practices are still rooted in the human-derived measurements or highly

subjective visual inspections conducted by trained personnel.

Although the concept of visual assessment is generally a sound practice, individual

subjectivity, the periodic nature of most inspection practices, and the associated costs

are among its current limiting factors (Chang et al. 2003). Furthermore, the vast

majority of the current infrastructure components were designed and built in an era

before the evolution of IHM and “smart city” (Anttiroiko et al. 2014) concepts, and

the requirement of a dense network of detection and monitoring devices is neither

feasible nor cost effective. Additionally, stakeholders would still be confronted with

resource limitations and the expertise required to interpret the collected data. These

challenges highlight the need for innovative solutions that are sustainable and

scalable across a diverse system of infrastructure components.

Crowd-sourcing: A Solution for Participatory Problem Solving

Crowd-sourcing is a relatively new concept that seeks to distribute the efforts to

execute a task or solve a problem between a group of volunteers usually through the

internet (Estellés-Arolas and González-Ladrón-De-Guevara 2012). With the increased

3

availability of network access and the interest of citizens in direct involvement with

scientific, social, and economic events, crowd-sourcing approaches have not only

become more popular, but often mission critical as evidenced by the analysis of

satellite data by volunteers that proved essential to humanitarian relief after a typhoon

devastated the Philippines in 2013 (Butler 2013, Gao et al 2011). Other examples of

popular crowd-sourcing applications on the internet includes Wikipedia, the online

encyclopedia, the open-content collaborative mapping project Wikimapia, and the

more commercial cases of the human intelligence task marketplace, Amazon

Mechanical Turk, or the internet image sale platform, iStockphoto. Further, many

designers have noted that mobile phones are powerful tools for facilitating crowd-

sourcing projects (Chatzimilioudis et al. 2012, Yan et al. 2009). For example, with the

popular mobile app Waze, users can collect and report traffic data to other nearby

users.

Of late, “citizen science” projects have been increasingly employed for large data

projects. Zooniverse, the largest citizen science portal, has projects ranging from

astronomy to biology to history (Borne and Team 2011). Many have noted the

positive scientific outcomes of these projects (Kim et al. 2011, Cohn 2008); however,

it is clear that for the citizens to truly appreciate the impact of their work and to gain

an increased appreciation for the underlying scientific process, much attention is

needed to the design of such projects (Brossard et al. 2005, Bonney 2009).

The Citizen engineer: A Crowd-Sourced Monitoring Paradigm

Building upon the successful experiences of crowd-sourcing in citizen science

projects, this paper seeks to introduce the concept of citizen engineering and frame it

as a viable contribution to the analysis and preservation of urban infrastructure. The

citizen engineer is a descriptor for a new kind of social actor modeled off of the

popular citizen scientist, a volunteer who engages in the collection of multimedia data

and, with proper training, is able to perform basic engineering analyses that benefit

design and health monitoring of an engineering system within their local community.

In the context of this investigation, the citizen engineer is a user who is actively

driving the identification of condition features and actively participating in their

resolution by collecting the data required for assessing and tracking degradation over

time. By their direct involvement in the detection process, citizen engineers

effectively become a network of human sensors tightly integrated within the urban

infrastructure (Fig. 1) and are enabled to contribute directly to decision-making

related to the infrastructure that they assess.

This approach provides not only the sensorial density required for effective sampling,

but also a level of redundancy within the proposed framework that ensures both

reliability and resilience of the monitoring process and guarantees lower user fatigue.

Within the citizen engineer framework, users not only directly identify the presence

of an issue, but also indirectly provide a quantification of the gravity of an issue and

the relative severity. For example, issues that warrant or “trigger” a sense of

“urgency” within the observations (large cracks, potholes, large corrosion, structural

distortions, etc.) will be imaged more often than smaller issues. Thanks to this natural

discrimination capability offered by the advantages of using citizen engineers, the

collected data will offer an a priori importance evaluation.

4

Fig. 1. The citizen engineer: a network of human sensors integrated with infrastructure

PROPOSED FRAMEWORK

In the proposed framework, citizens are trained to use mobile and wearable

technologies to collect multimedia data about their surrounding civil infrastructure

and provide user-driven knowledge to characterize its condition state. This user-

driven knowledge for characterization is the critical defining attribute that

distinguishes the citizen engineer from a citizen scientist. The key underlying

challenges for achieving this goal are identified as follows:

1- Acquiring crowd-sourced data through constantly engaging, training and

motivating the volunteer citizens to contribute multimedia data.

2- Ensuring the quality, usability and reliability of the data and enabling efficient

and automated processing of the data into actionable knowledge.

To address these challenges, a system architecture with two main components is

proposed in this paper and will be discussed in the next sections (Fig. 2).

Fig. 2. Proposed Plan for Project Execution

Although the proposed approach is generic in nature and can be applied to all aspects

5

of urban infrastructure, the description of system architecture and the formulation of

the solutions focuses on representative physical infrastructure such as buildings,

bridges, pavements, and sidewalks, which are common to a city environment. The

main types of defects and deteriorations experienced by such components during their

service lives have been studied and well-documented in recent years (FHWA 2012,

Adlinge and Gupta 2009, Gharaibeh et al. 1998, Richardson 2002) (Fig. 3).

Furthermore, the data collected by the citizen engineers can include a variety of types

from multimedia data to accelerations recorded by cellphones or simple text in the

form of user reports; however, this paper mainly focuses on images taken from these

representative defects as an example.

Fig. 3. Examples of target defects within urban infrastructure components

Component 1- Data Acquisition through Citizen Network Design

In designing the proposed citizen sensor system, alongside the development of the

data collection medium, questions of motivation and civic engagement need to be

addressed; therefore, the development and testing of the system should be carried out

with attention to both how this experience would benefit the user, and also how the

user can be trained to produce reliable information. Most significantly, like the citizen

scientist, the citizen engineer must be trained as an observer and learn to see

compromised infrastructure in the same way as a professional. In addition, designing

for the citizen engineer requires close examination of the interactions between the

users, the urban grid, and the mobile application. Interviews with stakeholders (the

design team, potential users, engineers, city planners) and participant observation can

effectively feed into the design process in order to make a tool that reflects the social

and technical needs. The design process can assume different roles for the citizen

engineer in monitoring infrastructure. A more active approach would entail the user

detecting and recording new features that he or she may think are in need of

servicing. Alternatively, users can take a more passive role by collecting images of

locations specifically requested by city workers. Further, understanding the self-

conception of the citizen engineer is a vital factor in successful realization of the

requirements of the design of the system. Why would citizens want to enroll

themselves into this system and what would they hope to gain from participation?

How might planners harness this application such that the citizen engineer takes an

active role in improving his or her city? These questions and the aforementioned

challenges are essential to the citizen engineering framework. The following sections

6

highlight and summarize key components associated with the data acquisition through

the citizen network design.

Design Requirements for a Mobile Data Acquisition Platform

Citing the ubiquitous use of mobile applications and the examples of data acquisition

from large groups of people, for the design of the citizen engineering experience, a

mobile application is deemed the obvious choice. The envisioned software should

include a mobile application for the citizen engineer (suitable for interfacing with

mobile platforms such as iOS and Android as well as wearable devices such as

Google Glass and Microsoft HoloLens) as well as the base station analysis that will

be compatible with Windows and Mac. The application should include a basic

structure that allows users to collect and upload data, assess quality and condition

state of the selected infrastructure component, and also to allow for two-way

interaction that enables feedback between the citizen engineer and infrastructure

owners. A basic example of a preliminary mobile application is presented in Fig. 4

and illustrates the functionality of geo-referenced crack detection and quantification.

The application would also enable the solicitation of user feedback through surveys

and an indirect measure of the application performance based on the quality and

consistency of data derived during the collection processes.

Fig. 4. Citizen engineer mobile application: collection (left), post-processing

User Training for Citizen Engineering

Within the citizen engineering framework, participants are expected to be non-experts

in the domain or amateurs; therefore, to make sure the users are able to fulfill the

tasks and to guarantee an acceptable level of data quality, a brief and targeted training

and instruction is necessary. To this end, the users need to undergo a preliminary

training to be introduced to the motivation behind their efforts, the concept of the

citizen engineer, the type of analysis the project is interested in, and basic

requirements for data collection. Further optional training can provide education

about the engineering fundamentals behind the monitored infrastructure, failure

7

modes that might occur, the associated risks of failure, as well as the techniques

currently used to detect such failures. For example, in training users of the app to

evaluate from the perspective of a citizen engineer, short tutorials would be provided

on aspects such as camera functionality, basic photography tips, what the most

common distortions are, their description and how they can be detected and corrected

using software.

Assessing the Citizen Engineer Network

Key to the success of this system is developing an understanding regarding what

motivates citizen engineers to participate in infrastructure monitoring and then

leveraging that understanding to improve the system design. Among the effective

tools are structured interviews and focus groups with relevant stakeholders, including

potential users, to understand the connections and potential feedback loops between

infrastructure, citizens, government, and improvement projects. The goal is to

understand both the constraints and limitations of this novel network, while also

defining the potential impact, new applications, and most importantly what the

system needs to include such that it will be seen as a useful tool for the citizen.

Integrating interview data from multiple stakeholders will create a more complete

picture of motivation and engagement that takes into account both user and project

needs. Moreover, characterizing potential social barriers to entry early in the design

process will ensure more successful implementation. One preliminary design that

speaks to the question of motivation, modeled off of game design, involves the

utilization of a point-based reward system where individual citizen engineers will

accumulate points based on the quality and relevance of the acquired data and

correctness of the failure evaluation. An alternative option includes monetary or

prize-based compensation for participation. Regardless of the motivation model, the

desired outcome involves the continued engagement of the user in the citizen

engineering process.

User Testing for Design

User testing is an important component in the design of the proposed cyber-human

system and should focus on evaluating the effectiveness of the application of citizen

engineering when put into practice. During the design phase, testing should be

conducted at incremental points using available participants. These participants will

be systematically monitored (through both self-reporting and in situ user studies) in

order to understand not only usability problems with the application, but also

motivation barriers with regards to participating as a citizen engineer. The testing

program should focus on developing a test matrix that elicits user feedback, thus

enabling the project team to better understand how the system can be used. In

addition, the feedback will allow for an assessment of the desirability of the system,

which in turn can be used to improve designs and outcomes. User testing should also

evaluate the efficacy of the user training by assessing how adequately this application

teaches citizens to “see” in a way similar to a professional engineer. Moreover,

assessing the viability of the network feeds directly into this user testing task and

facilitates the use of similar qualitative assessment means (e.g. focus groups,

interviews, and observations of system use). Where the network assessment focuses

8

primarily on motivation and understanding how to produce an engaged user base, the

user testing task focuses on how to cultivate a user that both benefits from and

provides benefits to the system and can be performed on different versions of the

application to best evaluate features that are most useful to the user and those that

produce a user capable of providing high fidelity data useful for infrastructure

assessment.

System Feedback to the User

Significant improvement to the performance of the system is expected by providing

feedback to the users about their performance. This feedback can cover all aspects of

user-system interaction. For instance, in an image based collection scheme, once an

image is received, the system can assess its quality and notify the user of the quality

score together with photography tips thus helping to improve future image quality.

Additionally, once an image is analyzed, the type of structure and defects and other

extracted information could be provided to the user, which helps familiarize the non-

expert citizen with a basic knowledge and terminology regarding infrastructure. This

will be a significant source of continued training for the users thus reinforcing the

efforts in user training.

Component 2 – Processing of the Crowd-Sourced Data

Key to the accurate evaluation of infrastructure condition state is the ability to extract

reliable information about the presence and characteristics of potential deterioration

and defects from the data collected by the citizen engineers. Full implementation of

the system in an urban community involves a sizeable population of users and thus

the collected data is expected to be derived from a variety of mobile platforms and of

varying quality and consistency. However, this sizeable population provides an

unprecedented opportunity to collect rich datasets that cannot be collected by

managing entities alone. In this work, the description of data is limited to images;

however, mobile platforms also have numerous other capabilities that can be

leveraged such as geo-location, acceleration, time, etc. For this work, images are

assumed to be acquired from a wide range of devices, such as professional or point

and shoot cameras, cellular phones, smart glasses, activity-tracking devices (such as

GoPro cameras), and tablets.

Blind Image Quality Assessment

Given the variety of devices from which the crowd-sourced imagery will be collected

and the variety of imaging conditions, one of the main challenges relates to vetting

the large amount of data to generate reliable information. A requirement is the

reduction or elimination of unusable or low quality images within the analysis

pipeline via an automated selection stage aimed at the discrimination between high

and low quality data. In general, an obvious problem with such quality assessment is

the lack of a pre-vetted reference ground truth, with which to compare the data

collected by the citizen engineers, making the assessment process effectively blind.

For the goal of providing a reliable infrastructure analysis metric, a numerical

measure of usability needs to be assigned to each collected image. The majority of

image quality assessment criteria such as the traditional mean-squared error and the

newer and more efficient structural similarity image measure (SSIM) (Zhou et al

2004) both require a reference, which will not be directly available for the proposed

9

framework. To mitigate this issue, natural image statistics provides an alternative to

measure image quality in terms of the presence/absence of such distortions as noise,

blur (from defocus and motion), fading, limited field of view, limited depth of field,

limited resolution, over-quantification and compression artifacts (Anish et al 2012).

Only images with quality higher than a set threshold, would be allowed through the

pipeline to the next analysis stage (Fig. 5). The automated blind image quality

assessment can be validated by comparison with mean opinion scores on the usability

of the images produced by professional inspectors on a sample of images.

Fig. 5. Pothole in pavement with different image quality scores

Defect Recognition

For data satisfying the image quality assessment, the next step would be the

identification of features of interest or defect recognition. This task is arguably the

most crucial step in complementing the non-expert citizen’s work with infrastructure

domain knowledge, thus realizing the real value of the citizen engineering concept.

Infrastructure domain knowledge for processing the crowd-sourced data can be

employed by manually assessing all images by qualified inspectors as data acquisition

proceeds. However, the successful implementation of the citizen engineering

paradigm entails handling a continuous stream of large amounts of data. Therefore,

manual processing by experts is prohibitively expensive and time-consuming.

As an automated and scalable solution, emerging computer vision techniques would

enable the encoding of domain knowledge into a supervised machine learning

scheme. Relatively limited literature on the application of computer vision to solve

infrastructure problems exists (Brilakis et al 2011, German et al 2012, Koch et al

2015, Halfawy and Hengmeechai 2014, Guo et al 2009); however, the of principles of

computer vision-based feature detection have been utilized in analogous domains

(Park et al 2011, Chi et al 2009, Ayache 1995, Du et al 2011, Danuser 2011). The

approach proposed in this framework would seek to train an object recognition model

on the training dataset of defect images such that the model can recognize the type of

defects in future images. To this end, a labeled training set is required in which

qualified engineers evaluate and label the defects in a dataset of sample images. Then,

distinctive feature sets are extracted from the images using a selection of the well-

established methods from the literature including SIFT and HOG (Dalal and Triggs

2005, Lowe 1999). State-of-the-art object detection classifiers such as Support Vector

Machines on the feature sets and Deep Convolutional Neural Networks would be

employed to located and characterize defects, considering their robustness to

environmental variations (such as illumination, viewpoint and scale) that are shown to

10

be major factors in visual defect detection tasks (Koch et al 2015, Halfawy and

Hengmeechai 2014). The output of this step is a label assigned to each image that

describes the type of defect captured in the user-provided image. Fig. 6 illustrates a

sample training dataset of common infrastructure deterioration mechanisms with the

respective labels.

(a) Corrosion in steel members (b) Potholes in roadway surface

(c) Cracks in concrete members (d) Spalling

Fig. 6. Sample labeled training image dataset

Quantification of Structural Defects

After the type of defect in the image is identified, a defect-specific algorithm is

required to quantify the size/severity of the defect (e.g. length and width of a crack vs.

the area of corrosion). Depending on the requirements of the specific model, a

concept such as scale analysis (Frangi et al 1998) needs to be applied to the data. This

approach would allow for the transition between the concept of “existence” of a

defect (answering question such as: “is there any defect?”) and actual “quantification”

of the defect (“how large is the defect?”, “what is the defect shape?”, “what is the

volume?”).

As a specific example, pothole detection has been achieved using segmentation of

potential image features identified using underlying assumptions about the

appearance, shape and texture (Koch and Brilakis 2011). In the case of corrosion,

several image processing techniques, based on edge detection (Pakrashi et al 2010),

3D shape analysis (Hanji et al 2003), and textural wavelet analysis (Jahanshahi and

Masri 2012a), have been developed and applied towards the detection and analysis of

corroded regions in various types of infrastructures. Automated crack detection

techniques for infrastructure health assessment have been successfully developed

both for the routine monitoring of simple structures such as subway tunnels (Zhang et

al 2014) as well as for the detection and 3D analysis of events within the more

challenging pavement environment (Zou et al 2012, Jahanshahi and Masri 2012b). As

far as structural distortion is concerned, several non-contact monitoring methods

based on continuous imagery and the use of edge detection (Jung 2014) and

11

embedded templates for motion tracking (Fukuda 2010) can be used to quantify

distortions.

Information Aggregation, Reporting and Feedback

To take advantage of the extracted information and help make appropriate

maintenance decisions, it is essential to visualize, present and disseminate the results

of the collected and processed condition state data. To this end, the geo-tagged

detected and quantified results can be charted on a “heat map” and provided to

infrastructure managing agencies (i.e. owners, maintenance offices, departments of

transportation, municipalities, etc.) to create a foundation for data-driven decision

making. Sharing the aggregated results with the users of the system is also expected

to help raise awareness about infrastructure, engage the citizens more actively in a

public demand for the enhancement of urban infrastructure, hold the responsible

parties accountable, and provide system users with validation of their contributions.

Moreover, responsible parties can also acknowledge the reported issues and inform

the public of their specific response or follow-up on the citizen engineer project

website and/or their respective websites to demonstrate to the public the value of

citizen participation and contributions and further promote the citizen engineering

concept.

Infrastructure Condition Rating

The results of the image analyses in the form of quantitative geometrical properties of

individual defects do not directly reflect the overall in-service condition of the

monitored components in terms of operational safety, serviceability, and the relative

priority/urgency of rehabilitation practices. If such overall measures are sought, the

crowd-sourced results must be converted into appropriate condition ratings or

performance indices that can be used as a foundation in prioritizing maintenance

operations and allocation of resources. The mapping of the image-based defect

measurements to the overall condition ratings (condition inference) requires a rational

and objective basis deeply rooted in long-term experience and knowledge, as well as

an automated framework to deploy it on components with any given set of

characteristics and level of physical conditions. This condition inference can be

achieved through the combination of a knowledge-based expert system and an

experience-based machine-learning scheme.

Expert systems have been successfully implemented in problem solving in a variety

of engineering practices (Adeli 1998, Zavadskas et al 2008, Norela et al 2009, Mohan

1990, Kaetzel and Clifton 1995). In the proposed citizen engineering framework, the

system needs to be fed image-based defect measurements, as well as knowledge with

respect to the overall properties of the monitored components, including their types

(i.e. bridge or pavement), type of materials, geometrical characteristics, and ages of

construction. The proposed expert system can then take advantage of the existing

knowledge in the field of infrastructure health monitoring by extracting rules from an

extensive interview program with domain experts, together with implementation of

the existing condition evaluation metrics for in-service structures and facilities within

the urban built environment.

12

Once the system has extracted the rules, including all the underlying patterns and

relationships, a machine learning scheme can be trained to reproduce condition

ratings or health grades (A to F or 1 to 10) for any specific type of urban

infrastructure component based on the analyzed image-based condition state data. In

order to train the model, condition ratings from existing evaluation metrics together

with expert solicitations can be used. Extensive literature exists on different machine

learning models used in infrastructure condition assessment and the reader is referred

to them for further reading (Meegoda et al 2006, Saitta et al 2009, Amiri et al 2015,

Harris et al 2015, Farrar and Worden 2012, Alipour et al 2016).

DISCUSSIONS AND CONSIDERATIONS

Leveraging the link between humans and the ubiquitously present portable and

wearable devices, this paper outlines the vision for a novel infrastructure health

monitoring framework for characterizing the health status of urban infrastructure

components centered on the concept of the citizen engineer. Within this framework

we propose to engage the untapped potential of a community of citizen engineers, and

their associated data acquisition and communication platforms, as a resource to

supply the required intelligence for real-time assessment of urban infrastructure

systems.

Implementation Considerations and Challenges

As outlined in the previous sections, user engagement and continued contribution is a

critical prerequisite for the successful implementation of the citizen engineering

framework. A variety of strategies were proposed in the Data Acquisition through

Citizen Network Design component to address this issue. The ubiquity of social media

in different forms can also be considered an opportunity for publicizing the initiative

and engaging new and current users. It is thus envisioned that volunteer citizens, in

the same way that they carry out data acquisition, can carry on a significant portion of

the constant user engagement efforts.

With data originating from amateur citizens, maintaining satisfactory data quality is

of major concern. Three mechanisms to address quality assurance were devised in the

framework. One mechanism is a blind image quality assessment using natural image

statistics, whereas alternative mechanism includes both basic training and continuous

feedback from the system on how the user performed with respect to image quality.

Reliability of the results for a certain structure is another area of concern. It can be

argued that due to the crowdsourced nature of the data acquisition, false alarms in the

network will be inevitable. However, for issues that might attract the attention of a

large set of citizen engineers due to their visibility characteristics, several images will

be expected from several vantage points. This will allow for cross-checking, where

infrastructure that receives highly discrepant scores from different users can be sent to

further investigation. Furthermore, more citizen engineers could be automatically

directed towards these structures to increase the number of available images and

improve the evaluation of the reported issue.

Scalability is another inevitable challenge for the full implementation of the citizen

engineer involving a sizeable population of users sending a daily stream of images

(and other types of data). First, the provision of the image recognition engine is

13

expected to significantly reduce expert involvement burden. It is also predictable that

the handling and analysis of large volumes of unstructured data from a network of

users in an urban area might necessitate Big Data solutions.

CONCLUSION

This paper outlines a new paradigm in urban infrastructure health monitoring that

seeks to fill the gaps that exist within the assessment of urban infrastructure by

proposing a novel cyber-human evaluation system. In the proposed framework, non-

expert citizens are engaged, trained and motivated to use their mobile phones to

capture and send images of defects in the urban infrastructure. These images will first

be automatically analyzed for quality, and then computer vision models are proposed

to identify and quantify the pictured phenomena.

The two main challenges facing the implementation of the proposed framework were

described as the acquisition of high-quality crowd-sourced data through constant user

engagement, and reliable and efficient processing of the data toward actionable

knowledge. Accordingly, two major components were proposed for the system with

detailed tasks. The proposed tasks were supported by relevant literature and are

expected to cover the main challenges. Further considerations on the implementation

of the system were also included.

The proposed approach pushes the boundaries through an exploration of the cyber-

human symbiotic relationship such that participants are not simply users, but

ultimately become intelligent sensor nodes themselves. This multi-objective approach

promises to train volunteers and provide them with a better understanding of the

surrounding infrastructure and, by leveraging the wealth of mobile technology

available, empower them to become active participants in their local community.

Beyond the near-term implications for preservation of the existing infrastructure, the

proposed approach will have a long-term impact by providing the essential guidance

required to enhance the design of the next generation of sustainable and intelligent

“smart cities”.

REFERENCES

Ayache, N. (1995). “Medical computer vision, virtual reality and robotics”. Image and

Vision Computing, 13(4), 295-313.

Adeli, H. (1988). Expert Systems in Construction and Structural Engineering, Taylor and

Francis.

Aktan, A., Catbas, F., Grimmelsman, K., and Tsikos, C. (2000). "Issues in Infrastructure

Health Monitoring for Management." Journal of Eng. Mechanics, 126(7), 711-724.

Adlinge, S. S., and Gupta, A. K. (2009). "Pavement Deterioration and its Causes." Journal

of Mechanical and Civil Engineering, 9-15.

ASCE (American Society of Civil Engineers) (2013). "Report Card for Amirica's

Infrastructure."

Anttiroiko, A.-V., Valkama, P., and Bailey, S. (2014). "Smart cities in the new service

economy: building platforms for smart services." AI and Society, 29(3), 323-334.

Amiri, M., Ardeshir, A., Fazel Zarandi, M.H., and Soltanaghaei, E. (2015). “Pattern

extraction for high-risk accidents in the construction industry: a data-mining approach”.

International journal of injury control and safety promotion. 1-13.

14

Alipour, M., Gheitasi, A., Harris, D.K., Ozbulut, O.E., Barnes, L.E. (2016). “A data-driven

approach for automated operational safety evaluation of the national inventory of

reinforced concrete slab bridges”. Transportation Research Board (TRB), 95th Annual

Meeting, Washington, DC.

Brossard, D., Lewenstein, B., and Bonney, R. (2005). "Scientific knowledge and attitude

change: The impact of a citizen science project." Intl. Journal of Science Education,

27(9), 1099-1121.

Bonney, R., Cooper, C. B., Dickinson, J., Kelling, S., Phillips, T., Rosenberg, K. V., and

Shirk, J. (2009). "Citizen science: a developing tool for expanding science knowledge

and scientific literacy." BioScience, 59(11), 977-984.

Borne, K. D., and Team, Z. (2011). “The Zooniverse: A framework for knowledge

discovery from citizen science data”. In AGU Fall Meeting Abstracts (Vol. 1, p. 0650).

Brilakis, I., German, S., Zhu, Z. (2011). “Visual pattern recognition models for remote

sensing of civil infrastructure”. Journal of Computing in Civil Eng., 25(5), 388-393.

Butler, D. (2013). “Crowdsourcing Goes Mainstream in Typhoon Haiyan Response”.

Nature News, 20.

Chang, P. C., Flatau, A., and Liu, S. C. (2003). "Review paper: Health monitoring of civil

infrastructure." Journal of Structural Health Monitoring, 3, 257-267.

Cohn, J. P. (2008). "Citizen science: Can volunteers do real research?" BioScience, 58(3).

Chi, S., Caldas, C. H., & Kim, D. Y. (2009). A methodology for object identification and

tracking in construction based on spatial modeling and image matching techniques.

Computer‐Aided Civil and Infrastructure Engineering, 24(3), 199-211.

Chatzimilioudis, G., Konstantinidis, A., Laoudias, C., and Zeinalipour-Yazti, D. (2012).

"Crowdsourcing with Smartphones." Internet Computing, IEEE, 16(5), 36-44.

Dalal, N., Triggs, B. (2005). “Histograms of Oriented Gradients for Human Detection”. In

Computer Vision and Pattern Recognition, IEEE, (Vol. 1, pp. 886-893).

Du, S., Shehata, M., Badawy, W. (2011). “Hard hat detection in video sequences based on

face features, motion and color information.” In Computer Research and Development

(ICCRD), 2011 3rd International Conference on (Vol. 4, pp. 25-29). IEEE.

Danuser, G. (2011). “Computer vision in cell biology.” Cell, 147(5), 973-978.

Park, M. W., Makhmalbaf, A., Brilakis, I. (2011). “Comparative study of vision tracking

methods for tracking of construction site resources”. Automation in Construction, 20(7),

905-915.

Estellés-Arolas, E., González-Ladrón-De-Guevara, F. (2012). “Towards an integrated

crowdsourcing definition.” Journal of Information science, 38(2), 189-200.

Frangi, A., Niessen, W., Vincken, K., Viergever, M. (1998). "Multiscale vessel

enhancement filtering." Medical Image Computing and Computer-Assisted

Interventation, Springer Berlin Heidelberg, 130-137.

Fukuda, Y., Feng, M. Q., and Shinozuka, M. (2010). "Cost‐effective vision‐based system

for monitoring dynamic response of civil engineering structures." Structural Control and

Health Monitoring, 17(8), 918-936.

Farrar, C.R. and Worden, K. (2012) Structural health monitoring: a machine learning

perspective. John Wiley and Sons.

FHWA (Federal Highway Administration) (2012). "Bridge Inspectors Refernce Manual

(BIRM)." Washington D.C.

Gharaibeh, N., Wilson, C., Darter, M., Jones, G. (1998). "Development of a bike path

Management System for the University of Illinois at Urbana-Champaign."

Transportation Research Record: Journal of the Transportation Research Board, 1636(1),

56-63.

15

GAO (Goverment Accountability Office) (2008). "Defense critical infrastructure: adherence

to quidance would improve DOD's approach to identifying and assuring the availability

of critical transporation assets." Defense Capabilities and Management Washington D.C.

Guo, W., Soibelman, L., Garrett, J. H. (2009). “Automated defect detection for sewer

pipeline inspection and condition assessment.” Automation in Construction, 18(5), 587-

596.

Gao, H., Barbier, G., Goolsby, R. (2011). "Harnessing the Crowdsourcing Power of Social

Media for Disaster Relief." IEEE Intelligent Systems, 26(3), 10-14.

German, S., Brilakis, I., DesRoches, R. (2012). “Rapid entropy-based detection and

properties measurement of concrete spalling with machine vision for post-earthquake

safety assessments.” Advanced Engineering Informatics, 26(4), 846-858.

Hanji, T., Tateishi, K., Kitagawa, K. (2003). "3-D shape measurement of corroded surface

by using digital stereography." Proc., 1st Int. Conf. on Structural Health Monitoring and

Intelligent Infrastructure,The Netherlands, 699–704.

Halfawy, M. R., Hengmeechai, J. (2014). “Automated defect detection in sewer closed

circuit television images using histograms of oriented gradients and support vector

machine.” Automation in Construction, 38, 1-13.

Harris, D.K., Ozbulut, O.E., Alipour, M., Usmani, S., and Kassner, B.L. (2015).

“Implications of load rating bridges in Virginia with limited design or as-built details.”

Proc. 7th Intl. Conf. on Structural Health Monitoring of Intelligent Infrastructure.

Torino, Italy.

Jahanshahi, M., and Masri, S. (2012a). "Parametric Performance Evaluation of Wavelet-

Based Corrosion Detection Algorithms for Condition Assessment of Civil Infrastructure

Systems." Journal of Computing in Civil Engineering, 27(4), 345-357.

Jahanshahi, M. R., and Masri, S. F. (2012b). "Adaptive vision-based crack detection using

3D scene reconstruction for condition assessment of structures." Automation in

Construction, 22(0), 567-576.

Jung, Y. (2014). "An Approach to Automated Detection of Failure in Temporary Structures

using Image Processing." Journal of Engineering and Architecture, 2(1), 49-61.

Kaetzel, L., and Clifton, J. (1995). "Expert/knowledge based systems for materials in the

construction industry: State-of-the-art report." Materials and Structures, 28(3), 160-174.

Kavinoky, J. F. (2007). "The importance of transportation infrastructure to the american

business community " House of Representatives Committee on Transportation and

Infrastructure, U.S. Cahmber of Commrce,Washington D.C.

Koch, C., and Brilakis, I. (2011). "Pothole detection in asphalt pavement images."

Advanced Engineering Informatics, 25(3), 507-515.

Kim, S., Robson, C., Zimmerman, T., Pierce, J., Haber, E. M. (2011, May). “Creek watch:

pairing usefulness and usability for successful citizen science.” In Proceedings of the

SIGCHI Conference on Human Factors in Computing Systems (pp. 2125-2134). ACM.

Koch, C., Georgieva, K., Kasireddy, V., Akinci, B., Fieguth, P. (2015). “A review on

computer vision based defect detection and condition assessment of concrete and asphalt

civil infrastructure.” Advanced Engineering Informatics, 29(2), 196-210.

Lowe, D. G. (1999). “Object Recognition from Local Scale-Invariant Features”. In the proc.

of the 7th IEEE Intl Conference on Computer Vision (Vol. 2, pp. 1150-1157).

Mohan, S. (1990). "Expert Systems Applications in Construction Management and

Engineering." Journal of Construction Engineering and Management, 116(1), 87-99.

Meegoda, J., Juliano, T., Banerjee, A. (2006). "Framework for Automatic Condition

Assessment of Culverts." Transportation Research Record, 1948(-1), 26-34.

Anish, M., Krishna Moorthy, A. Bovik. A.C. "No-reference image quality assessment in the

spatial domain." IEEE Transactions on Image Processing 21.12 (2012): 4695-4708.

16

Norlela, I., Ismail, A., Atiq, R. (2009). "An overview of expert systems in pavement

management." European Journal of Scientific Research 30(1), 99-111.

Perry, W. (2008). "Grand Challenges for Engineering." National Academy of Engineering

of the National Academics, Washington D.C.

Pakrashi, V., Schoefs, F., Memet, J. B., and O’Connor, A. (2010). "ROC dependent event

isolation method for image processing based assessment of corroded harbour structures."

Struct. Infrastruct. Eng., 6(3), 365–378.

Park, T., Lee, J., Hwang, I., Yoo, C., Nachman, L., Song, J. (2011). E-gesture: a

collaborative architecture for energy-efficient gesture recognition with hand-worn sensor

and mobile devices. Proc. 9th Conf. on Embedded Networked Sensor Systems (pp. 260-

273). ACM.

Richardson, B. (2002). “Defects and Deterioration in Buildings: A Practical Guide to the

Science and Technology of Material Failure.” Routledge.

Saitta, S., Benny, R., Smith, I. (2009). Data Mining: Applications in Civil Engineering.

VDM Verlag. Saarrucken, Germany.

Yan, T., Marzilli, M., Holmes, R., Ganesan, D., Corner, M. (2009). "mCrowd: a platform

for mobile crowdsourcing." Proc. 7th ACM Conf. on Embedded Networked Sensor

Systems, ACM, Berkeley, Ca, 347-348.

Zhou, W., Bovik, A. C., Sheikh, H. R., Simoncelli, E. P. (2004). "Image quality assessment:

from error visibility to structural similarity." Image Processing, IEEE Transactions on,

13(4), 600-612.

Zavadskas, E., Kazimieras, R. L., Turskis, a. Z. (2008). "Multi-attribute decision-making

methods for assessment of quality in bridges and road construction: State-of-the-art

surveys." The Baltic Journal of Road and Bridge Engineering 3(3), 152-160.

Zou, Q., Cao, Y., Li, Q., Mao, Q., Wang, S. (2012). "CrackTree: Automatic crack detection

from pavement images." Pattern Recognition Letters, 33(3), 227-238.

Zhang, W., Zhang, Z., Qi, D., Liu, Y. (2014). "Automatic Crack Detection and

Classification Method for Subway Tunnel Safety Monitoring." Sensors, 14(10), 19307-

19328.

The Photos were provided courtesy of the Virginia Department of Transportation.