This research was supported by the United States Department of Homeland Security (DHS) through the National Center for Risk and Economic Analysis of Terrorism Events (CREATE) at the University of Southern California (USC) under award number 2010-ST-061-RE0001. However, any opinions, findings, and conclusions or recommendations in this document are those of the authors and do not necessarily reflect views of the United States Department of Homeland Security, or the University of Southern California, or CREATE.
Analyzing the Return on Investment of Homeland Security Research
Detlof von Winterfeldt Director, CREATE
University of Southern California 1150 S. Olive Street, Suite 1700
Los Angeles, CA 90015 (213) 740-3863 [email protected]
May 15, 2019
ADSA20 Boston, MA
2
Carl Crawford asks: So what? Who cares?
• The Problem. The Office of University Programs at DHS has been challenged to show the return on investment of the research conducted by its Centers of Excellence (COEs)
• Who Cares? Congressional appropriations committees, OMB, DHS S&T leaders, OUP leaders, COE leaders, HS researchers
• The Solution. Conduct selected benefit-cost analyses of about 20 OUP-funded research products (TTKPs=Tools, Technologies, and Knowledge Products)
• The Results. – One research product (CT Datasets and ATR) produces over $100 million in
benefits at a cost of $5 million, approximate ROI = 2,000% – 10 other research products (across multiple COEs) produce about $400
million in additional benefits, ROI = 856% – Two research products from CREATE produced enough benefits to pay for
$45 million = 12 years of OUP funding of CREATE 2
3
Steps in the Benefit-Cost Analysis
• Selection of research products
• Baseline analysis
• Cost analysis
• Benefits analysis
• Sensitivity and uncertainty analysis
• Results: Net Benefits, BCR, and ROI
4
Application to 20 Research Products (Selected by USCG, TSA, and CBP)
1. The likelihood of a successful transition (transitioned and used = 1)
2. The beneficial impact on customer, if successfully transitioned
Note: Selections are not representative of all OUP-funded projects
5
Example: CT Datasets and Algorithms
• Improve detection technology at TSA checkpoints
• CT Datasets enable better detection algorithms
• No need for classified or sensitive data
• Reduced false (nuisance) alarms
6
Cost Calculations for CT Datasets
6
Cost Category Start EndAmount in
Real Dollars
Amount in
2017 DollarsSource
Pre-project costs (COE)
Pre-project costs (other funding)
Project costs (COE) 9/1/13 6/30/18 740,000$ 748,315$ COE/OUP
Project costs (university cost share)
Oversight cost at the COE 9/21/10 6/30/18 210,000$ 216,987$ COE/OUP
Oversight cost at OUP 9/21/10 6/30/18 160,000$ 165,324$ OUP Overhead
Transition development cost 9/21/10 2/27/15 3,555,000$ 3,768,300$ S&T, TSA
Implementation start up cost
Implementation cost (User)
Implementation cost (COE)
Implementation cost (Other users)
TOTAL COST 4,665,000$ 4,898,926$
7
The Benefits of CT Datasets
• Reduces false alarms
• Saves time of passengers
• Saves time of TSA officers
7
National Academy of Science: Each 1% reduction in false alarms saved $25 million
8
Benefit Calculations for CT Datasets
8
Input Variable BaseNumber of Travelers/Year 719,000,000
Current False Positive Rate 10.0%
Decrease in False Positives 25%
Time in Secondary (Min) 7.5
Cost of One Minute of Wait Time $0.79
Cost of One Minute of TSA Time $0.27
Percent attributable to CT Vendor 25%
Percent attributable to ATR 30%
Cost of a Successful Attack 25,000,000,000
Current Cost of False Positives/Year 571,605,000$
3D Cost of False Positives/Year 428,703,750$
Attributabe Benefits/Year 10,717,594$
Discount Factor 0.030
Total Value of 3D CT Scanners for 10 Years 107,175,938$
Total Net Benefits for Ten Years 102,277,012$
9
Costs vs. Benefits of CT Datasets
Cost (in 2017 $)
$4,898,926
Benefits (in 2017 $)
$107,175,938
BCR = 22 ROI = 2,087%
10
Summary of 10 Additional BCAs
10
Tool, Technology, or Knowledge Product
(TTKP)
Cost (in Thousands, in 2017 $)
Median Net Benefit (50th Percentile)
Years of Use for Net Benefit Calculations
Research Products with Past Applications PROTECT $710 $35,505 6 Past & 4 Future Years
ARMOR $1,057 $28,969 10 Years, Past Use
CgSARVA $803
$5,247 One Time (Sandy)
Research Products with Potential Future Applications ADCIRC $36,893 $256,411 10 Years, Future Use
Engineered Swabs $1,867 $22,528 10 Years, Future Use
GeoXray $273 $18,404 10 Years, Future Use
TraffiCop $1,413 $10,444 10 Years, Future Use
HOAX Calls $183 $4,646 10 Years, Future Use
BOARD $1,018 $2,435 10 Years, Future Use
E-CAT $942 $806 10 Years, Future Use
TOTAL $45,161 $386,395 ROI=856%
11
Conclusions
• Benefit-cost analyses can be successfully conducted on OUP-funded research products
• Uncertainty about future benefits • Nevertheless: Benefits for a dozen selected research
products exceed OUP funding by a wide margin • A few research products are the main contributors to
benefits (ADCIRC, IRIS, CT Datasets, Enhanced Millimeter Wave Scanner)
• Start considering success criteria and benefits and benefit-cost analysis early – at proposal stage
• Nine benefits criteria and models developed by CREATE help (e.g., reducing false alarms, increasing detection, reducing threat, informing decision-making)
12
Thank you! For more on CREATE, go to
create.usc.edu
13
Supplementary Slides
14
Sensitivity Analysis for CT Datasets
14
5.0%
5
25%
$0.50
20%
20%
$0.18
719,000,000
0.1
0.9
10,000,000,000
20.0%
10
40%
$1.00
30%
30%
$0.35
719,000,000
0.4
0.9
50,000,000,000
$20,000,000 $40,000,000 $60,000,000 $80,000,000 $100,000,000 $120,000,000 $140,000,000 $160,000,000 $180,000,000 $200,000,000 $220,000,000 $240,000,000
Current False Positive Rate
Time in Secondary (Min)
Percent attributable to ATR
Cost of One Minute of Wait Time
Decrease in False Positives
Percent attributable to CT Vendor
Cost of One Minute of TSA Time
Number of Travelers/Year
Number of Attack Attempts Per Year
Detection probability with current system
Cost of a Successful Attack
Net Benefits for 10 Years
15
Uncertainty Analysis for CT Datasets
15
• Using wide ranges for all uncertain inputs (min, mode, max) • Using triangular distributions for each input
$- $50,000,000 $100,000,000 $150,000,000 $200,000,000 $250,000,000 $300,000,000 $350,000,000 $400,000,000
0
500
1000
1500
2000
2500
Net Benefits for Ten Years
Freq
uen
cy
16
10 Additional Research Products •ADCIRC - ADvanced CIRCulation Next generation modeling suite to predict coastal flooding
CHC - Coastal Hazards Center of Excellence
•ARMOR - Assistant for Randomized Monitoring Over Routes Software that randomizes schedules, plans, or actions for security agencies CREATE – Center for Risk and Economic Analysis of Terrorism Events
•BOARD - Bus Operator Awareness and Research Development Training program for bus operators to respond to security threats NTSCOE – National Transportation Security Center of Excellence
•CgSARVA – Coast Guard Search and Rescue Visual Analytics Interactive analysis of trends, patterns, anomalies, and distribution of
Search and Rescue cases VACCINE – Visual Analytics for Command, Control and Interoperability Environments
• E-CAT – Economic Consequence Analysis Tool Provides rapid estimates of economic impact of threats
CREATE – Center for Risk and Economic Analysis of Terrorism Events 16
17
• Engineered Detection Swab Swabs to improve detection of trace explosives on luggage and persons
ALERT - Awareness and Localization of Explosives-Related Threats
• GeoXray Tool to display geographic area and overlay relevant emergency response
CCICADA – Command, Control and Interoperability Center for Data Analysis
• HOAX Calls State-of-the-art tool for identifying hoax emergency calls
CCICADA – Command, Control and Interoperability Center for Data Analysis
• PROTECT – Port Resilience Operational/Tactical Enforcement to Combat Terrorism
Software to randomize schedules in a port environment CREATE – Center for Risk and Economic Analysis of Terrorism Events
• TraffiCop Analysis of Twitter and social media to detect human trafficking
CCICADA – Command, Control and Interoperability Center for Data Analysis
18
ADCIRC - ADvanced CIRCulation Storm surge/inundation modeling
COE: CHC - Coastal Hazards Center of Excellence
• Storm surge model that combines rain, atmospheric pressure, and wind forecasts
• Predicts when, where, and
to what extent flooding will inundate a coastal community
Median Benefit (50th Percentile)
$293,804,000
Cost (in 2017 $)
$36,893,000
18
19
E-CAT – Economic Consequence Analysis Tool Rapid estimates of the economic impact of threats
COE: CREATE- Center for Risk and Economic Analysis of Terrorism Events
Median Benefit (50th Percentile)
$1,746,000
Cost (in 2017 $)
$942,000 • Provides quick estimates of the economic impact of terrorist attacks, natural disasters, and technological accidents
20
Summary of Costs and Benefits
20
Tool, Technology, or Knowledge Product
(TTKP)
Cost (in Thousands, in 2017 $)
Median Net Benefit (50th Percentile)
Years of Use for Net Benefit Calculations
Research Products with Past Applications PROTECT $710 $35,505 6 Past & 4 Future Years
ARMOR $1,057 $28,969 10 Years, Past Use
CgSARVA $803
$5,247 One Time (Sandy)
Research Products with Potential Future Applications ADCIRC $36,893 $256,411 10 Years, Future Use
Engineered Swabs $1,867 $22,528 10 Years, Future Use
GeoXray $273 $18,404 10 Years, Future Use
TraffiCop $1,413 $10,444 10 Years, Future Use
HOAX Calls $183 $4,646 10 Years, Future Use
BOARD $1,018 $2,435 10 Years, Future Use
E-CAT $942 $806 10 Years, Future Use
TOTAL $45,161 $386,395 ROI=856%
21
Nine Benefits Criteria and Models • Cost savings with the same risk level
– Savings in operational costs (ARMOR, E-CAT)
– Savings by stretching expenses (cgSARVA)
• Increased security at same cost
– Reduction of threat (HOAX Calls)
– Reduction of vulnerability (PROTECT)
– Reduction of negative consequences (TraffiCop)
• Improved signal detection
– Reduction of false alarm rates (Engineered Swabs)
– Improvement of detection rates (3D CT Scans)
• Value of information
– Reduction of uncertainty (GeoXray)
– Improvement of decision quality (ADCIRC)
21
Top Related