2016 Bad Bot Report: Quantifying the Risk and Economic Impact of Bad Bots
-
Upload
distil-networks -
Category
Internet
-
view
633 -
download
1
Transcript of 2016 Bad Bot Report: Quantifying the Risk and Economic Impact of Bad Bots
Quantifying the Risk and Economic Impact of Bad BotsDistil Networks 2016 Bad Bot Report
Our Speakers
Rami EssaidCEO & Co-founder
Distil Networks
Derek BrinkVP & Research
FellowAberdeen Group
2015 Bad Bot Landscape Report Methodology
Study is based on anonymized data from:
74 billion bot requestsReal web traffic from
hundreds of customers17 global datacenters
Key Findings Key Findings
Bad Bot, Good Bot and Human Traffic, 2015
Good Bots
Humans
Bad Bots
19% of Web Traffic Causes The Following Problems
Humans take back the Web with 54.35% of all
web traffic
But why?
2013 vs. 2014 vs. 2015
Humans internet users grew 8% in 2105
Especially in countries such as China, India,
Indonesia, etc.
2015 Saw Tremendous Growth in Human Users
Source: http://www.statista.com/statistics/273018/number-of-internet-users-worldwide/
Number of internet users worldwide from 2000 to 2015 (in millions)
Meanwhile, Bot Operators Were Updating their Software
Bot software used in 2015 was vastly more advanced than in previous
years
This was a shift in focus from quantity
of bots to quality
Key Findings Bad Bot
Targets
Traffic Distribution by Size of Site, 2014 and 2015
Traffic by Type of Site, 2014 vs 2015
In 2015 the most targeted verticals were digital
publishing and real estate
Traffic by Size and Type of Site, 2014 vs 2015
More specifically,small digital publishers and
large real estate sites were hardest hit in 2015
Defense Tactics - Know your Industry
Understand how great of a risk bots pose to your industry
Learn how bots attack sites similar to yours
Industry Most Common Bot Problem
Ecommerce Price scraping
Digital Publishing Content theft
Travel Aggregation and loss of up-sell / cross-sell opportunities
Finance Brute force attacks
Real Estate Scraping Listing Information
Bad Bot Origins
China and US Home to the Worst Bad Bot Originators
Companies from China and the US dominate the list of organizations with the most
bad bot traffic
The US is always on top of this list, China is new
China
ChinaChina
China
China
China
USUS
US
USUS
US
US
Worst Bad Bot Originators 2013 to 2015
Amazon makes the Top 5 forthree years in a row
Verizon Business and residential ISPs Comcast,
Time Warner Cable clean up their acts
Mobile Carriers with the Most Bad Bots
Dutch carriers emerge as a new hotbed for mobile client based bots
The four largest mobile carriers in the US are all present on this year’s list
● Verizon Wireless● AT&T● T-Mobile● Sprint PCS
Countries Originating the Most Bad Bots, 2014 vs 2015
The US still tops the list of countries with the most bad
bots
Israel, India, and the UK make the biggest gains
Germany, Canada, Russia, and the Netherlands move down
the list
Countries Most Often Blocked by Geofencing Rules
2014 saw customers blocking developing
countries and stereotypical “bad guys”
2015 saw customers blocking more
industrialized countries
Top “Bad Bot GDP’s” of 2014 and 2015
Maldives rules the roost with 526 bad bots per
human online user
The average number of bots per human user on this list increased from 26.1 bots/user to 99.2
bots/user
Defense Tactics - Know Their Origins
Does your business model support all regions?
Is it normal that your customer is originating from a commercial data center or cloud provider?
Are there any reasons visitors to your site should go through a TOR network?
Analyze your business. Then trim the fat.
Bad Bot Capabilities and Behavior
The Majority of Bots are Now APBs
Advanced Persistent Bots (APBs) are becoming more commonplace
APBs are defined as having one or more of the following abilities:
● Mimicking human behavior● Loading JavaScript and
external assets● Cookie support● Browser automation● IP spoofing and rotation● User agent spoofing and
rotation● Distributed attacks (using
many IP addresses at once)
Loading Assets & Bots Mimicking Humans % of bots able to load external Assets (e.g.
JavaScript) % of bots able to mimic
human behavior
These bots will skew marketing tools such as (Google Analytics, A/B testing,
conversion tracking, etc.)These bots will fly under the radar of
most security tools
That Majority of Bad Bots Now Use Multiple IP Addresses
Bots which dynamically rotate IP addresses, or distribute attacks are significantly harder to detect and mitigate
Bad Bots Obtain New User Agents to Persistently Attack Websites
Over 36% of bots use multiple user agents to evade detection and overcome blacklisting and custom blocking rules
Chrome Takes the Lead as Most Assumed User Agent
Defense Tactics - Defeat APBs with Fingerprinting
Real-analysis and device fingerprinting allows security solutions to track bots even if they
● Assume new identities
● Mimic human behavior
● Rotate IP Addresses
● Distribute their attack overMany IP addresses
29
Quantifying the Risk of Bad BotsDerek E. Brink, CISSP
Vice President and Research Fellow, Information Security and IT GRC
www.linkedin.com/in/derekbrink
April 2016
Derek E. Brink, CISSPVice President and Research Fellow, Information Security and IT
GRC
www.linkedin.com/in/derekbrink
April 2016
Quantifying the Risk of Bad Bots
30
Context: The Dual Roles of Modern Information Security Professionals
Subject Matter Experts Trusted Advisors
31
Two Questions Modern Information Security Professionals Must Answer
What is the risk of [x]? How does an investment in [y] quantifiably reduce that risk?
32
Three Challenges Modern Information Security Professionals Must Overcome
What is the risk of [x]?
• A language challenge
• A measurement challenge
How does an investment in [y] quantifiably reduce that risk?
• A communications challenge
33
The Threat of Bad Bots: A Material Percentage of Web Site Traffic
Bad Bots Good Bots Humans
18.6% 27.0% 54.4%
Source: Distil Networks, 2016 Bad Bot Landscape Report
34
Web Site Vulnerabilities and Exploits Related to Bad BotsBad Bot Vulnerabilities and Exploits (illustrative)
Web Security
Brute force login; account takeover; fraudulent account creation
Man-in-the-browser attacks
Reconnaissance attacks; application coding exploits
Application denial of service
Spam
Web Scraping
Content theft
Price scraping
API scraping
Competitive data mining
Waste and Abuse
Web site performance
Negative SEO
Skewed web site analytics
Fraud Fraudulent transactions
Digital ad fraud
Source: adapted from Distil Networks, 2016 Bad Bot Landscape Report; Aberdeen Group, April 2016
35
The Risk of Bad Bots: How Likely? What Business Impact?Bad Bot Vulnerabilities and Exploits (illustrative) Likelihood Impact
Web Security
Brute force login; account takeover; fraudulent account creation
How Likely is it that these
Vulnerabilities are Successfully Exploited?
What is the Business Impact, when Successful
Exploits Do Occur?
Man-in-the-browser attacks
Reconnaissance attacks; application coding exploits
Application denial of service
Spam
Web Scraping
Content theft
Price scraping
API scraping
Competitive data mining
Waste and Abuse
Web site performance
Negative SEO
Skewed web site analytics
Fraud Fraudulent transactions
Digital ad fraud
Source: adapted from Distil Networks, 2016 Bad Bot Landscape Report; Aberdeen Group, April 2016
36
Qualitatively, Four Categories for the Business Impact of Bad Bots
• Additional cost
• Data breaches
• Loss of current revenue
• Loss of future revenue
37
At a Qualitative Level, the Business Impact of Bad BotsBad Bot Vulnerabilities and Exploits (illustrative) Likelihood Incr.
CostData Loss
Curr.Rev.
Fut. Rev.
Web Security
Brute force login; account takeover; fraudulent account creation
How Likely is it that these
Vulnerabilities are Successfully Exploited?
X X X X
Man-in-the-browser attacks X X X X
Reconnaissance attacks; application coding exploits X X X X
Application denial of service X X X
Spam X X
Web Scraping
Content theft X X X X
Price scraping X X X X
API scraping X X X X
Competitive data mining X X X X
Waste and Abuse
Web site performance X X X
Negative SEO X X X
Skewed web site analytics X X X
Fraud Fraudulent transactions X X X
Digital ad fraud X X
Source: adapted from Distil Networks, 2016 Bad Bot Landscape Report; Aberdeen Group, April 2016
38
There are Many Approaches to Measuring and Communicating Risk that We’re All Familiar With … But These Don’t Really Work!
Techno-babble
about threats, vulnerabilities,
and exploitsHeadlines
of recent breach
disclosures
ALE-stylecalculations
Averages, based on surveys
Crackpot rigor
Qualitative “heat
maps”
“$201 / record”
39
With These Approaches, Most Decisions About Security-Related Risks are Still Made by the Intuition and Gut Instinct of the HiPPO …
(The Highest-Paid Person in the Organization)
40
Let’s Try to Raise the Bar for Making Important Decisions AboutSecurity-Related Risks, Beyond Mere Intuition and Gut Instinct!
Source: http://dilbert.com/strip/2016-03-24
41
Modeling the Risk of Bad Bots
• Let’s estimate the risk (both likelihood, and impact) of bad bots, using these four high-level categories:
• Additional cost• Data breaches• Loss of current revenue• Loss of future revenue
• Remember that risk is inherently about making decisions in the face of uncertainties
• Models are not about precision …• … they are about making better-informed
decisions about risk … • … most of which are based primarily on
intuition
42
Monte Carlo Modeling is a Proven, Widely Used Solution for our
Measurement Problem
• In a nutshell: we can carry out the same familiar estimates and computations we have traditionally made
• Except that we do this for many (say, ten thousand) scenarios, each of which uses a random value from our estimated ranges and distributions
• The results of these computations are likewise not a single, static number – which says nothing about risk
• The output is also a range and distribution, from which we can readily describe both probabilities and business impact
• I.e., the results can be expressed in terms of risk – which is exactly what we are looking for!
43
We’re All Familiar with This Approach, Too – Note the Inclusion of Both Likelihood and Impact in This Illustrative Example!
44
Just So Long As We Don’t Do This … Remember, All Models Are Wrong – But Some Can Be Useful!
Source: http://dilbert.com/strip/2016-04-01
45
Risk of Bad Bots
Additional Cost
Overprovisioning of web site infrastructure
Web site contribution to
annual revenue
Data breaches Loss of Current Revenue Loss of Future Revenue
Factoring the Risk of Bad Bots – Conceptual
$ $ $ $
Source: Aberdeen Group, April 2016
% of annual revenue spent
on web site infrastructure
% of web traffic
represented by bad bots
Web site contribution to
annual revenue
% of annual revenue spent
of website marketing
% of web traffic
represented by bad bots
# of “incidents”
represented by bad bots
(i.e., an attempt)
Likelihood of a “breach”
(i.e., a success)
Business impact of a
breach
Web site contribution to
annual revenue
Web site contribution to
annual revenue
Time that web site is
negatively affected (e.g., downtime or slowdown)
% of revenue lost during the
period of downtime or slowdown
% of web traffic
represented by bad bots
% of website revenue lost as a result of
fraud
Wasted web site marketing Cost of data breaching Downtime and slowdown Fraudulent transactions
$
46
Factoring the Risk of Bad Bots – Computational
Source: Aberdeen Group, April 2016
47
Run the Numbers – The Results Provide Invaluable Insights into the Risk of Bad Bots
Histogram
Probability Curve
Source: Aberdeen Group, April 2016
48
Quantifying the Risk of Bad Bots
Source: Aberdeen Group, April 2016
49
Quantifying the Risk of Bad Bots … and Addressing the Two Fundamental Questions
• For a web site contributing $100M / year in revenue
(% of web site annual revenue)
• Median annual reduction in risk: about 18 times
• Median annual return on investment: about 22 times
• Note: the risk owner still needs to decide …
Source: Aberdeen Group, April 2016
Distil Networks 2016 Bad Bot Report: Quantifying the Risk and Economic Impact of Bad Bots
Distil Networks has produced their third annual Bad Bot Report. It's the IT Security Industry's most in-depth analysis on the sources, types, and sophistication levels of last year's bot attacks -- and there are serious implications for anyone responsible for securing websites and APIs.
Join Derek Brink, Vice President of Research at Aberdeen Group and Rami Essaid, CEO of Distil Networks as they dive into the data to reveal:
● 6 high-risk lessons every IT security pro must know
● How to quantify the risk and economic impact of bad bots for your organization
● How bot activity varies across websites based on industry and popularity
● The worst offending countries, ISPs, mobile operators, and hosting providers
Bad bots are the key culprits behind web scraping, brute force attacks, competitive data mining, online fraud, account hijacking, unauthorized vulnerability scans, spam, man-in-the-middle attacks, digital ad fraud, and downtime. Register today to gain actionable insights on how to defend your websites and APIs for the coming year of threats.
Bonus: All registrants will receive a copy of Distil Networks’ 2016 Bad Bot Reports and a copy of the presentation slides.
Abstract
52
Modeling the Risk of Bad Bots: Additional Cost (1)
1. Web site contribution to annual revenue ($ / year)• For the purposes of this analysis, let’s model based on $100,000,000
2. % of annual revenue spent on web site infrastructure• “Infrastructure” = all related people, process, technologies• Model as 4% - 6%; uniform distribution (analyst estimates)
3. % of web traffic represented by bad bots• Model as 0% - 50%; most likely 18.6%; beta distribution (Distil Networks)
4. Annual cost of overprovisioning web site infrastructure• (1) x (2) x (3)
Source: Aberdeen Group, April 2016
53
Modeling the Risk of Bad Bots: Additional Cost (2)
1. Web site contribution to annual revenue ($ / year)• For the purposes of this analysis, let’s model based on $100,000,000
2. % of annual revenue spent on web site marketing• “Marketing” = all costs related to driving web traffic• Model as 5% - 15%; normal distribution (analyst estimates)
3. % of web traffic represented by bad bots• Model as 0% - 50%; most likely 18.6%; beta distribution (Distil Networks)
4. Annual cost of wasted web site marketing (e.g., negative SEO, skewed web site analytics, etc.) resulting from bad bots
• (1) x (2) x (3)
Source: Aberdeen Group, April 2016
54
Modeling the Risk of Bad Bots: Data Breaches
1. # of “incidents” represented by bad bots (i.e., an attempt)• One extreme: all bad bots = 1 incident• The other extreme: every bad bot = 1 incident• My modeling choice: 1 (one incident per year) to 12 (one incident per month); beta distribution
2. Likelihood of a “breach” (i.e., a success)• 0% - 100%; mostly likely 30%; beta distribution (Verizon DBIR)
3. Business impact of a data breach• Expressed as a function of the number of records (Verizon DBIR)• Use 100,000 – 1,000,000 records as the range (Privacy Rights Clearinghouse)
4. Annual cost of data breaches resulting from bad bots• (1) x (2) x (3)
Source: Aberdeen Group, April 2016
55
Modeling the Risk of Bad Bots: Loss of Current Revenue (1)
• Bad bots → negative impact on web site availability and performance
• Combination of downtime and slowdown results in web site customers abandoning what they were trying to do … which leads to lost revenue during this time of disruption
1. Web site contribution to annual revenue ($ / year)• For the purposes of this analysis, let’s model based on $100,000,000
2. Time that web site is negatively affected (e.g., downtime or slowdown) (hours / year)• For simplicity, assume 24x7x365 operation• Model as 0 – 720 hours; most likely 200 hours; beta distribution (Arbor Networks)
3. % of revenue lost during the period of downtime or slowdown• Model as 1% to 30%; most likely 3%; beta distribution (analyst estimates)
4. Loss of current revenue as a result of bad bots• (1) x (2) x (3)
Source: Aberdeen Group, April 2016
56
Modeling the Risk of Bad Bots: Loss of Current Revenue (2)
• Bad bots → fraudulent transactions
1. Web site contribution to annual revenue ($ / year)• For the purposes of this analysis, let’s model based on $100,000,000
2. % of web site traffic represented by bad bots• 0% - 50%; most likely 18.6%; beta distribution (Distil Networks)
3. % of web site revenue lost as a result of fraud from bad bot traffic• Model as 0% – 10%; most likely 1.4%; beta distribution (Kroll, Global Fraud Survey)
4. Loss of current revenue as a result of bad bots• (1) x (2) x (3)
Source: Aberdeen Group, April 2016
57
Final Important Detail: Effectiveness of Countermeasures for Bad Bots
• Status quo = manual blocking
• 0% - 50%; most likely 12%; beta distribution• Assume that the annual cost of manual blocking is already
baked in to the cost of overprovisioned web site infrastructure
• Future state = use the Distil Networks solution
• 90% - 100%; mostly likely 99.9%; beta distribution• The model for the future state must also incorporate
the annual cost of the Distil Networks solution
Source: adapted from Distil Networks, 2016 Bad Bot Landscape Report; Aberdeen Group, April 2016