The definitive source for network performance metrics Speedtest... · Benefits of a dedicated...

The definitive source for network performance metrics

methodology

2methodology

© 2006-2020 Ookla, LLC. All Rights Reserved

ContentsIntroduction 1

How Ookla® collects data 2

Speedtest measures the full capacity of an internet connection 3

Quantity + quality of test data 3

Benefits of a dedicated server network 4

Testing to a CDN alone does not provide an accurate picture of network performance 5

Understanding the last mile of a network connection 5

Testing 5G and super-fast fiber connections 6

Accurate 5G connection detection and identifcation 6

Background testing does not provide a reliable measurement of speed 7

The hidden downsides of background testing from other providers 7

Ookla provides a reliable, consistent test experience across devices 8

Testing coverage, availability and quality of service 8

How Ookla aggregates and analyzes data 9

Ookla metrics 10

Standard internet speed metrics measured by Speedtest 10

Ookla performance metrics for mobile and fixed broadband 11

Coverage metrics for mobile broadband 12

Operational definitions 15

The Ookla product suite 16

1methodology


Introduction

Ookla® is fiercely committed to measuring the performance and availability of the internet worldwide and transparently reporting on that performance. For over 14 years, Ookla has been perfecting the way we test all aspects of internet performance.

Speedtest®, the company’s flagship product, is the most reliable tool for measuring internet performance and providing network diagnostics. Every day, millions of people use Speedtest to better understand the performance and quality of their internet connections. This consumer-initiated testing is the most accurate way to understand a network connection because it reflects how people actually experience the internet, when and where it actually matters to them.

Ookla’s Speedtest Server Network™ also ensures the most accurate possible test results anywhere in the world. Because we have servers in every country and major population center, Speedtest users always test using a nearby server, reducing travel time of the data as it communicates between the device and server.

We subject this robust amount of data to rigorous aggregation and analysis, setting Speedtest apart and making Ookla the preferred data provider for telecommunications operators, regulatory bodies, trade bodies, analysts, journalists and nonprofits worldwide.

This document shares the details of Ookla’s methodology, including the metrics we cover, how our methods differ from other network testing providers, and why Ookla is uniquely capable of measuring super-fast speeds and 5G performance.

10,000,000+TESTS DAILY

300,000,000+COVERAGE SCANS DAILY

10,000+SERVERS GLOBALLYIN OVER 190 COUNTRIES

30,000,000,000+TESTS TO DATE

100%OF THE WORLD’S COUNTRIES REPRESENTED IN OOKLA DATA

2methodology


How Ookla collects data

Each time a user takes a Speedtest, a snapshot of the internet is captured for that given time, place, device and network. Because these tests are initiated by consumers, Speedtest gives users accurate information about internet speeds at the times and locations that are important to them. When aggregated, these measurements describe the network’s real-world performance, including information about locations, times, service providers and devices.

A test taken with Speedtest measures the characteristics of the communication network between a device and a server using our global server network. The fewer links between a device and a server, the more relevant the measurement is to quantifying and understanding the networking capability of a particular device. To that end, Ookla operates a vast testing infrastructure of over 10,000 servers worldwide which allows us to ensure an accurate and meaningful view of network performance by minimizing the number of links between device and server.

Along with network performance metrics such as download speed and latency, mobile Speedtest users on Android devices can also opt in to submit data from background coverage scans. Our users contribute billions of measurements on the quality and conditions of mobile networks in their area from over 300 million scans each day. These coverage scans provide real-time insights into signal conditions, spectrum usage and network equipment at a fine level of geographic detail.

Periodic background tests are limited in what they can measure

The dedicated foreground test from Speedtest measures full throughput

Speedtest is the most accurate measure of real-world network

performance and coverage

3methodology


Speedtest measures the full capacity of an internet connection When a consumer decides to take a test using Speedtest, it is run as a dedicated foreground service. This is the most accurate measure of internet performance capability because it is able to use enough data to flood the internet connection and measure the full capability of both the network connection and the device. The connection must be fully utilized to accurately assess network performance and quality metrics, such as download speed, upload speed, latency, packet loss, jitter and other indicators of network conditions.

Companies that rely on background tests hidden inside of other apps to measure speed only send small amounts of data back and forth. This is not a sound method for accurately measuring performance, particularly at high speeds. While useful for assessing metrics like signal and coverage, background tests are not able to flood an internet connection with a sufficient volume of data to measure the full throughput capability. Many companies claiming to test speed do so via background testing, which means that their measurements are incomplete and do not offer a true picture of performance.

Because Speedtest operates in the foreground and measures the full throughput capacity of a connection, we can properly measure the performance capability of each connection. This is why speeds measured with Speedtest are often higher than those measured with other methods. This difference can be substantial when testing the performance of newer technologies like 5G and multi-gigabit fiber. Furthermore, background tests that run on a predefined schedule that suits the data providers’ needs don’t always reflect accurate information about internet speeds at the times and locations that are important to consumers.

Quantity + quality of test data

Numbers matter in data collection, and it’s important to look deeply at what the numbers represent. If a testing provider were to run 100 background tests per day on 100 phones for 100 days that record 100 values each, they’d have 1 million measurements. That overall number might sound impressive — until you realize that 100 phones do not give you a very wide (or interesting) distribution of phone types, locations or even experiences. The numbers are even less impressive when you realize that the 100 data points they’re collecting include less significant details like the device’s screen width and battery type. Those might be interesting data points for tiny cross sections of our industry, but they contribute little valuable information about internet speed or performance at a global scale. When test providers bury their SDK in third-party apps, the number of “measurements” can be smoke and mirrors designed to distract from a low-quality user base.

Each and every day, over 10 million tests are actively initiated by pressing the “Go” button on Speedtest, and we receive additional data from over 300 million coverage scans. We see daily results on almost every mobile and fixed broadband network in the world, which provides us with a real-time view of how the internet is performing at a global scale. This constant flow of immense amounts of data allows us to precisely track how networks respond to events like large crowds, the capabilities of new devices, the impact of network upgrades and the rollout of new technologies like 5G.

4methodology


Benefits of a dedicated server network

Testing to a local server eliminates latency and bottlenecks that can skew performance metrics. Each link and node through which data is transferred can affect the final measurements as the link with the most constraining characteristics (highest latency, lowest bandwidth or highest packet loss) will typically limit the final measurements. Therefore, the fewer links between a device and a server, the more relevant the measurement is to qualifying and understanding the networking capability of a particular device.

Speedtest applications leverage a vast testing infrastructure with over 10,000 servers in more than 190 countries. By having multiple servers in every country and major population center, Speedtest ensures an accurate and meaningful view of networking performance by minimizing the number of data links between device and server.

Furthermore, by maintaining servers in a broad selection of networks, we are able to measure the performance of both the last mile connection and connections elsewhere on the internet. Ookla also takes comprehensive steps to ensure our traffic is indistinguishable from other applications or browser traffic to the server.

Nearby servers have optimal and full performance

Distant servers introduce latency and bottlenecks which reduce performance

5methodology


Testing to a CDN alone does not provide an accurate picture of network performance

Ookla is focused on providing the most accurate quality of service (QoS) measurement technology possible. To do this, it’s vital to measure the service provided to the end user by their ISPs and mobile operators. This means getting closer to the user than can be achieved by testing to a content delivery network (CDN) or multiple CDNs.

Although CDNs serve a large portion of internet content, they do not represent internet performance as a whole. Testing to a CDN means testing to a single provider, often in a distant location. This only measures the connection from that specific CDN. CDNs also introduce several additional variables that can affect how consumers experience the internet, including peering relationships, performance issues between provider networks and CDNs, and the geographic location of CDN servers.

By leveraging our network of over 10,000 high-performance test servers around the world, Ookla is uniquely able to evaluate the service provided by every ISP and mobile operator in the world — and to remove the variability that comes with measuring CDN performance.

We recognize the value in measuring a user’s performance beyond the last mile, so Speedtest allows testing to various providers in diverse locations. In this way we can assess various connection scenarios to understand the full potential of a connection, instead of being limited to a single provider. Only Ookla allows you to detect points of network congestion, and gain a better understanding of internet performance as a whole.

Understanding the last mile of a network connection

The “last mile” refers to the final leg of a telecommunications network that delivers services to an end user. Specifically, the last mile refers to the portion of the network chain that physically reaches the end-user’s premises, including coaxial cable service drops carrying signals from utility poles to subscribers’ homes and cell towers linking local mobile devices to the cellular network. Limitations to the last-mile connection can create bottlenecks for speed for individual internet users. This is the part of a user’s internet experience that a provider has the most control over — and responsibility for. Because Ookla connects to dedicated local servers, consumers can more accurately measure and troubleshoot the network connection they’re actually paying for, based on the location from which they are testing.

6methodology


Testing 5G and super-fast fiber connections

The ability to measure a network’s full capacity is especially critical with modern network speeds increasing rapidly across connection types like 4G, 5G and fiber broadband. Speedtest is uniquely capable of measuring the throughput of a 5G connection and other gigabit-and-beyond speeds. Speedtest uses a client and server testing engine that dynamically scales the number of connections to the server in order to saturate and accurately measure client-side connections up to 10 Gbps. This allows Ookla to measure the full extent of real-world performance and overcome the effects of network bottlenecks such as TCP slow start (a transmission control protocol that avoids sending more data than the network is capable of forwarding).

Accurate 5G connection detection and identification Not all 5G-capable devices natively identify the 5G connection type when reporting to applications. That’s why Ookla has directly partnered with device manufacturers worldwide to implement accurate in-app 5G detection in Speedtest. Consumers can reliably see when they’re testing a 5G connection in the Speedtest app, which provides a valuable feedback loop of 5G network service.

7methodology


Background testing does not provide a reliable measurement of speed

Other testing providers often report much slower speeds than Speedtest. That’s because their tests either run in the background of other apps without consumers’ knowledge or simply download a very small file from a content delivery network (CDN) or server. These quick tests cannot scale up to assess the full capacity of a connection or accurately determine upload speed — and the results do not indicate how a connection might actually perform while using large amounts of data over a 5G connection. Because a significant amount of demand on the network must be generated for 5G carrier aggregation to even take effect, shorter-duration tests are incapable of ensuring the network assigns the maximum number of component carriers to the user equipment (UE) to trigger a 5G connection.

The hidden downsides of background testing from other providers

Other internet testing solutions run in the background of third-party applications that users are often unaware of, such as messengers, call recorders, dating apps and media converters. Speedtest is used and trusted by hundreds of millions of consumers worldwide to do exactly what they expect: to provide an accurate, reliable measure of the performance and quality of their networks.

Due to protections Apple has in place for user privacy, iOS doesn’t surface information like connection type to apps that run in the background. It is vital that a testing solution run in the foreground on iOS to accurately test the network and collect information about the active SIM and access technology. Aggregate speeds you might see from other network testing providers do not adequately represent the results of iOS users, which make up a large and important segment of the market.

As Google continues to update their data privacy policies for Android, having a dedicated app with a transparent consumer experience and clear user permissions around location data collection is increasingly important.

8methodology


Ookla provides a reliable, consistent test experience across devices

Speedtest provides an accurate, consistent test experience across devices. We have mobile apps for both Android and iOS that are optimized to measure the specific intricacies of a mobile connection. There are dedicated Speedtest apps for computers (Mac and Windows), a browser extension for Chrome, an AppleTV app and embedded solutions that integrate with routers, hardware and other connected devices. For developers who want to use the trusted technology and global server network behind Speedtest to power their own solutions, we also offer a command-line interface tool (Speedtest CLI) and an SDK. Our original web test at Speedtest.net covers most other platforms, and we continue to explore new platforms for Speedtest.

Testing coverage, availability and quality of service

Fast speeds only matter when you actually have mobile coverage. Consumer-initiated testing is the best measure for speed and several other metrics, but to measure signal and coverage Speedtest also collects over 300 million daily scans of coverage data in the background, submitted by Speedtest Android app users. These coverage scans capture where service is offered and what the quality of service is at each location.

android

Android iOS Windows Mac Chrome AppleTV CLI

9methodology


How Ookla aggregates and analyzes data

We use a rigorous statistical sampling methodology to combat sampling bias and ensure data accuracy. Through consumer-initiated testing, Speedtest gives every user a voice to describe what the internet connection is like on their device at the times and places that matter most to them. Whether a user takes a Speedtest once per month or once per hour, our sampling methodology makes sure that each user’s voice is heard and is not drowned out by high-volume testers.

When we aggregate data from all of the tests we’ve received, each unique Speedtest user’s results are averaged to create a single sample that summarizes their internet experience for that time period and geographic area. We then evaluate each service provider based equally on the samples provided by each of its users. This removes the potential for results to be skewed by individual outliers or short-term fluctuations in service or user behaviors.

Network engineers frequently use Speedtest while developing next-generation technologies. To ensure that our results represent the true commercially available user experience, results from these tests, which include those taken from our CLI tool, are excluded from our published results. This is because these tests are often performed in controlled environments that do not necessarily reflect the real-world conditions consumers see. For various reasons, results from the following platforms are also excluded from our aggregates: mobile web tests taken on Speedtest.net, tests on Windows phones, the Chrome app and through our embedded solutions.

Finally, our data scientists and analysts employ a host of tools that allow us to identify and remove any tests that may intentionally or unintentionally bias our results.

Consumer-initiated = speedBackground = coverage

A user’s mobile internet experience is greatly impacted by coverage. With coverage scans, we also examine information about a mobile user’s “radio environment,” including the technology used (e.g., 5G, 4G LTE, etc.), the cellular infrastructure to which they are connected and the accompanying strength and quality of signal. This allows us to better understand coverage in that area and to provide more information about why particular Speedtest results were seen. We use the hundreds of millions of coverage scans that we receive each day to provide an accurate and up-to-date understanding of the coverage offered by mobile operators.

Consumer-initiated testing and coverage scans complement one another, and together they provide an unparalleled amount of data on performance and coverage. Here is an example of what that looks like in Brazil.

10methodology


Ookla metrics

Here at Ookla, we are continuously researching new and better metrics to describe real-world internet performance and coverage. Our metrics help consumers to understand their network performance and service providers to understand their customer experience, gain competitive insights and identify how to improve their networks.

Standard internet speed metrics measured by Speedtest

Download speed Upload speed Latency / Ping Jitter Packet loss

Download speed measures how quickly you can pull data from a server on the

internet to your device. Most connections are designed to download much faster than they upload. This is because

the majority of online activity, like loading web pages or streaming videos, is driven by downloading content.

Download speed is measured in megabits per second

(Mbps).

Upload speed measures how quickly you send data from

your device to the internet. A fast upload speed is helpful when sending large files via email, or in using video-chat

to talk to someone else online (since you have to send

your video feed to them). Upload speed is measured in megabits per second (Mbps).

Also called latency, ping is the reaction time of your connection — that is, how quickly your device gets a

response after you’ve sent out a request. A fast ping means a more responsive connection,

especially in applications where timing is everything (like video games). Ping is measured in milliseconds

(ms).

Also called Packet Delay Variation (PDV), jitter is a measure of the variability in ping over time. Jitter is

not usually noticeable when reading text, but when streaming and gaming a high jitter can result in buffering and other

interruptions. This metric shows the average difference between consecutive latency

measurements.

Packet loss occurs when a packet of data being sent

over the internet is not received or is incomplete.

This is described in the percentage of packets lost compared to packets sent. Packet loss can be a result

of network congestion, poor signal or line quality. Even small amounts of packet loss can severely degrade

the quality of experience for real-time applications such as videoconferencing and

gaming.

11methodology


Ookla performance metrics for mobile and fixed broadband

Speed Score™

We created Speed Score to fully account for the range of speeds a provider offers in a single metric. This makes it easier to compare mobile operators and ISPs on several measures of speed using one metric.

Speed Score incorporates a measure of each provider’s download and upload speed to rank network speed performance (90% of the final Speed Score is attributed to download speed and the remaining 10% to upload speed because online experiences are typically more affected by download speed). Speed Score uses a modified trimean to combine speeds from the 10th percentile, 50th percentile (also known as the median) and 90th percentile in a weighted average using a 1:8:1 ratio, respectively. We place the most emphasis on the median speeds as those represent what most network providers’ customers will experience on a day-to-day basis.

Consistency Score™

While fast speeds are paramount, a consistent experience is also a worthy measure of a network’s quality. Consistency Score is the metric we use to identify fixed broadband or mobile networks that provide a consistent quality of service. It reflects the percentage of a provider’s data samples that meet minimum thresholds for download and upload speeds, depending on the type of network. Consistency Score for fixed broadband uses thresholds of 25 Mbps minimum download speed (the recommended speed for streaming in 4K) and 3 Mbps minimum upload speed. Consistency Score for mobile uses thresholds of 5 Mbps minimum download speed (the recommended speed for streaming in HD) and 1 Mbps minimum upload speed. The higher a provider’s Consistency Score, the more likely a consumer will enjoy acceptable internet performance and quality.

Competitive Geography

The Competitive Geography filter, which is currently only applied to the United States, ensures a balanced comparison of national wireless operators by eliminating geographic outliers. To meet the definition of “competitive” in the U.S., a ZIP code must contain samples from at least three top national competitors (those who have at least 3% of market share at a national level), but no competitor can have more than 2/3 of the samples in that ZIP code. Operators are considered present in a ZIP code if they have at least 3% of the samples in the area and show samples on multiple devices. Limiting any operator from having more than 2/3 of samples ensures actual competition in a ZIP code rather than including areas where one competitor dominates the market.

12methodology


Coverage metrics for mobile broadband

Ookla determines an operator’s geographic coverage using a sample of scans received from devices on that operator’s network in each 100 m2 area. Because coverage is a spatially-focused metric, only scans with precise and legitimate location information are included as we build samples that normalize data by user, operator, location and timeframe.

Android Speedtest users can enable or disable coverage scan collection at any time. When enabled, Android devices running the Speedtest app periodically capture key characteristics of each user’s experience with the surrounding mobile networks throughout the day. Each coverage scan captures information about the mobile network that device is connected to, including information about the connection to that network.

As with our consumer-initiated tests, each coverage scan provides a wealth of information, including:

Operator Radio signal Connection Radio access network Device location

Information about the subscriber network and

the network to which the device is connected

Information about signal strength and quality (e.g., RSRP, RSRQ,

RSSI), frequency band (EARFCN, UARFCN, ARFCN) and other

properties of the cellular connection

Data connection type (e.g., LTE, WiFi) and

cellular connection type

Tracking area (LAC/TAC), base station ID, cell ID, physical cell ID

(PCI)

Latitude, longitude, altitude, accuracy, and age of location (when accurate GPS locations

are not available, IP address geolocation

is used)

13methodology


Coverage Score™

We created Coverage Score to account for both the quality and coverage of service for mobile operators. Coverage Score multiplies the proportion of locations in which an operator was seen with service (its footprint) against the average tile quality score (based on availability metrics) among all locations in which that operator is present. Coverage Score has a range of 0 to 1000 to avoid any potential for confusion that te Coverage Score represents a percentage of an area or population with coverage.

AvailabilityWhether or not service is available is closely tied to coverage. We divide our data on availability into three separate views: General Availability, 4G Availability, and On-Network Availability. These metrics indicate how likely a user, on average, is to have service available in the places they go. They are individually defined as:

• General Availability The percentage of an operator’s known locations where a device has access to any kind of service (including roaming).

• 4G Availability The percentage of an operator’s known locations where a device has access to 4G LTE service (including roaming).

• On-Network Availability The percentage of an operator’s known locations where a device has access to service from that operator.

Coverage Score and Average Tile Quality

14methodology


Time SpentWhile it might be easy to assume that a user connects to a 4G signal 100% of the time, actual experience varies. We use Time Spent to give mobile operators insight into the percent of time that an average user spends on a given cellular technology — both on and off of their subscriber (SIM) network. For example, we could see that users in one area are using 74% 4G, 6% Roaming 4G and 20% 3G. This can help pinpoint gaps in LTE coverage, among other things. Time Spent uses coverage scans to construct daily timelines for each device. Then, daily per-device durations on subscriber network, active network and cellular technology are calculated for a given area of interest. Next, the percent time spent on cellular technology per-device based on the area and time period of interest are determined. Finally, we aggregate the average percent time over devices for a subscriber network to give the final metrics seen in Speedtest Intelligence™.

Mean Percent Time Spent

15methodology


Operational definitions

It’s also important to define four of the key phrases we use in our articles and reports.

Top providersTo be designated as a top provider, an operator or ISPs must account for 3% or more of total test samples in the market for the period. We use this designation to ensure that most consumers in the area actually have access to the provider to qualify it as the fastest, most consistent or having the best coverage.

Modern chipsetsOokla maintains a list of device models built on chipsets capable of utilizing the latest network technologies. When calculating Speed Score and determining the winners of Speedtest Awards, we use results from devices built on modern chipsets so that an operator’s score is not negatively impacted if a portion of its subscriber base continues to use older technology.

FootprintFootprint is the fraction of locations within a given market, across all operators in that market, where a device has access to service. We use Footprint in calculating Coverage Score. Operators with a relatively small footprint will have a lower Coverage Score than competitors with equal availability and a larger footprint. We require that an operator’s footprint in their market is at least 30% before we will calculate a Coverage Score, however, to indicate that the operator is generally available to the public.

Mean (or average)We use “mean” and “average” interchangeably unless specifically stated otherwise.

16methodology


The Ookla Product Suite

Ookla’s enterprise solutions provide the most expansive view of worldwide network performance available through a suite of products trusted by Fortune 500 operators, governments, trade groups and other businesses around the world.

GLOBAL PERFORMANCEDATA AND ANALYSIS

Speedtest Intelligence provides current insights into global fixed broadband and mobile network performance, availability and quality. Leveraging billions of consumer-initiated tests, Speedtest Intelligence is an essential tool to help organizations make business and technical decisions with a full understanding of both the speed and quality of networks in their markets.

WIRELESS NETWORK ASSET INTELLIGENCE

TowerSource is the U.S. tower industry’s largest and most comprehensive independent vertical asset database.

YOUR CUSTOMIZED SPEEDTEST

Leverage the renowned methodology and worldwide server network of Speedtest in your own mobile-friendly, HTML5-based testing solution.

COMPREHENSIVE MONITORING FOR ONLINE SERVICE PROVIDERS

Leveraging over 22 million monthly problem reports, Downdetector Enterprise provides real-time outage detection, early alerting, historical data around service outages and insight into third-party outages, so your team can reduce downtime and increase customer satisfaction.

PERFORMANCE, COVERAGE AND SIGNAL MEASUREMENT DATA

Powered by billions of measurements from Speedtest, Cell Analytics provides insights about wireless service quality, RF measurements, data usage, user density, indoor vs. outdoor performance, cell site locations and much more — surfaced in a powerful, yet easy-to-navigate portal.

INTEGRATED NETWORK TESTING SOLUTIONS

Speedtest Powered brings the power and accuracy of Ookla testing to any product or service capable of connecting to the internet with an array of enterprise licensing options, including embedded testing solutions, hosted web testing, SDKs and more.

COVERAGE AND SPECTRUM LICENSE INFORMATION

With a curated global collection of coverage information publicly marketed by mobile operators, as well as data on mobile licensing in North America, our geospatial data helps organizations make more informed business decisions.

STRATEGIC NETWORK PLANNING

This award-winning cloud-based service visualizes wireless, cable, broadcast and landline network coverage intelligence to help organizations explore global networks, analyze spectrum depth, identify competitors and share research — all from one location.

PERSONALIZED NETWORK COVERAGE MAPS

CellMaps is a user-friendly, interactive mapping platform that showcases your company’s personalized network coverage map, letting customers know where they can expect to experience a quality connection on your network.

Speedtest Intelligence®

TowerSource™

Speedtest Custom™

Downdetector Enterprise®Cell Analytics™

Speedtest Powered™

GIS Solutions™ MapELEMENTS®

CellMaps®

17methodology


To learn more, inquire at https://ookla.com/enterprise

Marketing and brand programs Speedtest Awards™

Ookla licenses data access to help companies both understand and demonstrate the benefits of their networks. With claims backed by Ookla, you can

differentiate yourself in the market with meaningful, verifiable marketing and brand campaigns.

With millions of tests taken every day, Ookla has developed a robust and comprehensive view of

worldwide internet performance. The Speedtest Awards acknowledge the fastest and top performing ISPs and

mobile networks around the world.

The definitive source for network performance metrics Speedtest... · Benefits of a dedicated...

Documents

Transcript of The definitive source for network performance metrics Speedtest... · Benefits of a dedicated...