Superdome Benchmarking Report - ZK Serviceszkservices.net/files/Download/Superdome... · ZK...

February 22, 2013 1

Indoor Benchmarking Test and AnalysisConducted on Feb. 1, 2013

New Orleans Superdome

Superbowl XLVII

ZK Celltest, Inc256 Gibraltar Drive

Suite 109Sunnyvale, CA 94089

www.zk.com1-800-TESTCEL

February 22, 2013

Table of Contents

1.0 Executive Summary.................................................................................................11.1 Data Test ......................................................................................................11.2 Voice Calls ...................................................................................................21.3 Coverage ......................................................................................................21.4 Design Approaches ......................................................................................2

2.0 Introduction..............................................................................................................4

3.0 Scope........................................................................................................................4

4.0 Testing Plan..............................................................................................................5

5.0 Superdome DAS Networks......................................................................................55.1 The Superdome ............................................................................................55.2 DAS Networks .............................................................................................65.3 Design Approaches ......................................................................................7

6.0 Test Setup.................................................................................................................76.1 Scanner Setup ..............................................................................................86.2 Device Setup ................................................................................................96.3 Floor Plan Setup...........................................................................................96.4 Post-Processing..........................................................................................10

7.0 Data Analysis .........................................................................................................107.1 Coverage ....................................................................................................107.2 Sectorization ..............................................................................................127.3 Throughput.................................................................................................177.4 Voice Quality .............................................................................................19

8.0 Conclusion .............................................................................................................19

ZK Celltest, Inc. Superdome Benchmarking Test and Analysis

1.0 Executive Summary

Voice and data tests along with scanning data was collected in the Superdome for 4 major wireless Carriers Verizon, AT&T, T-Mobile and Sprint. The ZK-MPS/B product was used to collect the data.

Both AT&T and Verizon have spectrum at 700MHz, 850MHz and 1900MHz. They offer 2G, 3G and 4G service. T-Mobile has two bands in the AWS spectrum and Sprint has one band in the 1900MHz spectrum. Ultimately, the amount of available spectrum seems to have dictated the ability for each Carrier to deliver service.

If you were going to use our phones on game day to send photos or videos and/or to surf the web then we strongly suggest you have Verizon or AT&T service due to their superior data rates and spectrum capacity. If you are just texting and doing voice calls then all of the Carriers’ service should suffice.

1.1 Data Test

The following is the result of the data testing. Sprint was not tested for data as they have a 3G network that has limited speeds. Verizon and AT&T data rates are on par with 4G tech-nology.

Also note that AT&T used more sectors than Verizon for 4G and quite a few more than T-Mobile. This will be discussed later in the report.

February 22, 2013 1


1.2 Voice Calls

Repeating voice calls were made on all four Carriers. The performance of all the Carriers was good. There were some minor differences in signal levels and error rates.

1.3 Coverage

The scanning data shown below indicates that Verizon’s system transmitted at a much higher power than the others across all of their serving bands. All Carriers covered the Superdome well for the areas that we tested. We did not test concession areas, press boxes or suites though.

1.4 Design Approaches

In terms of design Verizon chose to have fewer sectors and transmit at a higher power than AT&T while AT&T had many more sectors at much lower transmit levels.

For 2G/3G we determined the approximate number of sectors for level 1.

AT&T: 12 SectorsVerizon: 4 SectorsSprint: 9 SectorsT-Mobile: 4 Sectors

For 4G Verizon has more sectors (see Section 1.1 Data Test on page 1 above) but still fewer than AT&T. AT&T also has a higher SINR measure which indicates lower noise from their network but Verizon’s SINR is still above acceptable levels. It is hard to tell which approach will perform better on game day. Verizon may cover more sections of the

February 22, 2013 2


Superdome better but AT&T has more sectors so they have a higher maximum number of connections to provide.

In terms of throughput they will most likely offer about the same and our assumption is that the user perceived service for AT&T and Verizon will be similar.

T-Mobile and Sprint will have much lower data speeds. Both of these Carriers have lim-ited spectrum compared to Verizon and AT&T. In addition they do not have a 4G technol-ogy deployed. T-Mobile has deployed HSPA technology and Sprint has deployed EVDO.

February 22, 2013 3


2.0 Introduction

Most wireless Operators have "baseline" products and services that measure the "user-per-ceived" performance of their network as compared to the other wireless Operators in the same market (their competition) in delivering wireless voice and data services.

While it seems recently that the top two Carriers, AT&T and Verizon command a domi-nant share of the wireless space there are many other national, regional and local wireless Operators competing for the monthly fees of the wireless user.

There are many factors in the user's decision on which carrier to go with: pricing plans, coverage and network quality being the top factors.

FIGURE 1. New Orleans Superdome

3.0 Scope

This paper focuses on coverage and quality factors in comparing networks in the space that is a growing area of customer complaints, indoors. Other than the most remote areas coverage outdoors is pretty much ubiquitous. There continues to be performance issues outdoors and as long as the industry struggles to keep up with the exponential demand for more data it will continue. That is not the focus of this paper though.

This paper keys in on the indoor delivery and demand for voice and data services with the specific application of an indoor benchmarking test and analysis of the New Orleans Superdome conducted two days before Superbowl XLVII between the Baltimore Ravens and the San Francisco 49ers.

February 22, 2013 4


ZK Celltest was not commissioned by any wireless operator to perform this test. We obtained access to the Superdome ourselves and we were not paid to perform the test. Our test and analysis of the data is completely unbiased. Our attempt is to present the data fairly and if any assumptions we make turn out to be incorrect then we apologize to those affected and will update any information or assumption as necessary.

4.0 Testing Plan

The plan is to use the ZK Celltest ZK-MPS/B product to walk the Superdome and measure the quality and performance of Carriers AT&T, Verizon, T-Mobile and Sprint. Further analysis will be discussed on the different RF design approaches of each of the Carriers’. Voice and data calls will be made and the key parameters will be presented and compared for each of the Carriers. All the frequency bands of the Carriers were scanned in order to analyze delivered signal levels and the number of sectors used to deliver services.

5.0 Superdome DAS Networks

5.1 The Superdome

The Superdome is located in New Orleans, Louisiana, USA. Plans were drawn up in 1967, by the New Orleans modernist architectural firm of Curtis and Davis. Its steel frame cov-ers a 13-acre expanse. Its 273-foot (83 m) dome is made of a Lamella multi-ringed frame and has a diameter of 680 feet (210 m), making it the largest fixed domed structure in the world.

The dome has an interior space of 125,000,000 cubic feet (3,500,000 m3), a height of 253 feet (77.1 m), a dome diameter of 680 feet (207.3 m), and a total floor area of 269,000 square feet (24,991 m2). It can hold nearly 80,000 people.

February 22, 2013 5


5.2 DAS Networks

There are two DAS networks inside the Superdome. A neutral host system that T-Mobile, AT&T and Verizon use and a separate system that Sprint uses. The neutral host system has approximately 800 antennas covering all levels and areas of the Superdome. The Sprint system is older although has been updated recently.

FIGURE 2. Typical DAS and WiFi antennas

AT&T and Verizon deliver at LTE at 700MHz and 2G/3G at 850/1900 MHz. T-Mobile delivers HSPA/UMTS at AWS frequencies and Sprint delivers 2G/3G at PCS frequencies. It was assumed that data rates for Sprint would be well below those for T-Mobile. AT&T and Verizon based on this information so 3G data for Sprint was not tested.

February 22, 2013 6


5.3 Design Approaches

We had some limited amount of information as to the number of sectors each carrier used to deliver wireless on the respective DAS networks. This coupled with the test results pro-vided some insight as to the design approach each carrier took. As you may expect each was different in its own way.

Based on the data we received from various sources the following table indicates the fre-quency bands and technologies implemented for each carrier.

Table 1: Frequency Bands by Carrier

6.0 Test Setup

The ZK-MPS/B was used as the walk-around testing device. The product contains dual high-speed scanners that can scan multiple technologies on 700/850/1900/AWS frequency bands. It also allows the user to connect up to eleven (11) cellular handsets and/or modems (Devices) in order to capture voice and data calling performance information.

FIGURE 3. ZK-MPS/B

LTE CDMA/EVDO HSPA/UMTSVerizon 700MHz Cellular850, PCS1900AT&T 700MHz Cellular850, PCS1900Sprint PCS1900Tmobile AWS

February 22, 2013 7


The ZK-MPS/B unit integrates scanning hardware and computer processing and display into a single integrated unit. An internal dual battery system provides 4 - 8 hours of power for the entire system including powering the scanners, display interface and devices. The unit weighs 10 lbs including batteries and comes with a side/backpack carrying case that can swing to the side to make connections then swing to the back for walking. It has many compartments for devices and cables.

The user does not have to worry about charging multiple batteries since a single dual bat-tery system powers everything. Additional batteries are included along with an external charger so they can be easily swapped when required.

Log files were stored to the ZK internal memory card. After completion of the walk test the log files were copied to a computer and brought back to the office for further analysis.

6.1 Scanner SetupThe ZK-MPS/B was configured to scan the following channels and technologies:

Table 2: Channels scanned with MPS/B

700MHzLTE

850MHzCDMA/EVDO

850MHzUMTS

1900MHzCDMA/EVDO

1900MHzUMTS

AWSUMTS

Verizon 5230 78, 160 425, 625AT&T 5780 4412 587Sprint 25, 75TMobile 1987, 2012

February 22, 2013 8


6.2 Device Setup

Multiple devices were connected to the MPS/B. The MPS/B can support up to 11 simulta-neously connected devices. The following device types were connected. Note that Port 1 is reserved for the scanners.

Port 2: Verizon LTE Device Port 3: AT&T LTE DevicePort 4: Verizon CDMA/EVDO Device (Voice)Port 5: Verizon CDMA/EVDO Device (Data)Port 6: AT&T UMTS Device (Voice)Port 7: AT&T UMTS Device (Data)Port 8: Sprint CDMA/EVDO Device (Voice)Port 9: T-Mobile UMTS Device (Voice)Port 10: T-Mobile UMTS Device (Voice)Port 11: OpenPort 12: Open

6.3 Floor Plan Setup

Floor plan images for each of the Superdome levels were imported into the ZK-MPS/B. The user displays the floorplan image on the screen while walking the Superdome and drops breadcrumbs by moving the cursor and pressing a button. Although the Superdome is quite large, holding 80,000 people, a detailed walk of the lower seating level can be con-ducted in about fifteen (15) minutes.

A sample of the lower level image is shown below:

FIGURE 4. Floorplan image used for data collection

February 22, 2013 9


6.4 Post-Processing

Although ZK log files have an open, ASCII format and can be imported with many other third party software packages including Actix, Windcatcher and IBWave, this data analy-sis was performed using the ZK KPI Mapper software and ZK Benchmarker software. The KPI Mapper software converts the ZK log data and images into MapInfo tables and work-spaces. Maps of KPI's are auto-generated simplifying the analysis tasks. KPI Mapper also converts the condensed ZK log file into separate CSV files for each device and scanner band/technology. This makes it easier for other software to import the ZK data. The ZK Benchmarker software was used to mine the data and create tabular comparison informa-tion for VP level type reports.

7.0 Data Analysis

The goal of the analysis was to compare the voice and data performance of the Operators AT&T, Verizon, Sprint and T-Mobile for 2G, 3G and 4G technologies. In addition, based on the data we briefly describe varying network design approaches for each.

7.1 Coverage

In order to determine coverage both scanner data and phone data were used. We will first look at scanner data as this provides the most complete coverage information. Later we will look at coverage information coming from the phone as it relates to throughput and voice quality.

The scanner was programmed to scan channels for each carrier, for each technology within their frequency bands. The channels and technologies scanned for each carrier can be seen in Table 2: Channels scanned with MPS/B on page 8.

The following table shows the average signal strength and signal to noise for the strongest PN (CDMA/EVDO0, PCI (LTE) or SC (UMTS).

Table 3: Strongest PN, PCI SC

One fairly obvious difference is the average signal levels for Verizon are much stronger than for the others. Some sample plots follow to discuss this variance.

February 22, 2013 10


The following two plots show the strongest sector for 850 bands for Verizon and AT&T. The Verizon plot shows nothing below -8dB while the AT&T plot shows quite a bit of data below -8dB. It is clear that Verizon’s design is to transmit at a much higher power than AT&T.

FIGURE 5. Verizon 850 Band Strongest Ec/Io



FIGURE 6. AT&T 850 Band Strongest Ec/Io

7.2 Sectorization

Another aspect of the network to look at from the scanning perspective is the sector plot. For LTE, sectors can be plotted by their PCI as networks have a different PCI for each sec-tor. The same is true for PN’s in CDMA/EVDO and Scrambling Codes (SC) in UMTS.

Below are sector plots for 2G/3G for each carrier for Superdome Level 1. These plots show the strongest serving sectors for each of the Operators for their 2G/3G coverage.

The legend shows the sector number and the number of measurement in the map with that sector. If we eliminate the sectors with 20 or fewer measurements (these could be coming from upper levels or from the outside) then we arrive at the following number of serving sectors for 2G/3G for level one of the Superdome for each carrier. Note that this is proba-bly not the exact amount but is an approximate estimate.

AT&T: 12 SectorsVerizon: 4 SectorsSprint: 9 SectorsT-Mobile: 4 Sectors



FIGURE 7. AT&T 850 Strongest Sector Plot



FIGURE 8. Verizon 850 Strongest Sector Plot



FIGURE 9. Sprint Strongest Sector Plot



FIGURE 10. T-Mobile Strongest Sector Plot

There is a connection capacity limit for each sector so the carrier with more serving sec-tors will be capable of a higher number of connections. However, as in all engineering design there are trade-offs. In this case, serving up more sectors also creates more opportu-nity for interference (spill over) from other sectors and is more challenging to the design engineer to clearly separate the sectors to minimize areas where there is no dominant sec-tor. In addition, there is more back room equipment cost for more sectors.

At some point adding more sectors becomes non-beneficial in terms of quality/capacity improvement. The structure has everything to do with these sector layout decisions. Obvi-ously, in this case each carrier decided on different approaches in terms of sector coverage.



7.3 Throughput

Data throughput tests were conducted for AT&T (LTE), Verizon (LTE) and T-Mobile (HSPA). We chose not to conduct throughput tests for Sprint as we assumed since this was 3G EVDO technology which typically maximizes out at or slightly above 1 Mbps data rates.

Table 4: Data Testing Results for Levels 1 and 6 combined

The throughput speeds were in the range that we expected. Note that our testing was con-ducted two days before the Superdome with only a few hundred people in the building. The performance would most likely be different on game day with 80,000 people inside. However, in terms of comparing networks things should not change too much relative to each other.

The signal levels are similar to what we saw for 3G where AT&T chose to have more sec-tors and lower signal levels. It seems Verizon increased the number of sectors (nearly dou-ble) for their 4G network most likely due to anticipated increase in the number of data connections required. It is also interesting to see that AT&T has more sectors on level 6 than on level 1. This could be because level 6 is physically larger than level 1 and/or per-haps because level 6 holds more people than level 1 and they are anticipating having to support more connections.

It is also interesting to note that the AT&T system had a SINR considerably higher than Verizon. It seems their design of more sectors and lower power creates a much lower noise level and thus a higher signal to noise ratio. This was an average though so there might be a lower SINR in sector boundary areas. More analysis would need to be done to determine this.

T-Mobile’s data rates are much lower than AT&T or Verizon due to limited spectrum and limits of the HSPA technology. They also have less sectors so the number of connections on game day might be limited. Since Sprint is at 3G data rates is assumed their rates will be less than half the T-Mobile rates.



The following plots show the serving PCI’s for AT&T and Verizon for level 1.

Table 5: Verizon 700MHz LTE Strongest Serving PCIs

Table 6: AT&T 700MHz LTE Strongest Serving PCIs



7.4 Voice Quality

Voice calls were conducted on all four Carriers. Thirty second calls were repeated with a 5 second down time in between calls. The intent was not to test dropped or blocked calls because we knew the voice stats would be good since the testing was not being done on game day.

Rather, our intent was to look at the over levels and make a determination of voice quality. Game day information would be valuable in determining call block and drop stats. Typi-cally, the Carriers will monitor the stats before and during the game and make any changes they can to maintain acceptable levels of quality and capacity.

There are subtle differences in signal levels and error rates but nothing out of the ordinary. It is interesting that Sprint is the only Carrier that has a stronger signal on level 6 than the others. Sprint is on its own DAS and the others are on the neutral host.

Although the FER/BLER error rates are all pretty good it is interesting that T-Mobile has a much lower rate than the others. They were able to attain these lower rates at a similar sig-nal level and Ec/Io level. They are in the AWS band which perhaps is less affected by external interference so this may have something to do with it.

8.0 Conclusion

The goal was to perform a benchmarking study of voice and data performance of the four major Carriers in the Superdome and to provide information on the different design approaches for each Carrier.

The ZK-MPS/B unit performed all the voice and data call measurements and scanning measurements across 4 frequency bands and for LTE, CDMA/EVDO, UMTS and HSPA technologies. ZK’s KPI Mapper and Benchmarker software was used for analyzing the log data.

Verizon and AT&T have spectrum at 700MHz, 850MHz and 1900MHz. T-Mobile has 2 of the AWS bands and Sprint has one of the PCS bands. Having more spectrum allows you to offer better service and capacity.



In terms of data throughput Verizon and AT&T far out deliver the others due to LTE tech-nology and more spectral capacity. T-Mobile was next in data rates and Sprint was assumed to be lower than T-Mobile.

All Carriers did equally well in terms of voice service and it is assumed they would offer equivalent voice service on game day. This does depend on usage and capacity though.

Verizon chose a design approach that transmitted at a higher power and used fewer sectors and AT&T chose to deliver lower power and provide more sectors. Results were about the same for each although AT&T was able to obtain a higher average SINR. Verizon’s approach requires less DAS hardware and is somewhat simpler to optimize but since there is a limit to the number of connections per sector their limit is less than AT&T. This may or may not be a factor on game day though.

AT&T chose a design that has more sectors at a lower power. This costs extra in terms of hardware and it a bit more complicated of a design but they have a higher connection capacity and a higher SINR which translates into a better quality signal. This was an aver-gae SINR over the entire walk which may or may not address sector boundary areas. Since AT&T has more sectors there is more potential for sector bleed over which may create pockets of low SINR. Further analysis would need to be done.

Finally, if you were going to the Superbowl this year and were planning to use your cell phone to send photos or videos or surf the web for player stats we recommend having AT&T or Verizon service. They have more spectrum and have deployed 4G technology. There is a good reason that these two companies command a large majority of the sub-scriber base, they invest dollars in equipment and testing to deliver superior performance to the end user. If you were just going to make phone calls and send/receive texts than most likely any of the Carriers’ service would suffice.


Superdome Benchmarking Report - ZK Serviceszkservices.net/files/Download/Superdome... · ZK...

Documents

Transcript of Superdome Benchmarking Report - ZK Serviceszkservices.net/files/Download/Superdome... · ZK...