International Comparisons of Network Time Protocol Servers

Michael A. Lombardi, NIST, USA, lombardi@nist.govJudah Levine, NIST, USA

J. Mauricio Lopez and Francisco Jiménez , CENAM, MexicoJohn Bernard and Marina Gertsvolf, NRC, Canada

Harold Sanchez and Oscar G. Fallas, ICE, Costa RicaLiz Catherine Hernández Forero, INM, Colombia

Ricardo José de Carvalho and Mario N. Fittipaldi, ONRJ, BrazilRaul F. Solis, CENAMEP, Panama

Franklin Espejo, IBMETRO, Bolivia

PTTI 2014, 12/03/2014, Boston

• The use of NTP servers at national timing labs in the SIM region

• Description of NTP measurement system

• Direct comparisons of NTP servers to UTC(NIST)

• Common-view comparisons of NTP servers

• Causes and effects of network asymmetry

• Summary

Outline

The Interamerican Metrology System (SIM) region covers North, Central, and South America, and the

Caribbean Islands

Introduction: The Use of NTP Servers in the SIM Region The Network Time Protocol (NTP) synchronizes clocks via the

public Internet.

Many national timing laboratories now utilize NTP as the dominant or only method for distributing the national time.

NTP is usually the first time service offered by a new national timing lab because The Internet has become ubiquitous for all types of

communication, including time transfer NTP servers are much more useful than telephone time

systems and much cheaper to acquire and maintain than radio time systems

All official providers of the national time should verify the accuracy of the time they disseminate via NTP. This requirement led to the international comparison system described in this presentation.

Current Participants in SIM NTP Comparisons

Laboratory Country Synchronization Source

CENAM Mexico 1 pps from local time scale

CENAMEP Panama 1 pps from local time scale

IBMETRO Bolivia 1 pps from oscillator disciplined to SIM Time Scale

ICE Costa Rica 1 pps from local time scale

INM Colombia 1 pps from local time scale

NIST United States 1 pps from local time scale or NIST Automated Computer Time Service

NRC Canada 1 pps from local time scale

ONRJ Brazil 1 pps from local time scale

Description of NTP Measurement System

NTP Measurement System The system went on-line on May 31, 2014 at NIST in Boulder, Colorado and has

run continuously since that date.

Can currently measure up to 20 servers and can be expanded as necessary Eight countries are currently participating in NTP comparisons, with each country

providing access to either one or two servers.

The reference for the measurements is a clock board synchronized with a 1 pulse per second (pps) signal from UTC(NIST)

The clock board has 100 ns resolution The cable from UTC(NIST) to the clock board has a compensated delay of 480 ns

The system sends an NTP request to every server every 10 s. Measurements are recorded in 10-minute segments using two different methods and then uploaded to the SIM web site. The two methods are:

AVG method – the system stores the average time difference between the server clock and UTC(NIST) based on 60 requests

MIN method – the system only stores the time difference when the round trip delay between the server and NIST was at a minimum (59 of the 60 NTP requests are discarded)

NTP Measurements The time difference between the server clock and UTC(NIST), or TDNIST, is

where

T1 is the time of the measurement system request, obtained from UTC(NIST)

T2 is the time when request was received by server, obtained from server clock

T3 is the time when the server sends is response, obtained from server clock

T4 is the arrival time in Boulder of the NTP packet, obtained from UTC(NIST)

The round trip network delay between the server and Boulder, or RTDelay, is determined using the same variables as

The “divide by 2” in the first equation assumes that the delay from the server to the client is half of the round trip delay. In practice, however, the network path always has some asymmetry that adds uncertainty to the measurement.

All data in this presentation, and live NTP measurements in progress, can be viewed at:

tf.nist.gov/sim

Direct Comparisons to UTC(NIST)

Direct Comparisons to UTC(NIST)

All server clocks are synchronized to UTC, at least to within ±1 ms

Most of the measured time error is not due to the server clocks, but instead is due to network asymmetry

The effects of network asymmetry are nearly always reduced by the MIN method

The effects of network asymmetry are not necessarily a function of the distance travelled by the NTP packets. For example, servers located in Brazil (~9500 km from Boulder), can be measured with less uncertainty than other servers located much closer to Boulder.

100-day comparison (ending 10/27/14) of NTP Servers to UTC(NIST)

ONRJ-1 – UTC(NIST) over 100-day period ending 10/27/14

Common-view NTP Measurements

Common-View Comparisons between Servers

The common-view method is implemented by directly comparing two server clocks to UTC(NIST) and then subtracting the results of the two measurements. This technique is a convenient way to estimate the time difference between two server clocks.

The apparent common-view time differences between two server clocks are dominated by network asymmetries, which usually do not cancel when the measurements are subtracted. 

This is especially true in the case of servers located in different countries that are being measured via very different network paths.

When two servers that are synchronized to the same source and connected to the same network are compared to each other, then most of the network asymmetry does cancel.

For example, the average time difference between two servers located at ONRJ in Brazil can be determined to within tens of microseconds by implementing the common-view method from Boulder.

Common-view NTP Measurements

Common-view comparison of ONRJ (Brazil) servers from Boulder, Colorado for 100-day period ending 10/27/2014

Common-View Measurement Uncertainty Assume that the “worst case” uncertainty of a direct server clock comparison equals 1/2 the

round trip delay (meaning that all of the delay is in one direction), and that the paths between NIST and each server involved in the common-view comparison are uncorrelated.

Then, the “worst case” uncertainty of the common-view comparison, Ucv, can be estimated as

where

RT1 is the round trip delay to server 1 and RT2 is the round trip delay to server 2

This means that the time error has be large before ruling out network asymmetry as the cause when making international comparisons with large round trip delays. Some general rules:

Time differences larger than about 200 ms are too large to be caused by network asymmetry even in a region as large as the SIM region, and are likely due to an unsynchronized server clock.

Time differences ranging from about 50 to 200 ms should be investigated to determine if the error is due to an unsynchronized server clock or to current network conditions. The SIM grid displays time differences larger than 50 ms in red. This arbitrary threshold is usually sensitive enough to call attention to potential problems.

Causes and Effects of Network Asymmetry

Inconsistent routing of the NTP packets

Based on link availability and routing tables, the NTP packets may be sent using different hubs at different times and when travelling in opposite directions.

Thus, the delay from server-to-client can differ significantly from the delay from client-to-server.

Network operators control the routing tables and end users are not allowed to request specific routes for applications such as NTP.

Network congestion

Occurs when the amount of traffic carried by the network exceeds the amount of available bandwidth (bandwidth saturation).

Periods when network congestion is worse than usual are usually identified by an increase in the round trip delay, because packets are being buffered or rerouted.

Sources of Network Asymmetry

Round trip delays between servers under test and NIST during September-October 2014 (61 days)

Server Location RTDelay (ms) 

Average Minimum Avg – Min

NIST-B Colorado, USA 3.2 2.1 1.1

NIST-G Maryland, USA 52.1 51.1 1.0

NRC Canada 47.9 46.9 1.0

CHU Canada 77.2 63.9 13.3

ICE-1 Costa Rica 133.5 126.1 7.4

ICE-2 Costa Rica 132.0 122.4 9.6

INM-1 Colombia 162.5 150.2 12.3

ONRJ-1 Brazil 188.7 187.4 1.3

ONRJ-2 Brazil 189.8 188.3 1.5

CENAM Mexico 106.0 72.7 33.3

CNM-2 Mexico 107.3 74.7 32.6

IBMET Bolivia 410.0 228.5 181.5

CNMEP Panama 114.8 98.5 16.3

Inconsistent Routing of NTP Packets

The number of network “hops” and the type of hops, can differ in both directions and at different times of day. They can also change as hardware and software changes are made to the network. The network routing may not change very often, but it is always subject to change.

The table shows how the round trip delay between NIST and NRC was different in July (81 ms) and October (53 ms). The number of hops was reduced from 13 to 10 and the shaded area marks the hops that were different on the two occasions.

July 2014 October 2014

hop Min RTDelay (ms) node IP hop Min RTDelay

(ms) node IP

1 <1 132.246.52.1 1 <1 132.246.52.1

2 <1 132.246.0.53 2 <1 132.246.0.53

3 <1 132.246.0.25 3 1 132.246.0.25

4 5 206.130.255.13 4 5 206.130.255.13

5 36 205.189.32.180 5 16 205.189.32.117

6 44 205.189.32.178

7 53 205.189.32.182

8 64 205.189.32.174

9 88 207.231.245.129 6 34 198.71.45.2

10 92 137.164.25.49 7 31 198.71.45.8

11 80 137.164.26.2 8 54 192.43.217.223

12 81 128.117.243.11 9 54 128.117.243.11

13 81 140.172.2.26 10 53 140.172.2.26

Time difference between NRC server in Canada and UTC(NIST) for September-October 2014 (61 days)

Round trip delay between NRC (Canada) and Boulder, Colorado for September-October 2014 (61 days)

Network Congestion – Case 1 (Mexico)

Internet access on the campus of the Centro Nacional de Metrología (CENAM) is obtained by a wireless microwave link from the nearby city of Querétaro.

Network traffic exceeds the available bandwidth during the normal working hours at CENAM.

This leads to excessive buffering of the NTP packets, causing both the network delays and the uncertainty of the time received by clients to substantially increase.

The time uncertainty is much lower on the weekdays than on the weekends.

Time difference between CENAM server in Mexico and UTC(NIST) for 12-day period ending 10/17/2014

Round trip delay between CENAM (Mexico) and Boulder, Colorado for 12-day period ending 10/17/2014

Network Congestion – Case 2 (Bolivia)

The most extreme example of network congestion recorded by the SIM system.

The difference between the round trip delay, as estimated with the AVG and MIN methods, was 181.5 ms during September/October 2014, much larger than the differences recorded for the other servers.

The IBMETRO server was connected to an asymmetric digital subscriber line (ADSL) provided by a local telecommunications provider.

ADSL is inherently asymmetric because its downstream rate is faster than its upstream rate, thus it is not well suited for NTP time transfer. The ADSL data transfer rate in Bolivia was not particularly fast, limited to 2.5 Mb/s downstream and 1 Mb/s upstream, so the bandwidth could easily be saturated.

Time difference between IBMETRO server in Bolivia and UTC(NIST) for 12-day period ending 9/26/2014

Round trip delay between IBMETRO (Bolivia) and Boulder, Colorado for 12-day period ending 9/26/2014

Summary Due to its cost effectiveness and the ubiquitous nature of the Internet, NTP has

become the dominant or sole method for distributing the national time in many SIM nations.

We have developed a measurement system that verifies the accuracy of NTP servers that distribute the official time of SIM nations. The system makes the measurement results instantly accessible to all users.

In addition to serving as a time verification tool, the system allows us to study the delay asymmetry of the public Internet.

Future work could include: Expanding the system to include more servers, from both within and outside of the

SIM region. Installing NTP client measurement system at multiple SIM laboratories. With a

periodic data exchange, this could allow the unambiguous measurement of server clocks by fully compensating for network asymmetries.