DADDS DCS LRIT PrototypeLRIT Latency Results
Presented by
Microcom Design, Inc.
May 2014
Microcom Design, Inc. 2
Task Description
Authorized under a DCS Sustaining Engineering Contract. Develop a DADDS Process to create and disseminate
LRIT DCS Message files. Transfer files to LRIT system via SFTP. Develop text based and graphical tools to analyze
latency. Utilize Microcom LRIT Receiver to monitor data flow and
determine end-to-end latency.• DAMS-NT to DADDS to LRIT to Reception
Include configurable settings to determine file generation:
• Message Count
• Byte Count
• Time – timeout to minimize latency
Microcom Design, Inc. 3
DOMAST versus LRIT Comparison
DOMSAT streams DCS messages, while LRIT transfers files. For efficiency, multiple DCS messages must be collected into
files. DOMSAT stream “dedicated” to DCS, LRIT stream shares
multiple NOAA products; DCS, EMWIN, Imagery. DOMSAT uses Domestic Satellite, LRIT broadcast over
GOES. DOMSAT requires annual funding, LRIT has no user recurring
satellite usage costs. DOMSAT has limited coverage outside of CONUS, LRIT has
hemispherical coverage. DOSAT primarily used for other data, GOES/LRIT dedicated to
environmental usage. DOMSAT frequency in Ku band, while LRIT in L band.
DOMSAT highly susceptible to weather fading, LRIT far more robust.
Microcom Design, Inc. 4
LRIT Operational Summary 1
Downlink Frequency: 1691 MHz (L Band) Information Data Rate:128 kbps (16 kBps) Forward Error Correction: Convolution and Block
Inner Code: Rate ½ Viterbi, Constraint Length 7 Outer Code: Reed Solomon (255,223) – 223 information
bytes and 32 check bytes; can correct up to 16 bytes in error
Data Files are Packetized and Framed Frame: 1024 Bytes
• 4-byte FSS, followed by four interleaved 255 byte RS blocks. • ~ 880 bytes of information in each frame.
Packets: Variable up to 8 kilobytes• Each Packet terminated in CRC-16.• Large Files require multiple packets.• First Packet contains LRIT Headers.
Microcom Design, Inc. 5
LRIT Operational Summary 2
Files can be sent on one of 63 Virtual Channels (VC) Each Frame begins with a 6-byte sequence that defines the VC
and includes a 3-byte Frame Counter. Frame Sequence Counters are unique to VC.
Virtual Channels Represent a Priority The lower the VC number, the higher the priority. Priority scheme allows smaller, high priority files to be
interspersed with larger, low priority files. EMWIN uses VC 00; highest priority.
LRIT is a Continuous Transmission A 64TH Virtual Channel (VC 63) is used for “Fill”, i.e. when there
is no data to send. Actual transmission rate is 292.7 ksps (kbps)
128 = (292.7 / 2) * (223/255)
Microcom Design, Inc. 6
LRIT DCS Initial Analysis – Frame Utilization 1
LRIT Packet Distribution - 9/20/2012
0
20
40
60
80
100
20:00 20:05 20:10 20:15 20:20 20:25 20:30 20:35 20:40 20:45 20:50 20:55 21:00
Time, Local
Pe
rce
nt
Uti
liza
tio
n
DCS Imagery EMWIN Fill
Microcom Design, Inc. 7
LRIT Initial Analysis – Frame Utilization 2
Initial analysis showed that the LRIT stream was dominated by long periods of either DCS, Imagery, or Fill, but …
EMWIN maintained steady flow of data.
LRIT Packet Distribution - 9/20/2012
0
20
40
60
80
100
20:15 20:16 20:17 20:18 20:19 20:20
Time, Local
Per
cen
t U
tiliz
atio
n
DCS Imagery EMWIN Fill
Microcom Design, Inc. 8
LRIT Initial Analysis – Product Utilization
Initial Frame Analysis Showed: EMWIN: ~ 7.3 kbps ~5.7% DCS: ~15.3 kbps ~12.0% Imagery: ~41.2 kbps ~32.2% Fill: ~64.2 kbps ~50.1%
Initially: EMWIN used VC00 DCS used VC18 (High), VC38 (Medium), VC53 (Low) Imagery used VCs 13, 23, 33, 41, and 43
More Recently: EMWIN continues to use VC00 DCS was using 1, 48, and 53 Imagery shared VC53 with Low Priority DCS Files
Microcom Design, Inc. 9
LRIT DCS File Summary
High Priority Files: Contain one of two platforms (or both) that transmit on six
minute intervals at the top of the minute. Being used by NOS and NWS to facilitate their latency testing. Files sent in 1 Packet/Frame.
Medium Priority Files: Consist of approximately 500 platforms. All platforms appear to transmit on DCS channels 40-49. Files typically require 1 Packet sent over several Frames.
Low Priority Files: Everything not in Low or Medium priority files. Files are quite large (~200 kilobytes, ~1000 messages). Files typically sent about every 2 minutes.
Microcom Design, Inc. 10
LRIT Baseline Latency
Typical composite latency ~100 seconds with regular peaks over 300 seconds; some as high as 600 seconds.
Peaks believed to be primarily due to DCS Low Priority files sharing same VC as imagery files.
Microcom Design, Inc. 11
Prototype Target Goals
Create compatible DCS LRIT files from DADDS.
Determine if the LRIT system can handle smaller more frequent files.
Send all DCS messages as High priority. Accurately measure overall LRIT DCS
latency. Breakdown latency into constituent
components. See if DADDS can provide improvement
over existing dissemination path.
Microcom Design, Inc. 12
Initial Live Test Run
Test was run on April 16, 2014 from approximately 14:24:30 to 14:29:30 UTC.
All data sent as High Priority. DCS data collected and sourced from NSOF DADDS. Messages collected into files of approximately 8 kilobytes
each. Total of 58 files generated and transmitted. Approximately 2450 DCS messages were transmitted and
received. Test run was performed while WCDA DCS LRIT was also
flowing. No anomalies were detected and test ran as expected. After confirming data was flowing and no issues were
detected, performance data was collected from approximately 14:27:00 to 14:29:30 UTC.
Microcom Design, Inc. 13
1ST Test – Frame Utilization Graph
LRIT DADDS Test - Frame Utilization
0
10
20
30
40
50
60
70
80
90
100
14:2
7:56
14:2
8:01
14:2
8:07
14:2
8:12
14:2
8:18
14:2
8:24
14:2
8:29
14:2
8:35
14:2
8:40
14:2
8:46
14:2
8:51
14:2
8:57
14:2
9:03
14:2
9:08
14:2
9:14
14:2
9:19
14:2
9:25
14:2
9:31
14:2
9:36
14:2
9:42
Time
Fra
mes
Per
Sam
ple
(E
very
100
Fra
mes
)
DADDS DCS EMWIN Other (WCDA DCS & Imagery) Fill
Microcom Design, Inc. 14
1ST Test – Frame Utilization Notes
LRIT Frames counted for DADDS DCS, EMWIN, Fill, and Other. Other includes both Imagery and the DCS Data coming from WCDA.
A data sample set occurs every 100 LRIT Frames. NSOF DADDS DCS Data accounted for approximately 10% of the
Total Frames. Average: 9.8 Frames per 100 Minimum: 0 Maximum: 28
EMWIN accounted for approximately 6% of the Total Frames. Insertion of a second DCS stream did not create any major issues.
Only impact was a slight reduction in the transmission rate of the
“Other” frames.
This short duration test showed minimal Fill packets (~7%). Longer term testing has shown that the LRIT typically only runs at 65-
70% utilization (30-35% Fill).
Microcom Design, Inc. 15
1ST Test – Overall Results
Data points generated and accumulated every 100 messages. Computer generating graph had clock running 5-6 seconds fast.
Microcom Design, Inc. 16
1ST Test – Overall Notes
Latency calculated from the time the message was received to time it was ingested by test computer at Microcom.
Average DCS message latency for this test run calculated on every 100 messages. Note: After the test run was complete, the test
computer’s clock was checked and was found to be 5-6 seconds fast.
Accounting for computer’s clock error, average latency for the initial test run was actually on the order of 45-50 seconds.
Microcom Design, Inc. 17
1ST Test – LRIT File Latency Histogram
LRIT File Distribution Latency
0
2
4
6
8
10
12
14
15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
File Generation to Received in LRIT Stream, Seconds
File
Co
un
ts
Microcom Design, Inc. 18
1ST Test – LRIT Latency Notes
Latency calculated in seconds from file generation time to reception by LRIT Receiver. Average: 25 seconds Minimum: 19 seconds Maximum: 35 seconds
This latency encompasses the transfer time from DADDS-to-LRIT, the LRIT processing time, and the transfer time from the LRIT processor to the LRIT uplink transmitter. Note: The times are approximately 3-5 seconds higher than actual. Microcom utilized the first DCS message in the file to generate the
filename timestamp. A better indication would be provided by using the time the last
message was added to the file.
Average file latency is predicted to be more on the order of 20 seconds or so.
Average Latency remained relatively consistent over initial run.
Microcom Design, Inc. 19
1ST Test – DADDS LRIT File Generation
DADDS LRIT File Generation Latency
0
5
10
15
20
25
11 12 13 14 15 16 17 18 19 20
First File Msg End Time To File Timestamp, Seconds
File
Co
un
ts
Microcom Design, Inc. 20
1ST Test – DADDS LRIT Latency Notes
Latency calculated in seconds from end time of first message in file to file timestamp: Average: 16.4 seconds Minimum: 13.5 seconds Maximum: 18.6 seconds
DADDS file latency on the order of 16 seconds. Higher than expected, but optimizing DADDS process was
not a primary goal of this task. Microcom believes DADDS latency can be easily reduced to
5-10 seconds. Latency was consistent over initial run.
Microcom Design, Inc. 21
Second Live Test Run
Test was run on April 23, 2014 from approximately 13:27:30 to 16:05:00 UTC.
All data sent as High Priority. DCS data collected and sourced from NSOF DADDS. Messages collected into files of approximately 8 kilobytes
each. A total of 2,035 files were generated, transmitted, and
received. Over 76,000 DCS messages were transmitted and received
(~29,000 messages per hour). Test run was performed without WCDA DCS LRIT data
flowing. No operational issues were detected; independent
confirmation of reception from NOS and NWS. Performance data was collected from almost the entire run
UTC.
Microcom Design, Inc. 22
2ND Test – Frame Utilization Graph
LRIT DADDS Test - Frame Utilization - Entire Test Run
0
10
20
30
40
50
60
70
80
90
100
13
:30
:39
13
:35
:17
13
:39
:56
13
:44
:35
13
:49
:14
13
:53
:52
13
:58
:31
14
:03
:10
14
:07
:49
14
:12
:27
14
:17
:06
14
:21
:45
14
:26
:23
14
:31
:02
14
:35
:41
14
:40
:20
14
:44
:58
14
:49
:37
14
:54
:16
14
:58
:55
15
:03
:33
15
:08
:12
15
:12
:51
15
:17
:30
15
:22
:08
15
:26
:47
15
:31
:26
15
:36
:05
15
:40
:44
15
:45
:22
15
:50
:01
15
:54
:40
15
:59
:18
16
:03
:57
Time
Fra
me
s P
er
Sa
mp
le (
Ev
ery
10
0 F
ram
es
)
DCS EMWIN Other (Imagery) Fill
Microcom Design, Inc. 23
2ND Test – Frame Utilization Notes
LRIT Frames counted for DCS, EMWIN, Fill, and Other; Other primarily includes Imagery.
A data sample set occurs every 100 LRIT Frames. DCS Data accounted for 10.24% of the Total Frames.
Average: 10.24 Frames per 100 Minimum: 0 Maximum: 45 Stnd Dev: 7.6
EMWIN accounted for 5.73% of the Total Frames. Other (Imagery) accounted for 57.14% of the Total Frames. Fill accounted for 26.89% of the Total Frames. EMWIN and DCS packets flowed at relatively consistent
rate, while Imagery & Fill dominated alternately.
Microcom Design, Inc. 24
2ND Test – Frame Utilization Hour 14
LRIT DADDS Test - Frame Utilization - Hour 14 UTC
0
10
20
30
40
50
60
70
80
90
100
14
:00
:00
14
:01
:52
14
:03
:43
14
:05
:35
14
:07
:26
14
:09
:18
14
:11
:09
14
:13
:01
14
:14
:52
14
:16
:44
14
:18
:35
14
:20
:27
14
:22
:18
14
:24
:10
14
:26
:01
14
:27
:53
14
:29
:44
14
:31
:36
14
:33
:27
14
:35
:19
14
:37
:10
14
:39
:02
14
:40
:53
14
:42
:45
14
:44
:36
14
:46
:28
14
:48
:19
14
:50
:11
14
:52
:02
14
:53
:54
14
:55
:45
14
:57
:37
14
:59
:28
Time
Fra
me
s P
er
Sa
mp
le (
Ev
ery
10
0 F
ram
es
)
DCS EMWIN Other (Imagery) Fill
Microcom Design, Inc. 25
2ND Test – Overall Results
Data points generated and accumulated every 100 messages.
Microcom Design, Inc. 26
2ND Test – Overall Notes
Latency calculated from the time the message was received to time it was ingested by test computer at Microcom.
Average end-to-end latency for the bulk of this run was on the order of 40 seconds.
From ~15:05 to ~15:25 the average latency gradually increased to a peak of over 65 seconds and then came back down to ~40 seconds.
Standard Deviation of Latency consistently remained under 5 seconds.
Microcom Design, Inc. 27
2ND Test – Existing Dissemination Comparison
Typical composite latency 60-80 seconds with peaks over 200 seconds. Improvement from initial baseline due to move to using VC 01 for all DCS
files.
Microcom Design, Inc. 28
2ND Test – LRIT File Latency Histogram
LRIT File Distribution Latency - Second Run
0
50
100
150
200
2501
011 1
21
31
41
51
61
71
81
92
02
12
22
32
42
52
62
72
82
93
03
13
23
33
43
53
63
73
83
94
04
14
24
34
44
5
File Generation to Received in LRIT Stream, Seconds
File
Co
un
ts
Microcom Design, Inc. 29
2ND Test – LRIT Latency Notes
Latency calculated in seconds from file generation time to reception by LRIT Receiver. Average: 22.9 seconds Minimum: 13 seconds Maximum: 42 seconds Stnd Dev: 4.2 seconds
Latency encompassed the transfer time from DADDS-to-LRIT, the LRIT processing time, and the transfer time from the LRIT processor to the LRIT uplink transmitter. Note: The times are approximate 3-5 seconds higher than actual. Microcom utilized the first DCS message in the file to generate the
filename timestamp. A better indication would be provided by using the time the last
message was added to the file. Average file latency is predicted to be more on the order of 20
seconds or so. Average Latency remained relatively consistent over entire run.
Microcom Design, Inc. 30
2ND Test – LRIT File Latency Over Run
LRIT File Latency Time Plot
0
5
10
15
20
25
30
35
40
45
13
:27
:52
13
:32
:47
13
:37
:17
13
:41
:39
13
:45
:59
13
:50
:56
13
:55
:19
13
:59
:47
14
:04
:00
14
:07
:37
14
:11
:15
14
:14
:47
14
:18
:23
14
:22
:11
14
:26
:11
14
:30
:43
14
:34
:53
14
:39
:19
14
:43
:39
14
:48
:00
14
:52
:50
14
:57
:22
15
:01
:53
15
:05
:48
15
:09
:31
15
:13
:12
15
:16
:39
15
:20
:12
15
:23
:50
15
:28
:01
15
:32
:13
15
:36
:37
15
:40
:53
15
:45
:07
15
:49
:53
15
:52
:39
15
:54
:54
15
:57
:12
15
:59
:18
16
:01
:37
16
:03
:36
Time, UTC
La
ten
cy
, Se
co
nd
s
Raw 10 Point Average
Microcom Design, Inc. 31
2ND Test – DADDS File Latency
DADDS LRIT File Generation Latency
0
5
10
15
20
25
30
35
40
45
13:2
7:27
13:3
3:57
13:4
0:38
13:4
7:12
13:5
4:13
14:0
1:10
14:0
6:48
14:1
2:04
14:1
7:22
14:2
3:00
14:2
9:32
14:3
6:07
14:4
2:34
14:4
9:22
14:5
6:15
15:0
2:53
15:0
8:29
15:1
3:59
15:1
9:17
15:2
4:43
15:3
1:11
15:3
7:44
15:4
4:03
15:5
0:56
15:5
4:14
15:5
7:31
16:0
1:01
16:0
3:58
Time, UTC
Lat
ency
, S
eco
nd
s
Microcom Design, Inc. 32
2ND Test – DADDS File Latency Notes Latency calculated from end timestamp of first DCS message in file to the
file generation time. Average: 18.9 seconds Minimum: 13 seconds Maximum: 42 seconds Stnd Dev: 6.4 seconds
Baseline average appears to be on the order of 16-17 seconds. Discounting three spiked up sections, latency remained relatively consistent
over entire run. Excessive rise in processing delay accounts for increased overall latency
shown in graph, i.e the point centered about 15:15 UTC where the overall latency increased to over 65 seconds.
Smaller spike at 14:10 in above graph also correlates to similar spike in overall graph.
At this time Microcom has no specific explanation for the increases in DADDS processing time. Most likely cause is server performance due to some other function that
required high CPU resources. Additional investigation will be required to determine exact cause. Should be readily addressable.
Microcom Design, Inc. 33
Target Goals Acheived
Testing was highly successful and achieved target goals: DADDS can flow DCS data to LRIT system. DADDS DCS data files were properly formatted:
• Message data was correctly received by Microcom’s LRIT Receiver, which also receives WCDA sourced DCS data.
• Independent confirmation provided by NOS (Mark Bushnell, Warren Krug) and NWS (Brian Jackson) during and post test run.
LRIT system has no issues in processing smaller more frequent files.
LRIT throughput not significantly affected by sending all DCS messages as High Priority.
Microcom Design, Inc. 34
Testing Conclusions
Testing provided solid baseline information on overall latency and distribution of latency in various systems. Overall typical latency is on the order of 35-45 seconds. Latency remains consistent over extended periods of
time. LRIT processing and distribution latency on the order of
20-25 seconds. Typical DADDS file generation latency is on the order of
15-18 seconds. The DADDS-to-LRIT prototype and subsequent analysis
by Microcom has clearly demonstrated capability to accurately measure and to pinpoint problem areas in the overall latency.
Microcom Design, Inc. 35
Possible Next Steps
The unusual and unexpected areas where latency above baseline will require additional investigation to determine the cause. However, Microcom is confident this can be mitigated.
Microcom is also confident that the DADDS file generation can be optimized and reduce the typical latency down closer to 10 seconds (or better).
Assuming an LRIT system latency of 20-25 seconds and a 5-10 second DADDS latency can be achieved, than it should be possible to achieve an average latency of 30 seconds.
Make suggested improvements and perform additional testing.
END OF PRESENTATION “THANK YOU” FOR YOUR ATTENTION
Top Related