Data Storage Considerations for the Tactical Field Collection of Digital Imagery - 2010
-
Upload
robert-watson -
Category
Documents
-
view
104 -
download
1
Transcript of Data Storage Considerations for the Tactical Field Collection of Digital Imagery - 2010
Research Paper
Data Storage Considerations for the Tactical Field Collection of Digital Imagery
INTL625 Imagery Intelligence
20 April 2010
Robert L. Watson
2
Table of Contents
Introduction ................................................................................................................................................. 3
Chapter 1 Introduction to Storage Media ................................................................................................ 5
Magnetic Tape ........................................................................................................................................... 6
Optical Disks .............................................................................................................................................. 7
Magnetic Hard Disks ............................................................................................................................... 11
Solid State Drives .................................................................................................................................... 12
Redundant Array of Inexpensive/Independent Disks (RAID) ................................................................. 17
Chapter 2 Introduction to Video Compression ...................................................................................... 25
Video Compression Standards ................................................................................................................. 28
Metadata ................................................................................................................................................... 30
Multimedia Players and Container Formats ............................................................................................. 34
Chapter 3 Recommendations ................................................................................................................... 37
Hardware Recommendations ................................................................................................................... 37
Software Recommendations .................................................................................................................... 39
Video Retention and Deletion Schemes ................................................................................................... 41
The Future ................................................................................................................................................ 44
Chapter 4 Conclusion ............................................................................................................................... 46
Bibliography .............................................................................................................................................. 53
3
INTRODUCTION
As a result of the Global War on Terror, the need for timely intelligence, both on a global
as well as local scale has facilitated the need for more sophisticated imagery intelligence
gathering systems that provide useful information from satellites, unmanned aerial vehicles and
stationary surveillance systems, to name but a few. This large data gathering effort is
unprecedented in history and presents new challenges to our country and its government.
A recent article in the New York Times stated: "Military deluged in intelligence from
drones: Remote-controlled planes produce about 24 years' worth of video in 2009."1 Other
recent news has described the U.S. Intelligence community’s inability to preempt the Christmas
Day 2009 terrorist attempt as a byproduct of "information overload."2
Out of necessity, the need for timely imagery intelligence that provides support to both
our military and government has created the requirement for large groups of skilled analysts to
evaluate the enormous amount of imagery data that is being collected to determine what is
significant to the intelligence effort. In light of this, large-scale digital storage systems are
necessary to provide data storage and retrieval.
However, not all of the imagery that is collected is useable or of significant intelligence
value. In some instances, the time or resources may not exist to have the imagery reviewed and
analyzed in a timely manner, so its value as an intelligence source may diminish.
1 Christopher Drew, “NYT: Military deluged in intelligence from drones - Remote-controlled planes produced
about 24 years’ worth of video in 2009,” New York Times, January 10, 2010, Sunday;
http://www.msnbc.msn.com/id/34798080/ns/world_news-he_new_york_times/ (accessed January 11, 2010).
2 Dan De Luce, “US spy agencies face information overload: experts,” AFP (via Yahoo News), January 7, 2010,
Thursday; http://news.yahoo.com/s/afp/20100107/pl_afp/ usattacksintelligence (accessed January 25, 2010).
4
The paper will explore the current storage technologies and methodologies in use, their
advantages and shortcomings, and provide some insights and possible solutions to this ever-
growing issue.
The research will focus primarily on solutions for real-time digital video gathering
systems, specifically stand-alone, tactical systems currently used in the field environment. Since
a system’s storage capacity and the means of increasing this capability may be limited due to
system design and mission constraints, newer storage technologies and methods will be
suggested to include specific metrics, both user-initiated and software-based, that could be used
as criteria in determining imagery retention and disposal.
5
CHAPTER 1
INTRODUCTION TO STORAGE MEDIA
Digital storage devices are the building blocks of storage in disk subsystems and
standalone server systems. Functioning in the microscopic realm, they perform the reading and
writing functions necessary for the storage of data on nonvolatile media. Digital image
processing of remote sensor data and its associated geographic information system (GIS) data
requires significant storage resources, which are also necessary for the collection of large
imagery databases.3
Table 1 illustrates some of the more common digital mass storage devices and their
average time to physical obsolescence – the point at which the media begins to deteriorate and
data loss can occur.
Table 1. LONGEVITY OF DIGITAL STORAGE MEDIA.
AVERAGE LONGEVITY OF DIGITAL STORAGE MEDIA
MEDIA TYPE AVERAGE OBSOLESCENCE
(IN YEARS) REMARKS
Optical Disk >100 Very cost efficient, presently best means of long-term, digital
storage
Magnetic Disk
(Hard Disk Drive)
20 Main system storage medium, but storage longevity normally
less than 20 years.
Magnetic tape 10-15 Cost efficient, but may become unreadable if not rewound and
properly stored in a cool, dry place.
Flash-Solid State
Disk*
10 Expensive, but provides faster seek time than magnetic hard
disks. Data retention for up to 10 years without power applied. Jensen, John R., Introductory Digital Image Processing: A remote Sensing Perspective, New Jersey: Prentice-Hall, 2005, 118-119.
*Tudor, Marius, “Are Flash Solid-State Disk Drives Ready for the Enterprise?,” Bit Micro Networks, Inc., 2009, 1 (accessed March
2, 2010).
The storage of remote sensor data can significantly impact system reliability and costs.
Substantial resources are required for even the most simple of systems. In light of this, mass
3 Marc Farley, Storage Networking Fundamentals: An Introduction to Storage Devices, Subsystems,
Applications, Management, and File Systems (Indiana: Cisco Press, 2007), 69.
6
storage media should permit rapid retrieval of required imagery data, provide longevity, and be
cost effective.4
MAGNETIC TAPE
Tapes are low cost and find frequent use as storage and backup media. They are linear
access devices as all data is written to or read from them sequentially.5 They come in several
types varying in size, storage capacity, density, length, thickness, number of tracks and reels, and
speed. They are composed of magnetic tape with single or dual reels contained in a plastic
enclosure.6
Data in older tape drive technology was written by multiple heads in a parallel track
across the entire tape with some drives using a helical scanning method which wrote the data
diagonally. A linear serpentine recording method is used for modern tape drives, which requires
more tracks and fewer tape drive heads. Data is written using the linear method except that data
is continually written with the head being adjusted and reversed once the end of the tape is
arrived at.13
Tape media is not without its drawbacks. Since access to data stored on magnetic tape is
linear or sequential, it is not considered efficient for random data access. The linear nature of
tape also makes recovery and back up operations time consuming. This is the reason why tape is
not considered for use as primary system storage, but is used mainly for offline data storage and
vaulting.7, 8
4 John R. Jensen, Introductory Digital Image Processing: A Remote Sensing Perspective, (New Jersey:
Prentice-Hall, 2005), 117.
5Shrivastava. Op. Cit. 269.
6Ibid. 33.
7Ibid.
7
Additionally, magnetic tapes slowly deteriorate over time developing surface cracks,
tears, and corrosion of the metal oxide coatings. Whether they are being used or stored for future
use, they should be maintained in environments having moderate temperatures and low
humidity.9 Improper maintenance (e.g., not rewinding the tape) and storage can cause magnetic
tape media to become unreadable within ten to fifteen years.10
OPTICAL DISKS
An optical disk is another recordable storage media which comes in a variety of types to
include compact disks (CDs), digital video/versatile disks (DVDs), magneto-optical (MO) disks,
and Blu-Ray disks (BDs).11 Optical disks comprise three broad categories, which determine their
usability. These categories are as follows12:
1. Read only optical disks, which are recorded when they are manufactured and cannot
be altered or erased. They include Compact Disk (CD), Compact Disk – Read Only
Memory (CD-ROM), Digital Versatile/Video Disk – Read Only Memory (DVD-
ROM), Digital Versatile/Video Disk – Video (DVD-Video), and Blu-Ray Disk (BD).
2. Write Once Read Many (WORM) optical disks can be recorded once and cannot be
erased. These include Compact Disk – Recordable (CD-R), Digital Versatile/Video
Disk – Recordable (DVD-R), and Blu-Ray Disk – Recordable (BD-R).
3. Rewriteable/Magneto optical disks, which can be written, erased and read from any
number of times. These include Compact Disk – Rewriteable (CD-RW), Digital
Versatile/Video Disk – Rewriteable, Blu-Ray, and magneto-optical (MO) disks.
Regardless of type, they are manufactured using similar technologies which incorporate a
thin polycarbonate disk which is impressed with microscopic bumps arranged in a continuous,
8Josh Judd, Principles of SAN Design: Design Build and Manage SANS, (Pennsylvania: Infinity Publishing,
2007), 18-19.
9Marc Farley, Storage Networking Fundamentals: An Introduction to Storage Devices, Subsystems,
Applications, Management, and File Systems (Indiana: Cisco Press, 2007), 83.
10
John R. Jensen, Introductory Digital Image Processing: A remote Sensing Perspective, (New Jersey:
Prentice-Hall, 2005), 118.
11Encyclopedia.com, "Optical Disk," The Columbia Encyclopedia, Sixth Edition, 2008,
http://www.encyclopedia.com/doc/1E1-optidisk.html (accessed March 4, 2010).
12Park, Oskar, “What is an Optical Disk?," Self SEO (Search Engine Optimization), September 10, 2006,
http://www.selfseo.com/story-18894.php (accessed March 11, 2010).
8
spiral track of data. Following this, the disk is coated with a thin, reflective layer of aluminum
(barium ferrite for magneto-optical disks) to cover the bumps. A thin acrylic layer is then
sprayed over the metallic coating for protection.13, 14 Common optical disk media and their
typical storage capacities are shown in Table 2.
Table 2. COMMON OPTICAL STORAGE MEDIA.
OPTICAL MEDIA TYPICAL STORAGE
CAPACITY
REMARKS
CD/CD-ROM 700Mb Recorded at time of manufacture and cannot be
altered/erased
CD-R 650Mb Write Once Read Many (WORM) media. Once
written, can only be read from.
CD-RW 650Mb Data can be written/erased/read from the disk
any number of times.
DVD/DVD-ROM 9.4Gb
(4.7Gb per side)
Recorded at time of manufacture and cannot be
altered/erased
DVD-R 9.4Gb
(4.7Gb per side)
Write Once Read Many (WORM) media. Once
written, can only be read from.
DVD-RW 9.4Gb
(4.7Gb per side)
Data can be written/erased/read from the disk
any number of times.
BD* 25Gb (single layer)
50Gb (double layer)
Recorded at time of manufacture and cannot be
altered/erased
BD-R* 25Gb (single layer)
50Gb (double layer)
Write Once Read Many (WORM) media. Once
written, can only be read from.
BD-RE* 25Gb (single layer)
50Gb (double layer)
Data can be written/erased/read from the disk
any number of times.
MO** 128Mb to 9.2Gb
depending on disk size
Data can be written/erased/read from the disk
any number of times.
Park, Oskar, “What is an Optical Disk?," Self SEO (Search Engine Optimization), September 10, 2006,
http://www.selfseo.com/story-18894.php (accessed March 11, 2010).
*Wikipedia contributors, "Blu-ray Disc recordable," Wikipedia, The Free Encyclopedia,
http://en.wikipedia.org/w/index.php?title=Blu-ray_Disc_recordable&oldid=346269932 (accessed March 12, 2010).
**Wikipedia contributors, "Magneto-optical drive," Wikipedia, The Free Encyclopedia,
http://en.wikipedia.org/w/index.php?title=Magneto-optical_drive&oldid=338693032 (accessed March 12, 2010).
Optical disk systems write the data to the media using a low-power laser that etches
binary bits into the reflective layer. In this technique, the bits are heated to 150 degrees
centigrade, which are then realigned when a magnetic field is applied, creating a binary bit one.
13Encyclopedia.com, "Optical Disk," The Columbia Encyclopedia, Sixth Edition, 2008,
http://www.encyclopedia.com/doc/1E1-optidisk.html (accessed March 4, 2010).
14Mediatechnics Systems, Inc, "How a CD is Made," Mediatechnics Systems, Inc., FAQ,
https://www.mediatechnics.com/cdfaqs.htm (accessed March 5, 2010).
9
The recording of new data requires that existing bits be reset to a binary bit zero.15 To read
information from the disk, polarized light from a low-power laser is rotated according to the
direction of the magnetic field and the original binary signal is reproduced.16
Together with an optical disk drive, optical disks function much the same way as hard
disk drives. Access time components (of the optical disk drives laser seeking a target track and
acquiring the target sector ) are similar to the seek and rotational latencies present in hard disk
drives.17
The use of optical disks over magnetic storage media offers many advantages to include
higher storage capacity, lower cost, high data stability, environmental tolerance, and long shelf
life.18 The long-term storage potential exceeds 100 years and provides the means of storing large
amounts of data on a relatively small media footprint.19
Optical disks have a constant linear velocity which provides a constant read/write
bandwidth when access is sequential. This makes the media appropriate for multimedia backup
and restore operations, which are generally sequential in nature.20 As such, rewriteable CD-RWs
15R. Jensen, Introductory Digital Image Processing: A Remote Sensing Perspective, (New Jersey: Prentice-
Hall, 2005), 118.
16Daintith, John, "Magneto-optic Storage," A Dictionary of Computing. 2004, Encyclopedia.com,
http://www.encyclopedia.com/doc/1O11-magnetoopticstorage.html (accessed March 5, 2010).
17
Huseyin Simitci, Storage Network Performance Analysis (Indiana: Wiley Publishing, Inc., 2003), 123.
18
Park, Oskar, “What is an Optical Disk?," Self SEO (Search Engine Optimization), September 10, 2006,
http://www.selfseo.com/story-18894.php (accessed March 11, 2010).
19
R. Jensen, Introductory Digital Image Processing: A Remote Sensing Perspective, (New Jersey:
Prentice-Hall, 2005), 118.
20Huseyin Simitci, Storage Network Performance Analysis (Indiana: Wiley Publishing, Inc., 2003).
10
and DVD-RWs have replaced tapes as the primary backup system in most remote sensing
laboratories.21
However, the recent advent of Blu-Ray, deemed as a future replacement for the DVD
format, offers even greater storage capacity. Having the same physical dimensions as CDs and
DVDs, a standard, double layer Blu-Ray disk can store up to 50GB of data. The Blu-Ray
standard is open-ended with theoretical storage limits left unspecified. Larger disk capacities of
100 and 200 GB are currently available.22
The use of optical disks is not without its drawbacks. In relation to hard disk drives, the
data access latencies are much higher in optical drives. The typical seek time for hard disk
drives is 10msec compared to 100-300msec for optical drives.23, 24
Constant improvements in hard disk drive technology (i.e., price, capacity, and speed)
make the decision to use optical disks less inviting. The development of DVDs did provide
some improvement in the storage capability, but applications are limited with many
organizations still relying on tapes and hard disk drives for business and scientific data storage.
However, the Blu-Ray disk may provide the impetus necessary to facilitate a greater move
towards the use of optical media for large database storage.25
21Jensen. Op Cit.
22
Wikipedia contributors, "Blu-ray Disc," Wikipedia, The Free Encyclopedia,
http://en.wikipedia.org/w/index.php?title=Blu-ray_Disc&oldid=349002833 (accessed March 12, 2010).
23
Huseyin Simitci, Storage Network Performance Analysis (Indiana: Wiley Publishing, Inc., 2003), 123.
24The Free On-line Dictionary of Computing, "Average Seek Time,"
http://encyclopedia2.thefreedictionary.com/average+seek+time (accessed March 12 2010).
25 Simitci. Op Cit.
11
MAGNETIC HARD DISKS
The Magnetic Hard Disk, also referred to as a Hard Disk Drive (HDD), is an
electromechanical device which controls the performance of the storage system environment,
providing the primary function of reading and writing the data that is stored on the media. They
are the primary storage medium used on computers for the storage and access of data and
software applications.26, 27
A magnetic hard disk consists of round, magnetic platters which encode digital data with
magnetically charged media. Each platter is arranged in cylinders/tracks with sectors on which
respective data is stored. The platters spin at a high rate of speed, up to 15,000 revolutions per
minute (RPM). Several platters, a read/write head, and a controller constitute the main
components of a HDD. The servo-controlled read/write head is attached to a rapidly moving arm
and is positioned over specific sectors to provide access to data. Since the HDD is a read/write
mechanism, data may be consistently written to and removed from the media.28, 29
HDDs provide quick and simultaneous access to arbitrary data locations and have large
capacities. Multiple disks may be configured into storage arrays providing increased capacity
and improved performance.30 In relation to other storage media like optical disks and tape,
26
Peter Alok Shrivastava and G. Somasundaram, eds., Information Storage and Management: Storing,
Managing, and Protecting Digital Information (Indiana: Wiley Publishing, Inc., 2009), 33.
27Marc Farley, Storage Networking Fundamentals: An Introduction to Storage Devices, Subsystems,
Applications, Management, and File Systems (Indiana: Cisco Press, 2007), 69.
28Dane Nelson and Sam Siewart, PhD, “Solid State Drive Applications in Storage and Embedded
Systems,” Intel Technology Journal, Volume 13, Issue 1, 2009, 30,
http://download.intel.com/design/flash/nand/extreme/ITJ_03_SSDs_for_Embedded.pdf (accessed March 2, 2010).
29Peter Alok Shrivastava and G. Somasundaram, eds., Information Storage and Management: Storing,
Managing, and Protecting Digital Information (Indiana: Wiley Publishing, Inc., 2009), 27.
30Ibid.
12
HDDs provide high data throughput and low latency data retrieval.31 However, as degradation
may occur in less than twenty years, HDDs have a comparatively short storage life. The
continual use of a HDD will reduce its life expectancy significantly.32
SOLID STATE DRIVES
Solid state drives (SSDs) use semiconductor flash memory chips for data storage and
retrieval. Unlike mechanical HDDs, they have no moving parts. All processes are operated via a
controller assembly that is also composed of semiconductor materials. SSDs have improved in
recent years and are used when high performance is required for mission-critical
applications.33,34
They use nonvolatile memory which supports persistent data storage and are constructed
in either single-level cell (SLC) or multi-level cell (MLC) configurations, which are used to store
data bits on respective memory cells. SLC storage is used in high performance memory and
stores one bit per cell. MLC memory stores multiple bits per cell providing slower data transfer
rates, but is cheaper to manufacture than SLC memory.35
SSDs using SLC technology and high reliability controllers imitate HDDs via a
traditional storage interface and provide ultra-fast read/write performance, high reliability, high
31Lawrence D. Bergman, and Vittorio Castelli, Image Databases: Search and Retrieval of Digital Imagery
(New York: John Wiley & Sons, Inc., 2002), 144.
32Jensen, John R. Introductory Digital Image Processing: A remote Sensing Perspective. (New Jersey:
Prentice-Hall, 2005), 118.
33Peter Alok Shrivastava and G. Somasundaram, eds., Information Storage and Management: Storing,
Managing, and Protecting Digital Information (Indiana: Wiley Publishing, Inc., 2009), 80.
34Dane Nelson and Sam Siewart, PhD., “Solid State Drive Applications in Storage and Embedded Systems,”
Intel Technology Journal, Volume 13, Issue 1, 2009, 30,
http://download.intel.com/design/flash/nand/extreme/ITJ_03_SSDs_for_Embedded.pdf
(accessed 03-02-10).
35
Peter Alok Shrivastava and G. Somasundaram, eds., Information Storage and Management: Storing,
Managing, and Protecting Digital Information (Indiana: Wiley Publishing, Inc., 2009), 80.
13
data integrity, and minimized power requirements. This makes them ideal for supporting
applications that require the rapid processing of large amounts of information such as real-time
data feeds.36
HHDs, on the other hand, use rotating media and servo-actuated read/write heads, which
introduce seek and rotation latencies. The device is also prone to mechanical failure. Their ever-
increasing capacity due to advances in the areal density of the recording surfaces and their low
cost are their main advantages, but fast random access limitations have always been an issue.37
As a rule of thumb, central processing units (CPUs) are able to process data much faster
than mechanical HDDs can transfer it. HDDs have been the mainstay of storage for decades
with a storage capacity that has increased over 200,000 times that of the first HDDs developed in
the 1950’s. During that same time period, decreases in price have also made the media more
affordable.38
However, HDD performance has not managed to keep up with processor development.
Within the last 30 years, processors have seen exponential increases in speed with only marginal
increases in the read/write response time of HHDs. This has resulted in a major break between
CPU access capabilities and those of HDDs.39
The performance differences between HDDs and SSDs concern the number of random
reads/writes per second that the device can perform and the time necessary to resume full
36 Ibid.
37
Dane Nelson and Sam Siewart, PhD., “Solid State Drive Applications in Storage and Embedded
Systems,” Intel Technology Journal, Volume 13, Issue 1, 2009,
http://download.intel.com/design/flash/nand/extreme/ITJ_03_SSDs_for_Embedded.pdf , 30 (accessed 03-02-10).
38Tom Coughlin, Neal Ekker, and Jim Handy, “Solid State Storage 101: An introduction to Solid State
Storage,” Storage Networking Industry Association (SNIA) Solid State Storage Initiative, January 2009,
http://www.snia.org/forums/sssi/knowledge/education/SSSI_Wht_Paper_Final.pdf , 1 (accessed March 13, 2010).
39 Ibid, 2.
14
operational power from a low power state.40 The number of read/write operations that can be
completed in one second is called IOPs, or inputs/outputs per second. This metric is used to
measure the random read/write performance of different storage media. High performance
HDDs (15K RPM) normally perform 300 IOPs of random 4-kilobyte (KB) data.41 On the other
hand, SSDs are presently capable of processing 25,000+ 4KB IOPs, which is a tremendous
increase.42
Performance in HDDs is also affected adversely by the resumption of operation following
inactivity. When a HDD is inactive for a certain period of time, it will assume a low power state,
moving the read/write head off to the side and stopping the spinning platters. Upon the next
read/write request, the read/write head must be moved back in place and the platters must be
reactivated. This normally takes on the order of a few seconds to occur. However, when an SSD
is inactive and goes into a low power state, reactivation takes only a few milliseconds.43 The
following characteristics are indicative of solid state storage devices44:
1. Lowest possible access times: 100-1000 times faster than mechanical drives.
2. High bandwidth: provides multiple gigabytes (GB) per second of random data
throughput.
3. High IOPS: provides extremely high random input/output (I/O) performance due to
low access times and high bandwidth.
40
Dane Nelson and Sam Siewart, PhD., “Solid State Drive Applications in Storage and Embedded
Systems,” Intel Technology Journal, Volume 13, Issue 1, 2009,
http://download.intel.com/design/flash/nand/extreme/ITJ_03_SSDs_for_Embedded.pdf, 30, (accessed 03-02-10).
41
Ibid.
42
Brook Crothers, “Seagate Enters Solid-State Drive Market,” CNet News, December 7, 2009,
http://news.cnet.com/8301-13924_3-10411097-64.html (accessed March 13, 2010).
43Nelson. Op. Cit.
44
Tom Coughlin, Neal Ekker, and Jim Handy, “Solid State Storage 101: An introduction to Solid State
Storage,” Storage Networking Industry Association (SNIA) Solid State Storage Initiative, January 2009,
http://www.snia.org/forums/sssi/knowledge/education/SSSI_Wht_Paper_Final.pdf, 10 (accessed March 13, 2010).
15
4. Low price for performance: provides the best possible price/performance of all
storage devices.
5. High reliability: provides the same levels of data integrity/endurance as other
semiconductor devices.
6. Provides more consistent I/O response times.
7. Provides predictable wear and lifespan characteristics.
These characteristics certainly favor solid state drives over mechanical hard disk drives.
However, barriers to the large-scale adoption of SSDs do exist. First, the cost of SSDs is more
than HDDs. These costs are decreasing with the increased demand by organizations that require
the performance, reliability, lower power, and/or resistance to shock and vibration that they
provide.45 Second, the capacity of SDDs is usually much smaller than that of mechanical HDDs.
However, flash density is improving rapidly to meet user demand for increased capacity with
200GB SSDs already commercially available.46,47 Finally, SSDs have a limited number of write
cycles per storage cell. After many program/erase operations, flash memory loses the capability
to retain data. However, increased density and improved write wear-leveling algorithms have
greatly improved the longevity of SSDs, with enterprise-grade SSDs providing as many as 1
million program/erase cycles.48,49
45
Dane Nelson and Sam Siewart, PhD., “Solid State Drive Applications in Storage and Embedded
Systems,” Intel Technology Journal, Volume 13, Issue 1, 2009,
http://download.intel.com/design/flash/nand/extreme/ITJ_03_SSDs_for_Embedded.pdf, 31(accessed 03-02-10).
46Ibid.
47
Brook Crothers, “Seagate Enters Solid-State Drive Market,” CNet News, December 7, 2009,
http://news.cnet.com/8301-13924_3-10411097-64.html (accessed March 13, 2010).
48 Nelson. Op. Cit.
49
Jonathan Thatcher, “NAND Flash Solid State Storage Reliability and Data Integrity: An In-Depth Look,”
Storage Networking Industry Association (SNIA) Solid State Storage Initiative, 2009,
http://www.snia.org/forums/sssi/knowledge/education/Solid_State_Storage_Reliability_and_Data_Integrity--An_In-
depth_Look.pdf, 26 (accessed March 13, 2010).
16
Notwithstanding the shortcomings in SSDs, they are finding increased usage in
environments requiring enhanced response times to include high performance computing,
military applications, and data storage.50 The improving performance of SSD technology
permits them to complement or replace HDDs in computer, communications, and consumer
electronics. They have become the storage media of choice in avionics, industrial, medical, and
military equipment, which requires higher reliability under adverse mechanical and
environmental conditions.51
Direct attached storage (DAS) is typically used in aerospace, industrial, government, and
military applications, which require small form factor, durability, reliability, performance, and
the ability to endure extreme field environments. Currently, SSDs are the only practical solution
for these requirements, finding use in aircraft, weather balloons, spacecraft, missiles, ships,
submarines, trains, tanks and other armored vehicles, and portable military computers to name
but a few.52
The increasing requirements for reliability and performance in military, industrial, and
business applications is driving standards for data storage to extraordinary levels with faster
access, mobility, and extreme reliability becoming more imperative. As the performance
limitations of HDDs are reached and the costs of SSDs continue to decrease, SSDs will play a
more critical role in an environment that demands faster, more reliable data storage.53
50Tom Coughlin, Neal Ekker, and Jim Handy, “Solid State Storage 101: An introduction to Solid State
Storage,” Storage Networking Industry Association (SNIA) Solid State Storage Initiative, January 2009,
http://www.snia.org/forums/sssi/knowledge/education/SSSI_Wht_Paper_Final.pdf , 10 (accessed March 13, 2010).
51Marius Tudor, “Are Flash Solid-State Disk Drives Ready for the Enterprise?,” BitMicro, 2009,
http://www.bitmicro.com/press_resources_flash_ssd_enterprise.php, 1 (accessed March 2, 2010).
52 Ibid. 3.
53Ibid.
17
At the present, the mechanical HDD is the main obstacle to storage performance today,
due to rotational and seek actuation latencies when accessing data. This problem is complicated
when data access is random and of small, distributed I/O. Most hard drives can deliver from a
few hundred to around 100 MB/sec of IOPs when accessing sequential large blocks of data.54
However, this issue can be alleviated via two methods. The first method involves the
direct replacement of HDDs with SSDs, which can provide a tenfold increase in performance.55
The second method involves the use of a Redundant Array of Inexpensive/Independent Disks
(RAID), which can improve HDD performance by allowing concurrent access to data over an
entire array of disks, thereby increasing I/O throughput.56 As such, it is expected that HDDs will
see continued usage where high capacity storage is necessary and SSDs will be used where
higher performance capability is required.57
REDUNDANT ARRAY OF INEXPENSIVE/INDEPENDENT DISKS (RAID)
The term “RAID” stands for “Redundant Array of Inexpensive (or Independent) Disks,”
and it is one of the most important technologies in storage networking. RAID is often associated
with hardware such as disk subsystems and RAID adapters, but it is actually a set of software
algorithms that combine storage input/output (I/O) operations across multiple storage address
locations. RAID is usually applied with disk drives in disk subsystems, but it can also be applied
54Dane Nelson and Sam Siewart, PhD., “Solid State Drive Applications in Storage and Embedded Systems,”
Intel Technology Journal, Volume 13, Issue 1, 2009,
http://download.intel.com/design/flash/nand/extreme/ITJ_03_SSDs_for_Embedded.pdf, 38 (accessed 03-02-10).
55
Ibid.
56Ibid.
57
Tom Coughlin, Neal Ekker, and Jim Handy, “Solid State Storage 101: An introduction to Solid State
Storage,” Storage Networking Industry Association (SNIA) Solid State Storage Initiative, January 2009,
http://www.snia.org/forums/sssi/knowledge/education/SSSI_Wht_Paper_Final.pdf, 10 (accessed March 13, 2010).
18
across multiple disk subsystems in a storage network. It should be noted that these algorithms
are autonomous and apply to any storage media as long as the storage capacities are the same.58
Original research defined five levels of RAID with varying characteristics of
performance, capacity, and redundancy. Of these, levels one and five are the most common.
RAID Level 1 (RAID-1) encompasses simple disk mirroring without the use of parity or striping.
This level does not provide the benefits of performance and scalability related to other RAID
levels. RAID Level 5 (RAID-5) calculates parity values for stored data resulting in more
efficient data redundancy than mirroring.59 Present day has seen the development of other RAID
levels. Table 3 describes the current RAID levels and their main characteristics.60
Table 3. RAID LEVELS.
RAID Type Characteristics
RAID 0 Striping, no redundancy
RAID 1 Mirroring, full redundancy
RAID 10 Mirrored stripes, full redundancy
RAID 2 ECC protection, not used
RAID 3 Byte-interleaved parity, single parity disk
RAID 4 Block-interleaved parity, single parity disk
RAID 5 Block-interleaved parity, rotated parity blocks
RAID 5DP RAID 5 with double parity blocks per stripe Huseyin Simitci, Storage Network Performance Analysis (Indiana: Wiley Publishing, Inc., 2003), 117.
RAID provides four main benefits for data storage. These are as follows61:
1. Data redundancy.
2. Large capacity storage.
3. Management consolidation of devices and subsystems.
4. Parallel processing for enhanced performance.
58
Marc Farley, Storage Networking Fundamentals: An Introduction to Storage Devices, Subsystems,
Applications, Management, and File Systems (Indiana: Cisco Press, 2007), 191.
59Ibid. 196.
60Huseyin Simitci, Storage Network Performance Analysis (Indiana: Wiley Publishing, Inc., 2003), 117.
61
Farley. Op. Cit. 193.
19
The main building block of the RAID system is the array. RAID architecture allows
combining multiple storage entities, called array members, into a single array that functions as a
single, virtual storage device. A RAID can have two or more arrays assembled from member
disk partitions as shown in Figure 1.62
3 U
RAID
CONTROLLER
MEMBER
MEMBER
MEMBER
MEMBER
MEMBER
ARRAY MEMBER
DISK PARTITIONS
Figure 1. RAID ARRAY WITH DISK PARTITION MEMBERS.
Marc Farley, Storage Networking Fundamentals: An Introduction to Storage Devices, Subsystems,
Applications, Management, and File Systems (Indiana: Cisco Press, 2007), 192.
The use of RAID in image and video servers provides fault tolerance, which is the
primary concern for imagery/video databases. These systems attain fault tolerance by either disk
mirroring or parity encoding. Disk mirroring provides fault tolerance by the duplication of data
62Ibid. 192.
20
onto separate disks. Parity encoding techniques use error-correcting codes to reduce storage
overhead.63
In disk mirroring, data is stored on two separate disks creating two copies of the data – a
mirrored pair, as shown in Figure 2. If one disk fails, the data remains intact on the other
(redundant) disk and the controller continues to provide requested data from the surviving disk.
Replacement of the failed disk automatically instructs the controller to copy data from the
remaining disk to the new one – an operation transparent to the host system.64
4
2 2
1
4
3
1
3
DATA 1
DATA 2
DATA 3
DATA 4
MIRRORING
Figure 2. MIRRORING.
Peter Alok Shrivastava and G. Somasundaram, eds., Information Storage and Management: Storing,
Managing, and Protecting Digital Information (Indiana: Wiley Publishing, Inc., 2009), 55.
Mirroring provides complete data redundancy and enables faster recovery time following
disk failure. On the other hand, it only provides data protection and is not to be confused as a
63Lawrence D. Bergman, and Vittorio Castelli, eds., Image Databases: Search and Retrieval of Digital
Imagery (New York: John Wiley & Sons, Inc., 2002), 145.
64
Peter Alok Shrivastava and G. Somasundaram, eds., Information Storage and Management: Storing,
Managing, and Protecting Digital Information (Indiana: Wiley Publishing, Inc., 2009), 55.
21
substitute for data backup. In this instance, mirroring continuously captures data changes, as
opposed to backups, which capture point-in-time images of the stored data.65
The amount of required storage is twice the amount of data needing to be stored, so
mirroring is considered expensive, being used on mission-critical applications which cannot
suffer data loss. The technique does provide improved read performance since read requests can
be read from both disks. However, there is an obvious deterioration in write performance since
each write request manifests itself as two writes on the disks.66
To understand the parity method of fault tolerance, the concept of “striping” must be
explained. In a RAID array, each disk has a predefined number of contiguously addressable
blocks called “strips.” The series of aligned strips spanning across all of the disks in the array is
called a “stripe.”67 This division and distribution of data across multiple disks is called
striping.68
RAID using this technique does not provide data protection unless mirroring or parity is
used in the process. However, use of striping can considerably increase I/O performance if the
RAID controller is configured to simultaneously access data across multiple disks.69
Parity is a mathematical technique used to re-create missing data. Parity RAID protects
striped data from disk failure through the addition of an extra disk. This disk is added to the
65Ibid. 55.
66Ibid. 56.
67Ibid. 54.
68Ibid. 433.
69Ibid. 54.
22
stripe width and is used to hold parity, which provides a redundancy check that ensures
protection of data without having to maintain a full set of duplicate data.70
As opposed to mirroring, the use of parity significantly reduces data protection costs. For
example, in a RAID configuration using five disks (Figure 3), one disk retains the parity
information while the other four retain data. In this case, parity would require only 25 percent
extra disk usage compared to mirroring, which would need 100 percent extra disk usage.71
However, the use of parity is not without its disadvantages. Since parity information is
created by the data currently residing in the disk, it is recalculated for every change in data. This
process is time consuming and negatively affects the performance of the RAID controller.72
3
13
1
3
1
11 2
2
3
1
2
1
1
2
7
5
7
9
PARITY DISK
DATA DISKS
Figure 3. PARITY RAID.
Peter Alok Shrivastava and G. Somasundaram, eds., Information Storage and Management: Storing,
Managing, and Protecting Digital Information (Indiana: Wiley Publishing, Inc., 2009), 56.
Additionally, in the case of a disk failure, the load increase on the surviving disks results
in deadline violations during video stream playback operations. To prevent this occurrence using
70Ibid. 56.
71Ibid. 56-57.
72Ibid.
23
conventional fault-tolerance methods, disk use must be decreased during the fault-free state.
This reduction may be accomplished prior to imagery compression by using partitioning
techniques that take advantage of the characteristics of imagery through the use of two general
methods.73
First, the sequential nature of video access may be used to reduce the system overhead of
on-line recovery in a RAID. This may be accomplished by the computation of parity over a
video stream block sequence, ensuring that retrieved data used to recover a block that was stored
on the failed disk would be requested by the system in the future. The blocks are temporarily
stored in and serviced from the buffer, minimizing the time necessary for the failure recovery
process.74
Second, since human perception is tolerant to minor image distortion, the inherent
redundancies in imagery may be used to reconstruct lost imagery data to a fairly accurate degree.
This method uses error-correcting codes to accomplish this. The imagery is divided into sub-
images that are stored across the array members. The failure of a single disk will result in
fractional losses for several images.75
However, if the sub-images are generated using pixels, and none of the neighboring
pixels belong to the same sub-image, all elements of the lost pixels will be accessible in the event
of a single disk failure. The relationship between the neighboring pixels will allow for an
73
Lawrence D. Bergman, and Vittorio Castelli, eds. Image Databases: Search and Retrieval of Digital
Imagery (New York: John Wiley & Sons, Inc., 2002), 146.
74Ibid.
75Ibid.
24
approximate reconstruction of the original imagery without the need for additional information
from any of the other surviving disks members.76
It should be noted that use of the aforementioned techniques will adversely affect the
compression of the imagery due to the reduction of the correlation between pixels allocated to
the same sub-images. This will increase the bit rate, imposing greater load requirements on all
disks in the RAID and reduce the number of video streams that may be retrieved
simultaneously.77 Post-compression partitioning algorithms have been developed to address this
limitation and are discussed in the following section.
76Ibid.
77Ibid.
25
CHAPTER 2
INTRODUCTION TO VIDEO COMPRESSION
Present day imagery technology has provided an endless amount of photographic and
video data, which presents a major challenge in its capture, processing and storage. As this
technology has evolved, the need to improve the quality and to reduce the scale of the imagery
has become of primary importance. This is particularly true since the development of high
resolution photography and video has created more complex files, which can be of extremely
large size, requiring more resources to use. This ever-growing database has required the
development of techniques to control the size and complexity of the imagery and to allow ease of
use. The science of digital compression technology was born out of this need.
Compression is the science of reducing the amount of data used to convey information
about an object. Because information has order and patterns, these can be extracted and
reconstructed to provide the fundamental nature of the original information using less data for
transmission and reception.78 Certain restrictions are required for the compression of
multimedia. The encoded/decoded data should provide the best possible quality. Complexity
should be minimal to provide limited data link delay and cost-effective implementation. Modern
compression techniques must compromise between these requirements.79
The goal of compression technology is the elimination of redundant data while retaining
only the information that is essential for the effective reproduction of the original data. In other
words, the number of bits represented by a signal is reduced to provide a smaller data file that
78Peter Symes, Digital Video Compression (New York: McGraw-Hill, 2004), 2.
79Wolfgang Effelsberg and Ralf Steinmetz, Video Compression Techniques, (Netherlands: KoninKlijke
Wöhrmann B.V., 1998), 45.
26
still represents the fundamental nature of the original data. This technique is accomplished using
either “lossless” compression, “lossy” compression, or a combination of both methods.80
Lossless data compression algorithms permit exact reconstruction of original data
following compression and allow for compression ratios of about four to one.81, 82 It is used
when it is critical that the original data be identical following decompression. This is important
for executable files, source code, and some image file formats like portable network graphics
(PNG) and graphics interchange format (GIF). It has many applications to include ZIP utilities,
and is often used in conjunction with lossy compression methods.83
As opposed to lossless compression, lossy compression does not allow for complete
recovery of the original signal. Only a near approximation is available due to the tradeoff
required to provide reasonable compression of the signal.84 Compression ratios ranging from
eight to one and twenty to one can provide images that are visually comparable to the original. It
is possible to produce higher ratios for lossy compression, but a noticeable difference will exist
between the original and the compressed image.85 The most common use of lossy compression
80Lawrence D. Bergman, and Vittorio Castelli, eds. Image Databases: Search and Retrieval of Digital
Imagery (New York: John Wiley & Sons, Inc., 2002), 211.
81Ibid.
82Wikipedia contributors, "Lossless data compression," Wikipedia, The Free Encyclopedia,
http://en.wikipedia.org/w/index.php?title=Lossless_data_compression&oldid=346709055 (accessed March 10,
2010).
83Ibid.
84Ibid.
85Lawrence D. Bergman, and Vittorio Castelli, eds. Image Databases: Search and Retrieval of Digital
Imagery (New York: John Wiley & Sons, Inc., 2002), 211.
27
is for the compression of multimedia data like audio, video, and still imagery. So, it finds greater
use in streaming media and internet telephony applications.86
A standard image compression method consists of three components: pixel-level
redundancy reduction, data discarding, and bit-level redundancy reduction. The normal image
compression process is shown in Figure 4. Note that lossless image compression systems do not
use the data discarding process as compression is achieved solely from the redundancy reduction
processes. Lossy image compression generally uses all three processes, but may omit bit-level
redundancy.87
Figure 4. IMAGE COMPRESSION COMPONENTS.
Lawrence D. Bergman, and Vittorio Castelli, eds. Image Databases: Search and Retrieval of Digital Imagery (New
York: John Wiley & Sons, Inc., 2002), 212.
The pixel-level redundancy process reverses the mapping of the image, disassociating the
original pixels from the output. This process enumerates the image pixilation (converts it to data
bits) in a way that more corresponds to the human visual system frequency response. This data
is then transmitted to the data discarding process.88
The data discarding process removes insignificant data from the original data stream that
has been divided into separate data bits during the pixel-level redundancy process. It then
discards the data bits that are deemed unnecessary, and retains those that are minimally required
86Wikipedia contributors, "Lossy compression," Wikipedia, The Free Encyclopedia,
http://en.wikipedia.org/w/index.php?title=Lossy_compression&oldid=347940008 (accessed March 10, 2010).
87Op Cit. 212.
88Ibid.
28
to produce a near-perfect rendition of the original, without loss of image fidelity. Statistical
image properties and human visual characteristics are considered in this approximation. This
data is then sent for bit-level redundancy reduction for final processing.89
The bit-level redundancy reduction process is considered a lossless process. It is used to
remove and/or reduce any unnecessary dependencies left over from the data discarding process.90
Following compression of the imagery by one or more compression standards such as those
developed by the Motion Pictures Experts Group (MPEG), the imagery is then sent to the
applicable storage location.
VIDEO COMPRESSION STANDARDS
Several video compression techniques exist that use different techniques to reduce data
size.91 The most commonly used family of digital video compression techniques is the Motion
Pictures Experts Group (MPEG) standards, which are briefly described in Table 4.
Table 4. MPEG STANDARDS.
STANDARD OBJECTIVE CHARACTERISTICS APPLICATIONS
MPEG-1 To provide a standard for encoding motion video at bit rates transportable over T1 data circuits and for
replay on CD-ROM.
Provided audio and video recording at the same
data rate.
Broadcasting in any form and
large-distribution CD-ROMs.
MPEG-2 To provide a standard that included broadcast
quality video.
Provides video interlacing, scalable syntax, a
system layer to handle multiple program
streams.
Most popular standard presently
in use. Used for HDTV and for compression of other
video/audio data.
MPEG-3 To provide a compression system suitable for high-
definition television (HDTV) broadcasting. Abandoned when determined that MPEG-2 was
able to accommodate the need for HDTV. N/A
MPEG-4 To provide for encoding of video and audio at very
low bit rates.
Provides for the combining of multiple video streams. Increased efficiency and error
forgiveness over MPEG-2.
Provides end-user interaction
with specific applications like
games, interactive TV, and educational systems.
MPEG-7 To provide a standard for applying metadata
technology to recorded video and audio.
Uses metadata technology to catalog/ index
video/audio files to accommodate search engine retrieval.
In work, but will provide ease of
retrieval of data in large-scale databases.
MPEG-21 To provide a complete structure for the management
and use of digital assets.
Provides all infrastructure support for
commercial transactions and rights
management.
In work, but will provide
intellectual property
management and protection.
Peter Symes, Digital Video Compression (New York: McGraw-Hill, 2004), 152,172, 196, 268, 276.
89Ibid.
90Ibid.
91
Peter Symes, Digital Video Compression (New York: McGraw-Hill, 2004), 2.
29
MPEG is a committee that was formed in 1988 under the Joint Technical Committee of
the International Standards Organization (ISO) and the International Electro-technical
Commission (IEC). Their formation was facilitated to derive a standard for encoding full motion
video at rates suitable for transport over T1 data circuits and for replay from CD-ROM at about
1.5Mbits/sec. This was the birth of the MPEG-1 standard, which ultimate goal was to record
sampled audio at 48 kHz together with video, at the same data rate.92
Following the development of MPEG-1, a standard was required that allowed for the
compression, storage, and digital transmission of television broadcast quality video. This project
became MPEG-2, which extended and improved MPEG-1.93 Among other things, MPEG-2
included the provision of video interlacing, a scalable syntax, and the addition of a system layer
to handle multiple program streams. In light of this, MPEG-2 has become the worldwide
standard for both standard- and high-definition transmissions.94
After development of MPEG-2, a standard that would support compression suitable for
high-definition television (HDTV) was deemed necessary. This effort became MPEG-3.
However, it was discarded very early into the program when it became clear that the new
compression algorithms and schemes used in MPEG-2 were capable of handling the HDTV
requirement.95
With the demise of MPEG-3, MPEG-4 was started to accommodate the encoding of
video/audio at very low data rates. Later amendments also provided applications that allowed
92Ibid. 152.
93Wolfgang Effelsberg and Ralf Steinmetz, Video Compression Techniques, (Netherlands: KoninKlijke
Wöhrmann B.V., 1998), 45.
94Symes. Op. Cit. 172-173.
95Effelsberg. Op. Cit. 45.
30
end-user interaction with the video. This was particularly useful for gaming, interactive
television, educational systems, and high-end data retrieval systems. However, the standard has
not seen wide acceptance due to the popularity of MPEG-2.96
The current amount of digital video and audio information presently available in the
public domain is enormous and continues to grow. To sort, maintain, and access required
information in a timely manner, MPEG-7 has been proposed and may provide the future means
by which digital video is cataloged, stored, and retrieved. The MPEG-7 standard uses metadata
technology to both catalog and index audio-video files using methods that complement data
retrieval.97
The latest effort in video compression by the MPEG organization is MPEG-21, which
also accommodates metadata technology. The standard was undertaken to provide a complete
framework for the use and management of digital resources. Infrastructure support for
commercial transactions and rights management is also an important part of the standard, which
focus is “to enable transparent and augmented use of multimedia resources across a wide range
of networks and devices.”98
METADATA
To better understand what the MPEG-7 and MPEG-21 standards can provide to the
organization and retrieval of multimedia data, the concept of metadata requires further
explanation. Metadata is the “data about the data” or the “information about the information.” It
96Op. Cit. 196, 222.
97Ibid. 268-269.
98Ibid. 276.
31
describes the various attributes of a given piece of information by providing meaning and
context. It also aids in the location, retrieval, use and management of the information.99, 100
Metadata provides many benefits to include101:
1. Aids in the search for required information.
2. Aids in organization of digital resources.
3. Facilitates interoperability and legacy resource integration.
4. Provides digital identification.
5. Supports data archiving and preservation.
Typical remote sensing systems have enormous amounts of imagery that must be
maintained. To keep track of this data, information about the images is necessary. This
metadata can be stored with the imagery or in a separate database using pointers to link the
applicable imagery with its respective data set. Various metadata may be used in search and
retrieval operations while other metadata may provide user information.102
Metadata, as used in remote sensing systems, consists of several characteristics with
some elements associated with a collection of images and others that specify smaller subsets
called “granules,” which can be a single image or a small set of images. Examples of these
characteristics are as follows103:
1. General Description, which includes the name, topic, collection version, etc.
2. Data Origin, which describes the platform, instrument, sensor used, etc
3. Spatial Coverage, which specifies the geographic area covered by the imagery in the
collection (e.g., location coordinates of the corners of a bounding rectangle).
99
Cornell University Library, “Moving Theory into Practice: Digital Imaging Tutorial, Chapter 5,
Metadata: Definition, Types, and Functions,”
http://www.library.cornell.edu/preservation/tutorial/metadata/metadata-01.html (accessed January 14, 2010).
100National Information Standards Organization, Understanding Metadata (Maryland: NISO Press, 2004),
http://www.niso.org/publications/press/UnderstandingMetadata.pdf (accessed January 20, 2010), 1.
101Ibid.
102Lawrence D. Bergman, and Vittorio Castelli, eds. Image Databases: Search and Retrieval of Digital
Imagery (New York: John Wiley & Sons, Inc., 2002), 62.
103Ibid.
32
4. Temporal Coverage, which specifies the time range during which the imagery was
obtained (e.g., start/end date/time).
Standards for the collection of geographic data have been established by the U.S. Federal
Geographic Data Committee (FGDC). All geographic data that is managed by federal
government agencies must comply with this standard. The standards serve as a guideline for the
use of metadata in remote sensing applications.104 The ISO has also established a working group
on metadata for the purpose of developing standards for the recording of metadata on a separate
track than the audio-video. This would provide a powerful means of retrieval for later access to
the digitized data and illustrates the importance of the MPEG-7 and MPEG-21 standards.105 The
rest of this section will discuss metadata in regards to the MPEG-7 standard.
The scope of the MPEG-7 standard is illustrated in Figure 5. From the figure, it is clear
that only the syntax (Standard Description) is standardized. Devices that might generate
metadata and how that data is represented are specified by the standard. Generation of metadata
(i.e., Feature Extraction) and the applications that might use it (i.e., Search Engines) are
unspecified and left to commercial developers to provide.106 Several groups are presently
developing video content analysis algorithms to automatically extract semantic information from
video data with the hopes of partially automating the metadata creation process.107
104Ibid. 63.
105Wolfgang Effelsberg and Ralf Steinmetz, Video Compression Techniques, (Netherlands: KoninKlijke
Wöhrmann B.V., 1998), 58.
106
Peter Symes, Digital Video Compression (New York: McGraw-Hill, 2004), 270.
107Effelsberg. Op. Cit.
33
FEATURE EXTRACTION STANDARD DESCRIPTION SEARCH ENGINE
MPEG-7 SCOPE
Figure 5. SCOPE OF THE MPEG-7 STANDARD.
Wolfgang Effelsberg and Ralf Steinmetz, Video Compression Techniques, (Netherlands: KoninKlijke Wöhrmann
B.V., 1998), 58.
Assuming metadata is recorded simultaneously with the audio-visual data and timing
information, search engines can be developed that can access specific audio or video data from
large digital archives or databases.108 This timely access to required material can provide
immediate use – an important consideration especially for mission critical applications.
The generation of metadata poses two main issues. The first is the amount of data that is
generated. Human interaction in the indexing process will result in only a fraction of the total
created content being indexed. In this case, the automated analysis of digital records is critical
and presents many technical challenges. The second issue regards the standardization of
metadata descriptions. The situation has improved with the widespread availability of
information.109
Conversely, practical application of metadata has been made problematic by the creation
of different standardization methods. The MPEG-7 standard calls for extensible markup
language (XML) schemas as opposed to the Society of Motion Picture and Television Engineers
(SMPTE) which uses key-length-value (KLV) coding. Other organizations have created their
108Ibid.
109Peter Symes, Digital Video Compression (New York: McGraw-Hill, 2004), 270.
34
own metadata sets that are based on their own needs. This proliferation of different metadata
methodologies creates interoperability issues, which was never the intent of MPEG-7. Efforts
are being led by MPEG and SMPTE to synchronize the different methodologies to allow
portability between them.110
MULTIMEDIA PLAYERS AND CONTAINER FORMATS
This discussion is not complete without providing a clear distinction between multimedia
players and multimedia container formats. Multimedia players are the software applications that
are used to playback video and audio data files. There are many of these applications available
to include Windows Media Player®, Real Player®, and QuickTime®, to name but a few.
Multimedia container formats provide for the actual formatting of the recorded video and audio.
Different methods exist and are signified by the actual data file extensions. These include, but
are not limited to the following format types: AVI, ASF, and MOV.
Many of the currently fielded video collection systems use digital video recorder utilities
that apply the audio video interleave (AVI) container format for playback operations. Playback
of applicable video is then performed using one of the many available multimedia players like
Windows Media Player®. The AVI format is proven over some years of usage and has continued
to be supported by many multimedia players. For purposes of this paper, only the AVI container
format will be discussed.
The AVI format has its origins in the Resource Interchange File Format (RIFF), which
divides the data into “blocks” that are each identified by a tag. The AVI file is a single block,
RIFF formatted file that is further divided into two mandatory sub-blocks and one optional sub-
block. The first sub-block is the file header, which contains video metadata such as frame width,
110Ibid. 275-276.
35
height and rate. The second sub-block contains the actual audio-visual data that is the AVI
movie file. The third, optional, sub-block is used to index data block offsets within the file.111
Via the RIFF format, the sub-block containing the actual audio-visual data is encoded
and/or decoded by a software algorithm called a “codec,” an abbreviation for (en) coder/decoder.
Upon file creation, the codec interprets between raw data and the compressed data format used
within the sub-block. AVI files can hold audio-visual information inside the data blocks in
almost any compression scheme to include full frame (uncompressed), Motion JPEG, editable
MPEG, and MPEG-4 to name a few.112
AVI was not originally intended to contain compressed video, which requires access to
video frame data beyond the currently used frame. Methods do exist that support modern video
compression use within the AVI framework (MPEG-4, et. al.), even though this is outside the
scope of the original specification. Problems can occur during playback with utilities that do not
anticipate any issues like not having the correct codec required to play the video file.113
However, most audio-video files can be reviewed for playback provided the necessary
codec files are installed on the computing machinery that is being utilized for this purpose. As
such, the AVI format continues to see major use in both commercial as well as military
applications.
The previous sections have described some of the concepts related to data storage. In
particular, storage media, data redundancy, and video compression technologies were discussed
111Wikipedia contributors, "Audio Video Interleave," Wikipedia, The Free Encyclopedia,
http://en.wikipedia.org/w/index.php?title=Audio_Video_Interleave&oldid=347331467 (accessed March 10, 2010).
112Ibid.
113Ibid.
36
in relation to the large scale storage of recorded audio-video data files. The discussion will now
turn to possible solutions to the storage of this data in relation to tactical video acquisition.
37
CHAPTER 3
RECOMMENDATIONS
HARDWARE CONSIDERATIONS
Although somewhat outside the original scope of this paper, it is important that the
system developer/user consider the computing machinery to be used in the tactical video
collection system. The personal computer (PC) has advanced to the point that powerful, robust
systems are available commercially at a fraction of the cost of high performance systems from
years past. Most PCs available for commercial-off-the-shelf (COTS) purchase are quite suitable
for digital video recording functions. Notwithstanding the obvious need for robust computing
machinery, given the critical nature of tactically fielded video collection systems, some minimal
hardware recommendations are as follows:
1. Robust PC-based system using a dual-processor, 64-bit bus motherboard with large
random access memory (RAM) capacity.
2. Best commercially available video graphics adapter using the most video RAM.
3. Main internal storage consisting of solid state hard drives (SSDs).
4. Redundant backup using an external (or internal) RAID-based system.
5. CD-RW/DVD-RW/Blu-Ray-RW for archiving purposes.
Following these deliberations, the primary, internal system storage should consist of solid
state drives (SSDs) of sufficient size to accommodate the operating system (OS), applicable
system and recorder applications, and onboard storage. The use of SSDs will allow for enhanced
system performance during mission critical operations. External (or internal) RAID storage is
recommended using standard hard disk drives, which provide more efficient mass storage
properties and should be sufficient for normal backup operations.
Regarding archival storage, the use of Blu-Ray optical disks is recommended given both
the large capacity and the storage longevity of optical disks. Compared to other optical storage
mediums as well as magnetic tape, Blu-Ray best accommodates the requirements necessary for
38
the storage of large amounts of captured video data, and it will also reduce hard storage
requirements. A simple hierarchy depicting these guidelines is shown in Figure 6 followed by
brief explanations for each level.
LEVEL 3
ANCILLARY STORAGE
(STORED ON-SITE)
LEVEL 2
SECONDARY STORAGE
(EXTERNAL/INTERNAL RAID)
LEVEL 1
PRIMARY STORAGE
(INTERNAL)
Figure 6. SIMPLE STORAGE HIERARCHY.
LEVEL 1:
Primary storage, internal to the computing machinery and using flash Solid State Drives (SSDs)
for their enhanced performance and data retrieval.
LEVEL 2:
Secondary storage for database back up (automated or manual) using RAID-based architecture
and consisting of mechanical Hard Disk Drives (HDDs) for their mass storage benefits.
LEVEL 3:
Ancillary storage using Blu-Ray optical disks for their large storage capacity, environmental
durability, and long-term storage properties.
39
SOFTWARE CONSIDERATIONS
Once the computing machinery is in place, the system developer/user must next consider
the software operating system (OS) and the required application software to be used with the
system. The Windows® OS is recommended for its ready availability and widespread use. The
digital video recorder (DVR) application may be selected from open or proprietary sources. It
should be robust, and its selection based on system considerations. In addition to its recording
function it must also provide post-mission playback and video editing utilities.
For backup operations, an automated backup and restore system utility is recommended.
It should have a built in scheduling capability and operate in the background, transparent to the
host system. It should provide the capability for different types of backup operations, the most
common of which are explained in Table 5. For mission critical operations, it is recommended
that Full backups be performed to provide complete restoration of all data in case of corruption
or catastrophic system failure. The utility should also provide the ability to automatically
overwrite older video data files when necessary, in the interest of freeing up hard drive space.
This means that any video that is intended to be kept for planning or historical purposes must be
manually copied and archived on separate media, preferably Blu-Ray optical disk.
Table 5. BACKUP OPERATIONS.
BACKUP
TYPE OPERATION CHARACTERISTICS
Full Provides full backup of all data. Time intensive, but provides all data for complete
restoration.
Incremental Only copies files that have changed since last
full backup.
Assuming weekly backups, subsequent backups
may be larger.
Differential Only copies data that has changed since last
differential backup.
Shortens backup time, but may increase
restoration time.
Snapshot Point-in-time copy of the data, also called
pointer-based backup. Copies pointers and file
metadata.
Very fast backup - copies only metadata and
pointers that point to the data blocks. Does not
protect against disk loss.
Schulz, Greg, Resilient Storage Networks: Designing Flexible Scalable Data Infrastructures (Massachusetts:
Elsevier Digital Press, 2004), 324.
40
A suggested backup scheme is shown in Figure 7. It is recommended that a Master
System disk be produced on Blu-Ray optical media following final system setup. This should be
maintained on site and will provide a backup in case the magnetic hard disk drive needs to be
rebuilt due to file corruption or catastrophic system events.
Additionally, a Full system backup that is stored in the external RAID storage assembly
should be scheduled as required to intermittently capture the present system configuration. This
will allow the user to recover system operation quickly by using a previously known working
system configuration. This is also useful in the event of system file corruption or other events
that may affect the system operation.
Recorded video should also be captured in the external RAID storage area to keep the
internal, primary storage medium free of unnecessary files that may adversely affect the overall
system performance. The RAID storage capacity should be in the terabyte range to
accommodate the large amount of data that will be generated by the video collection system. At
a minimum, RAID-1 architecture providing mirroring and full redundancy is recommended.
41
Figure 7. BACKUP SCHEME.
Given the nature of real-time, tactical video collection systems, large amounts of data
are produced within a short period of time. In the interest of freeing up storage space for the
recording of newer video, it is recommended that any video that is desired to be retained for
historical reference should be archived on optical media and stored on-site to facilitate its later
usage. A master system copy stored on optical disk should also be maintained for recovery
operations due to system corruption and/or catastrophic system failure. This should also be
maintained on-site and be updated as required.
VIDEO RETENTION AND DELETION SCHEMES
Tactical operations are defined as “military operations conducted on the battlefield,
generally in direct contact with the enemy.”114 Tactical data collection is indigenous to the area
of operations in which the tactical collection asset is fielded. Given the nature of tactical
114Tactical Operations. Answers.com. The Oxford Essential Dictionary of the U.S. Military,
Oxford University Press, 2001, 2002. (accessed March 07, 2010).
42
intelligence data such as real-time video, the usefulness of the data may diminish quickly within
a very short period of time.
However, this data is still retained in storage although it may not provide any useful
intelligence value. This is generally true for most video that is collected over time and the
burden on storage assets can be significant. This section outlines a selection process that may be
used to determine automatic retention and/or deletion of tactically collected video.
Figure 8 is an example of a simple, software driven selection process, which could use
criteria such as the number of times a file has been accessed within a given period of time to
determine retention or deletion. It might also be based on guidelines established by the
command to determine the long-term or short-term tactical intelligence value of specific video.
RAW VIDEOVIDEO
PROCESSING
VIDEO
DATABASEVIDEO FILE
VIDEO
SCREENING
PROCESS
RETENSION
DELETION
NO
TACTICAL
VALUE
TACTICAL
VALUE
Figure 8. A SIMPLE VIDEO RETENTION/DELETION SCHEME.
A more in depth selection process is shown in Figure 9. The displayed retention times
shown are arbitrary examples. The automatic deletion of stored video imagery would rely on
specific, established criteria to determine whether a file is a candidate for deletion. At a
minimum, criteria might include the date/time stamp (from metadata) and the number of times
the specific files have been accessed, which denotes level of importance. However, user
43
intervention might be required to determine the retention of specific video that contains
significant events such as:
1. Reconnaissance operations.
2. Surveillance operations.
3. Intelligence gathering operations.
4. Mission operations support.
5. IED incidents.
6. Counter-IED operations.
7. Area/terrain observations in support of missions.
8. Areas of interest (AI).
9. Any video showing direct/indirect contact with an enemy.
10. Other significant events that are deemed to provide intelligence value of a tactical
nature, as determined by the command.
VIDEO
DATABASEPROCESSED VIDEO VIDEO FILES
RETENTION
SCREENING
ALGORITHM
CRITICAL
VALUE
HIGH VALUE
MEDIUM
VALUE
LOW VALUE
NO VALUE
TAGGED VIDEO
RETAIN INDEFINITELY
RETAIN 6-12 MONTHS
RETAIN 3-6 MONTHS
RETAIN 1-3 MONTHS
RETAIN 30 DAYS
TAGGED
VIDEO
3-6MOS6-12MOS
1-3MOS
AUTOMATED
DELETION
ALGORITHM
30 DAYS
DELETED VIDEO
INDEFINITE
RETENTION
Figure 9. A MORE ROBUST VIDEO RETENSION/DELETION SCHEME.
Additionally, unless it is used for planning purposes or as an historical reference in
support of future operations, most tactically captured video may have a short shelf life for
44
providing up to date intelligence information. Of course, this is dependent upon the intelligence
requirements and the value placed on specific video by the command.
FUTURE CONSIDERATIONS
Current digital video recorder software applications use a date/time stamp to sort
recorded video. This does allow for automated sorting as the video data files are sent to a
predetermined storage location and are sorted according to the date and time they were recorded.
However, this convention does not allow for the sorting of video based on its content. The
search and retrieval of multiple related records is also not supported. In normal operations, the
user must make note of the specific time and date of an event and access the file manually,
usually having to manually parse through different file locations and numerous video files to
locate the one in question. The use of metadata can alleviate this issue.
The U.S. government’s Federal Geographic Data Committee (FGDC) describes metadata
as the data about the content, quality, condition, and other characteristics of the data.115 At this
time, the FGDC is the organization that has been tasked with developing the guidelines and
standards for digital metadata that is to be used in geospatial imagery. Although metadata is
being developed primarily for use in the support of geographic information system (GIS)
mapping data, it would serve equally useful from the perspective of tactical video collection.
Some minimal considerations for items that might be included as part of a data element set that
should be included with the metadata might be as follows:
1. Time/date stamp.
2. Geographic locations of known points.
3. Location Coordinate System (Military Grid Reference System (MGRS)).
4. Distance/altitude units (feet/meters).
115Federal Geographic Data Committee, Content Standard for Digital Geospatial Metadata,
http://www.fgdc.gov/standards/projects/FGDC-standards-projects/metadata/base-metadata/v2_0698.pdf, 64
(accessed January 20, 2010).
45
5. Altitude/elevation data in reference to location.
6. Populated areas: cities, towns, and villages.
7. Rural areas: farms, ranches, etc.
8. Infrastructure: schools, public buildings, places of worship, power stations, military
installations, police stations, telecommunications (antenna towers), water treatment
facilities, transportation (roads/highways both improved and unimproved,
intersections, traffic circles), airfields/airports/airstrips, dams, bridges, waterways
(lakes, oceans, seas, rivers, streams, man-made), etc.
Many of these items already have a place in the metadata guidelines developed by the
FGDC. Other important metadata considerations have also been outlined by the FGDC, which
include mission specific data items. These may easily be adapted in the metadata that is used in
support of tactical video collection efforts. Some of this mission specific data is as follows116:
1. Mission data/time.
2. Mission name.
3. Mission significant events.
4. Mission Platform and Instrumentation.
At the present, the use of metadata in many video applications has been limited. Its usage
may also necessitate user intervention for the manual input of some data items. However, the
advent of the MPEG-7 and MPEG-21 video compression standards may provide a path forward
for the intelligent organization of video data files. The adoption of these compression methods
in future video collection systems will allow video files to be “tagged” and automatically sorted,
catalogued, and organized according to specific attributes that are captured as part of the
metadata architecture. This will aid in the timely search and retrieval of related video data
applicable to specific software queries.
116Federal Geographic Data Committee, Content Standard for Digital Geospatial Metadata: Extensions for
Remote Sensing Metadata, http://www.fgdc.gov/standards/projects/FGDC-standards-
projects/csdgm_rs_ex/MetadataRemoteSensingExtens.pdf, 67-71 (accessed January 20, 2010).
46
CHAPTER 4
CONCLUSION
The purpose of this paper was to evaluate the issue of database storage for the tactical
video collection process. This is a major concern for end users of these systems given the large
amount of video that is collected on a daily basis. The previous discussion has attempted to
explain the necessary hardware that is available and the software related techniques that are used
in this process. Possible recommendations were also provided that may assist in easing the
concerns related to this issue.
First, large capacity storage media were discussed to include magnetic tape, optical disks,
magnetic disk drives, and finally solid state drives. These media types are of particular
importance since they provide the best large-scale storage capacity. Each was evaluated to
determine which of them were best suited to be used in a tactical environment. Redundant
storage in the form of the RAID architecture was also examined.
Magnetic tape media has been a mainstay for the backup of digital video and imagery as
it provides a low cost method of backing up data for archival purposes. However, the sequential
nature of magnetic tape media makes it very inefficient for random data access, and
recovery/backup operations. This is particularly important when fielding a tactical collection
system since the command cannot afford to be without the asset for long periods of time
following, for example, a catastrophic failure. The system must be recoverable very quickly.
Magnetic tape media also deteriorate over time and must be stored in moderate environments
with low humidity. The conditions in many of the current locations in which these assets are
fielded make the use of magnetic tape media problematic.
47
Optical disks have many advantages over magnetic tape to include lower cost, greater
storage capacity, data stability, environmental durability, and a long shelf life potential of over
100 years. Their constant linear velocity provides unvarying read/write bandwidth during
sequential access operations, which makes them ideal for multimedia backup and restore
operations. These reasons have made optical disk technology the primary backup media in many
remote sensing operations.
Magnetic hard disks, also known as hard disk drives, are the primary storage technology
used in most personal computer systems and many video collection systems. They provide fast,
concurrent access to data and have very large capacities. Because of this, they are still useful as
a storage medium in redundant systems such as RAID assemblies. However, as they are electro-
mechanical devices with moving parts, extreme environments together with continual use can
dramatically increase their mortality rate.
Solid state drives, on the other hand consist of high reliability flash memory technology
and have no moving parts, which makes them more durable in severe environments. They have
much faster response times, lower access times, and much higher throughput bandwidth than
magnetic hard disks. Lifespan characteristics are comparable to magnetic hard disks with on-
the-shelf data retention of up to ten years.
However, solid state drives are more expensive than magnetic hard disks and capacities
are usually much smaller. In spite of these shortcomings, solid state drives are fast becoming the
primary storage medium for applications that require quicker response time and proven
durability in extreme environments.
The discussion on RAID described its importance as a backup storage medium and some
of the methods used in the different RAID architectures. Two of the most prevalent methods
48
were examined and included RAID Level 1 (RAID-1) and RAID Level 5 (RAID-5). Both
methods provide a means of data redundancy and recovery in case of system failures and/or hard
disk drive loss.
RAID-1 architecture performs only disk mirroring, which provides full redundancy. It
also permits quicker recovery in the case of disk failure in the array, and offers increased read
performance as data is read across all disks in the array. However, it lacks in overall
performance as it is necessary to have twice the storage capacity to accommodate data storage,
which can add significantly to costs.
In comparison, RAID-5 architecture uses parity checking methods and provides more
efficient data redundancy than simple mirroring. It is also less expensive in regards to data
protection costs. However, every change in data facilitates a recalculation of parity, which
adversely affects system performance.
Following the discussion on storage media, the subject of video compression was
investigated. Video compression deals with the reduction of the amount of data required to
convey a useable facsimile of a specific data object. This is particularly important since most
present day imagery can produce extremely large files, which facilitates the need for larger
storage capacities. Both “lossless” and “lossy” compression techniques were examined and are
used to compress data according to the different standards that have been developed to include
those of the Motion Pictures Experts Group (MPEG), which are the most common.
Lossless compression techniques allow for complete reconstruction of the original data
signal and provide a slight amount of compression. Lossy compression techniques provide only
a near approximation of the original data signal since some tradeoff is required to provide
satisfactory compression. Imagery can be produced that is fundamentally and visually similar to
49
the original using ratios much greater than those of lossless compression. Lossy compression is
most commonly used for compressing multimedia data such as audio, video, and still images.
These compression algorithms are used in conjunction with different standards, with the
most prevalent being developed by the MPEG. The standards outline the basic requirements for
the compression of audio, video, and imagery data. Short discussions of the different MPEG
standards were presented with particular attention given to the MPEG-7 and MPEG-21
standards, which are still in their infancy.
These standards do provide for the necessary compression of multimedia data, which is
the primary consideration. However, it is their use of metadata techniques that is of particular
importance. Since metadata considers the peripheral, supporting information about the actual
data, its use for the indexing and cataloging of collected data is significant to the video collection
process.
Multimedia players and multimedia container formats were briefly discussed to provide a
distinction between the two. Multimedia players, like Windows Media Player®, are the
applications that are used to replay the recorded audio-video data files. Container formats like
audio video interleave (AVI) provide the actual formatting process for the video. The AVI
container format was explained solely because of its widespread usage and to provide an
elementary understanding of this technology. No recommendation has been made for these
items since there are many different types which work equally well.
Following discussion of the different storage media and software techniques used in the
storage process, recommendations were made that might assist in alleviating storage capacity
issues in tactical video collection systems. These recommendations were divided into three
50
groups: hardware, software, and future considerations. Recommendations were provided for
each of these categories.
In developing the collection system, hardware considerations were taken into account to
include the motherboard architecture, system memory, processing power, the graphics adapter
and internal storage. As such, these suggestions were offered mainly as guidelines with the
system storage being the primary focus. In this case, it was recommended that solid state drives
be used as the primary data storage media in the computing machinery for tactical video
collection systems due to their higher performance capability and durability. Magnetic hard
disks should still be used in secondary data storage and backup operations as part of a RAID
assembly since they provide higher capacity storage than solid state drives.
Redundant storage was also discussed in the form of a RAID assembly, which provides
for system backup and large external storage capacity. Magnetic drives should be used for this
assembly as they are more efficient than solid state drives for mass storage and should suffice for
normal backup operations. The tactical video collection system is a mission-critical platform
that cannot tolerate any loss in data, if at all possible. In light of this, it was recommended that
RAID-1, providing complete data redundancy, be used to accommodate quicker system recovery
in case of system corruption or total system failure.
The archiving of data was also examined using optical disks as the storage medium.
Their large capacity, data retention characteristics, and durability made them the best choice for
this issue. Blu-Ray optical disks were recommended because they presently provide the most
storage capacity of all optical disk types.
Software considerations included the operating system, the digital video recorder (DVR)
application, and a backup and restore utility. In these cases, the Windows® OS was
51
recommended due to its widespread usage. No DVR application was suggested as this should be
selected based on system requirements. At a minimum, it should provide the necessary
recording function as well as post-mission playback and video editing utilities. With respect to
the system backup operations, a reliable backup and restore application was also examined. Its
operation should be transparent to the host system and provide automated, scheduled backups as
deemed necessary. Full backups were recommended to provide complete repair of the system in
case of file corruption or system failure. These scheduled backups should be placed in the
secondary storage area (RAID assembly) to provide expedited system repair or rebuild as
required.
Following system setup, it was recommended that a master system disk be created on
Blu-Ray optical disk to facilitate system repair or rebuild in case of catastrophic system failure or
internal disk issues. Tactical video intended to be kept for historical purposes should also be
copied to optical media to facilitate its later usage. These items should be kept in the on-site
archive. It was also suggested that all recorded video be captured and stored on the external
RAID assembly in the interest of keeping the primary storage area free of any extraneous files
that might affect system operation.
Proposals were made for software metrics that could provide for the automated retention
or deletion of stored video in the interest of freeing up storage space. These metrics could use
items such as the time/date stamp and the number of times a file has been accessed to determine
retention or deletion of files from the video database. Other versions of this method might base
retention and deletion on the actual nature of the video and its importance to the command.
Different grades or levels of importance might be assigned to provide a clear distinction between
video that is considered of tactical value and worth retaining and video that is deemed as
52
unimportant and requiring deletion. Some user intervention may still be required for these
methods, but they could provide a basis for determining the importance of the collected video
and its subsequent tactical value.
Finally, future considerations were alluded to, in particular the use of metadata as an
automated cataloging and indexing tool. The metadata technology is used to describe the
background information attributed to the collected data. The use of this technology is
accommodated in the MPEG-7 and MPEG-21 compression standards, which are currently in
work by the Motion Pictures Experts Group (MPEG) organization.
The Federal Geographic Data Committee (FGDC) provides the oversight regarding the
use of metadata and is the current working body responsible for the standardization of the
metadata technology. The guidelines developed by the FGDC are currently used in conjunction
with the development of GIS applications.
However, these same guidelines might be easily adapted for use on tactical video
collection systems and may provide the means for the expeditious retrieval of video data sets that
correspond to specific events, times, and locations. This makes it a potentially powerful
intelligence tool and may provide a foundation for content-based video data retrieval in the
future.
53
BIBLIOGRAPHY
Bergman, Lawrence D., and Castelli, Vittorio. Image Databases: Search and Retrieval of
Digital Imagery. New York: John Wiley & Sons, Inc., 2002.
Christopher Drew. “NYT: Military deluged in intelligence from drones - Remote-controlled
planes produced about 24 years’ worth of video in 2009.” New York Times, January 10,
2010, Sunday; http://www.msnbc.msn.com/id/34798080/ns/world_news-
the_new_york_times/ (accessed January 11, 2010).
Cornell University Library. “Moving Theory Into Practice: Digital Imaging Tutorial, Chapter 5,
Metadata: Definition, Types, and Functions.”
http://www.library.cornell.edu/preservation/tutorial/metadata/metadata-01.html (accessed
January 14, 2010).
Coughlin, Tom, Ekker, Neal, and Handy, Jim. “Solid State Storage 101: An introduction to
Solid State Storage.” Storage Networking Industry Association (SNIA) Solid State
Storage Initiative, January 2009,
http://www.snia.org/forums/sssi/knowledge/education/SSSI_Wht_Paper_Final.pdf
(accessed March 13, 2010).
Crothers, Brook. “Seagate Enters Solid-State Drive Market.” CNet News, December 7, 2009,
http://news.cnet.com/8301-13924_3-10411097-64.html (accessed March 13, 2010).
Daintith, John. "Magneto-Optic Storage." A Dictionary of Computing. 2004. Encyclopedia.com.
http://www.encyclopedia.com/doc/1O11-magnetoopticstorage.html (accessed March 5,
2010).
Effelsberg, Wolfgang and Steinmetz, Ralf. Video Compression Techniques. Netherlands:
KoninKlijke Wöhrmann B.V., 1998.
Encyclopedia.com. "Compact Disk." The Columbia Encyclopedia, Sixth Edition, 2008,
http://www.encyclopedia.com/doc/1E1-compactd.html (accessed March 4, 2010).
Encyclopedia.com. "Digital Versatile Disc." The Columbia Encyclopedia, Sixth Edition,
2008, http://www.encyclopedia.com/doc/1E1-DVD.html (accessed March 4, 2010).
Encyclopedia.com. "Optical Disk." The Columbia Encyclopedia, Sixth Edition, 2008,
http://www.encyclopedia.com/doc/1E1-optidisk.html (accessed March 4, 2010).
Farley, Marc. Storage Networking Fundamentals: An Introduction to Storage Devices,
Subsystems, Applications, Management, and File Systems. Indiana: Cisco Press, 2007.
Jensen, John R. Introductory Digital Image Processing: A remote Sensing Perspective. New
Jersey: Prentice-Hall, 2005.
54
Judd, Josh. Principles of SAN Design: Design Build and Manage SANS. Pennsylvania: Infinity
Publishing, 2007.
Judd, Josh and O’Conner, Michael. Introducing File Area Networks Advance Edition: Your
First Look at FAN Technology. Pennsylvania: Infinity Publishing, 2007.
Luff, John. “Technology in Transition: Video storage.” Broadcast Engineering,
http://broadcastengineering.com/mag/broadcasting_video_storage/index.html (accessed
January 18, 2010).
Luff, John. “Video Storage.” Broadcast Engineering, April 1, 2003,
http://broadcastengineering.com/mag/broadcasting_video_storage/index.html (accessed
January 17, 2010).
Luff, John. “Video Storage.” Broadcast Engineering, July 1, 2004,
http://broadcastengineering.com/mag/broadcasting_video_storage/index.html (accessed
January 17, 2010).
Mediatechnics Systems, Inc. "How a CD is Made." Mediatechnics Systems, Inc., FAQ,
https://www.mediatechnics.com/cdfaqs.htm (accessed March 5, 2010).
Mediatechnics Systems, Inc. "How are DVDs Made for use in Copiers?" Mediatechnics
Systems, Inc., FAQ, https://www.mediatechnics.com/dvdcopiers.htm (accessed March 5,
2010).
National Information Standards Organization. “Understanding Metadata.” (Maryland: NISO
Press, 2004), http://www.niso.org/publications/press/UnderstandingMetadata.pdf
(accessed January 20, 2010).
Nelson, Dane and Siewart, PhD, Sam. “Solid State Drive Applications in Storage and Embedded
Systems.” Intel Technology Journal, Volume 13, Issue 1, 2009,
http://download.intel.com/design/flash/nand/extreme/ITJ_03_SSDs_for_Embedded.pdf
(accessed March 2, 2010).
Park, Oskar. “What is an Optical Disk?." Self SEO (Search Engine Optimization), September
10, 2006, http://www.selfseo.com/story-18894.php (accessed March 11, 2010).
Schulz, Greg. Resilient Storage Networks: Designing Flexible Scalable Data Infrastructures.
Massachusetts: Elsevier Digital Press, 2004.
Shrivastava, Alok, and Somasundaram, G., eds. Information Storage and Management: Storing,
Managing, and Protecting Digital Information. Indiana: Wiley Publishing Incorporated,
2009.
55
Simitci, Huseyin. Storage Network Performance Analysis. Indiana: Wiley Publishing, Inc.,
2003.
Symes, Peter. Digital Video Compression. New York: McGraw-Hill, 2004.
Thatcher, Jonathan. “NAND Flash Solid State Storage Reliability and Data Integrity: An In-
Depth Look.” Storage Networking Industry Association (SNIA) Solid State Storage
Initiative, 2009,
http://www.snia.org/forums/sssi/knowledge/education/Solid_State_Storage_Reliability_a
nd_Data_Integrity--An_In-depth_Look.pdf (accessed March 13, 2010).
The Free Dictionary. “Flash Memory.”
http://encyclopedia2.thefreedictionary.com/flash+memory (accessed March 2, 2010).
The Free On-line Dictionary of Computing. "Average Seek Time."
http://encyclopedia2.thefreedictionary.com/average+seek+time (accessed March 12
2010).
The Oxford Essential Dictionary of the U.S. Military. “Tactical Operations.” Answers.com.
The Oxford Essential Dictionary of the U.S. Military, Oxford University Press, 2001,
2002, http://www.answers.com/topic/tactical-operations (accessed March 07, 2010).
Tudor, Marius. “Are Flash Solid-State Disk Drives Ready for the Enterprise?” Bit Micro
Networks, Incorporated, 2009,
http://www.bitmicro.com/press_resources_flash_ssd_enterprise.php (accessed March 2,
2010).
Wikipedia contributors. "Audio Video Interleave." Wikipedia, The Free Encyclopedia,
http://en.wikipedia.org/w/index.php?title=Audio_Video_Interleave&oldid=347331467
(accessed March 10, 2010).
Wikipedia contributors. "Blu-ray Disc.” Wikipedia, The Free Encyclopedia,
http://en.wikipedia.org/w/index.php?title=Blu-ray_Disc&oldid=349002833 (accessed
March 12, 2010).
Wikipedia contributors. "Blu-ray Disc recordable." Wikipedia, The Free Encyclopedia,
http://en.wikipedia.org/w/index.php?title=Blu-ray_Disc_recordable&oldid=346269932
(accessed March 12, 2010).
Wikipedia contributors. "Lossless data compression.” Wikipedia, The Free Encyclopedia,
http://en.wikipedia.org/w/index.php?title=Lossless_data_compression&oldid=346709055
(accessed March 10, 2010).
Wikipedia contributors. "Lossy compression.” Wikipedia, The Free Encyclopedia,
http://en.wikipedia.org/w/index.php?title=Lossy_compression&oldid=347940008
(accessed March 10, 2010).
56
Wikipedia contributors. "Magneto-optical drive." Wikipedia, The Free Encyclopedia,
http://en.wikipedia.org/w/index.php?title=Magneto-optical_drive&oldid=338693032
(accessed March 12, 2010).