Moving media storage technologies : applications & workflows for … · 2011-10-18 · Metadata 388...
Transcript of Moving media storage technologies : applications & workflows for … · 2011-10-18 · Metadata 388...
Moving Media StorageTechnologiesApplications & Workflows
for Video and Media Server
Platforms
Karl Paulsen
CONTENTS
Acknowledgments xv
Introduction xvii
Chapter 1 Information, Data, and Video 1
Data Structure 1Information 2
Storing the Moving Image 3
Digital Video Recording 4
Analog Component Video 5
Colorimetry and Conversion 11A Digital Media World: Going Forward 12Further Readings 13
Chapter 2 Digital Disk Recording 15
Recording Moving Images 16Dawn of the Video Disk 17Fundamentals of Digital Disk Recording 18Core Technologies for Digital Disk Recording 21
DDR, DVR, or VideoServer 28
Comparing DDRs to VideoServers 31
Clip Servers 32DVR Marketplaces 35Home Media Network Disk Recorders 36Audio 37
Further Readings 38
Chapter 3 Video Media Servers 39
Carrying Over from the DDR 39Universal Server Approach 41
VideoServers as Media Servers 46The Future of VideoServers 66
Communications Protocols Used in Video Serving Platforms 66
Further Readings 71
Chapter 4 Hard Drive Anatomy 73
Magnetic Recording History 74
Drive Components 77
Disk Performance Optimization 85
Disk Scheduling 85
Access Time Components 87
Servicing and Scheduling Criteria 88
Capacity Factors, Transfer Time and Obstacles 95Read-Write Heads 97
Advanced Format Sector Technology 102
Superparamagnetic Limit 106
Heat-Assisted Magnetic Recording. 107
Further Reading 108
Chapter 5 Buses, Ports, and Interfaces 109
Carrying and Transferring Data 110
Bus Types 110
Storage Bus Standards 112
Storage Module Device 112
Serial Storage Architecture 114
High-Performance Parallel Interface 115
IDE 116
Parallel ATA 116
SCSI 117
Further Readings 124
Chapter 6 SATA, SAS, and RAID 125
Transitioning from Parallel to Serial Technologies 126Serial Attached SCSI (SAS) 127SATA 133RAID 139Alternative Disk Sets and Configurations 155Error Prevention Methods 159
Conclusions 163Further Readings 163
Chapter 7 Solid State Disks 165
Solid State Storage Evolution 166
Dynamic Random Access Memory 167
Flash Memory 167
Flash Memory Functionality 168
Portability and Green Perspectives 182
Supplementing Video on Demand with SSD 183
Further Readings 185
Chapter 8 Optical Storage Media 187
Defining Removable Media 188
Optical Media Definitions 188
Blu-rayDisc 195
Holographic Storage 202
Care and Handling 210
Further Readings 211
Chapter 9 Interchange, Transcoding, and Wrappers 213
Moving Toward Non-Real Time 214
Compressed Video 216
Issues with Interchange 217
Media File Vocabularies and Terminologies 218
Content 229
Material exchange Format (MXF) 229
Task Force for MXF 230
Operational Patterns 233
Specialized Operational Pattern Atom 234
Items and Packages 236
Essence Container 238Conversion and Transcoding 239
Converting Audio and Video Essence 240
Transcode Pipeline 244
Multiplatform Conversion and Distribution 246
Examples and Conclusions 252
Further Readings 253
Chapter 10 IP Storage 255
Internet Protocol 255Delivering Packets by IP 256Connected or Connectionless 256IP Address Classes 257Addresses
257
Domain Naming 258Ethernet 259Internet Engineering Task Force 262IP Storage 262iSCSI 263iSCSI RFCs 264iSCSI Protocol 264
Initiator-Target Messages 264Transfer Directions 265
Network Address Authority (NAA) . 265iFCP 265
Gateway Protocol to TCP/IP 266Provisions 266Fabric 266iFCP Services 268
Transport Services 268
Other 268
Security 268
Fibre Channel Over IP (FCIP) 269
Complementary 270
Nominal Distances 270
FCIP Link 271
Clarification 272
Internet Storage Name Service (iSNS) 272
iSNS Architectural Components 273
iSNS Protocol (iSNSP) 273
iSNS Client 273
iSNS servers 274
iSNS Database 274
What iSNS Does 275
Unified Discovery and Management 276
Fiber Channel Discovery 277
Zoning 277
Using SCN Functions 277Distributed Intelligence 278IP Storage Support 278
Quality of Service for IP Storage Networks 278Class of Service (CoS) 278Traffic Prioritization 279
TOS Precedence 279Differentiated Services 279
Performance Predictability 280Traffic Shaping 280
Theory and Research 280Variable Delay 281
Throttling Decisions 281Resource Reservation Protocol (RSVP) 282
Multiprotocol Label Switching (MPLS) 283MPLS Transport Profile 284
InfiniBand 285IP Storage Considerations 290IP Switches and Routers 291
Determining the Need 291
Logical Separation Points 291
Security Considerations 292Ports and Connectivity 292Other Features 293
Management Tools 293Acceleration and Compression 294
Security 294
Authentication 294
Fibre Channel—Security Protocol (FC-SP) 295
Device Identity 296Authorization 298
Encryption 298
Auditing 298
Integrity Checking 298
Quality of Service (QoS) 298Further Readings 299
Chapter 11 Object-Based Storage 301
Introduction: Objects 302
Storage Categories 307
Storage Interface Progression 310Parallel Heading 313
Applications of OSD 321Further Readings 325
Chapter 12 Archives, Backups, and Linear Tape 327
Organizational Requirements for Data 328
Legal Requirements for Archive or Backup 329
Repacking and Repurposing 330Added Operational Requirements 331
Backup or Archive: There are Differences 332
Backups 332
Acting like Archives 333Data Backups—Not Archives 333Internal Application-Specific Backups 334Archive and Transfer Management—Nonlinear Editing 335Differentiation 335
Archiving for Discovery 336Data Archiving 337
Tape or Disk 338Pros and Cons 339
Self-Protecting Storage Systems 340Disk Archiving Pluses 340
Tape Archiving Pluses 341
Storage Retention Period 341
Disadvantages of Tape 341
Legacy Tape Archive Problems 342
Standards-Based Archive Approaches 342Common Backup Formats 343
Determining the Right Archiving Method 343Cloud Data Protection 344Cloud Benefits 344
Scaling 345Services Provided 345
Holographic Storage for the Archive 345RAIT 347
Mirroring—Level 1 RAIT 349Data Striping—Level 3 RAIT 349
Data Striping—Level 5 RAIT 349
Not for Random Access 349
Automated Tape Libraries (ATL) 350
Stackers, Autoloaders, or Libraries 351
Server and Archive Components 351
Evolution in Digital Tape Archives 352
Tape Technology Basics 353
Tape Metrics 353Linear Serpentine 354
Helical Scan Recording 355
Historical Development of Data Tape 355
QIC and DAT 356
Travan 356
8 mm Recording Technologies 356
Tape Error Detection 358
Tape Media Formats and Fabrication Parameters 359
Metal Particle to Advanced Metal Evaporated 359
Advanced Intelligent Tape (AIT) 360
Super and Turbo AIT 361
Digital Data Storage (DDS) 362
Data Storage Technology (DST) 362
Digital Tape Format (DTF) 363
Advanced Tape Materials 364
Digital Linear Tape (DLT) 365
Error Detection- 365
Adaptive Cache Buffering 366
Compression Algorithms 366
Duty Cycle 366
DLT1 Drive 367
Super DLT 367
RoadmapsforSDLT 368
Linear Tape Open (LTO) - 369
Genealogy 369
Ultrium—First Generation 370
LTO Generation 1 370
LTO Generation 2 370LTO Generation 3 370LTO Generation 4 371LTO Generation 5 371LTFS 371
LTO Program Roadmap 372
Security WORM 373
Why Use Tape? 373Data Archiving and Tape Storage 375
Preparing for the Archive 376Future Standards 377
Chapter 13 Metadata and Media Asset Management 379
Media Management Systems 380Media Assets and Content 380
Content Management System 381
Web-Based CMS 382
Digital Asset Management 382
Asset Management System 383
Media Asset Management 384
Tape-Based MAM 384
Drawbacks to Tape-Based Systems 386
MAM for VideoServer Implementations 387Lack of Media Asset Integration 388
Collaboration Issues 388Metadata 388
Media-Centric Metadata 389
Structural Metadata 391
Descriptive Metadata 391
Interoperability 393Dublin Core 393
PBCore 394
Usage Metadata 396
Schemas 396
Standards, Practices, and Policies for Metadata 397
Storing Metadata 397
Metadata Services 398
Media Application Services 398
Messaging Services for MAM 400
Intelligent Integrated MAM 401
Workflow Management 402
Collaboration among Multiple Users 402
Tool Set Integration 402
Centralized Database 403
Multi-Tier Architecture 403
Scalability 404
Storage for MAM 404
Latency 405
Interfacing with Production Storage 405
Archive 407
Further Readings 408
Chapter 14 Clustering 409
Why Use Clusters? 409
Physical Components 411
Capabilities and Types of Clustering 412
Cluster Scaling Characteristics 415
Shared Structures 416
Managing Storage Clusters for Scalability 420
Grid Computing 422
Summary 428
Further Readings 428
Chapter 15 Networked Storage .. 429
Networked Storage—Fundamentals 430Networks 436
Network-Attached Storage 442
Direct Access File System 443
Storage Area Network 444
Fibre Channel SAN Components 460
Next Generation SANs 467
Further Readings 467
Chapter 16 High-Performance and Shared Storage 469
Applications for High-Performance Storage 470
Size Matters 474
Intelligent Adaptive Storage 475
Storage Accelerators 476
Storage Appliance Technologies 477
Real-World Performance 490
Shared Storage Implementations 498
Storage Scaling and Management 510
Further Readings 510
Chapter 17 Scalability, Extensibility, and Interoperability 511
Scalability 511
Flexibility 516
Reliability 520
Accessibility and Availability 526
Extensibility 533
Interoperability 539
Science and Technology 541
Further Readings 541
Chapter 18 Storage Management 545
Storage System Dependencies 546
Storage Management Concepts 549
Electronic Media Management 554
Fragmentation 554
SAN Management 560
NAS Management 564
Snapshots 571
Data Deduplication 574
Further Readings 579
Chapter 19 Resilience, Fault Tolerance, and Virilization 581
Resilience 582
Storage Tiering 583
Protecting Data and Storage 595
Visualization in File-Based Storage 605
Fault Tolerance 610
Conclusion 617
Further Readings 617
Index 619
Companion website: www.elsevierdirect.com/companions/9780240814483