Audio and Video Repositories at Scale - Indiana University’s Media Digitization and Preservation...
-
Upload
avalon-media-system -
Category
Technology
-
view
143 -
download
1
description
Transcript of Audio and Video Repositories at Scale - Indiana University’s Media Digitization and Preservation...
![Page 1: Audio and Video Repositories at Scale - Indiana University’s Media Digitization and Preservation Initiative](https://reader034.fdocuments.in/reader034/viewer/2022051412/548401a75906b5b7158b46f9/html5/thumbnails/1.jpg)
Audio and Video
Repositories at Scale Indiana University’s Media Digitization and Preservation Initiative Jon Dunn
Indiana University Libraries
Open Repositories 2014 Helsinki, Finland
June 12, 2014
![Page 2: Audio and Video Repositories at Scale - Indiana University’s Media Digitization and Preservation Initiative](https://reader034.fdocuments.in/reader034/viewer/2022051412/548401a75906b5b7158b46f9/html5/thumbnails/2.jpg)
THE END IS NEAR!!!
(for physical time-based media objects)
2
![Page 3: Audio and Video Repositories at Scale - Indiana University’s Media Digitization and Preservation Initiative](https://reader034.fdocuments.in/reader034/viewer/2022051412/548401a75906b5b7158b46f9/html5/thumbnails/3.jpg)
Degradation
All analog and physical digital media objects actively degrading
– some catastrophically
3
Vinegar Syndrome Sticky Shed Syndrome
Fungus
Delamination Plasticizer Exudation
Color Fading Curling
Windowing
Shedding Undiagnosed Unplayability Mechanical Issues Shrinkage
Scratches
Breaking
Hydrolysis
Oxidation
Efflourescence
Binder Breakdown Cupping
Crystalline Residue
![Page 4: Audio and Video Repositories at Scale - Indiana University’s Media Digitization and Preservation Initiative](https://reader034.fdocuments.in/reader034/viewer/2022051412/548401a75906b5b7158b46f9/html5/thumbnails/4.jpg)
Degradation
4
![Page 5: Audio and Video Repositories at Scale - Indiana University’s Media Digitization and Preservation Initiative](https://reader034.fdocuments.in/reader034/viewer/2022051412/548401a75906b5b7158b46f9/html5/thumbnails/5.jpg)
Obsolescence
• Media formats • Equipment (playback machines, test
devices) • Repair parts • Playback expertise • Repair expertise • Tools • Supplies
5
![Page 6: Audio and Video Repositories at Scale - Indiana University’s Media Digitization and Preservation Initiative](https://reader034.fdocuments.in/reader034/viewer/2022051412/548401a75906b5b7158b46f9/html5/thumbnails/6.jpg)
How long do we have?
• Short time window in which to migrate or lose forever
• 10-15 years?
6
![Page 7: Audio and Video Repositories at Scale - Indiana University’s Media Digitization and Preservation Initiative](https://reader034.fdocuments.in/reader034/viewer/2022051412/548401a75906b5b7158b46f9/html5/thumbnails/7.jpg)
Indiana University
State university Founded in 1820 8 campuses 110,000 students 18,000 faculty and staff
7
![Page 8: Audio and Video Repositories at Scale - Indiana University’s Media Digitization and Preservation Initiative](https://reader034.fdocuments.in/reader034/viewer/2022051412/548401a75906b5b7158b46f9/html5/thumbnails/8.jpg)
Indiana University Bloomington Flagship IU campus Major research university Strong arts and humanities focus 46,000 students 8500 faculty and staff
8
![Page 9: Audio and Video Repositories at Scale - Indiana University’s Media Digitization and Preservation Initiative](https://reader034.fdocuments.in/reader034/viewer/2022051412/548401a75906b5b7158b46f9/html5/thumbnails/9.jpg)
IU Repository and Digital Library Environment
• Started with audio, 1992 • Repository infrastructure:
• Fedora, Hydra, Dspace • Strong collaboration between
Libraries and University IT Services • Strong enterprise and research
storage infrastructure
9
![Page 10: Audio and Video Repositories at Scale - Indiana University’s Media Digitization and Preservation Initiative](https://reader034.fdocuments.in/reader034/viewer/2022051412/548401a75906b5b7158b46f9/html5/thumbnails/10.jpg)
Context: IU Collections
• School of Music • Archives of Traditional
Music • Kinsey Institute • Moving Image Archive • Lilly Library • University Archives • Athletics • International
collections
10
![Page 11: Audio and Video Repositories at Scale - Indiana University’s Media Digitization and Preservation Initiative](https://reader034.fdocuments.in/reader034/viewer/2022051412/548401a75906b5b7158b46f9/html5/thumbnails/11.jpg)
Sound Directions, 2007
IU, Harvard Best practices for audio preservation digitization and storage
11
![Page 12: Audio and Video Repositories at Scale - Indiana University’s Media Digitization and Preservation Initiative](https://reader034.fdocuments.in/reader034/viewer/2022051412/548401a75906b5b7158b46f9/html5/thumbnails/12.jpg)
Media Preservation Survey, 2008
12
Findings: • 569,000 items • Unique and valuable
collections • 51 physical formats • 80 units at IUB Recommendations: • Digitize within 15-20
years • Form task force to
plan
![Page 13: Audio and Video Repositories at Scale - Indiana University’s Media Digitization and Preservation Initiative](https://reader034.fdocuments.in/reader034/viewer/2022051412/548401a75906b5b7158b46f9/html5/thumbnails/13.jpg)
Meeting the Challenge Report, 2012
13
Recommendations: • Create center to
digitize 300K rare or unique objects within 15 years
• Audio, video for preservation
• Film for access
![Page 14: Audio and Video Repositories at Scale - Indiana University’s Media Digitization and Preservation Initiative](https://reader034.fdocuments.in/reader034/viewer/2022051412/548401a75906b5b7158b46f9/html5/thumbnails/14.jpg)
• Announced by President McRobbie in October 2013
• Goal: Digitize, preserve, provide access to rare and unique audio and video by 2020 – all IU campuses
• Expect to start in 2014
Media Digitization and Preservation Initiative (MDPI), 2013
14
![Page 15: Audio and Video Repositories at Scale - Indiana University’s Media Digitization and Preservation Initiative](https://reader034.fdocuments.in/reader034/viewer/2022051412/548401a75906b5b7158b46f9/html5/thumbnails/15.jpg)
Public-Private Partnership
Memnon Archiving Services Brussels, Belgium
15
![Page 16: Audio and Video Repositories at Scale - Indiana University’s Media Digitization and Preservation Initiative](https://reader034.fdocuments.in/reader034/viewer/2022051412/548401a75906b5b7158b46f9/html5/thumbnails/16.jpg)
Digitization and Preservation Phases
16
Pre-Digitization
Digitization Discovery
and Access
Inventory Catalog Prioritize
Batch & Queue IU Operation
Massive parallel digitization
Digitization of selected unique and highly vulnerable formats
Quality control
Memnon Metadata Rights Issues
Technical aspects
Metadata Technical
infrastructure Ongoing monitoring
and migration
Digital Preservation and
Storage
![Page 17: Audio and Video Repositories at Scale - Indiana University’s Media Digitization and Preservation Initiative](https://reader034.fdocuments.in/reader034/viewer/2022051412/548401a75906b5b7158b46f9/html5/thumbnails/17.jpg)
Bins, Boxes, Barcodes, Batches
17
![Page 18: Audio and Video Repositories at Scale - Indiana University’s Media Digitization and Preservation Initiative](https://reader034.fdocuments.in/reader034/viewer/2022051412/548401a75906b5b7158b46f9/html5/thumbnails/18.jpg)
Workflow for Picking and Packing
18
![Page 19: Audio and Video Repositories at Scale - Indiana University’s Media Digitization and Preservation Initiative](https://reader034.fdocuments.in/reader034/viewer/2022051412/548401a75906b5b7158b46f9/html5/thumbnails/19.jpg)
Batches: Format Based
19
![Page 20: Audio and Video Repositories at Scale - Indiana University’s Media Digitization and Preservation Initiative](https://reader034.fdocuments.in/reader034/viewer/2022051412/548401a75906b5b7158b46f9/html5/thumbnails/20.jpg)
Workflow Support: The POD
20
![Page 21: Audio and Video Repositories at Scale - Indiana University’s Media Digitization and Preservation Initiative](https://reader034.fdocuments.in/reader034/viewer/2022051412/548401a75906b5b7158b46f9/html5/thumbnails/21.jpg)
High Volume
• 1500+ objects per week • Peak of ~12 TB per day of digitization • Total storage requirement: 9 PB
21
![Page 22: Audio and Video Repositories at Scale - Indiana University’s Media Digitization and Preservation Initiative](https://reader034.fdocuments.in/reader034/viewer/2022051412/548401a75906b5b7158b46f9/html5/thumbnails/22.jpg)
Post-Digitization Workflow
22
![Page 23: Audio and Video Repositories at Scale - Indiana University’s Media Digitization and Preservation Initiative](https://reader034.fdocuments.in/reader034/viewer/2022051412/548401a75906b5b7158b46f9/html5/thumbnails/23.jpg)
Post-Digitization Workflow
23
![Page 24: Audio and Video Repositories at Scale - Indiana University’s Media Digitization and Preservation Initiative](https://reader034.fdocuments.in/reader034/viewer/2022051412/548401a75906b5b7158b46f9/html5/thumbnails/24.jpg)
Metadata for Preservation and Access
Descriptive – From MARC when available – EAD finding aids, local inventories
Technical – File and original object characteristics – Checksums
Process History – Digitization and preservation process
Structural – Support relationships between and
navigation within objects
24
![Page 25: Audio and Video Repositories at Scale - Indiana University’s Media Digitization and Preservation Initiative](https://reader034.fdocuments.in/reader034/viewer/2022051412/548401a75906b5b7158b46f9/html5/thumbnails/25.jpg)
IU Scholarly Data Archive (SDA)
• Central storage resource for IU since 1999
• Data replicated between Indianapolis and Bloomington
• Implemented using IBM HPSS software • 40PB IBM TS1140 based capacity • Available from desktop, web, high
speed transfer protocols • Currently holds ~6PB of data in 36
million files
25
![Page 26: Audio and Video Repositories at Scale - Indiana University’s Media Digitization and Preservation Initiative](https://reader034.fdocuments.in/reader034/viewer/2022051412/548401a75906b5b7158b46f9/html5/thumbnails/26.jpg)
Access Technology
• Discovery • IUCAT, Archives Online (EAD Finding
Aids), other environments? • Delivery
• Key requirements: usability, reusability, access control, performance
• Avalon Media System
26
![Page 27: Audio and Video Repositories at Scale - Indiana University’s Media Digitization and Preservation Initiative](https://reader034.fdocuments.in/reader034/viewer/2022051412/548401a75906b5b7158b46f9/html5/thumbnails/27.jpg)
Long-Term Preservation Technology
• 12 petabytes+ to be preserved • Local storage
• UITS Scholarly Data Archive • Fedora 4 repository layer
• Out-of-region storage • APTrust, DPN • Data swap agreements
27
![Page 28: Audio and Video Repositories at Scale - Indiana University’s Media Digitization and Preservation Initiative](https://reader034.fdocuments.in/reader034/viewer/2022051412/548401a75906b5b7158b46f9/html5/thumbnails/28.jpg)
MDPI Assets and Opportunities
• Collaborative planning process – Libraries, UITS, archives, faculty
• Research storage and IT resources – 40 petabyte mirrored HSM system – Experience with data-intensive workflows – High-performance networks
• Expertise in digitization, repositories, access systems – Variations, Avalon Media System
• Potential for external services
28
![Page 29: Audio and Video Repositories at Scale - Indiana University’s Media Digitization and Preservation Initiative](https://reader034.fdocuments.in/reader034/viewer/2022051412/548401a75906b5b7158b46f9/html5/thumbnails/29.jpg)
MDPI Challenges
• Dealing with rights issues at scale • Descriptive metadata and discovery • Quality control strategies for mass
digitization • Strategies for born-digital media • Out-of-region preservation storage • Approach for film
29
![Page 30: Audio and Video Repositories at Scale - Indiana University’s Media Digitization and Preservation Initiative](https://reader034.fdocuments.in/reader034/viewer/2022051412/548401a75906b5b7158b46f9/html5/thumbnails/30.jpg)
Stay Tuned…
mdpi.iu.edu
avalonmediasystem.org
30