Basic Principles of Digitisation

of 48 /48
30 October 2014 SIPAD Workshop Basic Principles of Digitisation – of audio, video and film Richard Wright SOIMA-LATAM, Mexico City, October 2014

Embed Size (px)

Transcript of Basic Principles of Digitisation

  • 30 October 2014SIPAD WorkshopBasic Principles of Digitisation of audio, video and filmRichard WrightSOIMA-LATAM, Mexico City, October 2014

    SIPAD Workshop

  • ReviewAudio and Video are signalsAudiovisual Archives hold recordings of signalsA signal is described by two attributes:Frequency range = bandwidthDynamic range = signal-to-noise ratioPure tones = only one frequency (tuning fork)All other sounds have a spectrum of frequenciesSpeech: 100 Hz to 10 000 Hz (= 10 kHz)

    30 October 2014SIPAD Workshop

    SIPAD Workshop

  • ReviewAll other sounds have a spectrum of frequenciesSpeech: 100 Hz to 10 000 Hz (= 10 kHz)Music: 30 Hz to above 18 000 HzHuman hearing (normal): 50 Hz to 18 kHzTelephone bandwidth: 300 Hz to 3.3 kHz (old)78 rpm recording (shellac): up to 8 kHz33 1/3 (vinyl; LP): up to 15 kHz (hi-fi)Audio tape (1/4): up to 18 kHz (at 15 ips; many different kinds of tape, and bandwidth varies with recording speed) CD: theoretical max= 22 kHz = (44.1)/2; practical max = 17.6 kHz = (44.1)/(2.5)

    30 October 2014SIPAD Workshop

    SIPAD Workshop

  • Dynamic RangeSpeech: typically 30 to 50 dB; max about 70 dBMusic: could be 70 dB in concert hallCould be 90 dB in a very quiet studioHuman hearing: 100 dB without risk of damage; 130 dB maxOld telephones: 20 to 30 dB78 rpm shellac recording: 40 to 50 dBSound on film: typically 40 to 50 dB33 1/3 rpm vinyl LP: 60 to 70 dB audio tape: just over 70 dB (best = 73 dB)CD: capable of over 90 dB range (96 dB in theory)30 October 2014SIPAD Workshop

    SIPAD Workshop

  • *The need for digitisation

    Preservation planningmapping your collections;setting priorities;making a collection strategy,a preservationstrategyand a preservation plan

    The Preservation Factory approach Conservation, Digitisation and Preservation

  • TerminologyPreservation Everything needed to ensure permanent access = maintenanceConservation: keeping what you have (for as long as you can) = safe storage and handlingPreservation actions: interventions. Changing what you have = repair and replaceMaking a new negative or internegCopying from an old carrier to a new carrierDigitisingWhich separates content from carrier = liberation*

  • Conservationpackaging, handling and shelvingenvironmental conditions: temperature and humidity; protection from pollution, dirt etcprotecting the masters; andcondition monitoring = checking the stockFull description on Preservation Guide wiki: http://wiki.prestospace.org/pmwiki.php?n=Main.PreservationStrategy#Conservation *

  • *The need for digitisationOnly three ways to preserve:Keep what you haveCopy using same technologyCopy using new technologyKeep what you have: only works for filmAudio, Video: analogue technology obsoleteCopy on same technology: only works for filmAudio, Video: analogue technology obsolete

  • *What about vinyl?Gramophone records: lacquer, shellac and vinylLacquer = a master recording (acetate, instantaneous disc); very fragile !

  • *More on gramophone recordsShellac was used for 78 rpm commercial recordingsAlso fragile the main risk is handling; the shellac itself is stable (compared to laquer)

  • *But, what about vinyl?Used for 45 and 33 1/3 rpm recordingsThey are fragile (though less fragile than shellac)Warp from heat (can be fixed with care)Easily scratched, and can get very dirtyThe groove can be damaged in playback (if the needle and tone arm adjustment is not right)Vinyl is soft !!!Vinyl and shellac can suffer chemical damage

  • Vietnam Film Institute Workshops*Gramophone damageDropping a needle onto a vinyl disc

    Oil coming to the surface on a lacquer disc

    Vietnam Film Institute Workshops

  • *Where was I?Only three ways to preserve:Keep what you have: only works for filmAudio, Video: analogue technology obsoleteCopy on same technology:only works for filmAudio, Video: analogue technology obsoleteSo we are left with only one option (for audio and video):Copy using new technology

  • *Preservation of FilmMany excellent example of conservationFilm technology still exists (but in decline)BUT the situation is rapidly changingKodak in severe economic troubleCommercial cinemas changing to digital projectionNorway changed completely in 2011-2012Commercial cinema will change or go out of businessCommercial cinema will not keep old projectors

  • *The end of film projection? We're About to Lose 1,000 Small Theaters That Can't Convert to Digital. Does It Matter? Indiewire, USA 2012

  • *Digitisation of FilmNow: needed for accessInternetDigital cinemaSoon: needed for preservationWhen manufacture of film stock is stoppedResult: ALL media needs to be digitised, audio and video and filmThat's a lot of content that has to be digitisedThere isn't enough timeThere isn't enough money

  • *PRESTO DigitisationBetter Faster Cheaper Daniel Teruggi, Institut National de l'Audiovisuel (INA), ParisBasic concepts:1) Dealing with the whole collection2) Developing a strategyFor the institution and its collectionFor the preservation work3) Developing a preservation plan

  • *Digitisation PlanningAll the Presto information on a wiki: www.preservationguide.co.uk/RDWiki/pmwiki.phpStrategy what does your institution do?What does the collection do?What can digitisation do?Planning making a preservation planHow to estimate a budgetBuilding a business case

  • *Collection StrategyLong-term purpose of the digitisationPreserving the collectionMaintaining services (old business)Creating new services (new business)Physical Outcomes:Digital filesMass storageCheaper, better maintenanceCheaper, better digital access copiesWeb access

  • *Access OutcomesCollection access from computer stationsMediatheque approachCollection access via the InternetWith restricted accessBritish Film Institute: higher education, public libraries and on YouTubeBritish Library Sound Archive: higher education institutions onlyOr even unrestricted accessINA has 30 000 hours of broadcast content online !

  • *Restrictions on AccessAttitudes: we don't do thatFear: somebody might complainRights: we don't hold the rightsA solution: 1) public institutions create public value by opening their collections as widely as possible2) non-fiction content has the most information and the least rights problemsWe have a public service obligation to create access to our collections !

  • *Access:Technical RequirementsA usable catalogueOnline support services: questions, linking to online sales, dealing with faultsLogins, access control, authentication, data protection Computing power to support what could be large numbers of people requiring accessUK National Archive crashed; Europeana crashedManaging lots of information technology

  • *Digital Archive: workflowEverything changes when shelves disappear:Acquisition: digital ingestCataloguing: complex metadata issues: embedded metadata, preservation metadata, mappingCuration: huge opportunity to create online collectionsResearch: self-research, fast scanning of 'hits' changes the requirements of cataloguingAccess: requirements need to be built into the digitisation workflow (access copies, public metadata, rights clearance, censorship?)

  • *Getting StartedMap The collection needs to be divided by format, condition, purpose ...Priorities arrange the areas on the map according to what needs to be done first

    wiki:Make a Map of your Collection Divide the collection by physical formats, and collect the following information on each format: age range storage history genre or value physical conditionGetting Started

  • *BBC film examples from wikiMap: http://www.preservationguide.co.uk/RDWiki/pmwiki.php?n=Main.GettingStarted Strategy: http://www.preservationguide.co.uk/RDWiki/pmwiki.php?n=Main.PreservationStrategyPlan: http://www.preservationguide.co.uk/RDWiki/pmwiki.php?n=Main.PresPlan Also: tutorials on PrestoCentre website collection strategy and preservation plans_V1.01_0.pdf mapping your collection_V1.01_0.pdf

  • Collection Strategy & PlanPDFs of wiki chaptersCollection StrategyPreservation StrategyPreservation Plan and BudgetCollection Strategy Preservation StrategyPreservation Plan and BudgetTutorials on PrestoCentrecollection strategy and preservation plans_V1.01_0.pdfmapping your collection_V1.01_0.pdf

    30 October 2014SIPAD Workshop

    SIPAD Workshop

  • *The Factory ApproachThe division of labour in pin manufacturing:(and the great increase in the quantity of work that results)

  • The division of labour in pin manufacturing: (and the great increase in the quantity of work that results)Approach: NOT about cutting corners or reducing qualityInstead: about cutting wasted time and wasted effortBatches: doing one thing at a time, and then doing it again (and again, and again)Problem: how to keep the work interesting!

    *The Factory Approach

  • *A BBC Preservation Factory

  • *The Factory ApproachDivision of labour requires having specialists, which means having a teamWhich means having a middle-sized or larger projectBut: even a single person can use a batch approach, and get some efficiencies

  • BBC Preservation: 5M per year (for 20 years)Audio: 7000 hours per year of inch tapeAlso digitised 63 000 sepmag elements: separate magnetic sound tracks for film;Transferred to CD (good!) and to polyester (bad!)Also 35 000 shellac and vinyl recordingsAlso DAT and MD digital recordings2 videotape: 46 000 tapes, transferred to D3 or Digibeta, done over 7 years1 videotape: 80 000 in 5 yrs, to D3/Digibeta U-matic: 40 000 in 3 yrs, to MPEG-2 files

  • *News film: 43 000 items in 3 yearsEktachrome; digitised to SD video (Digibeta)10% of total cost is cataloguing20 to 30% is quality control (checking)Since 2008: transfer 40k D3 videotape to files:Uncompressed; MXF wrapper = INGEXNow: transfer of BetaSP and Digibeta to filesMajor confusion over High Definition formats !

  • *DiscussionDigitisation projects of delegatesAudio, video, filmWhich physical formats (eg -inch audio tape, 1-inch video tape, 16mm B&W film ) ?Size: how much content, how many people, how many years ?

  • *Time for a breakAfter the break a choice !Low-level: standards and best practice for audio, video and filmHigher level: case studies for preservation and access3) Your questions on digitisationyour long-term plan?your immediate problems?

  • 30 October 2014SIPAD Workshop

    SIPAD Workshop

  • Digitisation Issues:Audio and Video: no problems at all with the digitisation technology the problems are with:Playback of originalsStandards: compression, formats

  • Audio holdings (TAPE data)

    Carrier%wax cylinders0.03coarse groove replicated disks (78s,shellacs)1.9instantaneous disks of any kind0.2microgroove disks (LPs)17.5open reel magnetic tape38.6compact cassettes20.6R-DAT2.5replicated CDs, DVDs16.5recordable and rewritable CDs, DVDs1.4Minidisks0.4Other0.3Total100

  • Audio: major problemsWax cylinders: condition, equipmentLacquer discs: conditionSticky shed on 6mm tape playbackDAT recordings: technical standards, condition, equipmentCD ripping common issue for digital-digital transfer: error concealment (and what to do about it)IASA TC-04: Guidelines on the Production and Preservation of Digital Audio Objects, March 2009.BWAV, 96kHz, 24bit; BBC do 44.1kHz, 16bit

  • How to digitise audioIASA TC-0448 kHz, 24 bit96 kHz recommendedSave as a wav fileBroadcast Wave Format recommended

  • Film: major problemsNo problem with playback equipment !Some film can last centuriesSome film already destroyedvinegar syndrome; colour fade; moldTelecine out, datacine in

  • Datacine for film digitisationBatch processHD or better for 16mmMany archives doing 2K scanningNo grading shot-by-shotso a higher dynamic range is needed: 14 bits or betterNeeds sprocketless transport to cope with fragile film, shrinkage, splicesInfrared technology for scratch detectionWet-gate for scratch concealment (on one side)Several new sprocketless systems, 80 to 240k

  • How to Digitise FilmB&G document on PrestoCentrehttp://www.prestocentre.eu/sites/dev.prestocentre.antenna.nl/files/filmscanning_v1.03.pdf

    White Paper: Film scanning considerationsBy Tom de Smet and Harm Jan Triemstra

  • Sound on FilmSuccessful digital processing of optical sound tracks now very well established by Chase Audio, HollywoodBut expensiveAnalogue capture most cost-effective if sound track in reasonable conditionMuch TV archive material has separate sound tracks: sepopt, sepmag (beware vinegar!)

  • Video: major problemsAs for audio: equipment, operators, fragile and decaying media; sticky-shedPlus: time base correction and dropout concealment not built into old equipmentColour: composite signals and their decoding; BBC: a composite original is PAL-decoded and saved as component, even though the original is compositeHead wear, head life, head replacement alternative technology for reading videotape has been propose

  • Video digitisation issuesWhat codec, what file format?Uncompressed, lossless, lossyLossless: JPEG2000, FFV1 (many others) MOV, AVI, MXF ... (cf LOC FADGI case studies)Endgame: uncompressed (no codec !)BBC: saving uncompressed from D3 transfers to files: INGEX open-source software (as used by Tate Galleries, London)

  • How to digitise videoSave the originalDigitise @ SDI = 4:2:2 = 200 megabits/secSave exactly as digitised = uncompressedUse an open source file format (MXF or ?)

    3b or lossless = lossless JPEG2000, 3c or mezzanine = high-quality lossy (but only for low quality originals)(digibeta is a mezzanine format)

  • Common issue: quality controlComputer processing existsin professional audio equipment: Quadrega, NOA fault detection technology from restoration workJoanneum video problem detector = VidiCertAudio Inspector https://www.audioinspector.com/New QC software from Dave Rice, BAVC http://www.bavc.org/qctools Key issues: workflow integration, statistics

  • How to Digitise FilmB&G document on PrestoCentrehttp://www.prestocentre.eu/sites/dev.prestocentre.antenna.nl/files/filmscanning_v1.03.pdf

    White Paper: Film scanning considerationsBy Tom de Smet and Harm Jan Triemstra

    ***Wright DPC report, p 9: Conservation of analogue content can be divided into the following concerns.packaging, handling and shelving (storing): the immediate environment of a physical item;environmental conditions: essentially control of temperature and humidity, and the stability of that control, but also protection from anything harmful in the environment, such as dust, pollutants, magnetic fields, excess light or infrared and ultraviolet radiation;protecting the masters; and condition monitoring.

    **http://www.unesco.org/webworld/virtual_exhibit/safeguarding/film.gif

    http://www.unesco.org/webworld/virtual_exhibit/safeguarding/texte10.html

    Before magnetic tapes came into general use, recordings were made on so called instantaneous records, which could be replayed immediately after the cut. The most popular form was the lacquer disk, or acetate disk used from the 1930s to the 1950s in broadcasting and scientific institutions for unique recordings. Chemical decomposition (hydrolysis) causes the information carrying lacquer layer to become brittle and thus to separate from the metal or glass core of the record. Of an internationally assessed total of three million records, many have already been lost, including numerous unique interviews and speeches from historically important politicians and artists. Photo: Phonogrammarchiv, Austrian Academy of Sciences ***http://www.theaudioarchive.com/TAA_Resources_Disc_Transcription.htm

    ********BL Sound Archive access:http://www.bl.uk/reshelp/bldept/soundarch/about/soundarchive.html

    Br Film Inst access: Cinema, IMAX, MediathequeSells DVDs and publications

    BFI Inview: http://www.jisc-collections.ac.uk/Catalogue/Overview/index/24

    BFI Online sales: http://filmstore.bfi.org.uk/

    Other online: http://www.bfi.org.uk/Archive-collections/introduction-bfi-collections/archive-resources-online******http://www.flickr.com/photos/tobanblack/3433957359/License Some rights reserved by Toban BlackThe division of labour in pin manufacturing:(and the great increase in the quantity of work that results)

    *********************