Download - Basic Principles of Digitisation

30 October 2014 SIPAD Workshop

Basic Principles of Digitisation – of audio, video and film

Richard WrightSOIMA-LATAM, Mexico City, October 2014

Review

Audio and Video are signalsAudiovisual Archives hold recordings of signalsA signal is described by two attributes:• Frequency range = bandwidth• Dynamic range = signal-to-noise ratioPure tones = only one frequency (tuning fork)All other sounds have a spectrum of frequenciesSpeech: 100 Hz to 10 000 Hz (= 10 kHz)


Review

All other sounds have a spectrum of frequencies• Speech: 100 Hz to 10 000 Hz (= 10 kHz)• Music: 30 Hz to above 18 000 Hz• Human hearing (normal): 50 Hz to 18 kHz• Telephone bandwidth: 300 Hz to 3.3 kHz (old)• 78 rpm recording (shellac): up to 8 kHz• 33 1/3 (vinyl; LP): up to 15 kHz (hi-fi)• Audio tape (1/4”): up to 18 kHz (at 15 ips; many

different kinds of tape, and bandwidth varies with recording speed)

• CD: theoretical max= 22 kHz = (44.1)/2; practical max = 17.6 kHz = (44.1)/(2.5)


Dynamic Range

• Speech: typically 30 to 50 dB; max about 70 dB• Music: could be 70 dB in concert hall

– Could be 90 dB in a very quiet studio

• Human hearing: 100 dB without risk of damage; 130 dB max

• Old telephones: 20 to 30 dB• 78 rpm shellac recording: 40 to 50 dB• Sound on film: typically 40 to 50 dB• 33 1/3 rpm vinyl LP: 60 to 70 dB• ¼” audio tape: just over 70 dB (best = 73 dB)• CD: capable of over 90 dB range (96 dB in theory)


5

The need for digitisation

Preservation planningmapping your collections;setting priorities;making a collection strategy,a preservation strategyand a preservation plan

The Preservation Factory approach

Conservation, Digitisation and Preservation

Terminology

Preservation – “Everything needed to ensure permanent access” = maintenance Conservation: keeping what you have (for as long

as you can) = safe storage and handling Preservation actions: interventions. Changing what

you have = repair and replace− Making a new negative or interneg− Copying from an old carrier to a new carrier− Digitising

Which separates content from carrier = liberation

6

Conservation

packaging, handling and shelving environmental conditions: temperature and

humidity; protection from pollution, dirt etc protecting the masters; and condition monitoring = checking the stock

Full description on Preservation Guide wiki: http://wiki.prestospace.org/pmwiki.php?n=Main.PreservationStrategy#Conservation

7

8

The need for digitisation

Only three ways to preserve:

1.Keep what you have

2.Copy using same technology

3.Copy using new technology

Keep what you have: only works for film• Audio, Video: analogue technology obsolete

Copy on same technology: only works for film• Audio, Video: analogue technology obsolete

9

What about vinyl?

Gramophone records: lacquer, shellac and vinyl Lacquer = a master

recording (acetate, instantaneous disc); very fragile !

10

More on gramophone records

Shellac was used for 78 rpm commercial recordings

Also fragile – the main risk is handling; the shellac itself is stable (compared to laquer)

11

But, what about vinyl?

Used for 45 and 33 1/3 rpm recordings They are fragile (though less fragile than

shellac) Warp from heat (can be fixed with care) Easily scratched, and can get very dirty The groove can be damaged in playback (if the

needle and tone arm adjustment is not right) Vinyl is soft !!! Vinyl and shellac can suffer chemical damage

Vietnam Film Institute Workshops 12

Gramophone damage

Dropping a needle onto a vinyl disc

Oil coming to the surface on a lacquer disc

13

Where was I?

Only three ways to preserve:

Keep what you have: only works for film• Audio, Video: analogue technology obsolete

Copy on same technology:only works for film• Audio, Video: analogue technology obsolete

So we are left with only one option (for audio and video):

Copy using new technology

14

Preservation of Film

Many excellent example of conservation Film technology still exists (but in decline) BUT – the situation is rapidly changing

Kodak in severe economic trouble Commercial cinemas changing to digital projection

− Norway changed completely in 2011-2012

Commercial cinema will change or go out of business

Commercial cinema will not keep old projectors

15

The end of film projection?

“ We're About to Lose 1,000 Small Theaters That Can't Convert to Digital. Does It Matter?”Indiewire, USA 2012

16

Digitisation of Film

Now: needed for access Internet Digital cinema

Soon: needed for preservation When manufacture of film stock is stopped

Result: ALL media needs to be digitised, audio and video and film That's a lot of content that has to be digitised There isn't enough time There isn't enough money

17

PRESTO Digitisation

“Better Faster Cheaper” Daniel Teruggi, Institut National de l'Audiovisuel (INA), Paris

Basic concepts:

1) Dealing with the whole collection

2) Developing a strategy For the institution and its collection For the preservation work

3) Developing a preservation plan

18

Digitisation Planning

All the Presto information on a wiki: www.preservationguide.co.uk/RDWiki/pmwiki.php

StrategyStrategy – what does your institution do? What does the collection do? What can digitisation do?

PlanningPlanning – making a preservation plan How to estimate a budget Building a business case

19

Collection Strategy

Long-term purpose of the digitisation Preserving the collection Maintaining services (old business) Creating new services (new business)

Physical Outcomes: Digital files Mass storage Cheaper, better maintenance Cheaper, better digital access copies Web access

20

Access Outcomes

Collection access from computer stations Mediatheque approach

Collection access via the Internet With restricted access

− British Film Institute: higher education, public libraries … and on YouTube

− British Library Sound Archive: higher education institutions only

Or even unrestricted access− INA has 30 000 hours of broadcast content online !

21

Restrictions on Access

Attitudes: we don't do that Fear: somebody might complain Rights: we don't hold the rights

A solution:

1) public institutions create public value by opening their collections as widely as possible

2) non-fiction content has the most information and the least rights problems

We have a public service obligation to create access to our collections !

22

Access:Technical Requirements

A usable catalogue Online support services: questions, linking to

online sales, dealing with faults Logins, access control, authentication, data

protection … Computing power to support what could be

large numbers of people requiring access UK National Archive crashed; Europeana crashed

Managing lots of information technology

23

Digital Archive: workflow

Everything changes when shelves disappear: Acquisition: digital ingest Cataloguing: complex metadata issues: embedded

metadata, preservation metadata, mapping Curation: huge opportunity to create online

collections Research: self-research, fast scanning of 'hits' –

changes the requirements of cataloguing Access: requirements need to be built into the

digitisation workflow (access copies, public metadata, rights clearance, censorship?)

24

Getting Started

Map – The collection needs to be divided by format, condition, purpose ...

Priorities – arrange the “areas on the map” according to what needs to be done first

wiki: Make a Map of your CollectionDivide the collection by physical formats, and collect the following information on each format:

age rangestorage historygenre or valuephysical condition

Getting Started

25

BBC film examples from wiki

Map: http://www.preservationguide.co.uk/RDWiki/pmwiki.php?n=Main.GettingStarted

Strategy: http://www.preservationguide.co.uk/RDWiki/pmwiki.php?n=Main.PreservationStrategy

Plan: http://www.preservationguide.co.uk/RDWiki/pmwiki.php?n=Main.PresPlan

Also: tutorials on PrestoCentre websitecollection strategy and preservation plans_V1.01_0.pdfmapping your collection_V1.01_0.pdf

Collection Strategy & Plan

• PDFs of wiki ‘chapters’

– Collection Strategy

– Preservation Strategy

– Preservation Plan and Budget– Collection Strategy Preservation Strategy

– Preservation Plan and Budget

• Tutorials on PrestoCentre– collection strategy and preservation plans_V1.01_0.pdf

– mapping your collection_V1.01_0.pdf


27

The Factory Approach

The division of labour in pin manufacturing:(and the great increase in the quantity of work that results)

The division of labour in pin manufacturing: (and the great increase in the quantity of work that results)

Approach: NOT about cutting corners or reducing quality

Instead: about cutting wasted time and wasted effort

Batches: doing one thing at a time, and then doing it again (and again, and again)

Problem: how to keep the work interesting! 28


29

A BBC Preservation Factory

30


“Division of labour” requires having specialists, which means having a team

Which means having a middle-sized or larger project

But: even a single person can use a batch approach, and get some efficiencies

BBC Preservation:£5M per year (for 20 years)

Audio: 7000 hours per year of ¼ inch tapeAlso digitised 63 000 “sepmag” elements: separate magnetic sound tracks for film;− Transferred to CD (good!) and to polyester (bad!)

Also 35 000 shellac and vinyl recordingsAlso DAT and MD digital recordings

2” videotape: 46 000 tapes, transferred to D3 or Digibeta, done over 7 years

1” videotape: 80 000 in 5 yrs, to D3/Digibeta U-matic: 40 000 in 3 yrs, to MPEG-2 files

32

News film: 43 000 items in 3 years Ektachrome; digitised to SD video (Digibeta)

10% of total cost is cataloguing 20 to 30% is quality control (checking) Since 2008: transfer 40k D3 videotape to files:

Uncompressed; MXF wrapper = INGEX Now: transfer of BetaSP and Digibeta to files Major confusion over High Definition formats !

33

Discussion

Digitisation projects of delegates Audio, video, film Which physical formats (eg ¼-inch audio tape,

1-inch video tape, 16mm B&W film … ) ? Size: how much content, how many people,

how many years ?

34

Time for a break

After the break – a choice !

1)Low-level: standards and best practice for audio, video and film

2)Higher level: case studies for preservation and access

3) Your questions on digitisation your long-term plan? your immediate problems?

Digitisation Issues:

• Audio and Video: no problems at all with the digitisation technology – the problems are with:– Playback of originals

– Standards: compression, formats

Audio holdings (TAPE data)Carrier %

wax cylinders 0.03

coarse groove replicated disks (‘78s’,‘shellacs’)

1.9

instantaneous disks of any kind 0.2

microgroove disks (LPs) 17.5

open reel magnetic tape 38.6

compact cassettes 20.6

R-DAT 2.5

replicated CDs, DVDs 16.5

recordable and rewritable CDs, DVDs 1.4

Minidisks 0.4

Other 0.3

Total 100

Audio: major problems• Wax cylinders: condition, equipment

• Lacquer discs: condition

• Sticky shed on 6mm tape playback

• DAT recordings: technical standards, condition, equipment

• CD ‘ripping’ – common issue for digital-digital transfer: error concealment (and what to do about it)

• IASA TC-04: Guidelines on the Production and Preservation of Digital Audio Objects, March 2009.

– BWAV, 96kHz, “24bit”; BBC do 44.1kHz, 16bit

How to digitise audio

• IASA TC-04

• 48 kHz, “24 bit”– 96 kHz recommended

• Save as a wav file

• Broadcast Wave Format recommended

Film: major problems

• No problem with playback equipment !

• Some film can last centuries

• Some film already destroyed

vinegar syndrome; colour fade; mold

• Telecine out, datacine in

Datacine for film digitisation

• Batch process• HD or better for 16mm• Many archives doing 2K scanning• No grading shot-by-shot– so a higher dynamic range is needed: 14 bits or better• Needs sprocketless transport to cope with fragile film,

shrinkage, splices• Infrared technology for scratch detection• Wet-gate for scratch concealment (on one side)• Several new sprocketless systems, €80 to 240k

How to Digitise Film

B&G document on PrestoCentre

http://www.prestocentre.eu/sites/dev.prestocentre.antenna.nl/files/filmscanning_v1.03.pdf

White Paper: Film scanning considerations

By Tom de Smet and Harm Jan Triemstra

Sound on Film

• Successful digital processing of optical sound tracks now very well established by Chase Audio, Hollywood– But expensive

• Analogue capture most cost-effective if sound track in reasonable condition

• Much TV archive material has separate sound tracks: sepopt, sepmag (beware vinegar!)

Video: major problems

• As for audio: equipment, operators, fragile and decaying media; sticky-shed

• Plus: time base correction and dropout ‘concealment’ not built into old equipment

• Colour: composite signals and their decoding;

– BBC: a composite original is PAL-decoded and saved as component, even though “the original” is composite

• Head wear, head life, head replacement – alternative technology for reading videotape has been propose

Video digitisation issues

• What codec, what file format?– Uncompressed, lossless, lossy

– Lossless: JPEG2000, FFV1 (many others)

– MOV, AVI, MXF ... (cf LOC FADGI case studies)

• Endgame: uncompressed (no codec !)

• BBC: saving uncompressed from D3 transfers to files: INGEX open-source software (as used by Tate Galleries, London)

How to digitise video

1. Save the original2. Digitise @ SDI = 4:2:2 = 200 megabits/sec3. Save exactly as digitised = uncompressed4. Use an open source file format (MXF or ?)

3b – or lossless = lossless JPEG2000, 3c – or ‘mezzanine’ = high-quality lossy

(but only for low quality originals)(digibeta is a ‘mezzanine’ format)

Common issue: quality control

• Computer processing exists– in professional audio equipment: Quadrega, NOA …

• ‘fault detection’ technology from restoration work

• Joanneum video problem detector = VidiCert

• Audio Inspector https://www.audioinspector.com/

• New QC software from Dave Rice, BAVC http://www.bavc.org/qctools

• Key issues: workflow integration, statistics

How to Digitise Film

• B&G document on PrestoCentre

• http://www.prestocentre.eu/sites/dev.prestocentre.antenna.nl/files/filmscanning_v1.03.pdf

White Paper: Film scanning considerations

By Tom de Smet and Harm Jan Triemstra