What is Good Video? Using data to do better with our content

Anna Chiaretta Lavatelli, Director of Digital Media, Museum of Contemporary Art Chicago (@annachiaretta)Trilce Navarrete, Erasmus University Rotterdam (@trilcenavarrete)Emily Robbins, Web and Digital Assistant, San Francisco Museum of Modern Art (@EmBRobbins) Elena Villaespesa, Digital Analyst, The Metropolitan Museum of Art (@elenustika)

Why Video Analysis?Social media use in museums is growing to connect to the younger

audiences, particularly to attract new audiences.Performance indicators serve as metric to compare with peers. There is no

the performance of an institution: metrics make sense when placed in context.

Yet we lack a harmonized method for measuring success, and for comparing institutional performance across the sector.

We invited the MCN community to participate in a survey to help us identify the current practice which led to this presentation.

Here the results of the survey: … but first a world view of the use of social media.

Increase use of videoInternet video traffic accounts for +70% of all content traffic, which will grow

x4 from 2015 to 2020 (CISCO).Google = #1, YouTube = #2, Facebook = #3, ... Wikipedia = #6 …. (Alexa)YouTube = +1 billion unique visitor per month, 1 billion video views per dayFacebook = +1 billion users, 8 billion video views per day Wikipedia = +15 million visitors per month

Video = information, communication and entertainment.

What do MCN museums do with video?

Survey methodology• Research objectives• Understand video content production strategies• Get an overview of the resources dedicated to video production• Identify the channels used for video distribution• List the main challenges to evaluate video content

• 26 responses• URL distributed among MCN members

Q: Does your museum produce video content?

Video production: 1-100 videos per year

Q: What production positions exist at your institution?

Q: What types of media products does your museum produce?

Q: When you have a new video to upload...

Q: How do you currently distribute video?

Q: What tools do you use to evaluate your videos performance?

Q: What metrics do you report to your team and senior management?

Q: What are your challenges when evaluating video content?

What does this all mean?From the 26 MCN institutions that responded to the survey we can say:

• Museums use major social media as channels for video distribution• #1 = YouTube, #2 = Facebook• # 3 = Vimeo + Instagram, #4 = Twitter

• There is interest in knowing more about the impact of video production• # views is most used metric

• Main challenges:• Limited know-how / resources to support video production / evaluation• Lack of an institutional strategy to guide analysis

• We need a 'how to' guide for video metrics

Quick guide on how to develop your own video production evaluation metrics Strategy of desired goal: What do you want to achieve with video?○Qualitative / Quantitative

Start point to compare after change in practice: Where are you at?○Reach○Engagement○ Impact

Some metrics gathered regularly: How are things changing?○Quantitative / Qualitative

Reflect on your progress: What does this mean?

Eventually, make up your own indicators (e.g. Key Intangible Performance Indicators)

Q: What metrics do you report to your team and senior management?

Reach

Engagement

ImpactUser surveys to see that strategic goal is being met

Reach

Engagement

Reach

Engagement

Impact

Video evaluation framework

Draft benchmark on video production Strategy of desired goal: What do you want to achieve with video?○Qualitative / Quantitative

Start point to compare after change in practice: Where are you at?○Reach○Engagement○ Impact

Some metrics gathered regularly: How are things changing?○Quantitative / Qualitative

Reflect on your progress: What does this mean?

Eventually, make up your own indicators (e.g. Key Intangible Performance Indicators)

Example - The Met (Elena)

Benchmarking on YouTube

Get the YouTube Channel ID1.Developer tools2.Select channel owner name3.Copy code from data-ytid field

Get the YouTube DataYouTube data scraping tool from Digital Methods:

https://tools.digitalmethods.net/netvizz/youtube/mod_videos_list.php

Selecting Benchmark DataWHO?

1. Peer Museumssimilar budget, subject and scale of program

2. Aspirational Museum/Content Producer

LearningsGood practices• Frequency of posts• Short duration• There are exceptions to every rule

Easy wins if popularity is your goal• Celebrities

ExampleGoal: more views and followers

Production goals to date• Capture program content • Create better produced content (edited narratives)

New practices• Post video clips (outtakes)• Timely release of content• Prototype informal content capture

Case Study: Video Production and Analysis

at SFMOMA

@EmBRobbins

BackgroundTwo teams produce three streams of video

Initial strategies were developed in preparation for big opening moment

Now in a moment of reflecting back

Web and Digital Content:

Institutional Storytelling

Interpretive Media:

Artist Interviews

Artist Interviews3-5 minute interviews with artists about their practice or an aspect of their

work

Goals• Provide insightful content to current web/museum audiences• Create a long-lasting public resource• Depth of impact > breadth of reach

Priorities• Artists in the collection• Older artists are prioritized• Relevance of production to on-site programming• Strength of video material

Analytical Process• Alternation between production and analysis phases

• Analytics findings• Video was more popular than other types of content• “Cinematic” videos were more viewed (compared to static talking-head interviews)• Videos published to coincide with current events got more views

• Next steps• Measure depth of impact using quantitative research

Institutional Storytelling3-5 minute videos telling stories about artists and museum workers, working

on installations, projects, programs and performances

Goals• Humanize the museum, artists and artworks • Increase depth of interest in the museum and its offerings through storytelling• Provide information about artists, artworks, exhibitions, and performances• Attract visitors to the museum/promote specific activities

Priorities• Alignment with institutional priorities/messaging• Strength of story/visually compelling elements• Budget

Analytical Approach• Entering analytics phase after a period of production

• Initial findings• Some platforms generate more views than others• Location on website affects viewership• Videos about iconic aspects of SFMOMA received more views

• Next Steps• Strategize about publishing platforms• Analyze website pathways to improve video placement on website• Do surveys with niche audiences (like teachers) to determine impact

Different approaches for different goals• View counts have different implications depending on video goals

• Qualitative analysis needed to measure “impact”• For some videos views are less important than who watches and what they get out

of it

• Evaluate platform, not just content

Conclusions

Develop Your Own Video Production Evaluation Metrics● Strategy of desired goal: What do you want to achieve with video?

○ Qualitative○ Quantitative

● Start point to compare output after change in practice: Where are you at?○ Reach○ Engagement○ Impact

● Some metrics gathered regularly: How are things changing?○ Quantitative○ Qualitative

● Reflect on your progress: What does this mean?

Next Steps/GoalsOutcomes from survey and evaluation• Iterating on our data capture• Create peer groups for benchmarking• Continue building evaluation tools

MCN community survey version 2• Additional questions based on the results of the first survey and responses

at the presentation• The new survey: http://bit.ly/2ezBl5B

Best PracticesIDENTIFY GOALS: define what impact means for you

PROTOTYPE: create content and distribute in different ways, analyze results

FRAMEWORK: Select evaluation methods

BENCHMARK: Identify peers and aspirational content producers

ITERATE

Thank you

Anna Chiaretta Lavatelli, Director of Digital Media, Museum of Contemporary Art Chicago (@annachiaretta)Trilce Navarrete, Erasmus University Rotterdam (@trilcenavarrete)Emily Robbins, Web and Digital Assistant, San Francisco Museum of Modern Art (@EmBRobbins) Elena Villaespesa, Digital Analyst, The Metropolitan Museum of Art (@elenustika)