The Art of Evaluation Combined Slides - Arts Midwest · – Etienne Wenger Touchstone Center ......
Transcript of The Art of Evaluation Combined Slides - Arts Midwest · – Etienne Wenger Touchstone Center ......
October 11
1
The Art of Evaluation:
Reflective Learning Loops
October 17, 2011
Welcome
Opening Remarks: Sharon DeMark
Minnesota Philanthropy Partners
David ScheiePresident
Touchstone Center for Collaborative Inquiry
Touchstone Center for Collaborative Inquiry
Building a Learning Culture within Your
Organization or Project
The Art of Evaluation: Reflective Learning Loops
October 17, 2011
Touchstone Center for Collaborative Inquiry
What is a “learning culture?”
“A culture of learning is an environment that supports and encourages the collective discovery, sharing, and application of knowledge.”
-- Stephen Gill (2008)
Touchstone Center for Collaborative Inquiry
Organizational or project culture:
“Shared history, expectations, rules (written & unwritten), values, relationships, and customs that affect everyone’s behavior.” (Conner & Clawson)
“The sum of solutions to yesterday’s problems.” (Schein)
October 11
2
Touchstone Center for Collaborative Inquiry
Organizational or project culture:
Appears at 3 levels:
Level 1: Physical artifacts
Level 2: Intangible policies, rituals, procedures, and networks of relationships
Level 3: Values, assumptions, beliefs, and expectations that underlie levels 1 and 2.
Touchstone Center for Collaborative Inquiry
A project with a learning culture is like a …
Learning organization: “an organization where people continually expand their capacity to create the results they truly desire, where new and expansive patterns of thinking are included, where collective aspiration is set free, and where people are continually learning to see the whole together.” -- Peter Senge
Community of practice: “a group of people who share a passion for something that they know how to do and who interact regularly to learn how to do it better.” – Etienne Wenger
Touchstone Center for Collaborative Inquiry
A learning culture benefits…
Individuals: improves performance, expands opportunities, enhances self-worth, prevents boredom
Organizations: improves morale, effectiveness, efficiency, impact
Communities: more assets, dynamic innovation, better problem analysis & resolution
Touchstone Center for Collaborative Inquiry
Assessing your learning culture: self-audit
Pro-learning culture
People ask questions and share stories about successes, failures, and what they have learned.
People take at least some time to reflect on what has happened and what may happen.
Senior managers are willing to explore their underlying values, assumptions, beliefs, and expectations.
Anti-learning culture
Information is shared on “need to know” basis. People keep secrets & don’t describe how events really happened.
Little time or attention is given to understanding lessons learned from projects.
Senior managers are defensive and unwilling to explore their underlying values, assumptions, beliefs, and expectations.
Touchstone Center for Collaborative Inquiry
Learning culture self-audit, continued…
Pro-learning culture
People are treated as complex individuals.
People are hired and promoted on the basis of their capacity for learning and adapting to new situations.
Performance reviews include attention to what people have learned
Anti-learning culture
People are treated like objects or resources without attention to their individuality.
People are hired and promoted based on their technical expertise as demonstrated by credentials.
Performance reviews focus almost exclusively on what people have done.
Touchstone Center for Collaborative Inquiry
Learning culture: healthy practices
New ways of…
Asking questions
Running meetings
Conducting performance reviews
October 11
3
Touchstone Center for Collaborative Inquiry
Learning culture: healthy practices…
Participating in group discussion
Having one-on-one conversations
Reading about new ideas
Watching experts duel over cutting-edge issues
Reflecting on new data
Touchstone Center for Collaborative Inquiry
Relevant metrics and data
Who’s coming, and how many? Demographics
Roles, e.g. , student, spectator, volunteer
What difference are we making for people? Attitudes
Behaviors
Relationships
Touchstone Center for Collaborative Inquiry
Relevant metrics…
What influence on other institutions and the broader community?
Shifts in programs, practices, policies, partnerships
Participants who go on to contribute to the community as artists, leaders, teachers, etc.
Touchstone Center for Collaborative Inquiry
Relevant metrics: gathering data
Talk-back sessions
Facebook, Twitter posts
Visual documentation, observation
Administrative, attendance records
Surveys
Interviews
Focus groups
Touchstone Center for Collaborative Inquiry
Learning culture: healthy practices…
Meetings include time for task coordination and joint reflection
Annual rhythm includes short meetings and longer learning conferences,
reflection sessions and retreats
1:1, small group and large group interactions
People interact in the private spaces between meetings
Touchstone Center for Collaborative Inquiry
Learning culture: healthy practices…
Program designs often start with fewer elements, more open space than in traditional program design Leaves room for discovery and innovation
Small, everyday interactions are among the most valuable community activities informal discussions to solve a problem one-on-one exchanges of information about a tool,
approach or situation
October 11
4
Touchstone Center for Collaborative Inquiry
Learning culture: healthy practices…
People move between multiple levels of participation:
Core members who steer, initiate and participate extensively
Active members who participate regularly
Peripheral members who watch the interactions of core & active members
Neighbors who have an interest
Touchstone Center for Collaborative Inquiry
Learning culture: healthy practices…
Successful learning communities offer the familiar comforts of a hometown, but also enough varied events to keep new ideas and new peoplecycling into the community.
Familiarity, stability => candor, relationships
New ideas, people, events (divergent thinking) => excitement, a sense of common adventure, & challenges to assumptions
Touchstone Center for Collaborative Inquiry
Learning culture: healthy practices…
Vibrant communities of practice have a rhythm.
If the beat is too fast, the community feels breathless; people stop participating because they are overwhelmed.
When the beat is too slow, the community feels sluggish.
When the beat is strong and rhythmic, the community has a sense of movement and liveliness.
Touchstone Center for Collaborative Inquiry
Tensions & challenges with a learning culture
Expansive reflection may conflict with task completion Do we have time to reflect? To wonder? To think?
Ambiguity & uncertainty can be stressful
Touchstone Center for Collaborative Inquiry
Learning culture in organizational and collaborative projects
Simpler to implement within an organizational project Collaborative partners must agree to invest in, commit
to the learning culture
But contrasts among partners offer rich fuel for learning! And collaborative partners must learn if they are to
collaborate successfully
Touchstone Center for Collaborative Inquiry
Learning culture: implications for leadership
Traditional view: leaders set vision, give direction, make key decisions, energize troops Assumes most people lack vision, need direction, can’t
make good decisions, lack energy Hierarchical, charismatic models of leadership
Learning culture view: leaders are designers, stewards, teachers Assumes people possess vision, imagination, capacity
to solve problems and create things of value Shared, flat, collaborative models of leadership
October 11
5
Touchstone Center for Collaborative Inquiry
Learning culture: implications for leadership…
Leaders as
Designers (designing environments for learning)
Stewards(recognizing and developing others’ talents)
Teachers (facilitating others’ learning)
Panelist 1: BARBARA PlunkettMarketing Director,
MacPhail Center for Music
MARKET RESEARCHFindings, Action, Results
M A C P H A I L C E N T E R F O R M U S I C
MacPhail Center for Music
WALLACE GOALS
Launch “Customer Centered Approach” organization-wide, resulting in…
Increased enrollment and event attendance among 25-54 year olds
OVERVIEW:Customer Centered Approach
STEP 1:Customer Service
Communication Systems
STEP 2:STEP 3:
RESULTS:
CUSTOMER
October 11
6
OVERVIEW:Customer Centered Approach
STEP 1:Customer Service
Communication Systems
STEP 2:STEP 3:
RESULTS:
CUSTOMER
OVERVIEW:Customer Centered Approach
STEP 1:•Customer Service•Communication
Systems
STEP 2:Apply findings about
customers: alter Product,
Place, Price and/orPromotion to better
meet needs
STEP 3:
RESULTS:
CUSTOMER
OVERVIEW:Customer Centered Approach
STEP 1:•Customer Service•Communication
Systems
STEP 2:Apply findings about
Customers: alter Product,
Place, Price and/orPromotion to better
meet needs
STEP 3:Incorporate new
Knowledge of External Variables
RESULTS:
CUSTOMER
OVERVIEW:Customer Centered Approach
STEP 1:•Customer Service•Communication
Systems
STEP 2:Apply findings
aboutCustomers: alter
Product,Place, Price and/orPromotion to better
meet needs
STEP 3:Incorporate new
Knowledge of External Variables
RESULTS:Satisfied customers
Positive teaching experienceIncreased
Enrollment
CUSTOMER
Data Collection
One Time: 26 focus groups; current and potential, 6 Event attendee surveys
Bi-Annual: Demographic Survey
Annual: New Student Satisfaction Surveys
Quarterly: Web and Google analytics
Monthly: Coupon tracking, Student Services feedback
Analyzing Data
Marketing Department: Compiles, conducts 1st
analysis, creates summary report.
Program Leadership: Deeper analysis by area.
Marketing/Leadership: Identify specific action, build into annual/weekly plans. Action plan created.
Staff and Board: Share findings, action and results
October 11
7
26 Adult Focus Groups –One time research
Current: Strong praise for their teacher, class offerings Feel front desk is intimidating at times Lack of communication issue
Potential: No awareness of classes/ensembles, assume
Individual Instruction only Assumed expensive – yet their “fair” price = ours Want clarity on skill levels
Sample Adult Catalog
Adult Focus Groups –Action and Result
Altered price listing, by session and semester
Added “great for beginners”
Created adult-specific ads on public radio and TV stations
Restructured Student Services area
Result: 5% increase in adults since FY09
Early Childhood Music Parents –One time researchCurrent: Praised high-quality program, grounded in child
development, LOVE teachers, reputation NOT clear pathway after ECM Take home materials mixed
Potential: One-third never heard of MacPhail Assumed just for older “serious” musicians Competitors take home materials better Concerns about website usability
Early Childhood Music Sequence
Early Childhood Music –E-lesson sample
October 11
8
Early Childhood Music –Action and Results
Launching new website next week
Cut a new ECM CD as give away
Created ECM specific ads on public radio and TV stations
Results: Positive response. Over 100% enrollment in 2 of 3 sites
EVENTS SURVEYS – One Time
450 events – Onsite 6 surveys at Spotlight, Jazz and Bach’s Lunch
EVENTS SURVEYS -- Findings
98% Agree/Strongly Agree “high-quality” performance
Detailed demographic info/event
Only 5% Likely/Very Likely to become students
Only 13% current students
EVENTS SURVEYS – Action, Results
Action
Continue success of Spotlight Series and Jazz
Maximize faculty performances, decrease guests
Add internal communication to increase students
Introduce Free Family Music Series
Results
Sold out 3 of 6 concerts
Increased student attendance at events
Student Satisfaction Survey –Annual Research
Selected communication and customer satisfaction items to measure longitudinally
By program
By service area
By individual teacher
Student Satisfaction Survey -ACTION
Faculty to/from Student Services New role
Student Services to Students New placement process (use script, send ques.)
Re-org space (sit in front, clear counters, attire)
Conduct Customer Service training
Staff to Faculty Intro new orientation
Spread the Word kits
Professional Development
October 11
9
Sample Student Satisfaction
49
ECM2010
ECM2011
Group 2010
Group2011
Suzuki
2010
Suzuki2011
Individual
Instruction
2010
Individual
Instruction
2011
Music Therapy
2011
Materials and repertoireExcellent
69% 72% 70% 82% 83% 78% 82% 83% 65%
Good 30% 26% 29% 15% 16% 20% 17% 16% 20%Fair
2%2%
1%3%
1%1% 1% 5%
PoorN/A 1% 1% 1% 10%Total 100% 100% 100% 100% 100% 100% 100% 100% 100%Preparedness of teacherExcellent
84% 79% 78% 83% 81% 88% 87% 88% 90%
Good 15% 21% 17% 15% 17% 10% 11% 11% 10%Fair
1%5% 2%
2%1%
2%1%
PoorN/ATotal 100% 100% 100% 100% 100% 100% 100% 100% 100%Guidelines and expectations for home learningExcellent
40% 49% 55% 65% 71% 75% 78% 80% 45%
Good 38% 35% 35% 29% 24% 22% 20% 17% 30%Fair
8%9% 7% 5%
5%3%
1%2% 15%
CONCLUSION:Customer Centered Approach
STEP 1:Customer Service
Communication Systems
STEP 2:STEP 3:
RESULTS:
CUSTOMER
CONCLUSION:Customer Centered Approach
STEP 1:•Customer Service•Communication
Systems
STEP 2:Apply findings about
customers: alter Product,
Place, Price and/orPromotion to better
meet needs
STEP 3:
RESULTS:
CUSTOMER
OVERVIEW:Customer Centered Approach
STEP 1:•Customer Service•Communication
Systems
STEP 2:Apply findings about
Customers: alter Product,
Place, Price and/orPromotion to better
meet needs
STEP 3:Incorporate new
Knowledge of External Variables
RESULTS:
CUSTOMER
CONCLUSION:Customer Centered Approach
STEP 1:•Customer Service•Communication
Systems
STEP 2:Apply findings
aboutCustomers: alter
Product,Place, Price and/orPromotion to better
meet needs
STEP 3:Incorporate new
Knowledge of External Variables
RESULTS:Satisfied customers
Positive teaching experienceIncreased
Enrollment
CUSTOMER
QUESTIONS
October 11
10
Panelist 2: PJ DoyleManaging Director, Mixed Blood Theatre
Radical EvaluationThe Art of Evaluation: Reflective Learning Loops
October 17, 2011
What is…
Mixed Blood Theatre
Our VisionTo be the definitive destination where theater artists and audiences representing the global village can create and share work that spawns a ripple effect of social change and revolutionizes access to theater.
57
What is…
Radical Hospitality
Radical Hospitalityprovides no-cost access to mainstage productions for allaudience members beginning with the 2011–12 season. An expansion of Mixed Blood’s egalitarian mission, Radical Hospitalityerases economic barriers in pursuit of building a truly inclusive, global audience.
58
Today’s…
Strategic Question
Howdo we implement and measure the Radical Hospitality model with particular attention to engaging individuals to whom cost is a barrier to an arts experience?
59
WHO are the stakeholders?
AudiencesContributors
Artists
Create theProgram
Deliver the Program
Assess the impact
Communicatewith stakeholders
Plan the Programming
Mixed
Blood staff
Board
of D
irectors
5
October 11
11
Data CollectionAsking questions•Traditional audience surveys• Intrinsic impact surveys• Free ForUms• Post performance panel discussions• Social media
6162
Analyze, Assess, Crunch
What does it all mean?• Statistical comparisons• Measurable outcomes• Demographics
…and to whom?• Staff• Board• Audience• Contributors• Artists
63
How findings influence Mixed
Blood’s stakeholders
Now what?• Program planning• Community connections• Resource allocation
• Financial• Staffing
• Strategic thinking• Reporting
…and how?• Funder reporting requirements• Annual report• Dashboards
63
WHO are the stakeholders?
AudiencesContributors
Artists
Create theProgram
Deliver the Program
Assess the impact
Communicatewith stakeholders
Plan the Programming
Mixed
Blood staff
Board
of D
irectors
9
Mixed Blood’s VisionTo be the definitive destination where theater artists and audiences representing the global village can create
and share work that spawns a ripple effect of social change and revolutionizes access to theater
65 66
October 11
12
67
Are any of the scenes, lines or music from the performance still bouncing around in your head? 3.88
• Post-Performance: Impact Echo - indicates the extent to which respondents report that scenes or lines from the play [or musical] are still present in their minds
Afterwards, did you discuss the performance with others who attended? 36% intense discussion
• Post-Performance: Meaning-Making through Dialogue with Others Who Attended - indicates the percentage of respondents who reported having an "intense exchange" about the work with others who attended (a form of intellectual stimulation)
Overall, at what level were your expectations fulfilled for this performance? 4.16
• Summative Satisfaction - percentage of respondents who report that their expectations for the experience were fulfilled
68
Questions
69
Panelist 3: Kristin PrestegaardDirector of Marketing and Communications,
Minneapolis Institute of Arts
To enrich the community by,
collecting, preserving and
making accessible
outstanding works of art from
the world’s diverse cultures.
VISION & MISSION
Inspiring wonder through the power of art.
October 11
13
How can we engage
the next generation of
art lovers, ages 20-45?
KEY STRATEGIC QUESTION METRICS
Target Visitor Focus Groups
Museum-wide Visitor Survey
Survey As We Go
Social Media Feedback
METRICS – Focus Groups
• Who is NOT coming?
• Arts interested people who said “the MIA isn’t for me”
• Shifting to a younger age –46 to 39
• 20 to 30 year-old crowd increases from 2010 (33%) to 2011 (45%)
• Increased size of 21 to 45-year-old Third Thursday audience to 85% vs. 55%
METRICS - Annual Visitor Survey
• Direct Feedback on a Day to Day Basis
• What do you do with this?
• How do you use it?
• What’s important?
METRICS - Social Media
• Avinash Kaushik, Analytics Evangelist for Google
• October 10, 2011 blog
•http://www.kaushik.net/avinash/best-social-media-metrics-conversation-amplification-applause-economic-value/
METRICS - Social Media
Conversion Rate, Amplification, Applause Rate, Economic Value
October 11
14
METRICS - Survey As We Go
Is Something Better Than Nothing?
• Third Thursdays
• Music in the galleries
• Audio guides
• Lectures
• Art Carts
Sharing Visitor Survey
1. Share with our senior leadership team
2. All staff brown-bag lunch
3. Top results shared at all staff meeting
4. More in depth information accessible via internal Intranet
INTERNAL LEARNING – Staff input
Partner With Groups That Know Their Networks Best
INTERNAL LEARNING – Partner input
Insert photos
• Flickr albums
• Reflect what we want the event to look like to external audiences
TELLING OUR STORY TO EXTERNAL AUDIENCES
• Programming reflects what we are saying in our communications… you really can “Come As You Are”
TELLING OUR STORY TO EXTERNAL AUDIENCES TELLING OUR STORY TO EXTERNAL AUDIENCES
• Come As You Are
• “The MIA isn’t for me”
• “I don’t see myself at the MIA”
• Literally tell them they are welcome here
October 11
15
B I L L B O A R D S B I L L B O A R D S
P R I N TD O W N T O W N M I N N E A P O L I S
S K Y W A Y A D S
• Mirror display ads:
T E L E V I S I O N
KARE 11 TV spot
Visitor Response:
• Visitor Tweets
•Visitor Video
• Age and Demographic Switch
TELLING OUR STORY TO EXTERNAL AUDIENCES