Who the museum evaluations are for: From museum visitors...
Transcript of Who the museum evaluations are for: From museum visitors...
Who the museum evaluations are for : From museum visitors to local societies
and collaborative project partners
Toru Sasaki Hokkaido University
1
2
The museum focuses on the Native Peoples of the North who live in both Eurasia and America.
Hokkaido Museum of Northern Peoples Abashiri City, Hokkaido Prefecture
Source: website of Hokkaido Museum of Northern Peoples
3
Tokyo
Kyoto
Hokkaido Museum of Northern Peoples Abashiri City
Sapporo
4
Permanent Exhibition
Presentation Outline
1. Background 2. Period of museum evaluation adoption 3. Period of reflection 4. Trial period ・Mie Prefectural Museum(MieMu) ・Special exhibition “Mushrooms! Mushrooms! Mushrooms!” Osaka Museum of Natural History ・Regional cooperative project “Singing Insects and the Township” Itami City in Hyogo Prefecture 5. Conclusion and prospects for the future
5
1. Background on the museum evaluation’s adoption in Japan
6
7
Trends in National and Local government
1999
2001 Central Government Reform
2002
1996
1997 Shizuoka Prefecture : the administrative evaluations using business inventory tables
National government
Local government
the Law Concerning Access to Information Held by Administrative Organs
the Government Policy Evaluations Act
M ie Prefecture : the office project evaluation system
8
Tokyo
Kyoto
Mie Pref. Shizuoka Pref.
Sapporo
9
1990 2000 2010 2020
introducing
evaluations
of their
operations
spreading /
being set in p lace
Adoption of the evaluation of adm inistrative bodies in Japan
| | | |
• The Comprehensive Evaluation Method → an evaluation of programs • The Performance Evaluation Method → an evaluation by measuring result • The Project Evaluation Method → an evaluation with a cost-benefit analysis.
10
Issues on evaluation methods :Tatsuya Ono (2009)
• Almost all of the local governments use the Performance Measurement(PM).
• The lack of process to assess the cause and effect relationship or logic in PM.
11
Issues on evaluation methods :Tatsuya Ono (2009)
The method most widely adopted by museums is the Performance Measurement(PM).
12
Museums in Japan and establishers
5690 museums
4489 museums → 79%
Presentation Outline
1. Background 2. Period of museum evaluation adoption 3. Period of reflection 4. Trial period ・Mie Prefectural Museum(MieMu) ・Special exhibition “Mushrooms! Mushrooms! Mushrooms!” Osaka Museum of Natural History ・Regional cooperative project “Singing Insects and the Township” Itami City in Hyogo Prefecture 5. Conclusion and prospects for the future
13
2. Period of museum evaluation adoption
14
15
Office project evaluation system by Tokyo Metropolitan government
This was the first time that a public museum was considered a management organization and was evaluated as a whole.
Achievement rate Efficiency
Performance of a museum Necessity Impartia l i ty
Com prehensive eva luation result( 5 grades)
16
Tokyo
Kyoto
Shizuoka Prefectural Museum of Art
Sapporo
17 Source: website of Shizuoka Prefectural Museum of Art
18
Exhibition & Workshop
• Began to create an evaluation system in 2001.
• A committee composed of third-party stakeholders and museum’s staff members in 2003
• “Museum Navi”(2005) created over roughly 2 years.
19
Shizuoka Prefectural Museum of Art
32 quantitative indicators 8 qualitative indicators
32 quan titative ind icators
40 ind icators
indicators that use
responses from visitors,
18, 56%indicator that uses
responses from residents, 1, 3%
indicators from data
derived from documents,
13, 41%
20
Evaluation indicators of “Museum Navi”, Shizuoka Prefectural Museum of Art
21
One of 8 qualitative indicators
“Did the museum change the lives of exhibition visitors or event participants.”
22
Tokyo
Kyoto
Sapporo
Shizuoka Prefectural Museum of Ar
Presentation Outline
1. Background 2. Period of museum evaluation adoption 3. Period of reflection 4. Trial period ・Mie Prefectural Museum(MieMu) ・Special exhibition “Mushrooms! Mushrooms! Mushrooms!” Osaka Museum of Natural History ・Regional cooperative project “Singing Insects and the Township” Itami City in Hyogo Prefecture 5. Conclusion and prospects for the future
23
3. Period of reflection on the evaluation of museums
24
25
The state of museum evaluation adoptions
Source: Created by the presenter using data from the Japanese Association of Museums(2009:4-5)
N=1,044
The requests from museums that have not adopted evaluations yet.
• “the introduction of specific examples of evaluations”
• “the provision of training, manuals, or guidelines to help with adopting evaluations.
→Lack of the know-how on adopting evaluations. →Lack of appropriate place to learn about it.
26
The opinions and requests from museums that appear to have already adopted evaluations. • “Most of the indicators are evaluations of measures to achieve
numerical goals but it is difficult to evaluate unseen effects.” • “If we can find a way to measure the economic effect of
museum activities or their cost-effectiveness, we can make evaluation activities even better.”
• “We should shift to a system that evaluates the contents of activities.”
• “The evaluation cannot be said to be analyzed or used enough, and we want the technical aspect to be covered better.”
→Skeptical opinions about the Performance Measurement (PM) →Opinions about improving the way the data was analyzed.
27
28
“The Progress of Activities Related to Evaluation and Current Issues.” : Takai (2012)
?
Reason that evaluations did not spread among museums. Takai(2012) 1) There are few museums that formulate midterm or
long-term project plans.
2) There are no tools available to plan midterm or long-term project plans and it is difficult to arrive at concrete indicators or goals from project plans.
→ Lack of midterm or long-term project plans that are connected to a museum’s mission and its self-evaluation. 29
Presentation Outline
1. Background 2. Period of museum evaluation adoption 3. Period of reflection 4. Trial period ・Mie Prefectural Museum(MieMu) ・Special exhibition “Mushrooms! Mushrooms! Mushrooms!” Osaka Museum of Natural History ・Regional cooperative project “Singing Insects and the Township” Itami City in Hyogo Prefecture 5. Conclusion and prospects for the future
30
4. Trial period for creating new evaluations and evaluation methods
31
Presentation Outline
1. Background 2. Period of museum evaluation adoption 3. Period of reflection 4. Trial period ・Mie Prefectural Museum(MieMu) ・Special exhibition “Mushrooms! Mushrooms! Mushrooms!” Osaka Museum of Natural History ・Regional cooperative project “Singing Insects and the Township” Itami City in Hyogo Prefecture 5. Conclusion and prospects for the future
32
33
(1)Designing an evaluation system that is linked to the deliberation of the project p Mie Prefectural Museum (MieMu) Opened in April 2014
34
Mie Osaka
Itami Kyoto
100km
35
Basic Exhibition
MieMu’s symbol Children’s Hands-on exhibition room
Mie’s real-life encyclopedia Learning Space
38
History of the Prefectural Museum in Mie 1953 Prefectural museum of M ie opend( ~ 2007)
1986 Concept for a new museum began being deliberated
2008 Basic Plan for a New Prefectural M useum, M useum M ission
April 2011 Reconsideration of all sorts of projects
June 2011 "3 directions” and “7 items"
2012 W orking group to devise an evaluation system
April 2014 New M useum( M ieM u) opend
MieMu’s mission in 2008
1. Preserve and transmit the assets related to the nature, history, and culture of Mie and make use of them for the future generations.
2. Contribute to developing people through learning and exchanges.
3. Nurture love and pride for the local areas and contribute to developing local areas.
39
<3 Directions> • Create a museum that would transmit the identity of Mie in an easy-to-understand way
• Create a museum for “me” • Create a museum that would support municipal or private museums and cooperate or partner with them to transmit information about Mie
40
“3 directions” & “7 items”
<7 Items> • Constantly make an effort to reduce spending, including total operating
expenses. Devise an income plan that incorporates a gradual rise in income and cut the prefecture’s burden on the annual operating cost of 450 million yen by 20%.
• Enhance the PR system to realize an increase in income by increasing the number of visitors and donations from companies.
• Set up a committee of third-party stakeholders and adopt a mechanism to evaluate and improve the business side of the museum’s operation from a third-party perspective from an early stage.
• Establish a management base with the participation of the public sector based on a variety of ideas.
• Present a solution regarding the current prefectural museum of Mie that will not put a financial burden on the prefecture.
• Expand the use of natural sources of energy even more than what was originally planned.
• Make clear the influences and effects on society that cannot be reflected in financial values and create a mechanism for evaluation and improvement to check on the progress toward this goal.
41
42
Working group for making an evaluation system FY2012 FY2013 FY2014
Opened
in Ap ri l 2014
w ork ing group to devise an evaluation system
・ m useum staff m em bers
・ third-party people including m yself
3. set 6 strategies
4. Evaluation system
1. SW OT analysis
2. Vision making
• Strategy 1:Enrich the contents of the business to extend the range of new visitors and get them to visit many times.
• Strategy 2:Have active public relations activities making use of museum’s opening hours to make the existence of the museum widely known.
• Strategy 3:Promote the participation of prefecture residents and museum visitors in the museum’s activities and operations to create a “museum that thinks together, works together, and grows together.”
• Strategy 4:Focus on preserving and conserving materials in the collection and local cultural properties to prevent the disintegration of materials that will serve as the base for museum activities and to make them available for future use.
• Strategy 5:Target and focus on comprehensive research that can contribute to developing local areas (specific areas that the museum decides on) to clarify the academic value of materials in local areas and transmit their significance.
• Strategy 6:Select projects using the evaluation system to distribute business resources effectively.
Every 3 years, the set of strategies is reviewed and the evaluation indicators are deliberated on as well. 43
6 strategies for the first 3 years
Presentation Outline
1. Background 2. Period of museum evaluation adoption 3. Period of reflection 4. Trial period ・Mie Prefectural Museum(MieMu) ・Special exhibition “Mushrooms! Mushrooms! Mushrooms!” Osaka Museum of Natural History ・Regional cooperative project “Singing Insects and the Township” Itami City in Hyogo Prefecture 5. Conclusion and prospects for the future
44
45
(2)Adopting a logic model that considers the outcomes of a project Osaka Museum of Natural History special exhibition “Mushrooms! Mushrooms! Mushrooms!” (from July 21, 2018 to October 21, 2018)
46
Mie Osaka
Itami Kyoto
100km
47
Special exhibition “Mushrooms! Mushrooms! Mushrooms!”
Exhibition
写真何枚か
Workshop for children
Museum goods
51
Logic model of the Osaka Museum of Natural History's Special Exhibition "Mushrooms"
Museum's mission Museum's mission
Mid-termoutcome
General visitorstake an interest inmushrooms andbecome amateurresearchers.
Medium-termoutcome
Immediateoutcome
General visitorsunderstand theexhibits.
Collaboratingmuseums gainvisibility.
Presenters'incentive toresearch isincreased.
Childrenunderstand theexhibits.
Children,students andteachersunderstand theexhibits.
Childrenunderstand theexhibits
Eventparticipantsdeepen theirknowledge andunderstandingabout nature andmushrooms asliving things andthe exhibits ingeneral.
Visitors who readthe handbooklearn how toobservemushrooms.
Visitors whopurchase goodsdevelop aninterest in nature.
Immediateoutcome
OutputVisitors observethe exhibits(mushrooms).
Archiveddocuments areopened to thepublic.
Presenters’researches areutilized in theexhibition andevents.
Library users seethe notificationand learn aboutthe exhibition.
The exhibitionbecomes knownto many people.
The exhibitionstimulatespeople's interest.
Kids' maps andkid's panels areused.
Worksheets areused.
Many visitorstake part in theevents.
Many visitorstake part in theevents.
Many visitorsread thehandbook.
Many visitorspurchase originalgoods.
Output
Specific activities
Exhibition thatutilizes variousmaterials andinformation(specimens,models, pictures,past research)
Collaborationwith othermuseums
Collaborationwith a naturalhistory circlespecializing inmushrooms andfungi (KansaiMycologicalClub)
Notification in thelibrary about themushroomexhibition
Publicityactivities 1(handouts,posters)
Publicityactivities 2(preparationphase disclosedon the websiteand via SNS)
Kid's maps andkid's panels
Worksheets forschools
Workshop forchildren
Lectures andother promotionevents
Handbook (withthe theme ofmushroomobservation)
Development andsales of originalgoods by themuseum shop
Specific activities
Input Input
People see the publicity and visit the exhibition.
(1) General operation staff for the special exhibition (museum personnel, etc.); (2) Costs (special exhibition budget), scientific research subsidies, etc.; (3) Facilities (exhibition place: Nature Hall, exhibition equipment, etc.); (4) Operationaltasks (preparations, during/after the exhibition); (5) Time resources (activity time for preparation, during/after the exhibition); (6) Information resources (information gathering and management prior to the exhibition, understanding of users’
needs, etc.); (7) Organizational resources (related meetings, collaboration with other organizations, etc.); (8) Network resources (connections on the personal and community levels, etc.)
The museum improves its research/study activity functionsas Osaka's nature information center. Various segments of the citizenry become interested in nature, develop scientific thinking or an inquisitive mind, and cultivate greater sensitivity.
Museums and related organizationscollaborate in research and study.
The special exhibition prompts people to learn aboutthe museum and visit more often.
Children, students and teachers develop an interest inthe natural environment and encourage the adultsaround them to have a similar interest.
People who attend the events, read the handbook orpurchase goods retain their interest in mushrooms andnature, and actively pursue their interest after visitingthe exhibition.
Two missions of this museum that had many expectations riding on it as a museum of Osaka City
1. Information hub on nature of Osaka, enrich the functions of the study and research activities of natural history museums.
2. Cultivate rich sensitivities, scientific ways of thinking and curiosities and allow many citizens of various categories to have an interest in nature.
52 Source: Created by the presenter using data from SAKUMA and Shaku(2019)
When a logic model is used as one method of evaluation 1. When at the stage when a special exhibition’s theme is
decided and plans for the exhibition or related projects and the PR strategy are being devised.→ the staff select the necessary activities or projects.
2. Projects will be implemented multiple times. → Before the project, a logic model is made and project plans are reviewed.
3. When reflecting on a completed project and evaluating it based on a logic model that was used. 53
• Using the two missions of this museum that come with many expectations with it being a museum of Osaka City.
• Causal correlations between the mid-term outcomes, immediate outcomes, outputs, specific activities, and resources.
• How the special exhibition draws out the values or significance of the beneficiaries.
54
A logic model for special exhibition “Mushrooms! Mushrooms! Mushrooms!”
Source: Created by the presenter using data from SAKUMA and Shaku(2019)
Presentation Outline
1. Background 2. Period of museum evaluation adoption 3. Period of reflection 4. Trial period ・Mie Prefectural Museum(MieMu) ・Special exhibition “Mushrooms! Mushrooms! Mushrooms!” Osaka Museum of Natural History ・Regional cooperative project “Singing Insects and the Township” Itami City in Hyogo Prefecture 5. Conclusion and prospects for the future
55
• http://nakumushi.com/ • https://www.youtube.com/watch?time_continue=46&v=CcUzVZh_ifI
56
(3)Taking on the challenge of evaluating social impacts The study of evaluations in the regional cooperative project “Singing Insects and the Township” in Itami City in Hyogo Prefecture
57
Mie Osaka
Itami Kyoto
100km
58
Itami Gochokan, view of the main venue from outside.
Crying insect’s exhibition scene
63
• Making logic model to clarify who the project targets are for, and what changes or values the project intends to bring about.
• Interviews of stakeholders such as organizers, supporters, and beneficiaries.
• Evaluation as a tool to create a common “language” among various stakeholders in the project or to improve the project.
64
“A Practical Study Regarding the Adoption of Social Impact Evaluations in a Museum’s Regional Cooperative Project” Cho (20
65
Logi c Model t o Eval uat e t he Soci al I mpact of I t ami Ci t y' s "Si ngi ng I nsect s and t he Townshi p"
Fi nal I mpact Mi dt er m Out come Shor t -t er m Out come
2. Gai n pl ace f or ci t i zens t o act i vel yconduct act i vi t i es
3. Under st and t he Museum of I nsect sact i vi t i es
6. Ci t i zens/par t i ci pant s f eel cl osert o i nsect s
Sour ce: Cho ( 2019)
5. Ci t i zens/par t i ci pant s hel p pr epar ei nsect s, r ai si ng awar eness about
pr oj ect
Ci t i zens under st and nat ur e bet t er ,nur t ur i ng l ove f or l ocal communi t y.Cr eat es a br and r epr esent at i ve of
I t ami
1. By l i st eni ng t o si ngi ng ofi nsect s, l ocal peopl e enj oy ar r i val
of aut umn
1. Par t i ci pant s/ci t i zens enj oy si ngi ngof i nsect s, cust oms of aut umn and
hi st or i cal cul t ur e
2. Exchanges bet ween l ocal peopl e andr el at ed or gani zat i ons become act i ve 3. Par t i ci pant s j oi n i n act i vi t i es,
r ai si ng i nt er est i n t hat f aci l i t y/shop
4. Each f aci l i t i y par t i ci pat es i npr oj ect , deependi ng r el at i ons wi t h
l ocal communi t y
66
Cabinet Office in Japan and social impact evaluations
2015 ・ Working group to consider social impact evaluations
・ Report “Studies and Research on Social Impact Evaluations.”
→ M ean ing of adopting socia l im pact eva luations
2016 ・ Social Impact M easurement Initiative
(1) M ak ing use of private resou rces
(2) M ak ing socia l va lues visib le
(3) M ak ing business operations m ore effective
67
21 examples of social impact evaluations made public by the Cabinet Office.
2the Random ized Controlled Trial
(RCT) method
the method is both costly and requires
specialized know ledge
9the Social Return on Investment
(SROI) method
the method is more about aim ing to get
funds
8the Im pact evaluation m ethod
usin g log ic m odels
"to create a common language among
various stakeholders in regional
cooperative projects or as a tool to
improve projects" Taku(2019)
2 other methods
21
exam ples
Presentation Outline
1. Background 2. Period of museum evaluation adoption 3. Period of reflection 4. Trial period ・Mie Prefectural Museum(MieMu) ・Special exhibition “Mushrooms! Mushrooms! Mushrooms!” Osaka Museum of Natural History ・Regional cooperative project “Singing Insects and the Township” Itami City in Hyogo Prefecture 5. Conclusion and prospects for the future
68
5. Conclusion and prospects for the future
69
70
Progress of the evaluation at museums in Japan 1990 2000 2010 2020
Evaluation by m useum
| | | |
Evaluation
by estab l isher( Office p roject eva luation system )
Peri
od o
f re
flecti
on
PM
Evaluation system
l in ked to the project p lan
Log ic m odel
l in ked the ou tcom es of a project
Evaluatin g social im pacts Perform ance
M easu rem en
( PM )
Conclusion and prospects for the future
• It has changed from the value to museum visitors to include the value to project collaboration partners or members of local societies who do not visit museums and there have been attempts to make that visible.
• In the future, it is expected that the mission of museums in Japan will expand. When that happens, we can expect the choices of evaluation method to become even more various and rich.
71
Acknowledgements
For this presentation, I would like to thank the following people: • Mr. Junichi Kitamura, a curator at the Mie Prefectural Museum
• Mr. Daisuke Sakuma, a curator at the Osaka Museum of Natural History and Ms. Chieko Shaku, Chief of the General Affairs Division
• Ms. Yenling Cho, a student in the doctoral program of Hokkaido University Faculty of Letters.
72