Who the museum evaluations are for : From museum visitors to local societies

and collaborative project partners

Toru Sasaki Hokkaido University

1

2

The museum focuses on the Native Peoples of the North who live in both Eurasia and America.

Hokkaido Museum of Northern Peoples Abashiri City, Hokkaido Prefecture

Source: website of Hokkaido Museum of Northern Peoples

3

Tokyo

Kyoto

Hokkaido Museum of Northern Peoples Abashiri City

Sapporo

4

Permanent Exhibition

Presentation Outline

1. Background 2. Period of museum evaluation adoption 3. Period of reflection 4. Trial period ・Mie Prefectural Museum（MieMu） ・Special exhibition “Mushrooms! Mushrooms! Mushrooms!” Osaka Museum of Natural History ・Regional cooperative project “Singing Insects and the Township” Itami City in Hyogo Prefecture 5. Conclusion and prospects for the future

5

1. Background on the museum evaluation’s adoption in Japan

6

7

Trends in National and Local government

1999

2001 Central Government Reform

2002

1996

1997 Shizuoka Prefecture : the administrative evaluations using business inventory tables

National government

Local government

the Law Concerning Access to Information Held by Administrative Organs

the Government Policy Evaluations Act

M ie Prefecture : the office project evaluation system

8

Tokyo

Kyoto

Mie Pref. Shizuoka Pref.

Sapporo

9

1990 2000 2010 2020

introducing

evaluations

of their

operations

spreading ／

being set in p lace

Adoption of the evaluation of adm inistrative bodies in Japan

| | | |

• The Comprehensive Evaluation Method → an evaluation of programs • The Performance Evaluation Method → an evaluation by measuring result • The Project Evaluation Method → an evaluation with a cost-benefit analysis.

10

Issues on evaluation methods ：Tatsuya Ono (2009)

• Almost all of the local governments use the Performance Measurement(PM).

• The lack of process to assess the cause and effect relationship or logic in PM.

11

Issues on evaluation methods ：Tatsuya Ono (2009)

The method most widely adopted by museums is the Performance Measurement(PM).

12

Museums in Japan and establishers

5690 museums

4489 museums → 79%

Presentation Outline

1. Background 2. Period of museum evaluation adoption 3. Period of reflection 4. Trial period ・Mie Prefectural Museum（MieMu） ・Special exhibition “Mushrooms! Mushrooms! Mushrooms!” Osaka Museum of Natural History ・Regional cooperative project “Singing Insects and the Township” Itami City in Hyogo Prefecture 5. Conclusion and prospects for the future

13

2. Period of museum evaluation adoption

14

15

Office project evaluation system by Tokyo Metropolitan government

This was the first time that a public museum was considered a management organization and was evaluated as a whole.

Achievement rate Efficiency

Performance of a museum Necessity Impartia l i ty

Com prehensive eva luation result（ 5 grades）

16

Tokyo

Kyoto

Shizuoka Prefectural Museum of Art

Sapporo

17 Source: website of Shizuoka Prefectural Museum of Art

18

Exhibition & Workshop

• Began to create an evaluation system in 2001.

• A committee composed of third-party stakeholders and museum’s staff members in 2003

• “Museum Navi”(2005) created over roughly 2 years.

19

Shizuoka Prefectural Museum of Art

32 quantitative indicators 8 qualitative indicators

32 quan titative ind icators

40 ind icators

indicators that use

responses from visitors,

18, 56%indicator that uses

responses from residents, 1, 3%

indicators from data

derived from documents,

13, 41%

20

Evaluation indicators of “Museum Navi”, Shizuoka Prefectural Museum of Art

21

One of 8 qualitative indicators

“Did the museum change the lives of exhibition visitors or event participants.”

22

Tokyo

Kyoto

Sapporo

Shizuoka Prefectural Museum of Ar

Presentation Outline

1. Background 2. Period of museum evaluation adoption 3. Period of reflection 4. Trial period ・Mie Prefectural Museum（MieMu） ・Special exhibition “Mushrooms! Mushrooms! Mushrooms!” Osaka Museum of Natural History ・Regional cooperative project “Singing Insects and the Township” Itami City in Hyogo Prefecture 5. Conclusion and prospects for the future

23

3. Period of reflection on the evaluation of museums

24

25

The state of museum evaluation adoptions

Source: Created by the presenter using data from the Japanese Association of Museums（2009：4-5)

N=1,044

The requests from museums that have not adopted evaluations yet.

• “the introduction of specific examples of evaluations”

• “the provision of training, manuals, or guidelines to help with adopting evaluations.

→Lack of the know-how on adopting evaluations. →Lack of appropriate place to learn about it.

26

The opinions and requests from museums that appear to have already adopted evaluations. • “Most of the indicators are evaluations of measures to achieve

numerical goals but it is difficult to evaluate unseen effects.” • “If we can find a way to measure the economic effect of

museum activities or their cost-effectiveness, we can make evaluation activities even better.”

• “We should shift to a system that evaluates the contents of activities.”

• “The evaluation cannot be said to be analyzed or used enough, and we want the technical aspect to be covered better.”

→Skeptical opinions about the Performance Measurement (PM) →Opinions about improving the way the data was analyzed.

27

28

“The Progress of Activities Related to Evaluation and Current Issues.” : Takai (2012)

?

Reason that evaluations did not spread among museums. Takai(2012) 1) There are few museums that formulate midterm or

long-term project plans.

2) There are no tools available to plan midterm or long-term project plans and it is difficult to arrive at concrete indicators or goals from project plans.

→ Lack of midterm or long-term project plans that are connected to a museum’s mission and its self-evaluation. 29

Presentation Outline

1. Background 2. Period of museum evaluation adoption 3. Period of reflection 4. Trial period ・Mie Prefectural Museum（MieMu） ・Special exhibition “Mushrooms! Mushrooms! Mushrooms!” Osaka Museum of Natural History ・Regional cooperative project “Singing Insects and the Township” Itami City in Hyogo Prefecture 5. Conclusion and prospects for the future

30

4. Trial period for creating new evaluations and evaluation methods

31

Presentation Outline

1. Background 2. Period of museum evaluation adoption 3. Period of reflection 4. Trial period ・Mie Prefectural Museum（MieMu） ・Special exhibition “Mushrooms! Mushrooms! Mushrooms!” Osaka Museum of Natural History ・Regional cooperative project “Singing Insects and the Township” Itami City in Hyogo Prefecture 5. Conclusion and prospects for the future

32

33

（１）Designing an evaluation system that is linked to the deliberation of the project p Mie Prefectural Museum (MieMu) Opened in April 2014

34

Mie Osaka

Itami Kyoto

100km

35

Basic Exhibition

MieMu’s symbol Children’s Hands-on exhibition room

Mie’s real-life encyclopedia Learning Space

38

History of the Prefectural Museum in Mie 1953 Prefectural museum of M ie opend（ ~ 2007）

1986 Concept for a new museum began being deliberated

2008 Basic Plan for a New Prefectural M useum, M useum M ission

April 2011 Reconsideration of all sorts of projects

June 2011 "3 directions” and “7 items"

2012 W orking group to devise an evaluation system

April 2014 New M useum（ M ieM u） opend

MieMu’s mission in 2008

1. Preserve and transmit the assets related to the nature, history, and culture of Mie and make use of them for the future generations.

2. Contribute to developing people through learning and exchanges.

3. Nurture love and pride for the local areas and contribute to developing local areas.

39

＜3 Directions＞ • Create a museum that would transmit the identity of Mie in an easy-to-understand way

• Create a museum for “me” • Create a museum that would support municipal or private museums and cooperate or partner with them to transmit information about Mie

40

“3 directions” & “7 items”

＜7 Items＞ • Constantly make an effort to reduce spending, including total operating

expenses. Devise an income plan that incorporates a gradual rise in income and cut the prefecture’s burden on the annual operating cost of 450 million yen by 20%.

• Enhance the PR system to realize an increase in income by increasing the number of visitors and donations from companies.

• Set up a committee of third-party stakeholders and adopt a mechanism to evaluate and improve the business side of the museum’s operation from a third-party perspective from an early stage.

• Establish a management base with the participation of the public sector based on a variety of ideas.

• Present a solution regarding the current prefectural museum of Mie that will not put a financial burden on the prefecture.

• Expand the use of natural sources of energy even more than what was originally planned.

• Make clear the influences and effects on society that cannot be reflected in financial values and create a mechanism for evaluation and improvement to check on the progress toward this goal.

41

42

Working group for making an evaluation system FY2012 FY2013 FY2014

Opened

in Ap ri l 2014

w ork ing group to devise an evaluation system

・ m useum staff m em bers

・ third-party people including m yself

3. set 6 strategies

4. Evaluation system

1. SW OT analysis

2. Vision making

• Strategy 1：Enrich the contents of the business to extend the range of new visitors and get them to visit many times.

• Strategy 2：Have active public relations activities making use of museum’s opening hours to make the existence of the museum widely known.

• Strategy 3：Promote the participation of prefecture residents and museum visitors in the museum’s activities and operations to create a “museum that thinks together, works together, and grows together.”

• Strategy 4：Focus on preserving and conserving materials in the collection and local cultural properties to prevent the disintegration of materials that will serve as the base for museum activities and to make them available for future use.

• Strategy 5：Target and focus on comprehensive research that can contribute to developing local areas (specific areas that the museum decides on) to clarify the academic value of materials in local areas and transmit their significance.

• Strategy 6：Select projects using the evaluation system to distribute business resources effectively.

Every 3 years, the set of strategies is reviewed and the evaluation indicators are deliberated on as well. 43

6 strategies for the first 3 years

Presentation Outline

1. Background 2. Period of museum evaluation adoption 3. Period of reflection 4. Trial period ・Mie Prefectural Museum（MieMu） ・Special exhibition “Mushrooms! Mushrooms! Mushrooms!” Osaka Museum of Natural History ・Regional cooperative project “Singing Insects and the Township” Itami City in Hyogo Prefecture 5. Conclusion and prospects for the future

44

45

（２）Adopting a logic model that considers the outcomes of a project Osaka Museum of Natural History special exhibition “Mushrooms! Mushrooms! Mushrooms!” (from July 21, 2018 to October 21, 2018)

46

Mie Osaka

Itami Kyoto

100km

47

Special exhibition “Mushrooms! Mushrooms! Mushrooms!”

Exhibition

写真何枚か

Workshop for children

Museum goods

51

Logic model of the Osaka Museum of Natural History's Special Exhibition "Mushrooms"

Museum's mission Museum's mission

Mid-termoutcome

General visitorstake an interest inmushrooms andbecome amateurresearchers.

Medium-termoutcome

Immediateoutcome

General visitorsunderstand theexhibits.

Collaboratingmuseums gainvisibility.

Presenters'incentive toresearch isincreased.

Childrenunderstand theexhibits.

Children,students andteachersunderstand theexhibits.

Childrenunderstand theexhibits

Eventparticipantsdeepen theirknowledge andunderstandingabout nature andmushrooms asliving things andthe exhibits ingeneral.

Visitors who readthe handbooklearn how toobservemushrooms.

Visitors whopurchase goodsdevelop aninterest in nature.

Immediateoutcome

OutputVisitors observethe exhibits(mushrooms).

Archiveddocuments areopened to thepublic.

Presenters’researches areutilized in theexhibition andevents.

Library users seethe notificationand learn aboutthe exhibition.

The exhibitionbecomes knownto many people.

The exhibitionstimulatespeople's interest.

Kids' maps andkid's panels areused.

Worksheets areused.

Many visitorstake part in theevents.

Many visitorstake part in theevents.

Many visitorsread thehandbook.

Many visitorspurchase originalgoods.

Output

Specific activities

Exhibition thatutilizes variousmaterials andinformation(specimens,models, pictures,past research)

Collaborationwith othermuseums

Collaborationwith a naturalhistory circlespecializing inmushrooms andfungi (KansaiMycologicalClub)

Notification in thelibrary about themushroomexhibition

Publicityactivities 1(handouts,posters)

Publicityactivities 2(preparationphase disclosedon the websiteand via SNS)

Kid's maps andkid's panels

Worksheets forschools

Workshop forchildren

Lectures andother promotionevents

Handbook (withthe theme ofmushroomobservation)

Development andsales of originalgoods by themuseum shop

Specific activities

Input Input

People see the publicity and visit the exhibition.

(1) General operation staff for the special exhibition (museum personnel, etc.); (2) Costs (special exhibition budget), scientific research subsidies, etc.; (3) Facilities (exhibition place: Nature Hall, exhibition equipment, etc.); (4) Operationaltasks (preparations, during/after the exhibition); (5) Time resources (activity time for preparation, during/after the exhibition); (6) Information resources (information gathering and management prior to the exhibition, understanding of users’

needs, etc.); (7) Organizational resources (related meetings, collaboration with other organizations, etc.); (8) Network resources (connections on the personal and community levels, etc.)

The museum improves its research/study activity functionsas Osaka's nature information center. Various segments of the citizenry become interested in nature, develop scientific thinking or an inquisitive mind, and cultivate greater sensitivity.

Museums and related organizationscollaborate in research and study.

The special exhibition prompts people to learn aboutthe museum and visit more often.

Children, students and teachers develop an interest inthe natural environment and encourage the adultsaround them to have a similar interest.

People who attend the events, read the handbook orpurchase goods retain their interest in mushrooms andnature, and actively pursue their interest after visitingthe exhibition.

Two missions of this museum that had many expectations riding on it as a museum of Osaka City

1. Information hub on nature of Osaka, enrich the functions of the study and research activities of natural history museums.

2. Cultivate rich sensitivities, scientific ways of thinking and curiosities and allow many citizens of various categories to have an interest in nature.

52 Source: Created by the presenter using data from SAKUMA and Shaku（2019)

When a logic model is used as one method of evaluation 1. When at the stage when a special exhibition’s theme is

decided and plans for the exhibition or related projects and the PR strategy are being devised.→ the staff select the necessary activities or projects.

2. Projects will be implemented multiple times. → Before the project, a logic model is made and project plans are reviewed.

3. When reflecting on a completed project and evaluating it based on a logic model that was used. 53

• Using the two missions of this museum that come with many expectations with it being a museum of Osaka City.

• Causal correlations between the mid-term outcomes, immediate outcomes, outputs, specific activities, and resources.

• How the special exhibition draws out the values or significance of the beneficiaries.

54

A logic model for special exhibition “Mushrooms! Mushrooms! Mushrooms!”

Source: Created by the presenter using data from SAKUMA and Shaku（2019)

Presentation Outline

1. Background 2. Period of museum evaluation adoption 3. Period of reflection 4. Trial period ・Mie Prefectural Museum（MieMu） ・Special exhibition “Mushrooms! Mushrooms! Mushrooms!” Osaka Museum of Natural History ・Regional cooperative project “Singing Insects and the Township” Itami City in Hyogo Prefecture 5. Conclusion and prospects for the future

55

• http://nakumushi.com/ • https://www.youtube.com/watch?time_continue=46&v=CcUzVZh_ifI

56

（３）Taking on the challenge of evaluating social impacts The study of evaluations in the regional cooperative project “Singing Insects and the Township” in Itami City in Hyogo Prefecture

57

Mie Osaka

Itami Kyoto

100km

58

Itami Gochokan, view of the main venue from outside.

Crying insect’s exhibition scene

63

• Making logic model to clarify who the project targets are for, and what changes or values the project intends to bring about.

• Interviews of stakeholders such as organizers, supporters, and beneficiaries.

• Evaluation as a tool to create a common “language” among various stakeholders in the project or to improve the project.

64

“A Practical Study Regarding the Adoption of Social Impact Evaluations in a Museum’s Regional Cooperative Project” Cho (20

65

Logi c Model t o Eval uat e t he Soci al I mpact of I t ami Ci t y' s "Si ngi ng I nsect s and t he Townshi p"

Fi nal I mpact Mi dt er m Out come Shor t -t er m Out come

2. Gai n pl ace f or ci t i zens t o act i vel yconduct act i vi t i es

3. Under st and t he Museum of I nsect sact i vi t i es

6. Ci t i zens/par t i ci pant s f eel cl osert o i nsect s

Sour ce: Cho ( 2019)

5. Ci t i zens/par t i ci pant s hel p pr epar ei nsect s, r ai si ng awar eness about

pr oj ect

Ci t i zens under st and nat ur e bet t er ,nur t ur i ng l ove f or l ocal communi t y.Cr eat es a br and r epr esent at i ve of

I t ami

1. By l i st eni ng t o si ngi ng ofi nsect s, l ocal peopl e enj oy ar r i val

of aut umn

1. Par t i ci pant s/ci t i zens enj oy si ngi ngof i nsect s, cust oms of aut umn and

hi st or i cal cul t ur e

2. Exchanges bet ween l ocal peopl e andr el at ed or gani zat i ons become act i ve 3. Par t i ci pant s j oi n i n act i vi t i es,

r ai si ng i nt er est i n t hat f aci l i t y/shop

4. Each f aci l i t i y par t i ci pat es i npr oj ect , deependi ng r el at i ons wi t h

l ocal communi t y

66

Cabinet Office in Japan and social impact evaluations

2015 ・ Working group to consider social impact evaluations

・ Report “Studies and Research on Social Impact Evaluations.”

　 → M ean ing of adopting socia l im pact eva luations

2016 ・ Social Impact M easurement Initiative

(1) M ak ing use of private resou rces

(2) M ak ing socia l va lues visib le

(3) M ak ing business operations m ore effective

67

21 examples of social impact evaluations made public by the Cabinet Office.

2the Random ized Controlled Trial

(RCT) method

the method is both costly and requires

specialized know ledge

9the Social Return on Investment

(SROI) method

the method is more about aim ing to get

funds

8the Im pact evaluation m ethod

usin g log ic m odels

"to create a common language among

various stakeholders in regional

cooperative projects or as a tool to

improve projects" Taku(2019)

2 other methods

21

exam ples

Presentation Outline

1. Background 2. Period of museum evaluation adoption 3. Period of reflection 4. Trial period ・Mie Prefectural Museum（MieMu） ・Special exhibition “Mushrooms! Mushrooms! Mushrooms!” Osaka Museum of Natural History ・Regional cooperative project “Singing Insects and the Township” Itami City in Hyogo Prefecture 5. Conclusion and prospects for the future

68

5. Conclusion and prospects for the future

69

70

Progress of the evaluation at museums in Japan 1990 2000 2010 2020

Evaluation by m useum

| | | |

Evaluation

by estab l isher（ Office p roject eva luation system ）

Peri

od o

f re

flecti

on

PM

Evaluation system

l in ked to the project p lan

Log ic m odel

l in ked the ou tcom es of a project

Evaluatin g social im pacts Perform ance

M easu rem en

（ PM ）

Conclusion and prospects for the future

• It has changed from the value to museum visitors to include the value to project collaboration partners or members of local societies who do not visit museums and there have been attempts to make that visible.

• In the future, it is expected that the mission of museums in Japan will expand. When that happens, we can expect the choices of evaluation method to become even more various and rich.

71

Acknowledgements

For this presentation, I would like to thank the following people: • Mr. Junichi Kitamura, a curator at the Mie Prefectural Museum

• Mr. Daisuke Sakuma, a curator at the Osaka Museum of Natural History and Ms. Chieko Shaku, Chief of the General Affairs Division

• Ms. Yenling Cho, a student in the doctoral program of Hokkaido University Faculty of Letters.

72