How to Assess Student Learning in Arts Partnerships

Part II: Survey Research

Revised April 2, 2012

Mary Campbell-Zopf, Ohio Arts CouncilMary.Campbell-Zopf@oac.state.oh.us

Craig Dreeszen Ph.D., Dreeszen & Associatescraig@dreeszen.com

Mary Campbell-Zopf is deputy director at the Ohio Arts Council and has been with the agency since 1989. Between 1989 and 2011, Ms. Campbell-Zopf helped secure $12 million for state-level arts, arts education and international programming. She also played a central role in an agency-wide effort to expand the OAC’s International Program through a $1.2 million U.S. Department of Education grant, for which she served as the evaluation manager and participated in professional

development activities with educators and arts administrators in Chile. In 2006, a major milestone in international work was the publishing of The Appreciative Journey: A Guide to Developing International Cultural Exchanges. Campbell-Zopf oversaw the guide’s development and writing and was a major contributor to its content. Ms. Campbell-Zopf has an enduring interest in strategic planning and program evaluation, which led to the publishing of the seven-volume series, Focusing the Light: the Art and Practice of Planning in 2009. Ms. Campbell-Zopf has also been active at the state and national levels in arts education, including serving on numerous state advisory committees for content standards, curriculum, learner assessment, as well as on national advisory committees for the U.S. Department of Education, the National Endowment for the Arts, the National Assembly of State Arts Agencies and the Arts Education Partnership.

Craig Dreeszen directs Dreeszen & Associates, a consulting firm in Northampton Massachusetts. He is a planner, educator, program evaluator, and organizational development consultant. For twelve years he directed the Arts Extension Service at the University of Massachusetts. Since 1986 he has evaluated 30 programs for foundations and state arts agencies and provided strategic

planning support to over 50 public and nonprofit organizations, mostly community based cultural organizations and the agencies that fund and support them.

Dreeszen is a contributing editor of Fundamentals of Arts Management, 5th Edition, author of the chapters, “Program Evaluation,” “Strategic Planning,” and “Board Development.” He wrote the online courses “Strategic Planning” and “Outcome Based Program Evaluation,” the “Learning Partnerships Planning and Evaluation Workbook” and other arts management books and articles. Dreeszen earned his Ph.D. in regional planning and M.A. in organizational development from the University of Massachusetts Amherst.

Your Instructors

Our intended learning outcomes

• Understand the use and limitations of written assessment instruments

• Write effective survey questions

• Understand survey distribution options

• Interpret survey data– Qualitative data analysis– Quantitative data analysis

• Report survey findings

Written Assessment Instruments

• Evaluation forms

• Tests– Pre- and post-tests– Standardized tests

• Surveys

This workshop had relevance for my understanding of seeing art materials as a language. Great relevance, Relevance, little relevance, None at all This workshop prepared me think about ways I might use materials in my classroom. Very prepared, Prepared, Somewhat prepared, Need more instruction Which of the following significant skills were addressed in this workshop? Observation skills, Listening, Evidentiary reasoning, Multiple perspectives

How do you envision yourself using this material in your classroom?

Using Written Assessments

Good reasons to use:

•Efficient•May reach large samples

•Test qualitative conclusions from small samples•Provide quantitative data

•Enables reticent people to participate

May not be appropriate if:

•Uncertain what to ask

•Questions are ambiguous

•Questions are complex

•Respondents not literate

Two Sequencing Options

Start with interviews or focus groups:•Test questions•Learn likely responses•Explore conclusions•Create fixed-response questions•Distribute to larger sample

Start with survey:•Explore ambiguous survey findings in interviews or focus groups•Investigate reasons for survey findings

Survey Steps

04/19/23

• Decide what data is needed

• Design questionnaire• Select sample• Acquire lists• Pretest• Distribute and remind• Analyze and report

Determine Needed Data

• Learning outcomes– To what extent were did

participants learn what was intended

– Unintended outcomes

• Program satisfaction– What liked best– What needs

improvement

• Recommendations• Marketing advice• Advocacy evidence

Write Questions

Open-ended questions•Short answer•Long answer•Prompted

Fixed-response questions•Yes/No•Rating Scale•Choice (one or multiple)•Matrix rated•Ranked

Ask basic demographic questions too

Review Sampling

• Select representative sample of larger population

• Or survey entire population

• Acquire mailing or email lists

Open-ended Questions

Fixed-Response Questions[Replace sample with OAC AP grantee examples]

Practice Writing Questions

• Try writing an open-ended question

• Then, write a fixed-response question and response choices

Paper and Online Surveys

• Mailed surveys– Envelopes– Address labels– Postage– Cover letter– Survey– Return envelope and

postage

• Handout surveys

• Online surveys• SurveyMonkey.com• Zoomerang.com

– Email invitation– Survey– May provide links to

survey results and organization website

– Survey closed notice

Online Survey Features

• Screening questions• Skip logic

• Quick response• Inexpensive• Track responses• Send reminders to

unresponsive email addresses

Test Survey

• Send early draft survey to partners

• Revise• Send next draft to

small sample of your population

• Ask for feedback• Look for ambiguity

or misunderstanding• Revise final survey

Distribute Surveys

Paper surveys•Address and mail•Remind (newsletter, announcements…)

Handout surveys•Distribute and collect

Online surveys•Time right•Email invitation with link to survey URL•Web site links•Social media links•Track responses•Send reminders

Statistical SignificanceResponse Rate

Number of responses divided by number of surveys distributed

90 responses200 surveys sent = 45% response rate

Deduct undeliverable surveys

Margin of error

When sampling populations, larger samples have smaller margin of errorUse online calculator

If you survey 200 students and 90 respond, margin of error = 7.7If 120 respond, margin of error is 5.7

Some apparent results may be due to chance – not statistically significant

No margin of error if entire population sampled, e.g., all participants in your program

Analysis of data

04/19/23

• Reduce large amount of data• Display it in notes, tables, or charts• Make sense of it

– Statistical analysis of numbers– Content analysis of words

Making sense of words

04/19/23

• For each question, read all answers• Look for patterns and clusters• Code similar statements• Count clusters of similar statements• Select typical quotes• Write summary• Do the same for next question

Making sense of numbers

04/19/23

• Count• Average• Note patterns and clusters• Note most frequent responses• Cross tabulate to compare variables• Check statistical significance• Write summary

Report Results with Text

Executive summaryInterpret results

Key-point summaryHeadingsExplanationDataQuotes

Appendix with detailed data and methods

“Findings: The Arts Partnership Program works as intended. The Arts Partnership program furthers the OAC strategic plan, especially Goal 3, to “Help citizens of all ages learn and thrive through the arts.”

Quantitative evidence cited Just over half of AP grantees (55%) cited quantitative evidence to support their evaluation results in the final report.”

Display Results with Graphics

• Data tables• Bar charts• Histograms• Pie charts• Line charts• Scatter plots

Report Results

Report intended learning objectives and actual learning outcomes

Summarize activities

Suggest best practices and needed improvements

Acknowledge funders and partners

Provide back-up data

Sample OLGA report form

Use Results• Draw conclusions

• Report to stakeholders

• Use data to improve programs and account to funders authorizers