The Role of Data in Becoming a High Performing School

1

The Role of Data in a High Performing School

House Keeping The recording and slides for today’s presentation will be made available as soon as possible. Please use the question window to submit questions throughout the webinar. We have time designated at the end for Q&A.

Webinar Presenters

Phil Cummins Managing Director

CIRCLE

Dave Vannette Principle Research Scientist

QUALTRICS

Today’s Agenda

•  7 rules of being data driven in your school

•  How to engage school community members, analyse data patterns, use data to construct better outcomes for more learners

•  Best practices for survey questionnaire design

•  K12 survey templates

Webinar Presenters


CIRCLE

Dave Vannette Principal Research Scientist

QUALTRICS

CONTEXT: ABOUT US CIRCLE – The Centre for Innovation, Research, Creativity and Leadership in Education

–  Working with over 1,750 schools internationally –  An educational agency that equips, empowers and enables schools and school leaders through

consultancy and educational services –  Achieving better outcomes for more learners by building cultures of excellence in leadership

and learning in communities of inquiry –  Strategic alliances with tertiary bodies (including the University of Tasmania) and professional

associations –  Creating educational software solutions for improving school performance including

Touchstones Dr Philip SA Cummins [email protected]

Managing Director, CIRCLE Adjunct Associate Professor, Faculty of Education, University of Tasmania Working in and with schools since 1988

www.circle.education

www.mytouchstones.com @CIRCLEcentral

FRAMEWORK FOR 21st CENTURY LEARNING PARTNERSHIP FOR 21st CENTURY SKILLS March 2011

MODEL FOR 21st CENTURY EDUCATION CENTER FOR CURRICULUM REDESIGN January 2016

Data-‐informed research & prac3ce

Teacher performance & professional development

Literacy & numeracy benchmarking

Con3nuous improvement

ICT in learning benchmarking

Standards-‐referenced curriculum

Forma3ve assessment

THE INTERNATIONAL EDUCATIONAL LANDSCAPE

7 PRINCIPLES FOR USING DATA IN SCHOOLS

Drawn from our work with over 1,750 schools internationally since the early 1980s

1. Mission alignment Understand your purpose and concentrate your activity on this goal; don’t spread your resources too widely.



2. Open inquiry Ask good, open-ended questions; don’t expect a particular outcome

3. Measure what you do Beware substituting a feeling or perception about a successfully run event or program for real data about long-term impact on practice and performance – most schools never measure the impact of PL, especially on teacher capability and student learning


4. Dynamic explication and iteration Define your processes, test and iterate; don’t lock things down too soon


5. Contextualised interpretation Analyse data by finding patterns that tell the real story; don’t let data speak for itself


6. Balanced judgment Temper data with intuition..

Educators must be experts in the evaluation of data. John Hattie, Visible Learning, 2009

Systems 1 Thinking (thinking that is fast, intuitive, often unconscious, relying on past association of ideas) complements Systems 2 Thinking (thinking that is slow, conscious, reasoned, full of effort and ultimately often lazy) – Systems 1 Thinking is the hero because it of its high correlation with the truth. Daniel Kahnemann, Thinking Fast and Slow, 2012

11% of decisions made by marketing professionals are based on data; 16% of decisions are made based on too much data. Harvard Business Review, September 2012


7. Collaborative improvement Use the findings to help engage all members of the community to construct better outcomes for more learners


1.  Mission alignment 2.  Open inquiry 3.  Dynamic explication and experimentation 4.  Wise measurement 5.  Contextualised interpretation 6.  Balanced judgment 7.  Collaborative improvement

What might this look like in your school?

What should this feel like in your school?

USING DATA IN YOUR SCHOOL

Your school’s data culture: Add up your scores and divide by 7. How did you rate yourself? What’s working well? What needs attention?

How do I rate my school in each of these? 1= Below expectation 2 = Meets expectation 3 = Above expectation

1.  Mission alignment 2.  Open inquiry 3.  Dynamic explication and experimentation 4.  Wise measurement 5.  Contextualised interpretation 6.  Balanced judgment 7.  Collaborative improvement

USING DATA IN YOUR SCHOOL

1.  Mission alignment: Understand your purpose and concentrate your activity on this goal; don’t spread your resources too widely.

2.  Open inquiry: Ask good questions; don’t expect a particular outcome. 3.  Dynamic explication and experimentation: Define your processes, test and iterate; don’t lock things down too

soon. 4.  Wise measurement: Use grand school averages and value-added models; avoid benchmarks where possible. 5.  Contextualised interpretation: Analyse data by finding patterns that tell the real story; don’t let data speak for

itself. 6.  Balanced judgment: Temper data with intuition. 7.  Collaborative improvement: Use the findings to help engage all members of the community to construct better

outcomes for more learners.

One thing: •  You know more about •  You feel more confident about •  You might use at your school tomorrow •  You might think about carefully for a long time before using at your school

Your takeaways …


Webinar Presenters

Dave Vannette Principal Research Scientist

QUALTRICS


CIRCLE

Writing Good Survey Questions

Writing Good Survey Questions

Data quality

The cognitive response process

Satisficing

Reliability refers to the extent to which our measurement process provides consistent and repeatable results.

–  Internal consistency (high inter-item correlation for measures of the same construct)

–  Temporal stability (test-retest reliability)

DATA QUALITY: RELIABILITY

Validity refers to the extent to which our measurement process is measuring what we intend to be measuring.

–  Content validity – how well does your sample of response options reflect the domain of possible responses to the question?

–  Criterion-related validity (aka “predictive” or “concurrent” validity) – what is the strength of the empirical relationship between question and criterion (“gold standard”)?

–  Construct validity – how closely does the measure “behave” like it should based on established measures or the theory of the underlying construct

–  Face validity – what does the question look like it’s measuring?

DATA QUALITY: VALIDITY

So what is going on in a parent or student’s head when they take a survey?

1.  Understand intent of question. What is meant by the question, as it may differ from the literal interpretation of the words

COGNITIVE STEPS IN PROVIDING AN ANSWER


2.  Search memory for information. Identifying relevant information stored in memory




3.  Integrate information into summary judgment. Synthesizing information from memory and making determinations about knowledge or attitudes





4.  Translate judgment onto response alternatives. Formatting the summarized information into an acceptable response based on the available question response options





4.  Translate judgment onto response alternatives. Formatting the summarized information into an acceptable response based on the available question response options


Optimising!

Do you really expect parent and students to optimise for every question?

Shortcutting the optimal response process:

Weak Satisficing: Incomplete or biased memory search and/or information integration

Strong Satisficing: Skipping memory search and/or information integration altogether and cueing off the question or context for plausible answers

SATISFICING

Task difficulty –  Interpretation (e.g., number of words, familiarity of words, multiple

definitions) –  Retrieval (e.g., current vs. past state, single vs. multiple objects or

dimensions) –  Judgment (e.g., absolute vs. comparative) –  Response selection (e.g., verbal vs. numeric scale labels, familiarity of

words, multiple definitions of words)

CAUSES OF SATISFICING

Task difficulty Respondent ability

–  Cognitive skills –  Experience thinking about the topic –  Pre-consolidated judgments


Task difficulty Respondent ability Respondent motivation

–  Need for cognition –  Accountability –  Personal importance of the topic –  Belief about survey’s importance –  Number of prior questions


•  Selecting the first reasonable response –  Order of response options can affect answers

“How awesome is Qualtrics?” »  Extremely awesome »  Very awesome »  Somewhat awesome »  Slightly awesome »  Not at all awesome

–  Visual presentation = primacy (the first reasonable response seen) –  Tip: Randomize the direction of the response scale whenever

possible

FORMS OF SATISFICING BEHAVIOUR

•  Selecting the first reasonable response

•  Agreeing with assertions –  Acquiescence bias

•  You may know people that run into this every time they order at Starbucks…




•  You may know people that run into this every time they order at Starbucks…

“Is that with soymilk?”

“Yes”




•  Agree-Disagree (Likert) scales •  True/False •  Yes/No

–  Generally avoid any form of these response scales




•  This can be avoided on every order at Starbucks…




•  This can be avoided on every order at Starbucks…

“Is that with regular or soy milk?”

“…yes?”


•  Selecting the first reasonable response •  Agreeing with assertions •  Straightlining •  Worst in matrix/grid question types

•  Tip: Avoid any use of matrix or grid questions


•  Selecting the first reasonable response •  Agreeing with assertions •  Straightlining •  Saying “don’t know” (DK)

–  Easier than thinking of an answer –  DK/no opinion is not the same as selecting a neutral or middle

alternative •  Tip: Generally avoid DK/no opinion response options


•  Selecting the first reasonable response •  Agreeing with assertions •  Non-differentiation in ratings •  Saying “don’t know” •  Mental coin-flipping


There are two primary levers that we can operate on to reduce satisficing:

1.  Task difficulty •  Make questions as easy as possible •  Minimise distractions •  Keep the duration short

COMBATING SATISFICING

There are two primary levers that we can operate on to reduce satisficing:

1.  Task difficulty •  Make questions as easy as possible •  Minimise distractions •  Keep the duration short

2.  Respondent motivation •  Leverage survey importance •  Keep the duration short •  Use incentives and encouragement to increase engagement

COMBATING SATISFICING

1.  Begin from desired data set and drill down to each individual survey question

2.  Be aware of the cognitive response process – and make it easy

3.  Satisficing is a big threat – don’t enable it with your questionnaires

REVIEW

1.  “Survey Research” by Krosnick (Ann. Rev. Psych, 1999) 2.  “The Psychology of Survey Response” by Tourangeau, Rips, & Rasinski (2000) 3.  “The Science of Asking Questions“ by Schaeffer & Presser (Ann. Rev. Soc, 2003) 4.  “Thinking About Answers” by Sudman & Bradburn (1996) 5.  “Question and Questionnaire Design” by Krosnick & Presser (in the Handbook of Survey

Research, 2010) 6.  “Answering Questions: A Comparison of Survey Satisficing and Mindlessness” by Vannette

& Krosnick (The Wiley Blackwell Handbook of Mindfulness, 2014) 7.  “The Palgrave Handbook of Survey Methodology” by Vannette & Krosnick (forthcoming from

Palgrave in 2016)

FURTHER READING

50

Q&A

51

Thank you

Phil Cummins [email protected] www.circle.education

www.mytouchstones.com @CIRCLEcentral

Dave Vannette [email protected]

www.qualtrics.com/k12 @Qualtrics

Do you have other questions? Do you want to know more?

The Role of Data in Becoming a High Performing School

Technology

Transcript of The Role of Data in Becoming a High Performing School