GEMS FEEDBACK SURVEYTHOUGHTS FROM THE FIELD

Presenters:

Miriam Resendez, MDUC Grant Evaluator

Dr. Robin Clausen, MDUC Stakeholder Liaison/Analyst

9/26/17

Presentation Goals

What is MDUC and

GEMS?

What are the findings from the survey?

How will findings be

used?

What is MDUC and GEMS?

The Creating Montana Data Use Culture Grant is the third grant

that Montana won to develop a Statewide Longitudinal Data

System (GEMS) https://nces.ed.gov/programs/slds/. Montana won

the 2015 grant in the Evaluation and Research Priority of the SLDS

Competition.

MDUC OverviewFY 2015 Montana Data Use Culture grant set goals to use all of this collected and

linked data to explore other research areas and influence data literacy across the

state.

3 Goals of Grant:

• Create long-term, direct partnerships with university researchers

• Increase educator and school leader use of data to enhance learning based

on data driven decision making

• Enhance the OPI’s understanding of its own program effectiveness and the

effectiveness of its partners

Groundwork for Developing Stakeholder Engagement and Sustainability

In FY 2009 OPI was awarded it’s first SLDS grant to create a data warehouse that

would publically and securely (depending on the detail of data) store and report all

data that OPI collects from districts (http://gems.opi.mt.gov)

The 2011 Legislature passed The Pathways to Excellence Program as part of Senate

Bill 329

• Section 4.2 of Senate Bill 329 states that: OPI must develop a publicly available

data system that displays an educational data profile for each school district.

(http://leg.mt.gov/bills/mca/20/7/20-7-104.htm)

At the same time, OPI won the support of multiple stakeholders across the state

supporting the FY 2012 SLDS application, including school district officials, university

presidents, university education department leaders, and state school board

officials. These stakeholders remain engaged during the FY 15 grant.

Evaluation is the process of collecting and

using information for decision-making.

Evaluation

MDUC Grant Evaluation Goals

• To determine the value / impact of

grant activities

• To provide continuous, useful feedback

to team on usage and quality of

resources and activities to help to

inform program status, planning, and

activities

• To document informational needs of

stakeholders

Activities for Building a Data

Use Culture

1. Supporting partnerships

between OPI & K-12 and

university researchers2. Educating future

teachers/educators on

data-driven

decisionmaking

3. Promoting data usage

(GEMS)

Org

an

ize

with

Se

ctio

ns

Continuous Program Improvement (CPI) Model

MDUC Grant Evaluation

• What are the informational needs of stakeholders of the MDUC project (i.e., teachers, other key stakeholders)? What do they know/understand and what area(s) are lacking as it relates to the activities and goals of MDUC?

Stakeholder Needs

•Does professional development provided by the MDUC project enhance educators’ knowledge and ability to make data-driven decisions? Is there a positive change in educators’ attitudes toward data as a support for learning?

Curriculum Evaluation

What is the usability of GEMS? What do MDUC participants think of GEMS? What suggestions do they have for improvement? To what extent are resources being accessed and applied?

Evaluation of Functionality

Survey Process

1. Identify goals & purpose of survey: What research questions will items address? How will

results be used? Who is the target respondent(s)?

2. Literature and document review: What has been done? Can we use the same/similar

items for comparison?

3. Identify method for distribution based on target respondent.

4. Draft items following best practices (e.g., keep it short, no biased/leading items, etc.)

5. Collaborate with team (review-revise)

6. Launch survey, including reminders to improve on response rate

Results of the GEMS Feedback

Survey

Demographics

What is your current position? Frequency

Percent of

Total

(N=80)

Counselor/Advisor 29 31.5%

School Administrator (Principal, VP, Dean, etc.) 15 16.3%

District Administrator (Superintendent, Curriculum Director,

etc.)

21 22.8%

College/University Administrator (President, Dean, etc.) 1 1.1%

Assessment/Data Specialist/Coordinator/Director 4 4.3%

IT Specialist/Coordinator/Director 1 1.1%

Government Employee 1 1.1%

Researcher/Evaluator 2 2.2%

Other - Write in 6 6.5%

* Other: Business Manager, Gear-up Liaison, Library Media/Graduation Matters Coordinator, State Specialist, Title 1 School Support Coordinator, TRIO Director

What types of data do you

access through GEMS? N

Percent of Total

(N=80)

Student achievement 50 62.5%

General school, district, or

institution information

48 60.0%

School profile 32 40.0%

Student characteristics 25 31.3%

School climate 20 25.0%

Program and course offerings 15 18.8%

Student services 13 16.3%

Other* 13 16.3%

School finance 7 8.8%

*Other: FASFA (4), don't know how to access/cannot access (2), EWS, Financial Aid,

post-secondary info.

GEMS Usage2016-17 (N=80) 2015-16 (N=51) 2014-15 (N=51)

Which best describes how you use

GEMS? N % N % N %

To access multiple kinds of data through

GEMS 41 51.3% 21 41.2% 36 70.6%

To access GEMS data in order to create

reports for leadership 31 38.8% 10 19.6% 18 35.3%

To use GEMS data to compare our

district to other school districts 30 37.5% 27 52.9% 42 82.3%

To view dashboards and reports

available through GEMS 29 36.3% 32 62.7% 51 100.0%

To use GEMS data to make decisions

regarding curriculum and instruction 22 27.5% 12 23.5% 18 35.3%

To access one kind of data through

GEMS 11 13.8% 7 13.7% 11 21.6%

To use GEMS data to make decisions

regarding policy 10 12.5% 4 7.8% 10 19.6%

To download data from GEMS for

analysis using other software 9 11.3% 5 9.8% 14 27.5%

To request and access customizable

reports from OPI 9 11.3% NA NA NA NA

Other – Write in 4 5.0% NA NA 15 29.4%

Data Use and Culture

2%

4%

2%

6%

11%

34%

23%

25%

17%

49%

42%

52%

52%

65%

40%

23%

21%

21%

13%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

My administrator(s) (or I) communicate the value

and importance of data use

My administrator(s) (or I) creates many

opportunities for teachers to use data

I am adequately supported in the effective use

of data.

There is a data-driven, collaborative culture

present in my institution.

Resources available through GEMS have

positively impacted my school, district, or

postsecondary institution.

Strongly disagree Disagree Agree Strongly agree

Data Use and Culture

2017 N=57; 2016 N=31; 2015 N=24

23.037.5 45.3

77.062.5 54.7

0%

20%

40%

60%

80%

100%

2016-17 2015-16 2014-15

Resources available through GEMS have

positively impacted my school, district, or

postsecondary institution.

Strongly Disagree/Disagree Strongly Agree/Agree

» A total of 64 GEMS users responded to the items associated with usability

» Source: Based on After Scenario Questionnaire: Lewis, J. R. (1995) IBM Computer Usability Satisfaction Questionnaires: Psychometric Evaluation and Instructions for Use. International Journal of Human-Computer Interaction, 7:1, 57-78.

GEMS Usability and Feedback

17.2

15.6

14.5

34.4

35.9

43.5

48.4

48.4

41.9

0% 20% 40% 60% 80% 100%

Overall, I am satisfied with the ease of

completing tasks in GEMS

Overall, I am satisfied with the amount

of time it takes me to complete tasks in

GEMS

Overall, I am satisfied with the support

information (online-line help,

messages, documentation) when

completing the tasks in GEMS

For the following items, "tasks" refer to the activities or

information you are accessing via GEMS

Disagree Neutral Agree

GEMS Usability and Feedback

3%

8%

11%

11%

5%

22%

19%

35%

46%

47%

54%

42%

46%

22%

17%

12%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%100%

GEMS contains information that is useful

and relevant to me

GEMS provides timely data for decision-

making.

My needs for data and reporting are met

by GEMS

The GEMS website is easy to navigate

Strongly disagree Disagree Agree Strongly agree

46.0 42.250.0

54.0 57.850.0

0%

20%

40%

60%

80%

100%

2016-17 2015-16 2014-15

The GEMS website is easy to navigate

Strongly Disagree/Disagree Strongly Agree/Agree

GEMS Usability and Feedback

What do you like about GEMS? Frequency

Percent of

Total

(N=65)

Variety of data and reports that are

available from single website (one stop

shop)

42 64.6%

Ability to look at trends / changes over

time

36 55.4%

Ability to make side by side comparisons

between schools / districts

28 43.1%

Ability to access data at multiple levels

(student, class, school, district)

23 35.4%

Ability to triangulate data across multiple

GEMS resources (e.g., achievement data

and school climate data)

20 30.8%

User-friendly dashboards / displays 14 21.5%

Ease of navigation 7 10.8%

Other* 4 6.2%*Other: Do not use (2), do not have access (2)

What do you not like about GEMS?

Freque

ncy

Percent of

Total

(N=65)

Too difficult to navigate/use 21 32.3%

Other* 9 13.8%

Requires too much additional analyses to be

useful

6 9.2%

Do not know how to use data contained within

GEMS

6 9.2%

Data/reports are not useful to me 3 4.6%*Other: Limited information displayed (2), data not updated

frequently (2), Lacks a data upload feature, lack of support, difficult to gain access

GEMS Training and Support Resources

What resource(s) did you access? Frequency

Percent of

Total

(N=9)

Data Definitions & Explanations6 66.7%

GEMS User Manual6 66.7%

Regional Training Materials3 33.3%

FAQs2 22.2%

In person Presentation/Demonstration2 22.2%

Online Training Videos2 22.2%

OPI Knowledge Base2 22.2%

GEMS Training and Support Resources

11.1

66.7

22.2

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Percent

To what extent are you satisfied with the

training and support provided on GEMS?

Very satisfied Satisfied A little satisfied Not at all satisfied

22.2

88.9

66.7

11.1 11.1

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

The GEMS training resources I

have been exposed to provided

me with the skills and knowledge I

was seeking

I am satisfied with the OPI

communications regarding GEMS.

Strongly disagree Disagree Agree Strongy agree

49.542.7

100.0

50.557.3

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

2016-17 2015-16 2016-17

The GEMS training resources I have been

exposed to provided me with the skills and

knowledge I was seeking

Strongly disagree/disagree Strongly Agree/Agree

GEMS Early Warning System

• The Early Warning System is a statistical model based on peer reviewed research that

uses everyday live data to predict the probability of a student dropping out of school,

and how this probability may change over time.

• This probability index consists of three Tiers of Prevention – 75.4% of student represent

the lowest risk category; 13.6% of students are seen as at-risk of dropping out of

school; 11.09% of student are in situations of extreme risk of not graduating.

Early Warning System

School Level View in GEMS

Early Warning System

Student Level View in GEMS

Early Warning System

Student Profile View in GEMS

GEMS Early Warning System

35%

65%

Within GEMS, have you used the Early Warning

System feature (High School Only)

Yes No

62.1%

37.9%

Would you like to learn more about the Early Warning

System -- a system that helps identify students at risk

of dropping out of school?

Yes No

GEMS Early Warning System

Within the past 12 months, approximately

how frequently did you access the Early

Warning System? Percent

Monthly (7-12 times per school year) 37.5%

Quarterly (3-5 times per year) 29.2%

2-3 times a month 20.8%

1-2 times per year 12.5%

1-5 times a week 0.0%

12.5

45.8

37.5

62.5

4.2

12.5

25

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Percent Percent

2016-17 2015-16

The Early Warning System provides adequate

information to help me identify students at risk of

dropping out.

Strongly Disagree Disagree Agree Strongly Agree No Opinion

2016-17 N=24; 2015-16 N=8

GEMS Early Warning System

12.5

45.8

41.7

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Valid Percent

The Early Warning System is a useful tool

for teachers and counselors.

Strongly disagree Disagree Agree Strongly agree

25.0

45.833.3

75.0

54.266.7

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

School Level Report

(provides counts for

school, districtwide and

statewide, results by

grade and prior two

runs)

Student Summary

Report (provides EWS

results for every student

in your school via an

excel file)

Student Detail Report

(displays all data used

by the EWS model for a

specific student)

To what extent are you satisfied with the following

Early Warning System (EWS) reports?

Very Satisfied Satisfied A little satisfied Not at all satisfied

GEMS Early Warning System

What do you like about the

Early Warning System? Frequency

Percent of

Total (N=24)

Information it conveys 21 87.5%

Display/platform 11 45.8%

Usefulness for data-based

decision making11 45.8%

Ease of use 10 41.7%

What do you not like about the Early Warning

System? Frequency Percent

Nothing – no response 16 66.7%

Other* 5 20.8%

Do not know how to use EWS 2 8.3%

Requires too much additional analyses to be

useful1 4.2%

Data/reports are not useful to me 0 0.0%

FAFSA Completion Report

• OPI works in conjunction with the Office of the Commissioner of Higher Education

(OCHE) to distribute FAFSA Completion data in a secure platform so that high

school counselors and administrators can determine which of their students had

completed the FAFSA; which students had completed it, but there were errors on

the report; which of their students had started but not completed it; and which

students had not begun the process at all.

• This tool allows counselors to encourage students to complete one of their first

college requirements and to get a better idea of the affordability of college. Plus,

this actionable data allows counselors to see the critical errors on reports that

prevent students from fulfilling this requirement.

FAFSA Completion Report

41.5%

58.5%

Within the GEMS Reports and Data menu, have

you used the FAFSA Completion Report?

Yes No

Within the past 12 months, approximately how

frequently have you accessed the FAFSA

Completion Report? Counselor Administrator

Weekly or more5.9% 0.0%

2-3 times a month23.5% 0.0%

Monthly (7-12 times per school year)29.4% 28.6%

Quarterly (3-5 times per year)23.5% 14.3%

1-2 times per year17.6% 57.1%

4.012.0

80.0

44.0

8.0

52.0

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

The FAFSA Completion Report

provides adequate details on

what actions need to be taken to

complete a student's FAFSA.

The FAFSA Completion Report is a

useful tool for school counselors.

Strongly disagree Disagree Agree Strongly agree

12

88100

0

20

40

60

80

100

2016-17 2015-16

The FAFSA Completion Report provides

adequate details on what actions need to be

taken to complete a student's FAFSA.

Strongly Disagree/Disagree Strongly Agree/Agree

2017 N=25; 2016 N=14

FAFSA Completion Report

FAFSA Completion Report

What do you like about the FAFSA

Completion Report? Frequency

Percent

of Total

(N=27)

Information it conveys 21 77.8%

Ease of use 12 44.4%

Display/platform 10 37.0%

Usefulness for data-based decision

making10 37.0%

What do you not like about the FAFSA

Completion Report? Frequency

Percent

of Total

(N=27)

Nothing – no response 14 51.9%

Not updated frequently 6 22.2%

Other* 4 14.8%

Requires too much additional analyses

to be useful

2 7.4%

Too difficult to navigate/use 1 3.7%

*Other: requirements are not specified, login expires, nothing,

difficulty accessing

RECOMMENDATIONS

EWS More information is needed on why less than half of EWS users (42%) reported that

the EWS provided adequate information to identify students at risk, as this is the goal

of early warning.

FAFSA

FAFSA Completion Report users indicated that they would like these reports to

updated with greater frequency in order to make it a more useful tool for evaluating

student completion of their FAFSAs (e.g., daily or at least weekly).

RECOMMENDATIONSGeneral

Review the recommendations provided in the survey – for example, two areas that were

consistently noted as needing improvement involved providing timely data for decision-

making and improving on the navigation of the system.

Continue to do outreach / marketing of GEMS as the majority of respondents were

interested in learning more about the newest features of GEMS. Related to this, GEMS users

do not often access the online GEMS training and support resources (only 14% reported

accessing). As such, in person trainings or via live online platforms (e.g., webinars) should

continue to be prioritized.

The lowest rated item in the “data use culture” scale was associated with administrators

providing teachers/staff with opportunities to use data. This suggests that more training may

be targeted at administrators on how best to help/support teachers as they incorporate

data into their practices.

What’s next?

How we will use findings

Evaluation in MDUC is much more important than a

grant requirement. Evaluation is at the core of our

continuous improvement process – the third goal of our

grant.

Evaluation Analyses Used to:

• Develop a common understanding among stakeholders about how

people use GEMS.

• This allows us to better understand our successes, what strategies

helped us to achieve success, and to better understand how to sustain

these successes into the future.

• The findings allow us to better understand our weaknesses, to

understand what factors made our project weak, and to prioritize the

process of addressing the weaknesses.

? Questions?