Damian Bebell, Ph.D.Assistant Research Professor

Center for the Study of Testing, Evaluation and Educational PolicyLynch School of EducationBoston College

Data Mining for Educational Assessment and FeedbackASSESS 2014KDD 2014

Strapping jet engines to the stage coach: Using technology to

push the boundaries of educational measurement

• Background• Defining success in schools

– Purpose of School– Examples from research

• Measurement matters– DVs– IVs– Scaling

• Leveraging digital technology– Data visualization– Feedback loops– Big data– Learner analytics– APIP– Redefining success

“For want of records, much useful knowledge is continually lost. Though many individuals have derived advantages to themselves from experiments, but few have recorded them. Even those who make experiments [are] liable to forget them, so as to give incorrect representations of them when they try to relate them.”

Lathrop, Leonard E. (1826). Remarks On the General Condition of Agriculture: The Farmer’s Library, Second Edition. Windsor, VT: Wyman Spooner.

Background and perspective

• Limited knowledge of computer science, pedagogy, or

instructional technology

• Background in psychometrics, measurement, and evaluation

• Leveraging technology to advance the state of the art in

educational research and measurement…

What is the technology that drives so much of what we do in educational research and measurement today?

IBM 805 Test Scoring Machine 1937

“the assessment of students and schools in the US began as early as 1969, with the first National Assessment of Educational Progress (NAEP) test. The test has developed over the years, but it is still based on the same methodology that was initially enabled by the IBM 805: multiple choice tests with automated scoring.”

http://www-03.ibm.com/ibm/history/ibm100/us/en/icons/testscore/transform/

• The IBM Optical Scanner allowed the bubble sheet and the rise of the 5 response question.

• To this day, the artifacts of this 1930’s computer technology restrict imagination and practice in how we design surveys and tests.

How can we use new and emerging technologies to advance the state of the art In our own research and evaluation work?

Bebell, D., O’Dwyer, L. M., Russell, M., & Hoffman, T. (2010). Concerns, considerations and new ideas for data collection and research in educational technology studies. Journal of Research on Technology in Education, 43(x).

Summative Assessment

FormativeAssessment

The information you collect about your program/initiative can serve to inform the day-to-day operation of your program (formative information) which over time can provide

evidence of success/goals met (summative information).

Leveraging Research and data

Why do schools bother collecting data or conducting research?

•Organize school stakeholders/constituents around project goals •Determine if educational investments are “paying off”•Better understand how teaching and learning is evolving in educational system (and where)

•Measure progress toward specific project and strategic goals

•Support action planning with data

•Provide greater community evidence of “success”•Provide the greater educational community evidence and examples of what works (and what doesn’t)

•Develop a culture and community of data-informed practices

What is the purpose of school?

• Constitutional perspective• Legal perspective• Business leaders perspective• Parent perspective

What about school’s perspective? www.purposeofschool.org

Categorize and quantify the wide spectrum of reasons we have schooling in the US. Use content analyses to focus is on the perspective of schools themselves. Mission statements are a common “data” point for most schools.

Mission Statement/School purpose and outcome identifier:

Cognitive/Academic Development

Social Development

Emotional Development

Civic Development

Physical Development

Vocational Development

Integrate Into Local Community

Integrate Into Global Community

Integrate Into Spiritual Community

Safe/Nurturing EnvironmentChallenging Environment

Mission Statement/School purpose and outcome identifier:

It is not adequate to simply focus exclusively on low hanging academic and cognitive outcomes, if the overall mission or program has a broader reach.

Severe limitations of letting methodological limitations define success

www.purposeofschool.org

• 724 students • 98% non-White• Highest concentration of public housing in USA

Pre/Post Comparison Design Study Outcomes:• Evolution of teaching practices• New pedagogical opportunities• Student engagement • Student retention and attendance• Student achievement• Digital Citizenship

1:1 laptop and Digital teaching platform across 4th and 5th grade

Time to Know (NYC) Study PS-- Brownsville, Brooklyn

T2K Study: Increased Student Attendance

Compared to past averages and those students in the non-1:1 Brooklyn setting, 1:1 students exhibited a statistically significant difference of nearly six school days in the number of missed school days during the 2010/2011 school year.

T2K Study OutcomesYear 1:

– Increase in variety of instructional strategies/resources employed by teachers– 79% of students reported more engagement in class. – Skype partnership with middle-eastern schools

T2K Study OutcomesYear 2:

– NYC DOE’s official measures of success is based on students’ ELA and Math test scores– Teacher names were published with average student score gains in NY Times– High Stakes results for school leadership and teachers

Who defines your success?

Project Ceibal-Uruguay– 1st country wide 1:1 program (2007)– 350,000 students, 16,000 teachers– 1,000,000th student devices was distributed in Fall 2013– X0-1 device, wireless Internet connection

http://www.youtube.com/watch?v=SnLo-j8GvPM

• How do measure the success of Ceibal?• Who is impacted by this program?• How do we know it is working?

“Plan Ceibal…does not see itself as an 'education project' per se, but rather as a project to help transform larger society, with the education system as just the initial vector through which the project hopes to infect all of Uruguay with a new level of 'connectedness'.”

-Michael Trucano

World Bank

Who defines your success?

Who defines your success?

Since Ceibal: “Encompassing the video game companies,

software development in Uruguay has evolved into a $600 million industry, making the country Latin America’s leader in per-capita software exports.”

Feb. 22, 2013

http://www.nytimes.com/2013/02/22/world/americas/uruguays-video-game-start-ups-garner-attention.html?pagewanted=all&_r=0

Who defines your success?

Project Ceibal-Uruguay– 1st country wide 1:1 program (2007-2009)– 350,000 students, 16,000 teachers– X0-1 device, wireless Internet connection

http://www.youtube.com/watch?v=SnLo-j8GvPM

How we define and measure something matters

Small decisions can have big impacts

Measuring the frequency of teachers technology use example

**2-3 min. small group breakout**

Quick discussion activity•Why do some teachers use given resources while other teachers do not?

•How do we illuminate the relationship between teacher practices and educational impacts? (impact of IV on DV)?

•How do we measure teachers practices to illuminate why some teachers adopt and use digital technology and others do not?

Oh, and we have 2500 teachers across 20 schools in 4 continents (they have internet connection- and we have little operating budget)

HOW DO WE OPERATIONALLY DEFINE AND MEASURE TEACHER PRACTICES?

7 Scales of Teacher Technology Use

Examining Teacher Survey Data: Why multiple measures are more revealing (1)

Examining Teacher Survey Data: Why multiple measures are more revealing (2)

Traditional Survey Question(8 response choices):

Sequential Assigned Values

Scaling the Assigned Values

Slider with Full Ratio Scale

Accessible Portable Item Protocol Standard (APIP)IMS Global Learning Consortium, Measured Progress, US State Consortia developed Data model for standardizing the interchange file format for digital test items

1) APIP allows digital tests and items to be ported across APIP compliant test item banks.

2) It provides a test delivery interface with all the information and resources required to make a test and an Item accessible for students with a variety of disabilities and special needs.

Interoperability across assessment items across students

Accessible Portable Item Protocol (APIP)

Data model for standardizing the interchange file format for digital test itemsInteroperability across assessment items across studentsMajor step forward in utilizing technology to advance assessmentWhat are we really trying to measure? Who own the assessments?

Thinking ahead, questioning assumptions, and staying in front of the curve

“All of us… must consciously break the habits we bring to thinking about the computer.

Computation is in its infancy.

It is hard to think about computers of the future without projecting onto them the properties and the limitations of those we think we know today. And nowhere is this more true than in imagining how computers can enter the world of education.”

—Seymour Papert, Mindstorms, 1980

www.asbunplugged.org/collaborativewww.intasc.org

www.purposeofschool.orgwww.bc.edu/T2K

www.bc.edu/nps21www.edx.org

http://www.measuredprogress.org/apip-standard

Questions/reflections

47

bebell@bc.edu

Thank you!