Measuring what

counts: Data-driven

decision making through

learning analytics to

enhance life skillsApril 2016

Mohamad Ridwan Othman

Vice President, AGILE

Taylors Education Group

Not everything that counts can be

counted, and not everything that

can be counted counts.

William Bruce Cameron, 1963

Taylor’s Education Group

2016

Garden International

School

Taylor’s International

School KL

Australian International

School Malaysia

NexusInternational

School Putrajaya & Singapore

Taylor’s International

School Puchong

Taylor’s University

Taylor’s College Subang

Jaya

Taylor’s College

Sri Hartamas

British University Vietnam

Schools Division 65 years in Schools

Higher Education Division 47 years in Higher Education

Est. 1969

Est. 2000Est. 1951 Est. 1991Est. 2008

& 2007

Est. 2014

AGILE was set up in 2015 to elevate and make consistent, the learning and teaching quality across all these

institutions and also for the learning community across the region

Our vision is to “To

Educate the Youth of the

World to Take Their

Productive Place as

Leaders in the Global

Community.”

Coined about 5 years

ago, one of the key

outcome was for Taylors

to educate for the

Head, Heart and Hand

i.e. a Holistic Education

THES2016.taylors.edu.my

Why ?

Generation Shift and

Change in Student Profiles

#1

Born 1995 -

2009

Gen Z Connected Locally and Globally

• As global consumers, Gen Z’s have interconnectedness that make them demographically

homogenous. Universally, they are exposed to the same brands and marketing, as

geographical location has become irrelevant

• As they come of age, businesses, employers and even governments will want to steel

themselves for this demanding generation: They want things fast, flexible and in tune with their

beliefs on a mobile device.

General Response:

21st Century skill-sets and attributes. MOE Malaysia's 2015 Blueprint for HE

Many schools are exposing their students to overseas study trips, mission trips, internships and community projects

Social/Collaborative learning (Problem based Learning, Project-based Learning) paradigms

Students trying their hand at being entrepreneurs or taking on part-time jobs

Parents can help them cope with harsh realities by not shielding them too much

Connect with personal touch via mobile

New World of Everywhere

Everyone Everything Technology

#2

Current and

Near

Future

TeacherDidactic

Communicator

Knowledge transfer

ContentTextbook

Courseware

LearnerReceptor

Past/Current

ModeContent-basedTeacher-centric

Education 1.0 Learners• Receiving• Responding• Regurgitating

Reference: https://usergeneratededucation.wordpress.com/tag/education-3-0/

TeacherFacilitator

Curator

ContentTextbook

Courseware

Learner

eContentInternet

Web

OER

Video Lectures

Emerging ModelLearning-based

Learner-centric

L2L

Learner

Education 2.0 Learners• Communicating• Connecting• CollaboratingEducation 3.0 Learners• Connectors• Creators• Constructivists

Reference: https://usergeneratededucation.wordpress.com/tag/education-3-0/

• Personal Learning

Networks

• Community of

Practice

Change in Work Place Expectations

&

Disconnect of Employers and

Graduates

#3

Source: https://robertjgardner.files.wordpress.com/2013/07/vuca-copy.png

5 Key Trends in the Work-place

New behaviors entering the workplace

being more public, sharing, collaborating, etc;

Mobility

being able to get your job done anywhere, anytime, and on any device;

Millennials

a new generation of digitally savvy and enabled workers;

Globalization

living and working in a world without boundaries;

Technology

big data, the cloud, internet of things, wearables, and the like

Source: Jacob Morgan: 5 Ways People Over 50 Can Stay Relevant in the Future of Work

Training Net-Gen Students for Jobs Yet to be Invented

Education 1.0

Education 2.0/3.0

It’s not the content but the pedagogical method

Source: Dancing with Robots: Human Skills for Computerized Workhttp://content.thirdway.org/publications/714/Dancing-With-Robots.pdf

Collaboration U: Business and University Partnerships To Secure Talent Pipelines

Employers are struggling to find college graduates with the necessary skills for their jobs

Employers are encountering a "significant and constantly growing skills gap in today's workforce" and

"there appears to be a

disconnect between higher education institutions that are preparing the next generation of workers and the employers who expect to hire them."

Author: Jenna Filipkowski, PhD

(Jenna.Filipkowski@HCI.org)

Publication date: June 30, 2015 v.2

Source: Human Capital Institute (2015), Collaboration U: Business and University Partnerships To Secure Talent Pipelines. http://www.hci.org/hr-research/collaboration-u-business-and-university-partnerships-secure-talent-pipelines

Changes in/of Learning

Innovation

Agile & responsive

Risk taking

Fail early, fail fast

Problem finding

Beyond problem solving

Sense-making

Learner via the Teacher

Lifelong Learner

Learning Organization

Learning Society

13% OF PEOPLE ARE ENGAGED IN

THEIR WORK

Source: Charlene Li, Altimeter, TED Talk 2014, Giving up control: leading in the digital era

Our Response

The WHAT to do next is easy

when you know

the WHY

TEG Holistic Education Framework

Attributes

Ou

tco

me

Educational Experience

(Value Proposition)

Life SkillsEmotional

Wellbeing

Academic

Excellence

Dim

en

sio

ns

Pro

ce

ss

Dimension

1. Assumed

2. Obvious

Source : Team Analysis

Co

re

Pu

rpo

se

To Educate Youth of the World to Take Their Productive Place as

Leaders in the Global Community

Dim

en

sio

ns

Academic Excellence Life Skills Emotional Wellbeing

Att

rib

ute

1. Discipline Specific Knowledge

2. Thinking & Problem Solving

3. Lifelong Learning

4. Communication Skills

5. Personal competencies (Self-awareness + Self-management – Goleman)

6. Social competencies (Social awareness + r’ship management – Goleman)

7. Entrepreneurialism (Create, Capture and Deliver Value)

8. Global perspectives

Three (3) dimensions have been identified

with eight (8) attributes supporting the

delivery of holistic education

From Ad-Hoc,

Supplementary

To Integration within

Curriculum

How do we MEASURE the efficacy of the programmes

to inculcate HE?

Integration approach

Requires a holistic way to also measure the Life

Skills and Emotional Well Being portion.

Academic Excellence is easy

How do we measure, easily, to ensure accuracy,

validity and reliability?

Rubrics aiding academics and allowing for qualitative analysis would be helpful.

PROBLEM: TIME

CONSUMING and

RESOURCE

INTENSIVE

The ‘Promise’ of Learning Analytics

and Data-Driven Decision Making

“Why is it that the output of powerful learning analytics reporting

processes, acknowledged by institutional leaders as giving new insight

into organizational patterns and practices, fail to influence institutional

planning and strategic decision-making processes? ….

….this may be the result of lack of attention to institutional culture within

higher education, lack of understanding of the degree to which

individuals and cultures resist innovation and change, and lack of

understanding of approaches to motivating social and cultural change.”

Macfadyen, L. P., & Dawson, S. (2012). Numbers Are Not Enough. Why e-Learning Analytics Failed to Inform an Institutional Strategic

Plan. Educational Technology & Society, 15 (3), 149–163.

• Institutional Culture,

• Innovation and Change Resistant

• Approaches to motivate social and

cultural change

Critical success factors

Biggest ‘Problem’ with Analytics in Education

• Great Engine but no or very little mapping

• BI force-fit into Edu Model

• Cumbersome & Difficult to administer

• Time consuming to map to:• Academic’s concerns,

• Departments concerns

• Dean’s and Edu Leaders concerns

• Not easy to integrate with other Data sources

Choice for this pilot in Taylor’s

• Great Engine but no or very little mapping

• BI force-fit into Edu Model

• Cumbersome & Difficult to administer

• Time consuming to map to:• Academic’s concerns,

• Departments concerns

• Dean’s and Edu Leaders concerns

• Not easy to integrate with other Data sources

Edu Mapping

Easily Integrated

Pedagogy Centric

Professor Peter Mortimore

Professor Knut Roald

Professor Kjell B. Hjertø

Professor Louise Stoll

Professor Andy Hargreaves

Professor Viviane Robinson

Professor Michael Fullan

Professor Lorna Earl

Professor Jan Merok Paulsen

Edu Mapping done with Academic Rigour:

Yngve Lindvig, Louise Stoll, Michael Fullan, Andy

Hargreaves, Jarl Inge Wærness, Lorna Earl and

Anders Fosnes. Toronto, 2015

Input

Parental Support

Parental Expectations

Parental socioeconomic background

Input grades

Professional qualityStudent-Teacher Relations

Formative assessment

Student Involvement

Teacher Feedback

Order in Class

Teacher Encouragement of Creativity

Career counselling

Learning environment

Social well-being

Bullying

Trust

Safety

Emotional intelligence

EQ – SEA (Self Emotion Appraisal)

EQ - OEA (Other Emotion Appraisal)

EQ – UOE (Use of Emotion)

EQ – ROE (Regulation of Emotion)

Mastery-approach goal items -MAP

Mastery-avoidance goal items – MAV

Performance-approach goal items - PAP

Performance-avoidance goal items - PAV

Collectivism

Individualism

Win-loose

Commitment

Student behaviour

Self-Assessment

Digital skills

Communication skills

Decision-making skills

Citizenship

Key capacities

Output

Intrinsic Motivation

Self-efficacy

Satisfaction

Tests and/or grades

School Attainment

ConflictsConflicts – (Emotional Relationship Conflict)

Conflicts – (Emotional Task Conflict)

Conflicts – CT (Cognitive Task Conflict)

Conflicts - (Cognitive Relationship Conflict)

Scientifically tested student survey

Why student survey?

Recent research has

shown that student

perception data are

predictive of student

achievement gains and

in many cases are more

reliable than classroom

observations and student

growth measures

(Schulz, Sud and Crowe 2014)

Students are at a good vantage point

to make judgements about classrooms

because they have encountered

many different learning environments

and have enough time in a class to

form accurate impressions. Also, even if

teachers are inconsistent in their day-to

day behaviour, they usually project a

consistent image of the long-standing

attributes of a classroom environment(Fraser 1998)

Unlike value-added data,

student survey data provide

actionable feedback that

teachers and supervisors

can use to target specific

areas for improvement

Example of Survey Questions

Self-emotion appraisal (SEA)

• I have a good sense of why I have certain feelings most of the time.

• I have good understanding of my own emotions.

• I really understand what I feel.

• I always know whether or not I am happy.

Others’ emotion appraisal (OEA)

• I always know my friends’ emotions from their behavior.

• I am a good observer of others’ emotions.

• I am sensitive to the feelings and emotions of others.

• I have good understanding of the emotions of people around me.

Use of emotion (UOE)

• I always set goals for myself and then try my best to achieve them.

• I always tell myself I am a competent person.

• I am a self-motivated person.

• I would always encourage myself to try my best.

Regulation of emotion (ROE)

• I am able to control my temper and handle difficulties rationally.

• I am quite capable of controlling my own emotions.

• I can always calm down quickly when I am very angry.

• I have good control of my own emotions.

Emotional

intelligence

Pilot Approach and Specifics

3000+ students in Taylor’s College Subang Jaya and Taylor’s College Sri Hartamas

4 key programmes

A Levels

South Australian Matriculation (SACE)

Canadian Pre-University

International Baccalaureate

Feb to April 2016 – Administer anonymous surveys via the system

April to May 2016 – Analysis of Results

Sharing with all teachers in TC

Train the teachers in slicing, filtering and managing results in the system

Presentation of Detailed Findings in THES2016, end of May 2016

Review and Fine-Tuning for roll-out in TC

Non-anonymous and Aggregrated at Programme level

Integrated with other systems such as the LMS (Student actual grades)

Possible Analysis and Data

For every factor, academics

in TC can easily plot two-

factor graphs like this, at

both the Institution Level and

even at their own

programme level. (Also by

Gender etc.)

Ability to see the different

student groups that maybe

require the different and

necessary interventions.

Possible Analysis and Data (Future)

Its also possible to

drill down to the

individual students

and give a detailed

map such as this for

every one.

Correlations

between these

data and other

data such as

Student Grades in

the SMS or LMS can

also be shown.

Using evidence and customized educational technology systematically for releasing the potential in every learner: Findings presented at ICSEI Jan 2016

1. Assemble and present data from multiple sources leads to better professional learning

2. In-depth processes with a broad evidence perspective leads to better professional learning

3. Use of digital reflection tools leads to better professional learning

4. This type of professional learning leads to better student outcome, measured with a broad set of indicators

Paper presented at the annual conference of the International Congress for School Effectiveness and Improvement, January 2016, Glasgow.

Evidence

“The dialog maps is an engaging way to inquire into our class-

climate. We are challenged to expand the way we think about learning

and well-being.”

Student

“For the first time the management was inquiring into our

professional learning activities, instead of just evaluating our

results.”

School leader

“Using the process designer with relevant prototypes led to better

prepared teachers that discuss teaching quality instead of data

quality.”

School Leader

“This is probably… No, this is for sure the most sophisticated and

elegant educational solution I’ve ever seen. And the thinking behind it is

very much in line with ours.”

North American, Deputy Education Minister

“After seeing the products and mission, we need to further develop our

strategy and change the focus of our agenda for buying tools for

schools.”

Group of Educational Directors, European Country

“These products is an excellent way to broaden the focus in education

to include well-being, 21 century skills and holistic learning.”

Asian group of Educational Directors

THES2016.taylors.edu.my

25-27 April 2016

Delivering Holistic Edu, Acquiring of LifeSkills is

key for students now

Choosing the right tools helps us measure and fine-

tune

Conexus is one possible approach. Happy

Exploring! Do come to THES2016 for further

results

Mohamad Ridwan Othman

mohamadridwan.othman@taylors.edu.my

THES2016.taylors.edu.my