Student Goals, Engagement and Outcomes: The Path to Success

Student Goals, Engagement and Outcomes: The Path to Success

Mark TroyYunhee Bae

Texas A&M UniversityAIR Forum, May 21, 2013

Goals and engagement are dimensions that facilitate student learning and lead to student success

• How are these dimensions related? • Are some more effective than others at

facilitating learning? • What combinations of goals and engagement

lead to what types of learning?

IEO Assessment Model

Instrument

• Student Experience in the Research University (SERU)• Administered Spring 2012• 42263 students in Six universities

– Minnesota– Pittsburgh– Rutgers– South Carolina– Texas A&M– Virginia

• 7603 responses from TAMU

Methods• Exploratory factor analysis and confirmatory factor analysis to specify:

– three goal factors (epistemic goals, intrinsic goals, and instrumental goals), – four engagement factors (attention and interest, class preparation,

consolidated learning activities, and extra interaction with faculty), – three outcomes factors (knowledge/comprehension, research skills, and

communication skills).• Multiple Indicators and Multiple Causes (MIMIC) CFA to validate

criterion validities through interrelationships between academic engagement constructs and academic outcomes including the three outcome factors and GPA

• Structural Equation Modeling (SEM) to identify relationships between factors

I: Inputs

• Knowledge and skill upon entering the university• Student goals

– Epistemic goal– Instrumental goal– Intrinsic goal

• Background information – Parents’ education– Parents’ household income

8

Measurement Constructs:Students’ College Goal Orientations (n=6,782)

<Epistemic Goal> Reliability (Alpha) = .64Confirmatory Factor

Analysis

Factor LoadingsBe in a position to give something back to my community after finishing my education. .55

Integrate spirituality into my life. .41

Develop a personal code of values and ethics. .71

Discover what kind of person I really want to be. .65

<Instrumental Goal> Reliability (Alpha) = .55Prepare for graduate or professional school. .57

Achieve a high GPA. .65

Be in a position to make a lot of money after finishing my education .45

<Intrinsic Goal> Reliability (Alpha) = .62Develop an in-depth understanding of a specific field of study. .52

Explore new ideas. .58

Acquire a well-rounded general education. .55

Obtain the skills I need to function in the international arena. .50

E: Engagement

• Academic Engagement– Attention and interest in class– Class responsibilities– Consolidated learning activities– Extra interactions with faculty

• Cognitive Strategies– Elaboration– Critical Thinking– Rehearsal

10

Measurement Constructs:Academic Engagement (n=33,880)

<Attention & Interest in Class> Reliability (Alpha) = .85Confirmatory Factor Analysis

Factor LoadingsFound a course so interesting that you did more work than was required. .57Chosen challenging courses, when possible, even though you might lower your GPA by doing so. .36

Contributed to a class discussion. .85Brought up ideas or concepts from different courses during class Discussions. .85Asked an insightful question in class. .86

Interacted with faculty during lecture class sessions. .75

Made a class presentation. .50

<Class Responsibilities >Class Preparation Reliability (Alpha) = .73

Not Gone to class without completing assigned reading. .62Not Gone to class unprepared. .76On average, how much of your assigned course reading have you completed this academic year? .50

Not Skipped class. .66Not Turned in a course assignment late. .50

Time Use Reliability (Alpha) = .53Studying and other academic activities outside of class. .77Attending classes, discussion sections or labs. .49

11

Measurement Constructs:Academic Engagement (Continued)

<Consolidated Learning> Reliability (Alpha) = .73Confirmatory Factor Analysis

Factor LoadingsWorked on class projects or studied as a group with other classmates outside of class. .50

Helped a classmate better understand the course material when studying together. .57

Sought academic help from instructor or tutor when needed. .62

Extensively revised a paper at least once before submitting it to be graded. .61

Raised your standard for acceptable effort due to the high standards of a faculty member. .53

<Extra-Interaction with Faculty> Reliability (Alpha) = .75

Had a class in which the professor knew or learned your name. .67

Talked with the instructor outside of class about issues and concepts derived from a course. .77

Communicated with a faculty member by e-mail or in person. .72

Worked with a faculty member on an activity other than coursework (e.g., student organization, campus committee, cultural activity). .47

12

Measurement Constructs:Emphasis on Cognitive Strategies in the Major

(n = 32,034)<Elaboration> Reliability (Alpha) = .84

Confirmatory Factor Analysis

Factor LoadingsExamined how others gathered and interpreted data and assessed the soundness of their conclusions. .84

Reconsidered your own position on a topic after assessing the arguments of others. .75

Incorporated ideas or concepts from different courses when completing assignments. .75

Used facts and examples to support your viewpoint. .68

<Critical Thinking> Reliability (Alpha) = .83Judge the value of information, ideas, actions, and conclusions based on the soundness of sources, methods and reasoning. .86

Break down material into component parts or arguments into assumptions to see the basis for different outcomes and conclusions. .82

Create or generate new ideas, products, or ways of understanding. .71

<Rehearsal> Reliability (Alpha) = .73

Explain methods, ideas, or concepts and use them to solve problems. .94

Recognize or recall specific facts, terms, and concepts. .62

O: Outcomes

• Knowledge and Comprehension Skills• Research Skills• Communication Skills• GPA

14

Measurement Constructs:Students’ Perceived Learning Outcomes

(n=28,681)<Knowledge & Comprehension> Reliability (Alpha) = .82, .80

Confirmatory Factor AnalysisFactor Loadings

At Starting At present

Analytical and critical thinking skills .77 .76 Ability to read and comprehend academic material .77 .75 Ability to be clear and effective when writing .73 .70 Understanding of a specific field of study .63 .65 Ability to understand international perspectives (economic, political, social, cultural) .58 54

Quantitative (mathematical and statistical) skills .40 .32

<Research Skills> Reliability (Alpha) = .81, .78Internet Skills .90 .86

Computer Skills .84 .77

Library Research Skills .55 .55

Other Research Skills .48 .48

<Communication Skills> Reliability (Alpha) = .75, .77Interpersonal (social) skills .66 .69

Ability to lead .67 .70

Ability to prepare and make a presentation .78 .76

Ability to speak clearly and effectively in English .51 .56

Criterion Validation

• MIMIC (Multiple Indicators Multiple Causes)• Correlations between factor scores and

criteria variables– Parents’ education– Parents’ household income– GPA

Correlations between the Factors and Criterion Variables Path Coefficients from Criterion

Variables to the Factors through MIMIC through MIMIC

GPAParent's Annual Income

Parent Education GPA

Parent's Annual Income

Parent Education

Multiple R

AE Factors (n=33,880)

Attention & Interest .12** .04* .07** .12** .00 .05** .13Class

Responsibilities .24** .01 .05** .24** -.03* .03* .24

Consolidated Learning .07** .03** .02* .07** .03* -.01 .08

Extra Interaction with Faculty .13** .04** .07** .13** .00 .05** .14

SCG Factors (n=6,782)

Epistemic Goal -.02 -.05** -.03* -.01 -.05* -.01 .05Intrinsic Goal .02 -.06** -.01 .02 -.07** .02 .06

Instrumental Goal .19** -.01 .02 .20** -.04* .01 .19

Correlations Between Factors and Criterion Variables

Note. ** : p < .001; * p < .01

SPLO-S Factors (n=28,681)Knowledge & Comprehension Skills .09** .02** .13** .07** .07** .08** .16

Research Skills -.02* .04** .05** -.03** .03** .04** .05Communication Skills .00 .12** .06** -.02* .12** .01 .12

SPLO-P Factors(n=28,681)Knowledge & Comprehension Skills .20** .15** .11** .18** .12** .02** .24

Research Skills .02** .03* .02* .02* .03** .00 .03Communication Skills .04** .13** .05** .02** .14** -.02* .14

ECSMCritical Thinking .01 .00 -.01 .01 .01 -.02* .00Elaboration .05** .02* .00 .05** .02* -.02* .05Rehearsal .04** .08** .03** .03** .08** -.01 .08






Parent Education

Multiple R

Note. ** : p < .001; * p < .01

Correlations Between Factors and Criterion Variables (continued)

Criterion VariablesGPAAnnual Parents’ Household Income .13**

Parent Education .16** .46**






Parent Education

Multiple R

Note. ** : p < .001; * p < .01

Correlations Between Factors and Criterion Variables

19

Summary• Cronbach’s alphas (.53 ≤α ≤.85; mean = .72) were acceptable for all constructs

indicating good internal consistency. • CFAs on each measurement model of AE, SCG, SRLO, ECSM yielded relevant factor

loadings (.32 ≤ λ ≤ .94; mean = .64) with good or acceptable model fits, supporting the construct validity of AE, SCG, and SPLO .

• The relationship of GPA with “Class Responsibilities” ,”Instrumental Goal”, “Knowledge and Comprehension” at the present, and “Elaboration” were the strongest in AE, SCG, SPLO, and ECSM respectively.

• Parents’ annual income is most likely to influence “Knowledge and Comprehension” at present, while, Parent Education is most likely to be related with students’ “Knowledge and Comprehension” at starting indicating a greater impact of parent income than parent education on students’ college learning outcomes

• Necessary for the investigation of the relationships among AE, SCG, SPLO, ECSM and GPA to understand the paths to the learning outcomes.

20

Paths to Student Success in the SERU: The Relationships among 5 domains of students’ Self-Reported Constructs and

their cumulative GPA through Structural Equation Modeling (SEM)

• To investigate the relationships among Academic Engagement, Students’ College Goal, Learning Outcomes, and Cognitive Strategies.--> How do those constructs lead to each learning outcome including student’s perceived learning outcomes (Knowledge & Comprehension, Research Skills, and communication Skills) and GPA ?

• To better understand students’ college experiences and to develop productive policies and programs to heighten students’ competencies for their paths to success.

Pre-Knowledge &

Comprehension

Pre- Research

Skills

Pre- Communication

Skills

Post-Knowledge &

Comprehension

Epistemic Goal

Intrinsic goal

Instrumental Goal

Attention & Interest

Class Responsibilities

Consolidated Learning

Extra-Interaction

with Faculty

Elaboration Critical Thinking Rehearsal

Students’ Perceived Learning Outcomes

at Starting


Now

Students’ College Goal Orientations

Academic Engagement

Emphasis on Cognitive Strategies in the Major

Pre-Knowledge &

Comprehension

Pre- Research

Skills

Pre- Communication

Skills

Post-Knowledge &

Comprehension

Epistemic Goal

Intrinsic goal

Instrumental Goal




Extra-Interaction

with Faculty



at Starting


Now


Academic Engagement


.54

.58

.59

R2 = .41

R2 = .01

R2 = .02

R2 = .02

R2 = .21

R2 = .08

R2 = .18

R2 = .15

.67

.46

.55

Pre-Knowledge &

Comprehension

Pre- Research

Skills

Pre- Communication

Skills

Post-Knowledge &

Comprehension

Epistemic Goal

Intrinsic goal

Instrumental Goal




Extra-Interaction

with Faculty


-.11

-.01

.13

.04.05

.06

.05-.06.18

.11

.05

.04

.07

.49.02.00.02

-.07-.04.14

-.01-.03.13

-.03.00

.00

.17

.11.09

.01.01

.16

.01(ns)

.03

.09

.16-.04

.07

.09

.00(ns)

.08

-.01(ns)

.00(ns)

-.01(ns)

.00(ns)

.00(ns)

.01(ns).01(ns)


at Starting


Now


Academic Engagement


.33.16 -.10

.06.07 .10

.25.10 .07

.28.14 -.07

.54

.58

.59

R2 = .41

R2 = .01

R2 = .02

R2 = .02

R2 = .21

R2 = .08

R2 = .18

R2 = .15

.67

.46

.55

Pre-Knowledge &

Comprehension

Pre- Research Skills

Pre- Communication

Skills

Epistemic Goal

Intrinsic goal

Instrumental Goal




Extra-Interaction with

Faculty


-.11

-.01(ns)

.13

.04.05

.06

.05-.06.18

.11

.05

.04

.07

.02.00(ns)

.02

-.07-.04.14

-.01(ns)

-.03.13

-.03.00(ns)

.00(ns)

.17

.11.09

.01(ns).01(ns)

.06

.00(ns)

.04

.10

.12-.04

.07

.09

.04

.01(ns)

Post- Research Skills

.64


at Starting


Now


Academic Engagement


.33.16 -.10

.06.07 .10

.25.10 .07

.28.14 -.07

.54

.58

.59

R2 = .50

R2 = .01

R2 = .02

R2 = .02

R2 = .21

R2 = .08

R2 = .18

R2 = .15

.67

.46

.55

Pre-Knowledge &

Comprehension


Pre- Communication

Skills

Epistemic Goal

Intrinsic goal

Instrumental Goal




Extra-Interaction

with Faculty


-.11

-.01

.13

.04.05

.06

.05-.06.18

.11

.05

.04

.07

.58.02.00.02

-.07-.04.14

-.01-.03.13

-.03.00

.00

.17

.11.09

.01.01

.12

-.05

.09

.11

.10-.04

.07

.09

.02

.04

Post- Communication

Skills

-.01(ns)

.00(ns)

-.01(ns)

.00(ns)

.00(ns)

.01(ns).01(ns)


at Starting


Academic Engagement


.33.16 -.10

.06 .07 .10

.25 .10 .07

.28.14 -.07

.54

.58

.59

R2 = .49

R2 = .01

R2 = .02

R2 = .02

R2 = .21

R2 = .08

R2 = .18

R2 = .15

.67.46

.55


Now

Pre-Knowledge &

Comprehension


Pre- Communication

Skills

Epistemic Goal

Intrinsic goal

Instrumental Goal




Extra-Interaction

with Faculty


-.11

-.01

.13

.04.05

.06

.05-.06.18

.11

.05

.04

.07

.02

.00

.02

-.07-.04.14

-.01-.03.13

-.03.00

.00

.17

.11.09

.01.01

.04

.11

-.02

.08

.01(ns)

-.04

.07

.09

-.08

.02

Cumulative GPA

Post- Research Skills

Post-Knowledge &

Comprehension

Post- Communication

Skills

-.10

.26-.10

-.01(ns)

.00(ns)

-.01(ns)

.00(ns)

.00(ns)

.01(ns).01(ns)


at Starting


Now


Academic Engagement


.33.16 -.10

.06.07 .10

.25.10 .07

.28.14 -.07

.54

.58

.59

R2 = .07

R2 = .01

R2 = .02

R2 = .02

R2 = .21

R2 = .08

R2 = .18

R2 = .15

.67

.46

.55

33

NOTE: • Model Fit - Χ2/DF: 3250.59/39; CFI: .99; RMSEA: .04• Each construct value (factor score) was composed of 1) standardized item scores to address the

difference in item-response scale, 2) the mean of item scores weighted by factor loadings, and 3) the resulting scores standardized into a mean of 5 and a standard deviation of 2 .

• The SEM includes the error-correlations, but not shown in the figure .• The color of each construct box is correspondent with that of each path coefficient box• The Italic numbers followed by “(ns)” indicate non-significant path coefficients• The numbers following “ “ indicate the strongest impact of each domain on each construct .

ConclusionStructural equation modeling (SEM) yielded several insights into the dimensions that have the greatest impacts on student learning.

• Not surprisingly, for any learning outcome, the student’s incoming ability has the greatest effect on current ability.

• Following that, the greatest effect on “knowledge and comprehension” is found on the path from “pre-communication skills”, to “intrinsic goal”, to “attention and interest in class”, and from “elaboration”.

• The most important factors leading to GPA were found to be the SPLO of “knowledge and comprehension,” and the AE of “class responsibilities”.

• “Elaboration” considered a high level of cognitive strategy did not contribute to GPA.

• The effect of “critical thinking” on GPA was significant, but negative. • “Elaboration” is the strongest factor of ECSM on all of SPLO.

Conclusion, continued

• There are multiple paths to any desired outcome.

• Not all engagement is equal. Engagement activities need to be appropriate to the desired outcome.

• Goals are important

Contact Us

• Mark Troy – [email protected]

• Yunhee Bae– [email protected]

• To download this presentation, go to:– http://dars.tamu.edu/SERU

mailto:[email protected]

mailto:[email protected]

Student Goals, Engagement and Outcomes: The Path to Success

Documents

Transcript of Student Goals, Engagement and Outcomes: The Path to Success