My name is Dr. Jonathan Sturm My email is [email protected].
ufdcimages.uflib.ufl.edu · 4 ACKNOWLEDGMENTS I thank my supervisor and chair Dr. Albert Ritzhaupt...
Transcript of ufdcimages.uflib.ufl.edu · 4 ACKNOWLEDGMENTS I thank my supervisor and chair Dr. Albert Ritzhaupt...
3D PRINTING INTEGRATION IN K-12 SCIENCE CLASSROOMS: THE RELATIONSHIP WITH STUDENTS’ STEM MOTIVATION, 21ST CENTURY SKILLS,
AND INTEREST IN STEM CAREERS
By
LI CHENG
A DISSERTATION PRESENTED TO THE GRADUATE SCHOOL OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT
OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY
UNIVERSITY OF FLORIDA
2019
© 2019 Li Cheng
To my loved ones
4
ACKNOWLEDGMENTS
I thank my supervisor and chair Dr. Albert Ritzhaupt for his tremendous support
on my dissertation and my entire Ph.D. journey. Dr. Ritzhaupt not only guided my
academic and professional development but also cared about me as an individual. I
also thank my co-chair Dr. Pavlo Antonenko for giving me the opportunity to conduct my
dissertation research through his funded project and I appreciate Dr. Antonenko’s
guidance and support throughout the dissertation process. I am grateful to my
committee members Dr. Kara Dawson and Dr. David Miller for their invaluable feedback
and suggestions on my dissertation. I also want to take this opportunity to thank Dr.
Carole Beal for all her guidance and support during my study in the program.
I am grateful to my husband, Peng Xu, for his love and care. No matter what I go
through, ups and downs, he is my rock to rely on. I would not have completed this
journey without his love and support. I thank Dr. Susan Herrick for all her prayers and
her mother-like love to me. I am grateful to Dr. Ann Gaudino, who always cares about
me. I would also like to thank all the professors who have taught me, the colleagues
who have worked with me, and my friends who have helped me. I especially thank
Wenru Zhou for all her help with SAS programming and multilevel modeling analysis.
She saved me from so many frustrations during my dissertation data analysis.
Lastly, I would like to thank my parents for their unspeakable love. They went
through numerous hardships to raise me and provide me access to education, and they
engraved perseverance in my heart, so I can grow up from a little girl in a poor rural
village in China to a Ph.D. in a prestigious university in the United States.
5
TABLE OF CONTENTS
page
ACKNOWLEDGMENTS .................................................................................................. 4
LIST OF TABLES ............................................................................................................ 9
LIST OF FIGURES ........................................................................................................ 12
ABSTRACT ................................................................................................................... 13
CHAPTER
1 INTRODUCTION .................................................................................................... 15
Research Context ................................................................................................... 15
Problem Statement ................................................................................................. 17 Research Purpose .................................................................................................. 18 Research Questions and Hypotheses..................................................................... 19
Significance of This Study ....................................................................................... 20 Definition of Key Terms ........................................................................................... 20
Organization of This Dissertation ............................................................................ 22
2 LITERATURE REVIEW .......................................................................................... 23
Technology Integration Background and Definition ................................................ 24 Nature of Technology ....................................................................................... 24
History of Technology in Education .................................................................. 25 Defining Technology Integration ....................................................................... 27
3D Printing Integration in K-12 Education ............................................................... 29
How 3D Printing Technology Has Been Integrated in K-12 Education ............. 30 Teaching students how to use 3D printing technology ............................... 31
Integrating 3D printing technology into disciplines ..................................... 33 The Influence of 3D Printing Integration ........................................................... 39 Benefits and Challenges of 3D Printing Integration .......................................... 40
Integrated STEM Education via 3D Printing Integration .......................................... 48 Models and Frameworks for Analyzing 3D Printing Integration .............................. 50
Technology Integration Matrix (TIM) ................................................................. 51
Technological Pedagogical Content Knowledge (TPACK) ............................... 53
Teacher Beliefs and Technology Integration ........................................................... 57 Pedagogical Beliefs .......................................................................................... 60 Self-Efficacy in Technology Integration ............................................................ 62 Technology Value Beliefs ................................................................................. 63
Teacher Beliefs and Student Learning Outcomes .................................................. 66 Students’ STEM Motivation, Interest in STEM Careers, and 21st Century Skills .... 68
Social Cognitive Career Theory ........................................................................ 69
6
Expectancy-Value Theory of Motivation and STEM ......................................... 71 Self-efficacy and STEM ............................................................................. 72
Value beliefs and STEM ............................................................................ 74 Students’ Interest in STEM Careers ................................................................. 75 21st Century Skills ............................................................................................ 77
Conceptual Framework ........................................................................................... 78
3 METHODOLOGY ................................................................................................... 82
Context and Participants ......................................................................................... 82 Instrumentation ....................................................................................................... 85
S-STEM Survey ................................................................................................ 85 Teacher Beliefs on 3D Printing Integration Survey ........................................... 86
Teacher pedagogical beliefs ...................................................................... 87 Teacher self-efficacy in 3D printing technology integration ........................ 88 Teacher 3D printing value beliefs ............................................................... 89
Lesson Plan Codebook .................................................................................... 90 3D printing integration levels ...................................................................... 90
STEM integration levels ............................................................................. 93 Data Sources and Data Collection .......................................................................... 94 Data Analysis .......................................................................................................... 94
Data Analysis for RQ1 ...................................................................................... 96 Descriptive statistical analysis .................................................................... 96
Lesson plan analysis .................................................................................. 96 Correlational analysis ................................................................................. 97 Thematic analysis ...................................................................................... 97
Data Analysis for RQ2 ...................................................................................... 98
Descriptive statistical analysis .................................................................... 98 Multilevel modeling analysis ....................................................................... 99 Multiple regression analysis ..................................................................... 106
4 RESULTS ............................................................................................................. 107
Descriptive Statistics of Variables ......................................................................... 107 Dependent Variables ...................................................................................... 109
Student-Level Independent Variables ............................................................ 109 Teacher-Level Independent Variables ............................................................ 110
Internal Consistency ............................................................................................. 111 Correlations between Variables ............................................................................ 113
Results for RQ1 .................................................................................................... 117 Correlations between Teacher Beliefs and 3D Printing Integration ................ 117 Open Responses in Teacher Beliefs Survey .................................................. 117
Results for RQ2 .................................................................................................... 121 Missing Data Evaluation ................................................................................. 121 Assumptions Testing ...................................................................................... 122 Results for Science Motivation ....................................................................... 123
Baseline model ........................................................................................ 124
7
Student-level models ............................................................................... 125 Adding teacher-level variables ................................................................. 129
Effect size calculation .............................................................................. 135 Results for Technology/Engineering Motivation ............................................. 136
Baseline model ........................................................................................ 137 Student-level models ............................................................................... 138 Adding teacher-level variables ................................................................. 141
Effect size calculation .............................................................................. 148 Results for Math Motivation ............................................................................ 149
Baseline model ........................................................................................ 149 Student-level models ............................................................................... 151 Adding teacher-level variables ................................................................. 154
Effect size calculation .............................................................................. 161
Results for 21st Century Skills ........................................................................ 162 Baseline model ........................................................................................ 162
Student-level models ............................................................................... 164
Adding teacher-level variables ................................................................. 168 Effect size calculation .............................................................................. 174
Results for Interest in STEM Careers ............................................................. 175
Baseline model ........................................................................................ 175 Student-level models ............................................................................... 176
Multiple regression ................................................................................... 178 Summary of Results ....................................................................................... 182
5 DISCUSSIONS ..................................................................................................... 190
Limitations and Delimitations ................................................................................ 190
Limitations ...................................................................................................... 190 Delimitations ................................................................................................... 192
Relationships between Teacher Beliefs and 3D Printing Integration .................... 193
Relationships between Teacher Variables and Student Outcomes ...................... 194 Relationships with Science Motivation ........................................................... 195 Relationships with Technology/Engineering Motivation .................................. 198
Relationships with Math Motivation ................................................................ 200 Relationships with 21st Century Skills ............................................................ 205 Relationships with Interest in STEM Careers ................................................. 207
Implications ........................................................................................................... 209 Implications for Practice ................................................................................. 210
Implications for Research ............................................................................... 213
Conclusions .......................................................................................................... 216
APPENDIX
A S-STEM SURVEY (UNFRIED ET AL., 2015) ....................................................... 218
B TEACHER BELIEFS ON 3D PRINTING INTEGRATION SURVEY ...................... 224
8
LIST OF REFERENCES ............................................................................................. 230
BIOGRAPHICAL SKETCH .......................................................................................... 249
9
LIST OF TABLES
Table page
2-1 How 3D printing has been integrated in K-12 education ..................................... 36
2-2 3D printing integration in K-12 education and the impacts on student ................ 43
3-1 Student demographics ........................................................................................ 84
3-2 Teacher demographics ....................................................................................... 84
3-3 Codebook for 3D printing integration levels ........................................................ 92
3-4 Codebook STEM integration levels .................................................................... 93
3-5 Overview of data analysis ................................................................................... 95
3-6 Phases of thematic analysis (from Braun & Clarke, 2006) ................................. 97
3-7 The meanings of symbols in equations (3-1), (3-2), and (3-3) .......................... 102
4-1 Variable names and their meanings ................................................................. 108
4-2 Descriptive statistics for dependent variables ................................................... 109
4-3 Descriptive statistics for student-level independent variables ........................... 110
4-4 Descriptive statistics for teacher-level independent variables .......................... 111
4-5 Cronbach’s alpha of rating scales ..................................................................... 112
4-6 Correlations between dependent variables ...................................................... 114
4-7 Correlations between value beliefs subscales .................................................. 114
4-8 Correlations between self-efficacy beliefs subscales ........................................ 115
4-9 Correlations between teacher beliefs ............................................................... 116
4-10 Correlations between Printing_Level, STEM_Level, and teacher beliefs ......... 117
4-11 Thematic analysis results of teachers’ open responses ................................... 120
4-12 Proportion of missing values for student-level variables ................................... 122
4-13 Skewness and kurtosis of dependent variables ................................................ 123
4-14 The meaning of symbols in equations (4-1), (4-2), (4-3) ................................... 125
10
4-15 Baseline model summary ................................................................................. 125
4-16 Random-intercept model summary ................................................................... 127
4-17 Random-slope model summary ........................................................................ 129
4-18 Multicollinearity of variables for science motivation posttest score ................... 130
4-19 Random-intercept model with teacher variables model summary .................... 134
4-20 Statistics for effect size calculation ................................................................... 136
4-21 The meaning of symbols in equations (4-18), (4-19), (4-20) ............................. 137
4-22 Baseline model summary ................................................................................. 138
4-23 Random-intercept model summary ................................................................... 140
4-24 Multicollinearity of variables for technology/engineering motivation posttest score ................................................................................................................. 142
4-25 Random-intercept model with teacher variables model summary .................... 147
4-26 Statistics for effect size calculation ................................................................... 148
4-27 The meaning of symbols in equations (4-34), (4-35), (4-36) ............................. 150
4-28 Baseline model summary ................................................................................. 150
4-29 Random-intercept model summary ................................................................... 152
4-30 Random-slope model summary ........................................................................ 154
4-31 Multicollinearity of variables for math motivation posttest score ....................... 155
4-32 Random-intercept model with teacher variables model summary .................... 160
4-33 Statistics for effect size calculation ................................................................... 161
4-34 The meaning of symbols in equations (4-51), (4-52), (4-53) ............................. 163
4-35 Baseline model summary ................................................................................. 163
4-36 Random-intercept model summary ................................................................... 165
4-37 Random-slope model summary ........................................................................ 167
4-38 Multicollinearity of variables for 21st century skills posttest score .................... 168
4-39 Random-intercept model with teacher variables model summary .................... 173
11
4-40 Statistics for effect size calculation ................................................................... 174
4-41 The meanings of symbols in equations (4-68), (4-69), (4-70) ........................... 175
4-42 Baseline model summary ................................................................................. 176
4-43 Multicollinearity of variables for interest in STEM careers posttest score ......... 178
4-44 Multiple regression model summary ................................................................. 181
4-45 Summary of results for student outcomes ........................................................ 186
4-46 Interactions between student pretest scores and teacher variables ................. 188
4-47 Interactions between student gender and teacher variables ............................ 189
12
LIST OF FIGURES
Figure page
2-1 Organization of the literature review ................................................................... 23
2-2 TIM with progression across levels of integration ............................................... 53
2-3 The TPACK framework and its knowledge components.. ................................... 54
2-4 Theoretical framework of relationships between teacher beliefs and technology integration ........................................................................................ 60
2-5 Theoretical framework for STEM motivation, interest in STEM careers, and 21st century skills ............................................................................................... 69
2-6 Conceptual framework of this study .................................................................... 81
13
Abstract of Dissertation Presented to the Graduate School of the University of Florida in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy
3D PRINTING INTEGRATION IN K-12 SCIENCE CLASSROOMS: THE RELATIONSHIP WITH STUDENTS’ STEM MOTIVATION, 21ST CENTURY SKILLS,
AND INTEREST IN STEM CAREERS
By
Li Cheng
August 2019
Chair: Albert D. Ritzhaupt
Cochair: Pavlo Antonenko
Major: Curriculum and Instruction
As an emerging technology in K-12 education, 3D printing has gained much
attention from educators and researchers. However, meaningful 3D printing integration
into K-12 STEM curricula is still scarce, and little is known about how teacher beliefs
influence 3D printing integration and how the integration may influence students’
learning outcomes. This study examined the relationship between teachers’ beliefs, 3D
printing integration, and students’ STEM motivation, 21st century skills, and interest in
STEM careers, which are essential for students to participate in STEM disciplines and
future STEM careers.
This study included 26 teachers across 6 states in the U.S. and their 1,501
students, who participated in the iDigFossils project. Teachers’ lesson plans were
analyzed to examine the 3D printing and STEM integration levels. Data on teachers’
beliefs and students’ STEM motivation, 21st century skills, and interest in STEM careers
were collected using scales adapted from previously validated surveys. This study
14
conducted correlational and multilevel modeling analyses to examine the relationships
between these variables.
Results indicated that teacher beliefs and 3D printing integration were generally
not correlated except for a negative relationship between teachers’ self-efficacy in
pedagogical content knowledge and STEM integration level. Teachers perceived 3D
printing integration as beneficial for students, but they encountered a few challenges
including logistic and technical issues, lack of time and resources, insufficient ability to
use 3D printers and connect 3D printing with curricula, and difficulty in teaching
students with individual differences. Furthermore, teachers’ STEM integration levels
were a positive predictor of students’ math motivation. Teachers’ 3D printing integration
levels were not significant for any student outcome variables. Teachers’ value beliefs
including interest in and perceived importance of 3D printing integration were not
significant, however, teachers’ perceived usefulness of 3D printing was a negative
predictor of students’ 21st century skills. Finally, interesting interaction effects were
observed between student variables (student gender and pretest scores) and teacher
variables (teacher beliefs and 3D printing integration). Future research may employ
experimental design to examine the effects of different 3D printing and STEM
integration levels on students’ learning outcomes, and how different levels may
influence students with individual differences.
15
CHAPTER 1 INTRODUCTION
Research Context
Science, technology, engineering, and mathematics (STEM) are critical for a
nation’s economic development and are fundamental aspects of our lives (NRC, 2011).
STEM knowledge and skills are not only important for professionals in STEM areas, but
many other jobs require a certain level of STEM knowledge and skills (NRC, 2011). The
employment projections 2016-26 released by the Bureau of Labor Statistics (BLS)
indicated there will be significant employment increases in STEM-related occupations
(BLS, 2017). There has been increasing need for an adequately prepared STEM
workforce and increasing demand for workers with STEM skills and competencies
(Honey, Pearson, & Schweingruber, 2014). Moreover, the United States found its STEM
workforce lagged behind some Asian countries (Friedman, 2005) and realized the
necessity to invest in STEM education.
Research suggests that students’ STEM motivation is influential to their STEM
learning and future career choices. Specifically, students’ motivation in STEM leads to
continuous engagement in STEM learning (Maltese, Melki, & Wiebke, 2014) and many
studies indicate that students’ interest and motivation in STEM are closely related to
their future career choices (e.g., Christensen & Knezek, 2017; Maltese & Tai, 2011;
Sadler, Sonnert, Hazari, & Tai, 2012; Tai, Liu, Maltese, & Fan, 2006). Furthermore,
students’ interest in STEM careers can potentially increase the possibility of selecting
STEM careers. Nonetheless, students need 21st century skills to succeed in STEM
learning and future careers (Unfried, Faber, Stanhope, & Wiebe, 2015). Therefore,
16
students’ STEM motivation, interest in STEM careers, and 21st century skills are all
essential for students to participate in STEM disciplines and STEM careers.
As one of the subject areas of STEM education, science has been facing the lack
of student motivation and engagement, and it is challenging for teachers to keep
students engaged in science learning (Schmidt, Rosenberg, & Beymer, 2018). Since
science is a core subject of STEM education and the subjects in STEM are
intercorrelated, integrated STEM education in science classrooms has great potential to
enhance students’ STEM motivation and interest in STEM careers, and also to increase
student participation and persistence in STEM learning and strengthen the future STEM
and STEM-related workforce.
With its rapid development and greater accessibility, technology has been
increasingly integrated into teaching and learning, and there have been tremendous
efforts of technology integration to promote STEM learning (Honey et al., 2014; Urban &
Favlo, 2016). As an emerging technology in K-12 education, 3D printing has gained
attention from teachers, administrators, and researchers, and many schools have
invested in 3D printing technologies. The iDigFossils project, namely the National
Science Foundation (NSF) funded project “iDigFossils: Engaging K-12 Students in
Integrated STEM via 3D Digitization, Printing and Exploration of Fossils” (PI: Dr. Pavlo
Antonenko. Award No. 1510410), was an integrated STEM education initiative aimed to
engage students in STEM learning through the integration of 3D printing technology in
K-12 science classrooms within the context of paleontology.
17
Problem Statement
The need for an adequately prepared STEM workforce has been increasing
(Honey et al., 2014). Meaningful integration of science, technology, engineering and
mathematics (STEM) in K-12 education is an important way to address the need, but it
is challenging (NRC, 2014). With the increasing access to 3D printing technology,
educators gradually discovered the power of integrating these technologies into
teaching, and many schools have 3D printers now (D. Thornburg, N. Thornburg, &
Armstrong, 2014). Integrating 3D printing technology into science classrooms is a
promising approach for meaningful STEM integration (Bull, Chiu, Berry, Lipson, & Xie,
2014). However, 3D printing is largely perceived as a recreational tool (Gonzalez &
Bennett, 2016). Furthermore, 3D printing technology has been mostly used in
undergraduate education, but the integration of 3D printing technology in K-12 science
classrooms is scarce. Although many schools have 3D printing technology, there is a
lack of meaningful integration of this technology into K-12 science classrooms.
Many external and internal factors may impact teachers’ intention and the
integration of 3D printing technology in their classrooms. Teachers’ beliefs about
technology integration have been found influential to teachers’ integration of
technologies in the classrooms (Ertmer, 2005; Ertmer et al., 2012). Especially when the
access to technologies, professional development, and technical and instructional
support are provided, teacher beliefs about technology integration become salient
barriers for teachers to meaningfully integrate technologies in their classrooms (Ertmer
& Ottenbreit-Leftwich, 2010). Moreover, relationships between teacher beliefs and
18
students’ cognitive and affective learning outcomes have been identified by previous
research (Zee & Koomen, 2016).
Regarding 3D printing technology, a newly emerged technology in education,
there is little evidence on how teachers’ beliefs about 3D printing technology integration
influence their integration of 3D printing technologies in the classrooms and how
teachers’ beliefs about 3D printing technology integration may potentially influence
students. Moreover, the influence of 3D printing technology integration in science
classrooms on students’ STEM motivation, 21st century skills, and interest in STEM
careers remains understudied.
Research Purpose
Guided by the social cognitive career theory, expectancy-value theory of
motivation, the Technology Integration Matrix (TIM), and the Technology Pedagogical
Content Knowledge (TPACK) framework, this study intended to examine the
relationships between teachers’ beliefs, their 3D printing technology integration in
science classrooms, and students’ STEM motivation, 21st century skills, and interest in
STEM careers within the context of the iDigFossils project. Specifically, this study
investigated: 1) how teachers’ beliefs are related to 3D printing technology integration;
and 2) how teachers’ beliefs and 3D printing technology integration predict students’
STEM motivation, 21st century skills, and interest in STEM careers.
The data sources for this study included: 1) teachers’ lesson plans for analyzing
teachers’ 3D printing integration in their science classrooms, including 3D printing
integration levels and STEM integration levels; 2) survey data on teachers’ beliefs,
including pedagogical beliefs, and value beliefs and self-efficacy beliefs about 3D
19
printing integration; and 3) pre-post survey data on students’ STEM motivation, 21st
century skills, and interest in STEM careers. The survey of teacher beliefs also included
open-ended questions to provide some qualitative data for understanding and
explaining the relationship between teachers’ beliefs and their 3D printing integration.
Correlational analysis was conducted to investigate the relationship between
teachers’ beliefs and their 3D printing integration levels and STEM integration levels.
Multilevel modeling analysis and multiple regression analysis were conducted to
examine how teacher beliefs, 3D printing integration levels, and STEM integration levels
predicted students’ STEM motivation, 21st century skills, and interest in STEM careers.
Research Questions and Hypotheses
This study was guided by the following overarching research question:
What are the relationships between teachers’ beliefs, their 3D printing integration
in science classrooms, and students’ STEM motivation, 21st century skills, and interest
in STEM careers? Specifically,
1. How are teachers’ beliefs correlated with their 3D printing integration in the science classrooms?
2. How do teachers’ beliefs and their 3D printing integration in the science classrooms predict students’ STEM motivation, 21st century skills, and interest in STEM careers?
It was hypothesized that teachers’ beliefs may be significantly and positively
correlated with their 3D printing integration including the 3D printing integration levels
and STEM integration levels. The higher teachers’ beliefs were, the higher the 3D
printing integration levels and STEM integration levels were. It was also hypothesized
that teachers’ beliefs, 3D printing integration levels, and STEM integration levels may
significantly and positively predict students’ STEM motivation, 21st century skills, and
20
interest in STEM careers. The higher teachers’ beliefs, 3D printing integration levels,
and STEM integration levels are, the higher students’ STEM motivation, 21st century
skills, and interest in STEM careers are.
Significance of This Study
The iDigFossils project was the first initiative and endeavor to systematically
integrate 3D printing in K-12 science classrooms across several states in the U.S. on a
large scale. The dissertation aimed to investigate how teachers’ beliefs are related to
their 3D printing integration in their science classrooms, and how teachers’ beliefs and
3D printing integration predict students’ STEM motivation, 21st century skills, and
interest in STEM careers within the context of the iDigFossils project. This dissertation
seeks to make the following contributions to educational practice and research: 1)
Investigate and analyze the practices of different teachers’ integration of 3D printing
technology in their science classrooms and contribute to the body of knowledge on 3D
printing integration in science classrooms; 2) Examine the relationship between
teachers’ beliefs and their 3D printing technology integration in the science classrooms
to inform educational practice on how to assist teachers with integrating 3D printing
technology in their science classrooms; 3) Shed light on the relationship between
teachers’ beliefs, their 3D printing integration, and students’ STEM motivation, 21st
century skills, and interest in STEM career, which may lay a foundation for future
research to examine the influence of teachers’ beliefs and their 3D printing integration
on students’ learning outcomes.
Definition of Key Terms
21st century skills – are important knowledge and skills students need to succeed in academic and career development, including critical thinking,
21
communication, collaboration, creativity, problem solving, and digital literacy (NRC, 2010; P21; PCAST, 2010).
3D printing technology: the “process of building a physical object, layer by layer, from a three-dimensional digital model” (Gonzalez & Bennett, 2016, p. 11).
First-order barriers – are external to teachers and are related to educational resources (including hardware and software), teacher training, and instructional support.
Integrated STEM education – is “an effort to combine some or all of the four disciplines of science, technology, engineering, and mathematics into one class, unit, or lesson that is based on connections between the subjects and real-world problems” (Moore et al., 2014, p. 38).
Pedagogy – is generally perceived as the science of teaching and it is “any conscious activity by one person designed to enhance learning in another” (Mortimore, 1999, p. 3).
Pedagogical beliefs – are commonly classified as beliefs on teacher-centered learning (behaviorist beliefs) and student-centered learning (constructivist beliefs) (Ertmer, Ottenbreit-Leftwich, Sadik, E. Sendurur, & P. Sendurur, 2012; Kim et al., 2013; Park & Ertmer, 2007; Tondeur, van Braak, Ertmer, & Ottenbreit-Leftwich, 2017).
Second-order barriers – are internal to the teacher and rooted in teachers’ underlying beliefs about teaching and learning (Ertmer, 1999), including teachers’ self-efficacy on technology integration, beliefs about how students learn, and perceived value of technology for teaching and learning (Ertmer et al., 2012).
Self-efficacy – refers to individuals’ beliefs in their abilities to complete tasks or achieve goals (Bandura, 1986, 1997).
Self-efficacy in technology integration – is teachers’ beliefs on their competence to integrate technology into teaching in order to facilitate student learning and achieve the teaching goals.
STEM – is primarily defined as composed of science, technology, engineering, and mathematics, but can also include other related areas.
Technology – 1) technology as tangible tools; 2) technology as techniques, strategies, and processes.
Technology integration – “the process of determining which digital tools and which methods for implementing them are the most appropriate responses to given educational needs and problems” (Roblyer & Doering, 2013, p.16).
22
Technology Integration Matrix (TIM) – a framework for effective technology integration with a focus on student-centered learning.
Technology Pedagogical Content Knowledge (TPACK) – a framework on teachers’ knowledge of technology integration, including content knowledge, pedagogy knowledge, and technological knowledge, and also the interactions between and among the three components, which are PCK (pedagogical content knowledge), TCK (technological content knowledge), TPK (technological pedagogical knowledge), and TPACK (Koehler & Mishra, 2009).
Technology value beliefs – are teachers’ beliefs about the value of integrating technology to facilitate teaching and learning and to achieve the instructional goals (Watson, 2006; Ottenbreit-Leftwich, Glazewski, Newby, & Ertmer, 2010).
Value beliefs – consist of intrinsic interest value, attainment value/importance, utility value/usefulness, and cost (Eccles et al., 1983). Intrinsic interest value is the enjoyment an individual gains from doing the task; attainment value is the importance of doing well on the task; utility value is how a task contributes to future plans; and cost refers to what the individual has to give up on other things in order to do the task as well as anticipated efforts the individual has to put into the task (Eccles et al., 1983; Wigfield, 1994).
Organization of This Dissertation
This dissertation consists of five chapters. Chapter 1 introduces the context of
the research, the problem statement, research purpose, research questions and
hypothesis, significance of this study, and definition of key terms that were used in this
dissertation. Chapter 2 provides a literature review on related research, theories, and
the conceptual framework. Chapter 3 expounds the methodology of this study. Chapter
4 presents the research findings. Chapter 5 discusses the research findings, provided
implications for educational practice and research, and generated conclusions of this
study.
23
CHAPTER 2 LITERATURE REVIEW
This literature review is structured based on the research purpose of this study:
how teachers’ beliefs correlated with their 3D printing integration and how teachers’
beliefs and their 3D printing integration predicted students’ STEM motivation, 21st
century skills, and interest in STEM careers in the context of the iDigFossils project. The
organization of this literature review can be viewed in Figure 2-1.
Figure 2-1. Organization of the literature review
This literature review begins with the general technology integration background
and definition to lay a foundation for the whole literature review. Following the
technology integration background and definition, this review analyzes and describes
Technology Integration Background and Definition
3D Printing Integration in K-12 Education
Integrated STEM Education via 3D Printing Integration
Models and Frameworks for Analyzing 3D Printing Integration
Teacher Beliefs and Technology Integration
Teacher Beliefs and Student Learning Outcomes
Students' STEM Motivation, Interest in STEM Careers, and 21st Century Skills
Conceptual Framework of This Study
24
3D printing integration in K-12 education, including how 3D printing technology has
been integrated in K-12 education, the influence of 3D printing technology integration,
and the benefits and challenges of 3D printing technology integration. After reviewing
3D printing integration in K-12 education, this review illustrates the iDigFossils project
which was an integrated STEM education project via 3D printing technology integration
in K-12 science classrooms and the benefits of integrated STEM education.
After illustrating the STEM integration of the iDigFossils project, this review
continues with the models and frameworks that can be used to analyze 3D printing
technology integration since a major component of this study was to analyze how
teachers have integrated 3D printing technology in their science classrooms. The
following two sections focus on teacher beliefs including pedagogical beliefs, self-
efficacy in technology integration, and technology value beliefs, and the relationship
with teachers’ technology integration and students’ learning outcomes. Finally, this
review focuses on students’ STEM motivation, interest in STEM careers, and 21st
century skills. This review concludes with the conceptual framework of the study, which
is informed by the literature and theories reviewed in this chapter.
Technology Integration Background and Definition
Nature of Technology
Before going further on the application of technology in education, it is necessary
to discuss the nature of technology. As we are immersed in all kinds of fancy digital
technologies nowadays, it is very easy to perceive technology as digital tools or
machines. However, technology is not merely a tool or machine that is tangible.
Technology should be seen as “a system of practical knowledge not necessarily
reflected in things or hardware” (Saettler, 1990, p. 3). Specifically, as Gendron (1977)
25
defined, technology is “any systematized practical knowledge, based on
experimentation and/or scientific theory, which enhances the capacity of society to
produce goods and services, and which is embodied in productive skills, organization,
or machinery” (as cited in Saettler, 1990, p.4). Therefore, we can understand
technology as a two-fold system, i.e., 1) technology as tangible tools; 2) technology as
techniques, strategies, and processes. In this vein, educational technologies can be
understood as educational tools, design, or instructional strategies that are used to
facilitate teaching and learning. In the context of 3D printing, the technology could be 3D
printers, 3D modeling software, 3D designing techniques, and also the strategies to
integrate the 3D printers, 3D modeling software, 3D designing techniques into teaching
and learning.
History of Technology in Education
Historically, many kinds of technological tools have been used for education,
starting from the very early murals, to paper, to radios, films, TVs, to computers,
projectors, smartboards, mobile phones, etc. The development of technology in
education gave rise to continuous debate on the influence of technology, specifically
media, on learning. The media-method debate originated from Clark and Kozma who
maintained different stances on the impact of media on learning and refuted each
other’s opinions back and forth in the 1980s and 1990s. Clark (1983) insisted that media
do not influence learning since no media comparison studies found effects of media on
learning. Furthermore, he suggested a moratorium on exploring the relationship
between media and learning and that future research should focus on the instructional
method instead of the media. Kozma (1991) disputed Clark’s separation of media and
method and thought of it as “an unnecessary schism between medium and method”
26
(Kozma, 1991, p. 205). Learning with media is “a complementary process within which
representations are constructed and procedures performed, sometimes by the learner
and sometimes by the medium” (Kozma, 1991, p. 179). Kozma (1991) reviewed
research on learning with a variety of media, including books, television, computers, and
multimedia and suggested that the technologies, symbol systems, and processing
capabilities of media can influence the learning process. He believed that media and
method are integral parts of instructional design. Responding to Kozma’s (1991)
criticism, Clark (1994a) published an article and concluded that media will never
influence learning, after examining media research of the past 70 years which showed
largely negative evidence.
It may be untenable to assert that media will never influence learning. Studies
reviewed by Clark (1983, 1994a) were overwhelmingly focused on the technology and
few authors indicated what the teachers did with the technology for their instruction at
the time when media were relatively new in education. Additionally, many researchers
were interested in studying which medium was more effective, neglecting what
instructional strategies were used to implement the media. Our attention should not be
on the technologies per se but how technologies are used to facilitate teaching and
learning. As Kozma (1991) stated, students will benefit most from the use of media with
certain capabilities if the capabilities are employed by the instructional method. Media
need to correspond to the particular learning situation to be effective for learning
(Kozma, 1991). Appropriate integration of media could enhance learning by taking the
advantages of media (Kozma, 1994). When designing and implementing instruction with
technology, it is important to consider the affordances of technology and under what
27
conditions the affordances of technology can influence learning. Teachers need to
understand the affordability of different technologies and integrate appropriate
technologies for instruction. The selection of media is critical in design because of
“learner preferences, available media, and the available time and funds” (Clark, 1994b,
p. 8). Technologies may have a novelty effect on learning as students are probably
most interested in the technologies when they first begin to use them. Teachers need to
carefully design instructions and use effective instructional strategies. Only by
integrating proper technologies and using effective instructional strategy will students’
learning be improved.
Nowadays, many emerging technologies such as virtual reality and augmented
reality, robotics, and 3D printing technology have gained much attention in education.
Researchers and educators have been studying how these technologies can be
integrated to make a real impact on students. While these technologies, especially 3D
printing in the context of this study, may seem cool and fascinating, we need to be
careful not to fall into the media-method debate of whether 3D printing can make an
impact on students. More importantly, it is how teachers integrate 3D printing into
teaching and learning that may facilitate students’ learning.
Defining Technology Integration
To define technology integration, it is first necessary to be clear about the
broader concept of educational technology. The Definition and Terminology Committee
of Association for Educational Communications and Technology (2018) issued a new
and soon to be published definition of educational technology,
Educational technology is the study and ethical application of theory, research, and best practices to advance knowledge as well as mediate and improve learning and performance through the strategic design,
28
management and implementation of learning and instructional processes and resources.
This omnibus definition suggests that educational technology is not confined to
educational tools or devices but the application of theory, research, and practices to
advance knowledge and improve learning. Therefore, technology integration is not just
about educational tools or devices per se but how they can be applied to theory,
research, and practices.
Researchers define technology integration in many different ways. Some are
very narrow and focus on the tools, but others not only addressed the tools, but also
methods for implementation, and the purpose of integration. According to Roblyer and
Doering (2013), technology integration refers to “the process of determining which
digital tools and which methods for implementing them are the most appropriate
responses to given educational needs and problems” (p.16). This definition indicates
that when integrating technology, teachers need to consider the educational needs and
problems and determine which tools and implementation methods can meet the
educational needs or solve the problems. Technology integration is not simply
incorporating digital technologies into teaching and learning. As Koehler, Mishra, & Cain
(2013) maintain, there are three core components for good integration of technology:
“content, pedagogy, and technology, plus the relationships among and between them”
(p. 14). A broad range of factors should be considered for effective technology
integration in the classrooms, including technology selection, instructional content,
learner characteristics, instructional strategies, and teaching and learning environment.
In order to select appropriate technology, it is important to match technology
affordances to teaching and learning need. Instructional content is critical for technology
29
selection in terms of what content will be taught and what technology has the capability
to meet with the need for delivering specific instructional content. We also consider the
interaction required during instruction for effective technology selection. We also have to
consider learner characteristics, including who the learners are, their age, their attention
span, motivation, and confidence to use technologies. Instructional strategy/pedagogy is
another important factor to be considered. We have to consider whether it is for
individualized instruction or for grouped instruction (Reiser & Gagné, 1983). Specific
learning tasks also impacts what functions of instructional strategy to select:
presentation, practice, or feedback (Richey, Tracey, & Klein, 2011). Gagné’s Nine
Events of Instruction can also guide instructional strategy selection. In addition,
teachers need to consider the learning context such as group size (Leshin, Pollack, &
Reigeluth, 1992), availability of resources (Romiszowski, 1981), and types of interaction
needed (Huddlestone & Pike, 2008).
3D Printing Integration in K-12 Education
Three-dimensional (3D) printing technology is the “process of building a physical
object, layer by layer, from a three-dimensional digital model” (Gonzalez & Bennett,
2016, p. 11). 3D printing technology uses materials such as plastics, ceramic, and
metals. Technically referred to as additive manufacturing or rapid prototyping, 3D
printing technology has revolutionized small-scale fabrication by allowing users to
create specialized objects with reasonable costs (Gonzalez & Bennett, 2016). People
have been excited about the possibilities of 3D printing technologies and regard it as
potentially revolutionary and disruptive. 3D printing is actually not a new technology. It
has been used in industry since the 1980s, but it was not until recently that 3D printing
technology has been adopted in education.
30
A search of literature related to 3D printing technology revealed that the majority
of the articles either focused on the functionality of 3D printing as a high-tech tool (such
as the use of 3D printing technology to design and/or develop tools for industry or
medical field, etc.), or integrating 3D printing technology in STEM classes in higher
education. There were very limited studies on 3D printing technology integration in K-12
education. During the past few years, the development in 3D printing technology and
decreasing cost makes it affordable to be introduced and explored in K-12 education.
Although 3D printing technology was invented about three decades ago, like many new
technologies, the availability of affordable 3D printing technology (for less than $5,000)
“suddenly changed the game from fantasy to reality” (Gonzalez & Bennett, 2016, p. 1)
until recently.
Nowadays, many schools have 3D printers and we can see all kinds of 3D
printed objects. However, many people think 3D printers are only for generating trinkets,
toys, and generally useless objects (Gonzalez & Bennett, 2016). There remains
deficiency in how 3D printing can be integrated for K-12 education, the evidence of
effectiveness of 3D printing integration in K-12 education, and the influence on teaching
and learning. In the following section, a review of the empirical literature related to 3D
printing integration in K-12 education is provided. The review is guided by three
questions: 1) How has 3D printing been integrated in K-12 education? 2) What is the
influence of 3D printing integration in K-12 education? 3) What are the benefits and
challenges of integrating 3D printing in K-12 education?
How 3D Printing Technology Has Been Integrated in K-12 Education
3D printing has been integrated in K-12 education in two major ways: 1) teaching
students about using 3D printing to create models or objects; 2) integrating it into
31
disciplines to teach other content areas. There are also studies that integrated 3D
printing to facilitate the needs of special education students, for example, students with
visual impairments (e.g., Al-Rajhi, Al-Abdulkarim, Al-Khalifa, & Al-Otaibi, 2015; Grice,
Christian, Nota, & Greenfield, 2015; Jo, 2016), and students with various disabilities
(e.g., Mcloughlin et al., 2016). This review focuses exclusively on the two major ways of
3D printing integration in K-12 education. Table 2-1 provides an overview of how 3D
printing technology was integrated in each study.
Teaching students how to use 3D printing technology
In several studies, the focus of 3D printing integration was to teach 3D printing to
students and have students design and create 3D objects (e.g., Bicer et al., 2017;
Chao, Po, Chang, & Yao, 2016; Chen, Zhang, & Zhang, 2014; Kwon, 2017; Nemorin &
Selwyn, 2017).
In Chen et al. (2014), 23 elementary students learned about how to design 3D
models and they constructed with 3D printers. Compared to a traditional teaching group
which did not use 3D printing, Chen et al. (2014) found significant improvement in the
spatial ability of boys. In a study that was conducted with 24 high school students who
used 3D printing software to design and print the models with 3D printers to make
cultural and creative products, Chao et al. (2016) found the students’ creativity was
enhanced after designing and creating the 3D printed products to show culture. In
another study, Bicer et al. (2017) taught 95 high school students in a summer camp how
to design and print 3D models. The students designed and printed 3D models and
explained the mathematics and science underlying the object, the strengths and
weaknesses of their designs, and the educational purpose of their 3D printed objects.
This study found that the students had significant and positive changes in their
32
perceptions about the need and importance for creativity and problem-solving skills in
STEM fields. Similarly, Kwon (2017) had 47 middle and high school students in a
summer camp use Google SketchUp and XYZware software to design and print their
own unique 3D objects. The students demonstrated a significant increase in
mathematical skills, learning motivation, and technical skills using the software.
Most of these reviewed studies that taught students how to use 3D printing and
had students design and print 3D objects found positive results either in students’
learning performance, learning motivation, spatial ability, technical skills, creativity, or
perceptions on the necessity of creativity and problem-solving skills. However, an
ethnographic study conducted by Nemorin and Selwyn (2017) with a high school
teacher and his students found negative results. This ethnographic study focused on the
experiences and perceptions of the teacher and individual students and provided in-
depth details. In this study, the teacher and students had an eight-week 3D printing
course in which students learned and designed race cars with a 3D modeling software
SketchUp and printed out the cars. However, the teacher and students were
disappointed that many cars looked great but were not working due to many reasons. A
student perceived SketchUp as boring because the student spent too much time on it
and was frustrated with the problems he had when designing the model. Another
student preferred making objects with hands instead of using computers and would not
do a 3D printing project again because the student felt 3D printing was nothing special.
Although teaching 3D printing to students and having students design and create
3D objects had positive influence on students in most of these reviewed studies, one
study that was reviewed found negative influence on students. Engaging students
33
deeply in the designing and printing process might be beneficial for some students in
some situations, but we cannot generalize that it would always be beneficial to all
students. All these reviewed studies focused on teaching students designing and
creating 3D objects regardless of disciplines. There were also studies that focused on
the integration of 3D printing technology to facilitate students’ learning in different
disciplines.
Integrating 3D printing technology into disciplines
Most of the studies integrated 3D printing into disciplines such as history/social
studies (Maloy, Trust, Kommers, Malinowski, & LaRoche, 2017), mathematics (Ng,
2017), science (Koehler, 2017), engineering-related disciplines (Chien, 2017; Chien &
Chu, 2018; Hsiao, Chen, Lin, Zhuo, & Lin, 2019; Weber, Kotsopoulos, & Senger, 2017),
and integrated STEM in the context of Paleontology (Grant, MacFadden, Antonenko, &
Perez, 2017).
In Maloy et al.’s (2017) yearlong study with 13 middle school in-service students,
10 pre-service teachers, and 4 classes of students, 3D printing was integrated into
history/social studies classes. The students participated in carefully designed 3D
printing projects which were connected to the curriculum standards. The students
designed 3D printing models and created objects for different history/social studies
topics. Maloy et al. (2017) found 3D printing allowed students to express their thinking
and ideas in a new way, helped students remember information, visualize causes and
consequences of events, and be creative in school projects. Integrating 3D printing into
middle school mathematics classes, Ng (2017) had students learned how to use
Tinkercad and design 3D models of keychains to facilitate their learning about the
volume of solids, a math topic. The students’ math learning was enhanced, and
34
students had positive attitudes about their experience that they could design, create,
and make personalized keychains for themselves. In another study which integrated 3D
printed objects in science classes with middle school students who had visual
impairments, Koehler (2017) compared the use of 3D printed objects with using
traditional tactile graphics for learning a science topic of plate tectonics. This study
found that although students in each group had increased conceptual understanding
and decreased misconceptions, there were no significant differences between the two
groups, indicating no absolute benefit of using 3D printed objects.
In the previously reviewed studies, 3D printing was integrated into
historical/social studies, mathematics, and science disciplines. However, 3D printing
was mostly integrated into engineering-related disciplines. In Chien’s (2017) study, 3D
printing was integrated into pre-engineering classes with 108 high school students.
Chien (2017) developed an eight-week course to teach the design of CO2 dragsters
using 3D printers and free 3D digital modeling software. The students learned 3D
printing and used 3D digital modeling software to design and create CO2 dragsters.
Chien (2017) compared the 3D printing group (n = 108) to a traditional handmade group
(n = 74) and found that the 3D printing group had significantly better engineering
concepts learning and creativity in terms of novelty and sophistication of their design
than the handmade group, but there was no significant difference in the overall learning
performance. Chien and Chu (2018) integrated 3D printing into a high school STEAM
engineering design curriculum. The students designed and created CO2 racing cars with
3D printing and participated in racing activities. Chien and Chu (2018) compared the 3D
printing group (n = 108) to a handmade group (n = 36) and found the 3D printing group
35
had significantly better creativity in terms of sophistication and appearance of the design
and accuracy of predictions regarding which car was going to win the race, but there
were no significant differences in overall learning performance and creativity in terms of
novelty.
Hsiao et al. (2019) integrated 3D printing into an 11-week pre-engineering course
with high school students. The students learned how to use 3D printing technology and
3D modeling software, and they used 3D printing technology to print out their 3D
designs of windmills. Hsiao et al. (2019) found that students who participated in the 3D
printing project outperformed the control group that used traditional hands-on tools with
lectures in abstract scientific concepts and hands-on ability. Weber et al. (2017)
integrated 3D printing into a 6-week design engineering course with middle school
students. The students designed and constructed cube puzzles and then printed the
models using a 3D printer. From the teacher and students’ reflection, Weber et al.
(2017) reported that the task provided opportunities for students to develop visual-
spatial reasoning and it inspired significant student engagement and further learning.
In the study of Grant et al. (2017), 3D scanning, printing, and analysis of teeth of
the Neogene shark Carcharocles megalodon were integrated into two middle and two
high school science classes in the context of paleontology. As an endeavor to integrate
all four STEM disciplines through 3D printing integration in science classes, Grant et al.
(2017) found increased students’ engagement and enthusiasm in the Megalodon topic.
36
Table 2-1. How 3D printing has been integrated in K-12 education
Study Subject/Content Area
how 3D printing was integrated Integration Category Student interaction with 3D printing
Chen et al. (2014)
NA Students learned about how to design 3D models and operated on 3D printers.
3D printing as the learning content (Students learn 3D printing to improve spatial ability)
design and print
Maloy et al. (2017)
History/Social studies
Teachers and preservice teachers partnered and participated in a 3D printing workshop and developed curriculum projects to connect 3D printing to their school curriculum with curriculum standards.
Teachers implemented 3D printing lessons.
Students designed 3D models and created 3D objects.
To integrate into the curriculum to teach other content areas
design, print, and use for learning
Koehler (2017)
Science Students used 3D printed objects to study plate tectonics.
To integrate into the curriculum to teach other content areas
use printed objects
Hsiao et al. (2019)
pre-engineering Students learned how to use 3D printing technology, 3D modeling software, and used 3D printing technology to print out their 3D designs of windmills.
To integrate into the curriculum to teach other content areas
design and print
Weber et al. (2017)
design engineering
Students designed and constructed cube puzzles and then printed these models using a 3D printer.
To integrate into the curriculum to teach other content areas
design and print
37
Table 2-1. Continued
Study Subject/Content Area
how 3D printing was integrated Integration Category Student interaction with 3D printing
Chao et al. (2016)
NA Students used 3D software to design models and then printed the models with the 3D printer to make cultural and creative products.
3D printing as the learning content (To create 3D products to show culture and creativity)
design and print
Ng (2017)
Mathematics Students learned how to use Tinkercad and designed 3D models of keychains to facilitate learning volumes of solids.
To integrate into the curriculum to teach other content areas
design
Bicer et al. (2017)
Informal STEM learning program (summer camp)
Students designed 3D models with CAD software and printed out the 3D models with 3D printers. Students explained the mathematics and science underlying the object, strengths and weakness of their designs, and explained the educational purpose for the object.
3D printing as the learning content (To improve creativity and problem solving skills)
design, print, and use
Grant et al. (2017)
Paleontology The activity used 3-D scanning, printing, and analysis of teeth of the Neogene shark Carcharocles megalodon as an example of how paleontology has the potential to integrate all four STEM disciplines and to help develop student interest and motivation in STEM.
To integrate into the curriculum to teach other content areas
print and use
Chien (2017)
Pre-engineering The researcher developed a course to teach the design of CO2 dragsters using 3D printers and free 3D digital modeling software. Students learned 3D printing and used 3D digital modeling software to design and create a CO2 dragster.
To integrate into the curriculum to teach other content areas
design, print, and use
38
Table 2-1. Continued
Study Subject/Content Area
how 3D printing was integrated Integration Category Student interaction with 3D printing
Kwon (2017)
summer camp Students used Google SketchUp and XYZware software to design and print 3D objects. The learning objective and outcome were to print students’ unique 3D object.
3D printing as the learning content
design and print
Nemorin and Selwyn (2017)
3D printing course
Students designed race cars with SketchUp, a 3D modeling software, and printed the cars.
3D printing as the learning content
design and print
Chien and Chu (2018)
STEAM engineering design curriculum
Students designed and created CO2 racing cars with 3D printing and participated in racing activities.
To integrate into the curriculum to teach other content areas
design, print, and use
39
The Influence of 3D Printing Integration
The 3D printing technology has been integrated in different grade levels with
varied implementation durations and has had diverse impacts on students (view Table
2-2 for details). From the studies that have been reviewed on how 3D printing has been
integrated in K-12 education, it was found that the 3D printing has influence in three
main aspects: 1) cognitive learning performance outcomes such as conceptual
understanding and overall learning performance; 2) abilities and skills such as spatial
ability, creativity, hands-on ability, technical skills; and 3) affect learning outcomes such
as perceptions, attitude, engagement, and motivation.
Especially, in the study of Maloy et al. (2017) which integrated 3D printing to
middle school History/Social Studies classrooms, the teachers reported that many of the
students expressed interest in future career paths in STEM fields, in part because the
students felt comfortable with the 3D modeling software. In addition to express
themselves using spoken and written words, 3D printing provided the students an
opportunity to generate a tangible object to communicate their thinking and ideas
(Maloy et al, 2017). As 3D printing was a new technology not just for students but also
for teachers, the teachers and students were working together as partners instead of
adult experts and novice learners, which altered the teacher-as-expert/student-as-
novice relationship (Maloy et al., 2017). Maloy et al. (2017) also indicated that 3D
printing design activities changed the traditional teacher-centered presentations in
which students passively receive and remember information to student-centered active
learning, problem solving, design-based thinking, and collaborative group work.
Students noted that creating designs and objects helped them remember the historical
facts and events and printing objects allowed them to visualize the causes and
40
consequences of historical events. 3D printing not only helps students remember
specific information but also provides students the opportunities to be creative in their
school projects and many of the students took initiatives to use 3D printing in their other
school projects (Maloy et al., 2017).
Trust and Maloy (2017) surveyed 51 teachers about the impact of their 3D
projects on student learning and the teachers reported that their students developed a
number of skills, including 3D modeling, creativity, technology literacy, problem-solving,
self-directed learning, critical thinking, and perseverance, which are the essential 21st
century skills as the authors summarized.
Benefits and Challenges of 3D Printing Integration
3D printing has great potential to engage students in hands-on learning and
experiential learning which brings something impossible or unreachable in real life to
students and allows students to hold and touch 3D objects. 3D printing can make
abstract knowledge in visualizable ways. Holding objects by hand enables students to
learn by doing and can potentially motivate students to learn. 3D printing can be a great
teaching resource to enhance students’ learning performance, abilities of problem-
solving, critical thinking, communication and collaboration, and affective learning
outcomes such as positive attitudes, engagement, and motivation. In addition, 3D
printing can be a powerful resource for integrated STEM education, and it can not only
serve as a tool to create models for students but also a tool for students to design and
create their own models.
Although 3D printing integration has many potential benefits, it has some
challenges that have to be considered and addressed when integrating it into teaching
and learning. 3D printing integration requires a large amount of time and preparation
41
because it takes many hours to print a relatively small object, which may make it not
feasible to have students print out 3D objects during regular class time. 3D printing
integration has high requirements on the temperature for the 3D model layers to build
layer by layer and the model may not be successfully printed out as designed. The 3D
modeling software can also be difficult for teachers and students. The technical
difficulties can pose challenges to teachers and students. In their educator workshops
(Maloy et al., 2017), the teachers struggled to use 3D printing, which was an unfamiliar
technology for them, and in class students were openly frustrated and needed
assistance to use the 3D modeling software. 3D printing can also cause “frustration,
physical fatigue, mental exhaustion, tedium and occasional panic” (p. 592) if the activity
is not appropriately integrated. Maloy et al. (2017) reported that teachers in their study
recommended providing students with time to explore Tinkercad (a 3D modeling
software) to get more comfortable with the tool before starting the projects.
In addition to the technical challenges, it can also be difficult for teachers to
integrate 3D printing to connect to mandated course standards. Maloy et al. (2017)
indicated that both teachers and students in the study found the 3D printing idea
intriguing, but they were unsure how to connect 3D printing to the school district-
mandated and standards-driven curriculum topics. In most of the studies, whether and
how the teachers were trained to integrate 3D printing in their classes were not
mentioned or not provided. Very few studies provided professional development for
teachers to learn how to integrate 3D printing. In the study of Maloy et al. (2017),
teachers and preservice teachers partnered and participated in a 3D printing workshop
42
and developed curriculum projects to connect 3D printing to their school curriculum with
curriculum standards, which was helpful for them to implement the 3D printing lessons.
Professional development for teachers is essential for teachers to effectively
integrate 3D printing in their classes but it does not guarantee positive impacts on
students. As Schelly, Anzalone, Wijnen, and Pearce (2015) pointed out:
Simply putting 3-D printers in schools does not automatically provide a better learning environment or make students into maker and creator. Providing teachers with the training necessary to transform them into makers may make this more likely, but again does not guarantee that students will be transformed into creators rather than passive consumers of knowledge. (p. 235)
Effective 3D printing integration can also be influenced by teachers’ knowledge and
beliefs on technology integration. It is a challenging but important issue of how 3D
printing can be effectively integrated and make positive impacts on students.
43
Table 2-2. 3D printing integration in K-12 education and the impacts on student
Study Participants Duration Research Design Outcome Variables
Results
Chen et al. (2014)
46 primary students (mean age = 10)
7 months
Pre-post experimental design with two groups: 3D printing group (n = 23) vs. traditional teaching group (n = 23)
Spatial ability Significant improvement in spatial ability
Maloy et al. (2017)
13 middle school (eighth grade) in-service teachers and 10 preservice teachers; 4 classes of students
1 year Pre-survey on teachers’ 3D printing knowledge, skills, attitudes, and ideas, and post-survey on how teachers learned to incorporate 3D printing into their lessons; observation of 3D printing lesson implementation; focus groups with students
Teacher and student experiences and perceptions
It was challenging to connect 3D printing to standards-driven curriculum topics.
Teachers and students experienced technical challenges.
3D printing allows students to express thinking and ideas in a new way; helps students remember information; helps students visualize causes and consequences of events; help students be creative in school projects
Koehler (2017)
5 middle school science classroom students with visual impairments
3 weeks Qualitative study with two groups: 3D printed objects group (n = 3 students) vs. traditional tactile graphics group (n = 2).
Pre-post student interviews, student journals, audiotaped instructional sessions.
Conceptual understanding of plate tectonics; Misconceptions related to plate tectonics and associated geoscience concepts.
Increased conceptual understanding and decreased misconceptions for each group;
No significant differences in student conceptual understanding between the two groups.
44
Table 2-2. Continued
Study Participants
Duration
Research Design Outcome Variables
Results
Hsiao et al. (2019)
184 10th-grade students from five classes
11 weeks (for a total duration of 960 min)
Quasi-experimental design: Experimental Group 1 (EG 1: n = 74) which used 3D printing with ELS (experiential learning strategies), Experimental Group 2 (EG2: n = 36) which used 3D printing with lecture, Control Group (n = 74) which used traditional hands-on tools with lecture.
EG1 and EG2 both learned how to use 3D printing technology, 3D modeling software, and used 3D printing technology to do the actual practices.
Comprehension of abstract scientific concepts; Hands-on ability
All three groups improved their comprehension of abstract scientific concepts.
Abstract scientific concepts understanding: EG 1 > CG, EG2 > CG, no significant difference between EG1 and EG2.
Hands-on ability: EG1 > EG2 > CG.
Weber et al. (2017)
one classroom teacher and 22 students in 7th grade
6 weeks
Teacher and student reflections Visual-spatial reasoning; student engagement
The authors reported the task provided opportunities for students to develop visual-spatial reasoning, the 3D printer inspired significant student engagement and further learning.
Chao et al. (2016)
24 high school students
18 hours in 9 classes
Assessment on students’ design
Visual design abilities of cultural and creative goods
Enhanced creativity
45
Table 2-2. Continued
Study Participants Duration Research Design Outcome Variables
Results
Ng (2017)
Junior secondary students (aged 13 - 15) (sample size not available)
4 days Video recordings of the students’ communications, students’ calculations, and written
reflections
NA Enhanced math learning and positive attitude about their experience that they could “design”, “create”, and “make” personalized keychains for themselves.
Bicer et al. (2017)
95 high school students
2 weeks Pre-post surveys Students’ perceptions about creativity and problem solving skills in STEM disciplines
Students’ perceptions about the need for creativity in STEM fields were statistically significantly changed. Students indicated creativity was important for STEM fields and for engineering in particular and problem solving skills were essential for being successful in a STEM career. Cohen’s d effect sizes for students’ perceptions about creativity and problem solving skills in STEM disciplines at d = 0.61 and d = 0.66
Grant et al. (2017)
2 middle and 2 high schools
NA Case study, survey and observation
Student engagement
Student engagement increased. Increased enthusiasm in the Megalodon topic.
46
Table 2-2. Continued
Study Participants Duration Research Design Outcome Variables
Results
Chien (2017)
182 Grade 10 students
8 weeks Quasi-experimental design: a 3D Printing group with 108 students and a Handmade group with 74 students.
Engineering concepts learning; Overall learning performance of STEM knowledge, engineering drawing exercises, etc.; Creativity.
Engineering concepts learning: 3D Printing group > Handmade group
Overall learning performance: No significant difference;
Creativity: 3D Printing group significantly outperformed the Handmade group in terms of novelty and sophistication of their dragsters.
Trust and Maloy (2017)
51 teachers NA Survey of asking teachers the impact of 3D projects on student learning.
The kinds of 3D projects teachers were doing with students;
Skills or knowledge students were developing by participating in the projects
Teachers reported their students developed several skills, including 3D modeling, creativity, technology literacy, problem-solving, self-directed learning, critical thinking, and perseverance.
Kwon (2017)
47 secondary school students
2 weeks Pre-post surveys Mathematical skills, motivation, technical skills
Statistically significant increase in mathematical skills, learning motivation, and technical skills.
47
Table 2-2. Continued
Study Participants Duration Research Design
Outcome
Variables
Results
Nemorin and Selwyn (2017)
High school teacher and students (sample size not available)
Eight weeks
Ethnographic research
NA Only six cars were functioning. Many cars looked great but were not working due to many reasons. Teachers and students were disappointed. Students were excited and having fun at first and then felt bored. A student said SketchUp was boring because he spent too much time on it and he felt like giving up sometimes because he kept having to go back and change things because the models he designed were the wrong size or they didn’t fit together, which made him frustrated a lot. A student preferred making things with hands instead of using computers and would not do a 3D printing project again because he felt 3D printing was nothing special.
Chien and Chu (2018)
144 high school students
Eight weeks (two 50-min classes per week for 800 min)
Experimental design: 3D Printing group (n = 108) and handmade group (n = 36)
Novelty and sophistication of design;
Consistency of design (whether the appearance of the car was consistent);
Accuracy of predictions of the winning car;
Overall learning outcomes.
Novelty: no significant difference;
Sophistication: 3D printing group > handmade group;
Consistency: 3D printing group > handmade group;
Accuracy of predictions of the winning car: 3D printing group > handmade group;
Overall learning outcomes: no significant difference
48
Integrated STEM Education via 3D Printing Integration
Sanders (2009) defined integrated STEM education as “approaches that explore
teaching and learning between/among any two or more of the STEM subject areas,
and/or between a STEM subject and one or more other school subjects” (p. 21) More
specifically, Moore et al. (2014) defined integrated STEM education as “an effort to
combine some or all of the four disciplines of science, technology, engineering, and
mathematics into one class, unit, or lesson that is based on connections between the
subjects and real-world problems” (p. 38) and indicated that the STEM learning content
objectives primarily focused on one subject and learning contexts can be from other
STEM subjects. Kelley and Knowles (2016) also emphasized the importance of focusing
on real-world problems by stating that “an integrated approach seeks to locate
connections between STEM subjects and provide a relevant context for learning the
content. Educators should remain true to the nature in which science, technology,
engineering, and mathematics are applied to real-world situations” (p.3).
Integrated STEM learning experiences can enhance students’ interest and
motivation in STEM which leads to continuous engagement in STEM learning (Maltese
et al., 2014). Tai et al. (2006) surveyed 12,000 middle school students and found
students’ interest in STEM was a significant predictor of their career choices. Sadler et
al. (2012) found that students’ interest in STEM at the start of high school was a key
predictor of STEM career interest when they graduated. Maltese and Tai (2011) also
found that students who had interests in STEM were more likely to select and complete
a bachelor’s degree in STEM disciplines.
The National Science Foundation (NSF) funded project “iDigFossils: Engaging K-
12 Students in Integrated STEM via 3D Digitization, Printing and Exploration of Fossils”
49
(PI: Dr. Pavlo Antonenko. Award No. 1510410) was established with the spirit of STEM
integration by integrating 3D printing technology in K-12 science classrooms in the
context of paleontology, a rich and interdisciplinary science that examines life in deep
time. Paleontology integrates a variety of disciplines such as geology, biology,
anthropology, environmental science, and oceanography, providing great opportunities
for integrated STEM learning (Maltese et al., 2014) and to enhance students’ interest in
STEM. According to Grant et al. (2016), learning activities that integrated 3D scanning
and printing technologies with paleontology are a pathway for students to not only
enhance STEM learning through gathering data by analyzing 3-D printed fossils,
developing 3D modeling knowledge by using software, using mathematical estimations,
and building connections among STEM disciplines, but also to develop 21st Century
skills, such as collaboration, communication, critical thinking, problem solving, and
creativity (Partnership for 21st Century Skills, 2011).
In the following, an example iDigFossils lesson plan was illustrated in terms of its
integrated STEM approach. In the lesson plan, the iDigFossils activities were designed
with the instructional approach of integrated STEM by integrating science
(paleontology), technology, and math. Students built math skills by measuring and
calculating the 3D printed Hominins teeth. By comparing and contrasting the tooth sizes
students learned about the hominins’ diet. In the inquiry-based learning activities,
students played the role of paleoanthropologists and performed hands-on activities to
investigate a real-world problem of how hominins adapt to a variety of environments.
Students also worked collaboratively in teams to measure and estimate the teeth size
and discuss the similarities and differences of the teeth and how the teeth would
50
indicate different diet of the hominins. The iDigFossils activities were consistent with
how the authors defined integrated STEM education and specifically what Johnson
(2013) stated about integrated STEM, “integrated STEM education is an instructional
approach, which integrates the teaching of science and mathematics disciplines through
the infusion of the practices of scientific inquiry, technological and engineering design,
mathematical analysis, and 21st century interdisciplinary themes and skills” (p. 367).
Integrated STEM activities make learning more connected and relevant for
students (Stohlmann, Moore, & Roehrig, 2012), encourage students’ imagination and
curiosity and increase their motivation to learn (Laboy-Rush, 2011), and support
students’ interest development (Honey et al., 2014). The iDigFossils activities have
great potential to promote students’ motivation and career interest in STEM through
integrated STEM activities. These activities were also very likely to enhance students’
21st century skills. For instance, students develop critical thinking when they compare
and contrast the teeth size and analyze the diet of different hominins; problem solving
skills when they work on the problem of how different hominins adapt to a variety of
environments; communication and collaboration skills when they work in teams to
measure and calculate the teeth, and analyze the hominins’ teeth size and their diet;
and increase digital literacy by using 3D printers to print out fossil models.
Models and Frameworks for Analyzing 3D Printing Integration
An important part of this study was to analyze teachers’ 3D printing integration in
the classrooms, i.e., in what ways the teachers integrated 3D printing technology and
the levels of integration. This study used the Technology Integration Matrix (TIM) and
Technological Pedagogical Content Knowledge (TPACK) as the frameworks for 3D
printing integration analysis.
51
Technology Integration Matrix (TIM)
There are many technology integration frameworks. One of the most influential
frameworks is the Technology Integration Matrix (TIM). Technology Integration Matrix
(TIM) was created by the Florida Center for Instructional Technology (FCIT) from 2003
to 2005 and updated in 2011 (Welsh, Harmes, & Winkelman, 2011) to support students
in learning skills that are necessary for success in the 21st century (Harmes, Welsh, &
Winkelman, 2016). TIM provides a framework for effective technology integration with a
focus on student-centered learning. TIM is a two-dimensional matrix with five
characteristics of meaningful learning environments as the row and five levels of
technology integration as the column, resulting in 25 cells with each cell representing a
level of technology integration for each of the characteristics of the learning
environment. The 25-cell matrix is available at the FCIT website
https://fcit.usf.edu/matrix/matrix/.
The two dimensions of TIM are pedagogy and technology, with the unit of focus
being a lesson (Harmes et al., 2016). The technology aspect is composed of five levels
of technology integration: Entry, Adoption, Adaptation, Infusion, and Transformation, a
continuum from teacher-centered passive instruction to student innovative use of
technology in higher-order learning activities (Harmes et al., 2016). At the entry level,
teachers use technologies to deliver instruction and students do not have direct access
to the technologies. Even if students have some access to the technologies, the
purpose is to learn facts or basic skills through rote practice. At the adoption level,
teachers still dominate the use of the technologies and students have some but very
limited access to the technologies and use them for discrete tasks, which only require
procedural understanding. At the adaptation level, technologies are more integrated in
52
the lesson and students have more access to the technologies. Teachers still determine
when to use the technologies and students may begin exploring how to best use the
technologies. At the infusion level, students have full access to the technologies through
their study and the technologies are for learning rather than the technology tools per se.
Teachers guide students’ decision-making when using the technologies for learning.
Transformation is the highest level of technology integration. Teachers guide students’
use of the technologies and students can self-direct their use of the technologies.
Different from infusion level, transformation level of technology integration facilitates
higher-order learning activities that would otherwise be impossible or difficult without
using the technologies.
The pedagogical aspect focuses on five characteristics of meaningful learning
environments: Active, Collaborative, Constructive, Authentic, and Goal-Directed, which
were based on the work by Howland, Jonassen, and Marra (2012). The five
characteristics of meaningful learning enable students to “engage in higher-order
thinking and focus on real-world skills” (Harmes et al., 2016, p. 144). The TIM levels
describe a continuum of pedagogical approaches along with the levels of technology
integration from Entry to Transformation, reflecting four underlying differences:
ownership of learning, characterization of knowledge, use of technology tools, and
instructional focus (Harmes et al., 2016). The progression across levels of technology
integration of TIM can be viewed in Figure 2-2.
53
Figure 2-2. TIM with progression across levels of integration
Technological Pedagogical Content Knowledge (TPACK)
A comprehensive and prevalent technology integration framework on teachers’
knowledge of technology integration is the Technological Pedagogical Content
Knowledge (TPACK) framework, which has been adopted nationally and even
internationally. Teachers’ knowledge of technology integration is essential for effectively
integrating technologies into their teaching. Technological Pedagogical Content
Knowledge (TPACK) framework (previously named as TPCK), was conceptualized by
Mishra and Koehler (2006) based on Shulman’s (1986, 1987) description of
pedagogical content knowledge (PCK). Mishra and Koehler (2006) extended the PCK
framework by adding the component of technological knowledge and the interaction of
technological knowledge with other domains of knowledge in PCK. Different from
Technology Integration Matrix (TIM) which only has two dimensions: pedagogy and
technology, the TPACK framework incorporates the dimension of content. In TPACK
framework, there are three basic components, namely content knowledge, pedagogy
54
knowledge, and technological knowledge, and also the interactions between and among
the three components, which are PCK, TCK (technological content knowledge), TPK
(technological pedagogical knowledge), and TPACK (Koehler & Mishra, 2009). In a
nutshell, TPACK framework addresses the integration of technology, pedagogy, and
content knowledge in teaching and the intersections between these three domains of
knowledge. Figure 2-3 is the TPACK framework developed by Mishra and Koehler
(2009).
Figure 2-3. The TPACK framework and its knowledge components. Reprinted from http://tpack.org.
Content knowledge is teachers’ knowledge about the content in the subject
matter that will be learned or taught, including “knowledge of concepts, theories, ideas,
organizational frameworks, knowledge of evidence and proof, as well as established
practices and approaches toward developing such knowledge” (Koehler & Mishra, 2009,
p. 64; Shulman, 1986). Accurate, comprehensive, and deep knowledge of the content is
55
a fundamental requirement for teachers to teach students with appropriate and
adequate knowledge.
Pedagogical Knowledge is teachers’ knowledge about the pedagogy, namely,
how to teach, related to the processes and methods of teaching and educational
purposes and values. Pedagogical knowledge includes knowledge about “techniques or
methods used in the classroom; the nature of the target audience; and strategies for
evaluating student understanding” (Koehler & Mishra, 2009, p. 64). Pedagogical
knowledge requires teachers to understand how students construct knowledge and
acquire skills and how they develop positive attitudes toward learning (Koehler &
Mishra, 2009).
Technological Knowledge is teachers’ knowledge about technologies and the
fluency of using information communication technologies (ICT). Teachers need to
understand ICT to apply technologies in their teaching.
Pedagogical Content Knowledge is the knowledge of pedagogy for teaching
specific content, consistent with Shulman’s (1986, 1987) idea of pedagogical knowledge
that is applicable to the teaching of specific content. This knowledge has requirements
on both pedagogical knowledge and content knowledge and also appropriate pedagogy
for the specific content.
Technological Content Knowledge is teachers’ knowledge about what
technologies are appropriate for teaching the specific content. Technologies can afford
or constrain the type of content that can be taught, and the content can limit the types of
technologies that can be used (Koehler & Mishra, 2009). Teachers need to understand
the content and select technologies that can afford to teach the content.
56
Technological Pedagogical Knowledge is teachers’ knowledge about the
functions of specific technologies when using technologies to fulfill their pedagogical
design and instructional strategies. Teachers need to consider the affordances and
constraints of the technologies and how the technologies can facilitate the way they
would like to teach.
Technological Pedagogical Content Knowledge is the highest and most
comprehensive level of knowledge for effective technology integration. It is a
combination of the basic components of content knowledge, pedagogical knowledge,
and technological knowledge, and all the interactions among content, pedagogy, and
technological knowledge. Specifically, as Koehler and Mishra (2009) explained:
TPACK is the basis of effective teaching with technology, requiring an understanding of the representation of concepts using technologies; pedagogical techniques that use technologies in constructive ways to teach content; knowledge of what makes concepts difficult or easy to learn and how technology can help redress some of the problems that students face; knowledge of students’ prior knowledge and theories of epistemology; and knowledge of how technologies can be used to build on existing knowledge to develop new epistemologies or strengthen old ones. (p. 66)
Teachers need to develop not just knowledge in the key domains (Content, Pedagogy,
and Technology) but also knowledge of how these domains interrelate so as to integrate
technology effectively into their teaching.
Teaching with technology is challenging, especially when integrating new
technologies. The TPACK framework provides a comprehensive view of knowledge of
effective technology integration, but successful teaching with technology requires
teachers to continually create, maintain, and re-establish a dynamic equilibrium among
all components of TPACK framework (Koehler & Mishra, 2009). In addition to teachers’
technology integration knowledge, whether teachers can continually create, maintain,
57
and re-establish the equilibrium of all components of TPACK also depends on a number
of factors such as teachers’ beliefs of technology integration.
Teacher Beliefs and Technology Integration
Teachers may have designed and implemented 3D printing integration in a
variety of ways. What factors influence their 3D printing integration? To lay a foundation
for understanding the potential factors that may have influenced teachers’ 3D printing
integration, it is necessary to review the factors that can influence teachers’ technology
integration in the classrooms.
With the rapid development of information and communications technology (ICT),
teachers have been integrating technology into classrooms to facilitate teaching and
learning, and a large body of research has been conducted to examine the
effectiveness of technology integration and what factors or barriers impact the practice
of technology integration in K-12 classrooms. According to Ertmer (1999), there are two
types of barriers that can impact teachers’ integration of technology into classrooms,
external (first-order) barriers and internal (second-order) barriers. First-order barriers
are external to teachers and are related to educational resources (including hardware
and software), teacher training, and instructional support. Second-order barriers are
internal to the teacher and rooted in teachers’ underlying beliefs about teaching and
learning (Ertmer, 1999). Second-order barriers include teachers’ self-efficacy on
technology integration, beliefs about how students learn, and perceived value of
technology for teaching and learning (Ertmer et al., 2012).
Two decades ago, teachers had limited access to technologies, professional
development, and instructional support. First-order barriers were significant obstacles
for technology integration in the classrooms (O’Mahony, 2003; Pelgrum, 2001, as cited
58
in Ertmer et al., 2012), however, these barriers have been reduced with the substantial
funding dedicated to increasing technology access in K-12 education (Culp, Honey, &
Mandinach, 2005). These funding support has ensured teachers’ access to computers
and internet (Gray, Thomas, & Lewis, 2010). Moreover, professional development and
instructional support for teachers have also been improving (Ertmer et al., 2012). In the
current K-12 education context, the second-order barriers have a more remarkable
influence on teachers’ technology integration (Ertmer et al., 2012). Moreover, in the
iDigFossils project, the teachers were provided with 3D printers and related software,
professional development to learn how to use 3D printers and how to integrate 3D
printing into science classes, and technical and instructional support (e.g., Teachers’
lesson plans were reviewed by experts and suggestions were provided to the teachers)
for their 3D printing integration in their science classrooms. The external barriers in the
context of the iDigFossils project are trivial. Therefore, the current study focuses on
teachers’ beliefs which may have influenced their 3D printing technology integration.
Teacher beliefs is an ill-defined concept and can be broadly defined as “teachers’
implicit assumptions about students, learning, classrooms, and the subject matter to be
taught” (Kagan, 1992, p. 66). As individuals’ beliefs strongly influence their behavior
(Pajares, 1992), the influence of teacher beliefs on teaching practice has been widely
studied since the 1990s. There is a strong association between teacher beliefs and their
teaching practice (Ertmer, 2005). As stated by Pajares (1992), there is a "strong
relationship between teachers' educational beliefs and their planning, instructional
decisions, and classroom practices" (p. 326). Researchers perceived teacher beliefs as
the most valuable psychological construct to teacher education (Pintrich, 1990) and
59
teacher beliefs are even more influential on teaching practice than teachers’ knowledge
(Ertmer & Ottenbreit-Leftwich, 2010; Kagan, 1992; Pajares, 1992). Teacher beliefs “tend
to be associated with a congruent style of teaching that is often evident across different
classes and grade levels” (Kagan, 1992, p. 66). Teacher beliefs have been widely
studied in more than fourteen countries around the world such as U.K., Spain, Portugal,
Israel, the Netherlands, Turkey, China, South Korea, Singapore, the U.S., and Canada,
etc., as Kim, Kim, Lee, Spector, and DeMeester (2013) reviewed.
To understand the role of teacher beliefs in teaching with technology,
researchers have focused on a myriad of aspects of teacher beliefs, including teachers’
pedagogical beliefs, self-efficacy in teaching with technology, and technology value
beliefs (e.g., Ertmer & Ottenbreit-Leftwich, 2010; Park & Ertmer, 2007). Teachers’ self-
efficacy and technology value beliefs in teaching with technology are directly associated
with their technology integration in the classrooms and teachers’ pedagogical beliefs are
underlying beliefs that can impact how teachers integrate technology into teaching and
learning. As Kim et al. (2013) stated, pedagogical beliefs are teachers’ fundamental
beliefs that are regardless of the technology involved but can impact how teachers
integrate technology for teaching. In the following, research on teachers’ pedagogical
beliefs, self-efficacy beliefs in technology integration, and technology value beliefs will
be reviewed. A theoretical framework of teacher beliefs on technology integration can
be viewed in Figure 2-4.
60
Figure 2-4. Theoretical framework of relationships between teacher beliefs and technology integration
Pedagogical Beliefs
Pedagogy is generally perceived as the science of teaching and it is “any
conscious activity by one person designed to enhance learning in another” (Mortimore,
1999, p. 3). Therefore, pedagogy can be perceived as an effective instructional strategy
or method to enhance learning. Teachers’ pedagogical beliefs are assumptions about
effective instructional strategies that teachers can use to enhance students’ learning.
Access to technologies does not guarantee teachers’ effective integration of technology
into classrooms. Teachers’ pedagogical beliefs play a key role in how teachers integrate
technology in their classrooms (Deng, Chai, Chin-Chung, & Min-Hsien, 2014; Ertmer et
Teacher Beliefs on Technology Integration
Teacher beliefs related to technology
(Expectancy-Value Theory of Motivation)
Self-efficacy in
technology integration
(competency beliefs)
Technology value
beliefs
Teacher fundamental beliefs
Pedagogical beliefs
Teacher technology
integration in the
classrooms
61
al., 2012, 2015; Hermans, Tondeur, van Braak, & Valcke, 2008; Inan & Lowther, 2010;
Niederhauser & Stoddart, 2001; Tondeur et al., 2017).
Teachers’ pedagogical beliefs are commonly classified as beliefs on teacher-
centered learning (behaviorist beliefs) and student-centered learning (constructivist
beliefs) (Ertmer et al., 2012; Kim et al., 2013; Park & Ertmer, 2007; Tondeur et al.,
2017). Teachers with teacher-centered beliefs tend to act as an authority and expert in
highly structured learning environments and supervise the learning process (Tondeur et
al., 2017). Teacher-centered technology integration often involves low-level technology
uses such as using technology just to deliver information (Harmes et al., 2016).
However, teachers with constructivist beliefs tend to integrate technology in more
meaningful ways to facilitate student active learning and enhance students’ higher-order
thinking and problem-solving skills (Tondeur et al., 2017), such as problem-based
learning activities which involve students in authentic disciplinary problems and
collaborative learning activities (Ertmer & Glazewski, 2015; Park & Ertmer, 2007). A
systematic review on teachers’ pedagogical beliefs and technology use in education
indicated that there is a bi-directional relationship between teachers’ pedagogical beliefs
and their technology use in the classrooms: on one hand, teachers’ technology-rich
learning experiences can potentially change teachers’ beliefs towards more
constructivist beliefs; on the other hand, teachers with constructivist beliefs tend to
integrate technology to facilitate student-centered learning (Tondeur et al., 2017).
However, teachers’ technology integration may not always be consistent with their
pedagogical beliefs especially when external barriers such as lack of resources and
62
time prevent them from integrating technology in the ways they would like to (Ertmer et
al., 2012).
Self-Efficacy in Technology Integration
Teachers’ pedagogical beliefs are essential for teachers to decide how they will
integrate technology in their classrooms. However, teachers may not integrate
technology in the way they would like to if they do not feel they have the competence.
Rooted in Bandura’s (1986, 1997) social cognitive theory, self-efficacy is an individual’s
beliefs in his or her competence to accomplish a task or reach a goal. Teachers’ self-
efficacy in technology integration is teachers’ beliefs on their competence to integrate
technology into teaching in order to facilitate student learning and achieve the teaching
goals. Teachers may have abundant knowledge and skills in technologies but may not
be confident to integrate the technologies for teaching. Research suggests that
teachers’ self-efficacy in technology integration may be more important than their
knowledge and skills in technologies for integrating technologies in the classrooms
(e.g., Bauer & Kenton, 2005; Wozney, Venkatesh, & Abrami, 2006).
Teachers’ self-efficacy in technology integration is a key predictor of their
implementation of technology in the classrooms (Albion, 1999; Ertmer & Ottenbreit-
Leftwich, 2010; Gonzales, 2013; Haight, 2011; Heineman, 2018; Li, Garza, Keicher, &
Popov, 2018; Manglicmot, 2015; Marcinkiewicz, 1994; Tweed, 2013). Increasing
teachers’ self-efficacy in technology integration can encourage teachers to more
effectively integrate technology into their classrooms (Heineman, 2018). However, even
if teachers have strong self-efficacy in technology integration, they may not integrate the
technology in their classrooms if they do not perceive the technology as valuable to
63
achieve the teaching goals. Therefore, it is also important to take teachers’ technology
value beliefs into consideration.
Technology Value Beliefs
According to expectancy-value theory (Wigfield, 1994), self-efficacy in technology
integration, namely teachers’ beliefs in their competence to integrate technology to
achieve instructional goals, is necessary but not sufficient for teachers to implement
technology in teaching; teachers’ beliefs on the value of the technology are critical for
them to take actions to integrate technology into their classrooms. Teachers’ technology
value beliefs are teachers’ beliefs about the value of integrating technology to facilitate
teaching and learning and to achieve the instructional goals (Watson, 2006; Ottenbreit-
Leftwich et al., 2010). Teachers’ technology value beliefs are critical for teachers to
integrate technology in their classrooms (Ertmer et al., 1999; Ertmer & Ottenbreit-
Leftwich, 2010; Ertmer et al., 2012; Mueller, Wood, Willoughby, Ross, & Specht, 2008;
Vongkulluksn, Xie, & Bowman, 2018; Wozney et al., 2006). When teachers have access
to technology, they make judgments on whether the technology can be used to enhance
teaching and learning. Teachers who have more positive beliefs on the affordability of
technology for instruction tend to integrate technology more frequently in their
classrooms (Anderson & Maginger, 2007; Becker, 1999; Ottenbreit-Leftwich et al.,
2010; Park & Ertmer, 2007; Zhao & Frank, 2003). Integrating a new technology into
current teaching practice requires teachers to spend time and effort learning how to use
the technology and how to meaningfully integrate it into the curriculum. If teachers
perceive the technology as not having enough value to address teaching and learning
needs, they are not likely to use it. When teachers believe the values of using
64
technology to achieve instructional goals, they tend to use it even when barriers exist
(Ertmer et al., 2012; Ottenbreit-Leftwich et al., 2010; Snoeyink & Ertmer, 2001).
In light of the subjective task value in the expectancy-value theory of motivation
(Eccles et al., 1983; Eccles & Wigfield, 1995), teachers’ technology value beliefs can be
composed of intrinsic value (interest in integrating the technology), perceived attainment
value (importance of integrating the technology for teaching and learning), and
perceived utility value (usefulness of integrating the technology for teaching and
learning). When teachers believe that technology integration is interesting and important
and useful to enhance teaching and learning, they are more motivated to learn how to
integrate technology in their classrooms (Cheng & Xie, 2018). Studies have focused on
the general teachers’ technology value beliefs and it is unclear which specific value
beliefs contribute to teachers’ technology integration in the classrooms (Ottenbreit-
Leftwich et al., 2010; Smarkola, 2008).
There is limited research on the relationship between teachers’ technology
integration and their specific technology value beliefs such as teachers’ interest in
technology integration and perceived importance and usefulness of integrating
technology for teaching and learning. In the study of Inan and Lowther (2010) with 1382
in-service teachers, it was found that teachers’ beliefs on the importance of technology
for teaching and learning predicted teachers’ technology integration in the classrooms.
Cheng and Xie (2018) assessed teachers’ technology value beliefs in terms of teachers’
perceived interest, importance, and usefulness about learning digital content evaluation,
examined the relationship between teachers’ technology values beliefs and
Technological Pedagogical Content Knowledge (TPACK), and found teachers’
65
technology value beliefs significantly predicted TPACK. However, this study aggregated
the three aspects of technology value beliefs as a whole and did not examine how each
aspect of teachers’ technology value beliefs predicted TPACK. This study did not
examine the relationship between teachers’ technology value beliefs and the actual
technology integration either. Teachers’ interest in technology integration and perceived
importance and usefulness of technology integration may not necessarily positively
relate to each other. For instance, a teacher may perceive technology integration as
interesting but may not believe it is important and useful for teaching and learning, while
another teacher may perceive technology integration as not interesting but believe it is
important and useful for teaching and learning. It would be necessary to investigate how
the specific aspects of teachers’ technology value beliefs are correlated with teachers’
technology integration in the classrooms.
Teachers’ pedagogical beliefs, self-efficacy in technology integration, and
technology value beliefs are all essential for teachers to meaningfully integrate
technology into their classrooms. These teacher beliefs on technology integration may
also interrelate with each other. In a study with 152 teachers in the midwestern United
States, Hsu (2016) found that teachers who held constructivist pedagogical beliefs had
high self-efficacy in using technology for teaching and had positive value beliefs on the
use of technology. Teachers who have higher technology value beliefs are also more
likely to embrace constructivist pedagogical beliefs and integrate technology for student-
centered learning such as designing higher-order thinking and critical thinking activities
(Ertmer et al., 2012; Hixon & Buckenmeyer, 2009; Hsu, 2016). In this current study with
K-12 teachers who integrated 3D printing technologies into their science classrooms, it
66
is important to investigate how teachers’ pedagogical beliefs, self-efficacy in technology
integration, and technology value beliefs were related to their 3D printing technology
integration, and how these teacher beliefs correlated with each other, to provide
implications on improving teachers’ 3D printing technology integration in science
classrooms by initially investigating the relationship between teachers’ beliefs related to
technology integration and their teaching practice.
Teacher Beliefs and Student Learning Outcomes
In addition to the relationships between teacher beliefs and technology
integration, research also found teacher beliefs is associated with student learning
outcomes. Teacher beliefs can influence students’ learning performance and motivation
both “directly through observable teacher behaviors and indirectly through more subtle
forms of communication” (Midgley, Feldlaufer, & Eccles, 1989, p. 247). Therefore, even
if teachers’ technology integration practice is not aligned with their beliefs, students may
still be influenced by teacher beliefs.
Although there was little research on the direct relationship between teachers’
pedagogical beliefs and student learning outcomes, research indicated that teachers
with different pedagogical beliefs may tend to facilitate either teacher-centered or
student-centered learning (Tondeur et al., 2017), and student-centered learning fosters
students’ affective learning outcomes such as student engagement and motivation
(Cornelius-White & Harbaugh, 2009). As a meta-analysis study (Cornelius-White, 2007)
on student-centered teacher-student relationship demonstrated, learner-centered
teacher variables consisting of empathy, warmth, genuineness, nondirectivity (student-
initiated and student-regulated activities), higher-order thinking, encouraging
learning/challenge, and adapting to individual and social differences, had positive
67
associations with student cognitive and affective or behavioral outcomes including
student motivation. Therefore, there might be associations between teachers’
pedagogical beliefs and students’ cognitive learning outcomes such as knowledge and
skills, and affective learning outcomes such as motivation and interest.
There was also a lack of studies on the relationship between teachers’
technology value beliefs and student learning outcomes. However, research indicated
that teachers are more likely to integrate technology to promote student learning when
they believe the technology is valuable to be integrated (Ottenbreit-Leftwich et al.,
2010), and teachers’ technology integration may influence student learning.
Furthermore, even if teachers’ actual technology integration is not aligned with their
technology value beliefs due to some external barriers, teachers’ technology value
beliefs may still have influences on students because teacher beliefs can impact
students in some subtle forms of communication (Brophy & Good, 1974; Good, 1981;
Heller & Parsons, 1981, as cited in Midgley et al., 1989). Teachers’ technology value
beliefs may be communicated to the students and influence students even if teachers
cannot integrate the technology at a higher level as they would like to. For instance, a
teacher who has high value beliefs on the technology may show great enthusiasm when
just talking about the technology, which may make the technology appealing to students
and increase students’ motivation to learn or even enhance students’ interest in
pursuing a related career.
Most studies about the relationship between teacher beliefs and student learning
outcomes focused on teachers’ self-efficacy. The meta-analysis study conducted by
Zee and Koomen (2016) indicated that teachers’ self-efficacy in teaching was positively
68
related to students’ academic achievement and motivation. Specifically, researchers
found that teachers’ higher self-efficacy was associated with students’ learning
performance (e.g., Ashton & Webb, 1986; Brookover, Beady, Flood, Schweitzer, &
Wisenbaker, 1979; Brophy & Evertson, 1977; Hoy & Davis, 2005; Shahzad & Naureen,
2017) and students’ affective learning outcomes such as enhanced motivation (e.g.,
Eccles & Wigfield, 1985; Lazarides, Buchholz, & Rubach, 2018; Mojavezi & Tamiz,
2012; Pan, 2014; Schiefele & Schaffner, 2015).
Teacher beliefs may impact how teachers integrate the technologies into their
classrooms and teachers’ different levels of 3D printing technology integration may also
have different influences on students. Teacher beliefs may also have influences on
students even if their 3D printing integration levels are not aligned with their beliefs. It is
important to investigate both the relationships between teachers’ levels of 3D printing
integration and students’ learning outcomes, and the relationship between teacher
beliefs and students’ learning outcomes if teacher beliefs and teachers’ 3D printing
integration levels are not highly correlated. As the teachers were from different states
and different school levels, and they used different learning content and assessments, it
was not feasible to compare students’ learning performance in a consistent and
systematic way. Therefore, the current study did not include students’ learning
performance but focused exclusively on students’ STEM motivation, interest in STEM
careers, and 21st century skills.
Students’ STEM Motivation, Interest in STEM Careers, and 21st Century Skills
The review on students’ STEM motivation, interest in STEM careers, and 21st
century skills is guided by the underlying theoretical foundation of social cognitive
69
career theory and the expectancy-value theory of motivation. The theoretical framework
can be viewed in Figure 2-5.
Figure 2-5. Theoretical framework for STEM motivation, interest in STEM careers, and 21st century skills
Social Cognitive Career Theory
Social cognitive career theory (SCCT) is derived from Bandura’s (1986) social
cognitive theory and it provides a framework to understand people’s career interests,
career choice, and performance (Lent & Brown, 1996). SCCT focuses on a few
constructs including self-efficacy, outcome expectations, and personal goals, and how
these constructs interrelate with personal factors and environments in the process of
career development (Lent & Brown, 1996). Students’ academic goals and career choice
Social Cognitive Career Theory
Expectancy-Value Theory of Motivation
STEM
Motivation
STEM self-efficacy
(competency beliefs)
STEM value beliefs
Interest in STEM
careers
21st century
skills
STEM academic
and career pursuit
70
are influenced by their academic and career interests, self-efficacy, outcome
expectations, and environmental support and barriers.
SCCT has been applied by researchers to study undergraduate students’ STEM
career pathways (e.g., Byars-Winston, Estrada, Howard, Davis, & Zalapa, 2010; Lent et
al., 2005; Lent, Lopez, Lopez, & Sheu, 2008; Lent, Lopez, Sheu, & Lopez, 2011). Lent
et al. (2005) used an SCCT-based model to predict engineering interests and major
choice goals with 487 students at three universities, who enrolled in introductory
engineering courses. Their study found that self-efficacy and outcome expectations
were predictive of student interest in engineering, interests and perceived barriers were
predictive of their engineering academic goals, and the supports and barriers were
correlated with engineering self-efficacy. The SCCT-based model was also used in
computing disciplines (e.g., Lent et al., 2008; Lent et al., 2011) and science discipline
(Byars-Winston et al., 2010) to predict college students’ academic interests and major
choice in STEM.
SCCT has also been applied to investigate high school students’ choices to
pursue STEM majors (e.g., Nauta & Epperson, 2003; Wang, 2013). Nauta and
Epperson (2003) investigated 204 high school female students who attended science,
math, and engineering (SME) career conferences in a 4-year longitudinal study to
predict the girls’ choice of college major in SME, and their SME self-efficacy and
outcome expectations in college. The results showed there was a positive relationship
between students’ self-efficacy and interest in science; interest in science, self-efficacy,
and outcome expectations were positively correlated with students’ college major choice
in SME. Wang (2013) applied SCCT theory to understand recent high school graduates’
71
entrance into STEM majors in 4-year institutions. The results suggested that students’
intent to major in STEM was directly affected by their 12th-grade math achievement,
exposure to math and science courses, and math self-efficacy beliefs (Wang, 2013).
Suggested by Unfried et al. (2015), in terms of STEM career development,
student STEM self-efficacy and expectancy-value beliefs and interest in STEM careers
are key components of social cognitive career theory (SCCT) (Lent & Brown, 2006;
Lent, Brown, & Hackett, 2000; Lent, Sheu, Gloster, & Wilkins, 2010), and students also
need 21st century skills for academic and career development.
Expectancy-Value Theory of Motivation and STEM
Expectancy-value theory of motivation was originated by Atkinson (1957) who
developed the theory to understand individuals’ achievement motivation. Atkinson
(1957) defined expectancies as individuals’ anticipations of success or failure of their
performance and defined value as attractiveness of success on a task. Other
researchers continued the research on expectancy-value and further investigated the
relationship between students’ expectancies for success, subjective task values, and
their task choice, performance, and persistence (e.g., Crandall, 1969; Crandall,
Katkovsky, & Preston, 1962; Eccles et al., 1983; Feather, 1982, 1988, 1992; Wigfield &
Eccles, 1992). Eccles et al. (1983) expanded the expectancy-value theory by creating
an expectancy-value model of achievement performance and choice with more internal
and external variables, including achievement behaviors (e.g., students’ achievement
performance and choice) and belief and value constructs such as subjective task
values, expectancies for success, achievement goals, and competence or ability beliefs.
The gist of expectancy-value theory is that “individuals’ expectancies for success and
72
the value they have for succeeding are important determinants of their motivation to
perform different achievement tasks” (Wigfield, 1994, p. 50).
As Eccles et al. (1983) suggested, students’ choice of achievement tasks,
achievement performance, and persistence are most directly influenced by their
expectancies for success on those tasks and their subjective value of success on those
tasks. Expectancies for success and subjective task values are two key components of
expectancy-value theory (Eccles et al., 1983; Eccles & Wigfield, 2002; Wigfield, 1994;
Wigfield & Eccles, 2000). Expectancies for success refer to individuals’ beliefs about
how well they will perform on a task (Eccles et al., 1983; Wigfield, 1994), namely a
person’s competency or ability beliefs to complete a task or achieve a goal, which can
be represented as a person’s self-efficacy. Therefore, according to the expectancy-
value theory of motivation, students’ STEM motivation can be influenced by their self-
efficacy and values beliefs in STEM.
Self-efficacy and STEM
Self-efficacy is grounded in social cognitive theory which suggests that students’
achievement depends on interactions between students’ behaviors, beliefs, and
environmental conditions (Bandura, 1986, 1997). Self-efficacy refers to individuals’
beliefs in their abilities to complete tasks or achieve goals (Bandura, 1986, 1997). Self-
efficacy concerns with the “judgements of how well one can execute courses of action
required to deal with prospective situations” (Bandura, 1982). Students’ judgment of
their self-efficacy can be obtained from their actual performance, vicarious experiences,
verbal persuasions received from others, and their physiological and affective states
(Bandura, 1997; Schunk & Pajares, 2002).
73
Self-efficacy influences students’ thought patterns and emotional reactions when
faced with challenges. Those who have low self-efficacy dwell on their personal
deficiencies and imagine the potential challenges as more formidable than they really
are (Beck, 1976; Lazarus & Launier, 1978; Meichenbaum, 1977; Sarason, 1975; as
cited in Bandura, 1982). While individuals with low self-efficacy focus their attention on
concerns of failings, individuals with high self-efficacy divert their attention to the
demands of the task challenges and take efforts to overcome obstacles (Bandura,
1982). Students’ self-efficacy can influence their task choices, levels of efforts,
persistence, and achievement (Bandura, 1997; Schunk, 1995; Schunk & Pajares,
2002). Self-efficacy is important for a student to tackle learning challenges and be
persistent in learning to achieve learning goals. Self-efficacy is positively related to
student interests and engagement in learning (Schunk & Pajares, 2002) and student
engagement and motivation are highly related (Martin, Ginns, & Papworth, 2017;
Reeve, 2012; Schunk & Mullen, 2012). Students with high self-efficacy spend greater
effort and are more persistent to complete a task (Pajares, 2005; Zimmerman, 2000).
Research indicates that student self-efficacy in STEM is positively related to their
STEM performance and their persistence in STEM disciplines (Britner & Pajares, 2006;
Miller, 2015; Pajares, 2005; Schunk & Pajares, 2002). As Britner and Pajares (2006)
found, students’ performance in science class is positively related to their science self-
efficacy. Grigg, Perera, McIlveen, and Svetleff (2018) investigated math self-efficacy of
students from grade 6 to grade 10 and found that the students’ math self-efficacy
positively predicted their improvement in math performance. Brown, Concannon, Marx,
Donaldson, and Black (2016) examined middle school students’ STEM self-efficacy and
74
their intentions to persist in STEM and found students’ STEM self-efficacy was a
significant positive predictor for students’ intentions to persist in STEM.
Research also indicates that students’ self-efficacy in STEM is associated with
their pursuit in STEM fields when choosing majors. Scott and Mallinckrodt (2005)
surveyed students who previously participated in a high school enrichment program and
found that students who later majored in science had significantly higher science self-
efficacy than students who did not choose science major. Wang (2013) investigated
factors that impacted postsecondary school students’ choice of majors and found that
students’ math self-efficacy was positively correlated with the intent to major in STEM
disciplines. Sahin, Ekmekci, and Waxman (2017) examined the relationship between
2,246 high school graduates’ math and science self-efficacy and their likelihood of
majoring in STEM in college and found that male students who had higher math self-
efficacy and female students who had higher science self-efficacy are more likely to
choose a STEM major than their counterparts who had lower math and science self-
efficacy.
Value beliefs and STEM
In the expectancy-value theory, value beliefs consist of intrinsic interest value,
attainment value/importance, utility value/usefulness, and cost (Eccles et al., 1983).
Intrinsic interest value is the enjoyment an individual gains from doing the task;
attainment value is the importance of doing well on the task; utility value is how a task
contributes to future plans; and cost refers to what the individual has to give up on other
things in order to do the task as well as anticipated efforts the individual has to put into
the task (Eccles et al., 1983; Wigfield, 1994).
75
Expectancy-value theory provides a theoretical framework to explain the
relationship between students’ psychological factors and their interest in STEM and
career choices (Wang & Degol, 2013). Self-efficacy and value beliefs may influence
students’ motivation and task selection in STEM. Students’ beliefs about how well they
can perform in an activity and how much they value the activity can influence their
choices, performance, and persistence in STEM learning. When students are confident
that they can perform well and be successful in subject areas such as STEM, they are
more likely to engage and persist in learning which may lead to better academic
performance and course enrollment (Wigfield & Eccles, 2002).
Students’ value beliefs in STEM are predictive of their choice behaviors and
beliefs such as persistence in STEM learning and intentions to select STEM courses
and choose STEM careers (Eccles, 2009; Wang & Degol, 2013; Wang & Eccles, 2013).
Research shows that high value beliefs are associated with students’ persistence in
learning advanced science and math (Fan, 2011; Simpkins, Davis-Kean, & Eccles,
2006). Tai et al. (2006) surveyed 12,000 middle school students and found students’
interest in STEM was a significant predictor of their future career choices. Simpkins et
al. (2006) found that students’ interest in math and science in elementary school was
predictive of their course selection in high school. Expectancy-value theory is a strong
theoretical framework that can investigate the factors that influence students STEM
learning persistence and career aspirations.
Students’ Interest in STEM Careers
The construct of interest began with Herbart’s view that education should foster
unspecialized and multi-faceted interests which would facilitate learning (Wigfield &
Cambria, 2010). Interest is an essential motivational and driving force for learning
76
(Dewey, 1913). Interest is a psychological state which is “a particular relation of that
individual in engagement with that play object/task, relative to the other activities with
which he or she engages” (Renninger, 1992, p. 362). Interest is a relational concept
consisting of the relationship between an individual and an object or activity (Krapp,
2002; Schiefele, 2009). Interest can be mediated by the interaction between the
individual and the object or activity and both personal factors and environmental factors
can influence interest (Mitchell, 1993; Renninger & Hidi, 2002). Researchers
categorized interest as individual interest and situational interest (e.g, Hidi & Renninger,
2006; Krapp, 2002; Schiefele, 2009). Situational interest is “a temporary state aroused
by specific features of a situation, task, or object” and individual interest is “a relatively
stable affective-evaluative orientation toward certain subject areas or objects”
(Schiefele, 2009, p. 198).
Research shows that student interest has a powerful influence on learning in
terms of students’ attention, goals, and levels of learning (Hidi & Renninger, 2006).
Interest is also related to students’ self-efficacy and students with stronger interest
usually have higher self-efficacy and can persist on challenging tasks (Hidi & Ainley,
2008; Sansone, 2009, as cited in Honey et al., 2014) In the context of STEM learning,
there are strong associations among students’ STEM interest, interest in STEM careers,
and intention to pursue a STEM major or career. Sadler et al. (2012) found that
students’ interest in STEM at the start of high school was a key predictor of their STEM
career interests when they graduated. Christensen and Knezek (2017) collected data
from over 800 middle school students who participated in a hands-on and real-world
application curriculum and examined the relationship between students’ STEM interests
77
and their intentions to pursue STEM careers. Results showed that students’ interest in
STEM positively aligned with their intent to pursue STEM careers. Maltese and Tai
(2011) found that eighth-grade students who had interest in a science career and
believed science would be useful in their future were more likely to earn a bachelor’s
degree in a STEM discipline.
Since students’ interest in STEM and STEM-related careers are strong predictors
of their participation in STEM disciplines and future STEM career, it is important to
foster students’ interest in STEM disciplines and STEM careers. Individual interest is
relatively more difficult to change but teachers could engage students in classroom
activities to foster students’ situational interest which may subsequently develop into
personal interest.
21st Century Skills
The construct of 21st century skills has been defined in many different ways.
Twenty-first century skills are important knowledge and skills that students need to
succeed in academic and career development (P21). The core components of 21st
century skills include critical thinking, communication, collaboration, creativity, problem
solving, and digital literacy (NRC, 2010; P21; PCAST, 2010). Twenty-first century skills
are necessary for academic and career development in the 21st century information
era. Twenty-first century skills are essential for students to participate and succeed in
STEM disciplines and STEM careers and students also develop 21st century skills
through STEM education. As Jones (2014) stated, 21st century skills and STEM
education are coalesced to educate students to become productive and technologically
literate citizens of tomorrow. Twenty-first century skills have been integrated into K-12
STEM curricula as instructional goals and course standards. There is a close alignment
78
between K-12 STEM standards and 21st century skills. National policymakers,
researchers, and educators have been calling for attention and taking efforts to improve
students’ 21st century skills (NRC, 2010).
As students’ STEM motivation and interest in STEM careers are associated with
students’ persistence in STEM learning and future career choice, and students need the
21st century skills for STEM learning and STEM career development, enhancing
students’ STEM motivation, interest in STEM careers, and 21st century skills altogether
is imperative to increase and strengthen students’ participation in STEM disciplines the
future STEM workforce.
Conceptual Framework
A conceptual framework is an argument based on theory and driven by evidence
with the purpose to “justify the research problem, define relevant concepts, establish
theoretical and empirical rationale, select appropriate methods, and interpret results
relative to theory” (Antonenko, 2015, p. 58). Based on the theories and literature
reviewed above, the conceptual framework (Figure 2-6) guiding this study posits that: 1)
Teacher beliefs (pedagogical beliefs, self-efficacy beliefs in 3D printing integration, and
3D printing value beliefs) may be correlated with teachers’ 3D printing technology
integration in K-12 science classrooms (teacher proximal outcome); and 2) Teacher
beliefs and teachers’ 3D printing technology integration in K-12 science classrooms may
predict students’ STEM motivation, 21st century skills, and interest in STEM careers
(student proximal outcome). Correlational and regressional analyses were conducted to
examine these relationships and multiple data sources were used for the analyses. As
indicated in the conceptual framework, teacher beliefs data were collected with a
teacher beliefs survey, teachers’ lesson plans were used to analyze teachers’ 3D
79
printing integration, and students’ STEM motivation, 21st century skills, and interest in
STEM careers were collected with the S-STEM survey (Unfried et al., 2015).
Although teacher beliefs on 3D printing technology integration may impact how
teachers integrate the technologies into their science classrooms, this study cannot
determine if there was a causal relationship between teacher beliefs and teachers’
technology integration practice. Therefore, this study focused on the correlation
between teacher beliefs and teachers’ 3D printing technology integration in the
classrooms. As extant literature indicated that teachers’ 3D printing integration can
influence students’ learning outcomes and teacher beliefs may also influence students’
learning outcomes, this study explored how both teacher beliefs and teachers’ 3D
printing integration might predict students’ learning outcomes including students’ STEM
motivation, 21st century skills, and interest in STEM careers.
Teachers’ different 3D printing technology integration involved students in varied
learning activities which may have different impacts on students. Some teachers may
have just printed out the 3D objects by themselves and showed the objects to students,
and students did not have much interaction with the 3D printing technology or the
printed objects; while some other teachers may engage students in the 3D printing
process for students to learn how 3D printers work or even have students design and
print out 3D models, which may have enhanced students’ engagement in learning.
Some teachers may just introduce what 3D printing technology is without adequate
connection to the curriculum while other teachers may involve students in problem-
based learning activities with 3D printing process or 3D printed objects to engage
students in learning. Additionally, some teachers may just integrate 3D printing
80
technology in science learning, while others may also integrate math learning or
engineering learning. Teachers may have integrated 3D printing technologies in a
myriad of ways, which could have different influences on students’ STEM motivation,
21st century skills, and interest in STEM careers.
Abundant research suggests that students’ STEM motivation and interest in
STEM careers are associated with students’ participation in STEM disciplines and
STEM careers. The 21st century skills have been integrated as curriculum standards
and curriculum frameworks and improving student motivation in STEM disciplines and
career skills have gained increasing attention from national policymakers (Unfried et al.,
2015). Students’ STEM motivation, 21st century skills, and interest in STEM careers are
essential for students to participate in STEM disciplines and STEM careers. Therefore,
it is important to improve students’ STEM motivation, 21st century skills, and interest in
STEM careers, in order to increase students’ likelihood of choosing majors in STEM
disciplines and participate in STEM careers and also enhance students’ persistence and
success in STEM disciplines and careers. The current study explained the importance
of enhancing students’ STEM motivation, 21st century skills, and interest in STEM
careers to increase students’ participation in STEM disciplines and STEM careers
(student distal outcome), but this potential influence was not examined empirically due
to time constraints and feasibility of this study.
81
Figure 2-6. Conceptual framework of this study
Teacher
Proximal
Outcome
Student
Proximal
Outcome
Student
Distal
Outcome
Teacher
Beliefs
Self-Efficacy
Beliefs in 3D
Printing Integration
STEM
Motivation
Interest in
STEM careers
Participation
in STEM
disciplines
and STEM
careers
21st Century
Skills
3D Printing
Integration
in K-12
Science
Classrooms
3D Printing Value
Beliefs
Pedagogical Beliefs
Data Source:
Lesson Plan
Date Source:
S-STEM Survey
Not
measured in
this study
Data Source:
Teacher Beliefs
Survey
82
CHAPTER 3 METHODOLOGY
This study used quantitative methodology by conducting correlational analysis to
investigate how teacher beliefs are related to their 3D printing integration in the science
classrooms and multilevel modeling analysis to examine how teacher beliefs and
teachers’ 3D printing integration predict students’ STEM motivation, 21st century skills,
and interest in STEM careers. This chapter describes the study context and
participants, instrumentation, data sources and data collection, and data analysis.
Context and Participants
This study used a convenient sample of teachers and students who participated
in the iDigFossil project, a 3-year National Science Foundation (NSF) funded project
“iDigFossils: Engaging K-12 Students in Integrated STEM via 3D Digitization, Printing
and Exploration of Fossils” (PI: Dr. Pavlo Antonenko. Award No. 1510410). The
iDigFossils project was an integrated STEM initiative aimed to engage students in
STEM learning through the integration of 3D printing technology in K-12 science
classrooms within the context of paleontology, a rich, interdisciplinary science that
examines life in deep time. A group of teachers and their students in different states
across the United States participated in this project. Each teacher was provided a 3D
scanner, a 3D printer, a laptop, and related resources. To facilitate teachers’ 3D printing
integration in their science classrooms, a week-long workshop during the summer of
each year of the project was provided for the teachers to learn how to use 3D printing
technologies and how to integrate 3D printing technologies in science classes within the
context of paleontology. All the teachers participated in the workshop. The teachers also
had assistance from experts in educational technology and paleontology for their lesson
83
plan design. After the teachers designed the lesson plans, they sent them to the experts
for review and they received feedback on how to improve the lesson plans. Throughout
teachers’ participation in the project, the professionals in the iDigFossils project were
available through face-to-face or online meeting for any assistance the teachers
needed.
The participants in this study were a portion of the teachers and students who
participated in the iDigFossils project. The selection of participants for this study
depended on whether there were adequate data from the teachers and students. If
teachers' lesson plans on how they integrated 3D printing into their science classrooms
were available and they had students who completed both the S-STEM survey (Unfried
et al., 2015) pretest and posttest, their data were included for analysis. In this study, 26
teachers met with these criteria. Among the 26 teachers, 24 teachers completed the
teacher beliefs survey. The 2 teachers who did not take the teacher beliefs’ survey were
treated as missing data when doing statistical analyses that involved data on teachers’
beliefs. After data cleaning and screening of students who completed both the pretest
and posttest with the S-STEM survey, a total number of N = 1,501 students were
included for data analysis.
The students (see demographics in Table 3-1) consisted of 712 males and 789
females, with 192 elementary students, 847 middle school students, and 462 high
school students. Students’ race/ethnicity included White (N = 719), Hispanic (N = 197),
Mixed race or multi-race (N = 146), Asian (N = 100), Black or African American (N = 96),
American Indian or Alaska Native (N = 66), Native Hawaiian or other Pacific Islander (N
= 19), other race/ethnicity (N = 122), and students who did not wish to answer (N = 36).
84
Table 3-1. Student demographics
Demographics category Student N (%)
Gender Male: N = 712 (47.44%); Female: N = 789 (52.56%).
School Level Elementary school: N = 192 (12.79%); Middle school: N = 847
(56.43%); High school: N = 462 (30.78%).
Race/Ethnicity White: N = 719 (47.90%); Hispanic: N = 197 (13.12%); Asian: N
= 100 (6.66%); Mixed race or multi-race: N = 146 (9.77%);
Black or African American: N = 96 (6.4%); American Indian or
Alaska Native: N = 66 (4.4%); Native Hawaiian or other Pacific
Islander: N = 19 (1.27%); Other: N = 122 (8.13%); Do not wish
to answer: N = 36 (2.40%).
Among the 26 teachers (view Table 3-2 for demographics), there were 4 males
and 22 females. Their ages ranged from 21 to above 60 with 3 teachers aged between
21 and 30, 9 teachers aged between 31 and 40, 7 teachers aged between 41 and 50, 5
teachers aged between 51 and 60, and 2 teachers aged 61 or above. There were 4
elementary teachers, 13 middle school teachers, and 9 high school teachers. The
majority of the teachers were White (N = 20) and there were 2 teachers being African
American, 2 teachers being Hispanic, and 2 teachers being mixed-race. The teachers
were across 6 states including Oklahoma (N = 4), California (N = 9), Georgia (N = 3),
Texas (N = 1), Louisiana (N = 1), and Florida (N = 8).
Table 3-2. Teacher demographics
Demographics Category Teacher N
Gender Male: N = 4; Female: N = 22.
Age 21-30: N = 3; 31-40: N = 9; 41-50: N = 7; 51-60: N = 5; 61+: N = 2.
School Level Elementary school: N = 4; Middle school: N = 13; High school: N = 9.
Race/Ethnicity White: N = 20; African American: N = 2; Hispanic: N = 2; Mixed race: N = 2.
State Oklahoma: N = 4; California: N = 9; Georgia: N = 3; Texas: N = 1; Louisiana: N = 1; Florida: N = 8.
85
Instrumentation
S-STEM Survey
The S-STEM survey developed and validated by Unfried et al. (2015) from the
Friday Institute for Educational Innovation at North Carolina State University was used
as the instrument to measure students’ attitude towards STEM, 21st century skills, and
interest in STEM careers. Although the authors used the construct of attitude, the items
in the survey either reflected students’ self-efficacy and expectancy value. For instance,
in the science attitude subscale, there were 9 items, and 4 items were about students’
self-efficacy and 5 items were about students’ expectancy value. The 4 self-efficacy
items consisted of “I am sure of myself when I do science”, “I know I can do well in
science”, “I can handle most subjects well, but I cannot do a good job with science”, and
“I am sure I could do advanced work in science”. The 5 items regarding expectancy
value were “I would consider a career in science”, “I expect to use science when I get
out of school”, “Knowing science will help me earn a living”, “I will need science for my
future work”, and “Science will be important to me in my life’s work”. Students’ self-
efficacy and expectancy value can be conceptualized as student motivation according
to Eccles and Wigfield’s (2002) motivational theory. Therefore, this study utilized the
construct of motivation instead of attitude. The latent variables of STEM motivation and
21st century skills consist of four constructs: math motivation, science motivation,
technology and engineering motivation, and 21st century skills, and the reliability of the
four constructs was .90, .89, .90, and .92 respectively, according to the validation study
by Unfried et al. (2015).
There were 8 items for math motivation, 9 items for science motivation, 9 items
for technology and engineering motivation, and 11 items for 21st century skills
86
(Appendix A). All the items were rated using a five-point Likert scale, from “Strongly
Disagree”, “Disagree”, “Neither Agree nor Disagree”, “Agree”, to “Strongly Agree”, with 1
indicating “Strongly Disagree” and 5 indicating “Strongly Agree”. The items No. 1, 3, 5 in
math motivation and item No. 8 in science motivation were expressed in reversed ways,
so these items were reverse coded. The S-STEM survey also included 12 items to
assess students’ interest in STEM careers. These items were descriptions of STEM-
related subject areas that involve math, science, engineering and/or technology, and
corresponding jobs connected to each subject area. The 12 STEM-related career
pathways included physics, environmental work, biology and zoology, veterinary work,
mathematics, medicine, earth science, computer science, medical science, chemistry,
energy, and engineering. The STEM career interest items used a 4-point Likert scale,
including “Not at all Interested”, “Not so Interested”, “Interested”, and “Very Interested”,
with 1 indicating “Not at all Interested”, and 4 indicating “Very Interested”. The S-STEM
survey (Friday Institute for Educational Innovation, 2012) is attached in Appendix A. A
pretest and a posttest of the online version of the S-STEM survey were administered
before the first activity and upon completion of the last activity by each teacher and in
each classroom.
Teacher Beliefs on 3D Printing Integration Survey
The teacher beliefs on 3D printing integration survey (Appendix B) consisted of a
demographic section including teachers’ names (for matching with their lesson plan and
reflection), gender, age range, and years of teaching in K-12 education; rating scales for
pedagogical beliefs, self-efficacy in 3D printing integration, 3D printing value beliefs; and
a few open-ended questions regarding teacher beliefs on 3D printing integration and the
challenges they encountered when designing and implementing the lessons.
87
Specifically, the open-ended questions included: 1. How do you feel about integrating
3D printing technology into your science teaching? 2. Do you think you have sufficient
knowledge and skills to integrate 3D printing technology in your science classes?
Please explain. 3. What are the biggest advantages in integrating 3D printing
technology in science teaching? 4. What are the biggest challenges in integrating 3D
printing technology in science teaching?
The following section focused on the scales for measuring: teacher pedagogical
beliefs, teacher self-efficacy in 3D printing integration, and teacher 3D printing value
beliefs.
Teacher pedagogical beliefs
Teacher pedagogical beliefs were measured with 15 items that assess teachers’
constructivist (student-centered learning) beliefs from three sub-scales (4 items in
subscale J1, 7 items in J2, and 4 items in J3) of the Teaching, Learning, and Computing
(TLC) survey (Becker, 2001; Ravitz, Becker, & Wong, 2000), which was developed by a
project funded by the National Science Foundation and the U.S. Department of
Education.
The three sub-scales (J1, J2, J3) assess teachers’ pedagogical beliefs with a
teacher-centered and student-centered learning continuum. Scale J1 asks teachers to
rate on how open-ended they think the class discussions should be. Scale J2 asks
teachers to indicate how much they disagree or agree with a few statements about
teaching and learning. The original scale of J2 used a 6-point Likert scale without the
neutral option (strongly disagree, moderately disagree, slightly disagree, slightly agree,
moderately agree, strongly agree). The current study modified it with a 5-point Likert
scale (strongly disagree, disagree, neither disagree nor agree, agree, strongly agree) to
88
make it consistent with the 5-point Likert scale in scales J1 and J3, and scales in the
other sections of the teacher beliefs on 3D printing technology integration survey. Scale
J3 provides two opposite teaching philosophy statements on student-centered learning
and teacher-centered learning with five radio buttons in the middle for teachers to select
which statement they are more inclined to. In J2, items No. 5, 6, 7, 10, and 11, and all
the items in J3 were expressed in reversed ways, so these items were all reversed
before further analysis.
The validity of the entire TLC survey was verified by classroom observation and
teacher interview with 72 teachers in 24 schools in the United States and the reliability
(Cronbach’s alpha) of 13 out of the 15 items (2 items in J1, 7 items in J2, and 4 items in
J3) of TLC was 0.83 (Ravitz et al., 2000). The reliability verification did not include two
other items in J1. However, in the study of Kim et al. (2013), the reliability (Cronbach’s
alpha) of all the four items in J1 was 0.92. Therefore, the current study included all the 4
items in J1, with 15 items in total. An overview of the scale with the 15 items is attached
in Appendix B.
Teacher self-efficacy in 3D printing technology integration
Teachers’ self-efficacy in technology integration was measured using an adapted
version of the TPACK survey that was developed and validated by Schmidt et al. (2009)
to measure pre-service teachers’ technology integration knowledge. The content validity
of the original TPACK survey was evaluated and verified by experts in TPACK. The
construct validity was verified using principal components factor analysis for each
knowledge domain which yielded acceptable factor loadings. The reliability (Cronbach’s
alpha) for each knowledge domain ranged from 0.80 to 0.92.
89
The adapted scale for this study did not add the sections designed specifically for
pre-service teachers. Since all teachers in this study were in-service science teachers
and the classes were science classes, the survey excluded the items about how pre-
service teachers’ professors model combining content, technologies, and teaching
approaches in their teaching of different subjects, and items focusing on content other
than science. Items about general technologies were also adapted as 3D printing
technology if they were possible to be modified. The other sections about teachers’ self-
efficacy in different types of TPACK knowledge also apply to in-service teachers.
Specifically, the survey for this study included teachers’ self-efficacy in Technology
Knowledge (TK), Content Knowledge (CK), Pedagogical Knowledge (PK), Pedagogical
Content Knowledge (PCK), Technological Content Knowledge (TCK), Technological
Pedagogical Knowledge (TPK), and Technological Pedagogical Content Knowledge
(TPACK). The scale on teacher self-efficacy in 3D printing technology integration is
attached in Appendix B.
Teacher 3D printing value beliefs
The teacher 3D printing value beliefs scale was adapted from Eccles and
Wigfield’s (1995) survey to measure teachers’ intrinsic interest value, attainment value
(importance), and extrinsic utility value (usefulness) regarding integrating 3D printing
technology into their science classrooms.
The items of the original scale were developed with the guidance of the
expectancy-value theory of motivation and were validated (Eccles & Wigfield, 1995).
The reliability (Cronbach’s alpha) of the original scale was acceptable with 0.76, 0.70,
and 0.62 for the intrinsic interest value, attainment value, and extrinsic utility value
respectively given there were only 2 to 3 items in each subscale. Cheng and Xie (2018)
90
adapted the original scale to measure teachers’ value beliefs about learning digital
content evaluation and treated the value beliefs as a unidimensional construct, which
showed good reliability with Cronbach’s alpha of 0.88 and 0.91 of the scale
administered at the beginning and end of their professional development program.
In the current scale on teachers’ 3D printing value beliefs, there were 2 items to
measure the intrinsic interest value in integrating 3D printing technology, 3 items to
measure the perceived importance of doing well in integrating 3D printing technology,
and 2 items to measure the perceived usefulness of integrating 3D printing technology
to enhance engage students and enhance student learning. All the 7 items in the scale
used a 5-point Likert scale. The scale is attached in Appendix B.
Lesson Plan Codebook
Lesson plan codebooks were developed to code the 3D printing integration
levels, STEM integration levels, and school levels including elementary, middle, and
high school levels. The original coding plan also included teachers’ 3D printing
implementation duration. However, the duration information provided in the lesson plans
did not allow consistent coding. For instance, some teachers just provided the number
of days but did not provide specific class periods information. Although some teachers
provided the class periods information, the time for one class period varied from teacher
to teacher. Therefore, the final coding only included teachers’ 3D printing integration
levels and STEM integration levels without the implementation duration.
3D printing integration levels
Researchers have developed a few technology integration assessment
instruments based on the TPACK framework to evaluate teachers’ technology
integration in the classrooms. The general technology integration assessment
91
instrument developed by Britten and Cassady (2005) and the TPACK-based technology
integration assessment instrument by Harris, Grandgenett, and Hofer (2010) did not
focus on any specific technology or content area. Pringle, Dawson, and Ritzhaupt
(2015) developed a technology integration assessment instrument based on the TPACK
framework to evaluate teachers’ technology integration in science classes. All the
technology integration assessment instruments mentioned here focus on the general
technology integration which encompasses different technologies teachers integrate in
their classes.
Because the current study focused on the integration of a specific technology –
3D printing technology in science classes, the assessment instrument for this study was
adapted from the general TPACK-based technology integration assessment instruments
to specifically focus on the integration of 3D printing technology in science classes. As
teachers’ technological pedagogical content knowledge is the core of the TPACK
framework, this instrument just focused on teachers’ technological pedagogical content
knowledge (TPACK) related to the integration of 3D printing technology. In addition to
referencing the technology integration assessment instruments developed by other
researchers, the instrument for this study was also guided by the Technology
Integration Matrix (TIM) which illustrated the levels of technology integration and
meaningful learning environments.
The levels of 3D printing integration included entry, adoption, adaptation,
infusion, and transformation (Harmes et al., 2016). Specifically, the levels were
determined by how 3D printing was used and how the integration engaged students in a
meaningful learning environment and facilitated higher-order learning activities such as
92
apply, analyze, evaluate, and create (Krathwohl, 2002). The levels were scored from 1
to 5. Table 3-3 provides the definition and the coding and scoring criteria.
Table 3-3. Codebook for 3D printing integration levels
Category Criteria Score
3D printing integration levels
Entry: The teacher just introduced 3D printing technology or brought in 3D printed objects, students had no access to 3D printing technology and had minimal access to 3D printed objects, 3D printing technology did not contribute to the learning environment or student learning.
1
Adoption: Students had access to 3D printing technology or 3D printed objects, but the technologies were not used during the learning activity. The use of 3D printing technology or 3D printed objects minimally contributed to a meaningful learning environment and student learning.
2
Adaptation: Students used 3D printed objects for some part of the learning activity but there was no deep interaction. The use of 3D printing technology or 3D printed objects contributed to a meaningful learning environment and student learning but not strong.
3
Infusion: Students used 3D printed objects throughout or for the most part of the learning activity. The use of 3D printing technology or 3D printed objects strongly contributed to a meaningful learning environment, but the 3D printing technology was not used to facilitate higher-order learning activities such as apply, analyze, evaluate, and create.
4
Transformation: Students participated in the 3D printing process, printed 3D models, and used 3D printed objects throughout or for the most part of the learning activity. The use of 3D printing technology and 3D printed objects strongly contributed to a meaningful learning environment and the 3D printing technology was used to facilitate higher-order learning activities such as apply, analyze, evaluate, and create.
5
93
STEM integration levels
A codebook with scoring criteria was created to evaluate teachers’ STEM
integration levels (Table 3-4). The codebook was created by focusing on the integration
of STEM content. As required by the iDigFossils project, all teachers indicated how they
designed learning activities to facilitate student collaboration in the classrooms and all
the teachers designed inquiry-based learning activities. Some teachers might have used
more instructional strategies than other teachers and some may have also used other
instructional strategies, however, there were no consistent ways to compare which
instructional strategy would be more effective. According to the systematic review and
meta-analysis articles on STEM integration (e.g., Mustafa, Ismail, Tasir, Said, &
Haruzuan 2016; Thibaut et al., 2018), there are a myriad of instructional strategies that
have been used in STEM integration, but there is no evidence on which instructional
strategy would be more effective than another one or whether STEM integration with
more instructional strategies would be necessarily more effective than STEM integration
with fewer strategies. Therefore, this study focused exclusively on the integration of
STEM content without including the instructional strategies used by teachers. The initial
coding found there were no integration of science, 3D printing technology, and
engineering, so the codebook included four criteria as shown in Table 3-4. The STEM
integration levels were scored from 1 to 4.
Table 3-4. Codebook STEM integration levels
Criteria Score
Science 1
Integration of STEM content
Science + 3D Printing Technology 2
Science + 3D Printing Technology + Math 3
Science + 3D Printing Technology + Math + Engineering 4
94
Data Sources and Data Collection
The data sources included: S-STEM survey, which assessed students’ science
motivation, technology/engineering motivation, math motivation, 21st century skills, and
interest in STEM careers before and after the 3D printing integration; teacher beliefs
survey, which investigated teachers’ pedagogical beliefs, 3D printing value beliefs, and
self-efficacy beliefs in 3D printing integration; and teachers’ lesson plans, which
provided data on teachers’ 3D printing integration levels and STEM integration levels. A
pretest of the S-STEM survey was administered by teachers in their science classroom
before the first activity, and a posttest with the same S-STEM survey was administered
upon completion of the last activity. Teachers’ lesson plans and teacher beliefs data
were collected after the teachers completed their 3D printing integrated science classes.
Data Analysis
This study used descriptive statistical analysis, lesson plan analysis, Pearson’s
Correlation analysis, multilevel modeling analysis, and multiple regression analysis to
answer the two research questions. To explain the relationship between teacher beliefs
and teachers’ 3D printing integration, this study also conducted thematic analysis for the
open responses in the teacher beliefs survey. Table 3-5 displays an overview of the
data analysis for each research question. This study intended to include school levels in
the data analysis, however, the sample size of different school levels was unbalanced.
Most of the classes were middle or high school levels and few were elementary levels
(Elementary school: N = 4 teachers with 192 students; Middle school: N = 13 teachers
with 847 students; High school: N = 9 teachers with 462 students.). Therefore, school
levels were only reported in the descriptive statistics without further analysis.
95
Table 3-5. Overview of data analysis
Research Questions Data Sources Data Analysis
RQ1: How are teachers’ beliefs correlated with their 3D printing integration in the science classrooms?
1. Lesson plan
2. Teacher beliefs on technology integration survey.
1. Descriptive statistical analysis for teacher beliefs survey;
2. Lesson plan analysis: 3D printing technology integration levels and STEM integration levels;
3.Pearson’s Correlation analysis with teacher beliefs, 3D printing integration levels, and STEM integration levels;
4. Thematic analysis for open responses in the teacher beliefs survey.
RQ2: How do teachers’ beliefs and their 3D printing integration in the science classrooms predict students’ STEM motivation, 21st century skills, and interest in STEM careers?
1. S-STEM pre- and post-survey;
2. Lesson plan.
1. Descriptive statistical analysis for S-STEM survey;
2. Lesson plan analysis: 3D printing technology integration levels and STEM integration levels;
3. Multilevel modeling analysis with student gender and pretest scores as student-level independent variables, 3D printing integration levels, STEM integration levels, and each component of teacher beliefs variables as teacher-level independent variables, and student posttest scores in science motivation, technology/engineering motivation, math motivation, and 21st century skills as the dependent variable respectively.
4. Multiple regression analysis with student posttest scores in their interest in STEM careers as the dependent variable, and all the student variables (student gender and pretest scores) and teacher variables (3D printing integration levels, STEM integration levels, and each component of teacher beliefs variables) as independent variables at the same level.
96
Data Analysis for RQ1
To address research question 1, teacher beliefs data and teachers’ lesson plans
were analyzed. First, descriptive statistical analysis was conducted; second, teachers’
lesson plans were analyzed; third, correlational analysis was conducted using
Pearson’s correlation; and lastly, thematic analysis was conducted for teachers’ open
responses in the teacher beliefs survey.
Descriptive statistical analysis
To conduct descriptive statistical analysis, an aggregated average score,
standard deviation, and the minimum and maximum were calculated for each
component of the teacher beliefs: pedagogical beliefs, self-efficacy beliefs, and 3D
printing value beliefs.
Lesson plan analysis
To investigate teachers’ 3D printing integration features, teachers’ lesson plans
were analyzed. The lesson plans were coded for 3D printing integration levels and
STEM integration levels. A few professors in educational technology reviewed the
codebooks and provided suggestions, and revisions were made accordingly, which
ensured the validity of the codebooks. To ensure scoring reliability, I scored all the
lesson plans using the codebooks and a week later I scored all the lesson plans again
to check the percentage of the consistency. There was about 92.3% consistency
between the scoring. The only 2 lesson plans that I scored differently in the previous
and later scoring were due to that I did not find some specific information when I scored
them the first time. The discrepancies were reviewed, and corrections were made
accordingly.
97
Correlational analysis
The reliability of the survey was evaluated with Cronbach’s alpha. Pearson’s
correlation was conducted to examine the directionality and degree of association
between teacher beliefs, 3D printing integration levels, and STEM integration levels.
Thematic analysis
Thematic analysis was conducted to analyze teachers’ responses to the open-
ended questions in the teacher beliefs survey. According to Braun and Clarke (2006),
thematic analysis is “a method for identifying, analysing and reporting patterns (themes)
within data” (p. 79). The analysis of the open responses followed the six phases of
thematic analysis (Table 3-6) developed by Braun and Clarke (2006).
To ensure reliability of the coding and themes searching and reviewing, after one
week of the original coding, I selected a random sample of one-third of the scripts to
code and search themes again. The coding and themes were compared to the original
coding and themes, and there were few discrepancies except using some different
expressions which had the same meanings, for instance, lack of time and insufficient of
time. The responses for each question were analyzed separately, then the themes for
each question were synthesized across the questions.
Table 3-6. Phases of thematic analysis (from Braun & Clarke, 2006)
Phase Description of the process
1. Familiarizing yourself with your data:
Transcribing data (if necessary), reading and re-reading the data, noting down initial ideas.
2. Generating initial codes:
Coding interesting features of the data in a systematic fashion across the entire data set, collating data relevant to each code.
3. Searching for themes: Collating codes into potential themes, gathering all data relevant to each potential theme.
98
Table 3-6. Continued
Phase Description of the process
4. Reviewing themes: Checking if the themes work in relation to the coded extracts (Level 1) and the entire data set (Level 2), generating a thematic ‘map’ of the analysis.
5. Defining and naming themes:
Ongoing analysis to refine the specifics of each theme, and the overall story the analysis tells, generating clear definitions and names for each theme.
6. Producing the report: The final opportunity for analysis. Selection of vivid, compelling extract examples, final analysis of selected extracts, relating back of the analysis to the research question and literature, producing a scholarly report of the analysis.
Data Analysis for RQ2
To address research question 2, the S-STEM survey data were analyzed, and
teacher beliefs survey data and lesson plans were already analyzed when addressing
research question 1. Descriptive statistical analyses were conducted using the S-STEM
survey data, then multilevel modeling analyses were conducted using the S-STEM
survey data including students’ science motivation, technology/engineering motivation,
math motivation, and 21st century skills, teacher beliefs survey data, and lesson plan
data. Multiple regression analysis was conducted for students’ interest in STEM careers
because it was found that multilevel modeling analysis did not work for it.
Descriptive statistical analysis
Descriptive statistical analyses were conducted with the pretest and posttest data
of STEM motivation, STEM career interests, and 21st century skills to provide the mean
and standard deviation for each construct. As STEM motivation and 21st century skills
were rated with 5-point Likert scales, an aggregated average score was calculated for
99
science motivation, math motivation, engineering/technology motivation, and 21st
century skills for each student.
As suggested by Wiebe, Unfried, and Faber (2018), students’ understanding of
specific occupations is not very likely to have matured in the younger grades and
students study STEM subject areas in school but not necessarily study specific STEM
careers in school. Therefore, an aggregated average score of students’ ratings on the
12 STEM career interest items was calculated for the pre-survey and post-survey and
this average score represented a student’s interest in STEM careers.
Multilevel modeling analysis
This study involved two levels of factors: level-1 factors, i.e., student-level
factors, and level-2 factors, i.e., teacher-level factors. Student-level factors included
student gender and the pretest scores of STEM motivation, interest in STEM careers,
and 21st century skills. Teacher-level factors included teachers’ 3D printing integration
levels and STEM integration levels. The response variables were students’ posttest
scores on STEM motivation, 21st century skills, and interest in STEM careers. Multiple
regression analysis with just the student-level factors or the teacher-level factors would
omit the variances of the other level of factors that were not included in analysis. Since
there were two levels of factors that were of interest in this study, a more advanced
regression analysis, i.e. multilevel modeling analysis, would be a necessary approach to
address the potential influence of the two levels of factors (Peugh, 2010). As there were
a few response variables in this study, a multilevel modeling analysis was independently
conducted with each response variable and its corresponding student-level factors, for
instance, students’ science motivation posttest score as the response variable, and
student gender and science motivation pretest score as the student-level factor. All the
100
teacher-level factors were included in the multilevel modeling analysis. The software
program SAS 9.4 was used for all the multilevel modeling analyses.
The multilevel modeling analyses of this study were guided by the major steps
proposed by Peugh (2010): clarifying the research question under investigation,
choosing the correct parameter estimation method (i.e., full information or restricted
maximum likelihood), assessing whether multilevel modeling is needed, building the
level-1 model, building the level-2 model, comparing nested models using likelihood
ratio test, and reporting multilevel effect sizes).
Clarify the research question. Research question 2 is: How do teacher beliefs
and their 3D printing integration in the science classrooms predict students’ STEM
motivation, 21st century skills, and interest in STEM careers? The primary interest was
to examine the relationship between teacher-level factors and student outcome
variables. As students were nested within teachers, the student-level factors (student
gender and pretest scores) were treated as covariates to be controlled when analyzing
the relationship between teacher-level factors and student outcome variables.
Choose parameter estimation method. After clarifying the research question,
the next step is to choose the correct parameter estimation method. Due to the small
sample size at the teacher level (N < 50), the restricted maximum likelihood (REML)
estimation was adopted because REML can estimate variances more accurately than
full information maximum likelihood (FIML) estimation when the higher-level sample size
is small (Peugh, 2010).
Assess whether multilevel modeling is needed. The next step is to assess
whether multilevel modeling was needed by building the baseline model since nested
101
datasets may not require multilevel modeling if there is no response variable variation at
level-2 (Peugh, 2010), which basically meant that the level-2 factors have no impact on
the response variable and the level-1 units can be treated as independent observations
using OLS multiple regression (Peugh, 2010). Therefore, whether multilevel modeling is
needed depended on how much response variable variation was in level 2 (namely the
teacher level), which involved calculating the intraclass correlation (ICC) and the design
effect statistics (Peugh, 2010). If there is variation in the mean response variable scores
(e.g., student science motivation posttest scores) across teachers, multilevel modeling
is needed to separately estimate the variance of response variable scores that occurs at
both across students and across teachers. To test the variance in the mean response
variable scores, the next step is to build an unconditional model, namely a baseline
model that examines the variation in mean response variable scores without including
the student-level and teacher-level factors yet.
The baseline model is shown by the following equations (Hox, 2002;
Raudenbush & Bryk, 2002, as cited in Peugh, 2010):
Student Level: 𝑌𝑖𝑗 = 𝛽0𝑗 + 𝑖𝑗 (3-1)
Teacher Level: 𝛽0𝑗 = 𝛾00 + 𝑢0𝑗 (3-2)
Combined: 𝑌𝑖𝑗 = 𝛾00 + 𝑢0𝑗 + 𝑖𝑗 (3-3)
To make it easier to illustrate the multilevel modeling analysis procedure, the
following analysis focuses on a specific response variable – students’ science
motivation posttest scores. The analysis procedures apply to the other response
variables. The meanings of the symbols in equations (3-1), (3-2), and (3-3) are provided
in Table 3-7.
102
Table 3-7. The meanings of symbols in equations (3-1), (3-2), and (3-3)
Symbol Meaning
𝑌𝑖𝑗 The science motivation posttest score of student i of teacher j.
𝛽0𝑗 The students’ mean science motivation posttest score for teacher j.
𝑖𝑗 A residual term – individual student differences around the mean of teacher j.
𝛾00 Grand-mean science motivation posttest score across all teachers.
𝑢0𝑗 Deviation/difference between the mean score for each teacher and the grand mean.
ICC is the proportion of student science motivation posttest score variance that
can be explained by the mean student science motivation posttest score differences
across teachers.
𝐼𝐶𝐶 = 𝜎𝑢02 / (𝜎𝑢0
2 + 𝜎𝜀2) (3-4)
where 𝜎𝑢02 is the variance of 𝑢0𝑗 and 𝜎𝜀
2 is the variance of 𝑖𝑗.
ICC values between 0.05 and 0.20 are common in multilevel modeling analysis
applications in social research studies (Peugh, 2010).
The design effect is “an estimate of the multiplier that needs to be applied to
standard errors to correct for the negative bias that results from nested data” (Peugh,
2010, p. 91) and the equation is:
Design Effect = 1 + (𝑛𝑐 - 1) ICC (3-5)
where 𝑛𝑐 is the average number of students per teacher.
103
A design effect estimates greater than 2.0 indicates a need for multilevel
modeling analysis (Muthén, 1991, 1994; Muthén & Satorra, 1989, 1995, as cited in
Peugh, 2010).
If the ICC values are lower than 0.05 and the design effect estimate is less than
2.0, multiple regression analysis will be conducted to examine research question 2. If
the ICC value for the response variable is greater than 0.05 and/or the design effect
estimate is greater than 2.0, the multilevel modeling analysis will be continued with
further procedures and the next step is to build the level-1 model.
Build the student-level model. The student-level model adds students’ science
motivation pretest scores into the model. As predictor variables were measured with
scales that did not contain zero, a score of zero would have no substantive meaning,
therefore, the method of centering which involves rescaling a predictor variable is
necessary to make a value of zero can be interpreted meaningfully (Peugh, 2010). A
grand mean centering is adopted because students’ pretest scores were independent of
each other’s scores. The student-level model starts with a random-intercept model
which estimates the impact of students’ pretest scores on the posttest scores as a fixed
effect, indicating the impact of students’ pretest scores on posttest scores did not vary
across teachers. The equations for the random-intercept model with grand-mean
centering are as follows:
Student Level: 𝑌𝑖𝑗 = 𝛽0𝑗 + 𝛽1𝑗(𝑃𝑅𝐸𝑖𝑗 − 𝑃𝑅𝐸) + 𝛽2𝑗𝐺𝑒𝑛𝑑𝑒𝑟_𝑆𝑡𝑢𝑑𝑒𝑛𝑡𝑖𝑗 + 𝑖𝑗 (3-6)
Teacher Level: 𝛽0𝑗 = 𝛾00 + 𝑢0𝑗 (3-7)
𝛽1𝑗 = 𝛾10 (3-8)
𝛽2𝑗 = 𝛾20 (3-9)
Combined: 𝑌𝑖𝑗 = 𝛾00 + 𝛾10(𝑃𝑅𝐸𝑖𝑗 − 𝑃𝑅𝐸) + 𝛾20𝐺𝑒𝑛𝑑𝑒𝑟_𝑆𝑡𝑢𝑑𝑒𝑛𝑡𝑖𝑗 + 𝑢0𝑗
+ 𝑖𝑗
(3-10)
104
where 𝛽1𝑗 is the regression coefficient that shows the impact of pretest scores on
posttest scores across all students of teacher j , 𝛽2𝑗 is the regression coefficient that
shows the impact of student gender on posttest scores across all students of teacher j
𝑃𝑅𝐸𝑖𝑗 is pretest score of student i of teacher j, 𝑃𝑅𝐸 is the grand mean of science
motivation pretest scores across teachers, 𝛾10 is the average effect of pretest scores on
posttest scores across all teachers, and 𝛾20 is the average effect of student gender on
posttest scores across all teachers.
However, if the impacts of student gender and students’ pretest scores on
posttest scores vary significantly across teachers, variance components (i.e., random
effects) would need to be added to the teacher-level slope equation to model this
variation (Peugh, 2010). The teacher level equations for the random-slope model are as
follows:
Teacher Level: 𝛽0𝑗 = 𝛾00 + 𝑢0𝑗 (3-11)
𝛽1𝑗 = 𝛾10 + 𝑢1𝑗 (3-12)
𝛽2𝑗 = 𝛾20 + 𝑢2𝑗 (3-13)
Combined: 𝑌𝑖𝑗 = 𝛾00 + 𝛾10(𝑃𝑅𝐸𝑖𝑗 − 𝑃𝑅𝐸) + 𝛾20𝐺𝑒𝑛𝑑𝑒𝑟_𝑆𝑡𝑢𝑑𝑒𝑛𝑡𝑖𝑗 +
𝑢1𝑗 (𝑃𝑅𝐸𝑖𝑗 − 𝑃𝑅𝐸) + 𝑢2𝑗𝐺𝑒𝑛𝑑𝑒𝑟_𝑆𝑡𝑢𝑑𝑒𝑛𝑡𝑖𝑗 + 𝑢0𝑗 + 𝑖𝑗
(3-14)
where 𝑢1𝑗 is a residual term to show a random effect, indicating the impact of
pretest score on posttest score can vary randomly across teachers, and 𝑢2𝑗 is a residual
term to show a random effect, indicating the impact of student gender on posttest score
can vary randomly across teachers. Multilevel modeling analysis does not estimate the
residuals, but the variance of the residuals (Peugh, 2010).
Before proceeding to the next model during each step, a likelihood ratio test
needs to be conducted to examine whether the more complicated model fits better than
105
the simpler model. If the more complicated model fits better, this model is continually
built on with more parameters. Otherwise, the simpler model will be kept.
Build the teacher-level model. The teacher-level model adds the teacher-level
factors into the student-level model. Teacher level factors include 3D printing integration
level, STEM integration level, and teacher beliefs, which consisted of pedagogical
beliefs, technology value beliefs, and self-efficacy beliefs in 3D printing integration, and
the value beliefs and self-efficacy beliefs each had a few subcomponents. Before
adding these variables into the teacher-level model, the multicollinearity between the
variables had to be assessed. Variables that have high multicollinearity would not be
added into the teacher-level model. The equations for each of the student outcome
variables might be different depending on which teacher-level variables would be
added, therefore, the specific equations for the teacher-level model are presented in the
results section along with the specific analysis steps. This teacher-level model is also
considered as a full model with both student-level and teacher-level variables.
Report multilevel effect sizes. In addition to the regression coefficients of the
student-level and teacher-level factors, the effect sizes at both levels will be calculated
and reported. The effect size at student-level can be obtained by comparing the
random-intercept model and the baseline model.
𝑅𝐿12 =
𝜎𝜀 𝑏𝑎𝑠𝑒𝑙𝑖𝑛𝑒2 − 𝜎𝜀 𝑟𝑎𝑛𝑑𝑜𝑚−𝑖𝑛𝑡𝑒𝑟𝑐𝑒𝑝𝑡
2
𝜎𝜀 𝑏𝑎𝑠𝑒𝑙𝑖𝑛𝑒2
(3-15)
𝑅𝐿12 measures how much student-level variance is explained by the random-
intercept model compared to the baseline model.
The effect size at teacher-level can be obtained by comparing the full model and
the random-slope model if the random-slope model fits better than the random-intercept
106
model. Otherwise, the effect size would be obtained by comparing the full model and
the random-intercept model.
𝑅𝐿22 =
𝜎𝜀 𝑟𝑎𝑛𝑑𝑜𝑚−𝑠𝑙𝑜𝑝𝑒2 − 𝜎𝜀 𝑓𝑢𝑙𝑙
2
𝜎𝜀 𝑟𝑎𝑛𝑑𝑜𝑚−𝑠𝑙𝑜𝑝𝑒2
(3-16)
𝑅𝐿22 measures how much teacher-level variance is explained by the full model
compared to the random-slope model, namely, how much variance is explained by the
teacher-level variables.
Multiple regression analysis
After a few initial steps with multilevel modeling analysis, it was found that the
nested structure of students within teachers did not contribute to explaining the
variances in teacher-level variables for students’ interest in STEM careers. Therefore,
multiple regression analysis was conducted by treating students as independent of each
other and all the student and teacher variables were at the same level without nesting.
The dependent variable was the posttest scores of students’ interest in STEM
careers. The independent variables were student gender, pretest scores of students’
interest in STEM careers, teachers’ 3D printing integration levels, STEM integration
levels, each component of the teacher beliefs variables, and interactions between the
student variables and teacher variables. Except for student gender, all the other
variables were centered with their grand mean. The assumptions of multiple regression
analysis were examined, including normality assumption of the dependent variable, no
multicollinearity, normal distribution of residuals, and no autocorrelation in the residuals.
SAS 9.4 was used for the multiple regression analysis. The specific procedures of the
data analysis with the results have been reported in Chapter 4.
107
CHAPTER 4 RESULTS
This chapter is organized by presenting the descriptive statistics of the variables
included in the correlational analysis, multilevel modeling analysis, and multiple
regression analysis; the internal consistency of the rating scales; correlations between
the variables; missing data evaluation; assumptions testing; and then the multilevel
modeling analysis results for each of the dependent variables: students’ science
motivation, technology/engineering motivation, math motivation, and 21st century skills;
and the multiple regression analysis results for students’ interest in STEM careers.
Descriptive Statistics of Variables
The variables included in the analyses consisted of dependent variables,
student-level independent variables, and teacher-level independent variables. The
dependent variables included students’ posttest scores in science motivation,
technology/engineering motivation, math motivation, 21st century skills, and interest in
STEM careers. The student-level independent variables included student gender and
students’ pretest scores in math, science, and technology/engineering motivation, 21st
century skills, and interest in STEM careers. The teacher-level independent variables
included teachers’ 3D printing integration levels, STEM integration levels, pedagogical
beliefs, 3D printing value beliefs (interest in 3D printing integration, perceived
importance of 3D printing integration, perceived usefulness of 3D printing integration),
and self-efficacy beliefs in 3D printing integration in science classrooms (self-efficacy in
TK, PK, CK, TPK, TCK, PCK, and TPACK). The variable names and their meanings are
presented in Table 4-1.
108
Table 4-1. Variable names and their meanings Variable Categories
Variable Names Variable Meanings
Dependent variables
Post_Science Science motivation posttest scores
Post_TechEngi Technology/Engineering motivation posttest scores
Post_Math Math motivation posttest scores
Post_21st 21st century skills posttest scores
Post_Career Interests in STEM careers posttest scores
Student-level independent variables
Pre_Math Math motivation pretest scores
Pre _Science Science motivation pretest scores
Pre _TechEngi Technology/Engineering motivation pretest scores
Pre _21st 21st century skills pretest scores
Pre_Career Interests in STEM careers pretest scores
Gender_Student Student gender
Teacher-level independent variables
Printing_Level 3D printing integration levels
STEM_Level STEM integration levels
Pedagogical_Beliefs Teachers’ pedagogical beliefs
Interest_Teacher Teachers’ intrinsic interest value in 3D printing integration
Importance_Teacher Teachers’ attainment value (perceived importance) of 3D printing integration
Usefulness_Teacher Teachers’ extrinsic utility value (perceived usefulness) of 3D printing integration
Self_Efficacy_TK Teachers’ self-efficacy in Technological Knowledge
Self_Efficacy_PK Teachers’ self-efficacy in Pedagogical Knowledge
Self_Efficacy_CK Teachers’ self-efficacy in Content Knowledge
Self_Efficacy_TPK Teachers’ self-efficacy in Technological Pedagogical Knowledge
Self_Efficacy_TCK Teachers’ self-efficacy in Technological Content Knowledge
Self_Efficacy_PCK Teachers’ self-efficacy in Pedagogical Content Knowledge
Self_Efficacy_TPACK Teachers’ self-efficacy in Technological Pedagogical Content Knowledge
109
Dependent Variables
Students’ posttest scores in science motivation, technology/engineering
motivation, math motivation, 21st century skills, and interest in STEM careers were the
average scores of the corresponding items in the subscales of the S-STEM survey. The
number of students who completed the subscales, the mean, standard deviation, the
minimum score, and the maximum score for all the students’ posttest scores can be
viewed in Table 4-2. Science motivation, technology/engineering motivation, math
motivation, and 21st century skills were measured with 5-point Likert scales and interest
in STEM careers were measured with a 4-point Likert scale. For the posttest scores of
all the students, the posttest scores in science motivation, technology/engineering
motivation, math motivation, and 21st century skills ranged from 1 to 5 and the posttest
scores of students’ interest in STEM careers ranged from 1 to 4.
Table 4-2. Descriptive statistics for dependent variables
Variable N M SD Min Max
Post_Science 1492 3.5747 0.7126 1.0000 5.0000
Post_TechEngi 1485 3.4941 0.7364 1.0000 5.0000
Post_Math 1501 3.5601 0.8760 1.0000 5.0000
Post_21st 1482 4.0748 0.5522 1.0000 5.0000
Post_Career 1476 2.4230 0.5124 1.0000 4.0000
Student-Level Independent Variables
Student gender was included in the multilevel modeling analysis. There were 712
males and 789 females. Consistent with how students’ posttest scores were calculated,
students’ pretest scores in science motivation, technology/engineering motivation, math
motivation, 21st century skills, and interest in STEM careers were the average scores of
110
the corresponding items in the subscales of the S-STEM survey. The number of
students who completed the subscales, the mean, standard deviation, the minimum
score, and the maximum score for all the students’ pretest scores are in Table 4-3.
Table 4-3. Descriptive statistics for student-level independent variables
Variable N M SD Min Max
Gender_Student 0 = Male: N = 712; 1 = Female: N = 789.
/ / / /
Pre _Science 1497 3.5761 0.6877 1.2222 5.0000
Pre _TechEngi 1490 3.5356 0.6911 1.0000 5.0000
Pre_Math 1501 3.5396 0.8448 1.0000 5.0000
Pre _21st 1482 4.0926 0.5009 1.4545 5.0000
Pre_Career 1475 2.4582 0.5139 1.0000 4.0000
Teacher-Level Independent Variables
Teachers’ 3D printing integration levels and STEM integration levels were rated
with their lesson plans. Teachers’ 3D printing integration levels ranged from 2 to 5, with
the mean of 3.6136 and standard deviation of 1.0411. Teachers’ STEM integration level
ranged from 2 to 4, with the mean of 3.2352 and standard deviation of 0.6831.
Among the 26 teachers whose lesson plans were analyzed, 24 teachers
responded to the survey on their pedagogical beliefs and value beliefs and self-efficacy
beliefs in 3D printing integration in science classrooms. Teachers’ pedagogical beliefs
ranged from 2.8000 to 4.4667, with the mean of 3.5577 and standard deviation of
0.4654. Teachers’ value beliefs consisted of interest, perceived importance, and
perceived usefulness of 3D printing integration in science classrooms. On average,
teachers scored high on the three subscales of the value beliefs. The average scores of
111
all the teachers’ self-efficacy beliefs varied from 3.5123 in self-efficacy in TK to 4.4966
in self-efficacy in PK. The mean, standard deviation, minimum score, and maximum
score of all the teacher-level independent variables can be viewed in Table 4-4.
Table 4-4. Descriptive statistics for teacher-level independent variables
Variable Teacher N M SD Min Max
Printing_Level 26 3.6136 1.0411 2.0000 5.0000
STEM_Level 26 3.2352 0.6831 2.0000 4.0000
Pedagogical_Beliefs 24 3.5577 0.4654 2.8000 4.4667
Interest_Teacher 24 4.3805 0.5389 3.5000 5.0000
Importance_Teacher 24 3.9764 0.5720 3.0000 5.0000
Usefulness_Teacher 24 4.4395 0.4678 3.5000 5.0000
Self_Efficacy_TK 24 3.5123 0.6985 2.2000 5.0000
Self_Efficacy_PK 24 4.4966 0.4378 3.4286 5.0000
Self_Efficacy_CK 24 4.2085 0.5139 2.7500 5.0000
Self_Efficacy_TPK 24 4.0540 0.4945 3.0000 5.0000
Self_Efficacy_TCK 24 4.0068 0.5182 3.0000 5.0000
Self_Efficacy_PCK 24 4.3256 0.5845 3.0000 5.0000
Self_Efficacy_TPACK 24 3.9263 0.4754 3.0000 5.0000
Internal Consistency
The internal consistency of the value beliefs scales, pedagogical beliefs scales,
self-efficacy beliefs scales, and S-STEM survey were assessed with Cronbach’s alpha
(see Table 4-5). According to commonly accepted rules, a Cronbach’s alpha of above .8
is good, between .7 and .8 is acceptable, and between .6 and .7 is questionable. The
intrinsic interest value and attainment value/importance had acceptable internal
112
consistency with Cronbach’s alpha of .776 and .803 respectively. The internal
consistency of extrinsic utility value/usefulness approached the acceptable level with
Cronbach’s alpha of .667, and there was a small number of items in this scale, so it was
considered as acceptable. The Cronbach’s alpha of the overall value beliefs scale was
.869. The Cronbach’s alpha of pedagogical beliefs scale was .727, which was
acceptable. In terms of the internal consistency of self-efficacy beliefs scales, the
Cronbach’s alpha of self-efficacy in TK, PK, CK, TPK, and TPACK were .844, .878,
.652, .841, and .607 respectively. The Cronbach’s alpha of the overall self-efficacy
beliefs scale was .909. As TCK and PCK only had one item in the scale, the Cronbach’s
alpha was not available. The Cronbach’s alpha of self-efficacy in CK and TPACK were
questionable, but they were considered as acceptable in this study due to the small
number of items in the scale. The Cronbach’s alpha of the S-STEM survey including the
pretest and posttest of students’ math motivation, science motivation,
technology/engineering motivation, 21st century skills, and interest in STEM careers
ranged from .801 to .916. All these Cronbach’s alphas were above .800, indicating good
internal consistency of the scales.
Table 4-5. Cronbach’s alpha of rating scales
Scale Cronbach’s Alpha
Intrinsic interest value (2 items) .776
Attainment value/importance (3 items) .803
Extrinsic utility value/usefulness (2 items) .667
Overall value beliefs (7 items) .869
Pedagogical beliefs (15 items) .727
113
Table 4-5. Continued
Scale Cronbach’s Alpha
Self-efficacy in TK (5 items) .844
Self-efficacy in PK (7 items) .878
Self-efficacy in CK (4 items) .652
Self-efficacy in TPK (5 items) .841
Self-efficacy in TCK (1item) /
Self-efficacy in PCK (1 item) /
Self-efficacy in TPACK (3 items) .607
Overall self-efficacy beliefs (26 items) .909
Math motivation (8 items) .908 (pre); .916 (post)
Science motivation (9 items) .881 (pre); .892 (post)
Technology/Engineering motivation (9 items) .866 (pre); .890 (post)
21st century skills (11 items) .866 (pre); .902 (post)
Interest in STEM careers (12 items) .801 (pre); .804 (post)
Correlations between Variables
The correlations between the dependent variables were significant and ranged
from .190 to .542, and only one correlation was slightly above .500. Because the
correlations between the outcome variables were low to moderate and there were no
high correlations, the multilevel models were fit separately for each outcome variable,
and = .05 was used as the significance level. The correlation coefficients can be
viewed in Table 4-6.
114
Table 4-6. Correlations between dependent variables
Post_Science Post_TechEngi Post_Math Post_21st Post_Career
Post_Science 1
Post_TechEngi .366** 1
Post_Math .291** .270** 1
Post_21st .360** .269** .321** 1
Post_Career .478** .542** .258** .190** 1
**. Correlation is significant at the 0.01 level (2-tailed).
The correlations between the subscales of the value beliefs survey were all
significant (see Table 4-7). The correlations between intrinsic interest value and
attainment value/importance and extrinsic utility value/usefulness were .546 and .460
respectively. The correlation between attainment value/importance and extrinsic utility
value/usefulness was .784. Since there was a high correlation between the attainment
value/importance and extrinsic utility value/usefulness, the collinearity between the two
variables was assessed to decide whether they both should be kept in the model when
conducting multilevel modeling analysis.
Table 4-7. Correlations between value beliefs subscales
Interest_Teacher Importance_Teacher Usefulness_Teacher
Interest_Teacher 1
Importance_Teacher .546** 1
Usefulness_Teacher .460* .784** 1
**. Correlation is significant at the 0.01 level (2-tailed). *. Correlation is significant at the 0.05 level (2-tailed).
115
The correlations between self-efficacy beliefs subscales (see Table 4-8) were all
positive and ranged from .031 to .706. The correlations between TK and TPK, TK and
TPACK, CK and PK, PK and PCK, PK and TCK, PK and TPACK, PCK and TCK, TCK
and TPK, TCK and TPACK, and TPK and TPACK were all significant with low to
moderate correlation coefficients. The correlations between TK and TPACK, PK and
PCK, and TPK and TPACK were relatively high compared to other correlations between
the subscales. When fitting the multilevel models with the variables of the subscales,
multicollinearity was assessed to determine whether some of the variables should be
removed from the models.
Table 4-8. Correlations between self-efficacy beliefs subscales
TK CK PK PCK TCK TPK TPACK
TK 1
CK .298 1
PK .393 .454* 1
PCK .282 .326 .687** 1
TCK .351 .031 .519** .468* 1
TPK .540** .382 .366 .165 .408* 1
TPACK .617** .242 .494* .395 .478* .706** 1
**. Correlation is significant at the 0.01 level (2-tailed). *. Correlation is significant at the 0.05 level (2-tailed).
There were no significant correlations between pedagogical beliefs and any of
the teacher value beliefs or self-efficacy beliefs (see Table 4-9). Teachers’ intrinsic
interest value significantly and positively correlated with teachers’ self-efficacy beliefs in
TPK, TCK, and TPACK with correlation coefficients of .438, .487, and .687 respectively.
116
Teachers’ perceived importance of 3D printing integration significantly and positively
correlated with teachers’ self-efficacy in TPACK with r = .565. Teachers’ perceived
usefulness of 3D printing integration had no significant correlations with teachers’
pedagogical beliefs or self-efficacy beliefs.
Table 4-9. Correlations between teacher beliefs
Pedagogical_Beliefs
Interest _Teacher
Importance _Teacher
Usefulness _Teacher
Pedagogical _Beliefs 1
Interest _Teacher .049 1
Importance _Teacher -.039 .546** 1
Usefulness _Teacher -.066 .460* .784** 1
Self_Efficacy_PK .287 .338 .004 -.103
Self_Efficacy_CK -.105 -.141 -.205 -.402
Self_Efficacy_TPK -.086 .438* .330 .194
Self_Efficacy_TCK .308 .487* .115 .088
Self_Efficacy_PCK .339 .210 .020 -.152
Self_Efficacy_TPACK .013 .687** .565** .373
**. Correlation is significant at the 0.01 level (2-tailed). *. Correlation is significant at the 0.05 level (2-tailed).
After reporting the fundamental statistics, the next step is to report the results for
the two research questions:
How are teachers’ beliefs correlated with their 3D printing integration in the science classrooms?
How do teachers’ beliefs and their 3D printing integration in the science classrooms predict students’ STEM motivation, 21st century skills, and interest in STEM careers?
117
Results for RQ1
Correlations between Teacher Beliefs and 3D Printing Integration
The correlations between 3D printing integration levels, STEM integration levels,
and teachers’ pedagogical beliefs, value beliefs, and self-efficacy beliefs are presented
in Table 4-10. The correlation between 3D printing integration levels and STEM
integration levels was significant with r = .673, a moderate correlation. The correlation
between teachers’ self-efficacy beliefs in PCK was negative and significant with r = -
.457, a low-to-moderate correlation. All the other correlations were nonsignificant.
Table 4-10. Correlations between Printing_Level, STEM_Level, and teacher beliefs
Printing_Level STEM_Level
Printing_Level 1
STEM_Level .673** 1
Pedagogical_Beliefs -.011 -.085
Interest_Teacher -.259 -.107
Importance_Teacher -.274 .031
Usefulness_Teacher -.040 .292
Self_efficacy_TK .006 -.122
Self_efficacy_PK -.170 -.353
Self_efficacy_CK .136 -.033
Self_efficacy_TPK -.352 -.344
Self_efficacy_TCK -.156 -.265
Self_efficacy_PCK -.135 -.457*
Self_efficacy_TPACK -.152 -.127
**. Correlation is significant at the 0.01 level (2-tailed). *. Correlation is significant at the 0.05 level (2-tailed).
Open Responses in Teacher Beliefs Survey
Four open-ended questions were included in the teacher beliefs survey to obtain
some detailed data to provide specific information on teachers’ experience and
118
perceptions on their 3D printing integration, which may potentially explain the
relationship between teacher beliefs and their 3D printing integration. The four
questions were: 1. How do you feel about integrating 3D printing technology into your
science teaching? 2. Do you think you have sufficient knowledge and skills to use 3D
printing technology in your science classes? Please explain. 3. What are the biggest
advantages of integrating 3D printing technology in science teaching? 4. What are the
biggest challenges in integrating 3D printing technology in science teaching?
The responses of each question were coded and then the themes were
generated with the codes. Afterwards, related themes were synthesized across the
questions. The analysis of the first question yielded three themes: teachers’ and
students’ positive attitude towards 3D printing, benefits for students, and challenges of
3D printing integration. The teachers felt integrating 3D printing into their science
classrooms was great, beneficial, fun, exciting, and amazing, and they also indicated
that students loved the 3D printing integrated activities and felt amazed at the 3D
printed objects. The teachers felt 3D printing integration was beneficial for students,
including engaging students by allowing students to hold and visualize objects,
promoting hands-on learning, and enhancing students’ cognitive learning. However, the
teachers mentioned that they encountered a few challenges, including the lack of 3D
printers, technical issues, the lack of time to print 3D objects, the lack of time to
implement lessons, difficulty in using 3D printers, difficult in connecting 3D printing to
curriculum standards, and difficulty in adapting to students’ different abilities.
For the second question, most teachers thought they had sufficient knowledge
and skills. A few teachers thought they had sufficient knowledge and skills but had
119
some challenges including the lack of 3D printers, technical support, ability to print 3D
objects, time to learn how to print 3D objects, and the difficulty in connecting 3D printing
to curriculum standards and creating lessons. Moreover, several teachers thought they
did not have sufficient knowledge and skills in integrating 3D printing in their science
classrooms.
For the third question, the teachers indicated a few benefits of 3D printing
integration for teaching and learning: convenience for learning with access to objects;
enabling hands-on learning and engage students; stimulating student interest and
motivation; enhancing students’ cognitive learning including science knowledge,
scientific investigation, shifting science thinking and learning, understanding of science
learning content, and creativity; and making connections with other disciplines such as
science and math.
The last question provided information on the challenges that teachers
encountered when integrating 3D printing into their science classrooms, including
logistical and technical issues, insufficient 3D printers and related resources, and the
lack of time, including the time to print 3D objects, the time to plan, develop, and
integrate 3D printing into curriculum, and the time to teach students how to use 3D
printing. According to Ertmer et al. (2012), these challenges were external barriers for
teachers. Additionally, the teachers had some internal barriers, including the lack of
ability to print 3D objects and connect 3D printing to curriculum standards, difficulty in
teaching and making sure all students were able to use the 3D printing software, and
also difficulty in teaching students who were not enthusiastic, motivated, or having
limited ability.
120
After analyzing the themes of the teacher responses for each question, it was
found that there were some common themes across the questions. The responses of
question 1 and question 3 both had themes regarding the benefits of 3D printing
integration in science classrooms. The responses of question 1, 2, and 4 all had themes
regarding the challenges of 3D printing integration in science classrooms. The common
themes were synthesized across the questions and finally there were five themes with
some sub-themes for all the teacher responses: teachers’ and students’ attitude
towards 3D printing integration; teachers’ self-efficacy in their knowledge and skills in
3D printing integration in science classrooms; teachers’ value beliefs in 3D printing
integration, i.e., 3D printing integration was beneficial for teaching and learning;
teachers’ external barriers to integrating 3D printing technology in their science
classrooms; and teachers’ internal barriers to integrating 3D printing technology in their
science classrooms. All the sub-themes have been illustrated earlier on. The themes,
sub-themes, and the frequency of the sub-themes can be viewed in Table 4-11.
Table 4-11. Thematic analysis results of teachers’ open responses
Themes Sub-themes Frequency
Teachers’ and students’ attitude towards 3D printing integration
Students’ positive attitude perceived by teachers: Teachers felt that students loved 3D printing and were amazed at the 3D printed objects.
2
Teachers’ positive attitude: Teachers felt 3D printing integration in the science classrooms were great, beneficial, fun, exciting, and amazing.
14
Teachers’ self-efficacy in their knowledge and skills in 3D printing integration in science classrooms
High self-efficacy: Teachers thought they had sufficient knowledge and skills.
13
Moderate self-efficacy: Teachers thought they had some knowledge and skills but had some challenges.
7
Low self-efficacy: Teachers thought they did not have sufficient knowledge and skills.
4
121
Table 4-11. Continued
Themes Sub-themes Frequency
Teachers’ value beliefs in 3D printing integration: 3D printing integration was beneficial for teaching and learning
Convenience for learning with access to objects.
15
Enable hands-on learning and engage students by allowing students to hold and visualize the 3D printed objects.
25
Stimulate student interest and motivation. 4
Enhance students’ cognitive learning including science knowledge, scientific investigation, shifting science thinking and learning, understanding of science learning content, and creativity.
11
Make connections with other disciplines such as science and math.
2
Teachers’ external barriers to integrating 3D printing technology
Logistical and technical issues. 8
Insufficient 3D printers and related resources. 10
Lack of time to print 3D objects. 9
Lack of time to plan, develop, and integrate 3D printing into curriculum.
9
Teachers’ internal barriers to integrating 3D printing technology
Lack of ability to print 3D objects and connect 3D printing to curriculum standards.
10
Difficulty in teaching students who were not enthusiastic, motivated, or having limited ability.
3
Results for RQ2
Missing Data Evaluation
In this study, there was a very small portion of missing data for student-level
variables and teacher-level variables. The proportions of missing data of student-level
variables were all less than 2 percent (see Table 4-12). There were 26 teachers in total
and 24 teachers completed the survey for their pedagogical beliefs, value beliefs, and
self-efficacy beliefs. The proportion of missing data of all the teacher-level variables was
7.6923%. The missing data were assumed as missing completely at random (MCAR)
122
since the proportions of missing data were small and the probabilities of missing data of
the variables were unrelated to the observed variables and the variable with missing
data per se. Therefore, full information maximum likelihood (FIML) was used to deal
with the missing data by using all the available data. The software program SAS 9.4
automatically uses FIML to deal with missing data.
Table 4-12. Proportion of missing values for student-level variables
Variable Total N of Students
N of Responses N of Missingness
Proportion of Missingness
Gender_Student 1501 1501 0 0
Pre _Science 1501 1497 4 0.2664%
Pre _TechEngi 1501 1490 11 0.7328%
Pre_Math 1501 1501 0 0
Pre _21st 1501 1482 19 1.2658%
Pre_Career 1501 1475 26 1.7322%
Post_Science 1501 1492 9 0.5996%
Post_TechEngi 1501 1485 16 1.0660%
Post_Math 1501 1501 0 0
Post_21st 1501 1482 19 1.2658%
Post_Career 1501 1476 25 1.6656%
Assumptions Testing
There are several assumptions for conducting multilevel modeling analysis. First,
the dependent variable has to be normally distributed. Second, there is no
multicollinearity between the independent variables. Third, the residuals of the models
have to be normally distributed. In this study, the assumptions of the multilevel models
were checked along with the analysis process. The normality of the dependent variable
123
was examined before the models were built. All the dependent variables in this study
including science motivation posttest scores, technology/engineering posttest scores,
math motivation posttest scores, 21st century skills, and interest in STEM careers met
with the normality assumption. The skewness of each outcome variable was well
between -1 and 1, and the kurtosis of each outcome variable was well between -3 and
3. The skewness and kurtosis statistics can be viewed in Table 4-13. The
multicollinearity was examined when teacher-level variables were added into the model
for each dependent variable. The residual normality was examined after the best model
was identified for each dependent variable. The results of the multicollinearity and
residual normality assumptions testing will be reported later.
Table 4-13. Skewness and kurtosis of dependent variables
Variables Skewness Kurtosis
POST_Science -0.1149 0.1147
POST_TechEngi -0.2659 0.2747
POST_Math -0.4892 -0.2634
POST_21st -0.5046 1.0369
POST_Career -0.3245 0.7069
Results for Science Motivation
Multilevel modeling analyses were conducted to examine how teachers’ 3D
printing integration levels, STEM integration levels, pedagogical beliefs, value beliefs,
and self-efficacy beliefs in 3D printing integration predicted students’ STEM motivation,
21st century skills, and interest in STEM careers respectively while controlling for
student gender and students’ pretest scores.
124
The total number of students was 1,501. The student sample size for all the 26
teachers was unbalanced and varied from 20 to 103, with an average of 𝑛𝑐 = 57.7308,
SD = 25.6976, Median = 54. Because the student sample size for each teacher was
unbalanced, and the teacher-level sample size was small (N = 26), Kenward-Roger
adjustment (abbreviated as kr) (Kenward & Roger, 2009) was used in the SAS program
to adjust for degrees of freedom.
A series of multilevel models were built to examine how teachers’ 3D printing
integration levels, STEM integration levels, pedagogical beliefs, value beliefs, and self-
efficacy beliefs in 3D printing integration predict students’ science motivation while
controlling for student gender and science motivation pretest scores. Student gender
and science motivation pretest scores were included in the model as covariates to be
controlled while interpreting the intercept and the coefficients of other variables.
Baseline model
The first step was to build the baseline model, which predicted a student’ science
motivation posttest score from the grand mean science motivation posttest score of all
the teachers’ students. There were no student-level or teacher-level predictors in this
model. The purpose of this model was to examine the ICC and design effect to
determine whether multilevel models were necessary for the analysis. The meanings of
symbols in the model are provided in Table 4-14.
The baseline model is shown by the following equations:
Student Level: 𝑌𝑖𝑗 = 𝛽0𝑗 + 𝑖𝑗 (4-1)
Teacher Level: 𝛽0𝑗 = 𝛾00 + 𝑢0𝑗 (4-2)
Combined: 𝑌𝑖𝑗 = 𝛾00 + 𝑢0𝑗 + 𝑖𝑗 (4-3)
125
Table 4-14. The meaning of symbols in equations (4-1), (4-2), (4-3)
Symbol Meaning
𝑌𝑖𝑗 The science motivation posttest score of student i of teacher j.
𝛽0𝑗 The students’ mean science motivation posttest score for teacher j.
𝑖𝑗 A residual term – individual student differences around the mean of teacher j.
𝛾00 Grand-mean science motivation posttest score across all teachers.
𝑢0𝑗 Deviation/difference between the mean score for each teacher and the grand mean.
The baseline model statistics summary can be viewed in Table 4-15. The ICC =
𝜎𝑢02 / (𝜎𝑢0
2 + 𝜎𝜀2) = 0.06968 / (0.06968 + 0.4559) = 13.2577%. The variance in teacher
means accounted for 13.2577% of the total variance in science motivation posttest
scores. The Design Effect = 1 + (𝑛𝑐 - 1) ICC. The average number of students per
teacher 𝑛𝑐 = 57.7308. The Design Effect was 7.5212. Both the ICC and the Design
Effect indicated it was necessary to conduct multilevel modeling analysis for students’
science motivation posttest scores. The next step was to build the student-level models.
Table 4-15. Baseline model summary
Parameter Estimate Standard Error Z Value DF t Value p
𝜎𝑢02 0.06968 0.02297 3.03 / / .0012
𝜎𝜀2 0.4559 0.01685 27.06 / / <.0001
𝛾00 3.5923 0.05526 / 23.7 65.01 <.0001
-2 Log Likelihood𝑏𝑎𝑠𝑒𝑙𝑖𝑛𝑒= 3122.2
Student-level models
Random-intercept model. First, a random intercept model was built to estimate
the impact of student-level variables including students’ science motivation pretest
scores and gender on students’ science motivation posttest scores as fixed effects,
126
indicating the impact (coefficient) of pretest scores and gender did not vary across
teachers. As science motivation pretest was measured with a scale that did not contain
zero, a score of zero would have no substantive meaning. Students’ pretest scores
were also independent of each other’s scores. Therefore, grand mean centering was
used to enable a value of zero to be interpreted meaningfully. The equations for the
random-intercept model with grand-mean centering are as follows:
Student Level: 𝑌𝑖𝑗 = 𝛽0𝑗 + 𝛽1𝑗(𝑃𝑟𝑒_𝑆𝑐𝑖𝑒𝑛𝑐𝑒𝑖𝑗 − 𝑃𝑟𝑒_𝑆𝑐𝑖𝑒𝑛𝑐𝑒)
+ 𝛽2𝑗𝐺𝑒𝑛𝑑𝑒𝑟_𝑆𝑡𝑢𝑑𝑒𝑛𝑡𝑖𝑗 + 𝑖𝑗
(4-4)
Teacher Level: 𝛽0𝑗 = 𝛾00 + 𝑢0𝑗 (4-5)
𝛽1𝑗 = 𝛾10 (4-6)
𝛽2𝑗 = 𝛾20 (4-7)
Combined: 𝑌𝑖𝑗 = 𝛾00 + 𝛾10(𝑃𝑟𝑒_𝑆𝑐𝑖𝑒𝑛𝑐𝑒𝑖𝑗 − 𝑃𝑟𝑒_𝑆𝑐𝑖𝑒𝑛𝑐𝑒)
+ 𝛾20𝐺𝑒𝑛𝑑𝑒𝑟_𝑆𝑡𝑢𝑑𝑒𝑛𝑡𝑖𝑗 + 𝑢0𝑗 + 𝑖𝑗
(4-8)
Besides with the symbols that have been explained previously, 𝛽1𝑗 is the
regression coefficient that shows the impact of science motivation pretest scores on
posttest scores across all students of teacher j , 𝑃𝑟𝑒_𝑆𝑐𝑖𝑒𝑛𝑐𝑒𝑖𝑗 is science motivation
pretest score of student i of teacher j, 𝑃𝑟𝑒_𝑆𝑐𝑖𝑒𝑛𝑐𝑒 is the grand mean of science
motivation pretest scores across teachers, 𝛾10 is the average effect (coefficient) of
pretest scores on posttest scores across all teachers, and 𝛾20 is the average effect
(coefficient) of student gender on science motivation posttest scores across all
teachers. In the model, student gender was treated as a categorical variable and male =
0 was the reference. The random-intercept model statistics summary can be viewed in
Table 4-16.
To determine whether the random-intercept model fit better than the baseline
model, a likelihood ratio (LR) test was used to evaluate the difference between the log
127
likelihood values for the nested models, i.e., the baseline model and the random-
intercept model.
LR = -2LogLikelihoodbaseline
Likelihoodfull
= (-2Log Likelihoodbaseline) - (-2Log Likelihoodfull)
In the baseline model, -2Log Likelihoodbaseline = 3122.2.
In the student-level random-intercept model, -2Log Likelihood𝑓𝑢𝑙𝑙1 = 2102.1.
Therefore, LR = 3122.2 - 2102.1= 1020.1. LR follows a 2 distribution with df = 2.
The df = 2 because the degree of freedom of the baseline model and the random-
intercept model differed by 2, which was the difference between the number of
parameters in the two models. The p value of LR was calculated with the CHIDIST (2
value, df) function in SPSS. The likelihood ratio test was significant with 2 (2) = 1020.1,
p < .0001, indicating the random-intercept model was significantly better than the
baseline model. At least one of the student-level variables, i.e. science motivation
pretest score and student gender, can significantly predict teacher mean science
motivation posttest score.
Table 4-16. Random-intercept model summary
Parameter Estimate Standard Error Z Value DF t Value p
𝜎𝑢02 0.007481 0.003474 2.15 / / .0156
𝜎𝜀2 0.2340 0.008663 27.02 / / <.0001
𝛾00 3.5672 0.02545 / 43.6 139.89 <.0001
𝛾10 0.7372 0.01897 / 1395 38.87 <.0001
𝛾20 0.009815 0.02531 / 1479 0.39 .6982
-2 Log Likelihoodfull1 = 2102.1
128
Random-slope model. As the random-intercept model fit significantly better than
the baseline model, a random-slope model was built to evaluate whether the impact of
students’ science motivation pretest scores and student gender on the posttest scores
varied significantly across teachers. Variance components (i.e., random effects) of
pretest score and student gender were added to the teacher-level slope equation to
model the variation. The equations for the teacher-level model are as follows:
Teacher Level: 𝛽0𝑗 = 𝛾00 + 𝑢0𝑗 (4-9)
𝛽1𝑗 = 𝛾10 + 𝑢1𝑗 (4-10)
𝛽2𝑗 = 𝛾20 + 𝑢2𝑗 (4-11)
After combining with the student-level model (4-4), the combined model is:
Combined: 𝑌𝑖𝑗 = 𝛾00 + 𝛾10(𝑃𝑟𝑒_𝑆𝑐𝑖𝑒𝑛𝑐𝑒𝑖𝑗 − 𝑃𝑟𝑒_𝑆𝑐𝑖𝑒𝑛𝑐𝑒)
+ 𝛾20𝐺𝑒𝑛𝑑𝑒𝑟_𝑆𝑡𝑢𝑑𝑒𝑛𝑡𝑖𝑗
+ 𝑢1𝑗 (𝑃𝑟𝑒_𝑆𝑐𝑖𝑒𝑛𝑐𝑒𝑖𝑗 − 𝑃𝑟𝑒_𝑆𝑐𝑖𝑒𝑛𝑐𝑒)
+ 𝑢2𝑗𝐺𝑒𝑛𝑑𝑒𝑟_𝑆𝑡𝑢𝑑𝑒𝑛𝑡𝑖𝑗 + 𝑢0𝑗 + 𝑖𝑗
(4-12)
In the random-slope model, 𝑢1𝑗 and 𝑢2𝑗 are residual terms to show random
effects, indicating the impact of pretest score and student gender respectively on
posttest score can vary randomly across teachers.
However, the SAS Log noted, “Convergence criteria met but final Hessian is not
positive definite”, “Estimated G matrix is not positive definite”, and “Asymptotic variance
matrix of covariance parameter estimates has been found to be singular and a
generalized inverse was used”, indicating the random-slope model did not fit well after
adding the random effects. After removing Pre_Science from the random effects, SAS
Log provided the same notes. After removing Gender_Student from the random effects
while keeping Pre_Science in the random effects, the notes disappeared, and the model
fit well. Therefore, the final random-slope model just contained the intercept and
129
PRE_Science in the random effects. The random term 𝑢2𝑗 was removed from Formula
𝛽2𝑗 = 𝛾20 + 𝑢2𝑗 (4-11) and the final combined model for the random-slope model is:
Combined: 𝑌𝑖𝑗 = 𝛾00 + 𝛾10(𝑃𝑟𝑒_𝑆𝑐𝑖𝑒𝑛𝑐𝑒𝑖𝑗 − 𝑃𝑟𝑒_𝑆𝑐𝑖𝑒𝑛𝑐𝑒)
+ 𝛾20𝐺𝑒𝑛𝑑𝑒𝑟_𝑆𝑡𝑢𝑑𝑒𝑛𝑡𝑖𝑗
+ 𝑢1𝑗 (𝑃𝑟𝑒_𝑆𝑐𝑖𝑒𝑛𝑐𝑒𝑖𝑗 − 𝑃𝑟𝑒_𝑆𝑐𝑖𝑒𝑛𝑐𝑒) + 𝑢0𝑗 + 𝑖𝑗
(4-13)
The random-slope model statistics summary can be viewed in Table 4-17. For
the random-slope model, -2 Log Likelihoodfull2= 2096.2.
LR = (-2Log Likelihoodfull1) - (-2Log Likelihoodfull2) = 2102.1 - 2096.2 = 5.9
The likelihood ratio test was nonsignificant with 2 (2) = 5.9, p = .0523, indicating
the random-slope model was not significantly better than the random-intercept model.
Therefore, it was not necessary to build the random-slope model. The next step was to
add teacher-level variables to the random-intercept model.
Table 4-17. Random-slope model summary
Parameter Estimate Standard Error Z Value DF t Value p
𝜎𝑢02 0.006467 0.003194 2.02 / / .0214
𝜎𝑢0𝑢1 0.000998 0.002535 0.39 / / .6938
𝜎𝑢12 0.006525 0.004246 1.54 / / .0622
𝜎𝜀2 0.2314 0.008611 26.87 / / <.0001
𝛾00 3.5649 0.02493 / 45.3 142.98 <.0001
𝛾10 0.7391 0.02544 / 25.9 29.05 <.0001
𝛾20 0.009479 0.02529 / 1477 0.37 .7079
-2 Log Likelihoodfull2 = 2096.2
Adding teacher-level variables
Before building the model with teacher variables, multicollinearity was evaluated
to determine whether all the teacher-variables could be included in the model. As shown
130
in Table 4-18, all the teacher-level variables had a Tolerance of higher than 0.1 and a
Variance Inflation Factor (VIF) of less than 10. Therefore, all the teacher-level variables
were included in the model for further analysis.
Table 4-18. Multicollinearity of variables for science motivation posttest score
Variable Tolerance Variance Inflation Factor
Intercept . 0
Pre_Science 0.93130 1.07377
Gender_Student 0.98416 1.01609
Printing_Level 0.38082 2.62590
STEM_Level 0.18647 5.36284
Pedagogical_Beliefs 0.33510 2.98423
Interest_Teacher 0.23051 4.33827
Importance_Teacher 0.16425 6.08843
Usefulness_Teacher 0.13881 7.20423
Self_Efficacy_TK 0.23528 4.25019
Self_Efficacy_PK 0.31878 3.13695
Self_Efficacy_CK 0.28754 3.47772
Self_Efficacy_TPK 0.14508 6.89290
Self_Efficacy_TCK 0.28801 3.47212
Self_Efficacy_PCK 0.22842 4.37798
Self_Efficacy_TPACK 0.10609 9.42587
All the teacher-level variables were centered with the grand mean of the variable.
The equations for the teacher-level model are:
Teacher Level: 𝛽0𝑗 = 𝛾00 + 𝛾01(𝑃𝑟𝑖𝑛𝑡𝑖𝑛𝑔_𝐿𝑒𝑣𝑒𝑙𝑖𝑗 − 𝑃𝑟𝑖𝑛𝑡𝑖𝑛𝑔_𝐿𝑒𝑣𝑒𝑙 ) +
𝛾02(𝑆𝑇𝐸𝑀_𝐿𝑒𝑣𝑒𝑙𝑖𝑗 − 𝑆𝑇𝐸𝑀_𝐿𝑒𝑣𝑒𝑙) + 𝛾03(𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠𝑖𝑗 −
𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠) + 𝛾04(𝐼𝑛𝑡𝑒𝑟𝑒𝑠𝑡_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 −
𝐼𝑛𝑡𝑒𝑟𝑒𝑠𝑡_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛾05(𝐼𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 −
(4-14)
131
𝐼𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛾06(𝑈𝑠𝑒𝑓𝑢𝑙𝑛𝑒𝑠𝑠_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 −
𝑈𝑠𝑒𝑓𝑢𝑙𝑛𝑒𝑠𝑠_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛾07(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐾) + 𝛾08(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾) + 𝛾09(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝐶𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝐶𝐾) + 𝛾010(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐾) + 𝛾011(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐶𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐶𝐾) + 𝛾012(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐶𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐶𝐾) + 𝛾013(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐴𝐶𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐴𝐶𝐾) + 𝑢0𝑗
𝛽1𝑗 = 𝛾10 + 𝛾11(𝑃𝑟𝑖𝑛𝑡𝑖𝑛𝑔_𝐿𝑒𝑣𝑒𝑙𝑖𝑗 − 𝑃𝑟𝑖𝑛𝑡𝑖𝑛𝑔_𝐿𝑒𝑣𝑒𝑙 ) +
𝛾12(𝑆𝑇𝐸𝑀_𝐿𝑒𝑣𝑒𝑙𝑖𝑗 − 𝑆𝑇𝐸𝑀_𝐿𝑒𝑣𝑒𝑙) + 𝛾13(𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠𝑖𝑗 −
𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠) + 𝛾14(𝐼𝑛𝑡𝑒𝑟𝑒𝑠𝑡_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 −
𝐼𝑛𝑡𝑒𝑟𝑒𝑠𝑡_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛾15(𝐼𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 −
𝐼𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛾16(𝑈𝑠𝑒𝑓𝑢𝑙𝑛𝑒𝑠𝑠_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 −
𝑈𝑠𝑒𝑓𝑢𝑙𝑛𝑒𝑠𝑠_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛾17(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐾) + 𝛾18(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾)
+ 𝛾19(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝐶𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝐶𝐾) +
𝛾110(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐾) +
𝛾111(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐶𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐶𝐾) +
𝛾112(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐶𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐶𝐾) +
𝛾113(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐴𝐶𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐴𝐶𝐾)
(4-15)
𝛽2𝑗 = 𝛾20 + 𝛾21(𝑃𝑟𝑖𝑛𝑡𝑖𝑛𝑔_𝐿𝑒𝑣𝑒𝑙𝑖𝑗 − 𝑃𝑟𝑖𝑛𝑡𝑖𝑛𝑔_𝐿𝑒𝑣𝑒𝑙 ) +
𝛾22(𝑆𝑇𝐸𝑀_𝐿𝑒𝑣𝑒𝑙𝑖𝑗 − 𝑆𝑇𝐸𝑀_𝐿𝑒𝑣𝑒𝑙) + 𝛾23(𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠𝑖𝑗 −
𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠) + 𝛾24(𝐼𝑛𝑡𝑒𝑟𝑒𝑠𝑡_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 −
𝐼𝑛𝑡𝑒𝑟𝑒𝑠𝑡_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛾25(𝐼𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 −
𝐼𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛾26(𝑈𝑠𝑒𝑓𝑢𝑙𝑛𝑒𝑠𝑠_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 −
𝑈𝑠𝑒𝑓𝑢𝑙𝑛𝑒𝑠𝑠_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛾27(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐾) + 𝛾28(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾) + 𝛾29(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝐶𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝐶𝐾) + 𝛾210(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐾) + 𝛾211(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐶𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐶𝐾) + 𝛾212(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐶𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐶𝐾) + 𝛾213(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐴𝐶𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐴𝐶𝐾)
(4-16)
After evaluating the cross-level interactions between the student-level variables
and teacher-level variables, it was found the cross-level interactions
132
Pre_Science*Pedagogical_Beliefs and Gender_Student* STEM_Level were significant.
Therefore, after combining the student-level model (4-4) and teacher-level models
including the significant cross-level interaction terms, the equation for the final model is:
Combined: 𝑌𝑖𝑗 = 𝛾00 + 𝛾10(𝑃𝑟𝑒_𝑆𝑐𝑖𝑒𝑛𝑐𝑒𝑖𝑗 − 𝑃𝑟𝑒_𝑆𝑐𝑖𝑒𝑛𝑐𝑒) + 𝛾20𝐺𝑒𝑛𝑑𝑒𝑟_𝑆𝑡𝑢𝑑𝑒𝑛𝑡𝑖𝑗
+ 𝛾01(𝑃𝑟𝑖𝑛𝑡𝑖𝑛𝑔_𝐿𝑒𝑣𝑒𝑙𝑖𝑗 − 𝑃𝑟𝑖𝑛𝑡𝑖𝑛𝑔_𝐿𝑒𝑣𝑒𝑙 ) + 𝛾02(𝑆𝑇𝐸𝑀_𝐿𝑒𝑣𝑒𝑙𝑖𝑗 −
𝑆𝑇𝐸𝑀_𝐿𝑒𝑣𝑒𝑙) + 𝛾03(𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠𝑖𝑗 − 𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠) +
𝛾04(𝐼𝑛𝑡𝑒𝑟𝑒𝑠𝑡_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 − 𝐼𝑛𝑡𝑒𝑟𝑒𝑠𝑡_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) +
𝛾05(𝐼𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 − 𝐼𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) +
𝛾06(𝑈𝑠𝑒𝑓𝑢𝑙𝑛𝑒𝑠𝑠_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 − 𝑈𝑠𝑒𝑓𝑢𝑙𝑛𝑒𝑠𝑠_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) +
𝛾07(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐾) +
𝛾08(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾) +
𝛾09(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝐶𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝐶𝐾) +
𝛾010(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑐𝑦_𝑃𝐾) +
𝛾011(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐶𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐶𝐾) +
𝛾012(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐶𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐶𝐾) +
𝛾013(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐴𝐶𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐴𝐶𝐾) +
𝛾13(𝑃𝑟𝑒_𝑆𝑐𝑖𝑒𝑛𝑐𝑒𝑖𝑗 − 𝑃𝑟𝑒_𝑆𝑐𝑖𝑒𝑛𝑐𝑒) ∗ (𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠𝑖𝑗 −
𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠) + 𝛾22𝐺𝑒𝑛𝑑𝑒𝑟_𝑆𝑡𝑢𝑑𝑒𝑛𝑡𝑖𝑗 ∗ (𝑆𝑇𝐸𝑀_𝐿𝑒𝑣𝑒𝑙𝑖𝑗 −
𝑆𝑇𝐸𝑀_𝐿𝑒𝑣𝑒𝑙) + 𝑢0𝑗 + 𝑖𝑗
(4-17)
For this model with teacher-level variables, -2 Log Likelihoodfull3 = 1921.2.
LR = -2 Log Likelihoodfull1 - 2 Log Likelihoodfull3 = 2102.1 – 1921.2 = 180.9.
The likelihood ratio test was significant with 2 (15) = 180.9, p < .0001. The
model with teacher-level variables was significantly better than the random-intercept
model. Therefore, the random-intercept model with teacher variables was the best
model.
The residual normality for student-level and teacher-level residuals were
evaluated. The student-level residual had a skewness of -0.4500 and a kurtosis of
2.3855. The teacher-level residual was just the intercept variance. The skewness of
teacher-level residual was 0.8258 and the kurtosis of it was 0.1792. The skewness and
133
kurtosis for the residuals were between -1 and 1, and -3 and 3 respectively. Therefore,
the residual normality assumptions were met.
The statistics summary for the model with teacher variables can be viewed in
Table 4-19. After grand mean centering, the average intercept 𝛾00 = 3.5802 and it was
statistically significant with t (14.1) = 126.82, p < .0001, indicating on average the
science motivation posttest score was 3.5802 when student gender was male
(Gender_Student = 0) and all other variables were equal to their grand mean, and the
average intercept was significantly different from zero. After accounting for all the
student-level and teacher-level variables, the teacher-level residual variance in the
teacher means, i.e., the intercept variance 𝜎𝑢02 = 0.008594 and it approached
significance with Z = 1.35, p = .0890, indicating the teacher means of student science
motivation posttest scores varied across teachers with approaching significance. On
average, the teacher means of student science motivation posttest scores varied from
the grand mean by √0.008594 = 0.0927. The student-level residual variance 𝜎𝜀2 =
0.2365 and it was statistically significant with Z = 25.38, p < .0001. Student science
motivation posttest scores significantly varied within each teacher. On average,
individual student’s science motivation posttest score varied from their teacher mean by
√0.2365 = 0.4863.
As the best fit model was a random-intercept model with teacher variables, each
teacher had a different intercept but the same slope for each variable. The slope of
science motivation pretest scores 𝛾10 = 0.7107, statistically significant with t (1235) =
34.33, p < .0001. The slope 𝛾10 indicated on average one score increase in science
motivation pretest score increased science motivation posttest score by 0.7107 when
134
controlling for other variables. After controlling for student variables, none of the teacher
variables were significant predictors of student science motivation posttest scores. The
slope of teachers’ perceived usefulness of 3D printing integration 𝛾06 = -0.2507 and it
approached significance with t (9.88) = -2.02, p = .0708. The slope 𝛾06 indicated one
score increase in teachers’ perceived usefulness of 3D printing integration decreased
student science motivation posttest score by 0.2507 with approaching significance when
controlling for other variables.
The interaction between student science motivation pretest score and teachers’
pedagogical beliefs was significant with slope 𝛾13 = -0.1291, t (1296) = -2.97, p = .0030.
One score increase in student science motivation pretest score decreased the slope of
teachers’ pedagogical beliefs by 0.1291. The interaction between student gender and
teachers’ STEM integration level was significant with slope 𝛾22 = 0.09404, t (1299) =
2.48, p = .0133. As male students were coded as 0 (reference level) and female
students were coded as 1, the slope of teachers’ STEM integration level for female
students (one level increase in student gender) was 0.09404 higher than the slope for
male students.
Table 4-19. Random-intercept model with teacher variables model summary
Parameter Estimate Standard Error
Z Value
DF t Value
p
Teacher-level intercept variance (𝜎𝑢02 ) 0.008594 0.006379 1.35 / / .0890
Student-level residual variance (𝜎𝜀2) 0.2365 0.009320 25.38 / / <.0001
Intercept (𝛾00) 3.5802 0.02823 / 14.1 126.82
<.0001
Pre_Science (𝛾10) 0.7107 0.02070 / 1235 34.33 <.0001
Gender_Student (𝛾20) -0.00002 0.02719 / 1296 -0.00 .9993
Printing_Level (𝛾01) -0.00819 0.03618 / 8.63 -0.23 .8262
135
Table 4-19. Continued
Parameter Estimate Standard Error
Z Value
DF t Value
p
STEM_Level (𝛾02) 0.01457 0.07678 / 9.77 0.19 .8533
Pedagogical_Beliefs (𝛾03) 0.1076 0.08125 / 9.98 1.32 .2148
Interest_Teacher (𝛾04) -0.07845 0.08367 / 9.97 -0.94 .3706
Importance_Teacher (𝛾05) 0.1069 0.09401 / 10 1.14 .2822
Usefulness_Teacher (𝛾06) -0.2507 0.1238 / 9.88 -2.02 .0708
Self_Efficacy_TK (𝛾07) 0.02327 0.06138 / 11.1 0.38 .7117
Self_Efficacy_PK (𝛾08) 0.1317 0.09701 / 7.94 1.36 .2118
Self_Efficacy_CK (𝛾09) -0.06820 0.08284 / 8.78 -0.82 .4322
Self_Efficacy_TPK (𝛾010) 0.07921 0.1215 / 9 0.65 .5308
Self_Efficacy_TCK (𝛾011) -0.1127 0.07729 / 10.6 -1.46 .1736
Self_Efficacy_PCK (𝛾012) -0.06159 0.07818 / 9.53 -0.79 .4500
Self_Efficacy_TPACK (𝛾013) 0.05178 0.1450 / 9.13 0.36 .7290
Pre_Science*Pedagogical_Beliefs (𝛾13)
-0.1291 0.04343 / 1296 -2.97 .0030
Gender_Student*STEM_Level (𝛾22) 0.09404 0.03793 / 1299 2.48 .0133
-2 Log Likelihoodfull3 = 1921.2
Effect size calculation
The effect size was calculated by assessing the R2 at the student level and R2 at
the teacher level. R2 at the student level measured how student-level variance was
explained by the final model compared to the baseline model. R2 at the teacher level
measured how teacher-level variance was explained by the final model compared to the
student-level random-intercept model. The final model was the random-intercept model
with teacher variables. The student-level variance of the baseline model and final model
and the teacher-level variance of the student-level random-intercept model and the final
model for effect size calculation are presented in Table 4-20.
136
Table 4-20. Statistics for effect size calculation
Parameter Estimate
baseline model 𝜎𝜀2 0.4559
student-level random-intercept model 𝜎𝑢02 0.007481
final model 𝜎𝜀2 0.2365
final model 𝜎𝑢02 0.008594
R2 at the student level = (baseline model 𝜎𝜀2 - final model 𝜎𝜀
2 ) / baseline model 𝜎𝜀2
= (0.4559 – 0.2365) / 0.4559 = 48.1246 %.
R2 at the teacher level = (student-level random-intercept model 𝜎𝑢02 - final model
𝜎𝑢02 ) / student-level random-intercept model 𝜎𝑢0
2 = (0.007481 - 0.008594) / 0.007481 = -
14.8777%.
The final model (random-intercept model with teacher variables) explained about
48.1246 % of the student-level variance but the teacher-level variables did not
contribute to explaining teacher-level variance.
Results for Technology/Engineering Motivation
A series of multilevel models were built to examine how student gender,
teachers’ 3D printing integration levels, STEM integration levels, pedagogical beliefs,
value beliefs, and self-efficacy beliefs in 3D printing integration predict students’
technology/engineering motivation while controlling for student gender and
technology/engineering motivation pretest scores. Student gender and
technology/engineering motivation pretest scores were included in the model as
covariates to be controlled while interpreting the intercept and the coefficients of other
variables.
137
Baseline model
The first step was to build the baseline model, which predicted a student’
technology/engineering motivation posttest score from the grand mean
technology/engineering motivation posttest score of all the teachers’ students. There
were no student-level or teacher-level predictors in this model. The purpose of this
model was to examine the ICC and design effect to determine whether multilevel
models were necessary for the analysis. The meanings of symbols in the model are
provided in Table 4-21.
The baseline model is shown by the following equations:
Student Level: 𝑌𝑖𝑗 = 𝛽0𝑗 + 𝑖𝑗 (4-18)
Teacher Level: 𝛽0𝑗 = 𝛾00 + 𝑢0𝑗 (4-19)
Combined: 𝑌𝑖𝑗 = 𝛾00 + 𝑢0𝑗 + 𝑖𝑗 (4-20)
Table 4-21. The meaning of symbols in equations (4-18), (4-19), (4-20)
Symbol Meaning
𝑌𝑖𝑗 The technology/engineering motivation posttest score of student i of teacher j.
𝛽0𝑗 The students’ mean technology/engineering motivation posttest score for teacher j.
𝑖𝑗 A residual term – individual student differences around the mean of teacher j.
𝛾00 Grand-mean technology/engineering motivation posttest score across all teachers.
𝑢0𝑗 Deviation/difference between the mean score for each teacher and the grand mean.
The baseline model statistics summary can be viewed in Table 4-22. The ICC =
𝜎𝑢02 / (𝜎𝑢0
2 + 𝜎𝜀2) = 0.01636 / (0.01636 + 0.5291) = 2.9993%. The variance in teacher
means accounted for 2.9993% of the total variance in technology/engineering
motivation posttest scores. The Design Effect = 1 + (𝑛𝑐 - 1) ICC. The average number of
138
students per teacher 𝑛𝑐 = 57.7308. The Design Effect was 2.7015. The ICC was
smaller than 0.5 but the Design Effect was larger than 2, so multilevel modeling analysis
was used. The next step was to build the student-level models.
Table 4-22. Baseline model summary
Parameter Estimate Standard Error
Z Value
DF t Value p
Teacher-level intercept variance (𝜎𝑢02 ) 0.01636 0.008364 1.96 / / 0.0253
Student-level residual variance (𝜎𝜀2) 0.5291 0.01962 26.96 / / <.0001
Intercept (𝛾00) 3.4843 0.03226 / 19.4 108.00 <.0001
-2 Log Likelihood𝑏𝑎𝑠𝑒𝑙𝑖𝑛𝑒= 3298.4
Student-level models
Random-intercept model. First, a random intercept model was built to estimate
the impact of student-level variables including students’ technology/engineering
motivation pretest scores and gender on students’ technology/engineering motivation
posttest scores as fixed effects, indicating the impact (coefficient) of pretest scores and
gender did not vary across teachers. As technology/engineering motivation pretest was
measured with a scale that did not contain zero, a score of zero would have no
substantive meaning. Students’ pretest scores were also independent of each other’s
scores. Therefore, grand mean centering was used to enable a value of zero to be
interpreted meaningfully. The equations for the random-intercept model with grand-
mean centering are as follows:
Student Level: 𝑌𝑖𝑗 = 𝛽0𝑗 + 𝛽1𝑗(𝑃𝑟𝑒_𝑇𝑒𝑐ℎ𝐸𝑛𝑔𝑖𝑖𝑗 − 𝑃𝑟𝑒_𝑇𝑒𝑐ℎ𝐸𝑛𝑔𝑖)
+ 𝛽2𝑗𝐺𝑒𝑛𝑑𝑒𝑟_𝑆𝑡𝑢𝑑𝑒𝑛𝑡𝑖𝑗 + 𝑖𝑗
(4-21)
Teacher Level: 𝛽0𝑗 = 𝛾00 + 𝑢0𝑗 (4-22)
𝛽1𝑗 = 𝛾10 (4-23)
𝛽2𝑗 = 𝛾20 (4-24)
139
Combined: 𝑌𝑖𝑗 = 𝛾00 + 𝛾10(𝑃𝑟𝑒_𝑇𝑒𝑐ℎ𝐸𝑛𝑔𝑖𝑖𝑗 − 𝑃𝑟𝑒_𝑇𝑒𝑐ℎ𝐸𝑛𝑔𝑖)
+ 𝛾20𝐺𝑒𝑛𝑑𝑒𝑟_𝑆𝑡𝑢𝑑𝑒𝑛𝑡𝑖𝑗 + 𝑢0𝑗 + 𝑖𝑗
(4-25)
Besides with the symbols that have been explained previously, 𝛽1𝑗 is the
regression coefficient that shows the impact of technology/engineering motivation
pretest scores on posttest scores across all students of teacher j , 𝑃𝑟𝑒_𝑇𝑒𝑐ℎ𝐸𝑛𝑔𝑖𝑖𝑗 is
technology/engineering motivation pretest score of student i of teacher j, 𝑃𝑟𝑒_𝑇𝑒𝑐ℎ𝐸𝑛𝑔𝑖
is the grand mean of technology/engineering motivation pretest scores across teachers,
𝛾10 is the average effect (coefficient) of pretest scores on posttest scores across all
teachers, and 𝛾20 is the average effect (coefficient) of student gender on
technology/engineering motivation posttest scores across all teachers. In the model,
student gender was treated as a categorical variable and male = 0 was the reference.
The random-intercept model statistics summary can be viewed in Table 4-23.
To determine whether the random-intercept model fit better than the baseline
model, a likelihood ratio (LR) test was used to evaluate the difference between the log
likelihood values for the nested models, i.e., the baseline model and the random-
intercept model.
LR = -2LogLikelihoodbaseline
Likelihoodfull
= (-2Log Likelihoodbaseline) - (-2Log Likelihoodfull)
In the baseline model, -2Log Likelihoodbaseline = 3298.4.
In the student-level random-intercept model, -2Log Likelihood𝑓𝑢𝑙𝑙1 =2180.9.
Therefore, LR = 3298.4- 2180.9 = 1117.5. LR follows a 2 distribution with df = 2.
The df = 2 because the degree of freedom of the baseline model and the random-
intercept model differed by 2, which was the difference between the number of
140
parameters in the two models. The p value of LR was calculated with the CHIDIST (2
value, df) function in SPSS. The likelihood ratio test was significant with 2 (2) = 1117.5,
p < .0001, indicating the random-intercept model was significantly better than the
baseline model. At least one of the student-level variables, i.e. technology/engineering
motivation pretest score and student gender, can significantly predict teacher mean
technology/engineering motivation posttest score.
Table 4-23. Random-intercept model summary
Parameter Estimate Standard Error Z Value DF t Value p
𝜎𝑢02 0.001456 0.001784 0.82 / / .2073
𝜎𝜀2 0.2525 0.009384 26.91 / / <.0001
𝛾00 3.5493 0.02123 / 64.6 167.22 <.0001
𝛾10 0.7507 0.01988 / 1427 37.76 <.0001
𝛾20 -0.1060 0.02731 / 1474 -3.88 .0001
-2 Log Likelihoodfull1 = 2180.9
Random-slope model. As the random-intercept model fit significantly better than
the baseline model, a random-slope model was built to evaluate whether the impact of
students’ technology/engineering motivation pretest scores and student gender on the
posttest scores varied significantly across teachers. Variance components (i.e., random
effects) of pretest score and student gender were added to the teacher-level slope
equation to model the variation. The equations for the teacher-level model are as
follows:
Teacher Level: 𝛽0𝑗 = 𝛾00 + 𝑢0𝑗 (4-26)
𝛽1𝑗 = 𝛾10 + 𝑢1𝑗 (4-27)
𝛽2𝑗 = 𝛾20 + 𝑢2𝑗 (4-28)
After combining with the student-level model (4-21), the combined model is:
141
Combined: 𝑌𝑖𝑗 = 𝛾00 + 𝛾10(𝑃𝑟𝑒_𝑇𝑒𝑐ℎ𝐸𝑛𝑔𝑖𝑖𝑗 − 𝑃𝑟𝑒_𝑇𝑒𝑐ℎ𝐸𝑛𝑔𝑖)
+ 𝛾20𝐺𝑒𝑛𝑑𝑒𝑟_𝑆𝑡𝑢𝑑𝑒𝑛𝑡𝑖𝑗
+ 𝑢1𝑗 (𝑃𝑟𝑒_𝑇𝑒𝑐ℎ𝐸𝑛𝑔𝑖𝑖𝑗 − 𝑃𝑟𝑒_𝑇𝑒𝑐ℎ𝐸𝑛𝑔𝑖)
+ 𝑢2𝑗𝐺𝑒𝑛𝑑𝑒𝑟_𝑆𝑡𝑢𝑑𝑒𝑛𝑡𝑖𝑗 + 𝑢0𝑗 + 𝑖𝑗
(4-29)
In the random-slope model, 𝑢1𝑗 and 𝑢2𝑗 are residual terms to show random
effects, indicating the impact of pretest score and student gender respectively on
posttest score can vary randomly across teachers.
However, the SAS Log noted, “Convergence criteria met but final Hessian is not
positive definite”, “Estimated G matrix is not positive definite”, and “Asymptotic variance
matrix of covariance parameter estimates has been found to be singular and a
generalized inverse was used”, indicating the random-slope model did not fit well after
adding the random effects. When deleting Pre_TechEngi from the random effects, the
SAS Log showed the same warning. When deleting Gender_Student from the random
effects, SAS Log indicated “Estimated G matrix is not positive definite”. These warnings
indicated the model did not fit well with either Pre_TechEngi or Gender_Student or both
of them in the random effects. Therefore, both Gender_Student and Pre_TechEngi were
deleted from the random effects and only the random-intercept model fit well.
Continuing with the random-intercept model, teacher-level variables were added to the
model for examining the influence of the teacher-level variables on student
technology/engineering motivation posttest scores.
Adding teacher-level variables
Before building the model with teacher variables, multicollinearity was evaluated
to determine whether all the teacher-variables could be included in the model. As shown
in Table 4-24, all the teacher-level variables had a Tolerance of higher than 0.1 and a
142
Variance Inflation Factor (VIF) of less than 10. Therefore, all the teacher-level variables
were included in the model for further analysis.
Table 4-24. Multicollinearity of variables for technology/engineering motivation posttest score
Variable Tolerance Variance Inflation Factor
Intercept . 0
Pre_TechEngi 0.90113 1.10972
Gender_Student 0.92117 1.08558
Printing_Level 0.38212 2.61699
STEM_Level 0.18995 5.26460
Pedagogical_Beliefs 0.33464 2.98826
Interest_Teacher 0.23076 4.33358
Importance_Teacher 0.16513 6.05599
Usefulness_Teacher 0.14148 7.06807
Self_Efficacy_TK 0.23357 4.28135
Self_Efficacy_PK 0.32277 3.09817
Self_Efficacy_CK 0.28849 3.46637
Self_Efficacy_TPK 0.14646 6.82764
Self_Efficacy_TCK 0.29099 3.43649
Self_Efficacy_PCK 0.22938 4.35967
Self_Efficacy_TPACK 0.10684 9.35970
All the teacher-level variables were centered with the grand mean of the variable.
The equations for the teacher-level model are:
Teacher Level: 𝛽0𝑗 = 𝛾00 + 𝛾01(𝑃𝑟𝑖𝑛𝑡𝑖𝑛𝑔_𝐿𝑒𝑣𝑒𝑙𝑖𝑗 − 𝑃𝑟𝑖𝑛𝑡𝑖𝑛𝑔_𝐿𝑒𝑣𝑒𝑙 ) +
𝛾02(𝑆𝑇𝐸𝑀_𝐿𝑒𝑣𝑒𝑙𝑖𝑗 − 𝑆𝑇𝐸𝑀_𝐿𝑒𝑣𝑒𝑙) + 𝛾03(𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠𝑖𝑗 −
𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠) + 𝛾04(𝐼𝑛𝑡𝑒𝑟𝑒𝑠𝑡_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 −
𝐼𝑛𝑡𝑒𝑟𝑒𝑠𝑡_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛾05(𝐼𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 −
𝐼𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛾06(𝑈𝑠𝑒𝑓𝑢𝑙𝑛𝑒𝑠𝑠_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 −
(4-30)
143
𝑈𝑠𝑒𝑓𝑢𝑙𝑛𝑒𝑠𝑠_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛾07(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐾) + 𝛾08(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾) + 𝛾09(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝐶𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝐶𝐾) + 𝛾010(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐾) + 𝛾011(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐶𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐶𝐾) + 𝛾012(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐶𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐶𝐾) + 𝛾013(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐴𝐶𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐴𝐶𝐾) + 𝑢0𝑗
𝛽1𝑗 = 𝛾10 + 𝛾11(𝑃𝑟𝑖𝑛𝑡𝑖𝑛𝑔_𝐿𝑒𝑣𝑒𝑙𝑖𝑗 − 𝑃𝑟𝑖𝑛𝑡𝑖𝑛𝑔_𝐿𝑒𝑣𝑒𝑙 ) +
𝛾12(𝑆𝑇𝐸𝑀_𝐿𝑒𝑣𝑒𝑙𝑖𝑗 − 𝑆𝑇𝐸𝑀_𝐿𝑒𝑣𝑒𝑙) + 𝛾13(𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠𝑖𝑗 −
𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠) + 𝛾14(𝐼𝑛𝑡𝑒𝑟𝑒𝑠𝑡_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 −
𝐼𝑛𝑡𝑒𝑟𝑒𝑠𝑡_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛾15(𝐼𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 −
𝐼𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛾16(𝑈𝑠𝑒𝑓𝑢𝑙𝑛𝑒𝑠𝑠_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 −
𝑈𝑠𝑒𝑓𝑢𝑙𝑛𝑒𝑠𝑠_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛾17(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐾) + 𝛾18(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾)
+ 𝛾19(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝐶𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝐶𝐾) +
𝛾110(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐾) +
𝛾111(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐶𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐶𝐾) +
𝛾112(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐶𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐶𝐾) +
𝛾113(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐴𝐶𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐴𝐶𝐾)
(4-31)
𝛽2𝑗 = 𝛾20 + 𝛾21(𝑃𝑟𝑖𝑛𝑡𝑖𝑛𝑔_𝐿𝑒𝑣𝑒𝑙𝑖𝑗 − 𝑃𝑟𝑖𝑛𝑡𝑖𝑛𝑔_𝐿𝑒𝑣𝑒𝑙 ) +
𝛾22(𝑆𝑇𝐸𝑀_𝐿𝑒𝑣𝑒𝑙𝑖𝑗 − 𝑆𝑇𝐸𝑀_𝐿𝑒𝑣𝑒𝑙) + 𝛾23(𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠𝑖𝑗 −
𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠) + 𝛾24(𝐼𝑛𝑡𝑒𝑟𝑒𝑠𝑡_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 −
𝐼𝑛𝑡𝑒𝑟𝑒𝑠𝑡_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛾25(𝐼𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 −
𝐼𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛾26(𝑈𝑠𝑒𝑓𝑢𝑙𝑛𝑒𝑠𝑠_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 −
𝑈𝑠𝑒𝑓𝑢𝑙𝑛𝑒𝑠𝑠_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛾27(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐾) + 𝛾28(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾) + 𝛾29(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝐶𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝐶𝐾) + 𝛾210(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐾) + 𝛾211(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐶𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐶𝐾) + 𝛾212(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐶𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐶𝐾) + 𝛾213(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐴𝐶𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐴𝐶𝐾)
(4-32)
After evaluating the cross-level interactions between the student-level variables
and teacher-level variables, it was found the cross-level interactions
Gender_Student*Importance_Teacher and Gender_Student*Self_Efficacy_PK were
144
significant. Therefore, after combining the student-level model (4-21) and teacher-level
models including the significant cross-level interaction terms, the equation for the final
model is:
Combined: 𝑌𝑖𝑗 = 𝛾00 + 𝛾10(𝑃𝑟𝑒_𝑆𝑐𝑖𝑒𝑛𝑐𝑒𝑖𝑗 − 𝑃𝑟𝑒_𝑆𝑐𝑖𝑒𝑛𝑐𝑒) + 𝛾20𝐺𝑒𝑛𝑑𝑒𝑟_𝑆𝑡𝑢𝑑𝑒𝑛𝑡𝑖𝑗
+ 𝛾01(𝑃𝑟𝑖𝑛𝑡𝑖𝑛𝑔_𝐿𝑒𝑣𝑒𝑙𝑖𝑗 − 𝑃𝑟𝑖𝑛𝑡𝑖𝑛𝑔_𝐿𝑒𝑣𝑒𝑙 ) + 𝛾02(𝑆𝑇𝐸𝑀_𝐿𝑒𝑣𝑒𝑙𝑖𝑗 −
𝑆𝑇𝐸𝑀_𝐿𝑒𝑣𝑒𝑙) + 𝛾03(𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠𝑖𝑗 − 𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠) +
𝛾04(𝐼𝑛𝑡𝑒𝑟𝑒𝑠𝑡_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 − 𝐼𝑛𝑡𝑒𝑟𝑒𝑠𝑡_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) +
𝛾05(𝐼𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 − 𝐼𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) +
𝛾06(𝑈𝑠𝑒𝑓𝑢𝑙𝑛𝑒𝑠𝑠_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 − 𝑈𝑠𝑒𝑓𝑢𝑙𝑛𝑒𝑠𝑠_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) +
𝛾07(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐾) +
𝛾08(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾) +
𝛾09(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝐶𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝐶𝐾) +
𝛾010(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑦_𝑇𝑃𝐾) +
𝛾011(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐶𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐶𝐾) +
𝛾012(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐶𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐶𝐾) +
𝛾013(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐴𝐶𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐴𝐶𝐾) +
𝛾25𝐺𝑒𝑛𝑑𝑒𝑟_𝑆𝑡𝑢𝑑𝑒𝑛𝑡𝑖𝑗 ∗ (𝐼𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 −
𝐼𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛾28𝐺𝑒𝑛𝑑𝑒𝑟_𝑆𝑡𝑢𝑑𝑒𝑛𝑡𝑖𝑗 ∗
(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾) + 𝑢0𝑗 + 𝑖𝑗
(4-33)
For this model with teacher-level variables, -2 Log Likelihoodfull2 = 2043.2.
LR = -2 Log Likelihoodfull1 - 2 Log Likelihoodfull2 = 2180.9 – 2043.2 = 137.7.
The likelihood ratio test was significant with 2 (15) = 180.9, p < 0.0001. The
model with teacher-level variables was significantly better than the random-intercept
model. Therefore, the random-intercept model with teacher variables was the best
model.
The residual normality for student-level and teacher-level residuals were
evaluated. The student-level residual had a skewness of -0.5696 and a kurtosis of
2.8656. The teacher-level residual was just the intercept variance. The skewness of
teacher-level residual was -0.2086 and the kurtosis of it was -0.6442. The skewness
145
and kurtosis for the residuals were between -1 and 1, and -3 and 3 respectively.
Therefore, the residual normality assumptions were met.
The statistics summary for the model with teacher variables can be viewed in
Table 4-25. After grand mean centering, the average intercept 𝛾00 = 3.5477 and it was
statistically significant with t (18.9) = 153.33, p < .0001, indicating on average the
technology/engineering motivation posttest score was 3.5477 when student gender was
male (Gender_Student = 0) and all other variables were equal to their grand mean, and
the average intercept was significantly different from zero. After accounting for all the
student-level and teacher-level variables, the teacher-level residual variance in the
teacher means, i.e., the intercept variance 𝜎𝑢02 = 0.001550 and it was not significant,
indicating the teacher means of student technology/engineering motivation posttest
scores did not vary significantly across teachers. On average, the teacher means of
student technology/engineering motivation posttest scores varied from the grand mean
by √0.001550 = 0.0394. The student-level residual variance 𝜎𝜀2 = 0.2644 and it was
statistically significant with Z = 25.27, p < .0001. Student technology/engineering
motivation posttest scores significantly varied within each teacher. On average,
individual student’s technology/engineering motivation posttest score varied from their
teacher mean by √0.2644 = 0.5142.
As the best fit model was a random-intercept model with teacher variables, each
teacher had a different intercept but the same slope for each teacher-level variable. The
slope of technology/engineering motivation pretest scores 𝛾10 = 0.7310, statistically
significant with t (1264) = 33.82, p < .0001. The slope 𝛾10 indicated on average one
score increase in technology/engineering motivation pretest score increased
146
technology/engineering motivation posttest score by 0.7107 when controlling for other
variables.
The slope of teachers’ self-efficacy in TK 𝛾07 = -0.09374 and it approached
significance with t (11.8) = -1.98, p = .0709. The slope 𝛾07 indicated one score increase
in teachers’ self-efficacy in TK decreased student technology/engineering motivation
posttest score by 0.09374 with approaching significance when controlling for other
variables. The slope of teachers’ self-efficacy in PK 𝛾08 = 0.03467 and it was significant
with t (9.44) = 2.68, p = .0241. However, student gender interacted with teachers’ self-
efficacy in PK and the slope of the interaction 𝛾28 = -0.1452, statistically significant with t
(1287) = -2.18, p = .0292. Male students were coded as 0 (reference level) and female
students were coded as 1. Therefore, the slope of teachers’ self-efficacy in PK for male
students was 0.1452 higher than the slope for female students. When student gender
was 0 (male students), the slope for male students was 0.03467, indicating one score
increase in teachers’ self-efficacy in PK increased male students’
technology/engineering motivation posttest score by 0.03467. When student gender
was 1 (female students), the slope for female students was -0.11053, indicating one
score increase in teachers’ self-efficacy in PK decreased female students’
technology/engineering motivation by 0.11053.
The slope of student gender 𝛾20 = -0.1029, statistically significant with t (1289) = -
3.45, p = .0006. Student gender interacted with teachers’ perceived importance of 3D
printing integration and the slope of the interaction 𝛾25 = 0.1189, statistically significant
with t (1288) = 2.35, p = .0191. The technology/engineering motivation posttest scores
of male students were 0.1029 higher than female students when teachers’ perceived
147
importance of 3D printing integration and self-efficacy in PK were at their grand mean
and also controlling for other variables. When controlling for other variables, the slope of
teachers’ perceived importance of 3D printing integration for female students was
0.1189 higher than the slope for male students. Therefore, although teachers’ perceived
importance of 3D printing integration was not a significant predictor, it had a more
positive effect on female students.
Table 4-25. Random-intercept model with teacher variables model summary
Parameter Estimate Standard Error
Z Value
DF t Value p
Teacher-level intercept variance (𝜎𝑢02 ) 0.001550 0.003506 0.44 / / .3292
Student-level residual variance (𝜎𝜀2) 0.2644 0.01046 25.27 / / <.0001
Intercept (𝛾00) 3.5477 0.02314 / 18.9 153.33 <.0001
Pre_Tech/Engi (𝛾10) 0.7310 0.02162 / 1264 33.82 <.0001
Gender_Student (𝛾20) -0.1029 0.02980 / 1289 -3.45 .0006
Printing_Level (𝛾01) 0.03631 0.02586 / 6.84 1.40 .2040
STEM_Level (𝛾02) -0.00529 0.05288 / 6.63 -0.10 .9233
Pedagogical_Beliefs (𝛾03) 0.06597 0.06079 / 8.85 1.09 .3065
Interest_Teacher (𝛾04) -0.06930 0.06256 / 9.3 -1.11 .2958
Importance_Teacher (𝛾05) -0.07549 0.07510 / 11.6 -1.01 .3353
Usefulness_Teacher (𝛾06) 0.08973 0.09140 / 9.08 0.98 .3517
Self_Efficacy_TK (𝛾07) -0.09374 0.04723 / 11.8 -1.98 .0709
Self_Efficacy_PK (𝛾08) 0.03467 0.05950 / 9.44 2.68 .0241
Self_Efficacy_CK (𝛾09) 0.2052 0.07648 / 7.3 0.58 .5777
Self_Efficacy_TPK (𝛾010) 0.05224 0.08817 / 7.04 0.59 .5720
Self_Efficacy_TCK (𝛾011) 0.001128 0.05869 / 9.43 0.02 .9851
148
Table 4-25. Continued
Parameter Estimate Standard Error
Z Value
DF t Value p
Self_Efficacy_PCK (𝛾012) -0.04992 0.05767 / 9.04 -0.87 .4091
Self_Efficacy_TPACK (𝛾013) 0.002202 0.1056 / 7.81 0.02 .9839
Gender_Student*Importance_Teacher (𝛾25)
0.1189 0.05068 / 1288 2.35 .0191
Gender_Student*Self_Efficacy_PK (𝛾28)
-0.1452 0.06647 / 1287 -2.18 .0292
-2 Log Likelihoodfull2 = 2043.2
Effect size calculation
The effect size was calculated by assessing the R2 at the student level and R2 at
the teacher level. R2 at the student level measured how student-level variance was
explained by the final model compared to the baseline model. R2 at the teacher level
measured how teacher-level variance was explained by the final model compared to the
student-level random-intercept model. The final model was the random-intercept model
with teacher variables. The student-level variance of the baseline model and final model
and the teacher-level variance of the student-level random-intercept model and the final
model for effect size calculation are presented in Table 4-26.
Table 4-26. Statistics for effect size calculation
Parameter Estimate
baseline model 𝜎𝜀2 0.5291
student-level random-intercept model 𝜎𝑢02 0.001456
final model 𝜎𝜀2 0.2644
final model 𝜎𝑢02 0.001550
149
R2 at the student level = (baseline model 𝜎𝜀2 - final model 𝜎𝜀
2 ) / baseline model 𝜎𝜀2
= (0.5291 – 0.2644) / 0.5291 = 50.0284%.
R2 at the teacher level = (student-level random-intercept model 𝜎𝑢02 - final model
𝜎𝑢02 ) / student-level random-intercept model 𝜎𝑢0
2 = (0.001456 - 0.001550) / 0.001456 = -
6.4560%.
The final model (random-intercept model with teacher variables) explained about
50.0284% of the student-level variance but the teacher-level variables did not contribute
to explaining teacher-level variance.
Results for Math Motivation
A series of multilevel models were built to examine how teachers’ 3D printing
integration levels, STEM integration levels, pedagogical beliefs, value beliefs, and self-
efficacy beliefs in 3D printing integration predict students’ math motivation while
controlling for student gender and math motivation pretest scores. Student gender and
math motivation pretest scores were included in the model as covariates to be
controlled while interpreting the intercept and the coefficients of other variables.
Baseline model
The first step was to build the baseline model, which predicted a student’ math
motivation posttest score from the grand mean math motivation posttest score of all the
teachers’ students. There were no student-level or teacher-level predictors in this
model. The purpose of this model was to examine the ICC and design effect to
determine whether multilevel models were necessary for the analysis. The meanings of
symbols in the model are provided in Table 4-27.
The baseline model is shown by the following equations:
150
Student Level: 𝑌𝑖𝑗 = 𝛽0𝑗 + 𝑖𝑗 (4-34)
Teacher Level: 𝛽0𝑗 = 𝛾00 + 𝑢0𝑗 (4-35)
Combined: 𝑌𝑖𝑗 = 𝛾00 + 𝑢0𝑗 + 𝑖𝑗 (4-36)
Table 4-27. The meaning of symbols in equations (4-34), (4-35), (4-36)
Symbol Meaning
𝑌𝑖𝑗 The math motivation posttest score of student i of teacher j.
𝛽0𝑗 The students’ mean math motivation posttest score for teacher j.
𝑖𝑗 A residual term – individual student differences around the mean of teacher j.
𝛾00 Grand-mean math motivation posttest score across all teachers.
𝑢0𝑗 Deviation/difference between the mean score for each teacher and the grand mean.
The baseline model statistics summary can be viewed in Table 4-28. The ICC =
𝜎𝑢02 / (𝜎𝑢0
2 + 𝜎𝜀2) = 0.09501 / (0.09501 + 0.6898) = 12.1061%. The variance in teacher
means accounted for 12.1061% of the total variance in math motivation posttest scores.
The Design Effect = 1 + (𝑛𝑐 - 1) ICC. The average number of students per teacher 𝑛𝑐 =
57.7308. The Design Effect was 7.8679. Both the ICC and the Design Effect indicated it
was necessary to conduct multilevel modeling analysis for students’ math motivation
posttest scores. The next step was to build the student-level models.
Table 4-28. Baseline model summary
Parameter Estimate Standard Error Z Value DF t Value p
𝜎𝑢02 0.09501 0.03150 3.02 / / 0.0013
𝜎𝜀2 0.6898 0.02541 27.15 / / <.0001
𝛾00 3.5589 0.06490 / 24 54.82 <.0001
-2 Log Likelihood𝑏𝑎𝑠𝑒𝑙𝑖𝑛𝑒= 3759.7
151
Student-level models
Random-intercept model. First, a random intercept model was built to estimate
the impact of student-level variables including students’ math motivation pretest scores
and gender on students’ math motivation posttest scores as fixed effects, indicating the
impact (coefficient) of pretest scores and gender did not vary across teachers. As math
motivation pretest was measured with a scale that did not contain zero, a score of zero
would have no substantive meaning. Students’ pretest scores were also independent of
each other’s scores. Therefore, grand mean centering was used to enable a value of
zero to be interpreted meaningfully. The equations for the random-intercept model with
grand-mean centering are as follows:
Student Level: 𝑌𝑖𝑗 = 𝛽0𝑗 + 𝛽1𝑗(𝑃𝑟𝑒_𝑀𝑎𝑡ℎ𝑖𝑗 − 𝑃𝑟𝑒_𝑀𝑎𝑡ℎ) + 𝛽2𝑗𝐺𝑒𝑛𝑑𝑒𝑟_𝑆𝑡𝑢𝑑𝑒𝑛𝑡𝑖𝑗
+ 𝑖𝑗
(4-37)
Teacher Level: 𝛽0𝑗 = 𝛾00 + 𝑢0𝑗 (4-38)
𝛽1𝑗 = 𝛾10 (4-39)
𝛽2𝑗 = 𝛾20 (4-40)
Combined: 𝑌𝑖𝑗 = 𝛾00 + 𝛾10(𝑃𝑟𝑒_𝑀𝑎𝑡ℎ𝑖𝑗 − 𝑃𝑟𝑒_𝑀𝑎𝑡ℎ) + 𝛾20𝐺𝑒𝑛𝑑𝑒𝑟_𝑆𝑡𝑢𝑑𝑒𝑛𝑡𝑖𝑗
+ 𝑢0𝑗 + 𝑖𝑗
(4-41)
Besides with the symbols that have been explained previously, 𝛽1𝑗 is the
regression coefficient that shows the impact of math motivation pretest scores on
posttest scores across all students of teacher j , 𝑃𝑟𝑒_𝑀𝑎𝑡ℎ𝑖𝑗 is math motivation pretest
score of student i of teacher j, 𝑃𝑟𝑒_𝑀𝑎𝑡ℎ is the grand mean of science motivation pretest
scores across teachers, 𝛾10 is the average effect (coefficient) of pretest scores on
posttest scores across all teachers, and 𝛾20 is the average effect (coefficient) of student
gender on math motivation posttest scores across all teachers. In the model, student
gender was treated as a categorical variable and male = 0 was the reference. The
random-intercept model statistics summary can be viewed in Table 4-29.
152
To determine whether the random-intercept model fit better than the baseline
model, a likelihood ratio (LR) test was used to evaluate the difference between the log
likelihood values for the nested models, i.e., the baseline model and the random-
intercept model.
LR = -2LogLikelihoodbaseline
Likelihoodfull
= (-2Log Likelihoodbaseline) - (-2Log Likelihoodfull)
In the baseline model, -2Log Likelihoodbaseline = 3759.7.
In the student-level random-intercept model, -2Log Likelihood𝑓𝑢𝑙𝑙1 = 2051.6.
Therefore, LR = 3759.7 - 2051.6 = 1708.1. LR follows a 2 distribution with df = 2.
The df = 2 because the degree of freedom of the baseline model and the random-
intercept model differed by 2, which was the difference between the number of
parameters in the two models. The p value of LR was calculated with the CHIDIST (2
value, df) function in SPSS. The likelihood ratio test was significant with 2 (2) = 1708.1,
p < .0001, indicating the random-intercept model was significantly better than the
baseline model. At least one of the student-level variables, i.e. math motivation pretest
score and student gender, can significantly predict teacher mean math motivation
posttest score.
Table 4-29. Random-intercept model summary
Parameter Estimate Standard Error Z Value DF t Value p
𝜎𝑢02 0.008004 0.003481 2.30 / / 0.0108
𝜎𝜀2 0.2232 0.008223 27.15 / / <.0001
𝛾00 3.5487 0.02552 / 44.1 139.06 <.0001
𝛾10 0.8603 0.01513 / 1420 56.85 <.0001
𝛾20 0.004700 0.02467 / 1491 0.19 .8490
-2 Log Likelihoodfull1 = 2051.6
153
Random-slope model. As the random-intercept model fit significantly better than
the baseline model, a random-slope model was built to evaluate whether the impact of
students’ math motivation pretest scores and student gender on the posttest scores
varied significantly across teachers. Variance components (i.e., random effects) of
pretest score and student gender were added to the teacher-level slope equation to
model the variation. The equations for the teacher-level model are as follows:
Teacher Level: 𝛽0𝑗 = 𝛾00 + 𝑢0𝑗 (4-42)
𝛽1𝑗 = 𝛾10 +𝑢1𝑗 (4-43)
𝛽2𝑗 = 𝛾20 +𝑢2𝑗 (4-44)
After combining with the student-level model (4-37), the combined model is:
Combined: 𝑌𝑖𝑗 = 𝛾00 + 𝛾10(𝑃𝑟𝑒_𝑀𝑎𝑡ℎ𝑖𝑗 − 𝑃𝑟𝑒_𝑀𝑎𝑡ℎ) + 𝛾20𝐺𝑒𝑛𝑑𝑒𝑟_𝑆𝑡𝑢𝑑𝑒𝑛𝑡𝑖𝑗
+ 𝑢1𝑗 (𝑃𝑟𝑒_𝑀𝑎𝑡ℎ𝑖𝑗 − 𝑃𝑟𝑒_𝑀𝑎𝑡ℎ)
+ 𝑢2𝑗𝐺𝑒𝑛𝑑𝑒𝑟_𝑆𝑡𝑢𝑑𝑒𝑛𝑡𝑖𝑗 + 𝑢0𝑗 + 𝑖𝑗
(4-45)
In the random-slope model, 𝑢1𝑗 and 𝑢2𝑗 are residual terms to show random
effects, indicating the impact of pretest score and student gender respectively on
posttest score can vary randomly across teachers.
However, the SAS Log noted, “Convergence criteria met but final Hessian is not
positive definite” and “Asymptotic variance matrix of covariance parameter estimates
has been found to be singular and a generalized inverse was used”, indicating the
random-slope model did not fit well after adding the random effects. After removing
Pre_Math from the random effects, SAS Log provided the same notes. After removing
Gender_Student from the random effects while keeping Pre_Math in the random
effects, the notes disappeared, and the model fit well. Therefore, the final random-slope
model just contained the intercept and Pre_Math in the random effects. The random
154
term 𝑢2𝑗 was removed from Formula 𝛽2𝑗 = 𝛾20 + 𝑢2𝑗 (4-44) and the final combined
model for the random-slope model is:
Combined: 𝑌𝑖𝑗 = 𝛾00 + 𝛾10(𝑃𝑟𝑒_𝑀𝑎𝑡ℎ𝑖𝑗 − 𝑃𝑟𝑒_𝑀𝑎𝑡ℎ) + 𝛾20𝐺𝑒𝑛𝑑𝑒𝑟_𝑆𝑡𝑢𝑑𝑒𝑛𝑡𝑖𝑗
+ 𝑢1𝑗 (𝑃𝑟𝑒_𝑀𝑎𝑡ℎ𝑖𝑗 − 𝑃𝑟𝑒_𝑀𝑎𝑡ℎ) + 𝑢0𝑗 + 𝑖𝑗
(4-46)
The random-slope model statistics summary can be viewed in Table 4-30. For
the random-slope model, -2 Log Likelihoodfull2= 2050.2.
LR = (-2Log Likelihoodfull1) - (-2Log Likelihoodfull2) = 2051.6 - 2050.2 = 1.4
The likelihood ratio test was nonsignificant with 2 (2) = 1.4, p = .4966, indicating
the random-slope model was not significantly better than the random-intercept model.
Therefore, it was not necessary to build the random-slope model. The next step was to
add teacher-level variables to the random-intercept model.
Table 4-30. Random-slope model summary
Parameter Estimate Standard Error Z Value DF t Value p
𝜎𝑢02 0.008191 0.003580 2.29 / / .0111
𝜎𝑢0𝑢1 -0.00186 0.002319 -0.80 / / .4228
𝜎𝑢12 0.002292 0.002938 0.78 / / .2176
𝜎𝜀2 0.2219 0.008264 26.85 / / <.0001
𝛾00 3.5523 0.02595 / 43.3 136.90 <.0001
𝛾10 0.8570 0.01865 / 13.4 45.94 <.0001
𝛾20 0.004782 0.02472 / 1489 0.19 0.8466
-2 Log Likelihoodfull2 = 2050.2
Adding teacher-level variables
Before building the model with teacher variables, multicollinearity was evaluated
to determine whether all the teacher-variables could be included in the model. As shown
in Table 4-31, all the teacher-level variables had a Tolerance of higher than 0.1 and a
155
Variance Inflation Factor (VIF) of less than 10. Therefore, all the teacher-level variables
were included in the model for further analysis.
Table 4-31. Multicollinearity of variables for math motivation posttest score
Variable Tolerance Variance Inflation Factor
Intercept . 0
Pre_Math 0.91060 1.09817
Gender_Student 0.98107 1.01930
Printing_Level 0.37711 2.65173
STEM_Level 0.18755 5.33194
Pedagogical_Beliefs 0.33494 2.98565
Interest_Teacher 0.22817 4.38271
Importance_Teacher 0.16668 5.99947
Usefulness_Teacher 0.14207 7.03858
Self_Efficacy_TK 0.22466 4.45115
Self_Efficacy_PK 0.31984 3.12660
Self_Efficacy_CK 0.28789 3.47351
Self_Efficacy_TPK 0.14480 6.90616
Self_Efficacy_TCK 0.29006 3.44759
Self_Efficacy_PCK 0.22750 4.39558
Self_Efficacy_TPACK 0.10296 9.71234
All the teacher-level variables were centered with the grand mean of the variable.
The equations for the teacher-level model are:
Teacher Level: 𝛽0𝑗 = 𝛾00 + 𝛾01(𝑃𝑟𝑖𝑛𝑡𝑖𝑛𝑔_𝐿𝑒𝑣𝑒𝑙𝑖𝑗 − 𝑃𝑟𝑖𝑛𝑡𝑖𝑛𝑔_𝐿𝑒𝑣𝑒𝑙 ) +
𝛾02(𝑆𝑇𝐸𝑀_𝐿𝑒𝑣𝑒𝑙𝑖𝑗 − 𝑆𝑇𝐸𝑀_𝐿𝑒𝑣𝑒𝑙) + 𝛾03(𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠𝑖𝑗 −
𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠) + 𝛾04(𝐼𝑛𝑡𝑒𝑟𝑒𝑠𝑡_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 −
𝐼𝑛𝑡𝑒𝑟𝑒𝑠𝑡_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛾05(𝐼𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 −
𝐼𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛾06(𝑈𝑠𝑒𝑓𝑢𝑙𝑛𝑒𝑠𝑠_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 −
𝑈𝑠𝑒𝑓𝑢𝑙𝑛𝑒𝑠𝑠_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛾07(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐾𝑖𝑗 −
(4-47)
156
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐾) + 𝛾08(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾) + 𝛾09(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝐶𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝐶𝐾) + 𝛾010(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐾) + 𝛾011(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐶𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐶𝐾) + 𝛾012(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐶𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐶𝐾) + 𝛾013(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐴𝐶𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐴𝐶𝐾) + 𝑢0𝑗
𝛽1𝑗 = 𝛾10 + 𝛾11(𝑃𝑟𝑖𝑛𝑡𝑖𝑛𝑔_𝐿𝑒𝑣𝑒𝑙𝑖𝑗 − 𝑃𝑟𝑖𝑛𝑡𝑖𝑛𝑔_𝐿𝑒𝑣𝑒𝑙 ) +
𝛾12(𝑆𝑇𝐸𝑀_𝐿𝑒𝑣𝑒𝑙𝑖𝑗 − 𝑆𝑇𝐸𝑀_𝐿𝑒𝑣𝑒𝑙) + 𝛾13(𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠𝑖𝑗 −
𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠) + 𝛾14(𝐼𝑛𝑡𝑒𝑟𝑒𝑠𝑡_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 −
𝐼𝑛𝑡𝑒𝑟𝑒𝑠𝑡_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛾15(𝐼𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 −
𝐼𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛾16(𝑈𝑠𝑒𝑓𝑢𝑙𝑛𝑒𝑠𝑠_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 −
𝑈𝑠𝑒𝑓𝑢𝑙𝑛𝑒𝑠𝑠_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛾17(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐾) + 𝛾18(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾) + 𝛾19(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝐶𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝐶𝐾) + 𝛾110(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐾) + 𝛾111(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐶𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐶𝐾) + 𝛾112(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐶𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐶𝐾) + 𝛾113(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐴𝐶𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐴𝐶𝐾)
(4-48)
𝛽2𝑗 = 𝛾20 + 𝛾21(𝑃𝑟𝑖𝑛𝑡𝑖𝑛𝑔_𝐿𝑒𝑣𝑒𝑙𝑖𝑗 − 𝑃𝑟𝑖𝑛𝑡𝑖𝑛𝑔_𝐿𝑒𝑣𝑒𝑙 ) +
𝛾22(𝑆𝑇𝐸𝑀_𝐿𝑒𝑣𝑒𝑙𝑖𝑗 − 𝑆𝑇𝐸𝑀_𝐿𝑒𝑣𝑒𝑙) + 𝛾23(𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠𝑖𝑗 −
𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠) + 𝛾24(𝐼𝑛𝑡𝑒𝑟𝑒𝑠𝑡_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 −
𝐼𝑛𝑡𝑒𝑟𝑒𝑠𝑡_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛾25(𝐼𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 −
𝐼𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛾26(𝑈𝑠𝑒𝑓𝑢𝑙𝑛𝑒𝑠𝑠_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 −
𝑈𝑠𝑒𝑓𝑢𝑙𝑛𝑒𝑠𝑠_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛾27(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐾) + 𝛾28(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾) + 𝛾29(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝐶𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝐶𝐾) + 𝛾210(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐾) + 𝛾211(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐶𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐶𝐾) + 𝛾212(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐶𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐶𝐾) + 𝛾213(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐴𝐶𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐴𝐶𝐾)
(4-49)
After evaluating the cross-level interactions between the student-level variables
and teacher-level variables, it was found the cross-level interactions PRE_MATH *
Usefulness_Teacher, PRE_MATH*Self_Efficacy_TPK, PRE_MATH*Self_Efficacy_PK,
157
and Gender_Student*Self_Efficacy_PK were significant. Therefore, after combining the
student-level model (4-37) and teacher-level models and including the significant cross-
level interaction terms, the equation for the final model is:
Combined: 𝑌𝑖𝑗 = 𝛾00 + 𝛾10(𝑃𝑟𝑒_𝑀𝑎𝑡ℎ𝑖𝑗 − 𝑃𝑟𝑒_𝑀𝑎𝑡ℎ) + 𝛾20𝐺𝑒𝑛𝑑𝑒𝑟_𝑆𝑡𝑢𝑑𝑒𝑛𝑡𝑖𝑗 +
𝛾01(𝑃𝑟𝑖𝑛𝑡𝑖𝑛𝑔_𝐿𝑒𝑣𝑒𝑙𝑖𝑗 − 𝑃𝑟𝑖𝑛𝑡𝑖𝑛𝑔_𝐿𝑒𝑣𝑒𝑙 ) + 𝛾02(𝑆𝑇𝐸𝑀_𝐿𝑒𝑣𝑒𝑙𝑖𝑗 −
𝑆𝑇𝐸𝑀_𝐿𝑒𝑣𝑒𝑙) + 𝛾03(𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠𝑖𝑗 − 𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠) +
𝛾04(𝐼𝑛𝑡𝑒𝑟𝑒𝑠𝑡_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 − 𝐼𝑛𝑡𝑒𝑟𝑒𝑠𝑡_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) +
𝛾05(𝐼𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 − 𝐼𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) +
𝛾06(𝑈𝑠𝑒𝑓𝑢𝑙𝑛𝑒𝑠𝑠_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 − 𝑈𝑠𝑒𝑓𝑢𝑙𝑛𝑒𝑠𝑠_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) +
𝛾07(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐾) +
𝛾08(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾) +
𝛾09(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝐶𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝐶𝐾) +
𝛾010(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐾) +
𝛾011(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐶𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐶𝐾) +
𝛾012(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐶𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐶𝐾) +
𝛾013(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐴𝐶𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐴𝐶𝐾) +
𝛾16(𝑃𝑟𝑒_𝑀𝑎𝑡ℎ𝑖𝑗 − 𝑃𝑟𝑒_𝑀𝑎𝑡ℎ) ∗ (𝑈𝑠𝑒𝑓𝑢𝑙𝑛𝑒𝑠𝑠_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 −
𝑈𝑠𝑒𝑓𝑢𝑙𝑛𝑒𝑠𝑠_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛾18(𝑃𝑟𝑒_𝑀𝑎𝑡ℎ𝑖𝑗 − 𝑃𝑟𝑒_𝑀𝑎𝑡ℎ) ∗
(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾) + 𝛾110(𝑃𝑟𝑒_𝑀𝑎𝑡ℎ𝑖𝑗 −
𝑃𝑟𝑒_𝑀𝑎𝑡ℎ) ∗ (𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐾) +
𝛾28𝐺𝑒𝑛𝑑𝑒𝑟_𝑆𝑡𝑢𝑑𝑒𝑛𝑡𝑖𝑗 ∗ (𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾) +
𝑢0𝑗 + 𝑖𝑗
(4-50)
For this model with teacher-level variables, -2 Log Likelihoodfull3 = 1902.8.
LR = -2 Log Likelihoodfull1 - 2 Log Likelihoodfull3 = 2051.6 – 1902.8 = 148.8.
The likelihood ratio test was significant with 2 (17) = 148.8, p < 0.0001. The
model with teacher-level variables was significantly better than the random-intercept
model. Therefore, the random-intercept model with teacher variables was the best
model.
The residual normality for student-level and teacher-level residuals were
evaluated. The student-level residual had a skewness of -0.3235 and a kurtosis of
1.5635. The teacher-level residual was just the intercept variance. The skewness of
158
teacher-level residual was 0.2018 and the kurtosis of it was -0.9543. The skewness and
kurtosis for the residuals were between -1 and 1, and -3 and 3 respectively. Therefore,
the residual normality assumptions were met.
The statistics summary for the model with teacher variables can be viewed in
Table 4-32. After grand mean centering, the average intercept 𝛾00 = 3.5446 and it was
statistically significant with t (19.1) = 137.99, p < .0001, indicating on average the math
motivation posttest score was 3.5446 when student gender was male (Gender_Student
= 0) and all other variables were equal to their grand mean, and the average intercept
was significantly different from zero. After accounting for all the student-level and
teacher-level variables, the teacher-level residual variance in the teacher means, i.e.,
the intercept variance 𝜎𝑢02 = 0.005908 and it approached significance with Z = 1.31, p =
.0944, indicating the teacher means of student math motivation posttest scores varied
across teachers with approaching significance. On average, the teacher means of
student math motivation posttest scores varied from the grand mean by √0.005908 =
0.0769. The student-level residual variance 𝜎𝜀2 = 0.2291 and it was statistically
significant with Z = 25.51, p < .0001. Student math motivation posttest scores
significantly varied within each teacher. On average, individual student’s math
motivation posttest score varied from their teacher mean by √0.2291 = 0.4786.
As the best fit model was a random-intercept model with teacher variables, each
teacher had a different intercept but the same slope for each variable. The slope of
math motivation pretest scores 𝛾10 = 0.8593, statistically significant with t (1279) =
51.66, p < .0001. The slope 𝛾10 indicated on average one score increase in math
motivation pretest score increased math motivation posttest score by 0.8593 when
159
controlling for other variables. The slope of STEM integration level 𝛾02 = 0.1674,
statistically significant with t (10.5) = 2.55, p = .0280. The slope 𝛾02 indicated one score
increase in STEM integration level increased student math motivation posttest score by
0.1674 when controlling for other variables. The slope of teachers’ perceived usefulness
of 3D printing integration 𝛾06 = -0.2413, statistically significant with t (12.3) = -2.19, p =
.0482. However, the interaction between student math motivation pretest score and
teachers’ perceived usefulness of 3D printing integration was significant with slope 𝛾16 =
-0.1199, t (1286) = -3.16, p = .0016. The slope 𝛾06 indicated one score increase in
teachers’ perceived usefulness of 3D printing integration decreased student math
motivation posttest score by 0.2413 when student math motivation pretest score was
equal to the grand mean and controlling for other variables. The interaction slope 𝛾16
indicated one score increase in student math motivation pretest score decreased the
slope of teachers’ perceived usefulness of 3D printing integration by 0.1199 when
controlling for other variables. Teachers’ self-efficacy in PCK 𝛾012 = 0.1354 and
approached significance with t (12.1) = 1.94, p = .0758. The slope 𝛾012 indicated one
score increase in teachers’ self-efficacy in PCK increased student math motivation
posttest score by 0.1354 with approaching significance when controlling for other
variables.
There were also some other cross-level interactions between the student-level
and teacher-level variables. The interaction between student math motivation pretest
score and teachers’ self-efficacy in PK was significant with slope 𝛾18 = -0.08292, t
(1193) = -2.18, p = .0298. One score increase in student math motivation pretest score
decreased the slope of teachers’ self-efficacy in PK by 0.08292. The interaction
160
between student math motivation pretest score and teachers’ self-efficacy in TPK was
significant with slope 𝛾110 = 0.1096, t (1303) = 2.89, p = .0039. One score increase in
student math motivation pretest score increased the slope of teachers’ self-efficacy in
PK by 0.1096. The interaction between student gender and teachers’ self-efficacy in PK
was significant with slope 𝛾28 = -0.2276, t (1308) = -3.71, p = .0002. As male students
were coded as 0 (reference level) and female students were coded as 1, the slope of
teachers’ self-efficacy in PK for female students (one level increase in student gender)
was 0.2276 lower than the slope for male students.
Table 4-32. Random-intercept model with teacher variables model summary
Parameter Estimate Standard Error
Z Value
DF t Value p
Teacher-level intercept variance (𝜎𝑢02 ) 0.005908 0.004496 1.31 / / .0944
Student-level residual variance (𝜎𝜀2) 0.2291 0.008980 25.51 / / <.0001
Intercept (𝛾00) 3.5446 0.02569 / 19.1 137.99 <.0001
Pre_Math (𝛾10) 0.8593 0.01663 / 1279 51.66 <.0001
Gender_Student (𝛾20) 0.008217 0.02666 / 1309 0.31 .7580
Printing_Level (𝛾01) 0.001953 0.03210 / 10.7 0.06 .9526
STEM_Level (𝛾02) 0.1674 0.06570 / 10.5 2.55 .0280
Pedagogical_Beliefs (𝛾03) 0.03716 0.07271 / 12.6 0.51 .6182
Interest_Teacher (𝛾04) -0.06060 0.07559 / 13.2 -0.80 .4370
Importance_Teacher (𝛾05) 0.1357 0.08381 / 12.5 1.62 .1304
Usefulness_Teacher (𝛾06) -0.2413 0.1100 / 12.3 -2.19 .0482
Self_Efficacy_TK (𝛾07) -0.04628 0.05611 / 15.4 -0.82 .4221
Self_Efficacy_PK (𝛾08) 0.1368 0.09106 / 12.3 1.50 .1582
Self_Efficacy_CK (𝛾09) -0.03538 0.07340 / 10.8 -0.48 .6394
Self_Efficacy_TPK (𝛾010) 0.1412 0.1079 / 11 1.31 .2173
Self_Efficacy_TCK (𝛾011) 0.02345 0.06911 / 13.4 0.34 .7397
161
Table 4-32. Continued
Parameter Estimate Standard Error
Z Value
DF t Value p
Self_Efficacy_PCK (𝛾012) 0.1354 0.06977 / 12.1 1.94 .0758
Self_Efficacy_TPACK (𝛾013) -0.1192 0.1305 / 11.9 -0.91 .3791
Pre_Math*Usefulness_Teacher (𝛾16) -0.1199 0.03797 / 1286 -3.16 .0016
Pre_Math*Self_Efficacy_PK (𝛾18) -0.08292 0.03812 / 1193 -2.18 .0298
Pre_Math*Self_Efficacy_TPK (𝛾110) 0.1096 0.03788 / 1303 2.89 .0039
Gender_Student*Self_Efficacy_PK (𝛾28)
-0.2276 0.06135 / 1308 -3.71 .0002
-2 Log Likelihoodfull3 = 1902.8
Effect size calculation
The effect size was calculated by assessing the R2 at the student level and R2 at
the teacher level. R2 at the student level measured how student-level variance was
explained by the final model compared to the baseline model. R2 at the teacher level
measured how teacher-level variance was explained by the final model compared to the
student-level random-intercept model. The final model was the random-intercept model
with teacher variables. The student-level variance of the baseline model and final model
and the teacher-level variance of the student-level random-intercept model and the final
model for effect size calculation are presented in Table 4-33.
Table 4-33. Statistics for effect size calculation
Parameter Estimate
baseline model 𝜎𝜀2 0.6898
student-level random-intercept model 𝜎𝑢02 0.008004
final model 𝜎𝜀2 0.2291
final model 𝜎𝑢02 0.005908
162
R2 at the student level = (baseline model 𝜎𝜀2 - final model 𝜎𝜀
2 ) / baseline model 𝜎𝜀2
= (0.6898 – 0.2291) / 0.6898 = 66.7875%.
R2 at the teacher level = (student-level random-intercept model 𝜎𝑢02 - final model
𝜎𝑢02 ) / student-level random-intercept model 𝜎𝑢0
2 = (0.008004 - 0.005908) / 0.008004 =
26.1869%.
Therefore, the final model (random-intercept model with teacher variables)
explained about 66.7875% of the student-level variance and the teacher-level variables
explained about 26.1869% of the teacher-level variance.
Results for 21st Century Skills
A series of multilevel models were built to examine how teachers’ 3D printing
integration levels, STEM integration levels, pedagogical beliefs, value beliefs, and self-
efficacy beliefs in 3D printing integration predict students’ 21st century skills while
controlling for student gender and 21st century skills pretest scores. Student gender and
21st century skills pretest scores were included in the model as covariates to be
controlled while interpreting the intercept and the coefficients of other variables.
Baseline model
The first step was to build the baseline model, which predicted a student’ 21st
century skills posttest score from the grand mean 21st century skills posttest score of all
the teachers’ students. There were no student-level or teacher-level predictors in this
model. The purpose of this model was to examine the ICC and design effect to
determine whether multilevel models were necessary for the analysis. The meanings of
symbols in the model are provided in Table 4-34.
163
The baseline model is shown by the following equations:
Student Level: 𝑌𝑖𝑗 = 𝛽0𝑗 + 𝑖𝑗 (4-51)
Teacher Level: 𝛽0𝑗 = 𝛾00 + 𝑢0𝑗 (4-52)
Combined: 𝑌𝑖𝑗 = 𝛾00 + 𝑢0𝑗 + 𝑖𝑗 (4-53)
Table 4-34. The meaning of symbols in equations (4-51), (4-52), (4-53)
Symbol Meaning
𝑌𝑖𝑗 The 21st century skills posttest score of student i of teacher j.
𝛽0𝑗 The students’ mean 21st century skills posttest score for teacher j.
𝑖𝑗 A residual term – individual student differences around the mean of teacher j.
𝛾00 Grand-mean 21st century skills posttest score across all teachers.
𝑢0𝑗 Deviation/difference between the mean score for each teacher and the grand mean.
The baseline model statistics summary can be viewed in Table 4-35. The ICC =
𝜎𝑢02 / (𝜎𝑢0
2 + 𝜎𝜀2) = 0.01283 / (0.01283 + 0.2933) = 4.1910%. The variance in teacher
means accounted for 4.1061% of the total variance in math motivation posttest scores.
The Design Effect = 1 + (𝑛𝑐 - 1) ICC. The average number of students per teacher 𝑛𝑐 =
57.7308. The Design Effect was 3.3776. The ICC was slightly less than 0.05 but the
Design Effect was greater than 2, therefore multilevel modeling analysis was conducted
to examine the influence of student-level and teacher level variables on students’ 21st
century skills. The next step was to build the student-level models.
Table 4-35. Baseline model summary
Parameter Estimate Standard Error Z Value DF t Value p
𝜎𝑢02 0.01283 0.005412 2.37 / / 0.0089
𝜎𝜀2 0.2933 0.01087 26.97 / / <.0001
𝛾00 4.0683 0.02693 / 23.1 151.06 <.0001
-2 Log Likelihood𝑏𝑎𝑠𝑒𝑙𝑖𝑛𝑒= 2423.6
164
Student-level models
Random-intercept model. First, a random intercept model was built to estimate
the impact of student-level variables including students’ 21st century skills pretest
scores and gender on students’ 21st century skills posttest scores as fixed effects,
indicating the impact (coefficient) of pretest scores and gender did not vary across
teachers. As 21st century skills pretest was measured with a scale that did not contain
zero, a score of zero would have no substantive meaning. Students’ pretest scores
were also independent of each other’s scores. Therefore, grand mean centering was
used to enable a value of zero to be interpreted meaningfully. The equations for the
random-intercept model with grand-mean centering are as follows:
Student Level: 𝑌𝑖𝑗 = 𝛽0𝑗 + 𝛽1𝑗(𝑃𝑟𝑒_21𝑠𝑡𝑖𝑗 − 𝑃𝑟𝑒_21𝑠𝑡) + 𝛽2𝑗𝐺𝑒𝑛𝑑𝑒𝑟_𝑆𝑡𝑢𝑑𝑒𝑛𝑡𝑖𝑗
+ 𝑖𝑗
(4-54)
Teacher Level: 𝛽0𝑗 = 𝛾00 + 𝑢0𝑗 (4-55)
𝛽1𝑗 = 𝛾10 (4-56)
𝛽2𝑗 = 𝛾20 (4-57)
Combined: 𝑌𝑖𝑗 = 𝛾00 + 𝛾10(𝑃𝑟𝑒_21𝑠𝑡𝑖𝑗 − 𝑃𝑟𝑒_21𝑠𝑡) + 𝛾20𝐺𝑒𝑛𝑑𝑒𝑟_𝑆𝑡𝑢𝑑𝑒𝑛𝑡𝑖𝑗
+ 𝑢0𝑗 + 𝑖𝑗
(4-58)
Besides with the symbols that have been explained previously, 𝛽1𝑗 is the
regression coefficient that shows the impact of 21st century skills pretest scores on
posttest scores across all students of teacher j , 𝑃𝑟𝑒_21𝑠𝑡𝑖𝑗 is 21st century skills pretest
score of student i of teacher j, 𝑃𝑟𝑒_21𝑠𝑡 is the grand mean of 21st century skills pretest
scores across teachers, 𝛾10 is the average effect (coefficient) of pretest scores on
posttest scores across all teachers, and 𝛾20 is the average effect (coefficient) of student
gender on 21st century skills posttest scores across all teachers. In the model, student
gender was treated as a categorical variable and male = 0 was the reference. The
random-intercept model statistics summary can be viewed in Table 4-36.
165
To determine whether the random-intercept model fit better than the baseline
model, a likelihood ratio (LR) test was used to evaluate the difference between the log
likelihood values for the nested models, i.e., the baseline model and the random-
intercept model.
LR = -2LogLikelihoodbaseline
Likelihoodfull = (-2Log Likelihoodbaseline) - (-2Log Likelihoodfull)
In the baseline model, -2Log Likelihoodbaseline = 2423.6.
In the student-level random-intercept model, -2Log Likelihood𝑓𝑢𝑙𝑙1 =1581.0.
Therefore, LR = 2423.6 - 1581.0 = 842.6. LR follows a 2 distribution with df = 2.
The df = 2 because the degree of freedom of the baseline model and the random-
intercept model differed by 2, which was the difference between the number of
parameters in the two models. The p value of LR was calculated with the CHIDIST (2
value, df) function in SPSS. The likelihood ratio test was significant with 2 (2) = 842.6,
p < .0001, indicating the random-intercept model was significantly better than the
baseline model. At least one of the student-level variables, i.e. 21st century skills pretest
score and student gender, can significantly predict teacher mean 21st century skills
posttest score.
Table 4-36. Random-intercept model summary
Parameter Estimate Standard Error Z Value DF t Value p
𝜎𝑢02 0.004259 0.002101 2.03 / / 0.0213
𝜎𝜀2 0.1677 0.006249 26.84 / / <.0001
𝛾00 4.0338 0.02076 / 50.7 194.33 <.0001
𝛾10 0.6941 0.02182 / 1463 31.80 <.0001
𝛾20 0.07722 0.02174 / 1458 3.55 0.0004
-2 Log Likelihoodfull1 = 1581.0
166
Random-slope model. As the random-intercept model fit significantly better than
the baseline model, a random-slope model was built to evaluate whether the impact of
students’ 21st century skills pretest scores and student gender on the posttest scores
varied significantly across teachers. Variance components (i.e., random effects) of
pretest score and student gender were added to the teacher-level slope equation to
model the variation. The equations for the teacher-level model are as follows:
Teacher Level: 𝛽0𝑗 = 𝛾00 + 𝑢0𝑗 (4-59)
𝛽1𝑗 = 𝛾10 + 𝑢1𝑗 (4-60)
𝛽2𝑗 = 𝛾20 + 𝑢2𝑗 (4-61)
After combining with the student-level model (4-54), the combined model is:
Combined: 𝑌𝑖𝑗 = 𝛾00 + 𝛾10(𝑃𝑟𝑒_21𝑠𝑡𝑖𝑗 − 𝑃𝑟𝑒_21𝑠𝑡)
+ 𝛾20𝐺𝑒𝑛𝑑𝑒𝑟_𝑆𝑡𝑢𝑑𝑒𝑛𝑡𝑖𝑗
+ 𝑢1𝑗 (𝑃𝑟𝑒_21𝑠𝑡𝑖𝑗 − 𝑃𝑟𝑒_21𝑠𝑡)
+ 𝑢2𝑗𝐺𝑒𝑛𝑑𝑒𝑟_𝑆𝑡𝑢𝑑𝑒𝑛𝑡𝑖𝑗 + 𝑢0𝑗 + 𝑖𝑗
(4-62)
In the random-slope model, 𝑢1𝑗 and 𝑢2𝑗 are residual terms to show random
effects, indicating the impact of pretest score and student gender respectively on
posttest score can vary randomly across teachers.
However, the SAS Log noted, “Convergence criteria met but final Hessian is not
positive definite”, “Estimated G matrix is not positive definite”, and “Asymptotic variance
matrix of covariance parameter estimates has been found to be singular and a
generalized inverse was used”, indicating the random-slope model did not fit well after
adding the random effects. When deleting Pre_21st from the random effects, the SAS
Log showed the same warning. After removing Gender_Student from the random
effects while keeping Pre_21st in the random effects, the notes disappeared, and the
model fit well. Therefore, the final random-slope model just contained the intercept and
167
PRE_21st in the random effects. The random term 𝑢2𝑗 was removed from Formula
𝛽2𝑗 = 𝛾20 + 𝑢2𝑗 (4-61) and the final combined model for the random-slope model is:
Combined: 𝑌𝑖𝑗 = 𝛾00 + 𝛾10(𝑃𝑟𝑒_21𝑠𝑡𝑖𝑗 − 𝑃𝑟𝑒_21𝑠𝑡) + 𝛾20𝐺𝑒𝑛𝑑𝑒𝑟_𝑆𝑡𝑢𝑑𝑒𝑛𝑡𝑖𝑗
+ 𝑢1𝑗 (𝑃𝑟𝑒_21𝑠𝑡𝑖𝑗 − 𝑃𝑟𝑒_21𝑠𝑡) + 𝑢0𝑗 + 𝑖𝑗
(4-63)
The random-slope model statistics summary can be viewed in Table 4-37. For
the random-slope model, -2 Log Likelihoodfull2= 1577.7.
LR = (-2Log Likelihoodfull1) - (-2Log Likelihoodfull2) = 1581.0 -1577.7 = 3.3
The likelihood ratio test was nonsignificant with 2 (2) = 3.3, p = .1920, indicating
the random-slope model was not significantly better than the random-intercept model.
Therefore, it was not necessary to build the random-slope model. The next step was to
add teacher-level variables to the random-intercept model.
Table 4-37. Random-slope model summary
Parameter Estimate Standard Error Z Value DF t Value p
𝜎𝑢02 0.004591 0.002230 2.06 / / .0198
𝜎𝑢0𝑢1 -0.00098 0.002671 -0.37 / / .7140
𝜎𝑢12 0.007788 0.005999 1.30 / / .0971
𝜎𝜀2 0.1658 0.006242 26.57 / / <.0001
𝛾00 4.0361 0.02110 / 48.1 191.30 <.0001
𝛾10 0.6943 0.02862 / 21.3 24.26 <.0001
𝛾20 0.07687 0.02168 / 1445 3.54 .0004
-2 Log Likelihoodfull2 = 1577.7
168
Adding teacher-level variables
Before building the model with teacher variables, multicollinearity was evaluated
to determine whether all the teacher-variables could be included in the model. As shown
in Table 4-38, all the teacher-level variables had a Tolerance of higher than 0.1 and a
Variance Inflation Factor (VIF) of less than 10. Therefore, all the teacher-level variables
were included in the model for further analysis.
Table 4-38. Multicollinearity of variables for 21st century skills posttest score
Variable Tolerance Variance Inflation Factor
Intercept . 0
Pre_21st 0.97464 1.02602
Gender_Student 0.96867 1.03235
Printing_Level 0.38434 2.60186
STEM_Level 0.19046 5.25047
Pedagogical_Beliefs 0.33224 3.00984
Interest_Teacher 0.23127 4.32401
Importance_Teacher 0.16419 6.09038
Usefulness_Teacher 0.14083 7.10066
Self_Efficacy_TK 0.23242 4.30262
Self_Efficacy_PK 0.32293 3.09665
Self_Efficacy_CK 0.29137 3.43206
Self_Efficacy_TPK 0.14746 6.78141
Self_Efficacy_TCK 0.29142 3.43151
Self_Efficacy_PCK 0.23004 4.34709
Self_Efficacy_TPACK 0.10754 9.29916
All the teacher-level variables were centered with the grand mean of the variable.
The equations for the teacher-level model are:
169
Teacher Level: 𝛽0𝑗 = 𝛾00 + 𝛾01(𝑃𝑟𝑖𝑛𝑡𝑖𝑛𝑔_𝐿𝑒𝑣𝑒𝑙𝑖𝑗 − 𝑃𝑟𝑖𝑛𝑡𝑖𝑛𝑔_𝐿𝑒𝑣𝑒𝑙 ) +
𝛾02(𝑆𝑇𝐸𝑀_𝐿𝑒𝑣𝑒𝑙𝑖𝑗 − 𝑆𝑇𝐸𝑀_𝐿𝑒𝑣𝑒𝑙) + 𝛾03(𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠𝑖𝑗 −
𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠) + 𝛾04(𝐼𝑛𝑡𝑒𝑟𝑒𝑠𝑡_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 −
𝐼𝑛𝑡𝑒𝑟𝑒𝑠𝑡_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛾05(𝐼𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 −
𝐼𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛾06(𝑈𝑠𝑒𝑓𝑢𝑙𝑛𝑒𝑠𝑠_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 −
𝑈𝑠𝑒𝑓𝑢𝑙𝑛𝑒𝑠𝑠_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛾07(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐾) + 𝛾08(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾) + 𝛾09(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝐶𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝐶𝐾) + 𝛾010(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐾) + 𝛾011(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐶𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐶𝐾) + 𝛾012(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐶𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐶𝐾) + 𝛾013(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐴𝐶𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐴𝐶𝐾) + 𝑢0𝑗
(4-64)
𝛽1𝑗 = 𝛾10 + 𝛾11(𝑃𝑟𝑖𝑛𝑡𝑖𝑛𝑔_𝐿𝑒𝑣𝑒𝑙𝑖𝑗 − 𝑃𝑟𝑖𝑛𝑡𝑖𝑛𝑔_𝐿𝑒𝑣𝑒𝑙 ) +
𝛾12(𝑆𝑇𝐸𝑀_𝐿𝑒𝑣𝑒𝑙𝑖𝑗 − 𝑆𝑇𝐸𝑀_𝐿𝑒𝑣𝑒𝑙) + 𝛾13(𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠𝑖𝑗 −
𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠) + 𝛾14(𝐼𝑛𝑡𝑒𝑟𝑒𝑠𝑡_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 −
𝐼𝑛𝑡𝑒𝑟𝑒𝑠𝑡_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛾15(𝐼𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 −
𝐼𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛾16(𝑈𝑠𝑒𝑓𝑢𝑙𝑛𝑒𝑠𝑠_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 −
𝑈𝑠𝑒𝑓𝑢𝑙𝑛𝑒𝑠𝑠_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛾17(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐾) + 𝛾18(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾)
+ 𝛾19(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝐶𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝐶𝐾) +
𝛾110(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐾) +
𝛾111(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐶𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐶𝐾) +
𝛾112(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐶𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐶𝐾) +
𝛾113(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐴𝐶𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐴𝐶𝐾)
(4-65)
𝛽2𝑗 = 𝛾20 + 𝛾21(𝑃𝑟𝑖𝑛𝑡𝑖𝑛𝑔_𝐿𝑒𝑣𝑒𝑙𝑖𝑗 − 𝑃𝑟𝑖𝑛𝑡𝑖𝑛𝑔_𝐿𝑒𝑣𝑒𝑙 ) +
𝛾22(𝑆𝑇𝐸𝑀_𝐿𝑒𝑣𝑒𝑙𝑖𝑗 − 𝑆𝑇𝐸𝑀_𝐿𝑒𝑣𝑒𝑙) + 𝛾23(𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠𝑖𝑗 −
𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠) + 𝛾24(𝐼𝑛𝑡𝑒𝑟𝑒𝑠𝑡_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 −
𝐼𝑛𝑡𝑒𝑟𝑒𝑠𝑡_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛾25(𝐼𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 −
𝐼𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛾26(𝑈𝑠𝑒𝑓𝑢𝑙𝑛𝑒𝑠𝑠_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 −
𝑈𝑠𝑒𝑓𝑢𝑙𝑛𝑒𝑠𝑠_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛾27(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐾) + 𝛾28(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾) + 𝛾29(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝐶𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝐶𝐾) + 𝛾210(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐾) + 𝛾211(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐶𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐶𝐾) + 𝛾212(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐶𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐶𝐾) + 𝛾213(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐴𝐶𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐴𝐶𝐾)
(4-66)
170
After evaluating the cross-level interactions between the student-level variables
and teacher-level variables, it was found the cross-level interactions
Pre_21st*Self_Efficacy_CK and Gender_Student*Pedagogical_Beliefs were significant.
Therefore, after combining the student-level model (4-54) and teacher-level models
including the significant cross-level interaction terms, the equation for the final model is:
Combined: 𝑌𝑖𝑗 = 𝛾00 + 𝛾10(𝑃𝑟𝑒_𝑆𝑐𝑖𝑒𝑛𝑐𝑒𝑖𝑗 − 𝑃𝑟𝑒_𝑆𝑐𝑖𝑒𝑛𝑐𝑒) + 𝛾20𝐺𝑒𝑛𝑑𝑒𝑟_𝑆𝑡𝑢𝑑𝑒𝑛𝑡𝑖𝑗
+ 𝛾01(𝑃𝑟𝑖𝑛𝑡𝑖𝑛𝑔_𝐿𝑒𝑣𝑒𝑙𝑖𝑗 − 𝑃𝑟𝑖𝑛𝑡𝑖𝑛𝑔_𝐿𝑒𝑣𝑒𝑙 ) + 𝛾02(𝑆𝑇𝐸𝑀_𝐿𝑒𝑣𝑒𝑙𝑖𝑗 −
𝑆𝑇𝐸𝑀_𝐿𝑒𝑣𝑒𝑙) + 𝛾03(𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠𝑖𝑗 − 𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠) +
𝛾04(𝐼𝑛𝑡𝑒𝑟𝑒𝑠𝑡_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 − 𝐼𝑛𝑡𝑒𝑟𝑒𝑠𝑡_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) +
𝛾05(𝐼𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 − 𝐼𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) +
𝛾06(𝑈𝑠𝑒𝑓𝑢𝑙𝑛𝑒𝑠𝑠_𝑇𝑒𝑎𝑐ℎ𝑒𝑟𝑖𝑗 − 𝑈𝑠𝑒𝑓𝑢𝑙𝑛𝑒𝑠𝑠_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) +
𝛾07(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐾) +
𝛾08(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾) +
𝛾09(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝐶𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝐶𝐾) +
𝛾010(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾) +
𝛾011(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐶𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐶𝐾) +
𝛾012(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐶𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐶𝐾) +
𝛾013(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐴𝐶𝐾𝑖𝑗 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐴𝐶𝐾) +
𝛾19(𝑃𝑟𝑒_21𝑠𝑡𝑖𝑗 − 𝑃𝑟𝑒_21𝑠𝑡) ∗ (𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝐶𝐾𝑖𝑗 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝐶𝐾) + 𝛾23𝐺𝑒𝑛𝑑𝑒𝑟_𝑆𝑡𝑢𝑑𝑒𝑛𝑡𝑖𝑗 ∗
(𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠𝑖𝑗 − 𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠) + 𝑢0𝑗 + 𝑖𝑗
(4-67)
For this model with teacher-level variables, -2 Log Likelihoodfull3 = 1488.6.
LR = -2 Log Likelihoodfull1 - 2 Log Likelihoodfull3 = 1581.0 – 1488.6 = 92.4.
The likelihood ratio test was significant with 2 (15) = 92.4, p < .0001. The model
with teacher-level variables was significantly better than the random-intercept model.
Therefore, the random-intercept model with teacher variables was the best model.
The residual normality for student-level and teacher-level residuals were
evaluated. The student-level residual had a skewness of -0.5718 and a kurtosis of
2.8598. The teacher-level residual was just the intercept variance. The skewness of
171
teacher-level residual was -0.5489 and the kurtosis of it was 0.3091. The skewness and
kurtosis for the residuals were between -1 and 1, and -3 and 3 respectively. Therefore,
the residual normality assumptions were met.
The statistics summary for the model with teacher variables can be viewed in
Table 4-39. After grand mean centering, the average intercept 𝛾00 = 4.0246 and it was
statistically significant with t (20.9) = 207.22, p < .0001, indicating on average the 21st
century skills posttest score was 4.0246 when student gender was male
(Gender_Student = 0) and all other variables were equal to their grand mean, and the
average intercept was significantly different from zero. After accounting for all the
student-level and teacher-level variables, the teacher-level residual variance in the
teacher means, i.e., the intercept variance 𝜎𝑢02 = 0.001683 and it was not significant,
indicating the teacher means of student 21st century skills posttest scores did not vary
significantly across teachers. On average, the teacher means of student 21st century
skills posttest scores varied from the grand mean by √0.001683 = 0.0410. The student-
level residual variance 𝜎𝜀2 = 0.1734 and it was statistically significant with Z = 25.20, p <
.0001. Student 21st century skills posttest scores significantly varied within each
teacher. On average, individual student’s 21st century skills posttest score varied from
their teacher mean by √0.1734 = 0.4164.
As the best fit model was a random-intercept model with teacher variables, each
teacher had a different intercept but the same slope for each teacher-level variable. The
slope of 21st century skills pretest scores 𝛾10 = 0.6943, statistically significant with t
(1278) = 29.53, p < .0001. Students’ pretest scores interacted with teachers’ self-
efficacy beliefs in CK. To interpret the slope of the pretest scores 𝛾10, the interaction
172
had to be ruled out, namely teachers’ self-efficacy beliefs in CK had to be at its grand
mean. Thus, the slope 𝛾10 indicated on average one score increase in 21st century
skills pretest score increased 21st century skills posttest score by 0.6943 when
teachers’ self-efficacy beliefs in CK was at its grand mean and also controlling for other
variables. The slope of the interaction between 21st century skills pretest scores and
teachers’ self-efficacy beliefs in CK 𝛾19 = 0.08887 and it was statistically significant with
t (1225) = 2.08, p = .0380. The interaction slope 𝛾19 indicated one score increase in
students’ 21st century skills pretest scores increased the slope of teachers’ self-efficacy
beliefs in CK by 0.08887 when controlling for other variables.
The slope of student gender 𝛾20 = 0.08505, statistically significant with t (1278) =
3.61, p = .0003. Student gender interacted with teachers’ pedagogical beliefs. Male
students were coded as 0 (reference level) and female students were coded as 1. To
interpret the slope of gender 𝛾20, the interaction had to be ruled out, namely teachers’
pedagogical beliefs had to be at its grand mean. Thus, the 21st century skills posttest
scores of female students were 0.08505 higher than male students when teachers’
pedagogical beliefs were at the grand mean and also controlling for other variables. The
slope of the interaction between student gender and teachers’ pedagogical beliefs 𝛾23 =
0.1186 and it was statistically significant with t (1280) = 2.36, p = .0186. The slope of
teachers’ pedagogical beliefs for female students was 0.1186 higher than the slope for
male students. Lastly, the slope of teachers’ perceived usefulness of 3D printing
integration 𝛾06 = -0.2280 and it was statistically significant with t (11.1) = -2.89, p =
.0147. The slope 𝛾06 indicated one score increase in teachers’ perceived usefulness of
173
3D printing integration decreased students’ 21st century skills posttest scores by 0.2280
when controlling for other variables.
Table 4-39. Random-intercept model with teacher variables model summary
Parameter Estimate Standard Error
Z Value
DF t Value p
Teacher-level intercept variance (𝜎𝑢02 ) 0.001683 0.002394 0.70 / / .2409
Student-level residual variance (𝜎𝜀2) 0.1734 0.006880 25.20 / / <.0001
Intercept (𝛾00) 4.0246 0.01942 / 20.9 207.22 <.0001
Pre_21st (𝛾10) 0.6943 0.02351 / 1278 29.53 <.0001
Gender_Student (𝛾20) 0.08505 0.02357 / 1278 3.61 .0003
Printing_Level (𝛾01) 0.006339 0.02248 / 8.61 0.28 .7846
STEM_Level (𝛾02) 0.05526 0.04611 / 8.44 1.20 .2633
Pedagogical_Beliefs (𝛾03) 0.01742 0.05838 / 17.1 0.30 .7691
Interest_Teacher (𝛾04) 0.005602 0.05383 / 11.3 0.10 .9189
Importance_Teacher (𝛾05) 0.05545 0.06078 / 11 0.91 .3811
Usefulness_Teacher (𝛾06) -0.2280 0.07903 / 11.1 -2.89 .0147
Self_Efficacy_TK (𝛾07) -0.02248 0.04063 / 14.1 -0.55 .5888
Self_Efficacy_PK (𝛾08) 0.06666 0.05916 / 7.41 1.13 .2950
Self_Efficacy_CK (𝛾09) -0.07383 0.05160 / 9 -1.43 .1863
Self_Efficacy_TPK (𝛾010) 0.03989 0.07648 / 8.94 0.52 .6146
Self_Efficacy_TCK (𝛾011) -0.02865 0.05062 / 11.8 -0.57 .5820
Self_Efficacy_PCK (𝛾012) 0.03742 0.04979 / 10.9 0.75 .4683
Self_Efficacy_TPACK (𝛾013) 0.06426 0.09153 / 9.69 0.70 .4991
Pre_21st*Self_Efficacy_CK (𝛾19) 0.08887 0.04278 / 1225 2.08 .0380
Gender_Student*Pedagogical_Beliefs
(𝛾23) 0.1186 0.05033 / 1280 2.36 .0186
-2 Log Likelihoodfull2 = 1488.6
174
Effect size calculation
The effect size was calculated by assessing the R2 at the student level and R2 at
the teacher level. R2 at the student level measured how student-level variance was
explained by the final model compared to the baseline model. R2 at the teacher level
measured how teacher-level variance was explained by the final model compared to the
student-level random-intercept model. The final model was the random-intercept model
with teacher variables. The student-level variance of the baseline model and final model
and the teacher-level variance of the student-level random-intercept model and the final
model for effect size calculation are presented in Table 4-40.
Table 4-40. Statistics for effect size calculation
Parameter Estimate
baseline model 𝜎𝜀2 0.2933
student-level random-intercept model 𝜎𝑢02 0.004259
final model 𝜎𝜀2 0.1734
final model 𝜎𝑢02 0.001683
R2 at the student level = (baseline model 𝜎𝜀2 - final model 𝜎𝜀
2 ) / baseline model 𝜎𝜀2
= (0.2933 – 0.1734) / 0.2933 = 40.8796%.
R2 at the teacher level = (student-level random-intercept model 𝜎𝑢02 - final model
𝜎𝑢02 ) / student-level random-intercept model 𝜎𝑢0
2 = (0.004259 - 0.001683) / 0.004259 =
60.4837%.
The final model (random-intercept model with teacher variables) explained about
40.8796% and the teacher-level variables explained about 60.4837% of the teacher-
level variance.
175
Results for Interest in STEM Careers
Multilevel models were built to examine how teachers’ 3D printing integration
levels, STEM integration levels, pedagogical beliefs, value beliefs, and self-efficacy
beliefs in 3D printing integration predict students’ interest in STEM careers while
controlling for student gender and interest in STEM careers pretest scores, which were
included in the model as covariates to be controlled while interpreting the intercept and
the coefficients of other variables.
Baseline model
The first step was to build the baseline model, which predicted a student’ interest
in STEM careers posttest score from the grand mean interest in STEM careers posttest
score of all the teachers’ students. There were no student-level or teacher-level
predictors in this model. The purpose of this model was to examine the ICC and design
effect to determine whether multilevel models were necessary for the analysis. The
meanings of symbols in the model are provided in Table 4-41.
The baseline model is shown by the following equations:
Student Level: 𝑌𝑖𝑗 = 𝛽0𝑗 + 𝑖𝑗 (4-68)
Teacher Level: 𝛽0𝑗 = 𝛾00 + 𝑢0𝑗 (4-69)
Combined: 𝑌𝑖𝑗 = 𝛾00 + 𝑢0𝑗 + 𝑖𝑗 (4-70)
Table 4-41. The meanings of symbols in equations (4-68), (4-69), (4-70)
Symbol Meaning
𝑌𝑖𝑗 The interest in STEM careers posttest score of student i of teacher j.
𝛽0𝑗 The students’ mean interest in STEM careers posttest score for teacher j.
𝑖𝑗 A residual term – individual student differences around the mean of teacher j.
𝛾00 Grand-mean interest in STEM careers posttest score across all teachers.
𝑢0𝑗 Deviation/difference between the mean score for each teacher and the grand mean.
176
The baseline model statistics summary can be viewed in Table 4-42. The ICC =
𝜎𝑢02 / (𝜎𝑢0
2 + 𝜎𝜀2) = 0.01133 / (0.01133 + 0.2533) = 4.2815%. The variance in teacher
means accounted for 4.2815% of the total variance in math motivation posttest scores.
The Design Effect = 1 + (𝑛𝑐 - 1) ICC. The average number of students per teacher 𝑛𝑐 =
57.7308. The Design Effect was 3.4289. The ICC was slightly less than 0.05 but the
Design Effect was greater than 2, therefore multilevel modeling analysis was conducted
to examine the influence of student-level and teacher level variables on students’
interest in STEM careers. The next step was to build the student-level models.
Table 4-42. Baseline model summary
Parameter Estimate Standard Error Z Value DF t Value p
𝜎𝑢02 0.01133 0.005009 2.26 / / .0118
𝜎𝜀2 0.2533 0.009419 26.89 / / <.0001
𝛾00 2.4277 0.02524 / 20.8 96.17 <.0001
-2 Log Likelihood𝑏𝑎𝑠𝑒𝑙𝑖𝑛𝑒= 2198.0
Student-level models
Random-intercept model. First, a random intercept model was built to estimate
the impact of student-level variables including students’ interest in STEM careers
pretest scores and gender on students’ interest in STEM careers posttest scores as
fixed effects, indicating the impact (coefficient) of pretest scores and gender did not vary
across teachers. As interest in STEM careers pretest was measured with a scale that
did not contain zero, a score of zero would have no substantive meaning. Students’
pretest scores were also independent of each other’s scores. Therefore, grand mean
centering was used to enable a value of zero to be interpreted meaningfully. The
equations for the random-intercept model with grand-mean centering are as follows:
177
Student Level: 𝑌𝑖𝑗 = 𝛽0𝑗 + 𝛽1𝑗(𝑃𝑟𝑒_𝐶𝑎𝑟𝑒𝑒𝑟𝑖𝑗 − 𝑃𝑟𝑒_𝐶𝑎𝑟𝑒𝑒𝑟)
+ 𝛽2𝑗𝐺𝑒𝑛𝑑𝑒𝑟_𝑆𝑡𝑢𝑑𝑒𝑛𝑡𝑖𝑗 + 𝑖𝑗
(4-71)
Teacher Level: 𝛽0𝑗 = 𝛾00 + 𝑢0𝑗 (4-72)
𝛽1𝑗 = 𝛾10 (4-73)
𝛽2𝑗 = 𝛾20 (4-74)
Combined: 𝑌𝑖𝑗 = 𝛾00 + 𝛾10(𝑃𝑟𝑒_𝐶𝑎𝑟𝑒𝑒𝑟𝑖𝑗 − 𝑃𝑟𝑒_𝐶𝑎𝑟𝑒𝑒𝑟)
+ 𝛾20𝐺𝑒𝑛𝑑𝑒𝑟_𝑆𝑡𝑢𝑑𝑒𝑛𝑡𝑖𝑗 + 𝑢0𝑗 + 𝑖𝑗
(4-75)
Besides with the symbols that have been explained previously, 𝛽1𝑗 is the
regression coefficient that shows the impact of interest in STEM careers pretest scores
on posttest scores across all students of teacher j , 𝑃𝑟𝑒_𝐶𝑎𝑟𝑒𝑒𝑟𝑖𝑗 is interest in STEM
careers pretest score of student i of teacher j, 𝑃𝑟𝑒_𝐶𝑎𝑟𝑒𝑒𝑟 is the grand mean of interest
in STEM careers pretest scores across teachers, 𝛾10 is the average effect (coefficient)
of pretest scores on posttest scores across all teachers, and 𝛾20 is the average effect
(coefficient) of student gender on interest in STEM careers posttest scores across all
teachers. In the model, student gender was treated as a categorical variable and male =
0 was the reference.
However, after running the random-intercept model in the SAS program, the SAS
log indicated that “Estimated G matrix is not positive definite” and “Asymptotic variance
matrix of covariance parameter estimates has been found to be singular and a
generalized inverse was used” and the teacher-level intercept variance was zero,
indicating the clustering of students within teachers did not help explain the variance
and students can be considered as independent of each other no matter they had the
same teacher or not. Therefore, a multiple regression model was built to evaluate the
influence of the student and teacher variables on students’ interest in STEM careers
posttest scores.
178
Multiple regression
Before building the multiple regression, multicollinearity was evaluated to
determine whether all the teacher-variables could be included in the model. As shown in
Table 4-43, all the teacher-level variables had a Tolerance of higher than 0.1 and a
Variance Inflation Factor (VIF) of less than 10. Therefore, all the teacher-level variables
were included in the model for further analysis.
Table 4-43. Multicollinearity of variables for interest in STEM careers posttest score
Variable Tolerance Variance Inflation Factor
Intercept . 0
Pre_Career 0.94111 1.06257
Gender_Student 0.98329 1.01699
Printing_Level 0.38622 2.58922
STEM_Level 0.19071 5.24354
Pedagogical_Beliefs 0.32934 3.03640
Interest_Teacher 0.23046 4.33909
Importance_Teacher 0.16277 6.14369
Usefulness_Teacher 0.13945 7.17109
Self_Efficacy_TK 0.23310 4.28993
Self_Efficacy_PK 0.32137 3.11167
Self_Efficacy_CK 0.28985 3.45006
Self_Efficacy_TPK 0.14815 6.75011
Self_Efficacy_TCK 0.29116 3.43457
Self_Efficacy_PCK 0.22932 4.36063
Self_Efficacy_TPACK 0.10785 9.27191
All the teacher-level variables were centered with the grand mean of the variable.
The interactions Pre_Career*Printing_Level, Pre_Career*STEM_Level,
179
Pre_Career*Pedagogical_Beliefs, Pre_Career*Usefulness_Teacher,
Pre_Career*Self_Efficacy_PCK, and Gender_Student* Self_Efficacy_CK were
significant. Therefore, after including the interaction terms and centering the variables at
their grand means, the equation for the final multiple regression model is:
𝑌 = 𝛽0 + 𝛽1(𝑃𝑟𝑒_𝐶𝑎𝑟𝑒𝑒𝑟 − 𝑃𝑟𝑒_𝐶𝑎𝑟𝑒𝑒𝑟) + 𝛽2𝐺𝑒𝑛𝑑𝑒𝑟_𝑆𝑡𝑢𝑑𝑒𝑛𝑡 +
𝛽3(𝑃𝑟𝑖𝑛𝑡𝑖𝑛𝑔_𝐿𝑒𝑣𝑒𝑙 − 𝑃𝑟𝑖𝑛𝑡𝑖𝑛𝑔_𝐿𝑒𝑣𝑒𝑙 ) + 𝛽4 (𝑆𝑇𝐸𝑀_𝐿𝑒𝑣𝑒𝑙 − 𝑆𝑇𝐸𝑀_𝐿𝑒𝑣𝑒𝑙) + 𝛽5
(𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠 − 𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠) + 𝛽6(𝐼𝑛𝑡𝑒𝑟𝑒𝑠𝑡_𝑇𝑒𝑎𝑐ℎ𝑒𝑟 −
𝐼𝑛𝑡𝑒𝑟𝑒𝑠𝑡_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛽7(𝐼𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒_𝑇𝑒𝑎𝑐ℎ𝑒𝑟 − 𝐼𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) +
𝛽8(𝑈𝑠𝑒𝑓𝑢𝑙𝑛𝑒𝑠𝑠_𝑇𝑒𝑎𝑐ℎ𝑒𝑟 − 𝑈𝑠𝑒𝑓𝑢𝑙𝑛𝑒𝑠𝑠_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) + 𝛽9(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐾 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐾) + 𝛽10(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐾) +
𝛽11(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝐶𝐾 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝐶𝐾) + 𝛽12(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐾 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐾) + 𝛽13(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐶𝐾 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝐶𝐾) +
𝛽14(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐶𝐾 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐶𝐾) + 𝛽15(𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐴𝐶𝐾 −
𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑇𝑃𝐴𝐶𝐾) + 𝛽16(𝑃𝑟𝑒_𝐶𝑎𝑟𝑒𝑒𝑟 − 𝑃𝑟𝑒_𝐶𝑎𝑟𝑒𝑒𝑟) ∗ (𝑃𝑟𝑖𝑛𝑡𝑖𝑛𝑔_𝐿𝑒𝑣𝑒𝑙 −
𝑃𝑟𝑖𝑛𝑡𝑖𝑛𝑔_𝐿𝑒𝑣𝑒𝑙) + 𝛽17(𝑃𝑟𝑒_𝐶𝑎𝑟𝑒𝑒𝑟 − 𝑃𝑟𝑒_𝐶𝑎𝑟𝑒𝑒𝑟) ∗ (𝑆𝑇𝐸𝑀_𝐿𝑒𝑣𝑒𝑙 − 𝑆𝑇𝐸𝑀_𝐿𝑒𝑣𝑒𝑙) +
𝛽18(𝑃𝑟𝑒_𝐶𝑎𝑟𝑒𝑒𝑟 − 𝑃𝑟𝑒_𝐶𝑎𝑟𝑒𝑒𝑟) ∗ (𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠 − 𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙_𝐵𝑒𝑙𝑖𝑒𝑓𝑠) +
𝛽19(𝑃𝑟𝑒_𝐶𝑎𝑟𝑒𝑒𝑟 − 𝑃𝑟𝑒_𝐶𝑎𝑟𝑒𝑒𝑟) ∗ (𝑈𝑠𝑒𝑓𝑢𝑙𝑛𝑒𝑠𝑠_𝑇𝑒𝑎𝑐ℎ𝑒𝑟 − 𝑈𝑠𝑒𝑓𝑢𝑙𝑛𝑒𝑠𝑠_𝑇𝑒𝑎𝑐ℎ𝑒𝑟) +
𝛽20(𝑃𝑟𝑒_𝐶𝑎𝑟𝑒𝑒𝑟 − 𝑃𝑟𝑒_𝐶𝑎𝑟𝑒𝑒𝑟) ∗ (𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐶𝐾 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝑃𝐶𝐾)
𝛽21𝐺𝑒𝑛𝑑𝑒𝑟_𝑆𝑡𝑢𝑑𝑒𝑛𝑡 ∗ (𝑆𝑒𝑙_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝐶𝐾 − 𝑆𝑒𝑙𝑓_𝐸𝑓𝑓𝑖𝑐𝑎𝑐𝑦_𝐶𝐾)
(4-76)
In addition to the normality assumption of the student interest in STEM careers
posttest scores and the multicollinearity assumption, multiple regression models also
assume the residuals are normally distributed and there is no autocorrelation in the
residuals. The residuals had a skewness of -0.2879 and a kurtosis of 2.2548. The
skewness and kurtosis of the residuals were between -1 and 1, and -3 and 3
respectively. Therefore, the residual normality assumption was met. The autocorrelation
in the residuals was evaluated with the Durbin-Watson test. A value of 2 of the Durbin-
Watson test indicates no autocorrelation. For this multiple regression model, the Durbin-
180
Watson statistic was 2.02, very close to 2. Therefore, there was no autocorrelation in
the residuals.
The main regression model was significant at F (21, 1264) = 45.67, p < .0001,
with an R2 of 0.4314, indicating 43.14% of the total variance in student interest in STEM
careers posttest scores was associated with the student and teacher variables. The
statistics summary for the multiple regression model can be viewed in Table 4-44. After
grand mean centering, the intercept 𝛽0 = 2.44112, indicating the posttest score was
2.44112 when student gender was male (Gender_Student = 0) and all other variables
were equal to their grand mean.
Except for student STEM career interest pretest scores, none of the other
variables were significant. The slope of student pretest score 𝛽1 = 0.62555 and it was
significant with t (1) = 28.56, p < .0001, when controlling for other variables. Student
STEM career interest pretest score had significant interactions with a few teacher
variables. The slope of the interaction between the pretest score and teachers’ 3D
printing integration level 𝛽16 = -0.06581 and it was significant with t (1) = -2.26, p =
.0243. The slope of the interaction between the pretest score and teachers’ STEM
integration level 𝛽17 = 0.10874 and it was significant with t (1) = 2.07, p = .0386. The
slope of the interaction between the pretest score and teachers’ STEM integration level
𝛽17 = 0.10874 and it was significant with t (1) = 2.07, p = .0386. The slope of the
interaction between the pretest score and teachers’ pedagogical beliefs 𝛽18 = 0.14733
and it was significant with t (1) = 2.82, p = .0048. The slope of the interaction between
the pretest score and teachers’ perceived usefulness of 3D printing integration 𝛽19 = -
0.12311 and it was significant with t (1) = -2.43, p = .0150. The slope of the interaction
181
between the pretest score and teachers’ self-efficacy beliefs in PCK 𝛽20 = 0.09823 and
it was significant with t (1) = 2.30, p = .0217. One score increase in students’ STEM
career interest pretest score decreased the slope of teachers’ 3D printing integration
level by 0.06581, increased the slope of teachers’ STEM integration level by 0.10874,
increased the slope of teachers’ pedagogical beliefs by 0.14733, decreased the slope of
teachers’ perceived usefulness of 3D printing integration by 0.12311, and increased the
slope of teachers’ self-efficacy beliefs in PCK by 0.09823, when controlling for other
variables. Lastly, there was an interaction between student gender and teachers’ self-
efficacy beliefs in CK. The slope of the interaction between student gender and
teachers’ self-efficacy beliefs in CK 𝛽21 = 0.11747 and it was significant with t (1) = 2.72,
p = .0066. As male students were coded as 0 (reference level) and female students
were coded as 1, the slope of teachers’ self-efficacy in CK for female students (one
level increase in student gender) was 0.11747 higher than the slope for male students.
Table 4-44. Multiple regression model summary
Variable DF Parameter Estimate
Standard Error
t Value p
Intercept (𝛽0) 1 2.44112 0.01592 153.37 <.0001
Pre_Career (𝛽1) 1 0.62555 0.02190 28.56 <.0001
Gender_Student (𝛽2) 1 -0.02974 0.02197 -1.35 0.1761
Printing_Level (𝛽3) 1 -0.01301 0.01719 -0.76 0.4493
STEM_Level (𝛽4) 1 -0.00483 0.03511 -0.14 0.8905
Pedagogical_Beliefs (𝛽5) 1 -0.03610 0.04150 -0.87 0.3845
Interest_Teacher (𝛽6) 1 -0.00008138 0.04270 -0.00 0.9985
Importance_Teacher (𝛽7) 1 0.01843 0.04788 0.39 0.7003
Usefulness_Teacher (𝛽8) 1 0.04856 0.06208 0.78 0.4342
Self_Efficacy_TK (𝛽9) 1 -0.01088 0.03313 -0.33 0.7425
182
Table 4-44. Continued
Variable DF Parameter Estimate
Standard Error
t Value p
Self_Efficacy_PK (𝛽10) 1 -0.04413 0.04467 -0.99 0.3234
Self_Efficacy_CK (𝛽11) 1 -0.00624 0.04671 -0.13 0.8937
Self_Efficacy_TPK (𝛽12) 1 -0.02654 0.05755 -0.46 0.6447
Self_Efficacy_TCK (𝛽13) 1 -0.00334 0.03920 -0.09 0.9322
Self_Efficacy_PCK (𝛽14) 1 -0.01669 0.03931 -0.42 0.6713
Self_Efficacy_TPACK (𝛽15) 1 0.05855 0.07112 0.82 0.4106
Pre_Career*Printing_Level (𝛽16) 1 -0.06581 0.02918 -2.26 0.0243
Pre_Career*STEM_Level (𝛽17) 1 0.10874 0.05252 2.07 0.0386
Pre_Career*Pedagogical_Beliefs (𝛽18) 1 0.14733 0.05220 2.82 0.0048
Pre_Career*Usefulness_Teacher (𝛽19) 1 -0.12311 0.05057 -2.43 0.0150
Pre_Career*Self_Efficacy_PCK (𝛽20) 1 0.09823 0.04275 2.30 0.0217
Gender_Student* Self_Efficacy_CK (𝛽21) 1 0.11747 0.04315 2.72 0.0066
Summary of Results
To provide an overview of the results, the main effects and interactions between
student and teacher variables for each student outcome are provided in Table 4-45. All
the interactions between student variables and teacher variables are organized in Table
4-46 and Table 4-47. As student gender and pretest scores were treated as covariates
and the emphasis was on the relationship between teacher variables and student
outcome variables, the summary of results and the interpretations focused on the main
effects of teacher variables and the interactions between student variables and teacher
variables. The specific results and interpretations with statistics can be referred to in the
results sections for each student outcome variable.
183
Teachers’ 3D printing integration level was not a significant predictor for any of
the student outcomes, however, it had a negative interaction with students’ prior interest
in STEM careers. Therefore, 3D printing integration level had a more positive effect on
students with lower prior interest in STEM careers. Teachers’ STEM integration level
was a positive predictor for students’ math motivation, indicating that when teachers’
STEM integration level increased, their students’ math motivation also increased.
Teachers’ STEM integration level also had a positive interaction with students’ prior
interest in STEM careers and a positive interaction with student gender in terms of
students’ science motivation. Therefore, teachers’ STEM integration level had a more
positive effect on students with higher prior interest in STEM careers and STEM
integration level had a more positive effect on female students’ science motivation.
Although teachers’ pedagogical beliefs did not significantly predict any of the
student outcomes, it negatively interacted with students’ prior science motivation,
positively interacted with students’ prior interest in STEM careers, and positively
interacted with student gender in terms of students’ 21st century skills. Teachers’
pedagogical beliefs had a more positive effect for students with lower prior science
motivation, students with higher prior interest in STEM careers, and female students in
terms of 21st century skills.
Teachers’ interest in and perceived importance of 3D printing integration were
not significant predictors for any of the student outcomes. However, teachers’ perceived
usefulness of 3D printing integration negatively predicted students’ 21st century skills
and it negatively predicted students’ science motivation with approaching significance.
Therefore, when teachers’ perceived usefulness of 3D printing integration increased,
184
students’ 21st century skills decreased, and students’ science motivation decreased
(approaching significance). Teachers’ perceived usefulness of 3D printing integration
was a negative predictor for students’ math motivation, and it had a negative interaction
with students’ prior math motivation, indicating that when teachers’ perceived
usefulness of 3D printing integration increased, students’ math motivation increased if
students’ prior math motivation was at the grand mean, and teachers’ perceived
usefulness of 3D printing integration had a more positive effect on students with lower
prior math motivation. Teachers’ perceived usefulness of 3D printing integration also
had a negative interaction with students’ prior interest in STEM careers, indicating that
teachers’ perceived usefulness of 3D printing integration had a more positive effect on
students with lower prior interest in STEM careers. Lastly, teachers’ perceived
importance of 3D printing integration had a positive interaction with student gender in
terms of students’ technology/engineering motivation. Therefore, teachers’ perceived
importance of 3D printing integration had a more positive effect on female students in
terms of their technology/engineering motivation.
Teachers’ self-efficacy in PCK positively predicted students’ math motivation with
approaching significance while teachers’ self-efficacy in TK negatively predicted
students’ technology/engineering motivation with approaching significance, indicating
when teachers’ self-efficacy in PCK increased, students’ math motivation increased
(approaching significance), however, when teachers’ self-efficacy in TK increased,
students’ technology/engineering motivation decreased (approaching significance). In
addition to these main effects, there were varied interactions between student variables
(pretest scores and student gender) and teachers’ self-efficacy beliefs.
185
With regard to the interactions with students’ pretest scores, students’ prior 21st
century skills had a positive interaction with teachers’ self-efficacy in CK, students’ prior
math motivation had a positive interaction with teachers’ self-efficacy in TPK, and
students’ prior interest in STEM careers had a positive interaction with teachers’ self-
efficacy in PCK. Therefore, teachers’ self-efficacy in CK had a more positive effect on
students with higher prior 21st century skills, teachers’ self-efficacy in TPK had a more
positive effect on students with higher prior math motivation, and teachers’ self-efficacy
in PCK had a more positive effect on students with higher prior interest in STEM
careers. However, students’ prior math motivation had a negative interaction with
teachers’ self-efficacy in PK, indicating teachers’ self-efficacy in PK had a more positive
effect on students with lower prior math motivation.
Student gender and teachers’ self-efficacy beliefs also had varied interactions.
Teachers’ self-efficacy in PK positively predicted students’ technology/engineering
motivation (coefficient = 0.03467), however, student gender and teachers’ self-efficacy
in PK had a negative interaction (coefficient = -0.1452), indicating teachers’ self-efficacy
in PK had a more positive effect on male students. After accounting for the negative
interaction, the coefficient for male students was 0.03467 and the coefficient for female
students was -0.11053. Therefore, teachers’ self-efficacy in PK positively predicted
male students’ technology/engineering motivation but negatively predicted female
students’ technology/engineering motivation, indicating when teachers’ self-efficacy in
PK increased, male students’ technology/engineering motivation increased, however,
female students’ technology/engineering motivation decreased. The interaction between
student gender and teachers’ self-efficacy in PK was also negative for students’ math
186
motivation, indicating teachers’ self-efficacy in PK had a more positive effect on male
students. Although teachers’ self-efficacy in PK was a positive predictor for male
students’ technology/engineering motivation and had a more positive effect on male
students’ math motivation, teachers’ self-efficacy in CK had a more positive effect on
female students’ interest in STEM careers since there was a positive interaction
between teachers’ self-efficacy in CK and student gender.
Table 4-45. Summary of results for student outcomes Student outcome
Main effects Interactions
Science motivation
Teachers’ perceived usefulness of 3D printing integration was a negative predictor with approaching significance. When teachers’ perceived usefulness of 3D printing integration increased, students’ science motivation decreased (approaching significance).
Students’ science motivation pretest scores and teachers’ pedagogical beliefs had a negative interaction. Teachers’ pedagogical beliefs had a more positive effect on students with lower prior science motivation.
Other predictors were non-significant. Student gender and teachers’ STEM integration level had a positive interaction. Teachers’ STEM integration level had a more positive effect on female students.
Technology/ Engineering motivation
Teachers’ self-efficacy in TK was a negative predictor with approaching significance. When teachers’ self-efficacy in TK increased, students’ technology/engineering motivation increased.
Student gender and teachers’ perceived importance of 3D printing integration had a positive interaction. Teachers’ perceived importance of 3D printing integration had a more positive effect on female students.
Teachers’ self-efficacy in PK was a positive predictor when student gender was male. When teachers’ self-efficacy in PK increased, male students’ technology/engineering motivation increased.
Student gender and teachers’ self-efficacy in PK had a negative interaction. Teachers’ self-efficacy in PK had a more positive effect on male students.
Other predictors were non-significant.
187
Table 4-45. Continued Student outcome
Main effects Interactions
Math motivation
STEM integration level was a positive predictor. When teachers’ STEM integration level increased, students’ math motivation increased.
Students’ math motivation pretest scores and teachers’ perceived usefulness of 3D printing integration had a negative interaction. Teachers’ perceived usefulness of 3D printing integration had a more positive effect on students with lower prior math motivation.
Teachers’ perceived usefulness of 3D printing integration was a significant and negative predictor when students’ math motivation pretest scores were at the grand mean. When teachers’ perceived usefulness of 3D printing integration increased, students’ math motivation increased when students’ math motivation pretest scores were at the grand mean.
Students’ math motivation pretest scores and teachers’ self-efficacy in PK had a negative interaction. Teachers’ self-efficacy in PK had a more positive effect on students with lower prior math motivation.
Teachers’ self-efficacy in PCK was a positive predictor with approaching significance. When teachers’ self-efficacy in PCK increased, students’ math motivation also increased (approaching significance).
Students’ math motivation pretest scores and teachers’ self-efficacy in TPK had a positive interaction. Teachers’ self-efficacy in TPK had a more positive effect on students with higher prior math motivation.
Other predictors were non-significant.
Student gender and teachers’ self-efficacy in PK had a negative interaction. Teachers’ self-efficacy in PK had a more positive effect on male students.
21st century skills
Teachers’ perceived usefulness of 3D printing integration was a negative predictor. When teachers’ perceived usefulness of 3D printing integration increased, students’ 21st century skills decreased.
Students’ 21st century skills pretest scores and teachers’ self-efficacy in CK had a positive interaction. Teachers’ self-efficacy in CK had a more positive effect on students with higher prior 21st century skills.
Other predictors were non-significant. Student gender and teachers’ pedagogical beliefs had a positive interaction. Teachers’ pedagogical beliefs had a more positive effect on female students.
188
Table 4-45. Continued Student outcome
Main effects Interactions
Interest in STEM careers
None of the predictors were significant.
Students’ interest in STEM careers pretest scores had negative interactions with teachers’ 3D printing integration levels and perceived usefulness of 3D printing integration. Teachers’ 3D printing integration levels and perceived usefulness of 3D printing integration had a more positive effect on students with lower prior interest in STEM careers.
Students’ interest in STEM careers pretest scores had positive interactions with teachers’ STEM integration levels, pedagogical beliefs, and self-efficacy in PCK. Teachers’ STEM integration levels, pedagogical beliefs, and self-efficacy in PCK had a more positive effect on students with higher prior interest in STEM careers.
Student gender and teachers’ self-efficacy in CK had a positive interaction. Teachers’ self-efficacy in CK had a positive effect on female students.
Note: Male students were coded as 0 and female students were coded as 1.
Table 4-46. Interactions between student pretest scores and teacher variables
Pre_Science Pre_TechEngi Pre_Math Pre_21st Pre_Career
Printing_Level -0.06581 (.0243)
STEM_Level 0.10874 (.0386)
Pedagogical_Beliefs -0.1291 (.0030)
0.14733 (.0048)
Importance_Teacher
Usefulness_Teacher -0.1199
(.0016)
-0.12311
(.0150)
Self_Efficacy_PK -0.08292
(.0298)
189
Table 4-46. Continued
Pre_Science Pre_TechEngi Pre_Math Pre_21st Pre_Career
Self_Efficacy_CK 0.08887
(.0380)
Self_Efficacy_TPK 0.1096
(.0039)
Self_Efficacy_PCK 0.09823
(.0217)
Note: The number in each cell is the interaction coefficient with the p value in the bracket.
Table 4-47. Interactions between student gender and teacher variables
Gender
(Science)
Gender
(TechEngi)
Gender
(Math)
Gender
(21st)
Gender
(Career)
Printing_Level
STEM_Level 0.09404
(.0133)
Pedagogical_Beliefs 0.1186
(.0186)
Importance_Teacher 0.1189
(.0191)
Usefulness_Teacher
Self_Efficacy_PK -0.1452
(.0292)
-0.2276
(.0002)
Self_Efficacy_CK 0.11747
(.0066)
Self_Efficacy_TPK
Self_Efficacy_PCK
Note: Gender (Science), Gender (TechEngi), Gender (Math), Gender (21st), and Gender (Career) indicate interactions between student gender and teacher variables for science motivation, technology/engineering motivation, math motivation, 21st century skills, and interest in STEM careers. The number in each cell is the interaction coefficient with the p value in the bracket.
190
CHAPTER 5 DISCUSSIONS
The purpose of this study was to examine the relationship between teachers’
beliefs, their 3D printing integration in science classrooms, and students’ STEM
motivation, 21st century skills, and interest in STEM careers. The research questions
were:
How are teachers’ beliefs correlated with their 3D printing technology integration in the science classrooms?
How do teachers’ beliefs and their 3D printing technology integration in the science classrooms predict students’ STEM motivation, 21st century skills, and interest in STEM careers?
This chapter begins with reviewing the limitations and delimitations of this study
and then the results are interpreted. First, the relationships between teacher beliefs and
3D printing integration are discussed, and then the relationships between teacher
beliefs, 3D printing integration, and student outcome variables are discussed. After
discussing the results for each of the student outcome variables, the themes and
patterns are discussed. Finally, implications for practice and future research are
proposed.
Limitations and Delimitations
Like many other studies, this study had a few limitations and delimitations that
have to be clarified. The limitations were potential issues that I was not able to address
in this study and the delimitations elucidated the boundaries of this study and the
situations or areas this study may not apply.
Limitations
First, this study analyzed teachers’ 3D printing technology integration through
their lesson plans without classroom observation. Lesson plan analysis may not exactly
191
reflect teachers’ actual implementation. There might be some discrepancy between the
lesson plans and the actual classes. Moreover, teachers may make adjustments as they
progressed with different classes and students may have had similar classes but with
some differences. Therefore, the lesson plans may not have completely captured these
nuances.
Second, most of the lessons were short; usually a few classes with the
implementation of 3D printing technology. Students’ learning outcomes including their
STEM motivation, 21st century skills, and interest in STEM careers may not have
significant change with short interventions. Moreover, the S-STEM survey with 5-point
or 4-point Likert scales may not be sensitive enough to detect the changes in students’
learning outcomes associated with the 3D printing technology activities even if there
were trivial changes.
Third, the 3D printing integration levels defined and coded in this study may not
have been sensitive enough to capture all possible differences among various
implementations of 3D printing technology in the science classrooms. The 3D printing
integration level was an important predictor but may not reflect all aspects that can
potentially influence students.
Finally, the teacher beliefs survey was administered after teachers’ 3D printing
technology integration in their classrooms. Therefore, the correlation between teacher
beliefs and their 3D printing technology integration practice can only account for the
relationship with teacher beliefs after the 3D printing integration. Although teacher
beliefs may not change significantly in a short time, the beliefs may have evolved to
192
some extent during their 3D printing integration. It was unknown how teacher beliefs
prior to or during the integration would have influenced their 3D printing integration.
Delimitations
First, this study focused on 3D printing integration in K-12 science classrooms
within the context of paleontology. It may not account for 3D printing technology
integration in other disciplines or in contexts other than paleontology.
Second, teacher beliefs only included pedagogical beliefs, self-efficacy in
technology integration, and technology value beliefs, which are salient factors that may
influence teachers’ technology integration practice. Some more fundamental beliefs
such as epistemological beliefs were not included in this study because teachers’
epistemological beliefs are beliefs about the nature of knowledge and learning, which
are correlated with teachers’ pedagogical beliefs but are not directly correlated with their
technology integration practice (Kim et al., 2013). Therefore, this study cannot account
for the relationship between teachers’ 3D printing integration and their epistemological
beliefs, which may potentially have relationships with their technology integration
practice.
Third, this study was conducted with teachers who voluntarily participated in the
iDigFossils project. It was assumed that these teachers were interested in integrating
3D printing technology integration in their science classrooms to some extent, as part of
the reason they participated in the project. Thus, the findings may not apply to teachers
who are not interested at all in 3D printing technology integration.
Lastly, this study utilized correlational and multilevel modeling analysis to
examine the relationship between teacher beliefs, 3D printing technology integration,
and students’ STEM motivation, interest in STEM careers, and 21st century skills.
193
These types of analyses could not determine causal relationships. Therefore, this study
accounted for the correlational or regressional relationship between the variables but
could not demonstrate the influence of teacher beliefs on 3D printing integration, and
the influence of 3D printing integration on student learning outcomes.
Relationships between Teacher Beliefs and 3D Printing Integration
Surprisingly, there were no significant correlations between teacher beliefs and
their 3D printing integration level and STEM integration level except that teachers’ self-
efficacy in PCK significantly and moderately correlated with STEM integration level in a
negative direction. This result countered previous research which indicated important
relationships between teacher beliefs and their technology integration. Tondeur et al.
(2017) suggested that teachers with constructivist beliefs are more likely to integrate
technology to facilitate student-centered learning. In addition, teachers’ self-efficacy is a
key predictor of their technology integration in the classrooms (Albion, 1999; Ertmer &
Ottenbreit-Leftwich, 2010; Gonzales, 2013; Haight, 2011; Heineman, 2018; Li et al.,
2018; Manglicmot, 2015; Marcinkiewicz, 1994; Tweed, 2013). Moreover, teachers’
technology value beliefs are critical for teachers to determine whether and how they will
integrate technology in the classrooms (Ertmer et al., 1999; Ertmer et al., 2012; Ertmer
& Ottenbreit-Leftwich, 2010; Mueller et al., 2008; Vongkulluksn et al., 2018; Wozney et
al., 2006).
The discrepancy between this result and previous research findings was
probably due to the barriers that teachers encountered when integrating 3D printing into
their science classrooms. As Ertmer et al. (2012) indicated, external barriers such as
lack of resources can influence how teachers actually integrate technology, thus
teachers’ technology integration may not be consistent with their beliefs. As reported in
194
the open responses of the teacher beliefs survey in this study, the teachers
encountered a few barriers including logistical and technical issues, an insufficient
number of printers and related resources, the lack of time, and constraints in the
curriculum which made it difficult to fit 3D printing integrated lessons into the current
curriculum. These barriers may have deviated teachers from how they would have
actually wanted to integrate 3D printing in their science classrooms.
It is also necessary to note that the duration of the teachers’ 3D printing
integration was relatively short (mostly a few classes) compared to the durations of
many of the previous studies on teacher beliefs and technology integration in which the
duration of integration was from a semester to a few years (e.g., Ertmer et al., 2012;
Ertmer & Ottenbreit-Leftwich, 2010; Tondeur et al., 2017; Vongkulluksn et al., 2018).
Teacher beliefs and technology integration may be more aligned with each other after a
long time of teaching practice, whereas the correlation between teacher beliefs and
technology integration may not emerge within a short period of time integrating a new
technology (3D printing) in a new context such as paleontology.
Relationships between Teacher Variables and Student Outcomes
In this section, the results of the relationship between teacher variables and
student outcome variables including math motivation, technology/engineering
motivation, 21st century skills, and interest in STEM careers respectively were
discussed. Teacher variables included teachers’ 3D printing integration levels, STEM
integration levels, and teacher beliefs. Teacher beliefs consisted of pedagogical beliefs,
value beliefs which consisted of teachers’ interest in, perceived importance, and
perceived usefulness of 3D printing integration, and self-efficacy beliefs in 3D printing
195
integration, which was operationalized as self-efficacy in TK, PK, CK, TPC, TCK, PCK,
and TPACK.
Relationships with Science Motivation
It was surprising that none of the 3D printing integration level and STEM
integration level had significant relationships with students’ science motivation. This was
similar to Swayne’s (2017) finding that technology integration levels did not have a
statistically significant relationship with student engagement in middle-school English
Language/Arts and mathematics classrooms that used 1:1 tablet technology. Previous
research indicated that technology integration in science classes can increase students’
science motivation (e.g., Xie & Reider, 2014) and STEM integration can also enhance
student motivation (Honey, 2014), but there was little empirical evidence about how
different 3D printing integration levels or STEM integration levels may influence student
motivation. Since 3D printing was integrated into science classes, it was assumed that
different levels of 3D printing integration and/or STEM integration may have different
influences on students’ science motivation. This may be due to the short duration of 3D
printing integration as most of the teachers only integrated 3D printing for a few classes.
The effect of 3D printing integration may need a longer time to emerge.
It was also possible that a higher level of 3D printing integration and/or STEM
integration may benefit students but may not always be good for students. A higher
level of STEM integration requires students’ sufficient knowledge in the individual
subjects in order to successfully connect the concepts and ideas across STEM
disciplines (Pearson, 2017). With regards to the non-significant relationship between 3D
printing integration levels and students’ science motivation, similar to what Nemorin and
Selwyn (2017) found in their study, students may encounter difficulties when using 3D
196
printing software to create models, which may make them feel frustrated. As the
interaction between teachers and students, the problems encountered, and students’
responses could not be identified through the lesson plans, future studies are highly
recommended to conduct classroom observations to obtain more comprehensive data.
Although teachers’ STEM integration level was not a significant predictor, it had a
positive interaction with student gender. For both male students (student gender = 0)
and female students (student gender = 1), STEM integration level had a positive
relationship with student gender and STEM integration level had a stronger effect on
female students’ science motivation. This finding is interesting as research suggested
that male students are typically more interested and motivated to learn STEM than
female students (Wang & Degol, 2013). Although student gender was not a significant
predictor of students’ science motivation in this study, the interaction suggested that as
STEM integration level increases, female students’ science motivation may have higher
increase than male students.
In addition, it was found that teachers’ perceived usefulness of 3D printing
integration had a negative relationship with students’ science motivation with
approaching significance. When teachers’ perceived usefulness of 3D printing
integration increased, students’ science motivation decreased. This result was
unexpected, and it countered previous research that teachers’ technology value beliefs
may positively correlate with their technology integration (Ottenbreit-Leftwich et al.,
2010), which may further have positive influences on student motivation. It is possible
that even if a teacher perceived 3D printing integration as useful, he or she may not be
able to integrate 3D printing in effective ways, and external barriers can impact how
197
they integrate, thus students’ science motivation may not increase or even decrease.
This finding suggested that although teachers’ perceived usefulness of 3D printing may
potentially have positive impacts on students’ science motivation, the ways teachers
actually integrate 3D printing to facilitate science learning is probably more important.
None of teachers’ pedagogical beliefs and self-efficacy beliefs were significant
predictors of students’ science motivation. Although teacher beliefs may be associated
with students’ learning motivation, many external barriers can impact the actual
technology integration, which eventually impact student motivation. The barriers include
but not limited to external barriers such as the lack of resources or insufficient of time
and technical support (Ertmer et al., 2012), which were also demonstrated in the
teachers’ open responses on the challenges they encountered when integrating 3D
printing into their science classrooms.
Although no significant relationship between teachers’ pedagogical beliefs and
students’ science motivation was identified, there was a negative interaction between
students’ prior science motivation and teachers’ pedagogical beliefs. For students with
higher prior science motivation, teachers’ pedagogical beliefs had less effect, but for
students with lower prior science motivation, teachers’ pedagogical beliefs had a
stronger effect. Similar to what Ross (1994) suggested that teachers with higher self-
efficacy attend to the lower ability students more closely, and lower ability students
typically have lower motivation, teachers with higher pedagogical beliefs may be more
student-centered and pay more attention to students with lower motivation when they
teach. This finding suggested that in order to increase students’ science motivation
198
especially students with lower prior science motivation, teachers probably need a higher
level of beliefs in student-centered learning.
Relationships with Technology/Engineering Motivation
It was surprising that 3D printing integration level was not a significant predictor
for students’ technology/engineering motivation, which countered the general findings of
previous research. Previous research indicated that technology integration has positive
impacts on student motivation (e.g., Francis, 2017; Heafner, 2004; Xie & Reider, 2014).
However, there was little research on how different levels of technology integration
influence student motivation. The non-significant effect may due to that the 3D printing
integration levels may not be sensitive enough to account for all the differences in
teachers’ implementation of 3D printing in the classrooms. Higher levels of 3D printing
integration might be beneficial for students, but the higher levels have more demand on
students’ knowledge and skills, which might make some students feel frustrated. In
addition, teachers’ 3D printing integration levels negatively interacted with students’
prior interest in STEM careers, indicating higher 3D printing integration levels may have
more influence on students with lower prior interest in STEM careers. This may due to
that higher 3D printing integration levels engage students in deeper learning with 3D
printing technology, which may increase students’ interest in STEM careers especially
for students with lower prior interest.
For students’ technology/engineering motivation, teachers’ self-efficacy in PK
was a significant predictor and it had a negative interaction with student gender.
Teachers’ self-efficacy in PK positively predicted both male students’ and female
students’ technology/engineering motivation, which was consistent with previous
research that teachers’ self-efficacy beliefs have a positive relationship with student
199
motivation (Zee & Koomen, 2016). Furthermore, teachers’ self-efficacy in PK had a
stronger effect on male students’ technology/engineering motivation. One explanation
was that teachers with higher self-efficacy in PK may implement more student-centered
learning activities and male students may be easier to be engaged in the student-
centered learning activities as male students are typically more motivated to learn
technology/engineering (Wang & Degol, 2013).
Although teachers’ self-efficacy in PK was a positive predictor, teachers’ self-
efficacy in TK was a negative predictor with approaching significance for students’
technology/engineering motivation. These findings suggested that teachers’ self-efficacy
in PK instead of TK was of more importance to potentially increase students’
technology/engineering motivation. As previous research on the relationship between
teachers’ self-efficacy and student learning outcomes focused on teachers’ general
teaching self-efficacy and there was little research on how teachers’ self-efficacy in PK
and TK may have different correlations on student motivation, the findings can only be
conjectured without an adequate empirical or theoretical base. One explanation was
that teachers’ self-efficacy in TK may not be consistent with their actual integration of
3D printing in the classrooms due to some external barriers, thus may not necessarily
have positive relationships with students’ technology/engineering motivation.
In addition to teachers’ self-efficacy, it was also found that student gender was a
significant predictor and male students had higher technology/engineering motivation
than female students when teachers’ perceived importance and self-efficacy in PK were
at the average scores of all the teachers. However, the direction of the significance of
gender was conditional and gender could be a negative or positive predictor when
200
teachers’ perceived importance of 3D printing integration and self-efficacy in PK
change. Since the direction of the significance of gender was conditional, it was
undetermined whether student gender was a positive or negative predictor in general,
thus it was unknown whether male or female students had higher
technology/engineering motivation. Although research indicated male students are more
motivated to learn STEM and may have higher STEM motivation (Wang & Degol, 2013),
the findings on the relationship between student gender and their
technology/engineering motivation cannot confirm the findings of previous research.
There might be other factors that were more influential on students’
technology/engineering motivation.
Relationships with Math Motivation
It was found that teachers’ STEM integration level significantly and positively
predicted students’ math motivation while controlling for other variables. In the science
classrooms, teachers’ integrated science, 3D printing technology, math, and
engineering at different levels. All of the teachers integrated 3D printing technology in
their science classrooms, but some teachers also integrated math or both math and
engineering with 3D printing integrated science classes. This result suggested that
when math was integrated with other subjects in STEM, higher STEM integration level
contributed to students’ math motivation. This finding was consistent with previous
research that in general integrated curriculum or instruction can increase students’
learning motivation (Bragow, Gragow & Smith, 1995; Gutherie, Wigfield, & VonSecker,
2000), and specifically, STEM integration can enhance students’ interest in STEM
(Mustafa et al., 2016; Riskowski, Todd, Wee, Dark, & Harbor, 2009) and STEM learning
motivation (Laboy-Rush, 2011; Wang, Moore, Roehrig, & Park, 2011).
201
Integrating math with other STEM subjects enabled students to learn math in
more interesting and connected ways. Students may not even realize they were doing
math while engaging in science learning activities supported by technologies. The
integration of math and other STEM subjects also allowed students to make
connections between different subjects which may potentially increase their knowledge
and skills in math and make them feel more confident and motivated to learn math. As
Furner and Kumar (2007) stated, STEM integration makes learning “more relevant, less
fragmented, and more stimulating experiences for learners” (p.186), which potentially
increases students’ STEM motivation. Although math has been a challenging subject for
many students and math anxiety has been a long-lasting educational issue (Ashcraft &
Ridley, 2005; Maloney & Beilock, 2012; Ramirez, Gunderson, Levine, & Beilock, 2013;
Wigfield & Meece, 1988) that can decrease students’ motivation to learn math, the
finding of this study was encouraging that integrating math with other STEM subjects
might be helpful to increase students’ motivation to learn math.
Although STEM integration level was a significant predictor of students’ math
motivation, this study did not find a significant relationship between teachers’ 3D printing
integration level and students’ math motivation. The coding of teachers’ 3D printing
integration level did not specifically include the component of math learning, so
teachers’ 3D printing integration levels may not have a significant relationship with
students’ math motivation. This finding suggested that students’ math motivation may
not have a direct relationship with the integration of just technology in science classes
and math has to be specifically designed and integrated with other STEM subjects in
order to increase students’ math motivation.
202
This study found that teachers’ pedagogical beliefs were not significant for
students’ math motivation. Teachers’ pedagogical beliefs may not necessarily determine
how teachers integrate technology and there might be some other factors that impact
how teachers actually taught such as limited resources and time (Ertmer et al., 2012).
As corroborated by the open responses in the teacher beliefs survey in this study, the
lack of 3D printers and related resources, the large amount of time required to print 3D
objects, and the time required to integrate 3D printing technologies into the current
curriculum were some of the major challenges encountered by the teachers. Although
teachers may have intended to integrate 3D printing technologies to facilitate student-
centered learning, the external barriers could have impacted how they actually
integrated 3D printing technologies into their classes. Thus, it was not surprising that
teachers’ pedagogical beliefs did not have a significant relationship with students’ math
motivation.
Among teachers’ value beliefs, teachers’ perceived usefulness of 3D printing
integration significantly but negatively predicted students’ math motivation when
students’ math motivation pretest scores were equal to or higher than the average score
of all students’ pretest scores. For students whose prior math motivation was higher
than the average, when teachers’ perceived usefulness was higher, students’ math
motivation was lower. However, because students’ math motivation pretest scores
negatively interacted with teachers’ perceived usefulness of 3D printing integration, for
students who had lower prior math motivation, the effect of teachers’ perceived
usefulness of 3D printing integration was stronger; for students who had higher prior
math motivation, the effect of teachers’ perceived usefulness of 3D printing integration
203
was weaker. It suggested that increasing teachers’ perceived usefulness of 3D printing
integration may be more beneficial for students who had relatively lower prior math
motivation. Although research indicated that teachers’ technology value beliefs are
critical factors for their technology integration, and teachers with high value beliefs even
tend to integrate the technology when faced with external barriers (Ertmer et al., 2012;
Ottenbreit-Leftwich et al., 2010; Snoeyink & Ertmer, 2001), there was little evidence on
the relationship between teachers’ technology value beliefs and student motivation, and
even little was research on how teachers’ technology value beliefs impact students with
individual differences such as prior math motivation. Future research may further
investigate the relationship between teachers’ technology value beliefs and how and
why teachers’ value beliefs may be more influential on students with lower prior
motivation.
Most aspects of teachers’ self-efficacy beliefs did not have significant
relationships with students’ math motivation, however, teachers’ self-efficacy beliefs in
PCK was a positive predictor approaching significance. The higher teachers’ self-
efficacy beliefs in PCK, the stronger students’ math motivation. Although there was little
literature regarding the relationship between teachers’ self-efficacy in PCK and
students’ learning motivation, a meta-analysis study conducted by Zee and Koomen
(2016) indicated that teachers’ self-efficacy has positive relationships with students’
academic achievement and motivation as shown by all the qualified studies identified by
the authors. Teachers’ self-efficacy not only influenced students' learning performance
(e.g., Ashton & Webb, 1986; Brookover et al., 1979; Brophy & Evertson, 1977; Hoy &
Davis, 2005; Shahzad & Naureen, 2017) but also students’ learning motivation (e.g.,
204
Eccles & Wigfield, 1985; Lazarides et al., 2018; Mojavezi & Tamiz, 2012; Pan, 2014;
Schiefele & Schaffner, 2015). The finding of this study suggested that increasing
teachers’ self-efficacy in PCK may potentially increase students’ math motivation.
This study also found that student gender negatively interacted with teachers’
self-efficacy beliefs in PK, indicating teachers’ self-efficacy beliefs in PK had a stronger
effect on male students and a lesser effect on female students. Additionally, students’
math motivation pretest scores had a negative interaction with teachers’ self-efficacy
beliefs in PK but a positive interaction with teachers’ self-efficacy beliefs in TPK, which
indicated that for students whose prior math motivation was higher, teachers’ self-
efficacy beliefs in PK had less effect but teachers’ self-efficacy beliefs in TPK had
stronger effect, comparing to students whose prior math motivation was lower, and vice
versa.
It suggested that teachers’ self-efficacy beliefs in PK may have more influence on
students with lower prior math motivation and teachers’ self-efficacy beliefs in TPK may
have more influence on students with higher prior math motivation. Although there was
no direct causal relationship between teachers’ self-efficacy beliefs and students’ math
motivation, teachers’ self-efficacy beliefs may influence how they teach and even the
learning environment in the classrooms. This finding suggested that for students with
relatively low prior math motivation, teachers probably need to focus more on the
pedagogy they use to teach math; for students with relatively high prior math motivation,
teachers probably need to focus more on effective ways that they use technology to
teach math.
205
Relationships with 21st Century Skills
In this study, 3D printing integration level and STEM integration level were not
significant predictors for students’ 21st century skills either, which was not consistent
with previous research. In Trust and Maloy’s (2017) study with teachers who integrated
3D printing into their classrooms, the teachers reported that 3D printing integration
enhanced their students’ creativity, technology literacy, problem-solving, self-directed
learning, critical thinking, and perseverance, essential 21st century skills as the authors
summarized. However, the findings in Trust and Maloy’s (2017) study were from
teachers’ report, which may not objectively reflect the real influence of 3D printing
integration. Nevertheless, research suggested that STEM integration can also
contribute to students’ 21st century skills (Honey et al., 2014). The non-significant
effects of 3D printing and STEM integration levels in this study were probably due to
that teachers’ 3D printing and STEM integration levels may not necessarily influence all
aspects included in the 21st century skills scale developed by Unfried et al. (2015), such
as items on students’ self-regulation in learning and leadership.
Consistent with the relationship between teachers’ perceived usefulness of 3D
printing integration and students’ science motivation, teachers’ perceived usefulness of
3D printing integration was also a negative predictor for students’ 21st century skills.
This finding was unexpected because previous research indicated that teachers’
technology value beliefs are critical for them to effectively integrate technology
(Ottenbreit-Leftwich et al., 2010), which may positively influence students. The negative
relationship between teachers’ perceived usefulness of 3D printing technology and
students’ 21st century skills may due to the limitations of the measurement. The rating
scale for 21st century skills developed by Unfried et al. (2015) included some
206
components that may not be influenced by teacher beliefs or teachers’ 3D printing
integration. For instance, the rating scale included items such as “I am confident I can
manage my time wisely when working on my own”, “When I have many assignments, I
can choose which ones need to be done first”, etc. These components may not be
influenced by teacher beliefs or how teachers integrate 3D printing technology.
Student gender was a significant predictor for students’ 21st century skills and
there was a positive interaction between student gender and teachers’ pedagogical
beliefs. When teachers’ pedagogical beliefs were average or above average, student
gender had a positive effect on students’ 21st century skills. It suggested that when
teachers’ pedagogical beliefs were relatively high, female students had higher 21st
century skills than male students. In the rating scale of students’ 21st century skills,
there were a few items regarding students’ self-regulation. For instance, “I am confident
I can set my own learning goals”, “I am confident I can manage my time wisely when
working on my own”, and “When I have many assignments, I can choose which ones
need to be done first”. Research indicated that female students have better self-
regulation than male students (Matthews, Ponitz, & Morrison, 2009). The items on self-
regulation might have contributed to female students’ higher ratings on the scale of the
21st century skills. However, there was little empirical evidence or theoretical
foundations to explain why teachers’ pedagogical beliefs had more impact on female
students’ 21st century skills. It is possible that female students were more motivated to
learn when teachers were student-centered.
Although teachers’ self-efficacy in CK was not a significant predictor, there was a
positive interaction between students’ prior 21st century skills and teachers’ self-efficacy
207
beliefs in CK. Teachers’ self-efficacy beliefs in CK had stronger effects for students who
had higher prior 21st century skills and vice versa. It suggested that teachers’ self-
efficacy in CK was more influential and may become a significant predictor for students
with higher prior 21st century skills. There was little empirical evidence or theoretical
foundations to explain why teachers’ self-efficacy in CK had more influence on students
with higher prior 21st century skills. It is possible that students with higher prior 21st
century skills were more motivated to learn and have better learning skills, which may
help them benefit more from teachers with higher self-efficacy in CK who probably have
more content knowledge as well, thus to have more improvement in their 21st century
skills than students with lower 21st century skills.
Relationships with Interest in STEM Careers
Surprisingly, 3D printing integration and STEM integration level were both non-
significant for students’ interest in STEM careers. Although there was little research
regarding the influence of 3D printing integration on students’ interest in STEM careers,
Xie and Reider (2014) found the integration of innovative technologies can enhance
students’ motivation for science career, which is related to students’ interest in science
career. Honey et al. (2014) suggested integrated STEM education can facilitate
students’ interest development. However, there was little research on how different
levels of 3D printing integration and STEM integration may influence students’ interest
in STEM careers.
The non-significant findings on 3D printing integration level and STEM integration
level indicated that higher level of 3D printing integration level or STEM integration level
may not necessarily be better for students. As discussed previously, a higher level of 3D
printing integration might cause higher challenges on both teachers and students
208
(Nemorin & Selwyn, 2017) and higher level of STEM integration requires students to
have sufficient knowledge in each subject (Pearson, 2017), thus higher levels of 3D
printing integration and STEM integration may not contribute to students’ interest in
STEM careers. Additionally, teachers may have not made specific connections between
the 3D printing integration and STEM career pathways, so students may not make
connections between what they learned in the 3D printing integrated classes and the
possible future STEM careers, thus 3D printing integration and STEM integration levels
were not significant predictors for students’ interest in STEM careers.
In this study, none of the teacher variables were significant predictors of
students’ interest in STEM careers. However, there were a few interactions between
students’ prior interest in STEM careers and some of the teacher variables and also
interaction between student gender and teachers’ self-efficacy beliefs in CK. There were
positive interactions between students’ prior interest in STEM careers and teachers’
STEM integration levels, pedagogical beliefs, and self-efficacy beliefs in PCK
respectively. There was a negative interaction between students’ prior interest in STEM
careers and teachers’ 3D printing integration levels and a negative interaction between
students’ prior interest in STEM careers and teachers’ perceived usefulness of 3D
printing integration. There was a positive interaction between student gender and
teachers’ self-efficacy in CK.
These interactions suggested that teachers’ STEM integration levels,
pedagogical beliefs, and self-efficacy beliefs in PCK may have stronger positive
influences on students with higher prior interest in STEM careers. However, teachers’
3D printing integration levels and perceived usefulness of 3D printing integration may
209
have stronger positive influences on students with lower prior interest in STEM careers.
Lastly, teachers’ self-efficacy in CK may have stronger positive influences on female
students. Although previous research found correlations between teacher beliefs and
students’ affective learning outcomes (Zee & Koomen, 2016) in general, there was little
research on how teachers’ pedagogical beliefs, technology value beliefs including
interest, perceived importance, and perceived usefulness, and self-efficacy beliefs in
TK, PK, CK, TPK, TCK, PCK, and TPACK might have relationships with students’
affective learning outcomes including interest in STEM careers, and also how these
teacher beliefs may interact with student gender or students’ prior interest. There were
no empirical evidence or theoretical foundations to explain these findings and future
research would be necessary to further explore and explain these relationships.
Implications
Although extant literature suggested teacher beliefs can impact teachers’
educational practice and teacher beliefs can be significantly associated with students’
cognitive and affective learning outcomes, this study did not find significant relationships
between teacher beliefs and teachers’ 3D printing integration or relationships between
many aspects of teacher beliefs and student learning outcomes. This was probably due
to the challenges that teachers encountered as they reported in the open-ended
questions in the teacher beliefs survey and also the short duration of 3D printing
integration. In order for teachers to integrate 3D printing effectively, first, the challenges
have to be addressed. It may also take a longer time and more practice for teachers’
beliefs and their 3D printing integration to be aligned with each other. In addition, the
findings on the relationship between teachers’ 3D printing integration levels, STEM
210
integration levels, teacher beliefs, and student outcomes shed some light on future
research.
Implications for Practice
In this study, the teachers perceived 3D printing integration as beneficial for
students, but it was challenging for teachers to integrate 3D printing technology in their
science classrooms due to the external barriers and internal barriers reported in chapter
4. To facilitate teachers’ technology integration, it is necessary to reduce external
barriers by providing adequate funding, equitable access to recourses, and technical
support (ISTE, 2019). The teachers encountered some external barriers, including
logistical and technical issues, insufficient 3D printers and related resources, lack of
time to print 3D objects, and lack of time to plan, develop, and integrate 3D printing into
curriculum. Schools need to provide necessary resources and technical support.
Specifically, schools may provide more 3D printers and relevant resources so teachers
can print 3D objects more efficiently and also engage students in activities in smaller
groups. Technical support can help teachers save time on troubleshooting and focus
more on integrating 3D printing with the curriculum. Moreover, as the teachers had
insufficient time to plan, develop, and integrate 3D printing into the curriculum, schools
may provide more support to help teachers balance the current workload and the
initiative of integrating a new piece of technology into their teaching.
Besides the external barriers, the teachers also had some internal barriers
including the lack of ability to print 3D objects and connect 3D printing to curriculum
standards, and also to teach students who were not enthusiastic, motivated, or having
limited ability. It is essential to provide professional development and instructional
support to equip teachers with the ability to print 3D objects, connect 3D printing with
211
curriculum standards, and also develop skills in teaching students with individual
differences, such as varied enthusiasm, motivation, and ability. Schools may consider
providing professional development opportunities and instructional support for teachers
to learn how to use 3D printing technology, how to meaningfully connect 3D printing
with curriculum standards, and how to teach students with individual differences.
Successful 3D printing integration not only requires teachers being proficient with how
to use the technology, but also how to connect it to the curriculum, especially how to
integrate their technological knowledge, pedagogical knowledge, and content
knowledge to enhance their teaching and student learning outcomes.
Furthermore, schools may need to invest more on training teachers to effectively
integrate 3D printing into STEM disciplines. Although teachers may have abilities to
teach some individual STEM disciplines, they may not have knowledge and skills in all
the STEM disciplines that are involved when they design and implement STEM
integrated lessons. STEM integration could also be challenging even if teachers have
the necessary knowledge and skills in different STEM disciplines. STEM integration is
not just simply combining the different content area. Effective STEM integration requires
teachers to 1) be explicit about the goals and design STEM integrated lessons to
purposefully achieve the goals, 2) make STEM connections explicit to students through
scaffolding and opportunities to engage in activities in order to address the connected
ideas, and 3) keep learning goals in mind and pay attention to students’ learning
outcomes in individual STEM subjects (Honey et al., 2014). Additionally, schools may
build networking between teachers by establishing a community of support to facilitate
communication and collaboration between teachers in the same school and/or across
212
different schools. Schools may also partner with a local university to improve teachers’
3D printing integration and STEM integration through research and practice.
In addition to actively participating in professional training, teachers need to apply
what they have learned in their daily teaching. It is necessary for schools to provide
professional development opportunities for teachers to learn how to teach students with
individual differences, and it is critical that teachers taking the challenges and practicing
what they have learned. In this study, STEM integration level positively predicted
students’ math motivation but didn’t predict other student outcomes and STEM
integration level had different effects for male and female students and also students
with different prior interest in STEM careers. Teachers may consider students’ individual
differences such as student gender and prior interest when designing STEM integrated
lessons. Teachers’ technology integration level was not a significant predictor for any of
the student outcomes, but it had interaction with students’ prior interest in STEM
careers. Teachers may also consider students’ prior interest when they consider how to
integrate 3D printing in the classrooms.
Although providing necessary resources, technical support, professional
development, instructional support, and building network between teachers and schools
could be necessary for teachers to integrate 3D printing technologies in their science
classrooms, these efforts do not guarantee the teachers can integrate the technologies
without difficulties. In this study, the teachers were provided with 3D printers and
relevant resources, professional development, technical support, instructional support,
and network with the iDigFossils project team and other teachers, however, the
teachers still encountered many external barriers and internal barriers, which may have
213
limited and impacted their 3D printing integration in the science classrooms. Schools
and teachers have to be aware that integrating an emerging technology such as 3D
printing is not easy. It can take innumerous efforts of the schools and teachers to make
3D printing integration successful.
Although some external and internal barriers can be addressed by providing
resources, technical support, professional development, instructional support, and
building networking between teachers and schools, the intrinsic technical issues within
3D printing technologies such as the necessity of a long time and appropriate
temperature to successfully print 3D objects may require schools and teachers to take
extra efforts and strategies to leverage these intrinsic technical challenges. Additionally,
the durations of the 3D printing integration in this study were relatively short, which may
have contributed to the inconsistency between teachers’ beliefs and their 3D printing
integration. Moreover, positive relationships between 3D printing integration and
students’ learning outcomes may not show immediately with a short time of intervention.
It can take a long time of investment and continuous support for the teachers to
successfully integrate 3D printing technologies in their science classrooms. Schools
have to be judicious when making the decisions on whether or not to make investments
on a large-scale 3D printing integration in their classrooms.
Implications for Research
In addition to the implications for practice, this study also had some implications
for future research. In this study, teachers’ 3D printing integration level was not a
significant predictor for any of the student learning outcomes, which was probably
because 3D printing integration level did not capture all the possible differences in
teachers’ 3D printing integration and also the short durations of the 3D printing
214
integration. Future studies may include more aspects of teachers’ 3D printing integration
to detect the potential influence on student learning outcomes. It would also be
necessary to test the effects of 3D printing integration with long-term interventions.
Although STEM integration level positively predicted students’ math motivation,
this study cannot assert a causal relationship between STEM integration level and
students’ math motivation. In addition, the interaction between student gender and
STEM integration level suggested that as STEM integration level increases, female
students’ science motivation may have a higher increase than male students. As the
results of this study could not depict causal relationship, future research may conduct
experimental studies to examine the effects of STEM integration level on students’ math
motivation, and also the influence of STEM integration levels on students with individual
differences such as student gender.
Furthermore, teachers’ STEM integration level was a significant predictor for
students’ math motivation, but it was not significant for other student learning outcomes.
As stated previously, STEM integration can make learning more connected and
meaningful for students, but it also requires sufficient knowledge and skills for individual
subjects, which may explain the varied relationship between STEM integration level and
students’ motivation in different STEM subjects, students’ 21st century skills, and
interest in STEM careers. However, this conjecture needs future research to verify.
Future studies may obtain more quantitative student data such as students’ prior
knowledge and skills in the different subjects involved in STEM integration and also
some qualitative data such as student interviews to further explore the relationship
between STEM integration levels and students’ learning outcomes.
215
In addition, STEM integration level had positive interactions with students’ prior
interest in STEM careers and with student gender for science motivation, which
suggested that STEM integration level also had different effects on students with
individual differences. All in all, the varied associations STEM integration levels and
students’ science, technology/engineering, and math motivation, 21st century skills, and
interest in STEM careers, and also the interactions between STEM integration level and
students with individual differences call for further investigation to determine how STEM
integration can positively influence students with individual differences.
In this study, there were no significant relationships between teacher beliefs and
their 3D printing integration, however, there were some significant relationships
between teacher beliefs and student outcomes. Future studies may further explore why
teacher beliefs may influence students even if teachers’ technology integration is not
influenced. This study was not able to include duration and school level as predictors.
Future studies may investigate the effects of these variables. It is also highly
recommended that future research use classroom observation to obtain more data to
ensure a full picture of how 3D printing was integrated and the interactions in the
classrooms.
The interactions between student variables (prior motivation and interest and
student gender) and teacher beliefs showed some consistency but also had
discrepancies on the effect of teachers’ 3D printing integration and teacher beliefs on
male and female students and on students with different prior motivation, interest, and
21st century skills. Previous studies that investigated the relationship between teacher
beliefs and student learning performance and motivation did not examine how teacher
216
beliefs may influence students with individual differences such as gender and prior
motivation and interest. Although previous studies found significant relationships
between teacher beliefs and student learning outcomes in general, the influence of
teacher beliefs on students with individual differences needs further investigation.
Future studies may consider students’ individual differences when examining the
relationships between teacher beliefs, 3D printing integration, and students’ learning
outcomes.
Conclusions
The purpose of this study was to investigate the relationship between teacher
beliefs, teachers’ 3D printing integration, and students’ STEM motivation, 21st century
skills, and interest in STEM careers. Specifically, the correlation between teacher beliefs
and their 3D printing integration, and how teachers’ 3D printing integration predict
students’ STEM motivation, 21st century skills, and interest in STEM careers.
This study produced several interesting results. First, teacher beliefs and 3D
printing integration were generally not correlated except that teachers’ self-efficacy in
pedagogical content knowledge and STEM integration level were significantly but
negatively correlated. Second, teachers perceived 3D printing integration as beneficial
for students, but they encountered several challenges including logistic and technical
issues, lack of resources, lack of time, and insufficient abilities to use 3D printers and
connect 3D printing with curriculum, and also difficulty in engaging and teaching
students with individual differences. Third, STEM integration level was a positive
predictor for students’ math motivation. Fourth, teachers’ 3D printing integration level
was not a significant predictor for any of the student learning outcomes. Fifth, teachers’
extrinsic utility value (perceived usefulness) of 3D printing was a negative predictor for
217
students’ 21st century skills. Finally, there were some interesting interaction effects
between student variables (student gender and pretest scores) and teacher variables
(teacher beliefs and 3D printing integration).
Although not able to claim causal relationships, this study laid a foundation for
future educational practice and research. Schools may provide adequate resources,
technical support, professional development, instructional support, and build networking
between teachers and schools to facilitate teachers’ 3D printing integration. Schools
also need to be judicious when making decisions on whether or not to invest in a large-
scale 3D printing integration. Teachers need to actively participate in professional
development and apply what they have learned in their classrooms. Moreover, teachers
need to consider students’ individual differences including but not limited to student
gender and students’ prior motivation, interest, and skills when integrating 3D printing
into their classrooms. Regarding future research, it would be necessary to employ other
types of research design such as experimental studies to examine the effects of
different 3D printing integration levels and STEM integration levels on students’ learning
outcomes, and also how the different levels may influence students with individual
differences. Last, future studies may further investigate the relationships between
different aspects of teacher beliefs and students’ cognitive and affective learning
outcomes, and how teacher beliefs influence students with individual differences such
as student gender and prior motivation, skills, and interest.
218
APPENDIX A S-STEM SURVEY (UNFRIED ET AL., 2015)
Math
Strongly Disagree
Disagree Neither Agree nor Disagree
Agree Strongly Agree
1. Math has been my worst subject. ○ ○ ○ ○ ○
2. I would consider choosing a career that uses math. ○ ○ ○ ○ ○
3. Math is hard for me. ○ ○ ○ ○ ○
4. I am the type of student to do well in math. ○ ○ ○ ○ ○ 5. I can handle most subjects well, but I cannot do a good job with math.
○ ○ ○ ○ ○
6. I am sure I could do advanced work in math. ○ ○ ○ ○ ○
7. I can get good grades in math. ○ ○ ○ ○ ○
8. I am good at math. ○ ○ ○ ○ ○
Science
Strongly Disagree
Disagree Neither Agree nor Disagree
Agree Strongly Agree
9. I am sure of myself when I do science. ○ ○ ○ ○ ○
10. I would consider a career in science. ○ ○ ○ ○ ○
11. I expect to use science when I get out of school. ○ ○ ○ ○ ○
219
12. Knowing science will help me earn a living. ○ ○ ○ ○ ○
13. I will need science for my future work. ○ ○ ○ ○ ○
14. I know I can do well in science. ○ ○ ○ ○ ○
15. Science will be important to me in my life’s work. ○ ○ ○ ○ ○ 16. I can handle most subjects well, but I cannot do a good job with science.
○ ○ ○ ○ ○
17. I am sure I could do advanced work in science. ○ ○ ○ ○ ○
Engineering and Technology
Please read this paragraph before you answer the questions.
Engineers use math, science, and creativity to research and solve problems that improve everyone’s life and to invent new products. There are many different types of engineering, such as chemical, electrical, computer, mechanical, civil, environmental, and biomedical. Engineers design and improve things like bridges, cars, fabrics, foods, and virtual reality amusement parks. Technologists implement the designs that engineers develop; they build, test, and maintain products and processes.
Strongly Disagree
Disagree Neither Agree nor Disagree
Agree Strongly Agree
18. I like to imagine creating new products. ○ ○ ○ ○ ○ 19. If I learn engineering, then I can improve things that people use every day.
○ ○ ○ ○ ○
20. I am good at building and fixing things. ○ ○ ○ ○ ○
21. I am interested in what makes machines work. ○ ○ ○ ○ ○ 22. Designing products or structures will be important for my future work.
○ ○ ○ ○ ○
220
23. I am curious about how electronics work. ○ ○ ○ ○ ○
24. I would like to use creativity and innovation in my future work. ○ ○ ○ ○ ○ 25. Knowing how to use math and science together will allow me to invent useful things.
○ ○ ○ ○ ○
26. I believe I can be successful in a career in engineering. ○ ○ ○ ○ ○
21st Century Skills
Strongly Disagree
Disagree Neither Agree nor Disagree
Agree Strongly Agree
27. I am confident I can lead others to accomplish a goal. ○ ○ ○ ○ ○
28. I am confident I can encourage others to do their best. ○ ○ ○ ○ ○
29. I am confident I can produce high quality work. ○ ○ ○ ○ ○
30. I am confident I can respect the differences of my peers. ○ ○ ○ ○ ○
31. I am confident I can help my peers. ○ ○ ○ ○ ○ 32. I am confident I can include others’ perspectives when making decisions.
○ ○ ○ ○ ○
33. I am confident I can make changes when things do not go as planned.
○ ○ ○ ○ ○
34. I am confident I can set my own learning goals. ○ ○ ○ ○ ○ 35. I am confident I can manage my time wisely when working on my own.
○ ○ ○ ○ ○
36. When I have many assignments, I can choose which ones need to be done first.
○ ○ ○ ○ ○
221
37. I am confident I can work well with students from different backgrounds.
○ ○ ○ ○ ○
Your Future
Here are descriptions of subject areas that involve math, science, engineering and/or technology, and lists of jobs connected to each subject area. As you read the list below, you will know how interested you are in the subject and the jobs. Fill in the circle that relates to how interested you are.
There are no “right” or “wrong” answers. The only correct responses are those that are true for you.
Not at all Interested
Not So Interested
Interested Very
Interested
1. Physics: is the study of basic laws governing the motion, energy, structure, and interactions of matter. This can include studying the nature of the universe. (aviation engineer, alternative energy technician, lab technician, physicist, astronomer)
○ ○ ○ ○
2. Environmental Work: involves learning about physical and biological processes that govern nature and working to improve the environment. This includes finding and designing solutions to problems like pollution, reusing waste and recycling. (pollution control analyst, environmental engineer or scientist, erosion control specialist, energy systems engineer and maintenance technician)
○ ○ ○ ○
3. Biology and Zoology: involve the study of living organisms (such as plants and animals) and the processes of life. This includes working with farm animals and in areas like nutrition and breeding. (biological technician, biological scientist, plant breeder, crop lab
○ ○ ○ ○
222
technician, animal scientist, geneticist, zoologist) 4. Veterinary Work: involves the science of preventing or treating disease in animals. (veterinary assistant, veterinarian, livestock producer, animal caretaker)
○ ○ ○ ○
5. Mathematics: is the science of numbers and their operations. It involves computation, algorithms and theory used to solve problems and summarize data. (accountant, applied mathematician, economist, financial analyst, mathematician, statistician, market researcher, stock market analyst)
○ ○ ○ ○
6. Medicine: involves maintaining health and preventing and treating disease. (physician’s assistant, nurse, doctor, nutritionist, emergency medical technician, physical therapist, dentist)
○ ○ ○ ○
7. Earth Science: is the study of earth, including the air, land, and ocean. (geologist, weather forecaster, archaeologist, geoscientist)
○ ○ ○ ○
8. Computer Science: consists of the development and testing of computer systems, designing new programs and helping others to use computers. (computer support specialist, computer programmer, computer and network technician, gaming designer, computer software engineer, information technology specialist)
○ ○ ○ ○
9. Medical Science: involves researching human disease and working to find new solutions to human health problems. (clinical laboratory technologist, medical scientist, biomedical engineer, epidemiologist, pharmacologist)
○ ○ ○ ○
10. Chemistry: uses math and experiments to search for new ○ ○ ○ ○
223
chemicals, and to study the structure of matter and how it behaves. (chemical technician, chemist, chemical engineer) 11. Energy: involves the study and generation of power, such as heat or electricity. (electrician, electrical engineer, heating, ventilation, and air conditioning (HVAC) technician, nuclear engineer, systems engineer, alternative energy systems installer or technician)
○ ○ ○ ○
12. Engineering: involves designing, testing, and manufacturing new products (like machines, bridges, buildings, and electronics) through the use of math, science, and computers. (civil, industrial, agricultural, or mechanical engineers, welder, auto-mechanic, engineering technician, construction manager)
○ ○ ○ ○
224
APPENDIX B TEACHER BELIEFS ON 3D PRINTING INTEGRATION SURVEY
Demographic Information
1. Your name: _______
2. Gender: a. Female b. Male c. Other
3. Age range: a. 21-30 b.31-40 c. 41-50 d. 51-60 e. 61+
4. Your race:
5. Your ethnicity:
Teacher Pedagogical Beliefs Survey (adapted from Ravitz, Becker, & Wong, 2000)
J1. The following paragraphs describe observations of two teachers’ classes, Ms. Hill’s and Mr. Jones’. Answer each question below by selecting the column that best answers that question for you.
Definitely
Ms. Hill
Tend
towards
Ms. Hill
Cannot
decide
Tend
towards
Mr. Jones
Definitely
Mr. Jones
1. Which type of class discussion are you more comfortable having in class?
○ ○ ○ ○ ○
2. Which type of discussion do you think most students prefer to have?
○ ○ ○ ○ ○
3. From which type of discussion do you think
○ ○ ○ ○ ○
Ms. Hill was leading her class in an
animated way, asking questions that the
students could answer quickly; based
on the reading they had done the day
before. After this review, Ms. Hill
taught the class new material, again
using simple questions to keep students
attentive and listening to what she said.
Mr. Jones’ class was also having a
discussion, but many of the questions
came from the students themselves.
Though Mr. Jones could clarify
students’ questions and suggest where
the students could find relevant
information, he couldn’t really answer
most of the questions himself.
225
students gain more knowledge?
4. From which type of discussion do you think students gain more useful skills?
○ ○ ○ ○ ○
J2. Indicate how much you disagree or agree with each of the following statements about teaching and learning.
Strongly Disagree
Disagree Neither Disagree nor Agree
Agree Strongly Agree
5. Teachers know a lot more than students; they shouldn't let students muddle around when they can just explain the answers directly.
○ ○ ○ ○ ○
6. A quiet classroom is generally needed for effective learning. ○ ○ ○ ○ ○
7. It is better when the teacher – not the students - decides what activities are to be done.
○ ○ ○ ○ ○
8. Students will take more initiative to learn when they feel free to move around the room during class.
○ ○ ○ ○ ○
9. Students should help establish criteria on which their work will be assessed.
○ ○ ○ ○ ○
10. Instruction should be built around problems with clear, correct answers, and around ideas that most students can grasp quickly.
○ ○ ○ ○ ○
11. How much students learn depends on how much background knowledge they have – that is why teaching facts is so necessary.
○ ○ ○ ○ ○
226
J3. Different teachers have described very different teaching philosophies to researchers. For each of the following pairs of statements, check the button that best shows how closely your own beliefs are to each of the statements in a given pair. The closer your beliefs to a particular statement, the closer the button you check.
12. “I mainly see my role as a facilitator. I try to provide opportunities and resources for my students to discover or construct concepts for themselves.”
○ ○ ○ ○ ○
"That's all nice, but students really won't learn the subject unless you go over the material in a structured way. It's my job to explain, to show students how to do the work, and to assign specific practice."
13. "The most important part of instruction is that it encourages “sense-making” or thinking among students. Content is secondary."
○ ○ ○ ○ ○
"The most important part of instruction is the content of the curriculum. That content is the community’s judgment about what children need to be able to know and do."
14. "It is critical for students to become interested in doing academic work—interest and effort are more important than the particular subject matter they are working on."
○ ○ ○ ○ ○
"While student motivation is certainly useful, it should not drive what students study. It is more important that students learn the history, science, math and language skills in their textbooks."
15. "It is a good idea to have all sorts of activities going on in the classroom. Some students might produce a scene from a play they read. Others might create a miniature version of the set. It's hard to get the logistics right, but the successes are so much more important than the failures."
○ ○ ○ ○ ○
"It's more practical to give the whole class the same assignment, one that has clear directions, and one that can be done in short intervals that match students' attention spans and the daily class schedule."
227
Teacher Self-Efficacy in 3D Printing Integration Survey (adapted from Schmidt et al., 2009)
All the items used a 5-point Likert scale: Strongly Disagree = 1; Disagree = 2; Neither Agree/Disagree = 3; Agree = 4; Strong Agree = 5.
Notes: The subtitles here are just to show the components of the survey and they were not included in the survey administered.
Self-efficacy in Technology Knowledge (TK)
1. I am confident that I know how to solve technical problems of 3D printing technology.
2. I am confident that I can learn 3D printing technology easily.
3. I am confident that I can keep up with 3D printing technology.
4. I am confident that I have sufficient knowledge about 3D printing technology.
5. I am confident that I have the technical skills I need to use 3D printing technology.
Self-efficacy in Content Knowledge (CK)
6. I am confident that I have sufficient knowledge about science.
7. I am confident that I have sufficient knowledge about paleontology.
8. I am confident that I can use a scientific way of thinking.
9. I am confident that I have various ways and strategies of developing my understanding of science
Self-efficacy in Pedagogical Knowledge (PK)
10. I am confident that I know how to assess student performance in a classroom.
11. I am confident that I can adapt my teaching based upon what students currently understand or do not understand.
12. I am confident that I can adapt my teaching style to different learners.
13. I am confident that I can assess student learning in multiple ways.
14. I am confident that I can use a wide range of teaching approaches in a classroom setting.
15. I am confident that I am familiar with common student understandings and misconceptions.
16. I am confident that I know how to organize and maintain classroom management.
Self-efficacy in Pedagogical Content Knowledge (PCK)
17. I am confident that I can select effective teaching approaches to guide student thinking and learning in science.
228
Self-efficacy in Technological Content Knowledge (TCK)
18. I am confident that I know about 3D printing technology for understanding and doing science.
Self-efficacy in Technological Pedagogical Knowledge (TPK)
19. I am confident that I can use 3D printing technology to enhance the teaching approaches for a lesson.
20. I am confident that I can use 3D printing technology to enhance students’ learning for a lesson.
21. I am confident that I can think deeply about how 3D printing technology could influence the teaching approaches I use in my classroom.
22. I am confident that I can think critically about how to use 3D printing technology in my classroom.
23. I am confident that I can use 3D printing technology for different teaching activities.
Self-efficacy in Technological Pedagogical Content Knowledge (TPACK)
24. I am confident that I can design and teach lessons that appropriately combine science, 3D printing technology, and teaching approaches.
25. I am confident that I can use 3D printing technology to enhance what I teach, how I teach, and what students learn.
26. I am confident that I can provide leadership in helping others to coordinate the use of science content, 3D printing technologies, and teaching approaches at my school and/or district.
Teacher 3D Printing Value Beliefs (Adapted from Eccles & Wigfield, 1995)
Intrinsic Interest Value
1. In general, I find integrating 3D printing technology in my science classrooms (very boring, boring, neither boring nor interesting, interesting, very interesting)
2. How much do you like integrating 3D printing technology in your science classrooms? (strongly dislike, dislike, neither dislike nor like, like, strongly like)
Attainment Value/Importance
3. Is the amount of effort it took to do well in integrating 3D printing into your science classrooms worthwhile to you? (not at all worthwhile, not worthwhile, neutral, worthwhile, very worthwhile)
229
4. I feel that, to me, being good at integrating 3D printing technology into my science classrooms is (Not at all important, not important, neutral, important, very important)
5. How important is it for you to do well in integrating 3D printing technology into your science classrooms? (Not at all important, not important, neutral, important, very important)
Extrinsic Utility Value/Usefulness
6. How useful is integrating 3D printing technology in your science classrooms to enhance students’ learning? (not at all useful, not useful, neutral, useful, very useful)
7. How useful is integrating 3D printing technology in your science classrooms to engage students? (not at all useful, not useful, neutral, useful, very useful)
Open-Ended Questions on Teacher Beliefs on 3D Printing Integration
1. How do you feel about integrating 3D printing technology into your science teaching?
2. Do you think you have sufficient knowledge and skills to integrate 3D printing technology in your science classes? Please explain.
3. What are the biggest advantages in integrating 3D printing technology in science teaching?
4. What are the biggest challenges in integrating 3D printing technology in science teaching?
230
LIST OF REFERENCES
Al-Rajhi, N., Al-Abdulkarim, A., Al-Khalifa, H. S., & Al-Otaibi, H. M. (2015, July). Making Linear Equations Accessible for Visually Impaired Students Using 3D Printing. In Advanced Learning Technologies (ICALT), 2015 IEEE 15th International Conference on (pp. 432-433). IEEE.
Albion, P. (1999). Self-efficacy beliefs as an indicator of teachers' preparedness for teaching with technology. In Proceedings of the 10th International Conference of the Society for Information Technology & Teacher Education (SITE 1999) (pp. 1602-1608). Association for the Advancement of Computing in Education (AACE).
Anderson, S. E., & Maninger, R. M. (2007). Preservice teachers' abilities, beliefs, and intentions regarding technology integration. Journal of Educational Computing Research, 37(2), 151-172.
Antonenko, P. D. (2015). The instrumental value of conceptual frameworks in educational technology research. Educational Technology Research and Development, 63(1), 53-71.
Ashcraft, M. H., & Ridley, K. S. (2005). Math anxiety and its cognitive consequences: A tutorial review. In J. I. D. Campbell (Ed.), Handbook of mathematical cognition (pp. 315-327). New York, NY: Psychology Press.
Ashton, P. T., & Webb, R. B. (1986). Making a difference: Teachers' sense of efficacy and student achievement. New York, NY: Longman.
Atkinson, J. W. (1957). Motivational determinants of risk-taking behavior. Psychological Review, 64(6), 359-372. http://dx.doi.org/10.1037/h0043445
Bandura, A. (1982). Self-efficacy mechanism in human agency. American psychologist, 37(2), 122-147.
Bandura, A. (1986). Social foundations of thoughts and action: A social cognitive theory. Englewood Cliffs, NJ: Prentice Hall.
Bandura, A. (1997). Self-efficacy: The exercise of control. New York, NY: W. H. Freeman and Company.
Bauer, J., & Kenton, J. (2005). Toward technology integration in schools: Why it is not happening. Journal of Technology and Teacher Education, 13, 519-546.
Beck, A. T. (1976). Cognitive therapy and the emotional disorders. New York, NY: International Universities Press.
231
Becker, H. J. (1999). Internet use by teachers: Conditions of professional use and teacher-directed student use. Irvine, CA: Center for Research on Information Technology and Organizations.
Becker, H. J. (2001, April). How are teachers using computers in instruction? In Paper presented at the annual meeting of the American Educational Research Association, Seattle, WA. Retrieved from https://msu.edu/course/cep/807/*cep240studyrefs/beckeraera2001howtchrsusing.pdf on December 17, 2018.
Bicer, A., Nite, S. B., Capraro, R. M., Barroso, L. R., Capraro, M. M., & Lee, Y. (2017, October). Moving from STEM to STEAM: The effects of informal STEM learning on students' creativity and problem solving skills with 3D printing. In Frontiers in Education Conference (FIE) (pp. 1-6). IEEE.
Bragow, D., Gragow, K. A., & Smith, E. (1995). Back to the future: Toward curriculum integration. Middle School Journal, 27, 39–46.
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative research in psychology, 3(2), 77-101.
Britten, J. S., & Cassady, J. C. (2005). The Technology Integration Assessment Instrument: Understanding planned use of technology by classroom teachers. Computers in the Schools, 22(3), 49-61.
Britner, S. L., & Pajares, F. (2006). Sources of science self-efficacy beliefs of middle school students. Journal of Research in Science Teaching, 43(5), 485–499.
Brookover, W., Beady, C, Flood, P., Schweitzer, J., & Wisenbaker, J. (1979). School social systems and student achievement: Schools can make a difference. New York, NY: Praeger.
Brophy, J. E., & Evertson, C. (1977). Teacher behaviors and student learning in second and third grades. In G. D. Borich (Ed.), The appraisal of teaching: Concepts and process (pp. 79-95). Reading, MA: Addison-Wesley.
Brophy, J. E., & Good, T. (1974). Teacher-student relationships: Causes and consequences. New York, NY: Holt, Rinehart, & Winston.
Brown, P. L., Concannon, J. P., Marx, D., Donaldson, C., & Black, A. (2016). An examination of middle school students’ STEM self-efficacy, interests and perceptions. Journal of STEM Education: Innovations and Research, 17(3), 27-38.
Bull, G., Chiu, J., Berry, R., Lipson, H., & Xie, C. (2014). Advancing children’s engineering through desktop manufacturing, in Mayer, R., and Alexander, P., eds., Handbook of Research on Educational Communications and Technology (pp. 675–688). New York, Springer.
232
Bureau of Labor Statistics (BLS). (2017). Employment projections – 2016-26. Retrieved from https://www.bls.gov/news.release/pdf/ecopro.pdf on Dec. 24, 2018.
Byars-Winston, A., Estrada, Y., Howard, C., Davis, D., & Zalapa, J. (2010). Influence of social cognitive and ethnic variables on academic goals of underrepresented students in science and engineering: A multiple-groups analysis. Journal of counseling psychology, 57(2), 205-218.
Chao, J. Y., Po, H. Y., Chang, Y. S., & Yao, L. Y. (2016, November). The study of 3D printing project course for indigenous senior high school students in Taiwan. In Advanced Materials for Science and Engineering (ICAMSE), International Conference on (pp. 68-70). IEEE.
Chen, M., Zhang, Y., & Zhang, Y. (2014). Effects of a 3D printing course on mental rotation ability among 10-year-old primary students. International Journal of Psychophysiology, 94(2), 240.
Cheng, S. L., & Xie, K. (2018). The relations among teacher value beliefs, personal characteristics, and TPACK in intervention and non-intervention settings. Teaching and Teacher Education, 74, 98-113.
Chien, Y. H. (2017). Developing a pre-engineering curriculum for 3D printing skills for high school technology education. Eurasia Journal of Mathematics, Science and Technology Education, 13(7), 2941-2958.
Chien, Y. H., & Chu, P. Y. (2018). The Different Learning Outcomes of High School and College Students on a 3D-Printing STEAM Engineering Design Curriculum. International Journal of Science and Mathematics Education, 16(6), 1047-1064.
Christensen, R., & Knezek, G. (2017). Relationship of middle school student STEM interest to career intent. Journal of Education in Science, Environment and Health, 3(1), 1-13.
Clark, R. E. (1983). Reconsidering research on learning from media. Review of Educational Research, 53(4), 445-459. Doi:10.3102/00346543053004445
Clark, R. E. (1994a). Media will never influence learning. Educational Technology Research and Development, 42(2), 21-29. Doi:10.1007/BF02299088
Clark, R. E. (1994b). Media and method. Educational Technology Research and Development, 42(3), 7-10. Doi:10.1007/BF02298090
Cornelius-White, J. (2007). Learner-centered teacher-student relationships are effective: A meta-analysis. Review of educational research, 77(1), 113-143.
233
Cornelius-White, J. H., & Harbaugh, A. P. (2009). Learner-centered instruction: Building relationships for student success. Thousand Oaks, CA: Sage.
Crandall, V. C. (1969). Sex differences in expectancy of intellectual and academic reinforcement. In C. P. Smith (ed.), Achievement-related motives in children. New York, NY: Russell Sage Foundation.
Crandall, V. J., Katkovsky, W., & Preston, A. (1962). Motivational and ability determinants of young children's intellectual achievement behavior. Child Development, 33: 643-661.
Culp, K. M., Honey, M., & Mandinach, E. (2005). A retrospective on twenty years of education technology policy. Journal of Educational Computing Research, 32(3), 279-307.
Definition and Terminology Committee of Association for Educational Communications and Technology. (2018). Definition of Educational Technology. Retrieved from https://www.aect.org/ on November 23, 2018.
Deng, F., Chai, C. S., Chin-Chung, T., & Min-Hsien, L. (2014). The relationships among Chinese practicing teachers' epistemic beliefs, pedagogical beliefs and their beliefs about the use of ICT. Journal of Educational Technology & Society, 17(2), 245.
Dewey, J. (1913). Interest and effort in education. Boston, MA: Houghton Mifflin Company. http://dx.doi.org/10.1037/14633-000
Eccles, J. S. (2009). Who am I and what am I going to do with my life? Personal and collective identities as motivators of action. Educational Psychologist, 44(2), 78-89.
Eccles, J. S., Adler, T. F., Futterman, R., Goff, S. B., Kaczala, C. M., Meece, J., and Midgley, C. (1983). Expectancies, values and academic behaviors. In J. T. Spence (ed.), Achievement and Achievement Motives. San Francisco, CA: W. H. Freeman.
Eccles, J. S., & Wigfield, A. (1985). Teacher expectations and student motivation. In J. Dusek (Ed.), Teacher expectancies (pp. 185-217). Hillsdale, NJ: Erlbaum.
Eccles, J. S., & Wigfield, A. (1995). In the mind of the actor: The structure of adolescents' achievement task values and expectancy-related beliefs. Personality and social psychology bulletin, 21(3), 215-225.
Eccles, J. S., & Wigfield, A. (2002). Motivational beliefs, values, and goals. Annual review of psychology, 53(1), 109-132.
234
Ertmer, P. A. (1999). Addressing first-and second-order barriers to change: Strategies for technology integration. Educational technology research and development, 47(4), 47-61.
Ertmer, P. A. (2005). Teacher pedagogical beliefs: The final frontier in our quest for technology integration? Educational Technology Research and Development, 53(4) 25-39.
Ertmer, P. A., Addison, P., Lane, M., Ross, E., & Woods, D. (1999). Examining teachers’ beliefs about the role of technology in the elementary classroom. Journal of Research on Computing in Education, 32(1), 54–72.
Ertmer, P. A., Conklin, D., Lewandowski,]., Osika, E., Selo, M., & Wignall, E. (2003). Increasing preservice teachers' capacity for technology integration through the use of electronic models. Teacher Education Quarterly, 30(1), 95-112.
Ertmer, P. A., & Glazewski, K. D. (2015). Essentials for PBL implementation: Fostering collaboration, transforming roles, and scaffolding learning. In A. Walker, H. Leary, C. Hmelo-Silver, & P. A. Ertmer (Eds.), Essential readings in problem-based learning (pp. 89-106). West Lafayette, IN: Purdue University Press.
Ertmer, P. A., & Ottenbreit-Leftwich, A. T. (2010). Teacher technology change: How knowledge, confidence, beliefs, and culture intersect. Journal of research on Technology in Education, 42(3), 255-284.
Ertmer, P. A., Ottenbreit-Leftwich, A. T., Sadik, O., Sendurur, E., & Sendurur, P. (2012). Teacher beliefs and technology integration practices: A critical relationship. Computers & Education, 59(2), 423-435.
Fan, W. (2011). Social influences, school motivation and gender differences: an application of the expectancy‐value theory. Educational Psychology, 31(2), 157-175.
Feather, N. T. (1982). Expectations and actions: Expectancy-value models in psychology. Hillsdale, NJ: Lawrence Erlbaum Association Incorporated.
Feather, N. T. (1988). Values, valences, and course enrollment: Testing the role of personal values within an expectancy-valence framework. Journal of educational psychology, 80(3), 381-391.
Feather, N. T. (1992). Values, valences, expectations, and actions. Journal of Social Issues, 48(2), 109-124.
Francis, J. (2017). The effects of technology on student motivation and engagement in classroom-based learning. Unpublished doctoral thesis. University of New England, USA. Retrieved from http://dune.une.edu/cgi/viewcontent.cgi?article=1120&context=theses
235
Friday Institute for Educational Innovation (2012). Upper Elementary School Student Attitudes toward STEM Survey. Raleigh, NC: Author.
Friedman, T. L. (2005). The world is flat: A brief history of the twenty-first century. New York, NY: Farrar, Straus and Giroux.
Furner, J., & Kumar, D. (2007). The mathematics and science integration argument: a stand for teacher education. Eurasia Journal of Mathematics, Science & Technology, 3(3), 185–189.
Gendron, B. (1977). Technology and the human condition. New York, NY: St. Martin’s.
Gonzales, S. (2013). Examining the relationship among high-school teachers' technology self-efficacy, attitudes towards technology integration, and quality of technology integration. Available from ProQuest Dissertations & Theses Global. UMI 3596649. Northern Illinois University.
Gonzalez, S. R., & Bennett, D. B. (2016). 3D printing: A practical guide for librarians (Vol. 22). Lanham, MD: Rowman & Littlefield.
Good, T. L. (1981). Teacher expectations and student perceptions: A decade of research. Educational Leadership, 38, 415-422.
Grant, C. A., MacFadden, B. J., Antonenko, P., & Perez, V. J. (2016). 3-D fossils for k–12 education: a case example using the giant extinct shark Carcharocles Megalodon. The Paleontological Society Papers, 22, 197-209.
Gray, L., Thomas, N., & Lewis, L. (2010). Teachers' Use of Educational Technology in U.S. Public Schools: 2009. Washington, DC: National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education. Retrieved from https://files.eric.ed.gov/fulltext/ED509514.pdf on December 11, 2018.
Grice, N., Christian, C., Nota, A., & Greenfield, P. (2015). 3D printing technology: A unique way of making Hubble Space Telescope Images accessible to non-visual learners. Journal of Blindness Innovation and Research, 5(1). Retrieved from https://nfb.org/images/nfb/publications/jbir/jbir15/jbir050101.html. doi: http://dx.doi.org/10.5241/5-66
Grigg, S., Perera, H. N., McIlveen, P., & Svetleff, Z. (2018). Relations among math self efficacy, interest, intentions, and achievement: A social cognitive perspective. Contemporary Educational Psychology, 53, 73-86.
Gutherie, J. T., Wigfield, A., & VonSecker, C. (2000). Effects of integrated instruction on motivation and strategy use in reading. Journal of Educational Psychology, 92, 331–341, http://dx.doi.org/10.1037/0022-0663.92.2.331
236
Haight, K. W. (2011). Teacher self-efficacy in the adoption and integration of technology within the classroom. Dissertation published by ProQuest. UMI 3445014. Walden University.
Harmes, J. C., Welsh, J. L., & Winkelman, R. J. (2016). A framework for defining and evaluating technology integration in the instruction of real-world skills. In Handbook of Research on Technology Tools for Real-World Skill Development (pp. 137-162). Hershey, PA: IGI Global.
Harris, J., Grandgenett, N., & Hofer, M. (2010). Testing a TPACK-based technology integration assessment instrument. In C. D. Maddux, D. Gibson, & B. Dodge (Eds.). Research highlights in technology and teacher education 2010 (pp. 323-331). Chesapeake, VA: Society for Information Technology and Teacher Education (SITE).
Heafner, T. (2004). Using technology to motivate students to learn social studies. Contemporary Issues in Technology and Teacher Education, 4(1), 42-53.
Heller, K. A., & Parsons, J. E. (1981) Teacher influences on students' expectancies for success in mathematics. Child Development, 52, 1015-1019.
Heineman, A. J. (2018). Instructional Coaching, Educational Technology, and Teacher Self-Efficacy: A Case Study of Instructional Coaching Programs in a California Public K-12 School District. Dissertation published by ProQuest. ProQuest Number: 10801469. University of Southern California.
Hermans, R., Tondeur, J., van Braak, J., & Valcke, M. (2008). The impact of primary school teachers’ educational beliefs on the classroom use of computers. Computers & education, 51(4), 1499-1509.
Hidi, S., & Ainley, M. (2008). Interest and self-regulation: Relationships between two variables that influence learning. In D. H. Schunk and B. J. Zimmerman (Eds.), Motivation and self-regulated learning: Theory, research, and application (pp. 77-109). Mahwah, NJ: Erlbaum.
Hidi, S., & Renninger, K. A. (2006). The four-phase model of interest development. Educational psychologist, 41(2), 111-127.
Hixon, E., & Buckenmeyer, J. (2009). Revisiting technology integration in schools: Implications for professional development. Computers in the Schools, 26(2), 130-146.
Honey, M., Pearson, G., & Schweingruber, H. (Eds.). (2014). STEM integration in K-12 education: Status, prospects, and an agenda for research. Washington, DC: National Academies Press.
Howland, J. L., Jonassen, D. H., & Marra, R. M. (2012). Meaningful learning with technology. Upper Saddle River, NJ: Pearson.
237
Hox, J. (2002). Multilevel analysis: Techniques and applications. Mahwah, NJ: Erlbaum.
Hoy, A. W., & Davis, H. A. (2005). Teachers’ sense of efficacy and its influence on the achievement of adolescents. In T. Urdan & F. Pajares (Eds.), Adolescence and education: Volume V: Self-efficacy beliefs during adolescence (pp. 117-137). Greenwich, CT: Information Age.
Hsiao, H. S., Chen, J. C., Lin, C. Y., Zhuo, P. W., & Lin, K. Y. (2019). Using 3D printing technology with experiential learning strategies to improve preengineering students' comprehension of abstract scientific concepts and hands‐on ability. Journal of Computer Assisted Learning. 35(2), 178-187
Hsu, P. S. (2016). Examining current beliefs, practices and barriers about technology integration: A case study. TechTrends, 60(1), 30-40.
Huddlestone, J., & Pike, J. (2008). Seven key decision factors for selecting e-learning. Cognition, Technology & Work, 10(3), 237–247.
Inan, F. A., & Lowther, D. L. (2010). Factors affecting technology integration in K-12 classrooms: A path model. Educational Technology Research and Development, 58(2), 137-154.
Internatinoal Society for Technology in Education (ISTE). (2019). Essential Conditions. Retrieved June 10, 2019, from https://www.iste.org/standards/essential-conditions
Jo, W. (2016). Introduction of 3D printing technology in the classroom for visually impaired students. Journal of Visual Impairment & Blindness, 110(2), 115-121.
Johnson, C. C. (2013). Conceptualizing integrated STEM education. School Science and Mathematics, 113(8), 367-368.
Jones, V. R. (2014). Teaching STEM: 21st century skills. Children's Technology & Engineering, 18(4), 11-13.
Kagan, D. M. (1992). Implication of research on teacher belief. Educational psychologist, 27(1), 65-90.
Kelley, T. R., & Knowles, J. G. (2016). A conceptual framework for integrated STEM education. International Journal of STEM Education, 3(1), 1-11.
Kenward, M. G., & Roger, J. H. (2009). An improved approximation to the precision of fixed effects from restricted maximum likelihood. Computational Statistics & Data Analysis, 53(7), 2583-2595.
Kim, C., Kim, M. K., Lee, C., Spector, J. M., & DeMeester, K. (2013). Teacher beliefs and technology integration. Teaching and teacher education, 29, 76-85.
238
Koehler, K. E. (2017). Examining the conceptual understandings of geoscience concepts of students with visual impairments: Implications of 3-D printing (Order No. 10702675). Available from ProQuest Dissertations & Theses Global. (1999311016). Doctoral dissertation, The Ohio State University.
Koehler, M. J., & Mishra, P. (2009). What is technological pedagogical content knowledge? Contemporary Issues in Technology and Teacher Education, 9(1), 60-70.
Koehler, M. J., Mishra, P., & Cain, W. (2013). What is technological pedagogical content knowledge (TPACK)? Journal of Education, 193(3), 13-19.
Kozma, R. B. (1991). Learning with media. Review of Educational Research, 61(2), 179-211. doi:10.2307/1170534
Kozma, R. B. (1994). Will media influence learning? Reframing the debate. Educational Technology Research and Development, 42(2), 7-19. doi:10.1007/BF02299087
Krapp, A. (2002). Structural and dynamic aspects of interest development: Theoretical considerations from an ontogenetic perspective. Learning and instruction, 12(4), 383-409.
Krathwohl, D. R. (2002). A revision of Bloom's taxonomy: An overview. Theory into practice, 41(4), 212-218.
Kwon, H. (2017). Effects of 3D printing and design software on students' overall performance. Journal of STEM Education: Innovations and Research, 18(4), 37-42.
Laboy-Rush, D. (2011). Integrated STEM education through project-based learning. Retrieved from https://www.rondout.k12.ny.us/common/pages/DisplayFile.aspx?itemId=16466975
Lazarides, R., Buchholz, J., & Rubach, C. (2018). Teacher enthusiasm and self-efficacy, student-perceived mastery goal orientation, and student motivation in mathematics classrooms. Teaching and Teacher Education, 69, 1-10.
Lazarus, R. S., & Launier, R. (1978). Stress-related transactions between person and environment. In L. A. Pervin & M. Lewis (Eds.), Perspectives in interactional psychology. New York, NY: Plenum Press.
Lent, R. W., & Brown, S. D. (1996). Social cognitive approach to career development: An overview. The Career Development Quarterly, 44(4), 310-321.
Lent, R. W., & Brown, S. D. (2006). Integrating person and situation perspectives on work satisfaction: A social-cognitive view. Journal of vocational behavior, 69(2), 236-247.
239
Lent, R. W., Brown, S. D., & Hackett, G. (2000). Contextual supports and barriers to career choice: A social cognitive analysis. Journal of counseling psychology, 47(1), 36-49.
Lent, R. W., Brown, S. D., Sheu, H.-B., Schmidt, J., Brenner, B. R., Gloster, C. S., Wilkins, G., Schmidt, L. C., Lyons, H., & Treistman, D. (2005). Social cognitive predictors of academic interests and goals in engineering: Utility for women and students at historically black universities. Journal of Counseling Psychology, 52(1), 84-92.
Lent, R. W., Lopez Jr., A. M., Lopez, F. G., & Sheu, H. B. (2008). Social cognitive career theory and the prediction of interests and choice goals in the computing disciplines. Journal of Vocational Behavior, 73(1), 52-62.
Lent, R. W., Lopez, F. G., Sheu, H., & Lopez, A. M. (2011). Social cognitive predictors of the interests and choices of computing majors: Applicability to underrepresented students. Journal of Vocational Behavior, 78(2), 184-192. doi: 10.1016/j.jvb.2010.10.006
Lent, R. W., Sheu, H. B., Gloster, C. S., & Wilkins, G. (2010). Longitudinal test of the social cognitive model of choice in engineering students at historically Black universities. Journal of Vocational Behavior, 76(3), 387-394.
Leshin, C. B., Pollock, J., & Reigeluth, C. M. (1992). Instructional design strategies and tactics. Englewood Cliffs, NJ: Educational Technology Publications.
Li, Y., Garza, V., Keicher, A., & Popov, V. (2018). Predicting High School Teacher Use of Technology: Pedagogical Beliefs, Technological Beliefs and Attitudes, and Teacher Training. Technology, Knowledge and Learning, 1-18.
Maloney, E. A., & Beilock, S. L. (2012). Math anxiety: Who has it, why it develops, and how to guard against it. Trends in cognitive sciences, 16(8), 404-406.
Maloy, R., Trust, T., Kommers, S., Malinowski, A., & LaRoche, I. (2017). 3D modeling and printing in history/social studies classrooms: Initial lessons and insights. Contemporary Issues in Technology and Teacher Education, 17(2), 229-249.
Maltese, A. V., Melki, C. S., & Wiebke, H. L. (2014). The nature of experiences responsible for the generation and maintenance of interest in STEM. Science Education, 98(6), 937-962.
Maltese, A. V., & Tai, R. H. (2011). Pipeline persistence: Examining the association of educational experiences with earned degrees in STEM among US students. Science Education, 95(5), 877-907.
240
Manglicmot, T. M. (2015). Relationship between teacher self-efficacy, outcome expectations and the level of technology integration in secondary classrooms. Dissertation published by ProQuest. UMI 3663020. The College of William and Mary.
Marcinkiewicz, H. R. (1994). Differences in computer use of practicing versus preservice teachers. Journal of Research on Computing in Education, 27(2), 184-197.
Martin, A. J., Ginns, P., & Papworth, B. (2017). Motivation and engagement: Same or different? Does it matter? Learning and Individual Differences, 55, 150-162.
Matthews, J. S., Ponitz, C. C., & Morrison, F. J. (2009). Early gender differences in self-regulation and academic achievement. Journal of educational psychology, 101(3), 689-704.
Mcloughlin, L., Fryazinov, O., Moseley, M., Sanchez, M., Adzhiev, V., Comninos, P., & Pasko, A. (2016). Virtual sculpting and 3D printing for young people with disabilities. IEEE computer graphics and applications, 36(1), 22-28.
Meichenbaum, D. H. (1977). Cognitive-behavior modification: An integrative approach. New York, NY: Plenum Press.
Midgley, C., Feldlaufer, H., & Eccles, J. S. (1989). Change in teacher efficacy and student self-and task-related beliefs in mathematics during the transition to junior high school. Journal of educational Psychology, 81(2), 247-258.
Miller, J. A. (2015). Predictors of student persistence in the STEM pipeline: Activities outside the classroom, parent aspirations, and student self-beliefs using NELS:88 data. Retrieved from ProQuest Dissertations & Theses Global.
Mishra, P., & Koehler, M. J. (2006). Technological pedagogical content knowledge: A framework for integrating technology in teacher knowledge. Teachers College Record, 108 (6), 1017-1054.
Mitchell, M. (1993). Situational interest: Its multifaceted structure in the secondary school mathematics classroom. Journal of educational psychology, 85(3), 424-436.
Mojavezi, A., & Tamiz, M. P. (2012). The Impact of Teacher Self-efficacy on the Students' Motivation and Achievement. Theory & Practice in Language Studies, 2(3), 483-491.
Moore, T., Stohlmann, M., Wang, H., Tank, K., Glancy, A., & Roehrig, G. (2014). Implementation and integration of engineering in K-12 STEM education. In S. Purzer, J. Strobel, & M. Cardella (Eds.), Engineering in pre-college settings: Synthesizing research, policy, and practices (pp. 35-60). West Lafayette, IN: Purdue University Press.
241
Mustafa, N., Ismail, Z., Tasir, Z., Said, M., & Haruzuan, M. N. (2016). A meta-analysis on effective strategies for integrated STEM education. Advanced Science Letters, 22(12), 4225-4228.
Muthén, B. O. (1991). Multilevel factor analysis of class and student achievement components. Journal of Educational Measurement, 28, 338-354.
Muthén, B. O. (1994). Multilevel covariance structure analysis. Sociological Methods and Research, 22, 376-398.
Muthén, B. O., & Satorra, A. (1989). Multilevel aspects of varying parameters in structural models. In R. D. Bock (Ed.), Multilevel analysis of educational data (pp. 87−99). San Diego: Academic Press.
Muthén, B. O., & Satorra, A. (1995). Complex sample data in structural equation modeling. In P. V. Marsden (Ed.), Sociological methodology, 1995 (pp. 267−316). Oxford: Blackwell.
Mortimore, P. (Ed.). (1999). Understanding Pedagogy and its impact on learning. Thousand Oaks, CA: Sage Publications Inc.
Mueller, J., Wood, E., Willoughby, T., Ross, C., & Specht, J. (2008). Identifying discriminating variables between teachers who fully integrate computers and teachers with limited integration. Computers & education, 51(4), 1523-1537.
National Research Council (NRC). (2010). Exploring the Intersection of Science Education and 21st Century Skills: A Workshop Summary. Washington, DC: The National Academies Press. Retrieved from https://doi.org/10.17226/12771.
National Research Council (NRC). (2011). Successful K-12 STEM education: Identifying effective approaches in science, technology, engineering, and mathematics. Washington, DC: National Academies Press.
National Research Council (NRC). (2014). STEM integration in K-12 education: Status, prospects, and an agenda for research. Washington DC: National Academy of Science.
Nauta, M. M., & Epperson, D. L. (2003). A longitudinal examination of the social-cognitive model applied to high school girls' choices of nontraditional college majors and aspirations. Journal of Counseling Psychology, 50(4), 448-457.
Nemorin, S., & Selwyn, N. (2017). Making the best of it? Exploring the realities of 3D printing in school. Research Papers in Education, 32(5), 578-595.
Ng, O. L. (2017). Exploring the use of 3D Computer-Aided Design and 3D Printing for STEAM Learning in Mathematics. Digital Experiences in Mathematics Education, 3(3), 257-263.
242
Niederhauser, D. S., & Stoddart, T. (2001). Teachers’ instructional perspectives and use of educational software. Teaching and teacher education, 17(1), 15-31.
O'Mahony, C. (2003). Getting the information and communications technology formula right: access+ ability= confident use. Technology, Pedagogy and Education, 12(2), 295-311.
Ottenbreit-Leftwich, A. T., Glazewski, K. D., Newby, T. J., & Ertmer, P. A. (2010). Teacher value beliefs associated with using technology: Addressing professional and student needs. Computers & education, 55(3), 1321-1335.
Pajares, M. F. (1992). Teachers’ beliefs and educational research: Cleaning up a messy construct. Review of educational research, 62(3), 307-332.
Pajares, M. F. (2005). Gender differences in mathematics self-efficacy beliefs. In A. M. Gallagher & J. C. Kaufman (Eds.), Gender differences in mathematics: An integrative psychological approach (pp. 294–315). New York: Cambridge University Press.
Pan, Y. H. (2014). Relationships among teachers’ self-efficacy and students’ motivation, atmosphere, and satisfaction in physical education. Journal of Teaching in Physical Education, 33(1), 68-92.
Park, S. H., & Ertmer, P. A. (2007). Impact of problem-based learning (PBL) on teachers’ beliefs regarding technology use. Journal of research on technology in education, 40(2), 247-267.
Partnership for 21st Century Skills (P21). Framework for 21st century learning. Retrieved from http://www.p21.org/our-work/p21-framework.
Pearson, G. (2017). National academies piece on integrated STEM. The Journal of Educational Research, 110(3), 224-226. https://doi.org/10.1080/00220671.2017.1289781
Pelgrum, W. J. (2001). Obstacles to the integration of ICT in education: results from a worldwide educational assessment. Computers & education, 37(2), 163-178.
Peugh, J. L. (2010). A practical guide to multilevel modeling. Journal of school psychology, 48(1), 85-112.
Pintrich, P. R. (1990). Implications of psychological research on student learning and college teaching for teacher education. In W. R. Houston (Ed.), Handbook of research on teacher education (pp. 826-857). New York, NY: Macmillan.
243
President’s Committee of Advisors on Science and Technology (PCAST). (2010). Prepare and inspire: K-12 education in Science, Technology, Engineering, and Math (STEM) for America’s future. Washington, DC: Executive Office of the President. Retrieved from https://nsf.gov/attachments/117803/public/2a--Prepare_and_Inspire--PCAST.pdf.
Pringle, R. M., Dawson, K., & Ritzhaupt, A. D. (2015). Integrating science and technology: Using technological pedagogical content knowledge as a framework to study the practices of science teachers. Journal of Science Education and Technology, 24(5), 648-662.
Ramirez, G., Gunderson, E. A., Levine, S. C., & Beilock, S. L. (2013). Math anxiety, working memory, and math achievement in early elementary school. Journal of Cognition and Development, 14(2), 187-202.
Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods, 2nd ed. Newbury Park, CA: Sage.
Ravitz, J. L., Becker, H. J., & Wong, Y. T. (2000, July). Constructivist-compatible beliefs and practices among U.S. teachers (Report # 4). Teaching, learning, and computing: 1998 National survey. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.307.315&rep=rep1&type=pdf on December 26, 2018.
Renninger, K. A. (1992). Individual interest and development: Implications for theory and practice. In K. A. Renninger, S. Hidi, & A. Krapp (Eds.), The role of interest in learning and development (pp. 361-395). Hillsdale, NJ: Erlbaum.
Renninger, K. A., & Hidi, S. (2002). Student interest and achievement: Developmental issues raised by a case study. In A. Wigfield & J. S. Eccles (Eds.), Development of achievement motivation (pp. 173-195). San Diego, CA: Academic Press.
Reeve, J. (2012). A self-determination theory perspective on student engagement. In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of research on student engagement (pp. 149-172). Boston, MA: Springer.
Reiser, R.A., & Gagné, R.M. (1983). Selecting media for instruction, Englewood Cliffs, NJ: Educational Technology.
Richey, R., Tracey, M. W., & Klein, J. D. (2011). The instructional design knowledge base: Theory, research, and practice. New York, NY: Routledge.
Riskowski, J. L., Todd, C. D., Wee, B., Dark, M., & Harbor, J. (2009). Exploring the effectiveness of an interdisciplinary water resources engineering module in an eighth grade science course. International Journal of Engineering Education, 25(1), 181–195.
244
Roblyer, M. D., & Doering, A. H. (2013). Integrating educational technology into teaching (6th ed.). Boston, MA: Pearson.
Romiszowski, A. J. (1981). Designing instructional systems: Decision making in course planning and curriculum design. London: Kogan Page, Ltd.
Ross, J. A. (1994). Beliefs that make a difference: The origins and impacts of teacher efficacy. Paper presented at the annual meeting of the Canadian Society for the Study of Education, Calgary. Retrieved from https://eric.ed.gov/?id=ED379216 on May 31, 2019.
Sadler, P. M., Sonnert, G., Hazari, Z., & Tai, R. (2012). Stability and volatility of STEM career interest in high school: A gender study. Science Education, 96(3), 411-427.
Saettler, P. (1990). The evolution of American educational technology. Englewood, CO: Libraries Unlimited.
Sahin, A., Ekmekci, A., & Waxman, H. C. (2017). The relationships among high school STEM learning experiences, expectations, and mathematics and science efficacy and the likelihood of majoring in STEM in college. International Journal of Science Education, 39(11), 1549-1572.
Sanders, M. (2009). STEM, STEM education, STEMmania. The Technology Teacher, 68(4), 20-26.
Sansone, C. (2009). What’s interest got to do with it? Potential trade-offs in the self-regulation of motivation. In J. P. Forgas, R. Baumeister, and D. Tice (Eds.), Psychology of self-regulation: Cognitive, affective, and motivational processes (pp. 35-51). New York, NY: Psychology Press.
Sarason, I. G. (1975). Anxiety and self-preoccupation. In I. G. Sarason & D. C. Spielberger (Eds.), Stress and anxiety (Vol. 2). Washington, D.C.: Hemisphere.
Schelly, C., Anzalone, G., Wijnen, B., & Pearce, J. M. (2015). Open-source 3-D printing technologies for education: Bringing additive manufacturing to the classroom. Journal of Visual Languages & Computing, 28, 226-237.
Schiefele, U. (2009). Situational and individual interest. In K. R. Wentzel & A. Wigfield (Eds.), Handbook of motivation in school (pp. 197-223). New York, NY: Taylor Francis.
Schiefele, U., & Schaffner, E. (2015). Teacher interests, mastery goals, and self-efficacy as predictors of instructional practices and student motivation. Contemporary Educational Psychology, 42, 159-171.
245
Schmidt, D. A., Baran, E., Thompson, A. D., Mishra, P., Koehler, M. J., & Shin, T. S. (2009). Technological pedagogical content knowledge (TPACK) the development and validation of an assessment instrument for preservice teachers. Journal of research on Technology in Education, 42(2), 123-149.
Schmidt, J. A., Rosenberg, J. M., & Beymer, P. N. (2018). A person‐in‐context approach to student engagement in science: Examining learning activities and choice. Journal of Research in Science Teaching, 55(1), 19-43.
Schunk, D. H. (1995). Self-Efficacy and Education and Instruction. In Maddux J.E. (eds), Self-Efficacy, Adaptation, and Adjustment - The Plenum Series in Social/Clinical Psychology (pp. 281-303). Boston, MA: Springer.
Schunk, D. H., & Mullen, C. A. (2012). Self-efficacy as an engaged learner. In S. Christenson, A. Reschly, C. Wylie (Eds.), Handbook of research on student engagement (pp. 219-235). Boston, MA: Springer.
Schunk, D. H., & Pajares, F. (2002). The development of academic self-efficacy. In A. Wigfield & J. S. Eccles (Eds.), Development of achievement motivation: A volume in the educational psychology series (pp. 15-31). San Diego, CA: Academic Press. Retrieved from: http://www.uky.edu/~eushe2/Pajares/SchunkPajares2001.PDF
Scott, A. B., & Mallinckrodt, B. (2005). Parental emotional support, science self‐efficacy, and choice of science major in undergraduate women. The Career Development Quarterly, 53(3), 263-273.
Shahzad, K., & Naureen, S. (2017). Impact of Teacher Self-Efficacy on Secondary School Students' Academic Achievement. Journal of Education and Educational Development, 4(1), 48-72.
Shulman, L. (1986). Those who understand: Knowledge growth in teaching. Educational Researcher, 15(2), 4-14.
Shulman, L. S. (1987). Knowledge and teaching: Foundations of the new reform. Harvard Educational Review, 57(1), 1-22.
Simpkins, S. D., Davis-Kean, P. E., & Eccles, J. S. (2006). Math and science motivation: A longitudinal examination of the links between choices and beliefs. Developmental psychology, 42(1), 70-83.
Smarkola, C. (2008). Efficacy of a planned behavior model: Beliefs that contribute to computer usage intentions of student teachers and experienced teachers. Computers in Human Behavior, 24(3), 1196-1215.
Snoeyink, R., & Ertmer, P. A. (2001). Thrust into Technology: How Veteran Teachers Respond. Journal of Educational Technology Systems, 30(1), 85-111.
246
Stohlmann, M., Moore, T. J., & Roehrig, G. H. (2012). Considerations for teaching integrated STEM education. Journal of Pre-College Engineering Education Research, 2(1), 28-34.
Swayne, J. M. (2017). Staying Connected: Measuring the Impact of Technology Integration on Student Engagement and Achievement at the Middle Level. Unpublished doctoral dissertation. Concordia University, Portland. Retrieved from https://commons.cu-portland.edu/cgi/viewcontent.cgi?referer=https://scholar.google.com/&httpsredir=1&article=1105&context=edudissertations
Tai, R. H., Liu, C. Q., Maltese, A. V., & Fan, X. (2006). Planning early for careers in science. Life science, 1, 0-2.
Thibaut, L., Ceuppens, S., De Loof, H., De Meester, J., Goovaerts, L., Struyf, A. ... & Hellinckx, L. (2018). Integrated STEM Education: A Systematic Review of Instructional Practices in Secondary Education. European Journal of STEM Education, 3(1), 02. https://doi.org/10.20897/ejsteme/85525
Thornburg, D., Thornburg, N., & Armstrong, S. (2014). The invent to learn guide to 3D printing in the classroom: Recipes for success. Constructing Modern Knowledge Press.
Tondeur, J., van Braak, J., Ertmer, P. A., & Ottenbreit-Leftwich, A. (2017). Understanding the relationship between teachers’ pedagogical beliefs and technology use in education: a systematic review of qualitative evidence. Educational Technology Research and Development, 65(3), 555-575.
Trust, T., & Maloy, R. W. (2017). Why 3D Print? The 21st-Century Skills Students Develop While Engaging in 3D Printing Projects. Computers in the Schools, 34(4), 253-266.
Tweed, S. R. (2013). Technology implementation: Teacher age, experience, self-efficacy, and professional development as related to classroom technology integration. Dissertation published by ProQuest. UMI 3570301. East Tennessee State University.
Unfried, A., Faber, M., Stanhope, D. S., & Wiebe, E. (2015). The development and validation of a measure of student attitudes toward science, technology, engineering, and math (S-STEM). Journal of Psychoeducational Assessment, 33(7), 622-639.
Urban, M. J., & Favlo, D. A. (Ed.). (2016). Improving K-12 STEM Education Outcomes through Technological Integration. Hershey, PA: IGI Global.
Vongkulluksn, V. W., Xie, K., & Bowman, M. A. (2018). The role of value on teachers' internalization of external barriers and externalization of personal beliefs for classroom technology integration. Computers & Education, 118, 70-81.
247
Wang, H. H., Moore, T. J., Roehrig, G. H., & Park, M. S. (2011). STEM integration: Teacher perceptions and practice. Journal of Pre-College Engineering Education Research, 1(2), 1-13. https://doi.org/10.5703/1288284314636
Wang, M. T., & Degol, J. (2013) Motivational pathways to STEM career choices: Using expectancy-value perspective to understand individual and gender differences in STEM fields. Developmental Review, 33(4), 304–340
Wang, M. T., & Eccles, J. S. (2013). School context, achievement motivation, and academic engagement: A longitudinal study of school engagement using a multidimensional perspective. Learning and Instruction, 28, 12-23.
Wang, X. (2013). Why students choose STEM majors: Motivation, high school learning, and postsecondary context of support. American Educational Research Journal, 50(5), 1081-1121.
Watson, G. R. (2006). Technology professional development: Long-term effects on teacher self-efficacy. Journal of Technology and Teacher Education, 14(1), 151-165.
Weber, J., Kotsopoulos, D., & Senger, N. (2017). Visual-spatial reasoning, design engineering, and 3D printing. In Galindo, E., & Newton, J., (Eds.), Proceedings of the 39th annual meeting of the North American Chapter of the International Group for the Psychology of Mathematics Education. Indianapolis, IN: Hoosier Association of Mathematics Teacher Educators.
Welsh, J., Harmes, J. C., & Winkelman, R. (2011). Florida’s Technology Integration Matrix. Principal Leadership, 12(2), 69-71.
Wiebe, E., Unfried, A., & Faber, M. (2018). The Relationship of STEM Attitudes and Career Interest. EURASIA Journal of Mathematics, Science and Technology Education, 14(10), em1580.
Wigfield, A. (1994). Expectancy-value theory of achievement motivation: A developmental perspective. Educational psychology review, 6(1), 49-78.
Wigfield, A., & Cambria, J. (2010). Students’ achievement values, goal orientations, and interest: Definitions, development, and relations to achievement outcomes. Developmental Review, 30(1), 1-35.
Wigfield, A., & Eccles, J. S. (1992). The development of achievement task values: A theoretical analysis. Developmental Review, 12(3), 265-310.
Wigfield, A., & Eccles, J. S. (2000). Expectancy-value theory of achievement motivation. Contemporary educational psychology, 25(1), 68-81.
248
Wigfield, A., & Meece, J. L. (1988). Math anxiety in elementary and secondary school students. Journal of Educational Psychology, 80(2), 210-216. http://dx.doi.org/10.1037/0022-0663.80.2.210
Wozney, L., Venkatesh, V., & Abrami, P. C. (2006). Implementing computer technologies: Teachers' perceptions and practices. Journal of Technology and Teacher Education, 14, 173-207.
Xie, Y., & Reider, D. (2014). Integration of innovative technologies for enhancing students’ motivation for science learning and career. Journal of Science Education and Technology, 23(3), 370-380.
Zee, M., & Koomen, H. M. (2016). Teacher self-efficacy and its effects on classroom processes, student academic adjustment, and teacher well-being: A synthesis of 40 years of research. Review of Educational research, 86(4), 981-1015.
Zhao, Y., & Frank, K. A., (2003). An ecological analysis of factors affecting technology use in schools. American Educational Research journal, 40, 807-840.
Zimmerman, B. J. (2000). Self-efficacy: An essential motive to learn. Contemporary Educational Psychology, 25(1), 82-91.
249
BIOGRAPHICAL SKETCH
Li Cheng received her bachelor’s degree in public administration in China in
2010. She found her interest in technology integration when she taught 5th grade
English in a summer program in China. In 2011, she came to West Liberty University
(WLU) in West Virginia, the United States to pursue her master’s degree in education
with the emphasis of technology integration. During her study in the master’s program,
she initiated a Chinese club on campus and served as the president. She offered free
Chinese language and culture classes and cultural events on campus. Her initiation of
the Chinese club not only promoted the language and cultural communication between
American, Chinese, and other international students but also provided her with the
opportunity to integrate technology and teach a diverse group of students.
After graduation from WLU, Li Cheng worked as a Chinese language professor
at Marietta College in Ohio. She built three Mandarin Chinese courses in the Moodle
learning management system. Using her knowledge and skills in technology integration,
she designed, developed, and implemented a series of technological resources
including animations to engage students and enhance their learning. She was also the
language lab director to design and implement learning activities by using various
technology resources. She gained much experience in teaching with innovative use of
technologies. Her abilities in working with and mentoring a diverse group of students
have also been naturally nurtured through her teaching and leadership process.
Li Cheng’s teaching at Marietta College further increased her interest in
educational technology. In August 2015, she was admitted into the educational
technology Ph.D. program at the University of Florida. In addition to fulfilling the
coursework requirements, she worked as a Graduate Instructor and a Research
250
Assistant and she passionately involved in teaching and research. She taught EME2040
Introduction to Educational Technology, a blended course with online and face-to-face
classes, to undergraduate students for three semesters. She was the leading instructor
of EME2040 for Fall 2016 and she mentored three new instructors and coordinated this
course. She also co-taught EME6606 Advanced Instructional Design, an online course,
to graduate students including Ph.D. and Ed.D. students with diverse backgrounds. She
effectively facilitated the online discussions on the weekly instructional design case
analysis.
After her teaching as an independent instructor, she had the opportunity to work
as a Research Assistant. She worked for several NSF or IES funded projects and she
was a vital member of the project teams. In addition to assisting the funded projects,
she has also been an independent and active researcher. As an instructor and a
researcher, she has always been inspired to enhance teaching and learning through
innovative use of technologies. She has conducted several studies to investigate
effective instructional strategies and learning strategies to enhance teaching and
learning in technology-enhanced learning environments. During her teaching of the
Introduction to Educational Technology course to undergraduate students, she
conducted a study on engaging students through a technology-enhanced peer feedback
activity in presentation classes. The learning activity highly engaged students and
received very good feedback from students. Almost all the students would like to do this
activity again in their future learning. To promote students’ deep learning, she
conducted an experimental study to examine the effects of two generative learning
strategies (i.e., student generated-drawing and imagination) on science text reading in a
251
computer-based learning environment and her paper received the Distinguished Paper
Award with $500 at the 2018 Annual Conference of Florida Education and Research
Association (FERA). She recently published an article entitled “Effects of student-
generated drawing and imagination on science text reading in a computer-based
learning environment” in Educational Technology Research and Development, one of
the top journals in Educational Technology field. In addition to the effectiveness of
learning strategies, she is also devoted to research on effective instructional strategies
in technology-enhanced learning environments. She conducted a meta-analysis study
on the effects of Flipped Classroom instructional strategy. This research gained her the
Best Poster Award at the 2017 Annual Conference of FERA. Her paper entitled “Effects
of the flipped classroom instructional strategy on students’ learning outcomes: A
meta‐analysis” has been published in the journal Educational Technology Research and
Development. In addition to research on learning strategies and instructional strategies
for effective teaching and learning in technology-enhanced learning environments, her
research also aims to address pressing educational problems such as reducing the
digital divide in education. She collaborated with her professor Dr. Albert Ritzhaupt on a
book chapter entitled “The Digital Divide in Formal Educational Settings: The Past,
Present, and Future Relevance”, which is in press in the prestigious Handbook of
Research on Educational and Communications Technology. Another book chapter
entitled “Using Technology to Address Individual Differences in Cognitive Processing”,
on which she collaborated with her professors Dr. Pavlo Antonenko and Dr. Kara
Dawson, is also in press in that handbook.
252
Along with her dedication to teaching and research, she has regularly presented
at international academic conferences such as American Education and Research
Association (AERA), the Association for Educational Communications and Technology
(AECT), and the Florida Educational Research Association (FERA). She has also been
devoted to providing professional services. She worked as a volunteer classroom
assistant for a 3rd-grade math inclusion class in which some students had special
needs. She served as the Graduate Student Coordinator for Florida Educational and
Research Association (FERA) for the year of 2017 and 2018. Since 2014, she has
served as an Assistant to the Editor for The Excellence in Education Journal. She
reviewed articles for the journal Computers & Education, the TechTrends journal, and
the Journal of Educational Computing Research. She has also regularly reviewed
proposals for academic conferences including AERA, AECT, and the International
Conference of the Learning Sciences (ICLS).
The excellent education and the opportunities for teaching and research have
equipped her as an independent and thoughtful instructor and researcher. She is
determined to pursue her career goal in educational technology and make contributions
to education through vigorous research and teaching.