Running head: ONLINE LEARNING ASSESSMENT 1
Online Learning Assessment: Self- Regulation
Maimoona H. Al Abri
George Mason University
Spring 2016
EDRS 630
Running head: ONLINE LEARNING ASSESSMENT 2
TABLE OF CONTNET
ABSTRACT…………………………………………………………………………. 3
PROBLEM…………………………………………………………………………… 5
PURPOSE AND OBJECTIVES………………………………………………………. 5
REVIEW OF LITERATURE…………………………………………………………. 7
The IMPLICATION OF ONLINE LEARNING ASSESSMENT AND SELF-
REGULATION………………………………………………………………………..
10
THE DEVELOPMENT OF GOAL SETTING AND SELF ASSESSMENT IN SELF
REULATED LEARNING……………………………………………………………..
12
ONLINE LEARNING ASSESSMENT……………………………………………….. 15
ASSESSMENT INSTRUMNETS…………………………………………………….. 17
PROPOSED SOLUTION……………………………………………………………... 22
ASSESSMENT DEVELOPMENT…………………………………………………… 23
PILOT TESTING……………………………………………………………………… 34
REFLECTIVE DISCUSSION………………………………………………………… 61
REFERENCE…………………………………………………………………………. 63
Running head: ONLINE LEARNING ASSESSMENT 3
Abstract
Self-regulation has become a crucial factor in online learning. The distance learners is the owner
of the learning process and is responsible for the regulation and control of it. The present
assessment paper aims to assess the self-regulation skills of master teachers through technology-
based training (TBT) in the Specialized Center for Professional Training of Teachers (SCPTT).
This assessment paper includes a summative test which consists of ten multiple-choice questions
and a performance task. The construct was applied to 28 individuals who have taken online
courses. The participants responded to the test voluntarily through an online survey software
namely Survey Monkey. Online learning requires students to consider their learning pathway
and how to approach it. Therefore, I measured two components of self- regulation: goal setting
and self- assessment. I statistically interpreted the results of the test by analyzing the Central
Tendency and variability, validity and reliability of the test, and the items difficulty and
discrimination. Based on these tests the examiner evaluates whether the test was valid in order
to allow decision-makers to judge the participants’ performance and thereby, decides to make an
improvement in the processes, practices, and materials that are used at the SCPTT. Further
development of the test is required. Therefore, improvement will involve changing the type of
scale and focusing on three factors of self-regulation, planning, reflection, and monitoring.
Keywords: technology-based training, self-regulation, goal setting, self-assessment
Running head: ONLINE LEARNING ASSESSMENT 4
Online Learning Assessment
Problem
The World Bank report (2012) indicated that the decline of students’ learning performance in Oman,
Middle East, was regarded to less-skilled teaching methodologies. Simultaneously, Trends in International
Mathematics and Science Study (TIMMS) revealed that Omani students demonstrated low performance in
mathematics and science assessments for the fourth and eighth grades (Mullis & Michael, 2015). As a result, the
government has made it a priority to raise the quality of education in Oman. Integrating technology in education
plays a significant role in the refinement of teaching skills and knowledge into classrooms. Consequently, a
Specialized Center for Professional Development of Teachers (SCPTT) has been established approximately two
years ago, which intends to qualify Omani teachers through experiencing teaching practices. The professional
development in the SCPTT is delivering through: (a) face to face training using a wide range of active learning
methods; (b) online learning through individual and collaborative work through discussion forums and wiki; (c)
workplace learning. These processes, collectively, are provided through an e-learning platform. Moreover,
according to the feedback of trainers of the (SCPTT), they are encountered difficulties in assessing teachers
through conducting professional development via collaborative learning environment. The data analysis of this
study revealed that there are several evaluation strategies are used in SCPTT to assess the learning performance
of trainees during professional development. They are formative assessment and summative assessment with
different assessment tools. They including pre-assessment and post-assessment, self-assessment, individual
reflection on experience, task-based learning, a written descriptive report (a case study), and peer-evaluation in
some situations. However, the finding of the study revealed that the SCPTT lacks a clear criteria of assessment.
Most participants indicated that the lack of assessment criteria refers to the absence of rubrics and checklists to
guide trainers in evaluating trainees effectively.
ONLINE LEARNING ASSESSMENT 5
Consequently, it is necessary for the (SCPTT) to have a clear criteria for online learning assessment.
Furthermore, Miller and Tucker (20155) indicated that the challenges encountered educators through online
learning are the development of online learning pedagogy and assessment instruments. Similarly, Oncu and
Cakir (2011), stated that the assessment of online learning environment (OLE) is still an issue, and the critical
issue of it is assessing the quality of learning while the learners spend a long time on learning.
In addition, online learning is a way of learning and teaching relies on the constructivism Learning
Theory. Miller and Tucker (2015) clarified that the constructivist approach to online learning, assessment and
instruction, should concentrate on the role of learners or a group of learners. Furthermore, Dabbagh and
Bannan-Ritland (2005) described that online learning promotes self-directed learning, where the online learner
has a capability to define learning needs, set learning goals, define the resources, and evaluate learning
outcomes. They pointed that the self-directed learning can be acquired by the process of self-regulation
including, for instance, goal setting and self-evaluation. Consequently, online learning can be achieved by
considering the skills and techniques of self- regulation.
Purpose and Objectives
The purpose of the paper is to assess the self-regulation skills with master teachers through technology-
based training (TBT). Specifically, the assessment focused on the metacognitive components of self-regulated
learning. The self-regulation assessment measured two scales: (a) goal setting, and (b) self-assessment. The
overall goals of the assessment are to ensuring the quality of learning; developing standards of assessment for
online courses; and attaining the validity of the assessment in order to promote decision makers in the SCPTT to
improve the weakness area in the training materials.
In addition, the desire of developing the pilot test is based on my specialization of Instructional Design
and Technology, in which I am studying different pedagogical models of learning. Also, according to my
ONLINE LEARNING ASSESSMENT 6
personal qualitative study exploring trainers’ experiences in technology-based training, to ascertain what
meaning has been manifested as a result, what are the practices and learning activities embedded in the
instruction, and how this approach connects to the constructivism learning theories, I found that trainers in the
SCPTT are lacking for standards of assessment, specifically in online learning courses. In addition, because the
SCPTT is a new project in the MOE, there is a demand to ensure the quality of learning outcomes and the
validity of the assessment.
Moreover, self-regulated learning is an important factor that controls individuals’ learning at their own
pace, but it is a challenge in assessing those learners through online courses in terms of do they have the skills
and abilities to manage their own learning, to state their goals setting, and to be self-evaluated and reflect on
their experience in order to achieve the intended skills. Law, Ge, and Eseryel (2016) stated that there are lack
of instruments have measured SR into a collaborative context. Therefore, developing a plausible instrument to
measure to what extent learners have SR skills to regulate and control their own learning through learning
communities is required. In addition, the pilot test will gain me knowledge on the process of developing an
assessment for online courses. Also, it is beneficial to improve the assessment processes in the (SCPTT) as it is
a new project. Thus, support teachers to be self- regulated learner will enhance the success of technology-
based training in the SCPTT.
Review of Literature
Theoretical Framework
Technology became complement professional development in terms of the alternative ways of
delivering the training and the interactive materials which are provided for employees. Technology-based
training constructs “student learning community” which is a notion implies the new way of gaining learning
experiences, providing new way of communication, and facilitating the interaction between students/ instructors
ONLINE LEARNING ASSESSMENT 7
(Kochtanek & Hein, 2000). “A learning community is a group of people (students and instructors) bound
together by shared learning experiences” (Kochtanek & Hein, 2000, 282). The advocacy of learning
community is to connect the ideas in classrooms with the real world. Thus, the evaluation of professional
development program is the systematic process to assess the worth, meaning and value of a programme (Ghosh
et al., 2012). Finally, gaining learning experiences through integrated technology into professional
development can be delivered by different approaches through online learning. For example, it can be delivered
through self-paced learning and collaborative learning. Those approaches require that students be capable to
regulate their own learning.
Online Learning. The widespread of employing educational technology into educational situations a
long with pedagogical learning models enhance delivering learning through different approaches. Dabbagh
defined online learning as “an open and distributed learning environment that uses pedagogical tools, enabled
by The Internet and Web-based technologies, to facilitate learning and knowledge building through meaningful
action and interaction.” (2005, p. 164). In addition, online learning is defined as a process of learning
capitalizing the instructional materials and activities which are delivered through the internet and online
technologies, and it occurs by teachers’ facilitation in helping learners to achieve their intended objectives.
Moreover, several studies (Cook & Steinert, 2013; & Milligan & Buckenmeyer, 2008) summarized the
characteristics of online learning that learners must master, into four points: (1) motivated, active and
autonomous learning, (2) supports independent learning, (c) bolsters self- regulation, and (d) increases verbal
and written communication capabilities. As a result, the learner is a self-directed learner on the online learning
environment.
Collaborative Learning. “Collaboration is a process of mutual engagement and interaction to create
shared knowledge which may not be achieved otherwise, and is the collective effort of the members that may
result in something more than the sum of its parts” (Kwan & Yunus, 2014, p.60). In addition, collaboration
ONLINE LEARNING ASSESSMENT 8
learning capitalizes the affordances of widespread social media such as discussion forums and Wiki. For
instance, Kwan and Yunus (2014) explained that Wiki supports collaborative learning, where it is a group
working and the participation can be monitored. Finally, collaborative learning encourages the group of
students to set the goals of their tasks and to monitor and evaluate their learning within the group.
Self-Regulation (SR). It is defined as a cyclic process through which students self-perform a task, plan
it, carry out, monitor, and assess their learning performance and identify the causes of the outcomes (Panadero,
Tapia, & Huertas, 2012). The researchers recognized the importance of SR by focusing on the roots of
individual differences such as self-efficacy, motivational goals, and self-assessing the performance of their own
learning process. Generally, self-regulated learning (SRL) refers to an individual’s ability to understand and
control his or her own learning environment (Dabbagh & Bannan-Ritland, 2005).
Zimmerman defined SRL is the degree in which learners are active participators in their learning process
(2008). The abilities in self-regulation learning involves goal settings, self-mentoring, self- instruction, and
self- reinforcement. Therefore, self-regulated learning (SRL) deemed self-directive processes and skills, the
learner transforms the mental ability to skills (Zimmerman, 2008). Thus, the self-regulated learning does make
less emphasize on learner’s mental ability. The formal theories of social cognition, metacognition, and
behaviorism emphasizes that the three properties (metacognitive ability, motive, behavior) are core elements for
self-regulated learning (Zimmerman, 2008). Therefore, Self- regulated learning strategies focus on three
components of content domain which are: cognition, metacognition, and motivation. The formal theory of
metacognition consists of two components: knowledge of cognition and regulation of cognition (Schwartz,
2010). It relies on three sources which are cultural learning, individual construction, and peer interaction. The
regulation of cognition refers to the activities that support learners to control their own thinking and learning. It
includes three essential skills: planning, monitoring, and evaluation. Evaluation refers to examining the
performance, products, and regulated learning processes.
ONLINE LEARNING ASSESSMENT 9
At the end, all the definitions of SRL assert the goal directed and use one or more of SRL properties, so
learners are acknowledgeable about their learning performance (Zimmerman, 2008). As a result, the construct
of SRL refers to the degree to which learners can regulate and control the aspects of belief, behavior, and
motivation through learning processes (Nicol & Macfarlane-Dick, 2006). Also, the researchers clarified that
learners in SRL require to practice different learning processes such as setting of learning goals, orientation
toward strategies that are used to accomplish goals, and management of the resources.
All of the above, self- regulation is skills which are necessary for distance learners to master them in
order to achieve the intended outcomes from learning processes.
Students Learning Community
Collaborative Learning
Self-Paced Learning
Online Learning
Self-Regulation
Self-Regulation in Students’ learning Communities
ONLINE LEARNING ASSESSMENT 10
The Implication of Online Learning Assessment and Self- Regulation
In recent years, integrating technology into teaching and learning has become a convenient pedagogical
strategy for human learning and development. Dabbagh and Kitsantas (2012) summarized how social media
supported self-regulated learning by illustrating the pedagogical framework of social media by Zimmerman's
model. For instance, students are requested to use Wiki for organizing and managing content, encouraging
collaborative editing and commenting on peer participations, and promoting self-evaluation of their learning
across time. Thus, assessing a student’s learning achievement is a fundamental aspect of instruction. However,
this assessment deemed a challenge occurs with the online learning. For instance, Kearns (2012) realized that
the significant area of focus for assessing students’ learning through online courses is the instructor’s method of
formative and summative assessment; specifically, the distribution of grades across activities for an entire
course and the provision of effective feedback. Therefore, Kearns (2012), conducted a study in two phases to
explore and examine the practices and challenges among a group of instructors delivering online courses in a
university. The construction of the online course assessment involved, in the first phase, a written assignment,
an online discussion, a fieldwork, tests, quizzes, exams, and a presentation. The second phase involved an
online focus-group session and telephone interviews with instructors of the online courses. The findings of the
study emerged with three themes: (a) the physical separation between students and instructors, (b) time
management and workload, and (c) the challenges in assessing students in communication through technology.
Kearns (2012), mentioned that a rubric becomes the effective method for assessing students’ learning in large
assignments, grading a target performance, and facilitating instructors in grading. The rubric is defined as “a
scoring guide that lists criteria against which assignment submissions will be evaluated” (p.203). Similarly,
another study was carried out to explore the method of assessing students in the process and the product of
online collaborative courses (MacDonald, 2003). The study was applied to two online courses in UK
universities which varied in the degree of integrating online collaboration in the courses. Online collaboration
ONLINE LEARNING ASSESSMENT 11
was the central form of the course which constructed the first two blocks and linked to the assessment, where
students were divided into 4-6 groups to interact with each other under the tutorial supervision, and they
engaged in online discussion and used conferences for the rest of the course. To assess the process of online
collaborative learning, students were required to submit five transcripts of their own contributions to an online
debate to demonstrate their participation individually and on peers commenting as well. The finding of
assessing collaborative learning showed that the assignment of online collaborative learning had a positive
impact on motivating students in engaging in collaboration, improved quality of contribution, and developed
online group working. Moreover, to assess the product of online collaborative learning, students were required
to submit a report of a critique of a fictional newspaper article. The other finding of the study with respect to the
product was “attracting individual marks” (MacDonald, 2003, 286). The significant aspect of the online
collaborative learning assessment was the transcript, which was deemed a useful assessment tool in several
ways. However, online collaborative assessment might lead to a lack of flexibility in the dependency on
groups. Overall, assessing learners through online learning environment is a challenge and it needs for
instruments in order to support instructors in assessing learners’ performance.
The Development of Goal Setting and Self- Assessment in Self- Regulated Learning
As suggested earlier, SRL promotes learners to practice the processes of regulation into different aspects
and support them to reflect on that practice and experiences of learning. Nicol and Macfarlane-Dick (2006)
identified that learners in SRL engage in monitoring gaps between the internal task goals and the external
outcomes that they produced. Therefore, the researcher suggested that teachers are required to build structured
opportunities for self-monitoring and assessing the progress of learning goals. The recommended activity to
promote reflection on learning progress and experience is self-assessment. In the last decades, self-assessment
has been the significant interest in higher education. It is a crucial process in self- regulation, where learners
compare their own performance and process with a certain criteria in order to understand what they achieved
ONLINE LEARNING ASSESSMENT 12
and what they need for change and to learn from the mistakes to perform better in future tasks (Panadero et al.,
2012). The criteria of self-assessment must be visible for students from the beginning of the learning process to
promote students to define the learning goals and plan accordingly and perform the intended tasks. There are
two types of self-assessment criteria: a rubric and a script. The rubric has three essential characteristics: a list of
criteria and standards to assess the goals of the tasks, a scale of grading, and a description of each qualitative
level. Thus, students can compare and grade their performance according to the rubric. Scripts “are specific
questions structured in steps to follow the expert model of approaching a task from beginning to end” (Panadero
et al., 2012, p. 807). This kind of self-assessment tasks is dependent on the goals of the task and teachers’
instructions and supports SRL.
In addition, self-regulation refers to the Self-Control Theory, highlights three core components of self-
regulation: goal setting, goal monitoring, and goal operation (Burnette, O’Boyle, VanEpps, Pollack, & Finkel,
2012). All these three processes are conceptualized and manifested into implicit theoretical literatures including
goal setting (performance goals and learning goals), goal operating (helpless-oriented strategies and mastery-
oriented strategies), and goal monitoring (negative emotions and expectations). The study examined the process
of goal setting with regard to the SOAM model to make a distinction between performance goals and learning
goals (Burnette et al., 2012). The learners who pursued the performance goal have demonstrated the ability
which is frequently relative to their peers. Whereas, learners who pursued the learning goals have demonstrated
mastering a skill, and usually it is an internal reflection of satisfaction.
Online Learning Assessment
First of all, summative evaluation is the impact of online materials after they are completed (Dabbagh &
Bannan-Ritland, 2005). Dabbagh and Bannan-Ritland (2005) indicated that determining the suitable methods of
evaluating technology-based training is a challenge that practitioners are encountered. They concluded that
evaluating online learning with traditional evaluation methods does not support the nature of online and
ONLINE LEARNING ASSESSMENT 13
distance education. As online learning is a constructivist learning, the evaluation of online learning is
developed under the constructivist evaluation. For instance, Dabbagh and Bannan-Ritland (2005) stated the
essential principles of the evaluation methods for online learning, which is established by the Pennsylvania
State University. They are: assessment instruments and activities must align with learning goals of the
delivered training through online learning environment; assessment and instrument strategies “ should be
integral parts of the learning experience—enabling learners to assess their progress, to identify areas for review,
and to reestablish immediate learning or lesson goals” (p.255); and the assessment instruments should be
aligned to the characteristics of distance learners, and the online learner should be given opportunities for
providing feedback on their learning progress.
Context. The project will be conducted in my country, Sultanate of Oman., specifically in the
Specialized Center for Professional Training of Teachers (SCPTT). The SCPTT is located in the capital city,
Muscat, and it is covered under the egis of the Ministry of Education. It is the overarching professional
development organization for the education workforce in Oman, which aims to:
• Develop the skills of teachers in order to improve students’ learning achievement and progress.
• Improve the quality of leadership at all levels so that the quality of learning will improve in all
schools.
• Provide policy-makers with information on the development of the education profession, and to
provide the MOE with data to support the management of the school system.
Overall, the SCPTT aims at accomplishing the above goals by providing an ongoing professional
development program for 8000 participants. For each participant, the program will last two years, with an
accredited Higher Education Diploma. The targeted groups in the SCPTT center are: subject expert trainers,
center associates trainers, supervisor trainers, and new teacher program trainers.
ONLINE LEARNING ASSESSMENT 14
Participants. The participants of this paper is the master teachers, who are trained in the (SCPTT) by
the center associate trainers to address areas for development in international comparative tests and basic
literacy. Master teachers including 1040 Mathematics teachers of grades 7-12 representing all cycle 2 and post-
basic schools, 520 Science teachers of grades 7-12 representing 520 cycle 2 and post basic-schools, and 1040
Arabic teachers cycle 1 representing all cycle 1 schools. Participants will receive 20 days of face-to-face
training, 140 hours of online learning, and 75 hours of workplace learning. The selected participants represent
over 20% of all teachers in their respective specializations.
Assessment Instruments
Several instruments have been developed for assessing SRL on constructs of cognition, metacognition,
and motivation. They including Learning and Study Strategies Inventory, Motivated Strategies for Learning
Questionnaires, Self-Regulated Learning Interview Scale, Metacognitions Questionnaire, Productivity
Measurement and Enhancement System, and selection, optimization, compensation (SOC).
Learning and Study Strategies Inventory (LASSI). First of all, Zimmerman (2008) demonstrated a
number of instruments measuring SRL under those constructs: LASSI, MLSQ, and SRLIS. The Learning and
Study Strategies Inventory (LASSI) consists of 80 items; self-report inventory is divided up into three domains.
It involves 10 scales to assess several skills of SRL including metacognitive (concentration, selecting main
ideas, and information processing), motivational (attitude and anxiety), and behavioral skills (time management,
study aids, self-test, and test strategies). The subscale using 5 points rating to rate the students’ responses from
not at all typical of me to very typical to me.
Motivated Strategies for Learning Questionnaires (MSLQ). The second instrument is a questionnaire
focused in motivated strategies of SRL (Motivated Strategies for Learning Questionnaires, MSLQ; Pintrich,
Smith, Carcia, & McKeachie, 1993). It is an 81-item questionnaire and is divided into two sections: learning
ONLINE LEARNING ASSESSMENT 15
strategies and motivation. Learning strategies is further divided into sections relevant to a cognitive and
metacognitive skills. The cognitive skills focused on, for instance, elaborate, rehearse, organized, and critical
thinking. The metacognitive skills include time management, environmental study, and effort management,
seeking for help, and learning with peer. The scale of this measurement was using 7-point rating from “not all
true of me” to “very true of me.”
Self-Regulated Learning Interview Scale (SRLIS). An additional instrument is Self-Regulated
Learning Interview Scale (SRLIS) to assess the cognitive, metacognitive, and behavior of SR. The study in
assessing SRL through online learning shows the interrelation of various processes in real time in terms of the
effect of goal setting on self-monitoring. From the open-ended questions, 14 codes, related to three components
of SL, have emerged. For example, students’ motivation shows self-evaluation reactions and self-
consequences. The metacognitive skills are demonstrated as goal setting and planning and environment
structure, which help seeking and monitoring that emerged to the behavioral aspect. SRLIS uses a 4-point scale
that ranges between seldom and all of the time. The questionnaires that are used to measure SRL through the
online event offer answers to how do students become masters of their own learning processes.
SR into a collaborative context. Furthermore, there was a recent study was conducted by Law, Ge,
and Eseryel (2016) to identify the dimensions of the construct of self-regulation in a collaborative context. The
construct examined the SR into a collaborative context in terms of three factors: (1) sets goals, (2) monitors, and
(3) reflects on his/her cognition through interpersonal interactions. One hundred and thirty-one undergraduate
students, who enrolled in the educational psychology course at a Midwestern university, administered the
assessment. The instrument that measured the self-regulation was developed based on “three theoretical
frameworks: self-regulation (Pintrich 2000; Zimmerman 2001), collaborative learning (Goos et al. 2002; Iiskala
et al.2011; Teasley & Roschelle 1993), and ill-structured problem solving (Jonassen 1997;Sinnott 1989; Voss &
Post 1988)” ( Law, Ge, & Eseryel, 2016, p. 6) . The items of the construct were adopted from existing self-
ONLINE LEARNING ASSESSMENT 16
regulation and problem-solving scales which are Motivated Strategies for Learning Questionnaire (Pintrich et
al. 1991) and a self-report questionnaire of ill-structured problem-solving skills (Ge, 2001). The points of the
scale was seven-point Likert scale ranging from 1 (strongly disagree) to 7 (strongly agree). Correlations among
the factors were calculated to examine the relationships among various factors. The result of the assessment
showed that students clarified and elaborated their own thoughts and their understanding of other members’
thoughts based on the behaviors are shown in the items. Moreover, the result identified that students co-
regulate in a problem-solving environment by refuting others and summarizing others’ inputs. This finding
indicated that problem solvers engaged in reflection and evaluation during the problem-solving processes.
Finally, the planning component of self-regulation was aggregated in the clarification and resolution factors.
That was because the planning related constructs were not found in the final structure.
All the above three instruments are used to assess SR in three different domains, and they are defined as
the aptitude measures for SR (Zimmerman, 2008). Zimmerman (2008) described that LASSI and MLSQ are
reflective reports, whereas SRLIS conveys multiple perspectives for hypothetical learning context.
Metacognitions questionnaire (MCQ). A common measure is used to assess the metacognition of
SRL. The metacognitions questionnaire (MCQ; Wells & Cartwright-Hatton, 2004). The measurement was
originally created by the joint effort of the University of Manchester and the Norwegian University of Science
and Technology in order to assess several dimensions of metacognition beliefs. MCQ consists of a 65-item
scale and encompasses five correlated factors that assess three domains of positive and negative metacognitive
beliefs, metacognitive monitoring, and judgments of cognitive confidence. The reliability and validity of the
MCQ-30 factors are evaluated by assessing internal consistency of the items and item-total correlations of each
factor, test–retest correlations of factors, and the correlations of factors with other scales that have been
demonstrated to correlate meaningfully with the full-scale MCQ in previous studies.
ONLINE LEARNING ASSESSMENT 17
Productivity Measurement and Enhancement System (ProMES). Another measure that is used for
goals setting is the Productivity Measurement and Enhancement System (ProMES; Pritchard, Young, Koenig,
Schmerling, & Dixon, 2012). This measure is created by the University of Central Florida. The component of
the measure is the measurement/feedback system that tracks students’ accomplishment. This measurement
consists of 4–6 objectives and 8–12 indicators. ProMES integrates learning goals in at least two ways: (1) it
cultivates the productivity and improvements for long-term processes and (2) it allows learners to focus on the
overall goals. It is a design team that consists of five to eight people, and the organizational unit consists of 5–
35 members. It uses 5-point Likert-type scales. The researchers ask the participants to respond to the
questionnaires, and then meta-analysis is collected. The approach of this measurement has both advantages and
disadvantages. The advantage is that the researchers are acknowledgeable about the study; hence, they could
provide rich information more than that the rater can provide. The disadvantage of the scale is that the
participants on the study cannot be trained and cannot be evaluated to confirm the reliability of the scale. The
major challenge of this study is the large number of variables assessed by the questionnaires and there are no
complete data to cover the questionnaires variables. As the multiple R for the WLS analysis is .77, the result of
the study is that the productivity improvements are high with the ProMES intervention.
Selection, optimization, compensation (SOC). Another measurement related to self-regulation and
goal setting is SOC, which is based on three fundamental process (selection, optimization, compensation, SOC;
Pritchard et al., 2012). SOC consists of 48 items on self- report questionnaires. Validity and reliability of the
construct are examined by subsequent studies. Validity is based on the factor structure, retest reliability, and
criterion-related validity (Pritchard et al., 2012).
In Conclusion, The above instruments have both negative and positive values. The ProMES instrument
is a comprehensive instrument, but it lacks for the reliability. SOC instrument has been examined from different
researchers and it showed a positive value for measuring SR. The MCQ instrument has been examined along
ONLINE LEARNING ASSESSMENT 18
spread several studies which showed the positive values from using it in measuring SR. The instruments,
LASSI, MLSQ and SRLIS, have been used from different studies and are comprehend in measuring different
factors of SR such as planning, monitoring, self-evaluation, and reflection. At the end, measuring SR through
learning communities is in demand in order to provide evidence how learners can control, direct and regulate
their own learning into learning processes to achieve the success.
Finally, by searching for an application of online assessment and goal setting assessment in database
(Mental Measurements Yearbook), there were 25 items according to PsychTests AIM, Inc. (2011). The items
are related to several areas; however, there are no items relevant to the area of interest, which is an online
assessment in terms of assessing self –regulation, goal setting and self-assessment. The items belong to
different skills such as management skills, leadership and sport. For instance, The Management Skills and
Styles Assessment (MANSSA) consists of 196 multiple-choice items. It covers eight skills in management, and
the scores on the MANSSA scales can be used for the purposes of identifying promising managers. The test is
administered online, and the response to each item is based on Likert scales (agree to disagree) and a progress
bar to report the result of test completion. The period of the test is 40 min. It has been reviewed by three
authors: Oswald, Reilly, and Richard . Consequently, the test that is developed during this course is required to
make a better understanding of teachers’ self-regulation learning skills in terms of their ability on setting their
learning goals and assessing themselves.
At the end, the SCPTT revealed the need for further development of instruments to assess trainees’
learning performance. This conclusion was revealed from the study was conducted in the SCPTT and reported
the lack of instruments for assessing learners through online learning. In addition, several studies showed that
assessing learning through online learning is still an issue. Finally, developing instruments to assess learning
performance using a digital medium is the focus of this paper.
Proposed Solution
ONLINE LEARNING ASSESSMENT 19
Assessing teachers in a collaborative learning through the SCPTT encountered challenges, this paper
will focus on measuring the metacognitive skills of self-regulation learning in terms of (a) goal setting, and (b)
teachers’ self-assessed abilities in order to ensuring the quality of learning; developing standards of assessment
for online courses; and attaining the validity of the assessment. The group that administered this test were the
master teachers who are trained in advanced teaching methodologies, professional skills, higher –order-
thinking skills and 21 the century skills. The overall goals are to ensure the quality of learning; developing
standards of assessment for online courses; and attaining the validity of the assessment. The number of
participants for the project was 28 participants. The pilot test was developed as a summative assessment.
Assessment Development
Constructs of the measurement
Nowadays, there are emergent models of teachers’ professional development that engaging teachers in
collaborative learning to making a meaningful shifting of teaching practices into classrooms. Collaborative
learning provides learning communities where teachers can discuss the theories’ principles and pedagogical
models, share instructional strategies and experiences into classrooms, and participate in discussion forums
collaboratively with different teachers to exchange ideas and feedback about their application into classrooms.
All those processes collectively put teachers into authentic learning that lead them to discover the learning from
multiple resources rather than only from one resource which is the instructor. Therefore, collaborative learning
of professional development of teachers is more about supporting teachers to construct a conceptual knowledge
of teaching and improving their practices in classrooms. Thereby, students’ learning performance will be
improved.
The evaluation process in the (SCPTT) is based on different aspects. They are (a) evaluating trainees,
(b) evaluating training programs, and (c) accreditation. Evaluating trainees is based on the retain of investment
ONLINE LEARNING ASSESSMENT 20
(ROI) in five different levels which are: assessing the satisfaction in face-to-face training, assessing learning
performance through online learning, assessing application in workplace, assessing students’ learning
performance, and assessing (ROI) through analysis of the impact of finance and human resources that been
invested in the project.
Conceptual Framework and Construct Measurement
Self-regulation learning emerged in the 1980s and continues to be explored by several researchers and
educators with diverse background until 21the century in order to answer the question of how students master
their own learning process at their own pace (Zimmerman, B. J., 2008). Zimmerman (2008) pointed that the
construct of self-regulation learning refers to the self-directive process which promotes learners to transfer their
mental abilities into academic skills. That leads them to be proactive learners and to have the capability to
manage and organize their own learning process rather than receiving and absorbing knowledge from the
instructor as passive learners. In addition, he explained the self-regulation learning as a social form of learning
rather than isolated learning, where learner seeks support from peers, coaches and teachers. Thereby, the self-
regulation learning emphasizes on three core elements which are: a metacognitive ability, a motivation ability,
and a behavioral ability.
Zimmerman (2000) conceptualized the SRL into three phases as a cyclic model to understand how and
why learners achieve learning. The first phase is called the forethought phase that includes task analysis (goal
setting and planning) and self-motivation beliefs (self-efficacy, task interest, goal orientation, and outcomes
expectation). The second phase is called performance phase that includes self-control (self-instruction,
imagery, and task strategies) and self-observation (self-recording and metacognitive monitoring). The third
phase is called self-reflection that emphasizes self-judgment (self-evaluation and casual attribution) and self-
reaction (satisfaction, and adaptive and defensive).
ONLINE LEARNING ASSESSMENT 21
In order to measure the students’ self-regulation in terms of goal setting and self-assessment, we will
understand how and why students achieve academically and achieve the desired goals. Also, it will explain how
students control and organize their own learning to success in their learning. In addition, measuring self-
regulation will acquire students the strategies and tools to assess their own performance and to set their learning
and performance goals. The selection of those components of SR, goals setting and self-assessment, are
essential skills for learner who learn through technology medium weather through the online learning,
independent learning, and collaborative learning. Moreover, studying trainers experiences in SCPTT showed
that instructors lack for instruments and criteria to assess their trainees’s learning progress through online
learning.
Description of the test
The test type that will be used to assess the self- regulated learning, goal setting and self- assessment, is
a summative assessment. It was conducted at the conclusion of a unit of an online learning course to summarize
the strengths and weakness of the instructions and teachers’ performance. The assumption from developing the
summative assessment of this paper is to provide information about teachers’ mastery of the training materials
after instruction and practice of the unit are completed at a given point in time. It included multiple choice
questions, and a performance task. The unit is taught within 2 months (8 weeks) through an online learning
mode. The summative test was conducted electronically through the web- based assessment via Survey
Monkey. Teachers were asked to select one response from three to four choices in the MCQ. In addition,
teachers were evaluated on the quality of contributions on the discussion forum by illustrating on the criteria of
the rubric, and they were judged on the performance task by writing a report which will focus on the quality of
their final performance based on constructed- responses. The following chart identifies the structure of the
assessment.
ONLINE LEARNING ASSESSMENT 22
Purpose for the test
The purpose of the test is to evaluate teachers’ ability in goal setting and self –assessment skills. Goal
setting skills for teachers in the (SCPTT) are assessed based on their abilities if they have set their goals in order
to perform the learning tasks? What are the strategies and practices they often do to approach their tasks through
Assessment
SR: to measure goal setting and self- assessment
Classification Schemes
Performance Task
50 Points
Numerical Scales (MCH)
50 points
Tests
Masters Teachers who are taking professional development online
Evaluation
Decision Making
ONLINE LEARNING ASSESSMENT 23
online learning? What are the criteria they rely on in order to participate in an online discussion? However, in
the performance task, the teachers are assessed if they have the skills of self-assessment in terms of do they
have reached their goals from the training programs? What do they want to improve? And where they see
themselves in future in developing their teaching skills? In general, the summative test provided insights about
the status of the students’ learning performance after teachers are completed the period of the training program.
Finally, this assessment is a critical assessment to evaluate the effectiveness of the professional training of
teachers that are designed in the (SCPTT) to determine its’ weakness and strengthens for future modification
and improvement.
Inferences and Decisions
The data of the test will be presented in tables. The information and data that will be obtained from the
test scores will show indicators about the effectiveness of the professional training programs which deliver
through the (SCPTT). That is because the content of the test’s items focused on two components of SR, goal
setting and self-assessment. Therefore, the finding of the data analysis of the test will add value to the
performance in the SCPTT. Specifically, results of the test will provide data about teachers’ performance in
terms of do they manage and organize their learning process within the determined period, what is their
feedback about the skills they learned from the training materials, are they motivated to improve their
profession in the future, how they applied knowledge and skills that they gained from the professional training
in classroom, do teachers see an improvement in students’ learning performance and their motivation by
applying different teaching methodologies, what are the kinds of the activities should be modified in the next
training materials.
According to the data which are obtained from the test scores, the (SCPTT) will make a decision to
improve the training materials of the training program, to design a variety of learning tasks that required more
ONLINE LEARNING ASSESSMENT 24
application into a classroom as well as authentic learning activities to allow teachers transfer the learning
outcomes into real- life settings. In addition, the (SCPTT) will consider adding other skills in the training
materials for self-regulation strategies to ensure that teachers are capable of continuing to develop their teaching
skills and setting their own goals after completing the training program.
The type of decision that can be made based on the obtained data is:
(a) Data Driven Decision Making: the data of the test will be used to improve the instructions relevant to the
training materials and to improve the teachers’ performance.
Report of Data
The total score of each participant was obtained to demonstrate the general level of self-regulation. The
scores for the two factors, goal setting and self-assessment, were reported independently to show students’
acquisition of features for the SR. Then, the scores were compared in order to define the level of SR among the
participants.
The data analysis of items difficulty and items discrimination were used in order to examine possible
relationship between scores of the participants in SR. Also, the reliability, Central Tendency and variability are
analyzed.
Running head: ONLINE LEARNING ASSESSMENT 25
Plan of the Test of SR (Goal Settings and Self –Assessment).
Content Type of Item Numbers of
Items
Points per item
(Weighting)
Total points
Self- regulation MCQ 10 5 50
Self-
Assessment
5 5 25
Goal Settings 5 5 25
A report PBT 1 50 50
Total 2 types 21 100
Test
Multiple choice questions of self- regulated learning (goal setting):
Read the following questions, click only the correct answer.
1. Goal setting focuses on how goals are related to job performance, which of the following represents the
practices of goal setting for your learning task:
A. Ask peers to share the timeline.
B. List all resources.
C. Take your time to make a better understanding of the task first.
ONLINE LEARNING ASSESSMENT 26
2. Once you are participating in a discussion forum to answer ill-structured problem, what skills should
you use to make accomplishment on an online learning within a period of two months?
A. Study extra materials.
B. Set goals for yourself.
C. Manage your time.
D. Study with partners
3. When you posting your response in the discussion forum, what criteria are you looking for:
A. Accuracy of your responses.
B. The aesthetic of the responses.
C. The number of responses for peer.
D. The number of words of others.
4. What are the strategies are you using to start your learning activities in the discussion forum:
A. Develop your learning plan.
B. Manage your time.
C. Study all the time.
D. Wait for the instructors’ notification.
5. What strategies are you using to evaluate your performance in the discussion forum?
A. Ask the instructor.
B. Check the criteria of the rubric.
C. Compare my participation to peer’s participation.
D. I asked questions when I don’t understand
ONLINE LEARNING ASSESSMENT 27
6. The instructor posted an assignment in the discussion forum. In order to respond to the assignment, you
have to complete all of the reading materials during 4 days only for this task. How are you going to set
you learning goal to perform the task of the assignment?
A. I always just started perform the task.
B. I can usually plan how to reach the learning goal by writing down the action plan.
C. I developed an action plan and I revised it sometimes.
D. Often, I developed an action plan on my mind so, I work on the task when I remember.
7. In addition to use a rubric to assess your performance on an online discussion, what else you will do:
A. Compare myself to peer.
B. Perform the task directly.
C. Using a rubric only.
8. Collaborative assignments through a teamwork have begun to rise organizational customers’
satisfaction. The managers have modified how they view and practice goal setting. There are unique
complications that makes goal setting in a teamwork environment more complex. In order to facilitate
team’s accomplishment toward the goals, what are the strategies the managers can provide (Redmond,
2016):
A. Ask the manager.
B. Utilize electronic dashboard.
C. Work individually.
9. When you developed the timeline of your learning task, what is the best approach to start your timeline:
A. Finish reading all materials.
B. Start developing open plan.
C. Start with checklist.
ONLINE LEARNING ASSESSMENT 28
10. What strategies often motivate you to learn?
A. The long period will be taken to study the materials.
B. The students’ motivation in the class.
C. The interactivity of the training materials.
D. The methods of learning.
Answer Key:
1. (B) List all resources.
2. (B) Set goal for yourself.
3. (A) Accuracy of your responses
4. (A) Develop your learning’s plan.
5. (B) Check the criteria of the rubric.
6. (B) I can usually plan how to reach the learning goal by writing down the action plan.
7. (C) Using the rubric only.
8. (B) Utilize electronic dashboard.
9. (C) Start with checklist.
10. (C) The interactivity of the training materials.
Performance based task:
1. As you have attended conferences, online courses, or online tutorials, Write an essay presenting the
following questions:
Have you reached your goals?
Do the skills and strategies you learned from the training materials worked with you?
What didn’t work?
ONLINE LEARNING ASSESSMENT 29
What do you want to improve?
Where do you see yourself in the future in developing your teaching skills?
Rubric
A Rubric for the report evaluation
Criteria Excellent Good Average Poor
Areas of
Improvement
Very clear strengths and
weakness, very clear
recommendations
Clear strengths and
weakness, clear
recommendations
Vague information,
some
recommendations
Absent,
Very little
recommendation
Quality of
responses
very clear responses,
illustrate examples
clear responses,
examples exist,
clear responses, no
examples.
not clear
responses, no
examples
Length of
words
3 paragraph 2 paragraph 1 paragraph few sentences
Points 50-40 30-20 20-10 10 or less
Pilot Testing
The present assessment paper is a class assignment. It aims to develop and administer a pilot test of
online learning assessment. The purposes of the assignment are to understand the assessment processes, to
analyze data from a pilot test, and make adjustments to the test. An Institutional Review Board (IRB) approval
is not required for this study, since it is not going to be published. However, an IRB approval is valuable for
students in order to practice the process. Therefore, I drafted a letter to get permission from the IRB. The
following is the draft of the letter:
ONLINE LEARNING ASSESSMENT 30
“I would like to develop an assessment about the metacognitive components of self-regulated learning in
terms of (a) goal setting and (b) teachers’ self-assessed abilities. The group that will take up the test consists of
master teachers, who are trained in advanced teaching methodologies, professional skills, higher thinking skills,
and 21st-century skills. The overall objectives of the test are (1) to ensure the quality of learning; (2) develop
standards of assessment for online courses; (3) and to attain the validity of the assessment. The number of
participants is 28. My interest in developing the pilot test is based on my specialization in Instructional Design
and Technology, which includes different pedagogical models of learning. Self-regulated learning is an
important factor that controls individuals’ learning at their own pace, but it is a challenge to assess these
learners who take up online courses in terms of whether they have the skills and abilities to manage their own
learning and also to manage their goal setting in order to achieve the intended skills. Thus, I am interesting in
understanding the assessment processes in this area. I will implement the pilot test at the Specialized Center for
Professional Training of Teachers (SCPTT) in my country, Sultanate of Omani. The purpose of focusing on the
proposed group is that these master teachers who take up training through online courses have earlier mentioned
in their feedback on a qualitative study, which was part of my doctor class assignment, that there was a lack of a
clear description of the evaluation criteria. Therefore, I am interested in evaluating their performance and
learning tasks through the online courses. Hence, developing the pilot test will help in gaining knowledge on the
process of developing an assessment for online courses and thereby would enhance my future career. In
addition, it is beneficial to improve the assessment processes at SCPTT, as it is a new project. The data of the
pilot test is only a class assignment. I also declare that I have read the documents of classroom projects, and the
main purpose for the request of this approval to this particular assignment is to administer the test in my
country, Oman, and through online medium”.
Data Collection
ONLINE LEARNING ASSESSMENT 31
The data of the study was collected online through technology medium, i.e., an online survey software.
SurveyMonkey was used as a medium for data collection. Ten multiple choice questions (MCQ) and a
performance task were developed. The content of the multiple choice questions focused on the skills and
abilities for goal setting, whereas the performance task aimed to assess participants’ abilities in assessing
themselves. The responses of the performance task were graded by using the rubric.
A link for the survey (test) was sent to participants through email and WhatsApp. The participants
responded to the survey over the course of 2 weeks. I have not received clarification from the participants
regarding the questions of the test. Table (1)
shows an example of participants’ responses
to MCQ.
Table (1)
Data Analysis and Findings
For the present measurement, the test was carried out through SurveyMonkey, and the summary data
obtained as different versions like chart, graph, or numerical table. Spreadsheet and SPSS software are used to
analyze the data of the test as frequency, scale, reliability, items difficulty and discrimination, and coefficient
correlation. Chart 1 shows an example of the analysis that was obtained from the SurveyMonkey.
Goal Setting
What strategies are you using to start your learning activities in the discussion forum:
Answer Options Response Percent Response Count
A. Develop your learning plan. 70.4% 19B. Manage your time. 18.5% 5C. Study all the time. 3.7% 1D. Wait for the instructors’ notification. 7.4% 2
answered question 27skipped question 1
ONLINE LEARNING ASSESSMENT 32
Chart (1): Individual Responses on multiple choice questions and constructed response.
ONLINE LEARNING ASSESSMENT 33
ONLINE LEARNING ASSESSMENT 34
ONLINE LEARNING ASSESSMENT 35
Chart (2) Responses of performance task.
The educational and psychological researchers contended that measurements must be conducted on a
number of variables (Johnson & Christensen, 2014). According to the summary data of the above SR
measurement, there were participants who have skipped the questions. Three participants have skipped the first,
ONLINE LEARNING ASSESSMENT 36
second, and the third questions of MCQ, and 16 participants have skipped the performance task, where only
twelve participants have reflected on it. The initial results show that these participants didn’t have the skills of
goal setting and of self-regulated learning, or else it would also be that they might have had unsuccessful online
learning experiences.
Statistics
Distributions. The following table shows raw scores and frequencies of participants’ responses.
GOAL SETTING
Frequency Percent
Valid
Percent
Cumulative
Percent
Valid 1 1 3.4 3.6 3.6
2 1 3.4 3.6 7.1
3 1 3.4 3.6 10.7
4 1 3.4 3.6 14.3
5 1 3.4 3.6 17.9
6 1 3.4 3.6 21.4
7 1 3.4 3.6 25.0
8 1 3.4 3.6 28.6
9 1 3.4 3.6 32.1
10 1 3.4 3.6 35.7
11 1 3.4 3.6 39.3
12 1 3.4 3.6 42.9
13 1 3.4 3.6 46.4
ONLINE LEARNING ASSESSMENT 37
14 1 3.4 3.6 50.0
15 1 3.4 3.6 53.6
16 1 3.4 3.6 57.1
17 1 3.4 3.6 60.7
18 1 3.4 3.6 64.3
19 1 3.4 3.6 67.9
20 1 3.4 3.6 71.4
21 1 3.4 3.6 75.0
22 1 3.4 3.6 78.6
23 1 3.4 3.6 82.1
24 1 3.4 3.6 85.7
25 1 3.4 3.6 89.3
26 1 3.4 3.6 92.9
27 1 3.4 3.6 96.4
28 1 3.4 3.6 100.0
Total 28 96.6 100.0
Missing System 1 3.4
Total 29 100.0
practices of setting goals
Frequency Percent
Valid
Percent
Cumulative
Percent
ONLINE LEARNING ASSESSMENT 38
Valid 1 9 31.0 33.3 33.3
2 10 34.5 37.0 70.4
4 8 27.6 29.6 100.0
Total 27 93.1 100.0
Missing NO
DATA
1 3.4
System 1 3.4
Total 2 6.9
Total 29 100.0
what skills should you use to make accomplishment on an online
learning within a period of two months?
Frequency Percent
Valid
Percent
Cumulative
Percent
Valid 1 16 55.2 57.1 57.1
2 10 34.5 35.7 92.9
3 1 3.4 3.6 96.4
4 1 3.4 3.6 100.0
Total 28 96.6 100.0
Missing System 1 3.4
Total 29 100.0
what criteria are you looking for:
ONLINE LEARNING ASSESSMENT 39
Frequency Percent
Valid
Percent
Cumulative
Percent
Valid 1 18 62.1 66.7 66.7
2 5 17.2 18.5 85.2
3 1 3.4 3.7 88.9
4 3 10.3 11.1 100.0
Total 27 93.1 100.0
Missing NO
DATA
1 3.4
System 1 3.4
Total 2 6.9
Total 29 100.0
strategies are you using to start your learning activities in the
discussion forum
Frequency Percent
Valid
Percent
Cumulative
Percent
Valid 1 3 10.3 10.7 10.7
2 8 27.6 28.6 39.3
3 8 27.6 28.6 67.9
4 9 31.0 32.1 100.0
Total 28 96.6 100.0
Missing System 1 3.4
ONLINE LEARNING ASSESSMENT 40
Total 29 100.0
strategies are you using to evaluate your performance in the
discussion forum
Frequency Percent
Valid
Percent
Cumulative
Percent
Valid 1 1 3.4 3.6 3.6
2 15 51.7 53.6 57.1
3 8 27.6 28.6 85.7
4 4 13.8 14.3 100.0
Total 28 96.6 100.0
Missing System 1 3.4
Total 29 100.0
How are you going to set you learning goals to perform the task of
the assignment
Frequency Percent
Valid
Percent
Cumulative
Percent
Valid 1 15 51.7 53.6 53.6
2 11 37.9 39.3 92.9
3 1 3.4 3.6 96.4
4 1 3.4 3.6 100.0
Total 28 96.6 100.0
ONLINE LEARNING ASSESSMENT 41
Missing System 1 3.4
Total 29 100.0
In addition to use rubric to assess your performance on an line
discussion, what else you will do
Frequency Percent
Valid
Percent
Cumulative
Percent
Valid 1 15 51.7 53.6 53.6
2 10 34.5 35.7 89.3
3 3 10.3 10.7 100.0
Total 28 96.6 100.0
Missing System 1 3.4
Total 29 100.0
In order to facilitate team’s accomplishment toward the goals,
what are the strategies managers can provide
Frequency Percent
Valid
Percent
Cumulative
Percent
Valid 1 6 20.7 21.4 21.4
2 3 10.3 10.7 32.1
3 19 65.5 67.9 100.0
ONLINE LEARNING ASSESSMENT 42
Total 28 96.6 100.0
Missing System 1 3.4
Total 29 100.0
When you developed the timeline of your learning task, what is
the best approach to start your timeline
Frequency Percent
Valid
Percent
Cumulative
Percent
Valid 1 6 20.7 21.4 21.4
2 5 17.2 17.9 39.3
3 17 58.6 60.7 100.0
Total 28 96.6 100.0
Missing System 1 3.4
Total 29 100.0
When you developed the timeline of your learning task, what is
the best approach to start your timeline
Frequency Percent
Valid
Percent
Cumulative
Percent
Valid 1 1 3.4 3.6 3.6
2 3 10.3 10.7 14.3
3 18 62.1 64.3 78.6
4 6 20.7 21.4 100.0
ONLINE LEARNING ASSESSMENT 43
Total 28 96.6 100.0
Missing System 1 3.4
Total 29 100.0
Measure of Central Tendency and Variability. The measure of Central Tendency was based on the
average of a set of scores with a distribution of scores. It is an average of mean, mode, and median. The
measure of variability refers to how much variation occurs among the data values. The common indexes of
variability that show the results of data analysis are range, variance, and standard deviation. Table 1 shows the
participants’ responses to the multiple-choice questions.
Table 1: Descriptive Statistics for Respondents in MCQ.
ONLINE LEARNING ASSESSMENT 44
Item Statistics
Mean Std.
Deviation
N
practices of setting
goals
2.19 1.201 26
what skills should you
use to make
accomplishment on an
online learning within a
period of two months?
1.42 .578 26
what criteria are you
looking for:
1.50 .906 26
strategies are you using
to start your learning
activities in the
discussion forum
2.77 1.032 26
strategies are you using
to evaluate your
performance in the
discussion forum
2.58 .809 26
ONLINE LEARNING ASSESSMENT 45
How are you going to
set you learning goals
to perform the task of
the assignment
1.54 .761 26
In addition to use
rubric to assess your
performance on an line
discussion, what else
you will do
1.50 .648 26
In order to facilitate
team’s accomplishment
toward the goals, what
are the strategies
managers can provide
2.50 .812 26
When you developed
the timeline of your
learning task, what is
the best approach to
start your timeline
2.38 .852 26
ONLINE LEARNING ASSESSMENT 46
When you developed
the timeline of your
learning task, what is
the best approach to
start your timeline
3.12 .588 26
According to the tables, the difference in the number of respondents between the tables of statistics and
item statistics refers to the fact that there were a few students who did not respond to all the questions. The data
analysis showed that the average mean among scores was between .690 and 1.228. The highest score was for
Q1 (2.19), and the lowest score was for Q2 (1.40). That indicates the respondents lack for time management
skills. However, they had the ability to set goals for their learning. There is a contradiction among responses
that could be a result of participants’ not understanding the questions. Hence, mean is the best measurement of
Central Tendency because it provides precise data and evidence. Also, it measures the magnitude of all scores.
However, the median is a better choice for showing “what is typical” on Central Tendency when the data are
highly skewed because of the need for describing the measures of Central Tendency, which are less affected by
extreme scores (Johnson & Christensen, 2014).
Regarding the measures of variability, the numerical index provided information that showed that the
numbers were homogenous. The standard deviation (SD) was positive across all scores, and there was no zero
(no identical scores). This shows the variability was high among scores. The SD is 8.226, and the mean is
14.50, which highlighted a large variation between them. Revision of the questions is necessary to examine the
variability.
Item Difficulty and Discrimination for Multiple Choice Items
ONLINE LEARNING ASSESSMENT 47
Item Difficulty. The item difficulty refers to the percentage of students who answered the items of the
test correctly. The percentage should be between 60% and 80%. The ratio of item difficulty has to be between 0
and 1.0. The optimal ratio is 0.5. The following table (3) shows the item difficulty.
Table (3)
Scoring
groups
Studen
t q1 q2
q
3 q4
q
5 q6 q7 q8
q
9 q10
Total
scoring%
Highes
t
1 70% 0 1 0 0 1 1 1 1 1 1
2 60% 1 0 1 0 1 1 0 1 0 1
3 60% 1 1 1 1 1 0 0 0 1 0
Total High 2 2 2 1 3 2 1 2 2 2
4 50% 1 1 1 0 0 0 0 0 1 1
5 50% 0 1 1 1 1 0 0 0 0 1
6 50% 1 0 1 1 1 0 0 0 0 1
7 50% 0 1 0 0 1 1 1 0 0 1
8 50% 1 0 1 0 1 0 0 0 1 1
9 50% 1 1 1 0 0 0 0 0 1 1
Middle 10 40% 0 1 1 0 1 0 0 0 1 0
11 40% 0 0 1 0 0 1 0 0 1 1
12 40% 0 0 1 0 0 0 1 0 1 1
13 40% 1 0 1 0 1 1 0 0 0 0
ONLINE LEARNING ASSESSMENT 48
14 40% 0 1 1 0 1 1 0 0 0 0
15 30% 1 0 0 0 1 0 0 0 0 1
16 30% 0 0 1 0 0 0 0 0 1 1
17 30% 0 0 0 0 1 1 0 0 1 0
18 30% 0 0 0 0 0 1 0 0 1 1
19 30% 0 1 0 0 1 0 0 0 1 0
20 30% 0 0 1 0 0 1 0 0 0 1
21 30% 1 0 0 0 1 0 0 1 0 0
22 30% 0 0 1 0 0 0 0 0 1 1
23 30% 0 0 0 0 0 1 0 0 1 1
24 30% 0 1 0 0 0 0 0 0 1 1
25 30% 0 0 1 0 0 0 0 0 1 1
Middle 6 6 10 1 10 8 2 1 13 14
Lowest
26 20% 0 0 1 0 0 0 0 0 1 0
27 20% 0 0 1 0 1 0 0 0 0 0
28 20% 1 0 0 0 0 1 0 0 0 0
Total Low 1 0 2 0 1 1 0 1 1 0
Discriminatio
n 0 .04 0 .04 0 .04 .04 0 0 .04
Total correct 9 8 14 2 14 11 3 4 15 16
Difficult .32 .3 .5 .1 .5 .4 .11
.1
4 .5 .6
ONLINE LEARNING ASSESSMENT 49
The correct responses to questions 3, 5, and 9 are optimal in the range of 0.5. This shows that those
items were easy for participants. The rest of the responses were in the range of (0.11) and 0(.4), which mean
the ratio is less than (>.5), and the items were difficult for several reasons. It could be that the items were too
difficult or confusing, or the items were miskeyed.
Item Discrimination. Discrimination refers to how well the item serves to discriminate between
students with high skills and knowledge of goal setting or who received the high scores to students with low
skills and knowledge of goal setting or who received the low scores. Within a group, the discrimination index
refers to the differences between a group of high-scoring students who responded with the correct answers and a
group of low-scoring students who responded with the incorrect answers. Generally, 0.30 or greater shows a
good discrimination. According to Table 3, all of the ratios showed a positive discrimination index, and the
ratio was between 0 and 1. That is, for the given test questions, the participants knew the content of the test,
and they responded with the correct answers. The responses to the items were between 0 and 0.04, which
showed a good discrimination. Finally, the responses per student are in the range of 0 and .04, which indicates
that the items of the test were not too difficult, and the students knew the content of the items; therefore, they
answered correctly. The data of the table (3) does not indicate a negative discrimination index, so almost all the
students knew the content of the test.
Reliability and Validity
Reliability. To ensure the reliability of the measure, internal consistency reliability will be used for this
construct. That is because the group of individuals was required to take the test one at a time. Therefore, the
coefficient alpha measured the reliability of the test scores. The reliability coefficient of a set of test scores is
(.330), which indicates that there is a high relationship between all responses. This result indicated that the test
was poorly developed, and it needed to be dropped or replaced. Therefore, test homogeneity is recommended to
ONLINE LEARNING ASSESSMENT 50
measure the internal consistency of the items’ scores because it provides precise test score interpretation. The
following data shows the reliability and validity of the test scores.
Case Processing Summary
N %
Cases Valid 26 89.7
Excludeda 3 10.3
Total 29 100.0
Reliability Statistics
Cronbach's
Alpha
Cronbach's
Alpha Based
on
Standardized
Items N of Items
.330 .382 10
Item Statistics
Mean
Std.
Deviation N
ONLINE LEARNING ASSESSMENT 51
practices of setting
goals
2.19 1.201 26
what skills should you
use to make
accomplishment on an
online learning within a
period of two months?
1.42 .578 26
what criteria are you
looking for:
1.50 .906 26
strategies are you using
to start your learning
activities in the
discussion forum
2.77 1.032 26
strategies are you using
to evaluate your
performance in the
discussion forum
2.58 .809 26
How are you going to
set you learning goals
to perform the task of
the assignment
1.54 .761 26
ONLINE LEARNING ASSESSMENT 52
In addition to use
rubric to assess your
performance on an line
discussion, what else
you will do
1.50 .648 26
In order to facilitate
team’s accomplishment
toward the goals, what
are the strategies
managers can provide
2.50 .812 26
When you developed
the timeline of your
learning task, what is
the best approach to
start your timeline
2.38 .852 26
When you developed
the timeline of your
learning task, what is
the best approach to
start your timeline
3.12 .588 26
ONLINE LEARNING ASSESSMENT 53
Item-Total Statistics
Scale Mean if
Item Deleted
Scale
Variance if
Item Deleted
Corrected
Item-Total
Correlation
Squared
Multiple
Correlation
Cronbach's
Alpha if Item
Deleted
practices of setting
goals
19.31 8.862 -.040 .395 .414
ONLINE LEARNING ASSESSMENT 54
what skills should you
use to make
accomplishment on an
online learning within a
period of two months?
20.08 9.194 .141 .218 .304
what criteria are you
looking for:
20.00 8.800 .074 .467 .329
strategies are you using
to start your learning
activities in the
discussion forum
18.73 8.125 .141 .251 .297
strategies are you using
to evaluate your
performance in the
discussion forum
18.92 8.394 .208 .207 .268
How are you going to
set you learning goals to
perform the task of the
assignment
19.96 7.878 .366 .570 .202
ONLINE LEARNING ASSESSMENT 55
In addition to use
rubric to assess your
performance on an line
discussion, what else
you will do
20.00 8.080 .413 .275 .203
In order to facilitate
team’s accomplishment
toward the goals, what
are the strategies
managers can provide
19.00 9.760 -.079 .426 .389
When you developed
the timeline of your
learning task, what is
the best approach to
start your timeline
19.12 8.506 .158 .501 .289
When you developed
the timeline of your
learning task, what is
the best approach to
start your timeline
18.38 9.606 .019 .320 .341
According to the above data, participants responded differently to the MCQ. The relationship among
responses indicated vast differences. Therefore, deleting items could increase the reliability of the test, but it is
ONLINE LEARNING ASSESSMENT 56
not necessary to delete items because the distance learners should have self-regulation skills. Revising items
would be much better. Adding items related to the construct will increase the reliability of the test.
Validity. In addition, to examine the validity, accuracy, and interpretation of the test scores, evidence
based on the internal structure will be used. Therefore, test homogeneity is obtained to measure the degree to
which different items measure the same construct of the test. It is obtained by calculating the coefficient alpha
for the test. Factor analysis shows the relationship and correlation among test items and determines the number
of factors and the kind of construct of the test, such as unidimensional or multidimensional, as in Table 2.
Table 2 shows that there is no consistency among the items of the test. The relationship is negative, so it is in
the opposite direction, and it does not measure one construct. Revising the blueprint of the test is required to
improve the reliability of the test.
According to the coefficient alpha, there are differences between the scores of test items. For example,
the Q1 is low >.5 (-.040), and the coefficient alphas of the rest are positive, but still low (.019) and (.366). This
shows that the items of the test might be measured within different constructs, or participants might have been
confused. Thus, examining the items that showed the low coefficient alpha led examiner to eliminate or revise
the items.
Component Matrix
Component
1 2 3 4
practices of setting
goals
.462 -.015 .633 -.267
ONLINE LEARNING ASSESSMENT 57
what skills should you
use to make
accomplishment on an
online learning within a
period of two months?
-.240 .232 .615 -.294
what criteria are you
looking for:
.575 .164 .091 .662
strategies are you using
to start your learning
activities in the
discussion forum
.261 .577 -.331 -.188
strategies are you using
to evaluate your
performance in the
discussion forum
.089 .560 -.311 .281
How are you going to
set you learning goals
to perform the task of
the assignment
.613 .545 .143 -.021
ONLINE LEARNING ASSESSMENT 58
In addition to use
rubric to assess your
performance on an line
discussion, what else
you will do
-.134 .691 .308 -.021
In order to facilitate
team’s accomplishment
toward the goals, what
are the strategies
managers can provide
-.676 .255 -.117 .194
When you developed
the timeline of your
learning task, what is
the best approach to
start your timeline
-.729 .466 .166 .073
When you developed
the timeline of your
learning task, what is
the best approach to
start your timeline
-.169 -.233 .519 .646
Result Performance Task
ONLINE LEARNING ASSESSMENT 59
The performance task involved writing a report that showed the participants’ self- assessment abilities.
The evaluation of the self-assessment task was based on a rubric. The result is shown in Table 4.
Table 4
NO Rubric Grade
1 20 Average
2 1 Poor
3 1 Poor
4 5 Poor
5 5 Poor
6 20 Average
7 5 Poor
8 10 Average
9 2 Poor
10 30 Good
11 10 Poor
12 5 Poor
The overall results showed that participants’ performances on the performance task were poor in terms
of determining the area of development, the quality of response, and the length of the report. Almost all
participants responded vague and uncertain responses, even though there were no responses provided. The
participants who responded to the performance task, they did not support their ideas with examples, and they
reflected only a few words. Participants who scored average performed better in terms of the length of the
report (a paragraph) and supported their ideas with a few examples.
ONLINE LEARNING ASSESSMENT 60
In conclusion, the data analysis of both MCQ and the performance task show variability among the
scores of the participants. Therefore, revising and improving the construct of the test is required in order to
provide the decision maker in the SCPTT an evidence for improving particular areas in the training materials
and practices of professional training.
Reflective Discussion
According to the results of the data analysis of the MCQ, the test is required for improvement from
different angels. The test will be developed with multidimensional constructs. Also, it will focus on three
factors related to SR: planning, reflection, and monitoring. Furthermore, the type of scale will be changed. The
scale will consist of a 7-point Likert scale: 1 (strongly agree) to 7 (strongly disagree). Moreover, the test will be
improved by developing the same number of response options for all items on the test. The results of the
current test identified the ineffectiveness of the test. Because it is a pilot test, it provides the examiner with
sufficient and effective knowledge and skills about developing online learning assessment. Moreover, it
informs the examiner about which skills distance learners must acquire to be successful in online learning. In
addition, as the test will be administered by the trainees in the SCPTT, the results of the test will add value to
the overall performance of professional training in the SCPTT. Specifically, learning activities and evaluation
strategies will be improved to help trainees master SR skills.
In addition, the SCPTT has an accreditation from the University of London. Thus, the teachers are
required to achieve certain standards of performance across the training programs. Therefore, adding items to
the test that focus on certain standards will support the process of accreditation.
Developing educational assessment paper must be done through systematic procedures including
developing, reviewing, administering, and also using information about educational programs undertaken for
the purpose of improving students’ learning performance and development. In fact, personally, I am new to
educational assessment field and I was unaware about the procedures of developing an assessment paper. I was
ONLINE LEARNING ASSESSMENT 61
struggling at the beginning of the course, and I decided to drop it. However, after I started reading the
materials and then pursued the classroom activities including discussion, reasoning, and sharing experiences,
then, the subject became easier. In addition, attending AERA annual meetings helped me gain deep knowledge
about the kind of assessments that could be developed related to my study area. For instance, TPACK
instrument (Technological Pedagogical Content Knowledge) can be used to assess teachers’ skills in evaluating
digital content. Furthermore, using the TPACK instrument it is possible to assess the differences in the
knowledge and skills between novice teachers and experienced teachers through taking a professional
development. Another lesson that I learned is to use an instrument called STAAR to assess the self-efficacy of
teachers in terms of personal efficacy and teaching efficacy. Readiness was observed in mathematics teachers
for using this technology in order to develop students’ performance in mathematics.
I faced a challenge in developing the assessment paper. The difficulty was occurred during analyzing
and discussing the data. That is because I have not taken the statistics and quantitative courses so, it was
difficult for me. Overall, it has been a good experience to practice the whole process of the test development.
Finally, the concept of developing measurement instruments is very clear and I understood the flaws of
the current constructs. Therefore, I will rely on the standards of education and psychology in developing the
items of the test. Also, I will use the Likert-point scale, and will study the content of the construct profoundly.
As I have enough background about the content of the test, I can develop a good and effective test.
In my opinion, developing an assessment paper is difficult for a new student, but it is beneficial to
experience the process of developing it. In addition, self-regulation is a new notion, so it was an excellent
opportunity for me to understand the background, theoretical framework, and studies that was done around it.
Moreover, I gained knowledge about the theory of SR and the application and implication of SR in online
learning environments. I am now confident in my ability to measure different topics related to educational
technology.
ONLINE LEARNING ASSESSMENT 62
Running head: ONLINE LEARNING ASSESSMENT 63
References
Brookhart, S. M., & Niko, A.J. (2015). Educational assessment of students. Upper Saddle River,
NJ: Pearson Education, Inc.
Johnson, R. B., & Christensen, L. (2014). Educational research: Quantitative, qualitative,
and mixed method. Los Angeles, LA: SAGE Publication, Inc.
Cook, D. A., & Steinert, Y. (2013). Online learning for faculty development: A review of the
literature. Medical Teacher, 35(11), 930-937. doi:10.3109/0142159X.2013.827328
Burnette, J. L., O'Boyle, E. H., VanEpps, E. M., Pollack, J. M., & Finkel, E. J. (2013). Mind-sets
matter: A meta-analytic review of implicit theories and self-regulation. Psychological
Bulletin, 139(3), 655-701. doi:10.1037/a0029531
Dabbagh, N., & Bannan-Ritland, B. (2005). Online learning: Concepts, strategies and application. Upper
Saddle River, NJ: Prentice Hall.
Dabbagh, N. & Kitsantas, A. (2012). Personal learning environments, social media, and self-
regulated learning: A natural formula for connecting formal and informal learning. The Internet and
Higher Education, 15(1), 3-8. doi:10.1016/j.iheduc.2011.06.002
World Bank. 2013. Main report. Washington DC: World Bank.
http://documents.worldbank.org/curated/en/2013/01/17406411/education-oman-drive-quality-vol-2-2-
overview
Ghosh, P., Jagdamba, R. S., Rashmi, P. J., & Singh, R. P. (2012). Towards more effective
ONLINE LEARNING ASSESSMENT 64
training programmes: A study of trainer attributes. Industrial and Commercial Training, 44(4), 194 –
202. doi: http://dx.doi.org.mutex.gmu.edu/10.1108/00197851211231469
Kearns, L. R. (2012). Student assessment in online learning: Challenges and effective
practices. Journal of Online Learning and Teaching, 8(3), 198-208. Retrieved from
http://search.proquest.com.mutex.gmu.edu/docview/1500356017?accountid=14541
Kochtanek, T. R., & Hein, k. k. (2000). Creating and nurturing distributed asynchronous
learning environments. Online information review, 24(4), 280-294. Retrieved from
http://search.proquest.com.mutex.gmu.edu/docview/194493715?accountid=14541
Kwan, L. S. L., & Yunus, M. M. (2014). Group participation and interaction in ESL wiki
collaborative writing among malaysian gifted students. Asian Social Science,11(2),
59-68. doi:10.5539/ass.v11n2p59
Law, V., Ge, X. & Eseryel, D. (2016). The Development of a self-regulation in a collaborative
context scale. Springer Science+Business Media, 21, 1-12. doi: 10.1007/s10758-016-9274-z
MacDonald, J. (2003). Assessing online collaborative learning: Process and product. Computer
& Education, 40(4), 377–391. doi:10.1016/S0360-1315(02)00168-9
Miller, J.W., & Tucker, J. S. (2015). Addressing and assessing critical thinking in intercultural
contexts: Investigating the distance learning outcomes of military leaders. International Journal for
Intercultural Relations, 48, 120-136. doi:10.1016/j.ijintrel.2015.07.002
Milligan, A. T. & Buckenmeyer, J. A. (2008). Assessing Students for Online Learning.
ONLINE LEARNING ASSESSMENT 65
International Journal on E-Learning, 7(3), 449-461. Retrieved from
http://search.proquest.com.mutex.gmu.edu/docview/210337772?accountid=14541
Oncu, S., & Cakir, H. (2011). Research in online learning environments: Priorities and
methodologies. Computer and Education, 57(1), 1098–1108. doi:10.1016/j.compedu.2010.12.009
Redmond, B. F. (2106, Feb 20). Goal Setting Theory. Atlassian News. Retrieved from
https://wikispaces.psu.edu/display/PSYCH484/6.+Goal+Setting+Theory
Nicol, D.J. & Macfarlane‐Dick, D. (2006). Formative assessment and self‐regulated learning: A
model and seven principles of good feedback practice. Studies in Higher Education, 31(2), 199-218. doi:
10.1080/03075070600572090
Panadero, E, Tapia, J. A., Huertas, J. A. (2012). Rubrics and self-assessment scripts effects on
self-regulation, learning and self-efficacy in secondary education. Learning and Individual References,
22, 806-813. doi:10.1016/j.lindif.2012.04.007
Pritchard, R. D., Young, B. L. Koenig, N., Schmerling, D., & Dixon, N.W. (2012). New
development in goal setting and task performance. Florida, FL. Routledge.
Wells, A. Cartwright-Hatton, S. (2004). A short form of the metacognitions questionnaire:
Properties of the MCQ-30. Behavior Research and Therapy, 42, 385–396. doi:10.1016/S0005-
7967(03)00147-5
Zimmerman, B. J. (2008). Investigating self-regulation and motivation: Historical background,
ONLINE LEARNING ASSESSMENT 66
methodological developments, and future prospects. American Educational Research Journal, 45(1),
166–183. doi: 10.3102/0002831207312909
Running head: ASSESSING ONLINE LEARNING ENVIRONMENT 67
Top Related