The Framework of Qualitative Evaluation in Online HRD of In-service Teachers Yoshida, Masami, Ph.D....
-
Upload
kory-george -
Category
Documents
-
view
215 -
download
0
Transcript of The Framework of Qualitative Evaluation in Online HRD of In-service Teachers Yoshida, Masami, Ph.D....
The Framework of Qualitative Evaluation in Online HRD of
In-service Teachers
Yoshida, Masami, Ph.D.Professor
Faculty of Education, Graduate School of Humanities and ScienceChiba University, Japan
2
Study Outline
• In-service teachers’ training– Computer room– E-Learning (TCU-LMS)– Online assignment
• Qualitative evaluation– Forth Generation– Criteria for Validity: Guba’s criteria– QSR NVivo – Interpretive Approach– Scoring Policy: TRIZ
• Action Chart
3
Teachers’ License
• Personalization of education– Small sized school– Small sized classroom– Team teaching– Increasing special education
• Need expand adaptabilities of each teacher in terms of expertise– From pre-service (number) to in-service training of
enhancement• Continuing education and HRD for teachers
– Summer vacation– Credit– License
4
Incoherence
1. Entire distance method was not permitted by stipulations2. Objective-oriented evaluation was not fairly suitable to
evaluate teacher-students learning under constructivist paradigm of education
This study argues effects of new way of absolute scale evaluation
5
The Course Details
• “Educational Methodology and Skills” : 1 credit• Use two days (9:20-16:40) • E-learning was used as prime method of learning and all le
sson contents were accessed through online (LMS: TCU-LMS, the Japanese-language version)
• The 38 teacher-students who had at least 12 years teaching experience
• Content: 15 sections, include 182 Web links (knowledge resource: 22, others are vital communication pages), 94 pages in printed volume
• Motivate teacher-students to activate potential resources & expansive learning activity through net surfing
Historical Changes of Evaluation Target
7
First Generation: Period of Educational Measurement
• 1900-• To determine potential ski
lls of people• Indicators of social skills• Used to enforce conscripti
on, arrangement of study groups in a school
• Intelligence quotient, aptitude tests, personality tests were developed
• Influence of Thorndike’s theory of learning: law of readiness (Hergenhahn & Olson, 2000)
learner
10 Widely recognized view pointsWide adaptability of results
Universal property
・Intelligence Quotient・Aptitude Test
・Personality Test
Indicators of Social Skills
First Generation
8
Second Generation: Period of Educational Evaluation
• 1930-• Expanded secondary and
tertiary education• Educational Indicators• To know changes of
students’• Needs to value
educational effectiveness• Continuous monitoring:
time series based study• Statistical methods to
judge significance• Achievement test,
standardized test• Influence of theory of
John Dewey
learner
10Time Series
Qualitative & QuantitativeOverall view
Improvement of Education
・Achievement Test・National Exam.
・Standardized Test・Inferential Statisticslearner
Time Elapsed
Educational Indicators
Second Generation
9
Third Generation: Objective Based Evaluation
• 1960-• Accountability of
educational institutes • Indicators from
curriculum • Objectives based
evaluation• Used various educational
materials and media (Stake, 1967)
• Formative evaluation and masterly learning
• Standard score (T-score) for education
Learner
10 Objectives of InstructionCurriculum Improvement
・Objective Oriented Evaluation・Formative Evaluation・Masterly Learning
・Standard Score for Education
Indicators from Curriculam
Third Generation
10
Forth Generation: Responsive Constructivist Evaluation
• 1990-• Scientific evaluation frequently le
d managerialism (Guba & Lincoln, 1989)
• Indicators from specific classroom
• Constructivist paradigm• Qualitative evaluation; Naturalisti
c inquiries• Evaluation criteria were anti-preo
rdinate, determined through observation and discussion of stakeholders
• Pay more attention on validity than reliability
Leaner
10 Qualitative AnalysisThick Description
Inductive Data AnalysisOutcomes
・Action Research・Naturalistic Inquiries
・Validity
Indicators in Specific Classroom
Forth Generation
11
Inductive Procedure
Positivism Approach (Generation 1st-3rd):Positivism Approach (Generation 1st-3rd): Evaluation = Rule (situation, action)This aims to explain action and predict another situation to define a rule that is available in other cases (Belk, Sherry Jr., & Wallendorf, 1988)Use deductive data analysis.
Interpretive Approach (Generation 4th):Interpretive Approach (Generation 4th):Evaluation = Interpretation (situation, action)This aims to interpret action and recognize specific situation.Use inductive data analysis
Methods
13
Purposive Sampling
• Final assignment: designed each student to post proposals in a Web board as homework
“Describe your proposal that improves learning of your students to be more self-directed. Especially, explain the way how you intend to apply educational methodology and knowledge of technologies.”
• Purposive Sampling– All descriptions of proposals were introduced into QSR Nvivo7, a
nd processed qualitative data analysis– Valued by stakeholders: an instructor and a coordinator– Use naturalistic inquiries, that is similar to moderation (Gipps, 199
4)– 7 x 4 treatment by seven action criteria (action chart) and four ade
quacy criteria (validity).
14
Action Criteria
All proposals were arranged into the table framework of the action chart that was postulated by Mills (2000), where the following elements are positioned.
1. Research question (findings)2. Recommended action that targeted a given finding.3. Who responsible for specific actions. (responsibility)4. Who needs to be consulted or informed about the findings
of the lesson and the associated actions?5. Who will monitor or collect the effects of actions.6. Dates when the actions and monitoring will occur.7. Any resources and media that will be needed to carry out
the action.
15
Adequacy Criteria
Each element of an action chart was reviewed based on Guba’s four criteria of the qualitative paradigm, and valued trustworthiness.
1. Credibility (CR): roughly parallel to internal validity. It poses the questions of whether the explanation fits the description and whether the description is credible. It refers teacher’s ability to take into account all of the complexities.
2. Transferability (T): roughly parallel to external validity, generalizability of explanation. This concern only case-to-case domain specific data transfer in a naturalistic qualitative study view. It is teachers’ beliefs that everything they propose is context bound.
3. Dependability (D): roughly parallel to reliability. Proposal is responsible for ensuring that the process of lesson is logical, traceable and clearly documented.
4. Confirmability (CO): roughly parallel to objectivity. Data and interpretations of the proposals are not figments of the teacher-student’s imagination, but are clearly derived from the learned contents.
16
Scoring Policy• Developed through the discussion of stakeholders• TRIZ and invention cards were used to determine the warrantable scoring policy
(Altshuller & Altov, 1994).– No.2: taking out– No.5: merging– No.40 composite materials
• Decided a way to count positive points in valid measures that were applied from lesson contents as long as negative points did not exceed positive points
• If a negative point exceeds over a positive point, we valued the action chart one level below.
Evaluation Positive Point
A 5 more
B 4
C 3
D 2
F 1 or less
Findings
Possible adequacy criteria were marked on elements.
Then, positive and negative adequacy criteria were assigned.
18
Action Chart
19
Results
• For reference, scores of achievement test were compared (Table above).– The mean of achievement test
on the evaluation levels were not significantly different at the alpha =.05 level with one-way ANOVA procedures.
• The calculation results of mean positive score by criterion (Table below).– Credibility and Confirmability
had higher score than Transferability and Dependability.
Evaluation NumberMean Achievement
Test Score
A 14 81.2
B 17 81.6
C 5 78.5
D 2 80.0
F 0 -
Evaluation-Criterion
CR T D CO
A 1.9 1.2 1.2 1.6
B 1.6 1.1 1.0 1.4
C 1.2 1.0 1.0 1.0
D 1.0 0 0 1.0
Mean 1.6 0.71 0.66 1.4
Discussion
21
Appeared Expansive Learning
• Vigorous learned things were appeared as their applied methods
• An expansive learning process in which lesson contents and information of other communities are used together for generating new idea.
• It was prominent that using e-learning and Web surfing enabled them to invoke potential information that could not be possible to overview in the traditional F2F lecture with limited knotworking.
• The author sometimes sensed insufficient descriptions – Insufficient context– Lack: details explanation, situational suitability, cares about
limitation of methodology, and involvement of other staff in the process
22
Appeared Changes
• Qualitative evaluation method was vigor for interpretation and easy to lead diagnostic opinions.
• The proposals were made in the situation of their peculiar classroom, but adequacy criteria have enabled to value their varied proposals
• Figure shows summed number of how many plural criteria were marked together in one element of an action chart– This data showed that their plans
were valid to show combined marking
Credibility55.0%
Transferability85.2%
Confirmability75.5%
Dependability72.0%
9319
11
3
12