By Roy Clariana personal.psu/rbc4 University of Oulu, Finland EDTECH Team seminar
Semantic Map Assessment Project Overview for McREL Dr. Roy B. Clariana Penn State, Great Valley...
-
Upload
emerald-little -
Category
Documents
-
view
226 -
download
1
Transcript of Semantic Map Assessment Project Overview for McREL Dr. Roy B. Clariana Penn State, Great Valley...
Semantic Map Assessment Project Overview for McREL
Dr. Roy B. ClarianaPenn State, Great Valley
[email protected]://www.personal.psu.edu/rbc4
12/18/02
Project intent Semantic maps (SMs) are considered to be
valid and reliable measures of science content knowledge (Ruiz-Primo, Schultz, Li, Shavelson).
SMs are described as “authentic”, teachers and students use semantic maps in the science classroom as a tool to represent their understanding of that content (Novak).
The intent of this project is to establish an automatic computer system that scores semantic maps by comparing students’ maps to an “expert” map.
EARLYDAYS
Student-developed SMs
The student recalls important terms from a science lesson and drags related terms closer together and unrelated terms further apart to form clusters or categories, and then draws lines between directly related terms.
lungs
oxygenateblood
removeCO2
pulmonaryartery
pulmonaryvein
leftatrium
rightventricle
lungs
oxygenateblood
removeCO2
pulmonaryartery
pulmonaryvein
leftatrium
rightventricle
Scoring Semantic Maps To date, SMs are scored by teachers or
trained raters using scoring rubrics (Lomask) Although this marking approach is time
consuming and fairly subjective, map scores usually correlate well with more traditional measures of science content knowledge (multiple choice, fill-in-the blank, and essays)
No one has tried automatic assessment yet To automatically mark SMs, the graph is
converted into an array (matrix)
SMs can be represented by two kinds of arrays Link array – if a line (link) is used to
connect two terms, a “1” is placed in the corresponding array cell, and a “0” is used in array cells to show that there is not a link between terms.
Association array – represents the strength of relationship between pairs of terms as distances scaled from 0 (highly related) to 1 (unrelated). This array can be converted into a link array of implicit clusters equivalent to a PathFinder neighborhood.
Semantic map w/ link array
lungs
oxygenateblood
removeCO2
pulmonaryartery
pulmonaryvein
leftatrium
rightventricle
lungs
oxygenateblood
removeCO2
pulmonaryartery
pulmonaryvein
leftatrium
rightventricle
LA L OB PA PV RCO RVleft atrium (LA) 1Lungs (L) 0 1oxygenated blood (OB) 0 1 1pulmonary artery (PA) 0 1 0 1pulmonary vein (PV) 1 1 0 0 1remove CO2 (RCO) 0 1 0 0 0 1right ventricle (RV) 0 0 0 1 0 0 1
Most studies use link information,usually called “propositions”.
Semantic map w/ distance array
lungs
oxygenateblood
removeCO2
pulmonaryartery
pulmonaryvein
leftatrium
rightventricle
lungs
oxygenateblood
removeCO2
pulmonaryartery
pulmonaryvein
leftatrium
rightventricle
LA L OB PA PV RCO RV left atrium (LA) 0 0.20 0.25 0.18 0.12 0.26 0.11 Lungs (L) 0.20 0 0.06 0.14 0.17 0.07 0.17 oxygenated blood (OB) 0.25 0.06 0 0.20 0.19 0.09 0.23 pulmonary artery (PA) 0.18 0.14 0.20 0 0.23 0.14 0.07 pulmonary vein (PV) 0.12 0.17 0.19 0.23 0 0.24 0.19 remove CO2 (RCO) 0.26 0.07 0.09 0.14 0.24 0 0.20 right ventricle (RV) 0.11 0.17 0.23 0.07 0.19 0.20 0
0.17
Some use association information,usually called “neighborhoods”.
SM automatic scoring Pilot A group of 12 practicing teachers enrolled in
CI 400 at PSU completed SMs to describe the structure and function of the heart and then wrote essays on this topic from their maps.
SM “distance” data were obtained using Concept Mapper software (available at: http:/www.personal.psu.edu/rbc4/cm1.htm) that I developed last March for this purpose.
SM “link” array data were colleted by manually entering 1’s for linked terms and 0’s for unlinked terms into an Excel spreadsheet.
. . . Pilot Computer-derived LSA Essay scores were
obtained by pasting the participant’s essays into the Web-based form available at: http://www.personal.psu.edu/rbc4/frame.htm
Manually determined SM and essay scores were determined by 5 pairs of judges
Variables and correlation results are shown on the next slides
Internal variables Links (L) – arithmetic sum of all links in the map Link agreement with an expert (L/Exp) – the
arithmetic sum of the links that exactly match the expert map
Associations (A) – first convert the proposition closeness to a link array (.13 as cut-off), then the arithmetic sum of all links in the map
Association agreement with an expert (A/Exp) – convert proposition closeness to links (.13 as cut-off), then the arithmetic sum of the links that exactly match the expert map
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.00 0.05 0.10 0.15 0.20 0.25
Cut-off distance
Lin
k vs
Ass
ocia
tion
corr
elat
ion
(r)
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.00 0.05 0.10 0.15 0.20 0.25
Cut-off distance
Lin
k vs
Ass
ocia
tion
corr
elat
ion
(r)
Criterion Variables LSA Essay – Essay score established by
Latent Semantic Analysis software, using Landau and Kintsch's Web site
Semantic Map (Map) – Map scores established by averaging the scores from 6 pairs of judges using Lomask et al., 1992 rubric for assessing semantic maps
Essay – Overall essay scores established by averaging the scores from 5 pairs of judges (using a rubric)
HUMAN
Correlation matrix
Judge Metric L L/Exp A A/Exp LSA Map Essay
computer L 1
L/Exp 0.74 1
A 0.89 0.65 1
A/Exp 0.79 0.81 0.91 1
LSA 0.69 0.83 0.78 0.82 1
human Map 0.68 0.87 0.76 0.87 0.74 1
Essay 0.61 0.80 0.76 0.82 0.75 0.98 1
Computer
HumanHuman
Significant correlations shown in bold.
Maps and Essays The strongest correlation, r = 0.98, was
shown for “Map” and “Essay”, both human derived metrics.
Since the semantic maps were used as an aid in writing the essays, it seems reasonable that the two should be highly related.
This high correlation (based on human raters) provides criterion-related evidence of semantic map validity. If confirmed in later studies, science teachers may reasonably use semantic map scores for student assessment.
A/Exp The automatically derived variable
association agreement with an expert (A/Exp) was significantly correlated with the human derived “Map” (.87) and “Essay” (.82) scores, as well as with the computer derived “LSA essay” score (.82).
Thus the first pilot suggests that association agreement with an expert is a promising automatic measure of science content knowledge (like PathFinder C scores).
The automatically derived variable link agreement with an expert (L/Exp) was significantly correlated with the human derived “Map” score (.87), as well as with the computer derived “LSA essay” score (.83).
Thus, link agreement with an expert is also a promising automatic measure of science content knowledge (most extant studies involving SMs use some variant of L/Exp).
L/Exp
Links VS. associations In a multiple regression analysis of these
two automatic variables to human derived essay scores, association agreement with an expert accounted for 67% of the variance, link agreement with an expert accounted for an additional 6%, so the two together accounted for 73% of the variance in human rater essay scores (multiple r = 0.85).
So association (neighborhood) and link (propositions) information each account for some unique components of the essays.
Next steps
Field trials – Confirm pilot results; examine criterion-related validity for SMs to MC, CR, essay, and other test forms; determine cut-score approach for association arrays; find the best algorithm for score generation and automate it; improve the software, especially automating link capture; much more…
Present follow-up investigation 60 undergraduate students in intro
EdPsyc completed an instructional text on the heart, developed concept maps of the content on paper, then completed a verbatim- and an application-level multiple-choice posttest on the lesson content.
I am using the Concept Mapper software to establish the distance arrays, and the same manual procedure for link arrays.
Example concept map
Student TN-10 Concept Map
..follow-up investigation So far, data is collected, I’m establishing the arrays now, Then I will determine the cut-score for
transforming the distances to link arrays, Then, I’ll calculate correlations with the
MC tests, and finally Submit a manuscript early 2003, and use
these two as a basis for obtaining a grant for a larger field-trial and for software development
What is the potential of automatic SM assessment? Can automatic semantic map marking
support higher-level learning? Higher-level assessment?
How could teachers/districts use an automatic semantic map marking system?
What value would automatic semantic map marking have for “test” companies?
Next steps . . .
Is there a fit here at McREL?NCLB:-State standards must be developed for science by the 2005-06 school year.-Beginning in the 2005-06 school year, tests must be administered every year in grades 3 through 8 in math and reading.-Beginning in the 2007-08 school year, science achievement must also be tested.
A Roadmap to Professional Practice, Norm 5:- Choose teaching and assessment strategies that help students develop understandings of math and science.- Choose teaching and assessment strategies that are compatible to one another.- Use multiple methods and tools and systematically gathering data about students’ scientific and mathematical reasoning skills and their understandings about math and science concepts.- Incorporate ongoing, embedded, diagnostic, prescriptive, and summative assessment into instruction.- Provide opportunities for students to demonstrate their knowledge, understandings, and skills in a variety of ways.- Use the results of assessments at different levels and in a variety of ways to improve teaching and learning.- Communicate student progress to the student and his/her parent(s) or guardian.- Review assessment tasks for the use of stereotypes, offensive or irrelevant language, or assumptions that reflect the perspectives or experiences of a particular group.- Recognize that the purpose of an assessment may be different in different situations.
Next steps . . .
Is there a fit here at McREL?
If so, what are our next steps?