A SSESSING GE G OALS AND C APACITIES WITH C ONCEPT I NVENTORIES : O NE P ATH F ORWARD... P ART I...
-
Upload
loreen-craig -
Category
Documents
-
view
214 -
download
1
Transcript of A SSESSING GE G OALS AND C APACITIES WITH C ONCEPT I NVENTORIES : O NE P ATH F ORWARD... P ART I...
ASSESSING GE GOALS AND CAPACITIES WITH CONCEPT INVENTORIES:ONE PATH FORWARD . . . PART I
1. Overview: General Education
2. Overview: Concept Inventories
3. Confluence: SLCI Results from S13 GE Natural Sciences Courses
4. Confluence: Math Concept Inventories for GE Quantitative Reasoning
5. Moving Forward: Next steps and segue into Part II
Assessment takes a village . . . Catherin Atkins, College of Science
Janet Bowers, MathematicsGeoff Chase, Undergraduate Studies
Douglas Deutschman, BiologyReynaldo Monzon, STAR
Chris Rasmussen, MathematicsStephen Schellenberg, Geological Science
Kathy Williams, CTL. . . with thanks to S13 GE Nat. Sci. Faculty
and Ed Nuhfer of Humboldt State
Fun facts:
~1/3 of BS/BA is GE
WASC is interested in GE
GE is integral part of mission
Goals and Capacities (G&Cs) of SDSU General Education Program
Essential Capacities Developed through General Education1. Construct, analyze, and communicate arguments2. Apply theoretical models to the real world3. Contextualized phenomena4. Negotiate differences5. Integrate global and local perspectives6. Illustrate relevance of concepts across boundaries7. Evaluate consequences of actions
Goals for Natural Sciences (Three Foundation, One Exploration)1. Explain basic concepts and theories of the natural sciences2. Use logic and scientific methods to analyze the natural world
and solve problems3. Argue from multiple perspectives about issues in natural
sciences that have personal and global relevance4. Use technology in laboratory and field situations to connect
concepts and theories with real-world phenomena
Goals for Quantitative Reasoning (One Foundation)1. Apply appropriate computational skills and use basic
mathematical concept to analyzed problems in the natural and social sciences
2. Use methods of quantitative reasoning to solve and communicate answers to real-world problems
Are these G&Cs being assessed at the
course level?If so, how?
How can these G&Cs be introduced, developed,
and demonstrated within and across courses?
How can we assessthese G&Cs at aprogram level?
One Path Forward for Programmatic Assessment andCurricular Revision is the Concept Inventory
Definition:Collection of questions designed to assess student understanding of the
foundational knowledge, concepts, and procedures for a given topic, discipline, etc.
We hypothesize that such concept inventories provide a means for programmatic assessment and curricular revision that will serve our students, faculty, and institution
Goals: • Assess scientific habits of mind and literacy
• Identify gaps in approach understanding• Inform curricular reform
Inform on multiple levels in multiple ways when linked with demographics:From synoptic overview of student population and needs to
insights from distribution of distractor responses
Concept inventories have traditionally been discipline-focused,but recent efforts include metadisciplinary assessment of science literacy
Two Example Questions from 25-Item Science Literacy Concept Inventory(SLCI of Nuhfer et al., pers. comm.)
6. To help us to understand the lunar phases, we have set up a basketball, a baseball, and a golf ball to represent respectively the Sun, Earth and the moon. What method of science are we employing?
A. Experiment. Moving the balls can allow us to measure the size of the shadow that one ball casts on another ball.
B. Modeling. Moving the balls helps us to perceive the positions of the celestial bodies that might explain the observed phases.
C. Multiple working hypotheses. Moving the balls can allow us to determine whether the lunar phases were different during the ice ages.
D. None. Moving the balls differs from reality to such an extent that it is an ineffective way to understand lunar phases.
24. Which of the following assumptions is important to all sciences?
A. Humans can understand the physical world through laws they can discover.
B. The experimental method is the only valid way to test hypotheses.
C. Life is not governed by the same physical laws as non-living systems.
D. Random events have no role in the actual physical world.
GE Capacities:
1 2 3 4 5 6 7
GE Goals (Nat. Sci.):
1 2 3 4
GE Capacities:1 2 3 4 5 6 7
GE Goals (Nat. Sci.):1 2 3 4
2,794 InitialRecords
SLCI?870(31%) No
Yes
Non-Response Bias:Female 25.2%Male 39.1%
GEOL303 9.2%BIOL100 18.3%BIOL326 43.3%ASTR310 56.4%
Native?548
(20%)No
Yes
CHEM100?216(8%) Yes
No
Missing Data67
(2%)Yes
No
1,093 Retained Records (39%)
SAT/ACT Not Reported:Native 3.8%Transfer 60.8%
Taking Foundations:Native 71.6%Transfer 31.4%
First Generation:CHEM100 29.3%All Others 19.0%
Minority:CHEM100 73.7%All Others 54.3%
Summary of Results from SCLI Deployment across Suite ofGE Foundation and Exploration Courses during early Spring 2013
Demographics Student Preparedness and Performance
SDSU “Treatment”
Age SAT (ACT) Score Student Level
Gender GPA Foundations, Exploration
Ethnicity, Race Total Units Completed
First Generation Course Modality
English as 1st Language Major (Sci., Eng., Other)
Service Area Residence Hall
Overall SLCI Performance and
Potential Explanatory Variables
Correlations of Explanatory Variables with SLCI Performance
Univariate (Unadjusted) Multivariate (R2=20.5%)Explanatory
Variable R2 F p R2 F p
Age 0.5% 1.93 0.12 0.0% 0.19 0.91
Gender <.1% 0.42 0.51 0.2% 2.25 0.13
Ethnicity / Race 4.1% 4.57 <.001 1.2% 1.63 0.09
First Generation 1.3% 7.14 <.001 0.0% 0.33 0.72
English as 1st Lang. 0.4% 4.78 0.03 0.1% 1.27 0.26
Service Area <.1% 0.15 0.70 0.2% 2.66 0.10
SAT (ACT) Score 12.6% 157 <.001 7.1% 95.3 <.001
GPA 6.3% 73.4 <.001 1.9% 25.2 <.001
Student Level 1.8% 6.70 <.001 0.1% 0.52 0.67
Found/Exploration 1.1% 11.6 <.001 0.0% 0.11 0.74
Units Completed 2.4% 27.1 <.001 0.2% 2.93 0.09
Course Modality 0.3% 1.76 0.17 0.6% 3.74 0.02
Major (Sci, Eng, Oth) 0.4% 1.98 0.14 0.1% 0.74 0.48
Residence Hall <.1% 0.19 0.66 0.1% 1.21 0.27
Correlations of SAT (ACT) and GPA with SLCI Performance
R2 = 12.6% R2 = 6.3%
Freshman Sophomore Junior Senior Total
Exploration2
(<1%)66
(21%)154
(49%)91
(29%)313
Foundation340
(44%)351
(45%)62
(8%)27
(3%)780
Aside: Snapshot of when
Foundation and Exploration
courses are being taken
by our students . . .
Implications for
scaffolding, advising, and
degree progress?
Exp
lorations
R2 = 5.2%R2 = 6.0%
Foundations
R2 = 7.0%R2 = 15.0% Patterns,
trends, and
explanatory
hypotheses?
SLCI Performance Across Courses
Course 18 19-21 22+ Sci. Major Units Earned
BIOL32x 0% 76% 24% 30% 95 (+/- 29)
ENVS100 43% 53% 4% 14% 42 (+/- 23)
Course 18 19-21 22+ Science Major
BIOL32x 0% 76% 24% 30%
ENVS100 43% 53% 4% 14%
Course 18 19-21 22+
BIOL32x 0% 76% 24%
ENVS100 43% 53% 4%
ENVS100BIOL32x
COURSE2$
700 800 900 1,000 1,100 1,200 1,300 1,400
SAT (ACT) Score
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
1.1
SL
CI
ENVS100BIOL32x
COURSE2$
700 800 900 1,000 1,100 1,200 1,300 1,400
SAT (ACT) Score
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
1.1
SL
CI
Performance difference cannot be attributed to incoming SAT scores
Performance difference may be partially explained by differences in age, major, units earned, and
other confounded variables
End-Member Comparison
Performance difference cannot be attributed to incoming SAT scores
Performance difference may be partially explained by differences in age, major, units earned, and
other confounded variables
End-Member Comparison
ENVS100BIOL32x
COURSE2$
700 800 900 1,000 1,100 1,200 1,300 1,400
SAT (ACT) Score
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
1.1
SL
CI
ENVS100BIOL32x
COURSE2$
700 800 900 1,000 1,100 1,200 1,300 1,400
SAT (ACT) Score
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
1.1
SL
CI
Univariate (Unadjusted) Multivariate (R2=35.9%)
Variable R2 F p R2 F p
Course 19.9% 27.4 <.001
Gender
SAT (ACT) Score
GPA
Student Level
Units Completed
Total 19.9%
End-Member Comparison
Performance difference cannot be attributed to incoming SAT scores
Performance difference may be partially explained by differences in age, major, units earned, and
other confounded variables
End-Member Comparison
ENVS100BIOL32x
COURSE2$
700 800 900 1,000 1,100 1,200 1,300 1,400
SAT (ACT) Score
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
1.1
SL
CI
ENVS100BIOL32x
COURSE2$
700 800 900 1,000 1,100 1,200 1,300 1,400
SAT (ACT) Score
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
1.1
SL
CI
Univariate (Unadjusted) Multivariate (R2=35.9%)
Variable R2 F p R2 F p
Course 19.9% 27.4 <.001 4.7% 8.22 0.01
Gender 0.1% 1.00 0.32
SAT (ACT) Score 3.7% 5.75 0.02
GPA 2.6% 3.93 0.03
Student Level 0.1% 0.14 0.94
Units Completed 0.1% 0.10 0.77
Total 19.9% 11.3%
End-Member Comparison
Performance difference cannot be attributed to incoming SAT scores
Performance difference may be partially explained by differences in age, major, units earned, and
other confounded variables
End-Member Comparison
ENVS100BIOL32x
COURSE2$
700 800 900 1,000 1,100 1,200 1,300 1,400
SAT (ACT) Score
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
1.1
SL
CI
ENVS100BIOL32x
COURSE2$
700 800 900 1,000 1,100 1,200 1,300 1,400
SAT (ACT) Score
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
1.1
SL
CI
Majority of the R2 is not uniquelyassignable since it is shared amongrelated variables (i.e., confounded)
End-Member Comparison
Univariate (Unadjusted) Multivariate (R2=35.9%)
Variable R2 F p R2 F p
Course 19.9% 27.4 <.001 4.7% 8.22 0.01
Gender 0.1% 1.00 0.32
SAT (ACT) Score 3.7% 5.75 0.02
GPA 2.6% 3.93 0.03
Student Level 0.1% 0.14 0.94
Units Completed 0.1% 0.10 0.77
Total 19.9% 11.3%
Demographics Student Preparedness and Performance
SDSU “Treatment”
Age SAT (ACT) Score Student Level
Gender GPA Foundations/Exploration
Ethnicity / Race Total Units Completed
First Generation Course Modality
English as a 1st Language Major (Sci, Eng, Other)
Service Area Residence Hall
A Working Model . . . Limited by the Nature of the Current Data
Next Steps: Deploy more broadly over multiple semesters to allow focuson science literacy gains through time within individuals
SLCI
Alternate or Parallel Concept Inventory?Test of Scientific Literacy Skills (TOSLS) – Gormally et al. (2012)
Alternate or Parallel Path?Test of Scientific Literacy Skills (TOSLS) – Gormally et al. (2012)
Alternate or Parallel Path for GE Natural Sciences?Test of Scientific Literacy Skills (TOSLS) – Gormally et al. (2012)
Ongoing Efforts:Programmatic: Mapping Concept Inventory skills onto GE G&Cs
Administrative: Implementing deployment across GE Nat. Sci. coursesGrass-Roots: Using course content to introduce, practice, and master GE G&C
TECH
TRADITIONAL
EXPERIENCED
FLIPPED
September 2012Pre-Test
December 2012Post Test
TECH
TRADITIONAL
EXPERIENCED
FLIPPED
From GE Natural Science to GE Quantitative Reasoning:Fall 2012 Calculus Concept Inventory
Goals: • Assess scientific habits of mind and literacy
• Identify gaps in approach understanding• Inform curricular reform
Operate and inform on multiple levels in multiple ways:Synoptic overview of student population to
insights from distribution of distractor responses
Pre-Post Test Gains:Fall 2012 Calculus Concept Inventory
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
-.100
-.050
.000
.050
.100
.150
.200
.250
16. The drawing represents a loaf of bread with a slice shown x inches from the left-hand end of the bread. Which of the following graphs could represent the volume V of the bread to the left of the slice as a function of the distance x from the left-hand end of the slice?
Example Question
Operate and inform on multiple levels in multiple ways:Synoptic overview of student population
ASSESSING GE GOALS AND CAPACITIES WITH CONCEPT INVENTORIES:ONE PATH FORWARD . . . PART I
1. Overview: General Education
2. Overview: Concept Inventories
3. Confluence: SLCI Results from S13 GE Natural Sciences Courses
4. Confluence: Math Concept Inventories for GE Quantitative Reasoning
5. Moving Forward: Next steps and segue into Part II
Assessment takes a village . . . Catherin Atkins, College of Science
Janet Bowers, MathematicsGeoff Chase, Undergraduate Studies
Douglas Deutschman, BiologyReynaldo Monzon, STAR
Chris Rasmussen, MathematicsStephen Schellenberg, Geological Science
Kathy Williams, CTL. . . with thanks to S13 GE Nat. Sci. Faculty
and Ed Nuhfer of Humboldt State
Fun facts:
~1/3 of BS/BA is GE
WASC is interested in GE
GE is integral part of mission