C R E S S T / U C L A

Evaluating the DoD Presidential Technology Initiative:

Innovative Methods to Measure Student Outcomes

Davina C. D. Klein & Christina GlaubkeCRESST/UCLA

Louise YarnallCenter for Technology in Learning, SRI International

Harold F. O’Neil, Jr.CRESST/USC

Paper presented as part of symposium “Quantitative and Qualitative Strategies for Evaluating Technology Use in Classrooms”

AERA New Orleans—April 2000

C R E S S T / U C L A


2

PTI Program Background

In 1995 President Clinton set goals: computer access for students and teachers connectivity to the Internet for classrooms courseware to support quality curriculum competent teachers trained in technology

DoDEA’s response was the Presidential Technology Initiative


3

PTI Participants

PTI program implemented at 11 selected DoDEA school testbed sites across the world

Selected testbed sites required: Minimum hardware and connectivity configurations Technology implementation plans School-wide support (e.g., staff)


4

PTI Project Goals

“To develop and implement effective strategies for curriculum and technology integration”

Local site objectives included: Evaluation and alignment of courseware Development of technology integration plans Integration of software and PTI courseware

tools into the DoDEA curriculum


5

Evaluation Steps

Step 1: Identify program goals Specific expectations Our focus was on PTI students

General achievement measures Student attitudinal measures Content-specific performance measures Technology-specific performance measures


6

Examining Technology Outcomes

Classroom OutcomesIntegration of technology and curriculum

New instructional practices

System OutcomesComputers, connectivity, courseware

Professional developmentSupport for innovative teaching

Teacher OutcomesSkilled teachers

Student OutcomesIncreased performance

Better attitudes


7

Evaluation Steps (cont.)

Step 2: Describe how program plans to achieve goals Theory of Action for PTI program


8

Achieving Technology Goals

Systemoutcomes

Student outcomes

Teacheroutcomes

Classroom outcomes


9

Evaluation Steps (cont.)

Step 3: Measure intended outcomes Students’ attitudes toward technology Students’ content-specific knowledge

(focus on courseware tools) Students’ Web fluency Student-perceived classroom practices


10

Measurement Instrumentation Common measures

General impact of PTI on all students Technology Questionnaire On-line Web Expertise Assessment (WEA) Student interviews

Courseware-specific measures Detailed, courseware-by-courseware examination

of tool impact on students Content-specific performance-based assessment PTI courseware usability studies


11

Technology Questionnaire

Purpose: Measure students’ attitudes toward technology and perceptions of classroom practices 36-item paper-and-pencil survey Students rated statements on scale of 1 (“I really don’t

agree”) to 5 (“I really agree”) “I feel comfortable using computers” “In class we use computers to solve problems or answer

questions”


12

Web Expertise Assessment

Purpose: Examine effects of Web usage in the classroom Student training, then 20-minute session Presented students with authentic search tasks Asked students to navigate and search for relevant

information in a closed Web-based environment, then bookmark relevant findings

All measures logged and coded


13

WEA Search Task

Imagine you are learning about the U.S. presidents in your history class. Your teacher has asked you to write a report about what presidents said during their speeches when first elected to office. She has asked you to find out which presidents spoke of the importance of an educational system available to all without charge. Use WEA to find this information for your report. Find as many useful pages as you can. Bookmark pages by clicking on the Add Bookmark button

near the top of your screen. You may bookmark as many useful pages as you think

necessary.


20

WWW Background Questionnaire

Purpose: Evaluate students’ background knowledge regarding the World Wide Web 7-item paper-and-pencil survey Students rated statements on scale of 1 (“I

really don’t agree”) to 5 (“I really agree”) “The information on the World Wide Web is

not very useful”


21

Student Interviews

Purpose: To obtain further information about students’ attitudes toward technology and their perceptions of classroom practices Brief 5- to 10-minute interviews Three students interviewed per class Qualitative data supplements quantitative findings


22

Evaluation Participants

6 schools at 2 DoDEA sites 3 elementary schools 2 middle schools 1 middle/high school

21 classrooms

181 students participated in both pre- and posttest sessions


23

Pre-Post Comparisons

Data aggregated to PTI program intervention level (classroom)

N = 14 classrooms 4 of 21 classrooms not included because they

completed modified questionnaire due to the young age of the students

3 additional classrooms dropped because of lack of overlap between pretest and posttest samples


24

Results: Technology Questionnaire

140 students completed TQs

Two scales created Attitudes toward technology

19 items = .92

Student perceptions of classroom practices 8 items = .79


25

Attitudes Toward Technology

In general, positive attitudes held (pre/post; 1-5 scale): Students agreed it is fun figuring out how things work on a

computer (4.1/3.9) Students agreed/strongly agreed they feel comfortable using

computers (4.3/4.4) Students disagreed/strongly disagreed schoolwork on

computer is waste of time (1.5/1.6) Students agreed/strongly agreed it would be helpful to learn

how to use WWW (4.4/4.4)


26

Reported Classroom Practices

In general, limited computer use reported

High use (pre/post; 1-5 scale) Used presentations, essays, portfolios (4.1/3.7) Typed reports on computer after writing (3.8/3.6) Worked in small groups (3.6/3.4)


27

Classroom Practices (cont.)

Moderate use (pre/post; 1-5 scale) Computers used for different assignments

(3.3/3.5) Computers used to explore things (3.2/3.2)

Low use (pre/post; 1-5 scale) Computers used to practice basics (3.0/2.9) Computers used to solve problems (3.2/3.0) Many computer programs used (3.0/2.9)


28

TQ Pre-Post Comparisons

No significant differences found from fall to spring in: Attitudes toward technology

(t(13) = -0.92, p = .37) Reported classroom practices

(t(13) = -1.3, p = .21)


29

Results: WEA

142 students completed WEA Four scales created

Students’ background Web knowledge 4 items, =.77

Students’ finding ability 3 items, =.88

Students’ searching expertise 3 items, =.68

Students’ navigational strategies 2 items, =.72


30

Background Web Knowledge

In general, students familiar with Web Students were neutral/agreed that information on

WWW is accurate (3.6/3.3) Students disagreed/strongly disagreed that

information on WWW is not useful (1.6/1.8) Students disagreed that there is not a lot of

detailed or in-depth information on WWW (2.0/2.1) Students agreed/strongly agreed that WWW is

helpful in finding information (4.5/4.3)


31

Finding Ability

In general, students able to find info Average bookmark peripherally relevant to

task (2.2/2.0 on 0-3 scale) Quality of bookmark response set was good

(2.2/2.0 on 0-3 scale) About one third of pages bookmarked

appropriately (efficiency of .31/.32)


32

Searching Expertise

In general, students had difficulty searching (consistent with literature) Quality of keyword searching set rather

poor (1.6/1.6 on 0-3 scale) Number of good searches low (3.0/1.7) Students redirected searches, browsing

search output before selection (2.2/2.2)


33

Navigational Strategies

In general, students navigated well Students revisited over half the information pages

visited, orienting themselves in the Web space (7/6)

Students completed more steps, a sign of better searching (86/113)

[Use of back missing]


34

WEA Pre-Post Comparisons

No significant differences found from fall to spring in: Students’ Web knowledge (t(13) = 0.61, p = .55) Students’ finding ability (t(13) = 0.43, p = .68) Students’ searching expertise (t(13) = 0.54,

p = .60) Students’ navigational strategies (t(13) = 0.15,

p = .88)


35

Evaluation Steps (con’t.)

Step 4: Review implementation of plans If antecedents don’t occur, expected

outcomes won’t occur With technology, pay close attention to:

Hardware/software Measures of use or exposure Technology integration


36

PTI Implementation

Only 9 evaluation teachers planned to use courseware Of these 9, only 5 used courseware Courseware usage for these 5 was sparse

Teacher training/support was an issue

Student-reported classroom technology integration was weak


37

Evaluation Steps (con’t.)

Step 5: Evaluate progress toward goals No progress yet... Not surprising that we found no student

effects of the PTI program, as teacher- and classroom-level effects were not evident


38

Conclusions Need to find sensitive, innovative measures to

reveal best use of technology to instruct, assess, evaluate WEA and TQ are sample approaches

Our general approach involves: Defining where benefits are expected based on

particular high-technology environment Creating/finding innovative measures that will be

sensitive to changes within given area Ensuring that expectations required “below” or before

goal levels are being met


39

For More Information

Visit our Web site at: http://www.cse.ucla.edu/CRESST/pages/aera00.htm

Available: Overheads of this presentation Full paper And much, much more...

C R E S S T / U C L A

Documents

Transcript of C R E S S T / U C L A