Evaluating Technology’s Impact on Teaching and Learning
Educational Technology Integration
Contact Information
[email protected] 978-453-3070
www.sun-associates.com/eval/sample This presentation is linked to that page
Where Do We Stand?
Hot topics Your District State National
Your expectations for today
Workshop Goals
To overview a scheme for creating a formative evaluation of technology’s impact
To review related issues and developments visa vis our experience in a particular school district (Fayette County, KY)
To create an action plan for conducting a similar assessment in your own district
A Framework for Review
From Designing ProfessionalDevelopment for Teachers of Scienceand Mathematics, Loucks-Horsley,Hewson, Love, and Stiles. CorwinPress Inc. 1998
Why Evaluate?
To realize your investment in technology What sort of “difference” has all of this
technology made?
To continue your school or district technology planning effort Evaluation is a key component of ongoing
technology planning!
Evaluation Starts with Technology Goals
Evaluation should be rooted in a strong educational technology plan which... Is more than an infrastructure plan Focuses on technology’s impact on teachers
and students Has clear goals and objectives for what you
want to see happen
Skeletal Plan Format
Acitivity 1 Activity 2
Goal 1
Activity 1
Goal 2
Curricular Goals
Activity 1
Goal 1
Activity 1 Activity 2
Goal 2
Staff Development Goals
Activity 1
Goal 1
Activity 1
Goal 2
Community Goals
VISION
Your Goals?
Using the Plan Mapping Worksheet, map your plan in terms of vision, goals, and actions.
Goals Lead to Questions
What do you want to see happen? These are your goals
Achieving these goals requires a process Are you performing the process steps?
All of this can be measured through a formative evaluation
The Evaluation Process
Create meaningful evaluation questionsDesign relevant indicatorsOrganize those indicators in a performance
rubric which can be used to measure progress and achievement
Collect data to score the rubricsCreate a report which applies the
evaluation in a formative manner
Take a break...
FCPS Example Questions
How have students been impacted by technology integration?
Are teachers using technology in ways that match district goals for technology and use the potential for instructional technology?
Have we adequately allocated district technology resources?
Processes of Technology Evaluation
What are you looking for? Ways to make the implementation of
technology better Teaching and learning impacts — this is not
about counting machines!
Developing Indicators
What is it that you want to measure? Impact on teaching and learning Change Progress towards desired outcomes
You need indicators for progress to be measured
Indicators should reflect your school’s unique goals and aspirations Arise from your plan Rooted in your vision and goals Indicators must be indicative of your unique
environment...what constitutes success for you might not for someone else.
Indicators of Engaged Learning
Engaged learners are Responsible for their own learning Strategic problem-solvers Energized by learning Collaborative learners
These are all observable characteristicsThese indicators can help inform your
student impact rubric
Levels of Teacher Appropriation
Stage Teacher BehaviorEntry Teachers learn the "basics" of using new
technology.Adoption Teachers use new technology to support
traditional instructional methods.Adaptation Teachers integrate new technology into
traditional classroom practice, focusing onincreased student productivity andengagement through the use of tools such asword processors, spreadsheets, and graphicstools.
Appropriation Teachers focus on cooperative, project-based,and interdisciplinary work which incorporatestechnology as needed and as one of manytools.
Invention Teachers discover new uses for technologytools often by designing projects that combinemultiple technologies.
Adapted from "A Report on 10 Years of ACOT Research" (Apple Computer, 1996)
Point is... There are things we can look for in terms of student and teacher behavior.
The goal in creating indicators is to link aspects of behavior to levels of achievement
What’s Realistic?
When developing questions and indicators, what makes sense visa vis the research on technology’s potential impacts?
www.sun-associates.com/eval/resources.html
What the Research Says
CBI generates measurable increases in mastery of core content Kulik’s meta-analysis Fletcher’s evaluation of military training
Pace, repetition, and standardization promote memorization of “facts”
Kulik’s work shows that CBI did virtually nothing in terms of changing student “attitude” toward subject matter
Productivity tools -- particularly tools for writing -- encourage students to produce more and pay more attention to the quality of their work. This more free-form use of technology cuts
directly to the heart of the connection between attitude toward learning and student achievement
Students will do more of what they like to do
Technology improves the learning environment Motivates students Makes learning more exciting Makes learning more relevant to real-world
tasks
None of these benefits come solely through technology!
It gets more gray when considering “cognitive uses” Cooperative and collaborative tasks Communication Problem solving Using technology as a key element of a rich,
supportive, educational environment
Student Performance in Rich Educational Environments
Higher rates of attendance and graduation (ETS report)
Higher qualification rates for honors programsGreater development of critical thinking and
problem-solving skillsGenerally higher scores on traditional
assessmentsLower numbers of disciplinary incidents
These benefits have been observed/documented specifically in those classrooms where technology has been a key intervention...but never the only intervention!
What Other Interventions?
More teacher training Usually in relation to using the technology tools
as part of an overall reformed instructional environment
More individualized, student-centered, learning Usually as a result of problem-based,
cooperative learning environments where technology is the catalyst for change
So, What Changes?
The student outcomes from participation in a technology-enriched environment are indeed positive
Measures are usually in terms of new things that can be done versus increases on traditional assessments
Nevertheless, traditional assessments do show improvement...it just occurs over time and is part of a broader, system-wide, picture of change.
Try a Sample Indicator
Using the Developing Indicators worksheet, let’s develop a few sample indicators Work in several groups Select someone to report out Try to develop some consensus on indicators
which would work across districts
FCPS’s Indicators
Use of technology positively impacts and fosters the student’s motivation to engage in learning practices that lead to new ways of thinking, understanding, constructing knowledge, communicating results, and acquiring basic skills
The faculty and staff are proficient, knowledgeable, and current with available technology and translate that knowledge into relevant learning opportunities for students
Teachers create learning opportunities and physical environments that allow students to assume more independent roles in their own learning through their use of technology.
All teachers and learners throughout the district have sufficient access to technology-based productivity tools, on-line services, media-based instructional materials, primary sources of data, and adequate support for using these resources so as to enrich and extend their learning goals.
Rubrics
Student AchievementQuestion How has technology positively impacted student achievement?
BasicIndicator
Use of technology positively impacts and fosters the student’s motivation to engage in learning practices thatlead to new ways of thinking, understanding, constructing knowledge, communicating results, and acquiringbasic skills
Level 4 Technology is used routinely and seamlessly as a tool for learning. Technology positively impacts and fostersthe student’s motivation to engage in learning practices that lead to new ways of thinking, understanding,constructing knowledge, communicating results, and acquiring basic skills. Technology is used to inspirecritical thinking and the solving of problems relevant to real-life skills with recognition of the tradeoffs inherent inthe application of technology in society. Student work parallels the way in which professionals in the work forceuse technology.
Students exercise a high degree of personal judgement in the choice and application of technology to theirlearning. Teachers support this choice by designing and facilitating a student-centered learning environmentwhich makes use of a wide variety to technology tools.
Level 4Evidence
Technology is used in all of the ways documented in the previous levels.
In addition, the choice of technology tools used in learning is primarily student directed. Students makeappropriate choices with regard to when and where to use technology. For example…
1. The student exercises personal judgement in the maximum and most appropriate use of technology toassist problem solving, reasoning, and thinking.
2. Students demonstrate critical thinking and media literacy skills in the use of technology as a research tool.
www.fayette.k12.ky.us/central/edtech/suneval.asp
Level 3 Students regularly use technology within their learning activities. Fluent use of technology positivelyimpacts and fosters the student’s motivation to engage in learning practices that lead to new waysof thinking, understanding, constructing knowledge, communicating results, and acquiring basicskills.
Use of technology is equally student directed as it is teacher directed. Students exercise somedegree of personal choice and judgement in their use and application of technology to learning.
Level 3Evidence
As for Level 2, but considerably more advanced uses of technology are demonstrated. Forexample...
1. Students use technology tools to solve problems which require the organization and analysis ofdata (i.e., a graphing calculator, spreadsheet, database, etc.)
2. Students use software to create presentations to communicate effectively (i.e. PowerPoint slideshows, etc.)
3. Students use email to contact experts and communicate with peers about a specific area ofinterest in order to expand knowledge.
Level 2 Students use some technology nearly every day in relation to learning activities. The primaryemphasis of technology use is still in the acquisition of basic skills, but there is evidence that thestudent is beginning to use technology to engage in learning practices that lead to new ways ofthinking, understanding, constructing knowledge and communicating results.
The vast majority of student technology use is teacher-directed, with students assigned to useparticular technology tools, applications, or resources as part of their academic work.
Level 2Evidence
As with Level 1, except technology use becomes more routine and regular. More advanced usesbegin. For example…
1. Students use a word processor or appropriate software to enhance the organization ofproducts, and to improve quality (i.e., cut-paste, spell check, grammar checking, etc)
2. Students use technology to solve problems (i.e., a graphing calculator, spreadsheet, database,etc.)
3. Students use technology to improve problem solving, reasoning, and thinking.4. Students use application programs to discover concepts and relationships, especially in
science, math, and social studies.5. Students use the Internet (download and copy, know useful web sites, and use search
engines) and appropriate technologies, such as CD-ROM encyclopedias, as research tools.
Level 1 The student’s initial use of technology supports the acquisition of basic skills and increasedproductivity.
All uses of technology are teacher-directed. Teachers weave the use of some technology toolsand devices into student lessons and activities.
Level 1Evidence
1. Students use calculators, games, tutorial, and application programs to practice and acquirebasic skills and to increase the depth of understanding of particular subject area curriculum.
2. Students use word processors to write (i.e., cut-paste, spell check, grammar checking, etc) andproduce products.
3. Students demonstrate the knowledge of basic application/productivity software such asspreadsheets, databases, and presentation tools.
4. Students begin to explore the Internet and become acquainted with email.
Rubric Tips
Even number of levelsCreate from ideal to lesser Yes, Yes But, No But, NoBe as descriptive as possibleDo not get excessively quantitative
Take a break...
FCPS Rubric Scores
0
0.5
1
1.5
2
2.5
3
3.5
4
Indicators
Student Impact
Teacher Fluency
Teacher Use withStudentsTeacherProductivityAdministratorSupportInfrastructureAllocation
Student Impact, 2
Technology is being used daily as a basic tool within a range of traditional learning activities
Some evidence that technology is supporting new ways of thinking and learning
Most student technology use is very teacher-directed and occurs in settings outside of the classroom
Teacher Fluency
Teacher Fluency = 2.5 Teachers are reasonably fluent, but need to
do more to connect technology to the core curriculum
Use of Technology with Students = 2.25 Teachers direct students to use technology,
but need to use technology to empower students as independent, engaged, learners
Teacher Productivity = 4 Teachers are very proficient in using
technology as a tool for personal and professional productivity
Administrator Support = 2 Administrators use technology, but need to
advocate and support more integrated uses among their teaching staff
Infrastructure Allocation, 2.75
The basic infrastructure -- hardware, network, software -- is complete and adequate
Individuals in Technology Support functions (schools and district) are doing an exemplary job and should receive much of the credit for the advances we have seen thus far
Schools need even more technology resource teacher support
Efforts must continue to map technology on to the core curriculum
Sum It All Up?
Fayette County is doing very well! Based on our research nationwide, FCPS is
clearly above average in most areas of its technology implementation
Teachers are using technology in ways that are consistent with their overall instructional approach Directed instruction Technology as a “skill” versus an integrated
element of the learning process
Technology can do more, and teachers can do more Professional development and increased
levels of resource teacher support will make this happen
The full report contains a number of recommendations for professional development and policy designed to positively impact performance on the indicators
To Summarize...
Start with your planFrom your goals, develop indicators and a
performance rubricAccumulate data and/or develop your data
collection tools How will you collect the data necessary to
document your stage of fulfillment?
Data Collection
Review Existing Data Current technology plan Curriculum District/school improvement plans
www.sun-associates.com/eval/sampleCreate a checklist for data collection
Collect New Data Focus Groups/Interviews
TeachersParentsStudentsAdministratorsOther stakeholders
Classroom Observations
Surveys Creating good surveys — what do you want to
know• length• differentiation (teachers, staff, parents, community, etc..)• quantitative data• attitudinal data• timing/response rates (getting returns!)
www.sun-associates.com/eval/samples/samplesurv.html
Solicitation of teacher/parent/student stories This is a way to gather truly qualitative data What does the community say about the use
and impact of technology?
Compile the reportDetermine how to share the report
School committee presentation Press releases Community meetings
Conclusion
Build evaluation into your technology planning effort
Remember, not all evaluation is quantitative
You cannot evaluate what you are not looking for, so it’s important to —
Develop expectations of what constitutes good technology integration
Technology and Traditional Assessment Methods
Does technology use result in higher test scores?
Probably, but there is not a direct linkTechnology alone is not helpful in
improving student performanceA high degree of technology integration is
more than likely to be an indicator itself of high performing schools and students
Is There a Connection?
High scores on technology integration rubrics generally equate to successful teaching and learning practices -- regardless of whether these practices have any specific relation to technology e.g., Constructivism, Engaged Learning, and
Student-Centered Classrooms
Technology supports these practices and these practices are shown to improve student learning
Schools where these practices are in place and technology is used appropriately are high performing schools
It’s not the technology, it’s the teaching!
We measure how well the technology is used to support successful teaching practices.
Our formative evaluations highlight how to improve teaching, technology integration, and ultimately student success
Next Steps?
Awareness building Establish the need for evaluation Establish a process
Division leadership As with technology implementation in general,
this is very important The evaluation will only be as good as it is
taken seriously
Evaluation committee membership Stakeholders!
Are tech plans in shape? Ideally, the evaluations will be based in sound
visions and goals and will take into account existing data on completed actions
Do you divisions have this capacity?
Evaluation work can be used to rejuvenate the local revision process, but this will take time and organization
Training It takes about a day to train people in this
approach/process Regional workshops? District teams?
Standardizing and cross-division review People should share and review their
questions, indicators, and rubrics before setting out to conduct an evaluation
Think about data collection Integrating a meaningful technology impact
survey into existing survey efforts Integrating observation data into existing
teacher/class observations
More Information
[email protected] 978-453-3070
www.sun-associates.com/eval/sample This presentation is linked to that page
Top Related