Introduction To Canvas LMS- Evaluation Planning
-
Upload
jefftoorongian -
Category
Documents
-
view
213 -
download
0
Transcript of Introduction To Canvas LMS- Evaluation Planning
-
7/29/2019 Introduction To Canvas LMS- Evaluation Planning
1/10
EvaluationPlanning
Document
JeffToorongianEDTECH512,Fall2011
Prof.K.DianeHall
-
7/29/2019 Introduction To Canvas LMS- Evaluation Planning
2/10
Evaluation Criteria for WBI: An Introduction to the Canvas Learning Management
System
Evaluation Criteria Explanation Data Sources
Effectiveness:
Mastery of WBIlearning goals as
indicated throughcompletion of Module
lessons and activities
WBI modules will have video
tutorials and demonstrations.Effectiveness will be demonstrated
through successful completion ofmodule activities
-Web-based
assessments -Completed module
learning objects andproducts
Appeal: Self-paced
WBI is used and wellreceived. Learners
work through alltraining modules.
Content is engagingand well received.
Video tutorials wellreceived.
As self-paced instruction, the WBI
will need to attract and retain learnerinterest to keep them engaged
through the entire course. Instructionwill need to establish itself as a
valued resource for all instructorsusing the new LMS for their teaching
activities.Modules will feature video tutorials
reviewing important material in eachmodule. These resources will keep
learners engaged and provideimportant visual learning
opportunities.
-Learner comments and
feedback.
-Course facilitatorcomments and
feedback.
-Usage data onparticipant progress
through course content(did learners complete
all modules, or start andquit?)
-General feedback fromlearners, facilitators,
and SME, regarding
course content, deliverymethods, and levels oflearner engagement and
overall impressionswith course content,
structure, and deliverywithin Canvas LMS.
Efficiency: WBI
provided a short,engaging, and content
rich experience for
busy instructors
Participants are busy. The goal of
the WBI is to deliver self-paced,efficiently presented information that
they need to get up and running with
a new LMS product. By deliveringthe instruction within the LMS theyreceive additional hands-on, practical
experience that will further supporttheir successful completion of
learning goals.
-Feedback from course
evaluation will providedetails on time invested
by learners. Open-
ended comments inevaluation survey willallow learners to
comment further ondelivery methods,
course structure, andcontent.
-
7/29/2019 Introduction To Canvas LMS- Evaluation Planning
3/10
Evaluation Matrix for Formative Evaluation Questions for WBI: An Introduction to
the Canvas Learning Management System
Evaluation Criteria andCategories
Questions Methods and Tools
Effectiveness/Goals Are the learning goals alignedwith instructional materials?
Are the learning goals alignedwith instructional activities and
assessments?
Are the activities and assessments
achievable in a time that isreasonable for a self-pacedcourse?
Do the learners have feedback
mechanisms available to them ifneeded?
End-user survey, Instructor
SME review, Designer expertreview.
Effectiveness/Content Do the video tutorials provideadequate content to achieve
module learning objectives?
Are the instructional materialsaccurate and comprehensiveenough to support learning goals?
Are the instructional materials
aligned with learning activitiesand goals?
End-user survey, InstructorSME review, Designer expert
review, Extant data.
Effectiveness/Technology Are learners able to access allinstructional materials needed?
Does the course navigation
function properly?
Do the video tutorials open and
play properly?
Do other learning resources(links) work when accessed?
End-user survey, Instructor
SME review, Designer expertreview.
-
7/29/2019 Introduction To Canvas LMS- Evaluation Planning
4/10
Does downloadable content open
properly in the target application?
Effectiveness/Message
Design
Is the course structure and
navigation clear, intuitive, andfunctional within the LMS?
Are the instructional video
tutorials well designed andreadable?
Is the audio quality of the tutorials
conducive to learning withoutmajor distractions or technical
problems?
Is the video quality of the tutorialsconducive to learning without
major distractions or technicalproblems? Are visuals clear and
easy to read?
Are the instructional materialsclearly and consistently labeled
and linked?
Are the lesson activities andassignments clearly described and
easily comprehended?
End-user survey, Instructor
SME review, Designer expertreview.
Appeal/Goals Do the goals encourage
participation from the learners?
Are the learning objectivesappropriate for the most important
skills needed by learners?
End-user survey, Instructor
SME review, Designer expertreview.
Appeal/Content Is the web-based instruction
engaging to the learner?Do the learning goals provide for
active learning opportunities forparticipants?
Does the instruction encourage
End-user survey, Instructor
SME review, Designer expertreview, Extant data.
-
7/29/2019 Introduction To Canvas LMS- Evaluation Planning
5/10
learners interact with the LMS asthey develop new skills?
Appeal/Technology Is the instructional content easyand intuitive to navigate?
Are there any grammatical or
typographic errors that distractfrom the learning objectives?
Do media files and external links
work properly?
End-user survey, Designerexpert review.
Appeal/Message Design Is the instructional messagealigned to the intended learnergroup (university instructors)?
Is the instructional web
environment intuitive, clear, andconducive to learning?
Is navigation through course
content easy and intuitive?
Are graphical elements within theweb environment attractive,
effective and not distracting to the
learner.
End-user survey, Designer
expert review.
Efficiency/Goals Are learning goals clearly andsimply stated?
Are the learning activities inalignment with course objectives?
End-user survey, Instructor
SME review, Designer expertreview.
Efficiency/Content Is the instruction presented clearlyand without distractions?
Does each learning module
represent unique learningobjectives?
End-user survey, InstructorSME review, Designer expert
review, Extant data.
Efficiency/Technology Does the web-based instruction
function effectively and withoutdistraction or complications to the
learner?
End-user survey, Instructor
SME review, Designer expertreview.
-
7/29/2019 Introduction To Canvas LMS- Evaluation Planning
6/10
Do the video tutorials coverimportant and necessary
information to help learnersachieve goals?
Is the audio in instructional mediaof high quality and support thelearner and facilitate a transfer of
information?
Efficiency/Message
DesignAre the learning modules unified
in design and overall learninggoals?
Are the learning objectives
presented logically and in a
manner appropriate for learnersuccess?
End-user survey, Designer
expert review.
Formative Evaluation Plan
Who are the Stakeholders? Primary Stakeholders:
-Course trainer/facilitator will have responsibilityfor developing course content in instructional
modules, designing course layout within the CanvasLMS, and facilitating course participants during
course.
-eLearning support staffwill be responsible forcourse design (in conjunction with course trainer),
support content development and delivery, providetechnical support to course participants, and facilitate
course administration for course participants.
-Course participants are the focus of the WBI.Their success in achieving learning objectives and
gaining a strong working knowledge of their newlearning management system will be the critical
factor that will determine the success of the trainingeffort.
-
7/29/2019 Introduction To Canvas LMS- Evaluation Planning
7/10
Secondary Stakeholders:
-IT staffwho support the server-side LMS
installation, back end support, and campus
application integrations are responsible for keepingthe LMS functional.
-Senior Administrators, including CIO,Library/eLearning Director, and executive team
members are ultimately responsible for the selectionand support of the LMS, and providing the necessary
resources (staff and funding) to provide training toinstructors.
What is being evaluated? - Course Objectives: Are the objectives that havebeen developed for the training course appropriate to
assure that participants will have sufficient skills withthe new LMS to begin developing their course in it?
- Course Assignments and Exercises: Do the
assignment and exercises developed and presented inthe WBI sufficient to assure learner mastery of the
skill(s) in question?
- Course Assessments: Do the assessments toolsadequately measure achievement of the learning
objectives?
- Design/Navigation: Is the navigation within thecourse clear, intuitive, and consistent? Are design
elements and the graphical display of course content(as experienced within the LMS) appealing,
appropriate to the content and learner audience, andconvey course information effectively?
- Course Content: Are the course materials of highquality, free of errors (grammatical and technical),and robust in breadth and depth of required content?
- Delivery Methods: Do the instructional delivery
technologies work properly? Does all linked contentfunction? Do all media files play as intended within
or external from the LMS, as designed?
-
7/29/2019 Introduction To Canvas LMS- Evaluation Planning
8/10
Who are the Evaluators and
Reviewers?Evaluator/Designer: An instructional designer, onstaff in the eLearning group will provide expert
review of the WBI, identify any areas of designweakness, suggestions to improve the design. This
instructional designer has more than seven years of
practical experience supporting faculty on campus inthe design of their course elements within the LMS.They also have extensive technical expertise with
troubleshooting and user LMS technical issues.
Expert Reviewer/Instructor: An AssociateProfessor in the university chemistry department will
provide technical expertise relevant to the Canvaslearning management system. He has piloted several
of his online and campus-based courses in the CanvasLMS over the last two semesters. His practical
knowledge of Canvas will provide invaluable insightinto the WBI content from the viewpoint of an
instructor.
End-User reviewers: A small group (no more than10) recent participants in the Canvas WBI course will
complete an online user-feedback survey. Thisreviewer group will be university instructors of
various ranks. Their feedback will offer importantinformation about course content, organization, and
design from a user perspective. Follow-up interviews
with a subset of these reviewers may be conducted iftime and resources permit, and are deemed necessary.
What are the Evaluation
Methods and Tools?-Expert reviews from designer and instructor via
checklist or written comments.
-End-user survey data.
-Extant data(see Evaluation Matrix for detailed breakdown each
criteria)
-
7/29/2019 Introduction To Canvas LMS- Evaluation Planning
9/10
When and how should formative evaluation take place?
The instructional designer and instructor reviewer will conduct initial review ofWBI design plans. At this early stage their input will likely involve observations and
suggestions related to course content, organization, and identified learning objectives.
An analysis of instructional activities and learner motivational strategies may also bereviewed. As course development nears completion, the instructional designer andinstructor reviewer will review course objectives, content, and message design issues to
find areas of weakness and identify errors that need addressing.
Due to a very small e-Learning support staff and the relatively short format of thisweb-based course, a limited try-out cycle will be conducted, consisting of a small-group
evaluation. This group of up to ten university instructors or other individuals withteaching experience and responsibilities, such a graduate students, will be asked to
engage the course content as it is designed and delivered within the Canvas learningmanagement system. The try-out cycle with end-user reviewers will also provide
important feedback on the technical quality of the instructional content and theapproximate time it takes to complete all course materials.
What decisions need to be made as the WBI design plan and prototype are
developed and revised?
Results from formative evaluation will primarily be qualitative data and will becompiled from survey results and evaluation comments. These findings will guide the
design team in their decision-making as they identify the areas of concern and resultantchanges to the course materials. Instances where there are divergent opinions regarding
needed changes, as identified in the evaluation results, may require consultation withstakeholders for guidance. Comments and completed course assignments from end-user
reviewers in the small group try-out cycle will provide important feedback regardingattitudes and newly developed skills. This information should prove valuable in helping
to determine needed changes to course assignments, course instructional content, andcourse design and navigation.
Preliminary Plan for Summative Evaluation
This is a new web-based training course, so no preexisting data from an original
course is available. Results from participant end-user surveys accompanied by theircompletion of course activities will be the primary methods of confirming success of the
instruction in preparing participants to use the Canvas LMS. Preliminary summativeevaluation plan criteria, questions, and related data sources are provided in the table
below. Results of the summative evaluation findings will be presented in a written reportthat will be submitted to the Director of the Center for Teaching and Learning and the
designated primary course instructor/facilitator. Information on the structure of theevaluation process, data sources, results and any recommendations will be included in the
report.
-
7/29/2019 Introduction To Canvas LMS- Evaluation Planning
10/10
Evaluation Criteria Main Questions Data Sources
Effectiveness - Did course help learnersto achieve learning
objectives?
- Were the activities andassessments achievable in a
time allocated for the self-paced course?
- Do learner perceptions
indicate they are preparedto use the Canvas LMS
upon completion of the
training course?
Course assessmentactivities
End-user survey results,and LMS student tracking
data.
End-user survey results
Efficiency - Did each learning moduleprovide unique learning
objectives as part of thetraining course?
Expert reviewer responses,end-user survey results
Appeal - Were course instructional
materials effective andengaging to the users?
- Did the web course design
and navigation supportparticipant learning
objectives?
- Do course objectivesencourage learning
participation in WBI?
End-user survey results
Expert reviewer responses,
end-user survey results
Expert reviewer responses,end-user survey results
2011JeffToorongian