Enhancing & Enhancing & Evaluating Student Evaluating Student

Success: Integrating Success: Integrating LMS and AssessmentLMS and Assessment

Delia HeckDelia HeckDirector, Assessment and Institutional ResearchDirector, Assessment and Institutional ResearchAssistant Professor of Environmental ScienceAssistant Professor of Environmental Science

Christine StinsonChristine StinsonChief Information OfficerChief Information Officer

Ferrum College, Ferrum, VirginiaFerrum College, Ferrum, VirginiaPresentation at AIR Forum, June 6, 2007Presentation at AIR Forum, June 6, 2007

www.ferrum.edu/dheckwww.ferrum.edu/dheck

MotivationMotivation

• Many educational institutions use Learning Many educational institutions use Learning Management Systems (LMS) to deliver Management Systems (LMS) to deliver and enhance student learning. and enhance student learning.

• These same institutions are under These same institutions are under increasing pressure to establish effective increasing pressure to establish effective processes for evaluating (and improving) processes for evaluating (and improving) student learning outcomes.student learning outcomes.

Motivation continuedMotivation continued

• How do you use your LMS to get How do you use your LMS to get assessment data?assessment data?

• Can LMS aggregate and export Can LMS aggregate and export information useful in evaluating student information useful in evaluating student success?success?

OverviewOverview• Purpose of today’s presentation:Purpose of today’s presentation:

– Review how we are integrating Assessment Process and LMS.Review how we are integrating Assessment Process and LMS.– Share what we are doing, get your feedback and suggestions.Share what we are doing, get your feedback and suggestions.

• Background on Ferrum CollegeBackground on Ferrum College• Developing a Culture of Assessment at Ferrum College.Developing a Culture of Assessment at Ferrum College.• Process of Choosing Assessment SoftwareProcess of Choosing Assessment Software• Ferrum’s experience with LMS: 3 systems in last 5 years.Ferrum’s experience with LMS: 3 systems in last 5 years.

• What What areare we doing now: Assessment, ANGEL 7.1, and ePortfolio. we doing now: Assessment, ANGEL 7.1, and ePortfolio.• What What willwill we be doing: Assessment, ANGEL 7.2, and ePortfolio we be doing: Assessment, ANGEL 7.2, and ePortfolio

• Your suggestions, questions, comments, and discussion.Your suggestions, questions, comments, and discussion.

Ferrum College

• Liberal arts college affiliated with the Methodist Church

• Mainly residential undergraduate population.• 1000-1100 students, 60% male, 40% female• SW Virginia (eastern Appalachian region).• Member of 36-school Appalachian College

Association.• Accredited by Southern Association of Colleges and

Schools (SACS).

Overview and review of Overview and review of Assessment.Assessment.

AssessmentAssessment

• What is assessment?What is assessment?– Systematic process to gather, analyze & Systematic process to gather, analyze &

interpret information for continuous interpret information for continuous improvement of student learningimprovement of student learning

• What are learning outcomes?What are learning outcomes?– Know, Do, & Be LikeKnow, Do, & Be Like

• What do we do with assessment What do we do with assessment results?results?– Improve student learningImprove student learning– Improve pedagogyImprove pedagogy– Collect learning artifactsCollect learning artifacts

Assessment ProcessAssessment Process

• Assessment should:Assessment should:– Set standards for programs & coursesSet standards for programs & courses– Evaluate progress & provide feedback Evaluate progress & provide feedback

to students and program coordinatorsto students and program coordinators– Motivate & improve student learningMotivate & improve student learning– Help the institution enhance its Help the institution enhance its

effectivenesseffectiveness

Assessment SystemsAssessment Systems• Assessment Systems provide valuable Assessment Systems provide valuable

feedback feedback – How students are being shaped and transformed How students are being shaped and transformed – Faculty learn how they can improve their own Faculty learn how they can improve their own

pedagogypedagogy• Coordination of LMS and Assessment Systems Coordination of LMS and Assessment Systems

is potentially valuable, but…is potentially valuable, but…– faculty can be reluctant to embrace new programs faculty can be reluctant to embrace new programs

that require large investments of time.  that require large investments of time.  • Need to design assessment systems that meet Need to design assessment systems that meet

the simultaneous needs ofthe simultaneous needs of– Individual facultyIndividual faculty– Programs or departmentsPrograms or departments– Divisions or unitsDivisions or units– Overall institution Overall institution

Evaluation of Assessment Evaluation of Assessment software alternatives.software alternatives.

Ferrum College’s ProcessFerrum College’s Process

• • Determined our selection criteriaDetermined our selection criteria

• • Critical criterion: Ease of use for faculty and Critical criterion: Ease of use for faculty and staff.staff.

• • Critical criterion: Interfaces with LMS.Critical criterion: Interfaces with LMS.

• • Critical criterion: Curricular Critical criterion: Curricular andand co-curricular co-curricular use.use.

Ferrum College’s ProcessFerrum College’s Process

• Key issues:Key issues:– How to capture learning artifactsHow to capture learning artifacts– How to tie learning artifacts to student How to tie learning artifacts to student

learning outcomeslearning outcomes– How to aggregate data about student How to aggregate data about student

learning outcomes across courses, learning outcomes across courses, programs & institutionprograms & institution

– Any redundancy of work between LMS Any redundancy of work between LMS and assessment system will render and assessment system will render process “dead-in-the-water”process “dead-in-the-water”

Assessment Assessment software products software products

we consideredwe considered• Evaluated various systems through on-line Evaluated various systems through on-line

demos with Assessment Taskforce and demos with Assessment Taskforce and Director of Instructional TechnologyDirector of Instructional Technology

• NuventiveNuventive • TaskStream• TaskStream• True OutcomesTrue Outcomes • WEAVE Online• WEAVE Online• Powersight (WebCT/Blackboard)Powersight (WebCT/Blackboard)• LiveTextLiveText

Ferrum College’s ProcessFerrum College’s Process

• Two on-site, open demonstrations Two on-site, open demonstrations for campus communityfor campus community– TaskStreamTaskStream– True OutcomesTrue Outcomes

• Participants at demonstrations Participants at demonstrations completed evaluation sheets for completed evaluation sheets for each product.each product.

Sample Evaluation FormSample Evaluation FormAssessment Software Evaluation Form

Software Package : Evaluator :

Criteria Ranking

1 = poor, 5 = excellent Develops a matrix that tracks stated outcome objectives from the course to the program to the institution

1 2 3 4 5 NA

Ease of inputting learning goals & outcome objectives at the course level 1 2 3 4 5 NA Ease of inputting learning goals & outcome objectives at the program level 1 2 3 4 5 NA Ease of inputting learning goals & outcome objectives at the campus level 1 2 3 4 5 NA Provides a library of learning outcome samples 1 2 3 4 5 NA Assess any program, whether curricular or co-curricular 1 2 3 4 5 NA Provides online instructions for writing learning goals and outcome objectives 1 2 3 4 5 NA Provides online instructions for entering data and/or completing forms 1 2 3 4 5 NA E-portfolio module 1 2 3 4 5 NA Track and analyze individual students 1 2 3 4 5 NA Specific measurement results by student 1 2 3 4 5 NA Track summative or formative evaluation 1 2 3 4 5 NA Track data across multiple programs or outcomes 1 2 3 4 5 NA End-user surveys (can administrator, faculty or others test or survey others, e.g., students, alumni, businesses, etc.)

1 2 3 4 5 NA

"Closing the loop" reports 1 2 3 4 5 NA Time per person/per semester required to administer 1 2 3 4 5 NA Interface with our current student records and registration? 1 2 3 4 5 NA Server platform and requirements, if any? 1 2 3 4 5 NA Does it require on-site technical support person? 1 2 3 4 5 NA If so, what percentage of that person's workload should be allocated? 10% 20% 30%

40% 50% 60% 70% 80% 90%

100% NA Cost - First year including setup with data input? 1 2 3 4 5 NA Cost - Following years? 1 2 3 4 5 NA Other costs? 1 2 3 4 5 NA Overall Impression of the software 1 2 3 4 5 NA Comments:

• Learning Outcomes • Sample Learning Outcomes• National Standards• Co-curricular• On-line instructions• ePortfolio• Analyze & Track individual outcomes• Summarize across programs• Track data across outcomes• Interface with ERP• Technical Support• Time requirements• Costs – initial, direct, indirect

Ferrum College’s ProcessFerrum College’s Process

• All assessment software products All assessment software products were either too limited or too were either too limited or too complex.complex.

• If the assessment process isn’t If the assessment process isn’t simple, faculty and staff members simple, faculty and staff members won’t do it well or happily.won’t do it well or happily.

Overview and review of Overview and review of Learning Management Learning Management

Systems.Systems.

Ferrum College and LMSFerrum College and LMS

• We had been searching for LMS that combined flexibility, ease of use, and allowed us to achieve institutional goals.

• 2003-2004, 2004-2005 – used a Moodle-based system.

• 2005-2006 – WebCT Vista (with Appalachian College Association)– Not pleased with cost/performance combination.– Needed LMS that integrated with our Assessment Process.

Evaluation of LMS Evaluation of LMS AlternativesAlternatives

<--- <--- Open sourceOpen source ---- ---- ----- ----- Private butPrivate but ----- -----

collaborativecollaborative

--- --- Private andPrivate and -----> ----->

closedclosed

SakaiSakai

MoodleMoodle

ANGELANGEL

Desire2LearnDesire2Learn

BlackboardBlackboard

WebCTWebCT

Least costly.Least costly.

““Free software”Free software”

as in “free puppy”.as in “free puppy”.

Intermediate cost.Intermediate cost. Highest cost.Highest cost.

Database open to Database open to usersusers

Database completely Database completely accessible to usersaccessible to users

Database partially Database partially accessible for feeaccessible for fee

No current support for No current support for Learning Outcomes.Learning Outcomes.

Newest/pending Newest/pending releases support releases support multiplemultiple LOs per LOs per

Assignment.Assignment.

BlackboardBlackboard: next : next release supports release supports oneone LO per assignment.LO per assignment.

WebCTWebCT: no LOs.: no LOs.

Ferrum College and LMSFerrum College and LMS

• We have been searching for LMS that combined flexibility, ease of use, and allowed us to achieve institutional goals.

• 2003-2004, 2004-2005 – used a “free” Moodle-based system.

• 2005-2006 – WebCT Vista (with Appalachian College Association)– Not pleased with cost/performance combination.– Needed LMS that integrated with our Assessment Process.

• 2006-2007, 2007-2008 – Happly home with ANGEL 7.1 (and, soon, 7.2).

Outcome and progress Outcome and progress at Ferrum College.at Ferrum College.

Ferrum College and Ferrum College and ANGELANGEL

Faculty Perspective…Faculty Perspective…

• Extensive training available for faculty.Extensive training available for faculty.• Strong (voluntary) adoption in first Strong (voluntary) adoption in first

year.year.• 60% of sections used ANGEL.60% of sections used ANGEL.• High faculty (and student) satisfaction.High faculty (and student) satisfaction.

How are we integrating How are we integrating Assessment and ANGEL?Assessment and ANGEL?

• Developing faculty culture of Developing faculty culture of assessment.assessment.

• Encouraging and supporting use of Encouraging and supporting use of ANGEL 7.1 to deliver course content.ANGEL 7.1 to deliver course content.

• Encouraging and supporting use of Encouraging and supporting use of ePortfolio to store artifacts (evidence ePortfolio to store artifacts (evidence of Learning Outcomes).of Learning Outcomes).

Faculty Culture of Faculty Culture of AssessmentAssessment

• Faculty owned and managed the process of Faculty owned and managed the process of developing assessment goals and methods.developing assessment goals and methods.

• Used pilot projects (course-level and program-Used pilot projects (course-level and program-level) to gradually introduce process.level) to gradually introduce process.

• Identified key players, both on early adopters-Identified key players, both on early adopters-side as well as the objectors and worked to get side as well as the objectors and worked to get their buy-in and find out their concerns.their buy-in and find out their concerns.

Assessment TimelineAssessment Timeline2004-052004-05 2005-2005-

06062006-072006-07 2007-082007-08 2008-2008-

09092009-102009-10 2010-112010-11

•Charge Charge to to AssessmeAssessment nt TaskforceTaskforce

Course Course & &

PrograProgram Pilot m Pilot Phase IPhase I

SACS SACS VisitVisit

•Mission Mission AlignmentAlignment

Course & Course & Program Program

Pilot Phase Pilot Phase IIII

•HallmarkHallmarks of a s of a Ferrum Ferrum College College GraduateGraduate

General General EducationEducation

AssessmentAssessment

•Student Student Learning Learning OutcomesOutcomes

Course & Course & Program Program

AssessmenAssessmentt

Example of using ANGEL Example of using ANGEL to undertake Course to undertake Course

AssessmentAssessment

• Professor used ANGEL Gradebook to Professor used ANGEL Gradebook to record grades for subsets of major record grades for subsets of major exams.exams.

• Each subset corresponded to course Each subset corresponded to course Learning Outcome.Learning Outcome.

ACC 202 A - Learning ACC 202 A - Learning OutcomesOutcomes

1. Be able to record economic events as accounting journal entries.

2. Be able to aggregate journal entries into a new Income Statement, Balance Sheet, and Statement of Cash Flows.

3. Be able to compute and interpret simple financial accounting ratios.

ACC 202 A - ResultsACC 202 A - Results

1. 19 out of 20 students who took the Final Exam scored 70% or higher on Learning Outcome 1.

2. Only 3 of those 20 students scored 70% or higher on Learning Outcome 2.

3. 14 of 20 students scored 70% or higher on Learning Outcome 3 (5 of the 6 who scored poorly ran out of time on the exam).

ePortfolio and ePortfolio and AssessmentAssessment

• Used by independently-accredited Used by independently-accredited program now (Teacher Education).program now (Teacher Education).

• Used by Director of Assessment now Used by Director of Assessment now to collect evidence for College-level to collect evidence for College-level Learning Outcomes.Learning Outcomes.

   Core Curriculum Core Curriculum Major/Minor Major/Minor

Program Program Co-curricular Co-curricular

Activities Activities

••Explore an area of interest Explore an area of interest through reading and research through reading and research

••Evaluate the validity, Evaluate the validity, perspectives, and contexts of perspectives, and contexts of information information

••Use problem-solving skills Use problem-solving skills creatively to synthesize an creatively to synthesize an effective response effective response

••Articulate a personal Articulate a personal aesthetic with regard to aesthetic with regard to human expression and the human expression and the natural worldnatural world

Creative Inquiry and Critical Thinking

   Core Curriculum Core Curriculum Major/Minor Major/Minor

Program Program Co-curricular Co-curricular

Activities Activities

••Demonstrate a depth of Demonstrate a depth of knowledge in their chosen knowledge in their chosen fields fields

••Demonstrate the skills to Demonstrate the skills to enhance the transition to enhance the transition to employment and/or graduate employment and/or graduate and professional educationand professional education

••Work cooperatively and Work cooperatively and effectively with others. effectively with others.

••Use technology appropriate Use technology appropriate to their chosen fields to their chosen fields

••Integrate personal integrity Integrate personal integrity with professional ethics with professional ethics appropriate to their fields appropriate to their fields

••Apply the concepts and skills Apply the concepts and skills gained in the liberal arts core gained in the liberal arts core to the major to the major

Professional Capability

ePortfolio and ePortfolio and AssessmentAssessment

• Used by independently-accredited programs Used by independently-accredited programs now.now.

• Used by Director of Assessment now to collect Used by Director of Assessment now to collect evidence for College-level Learning Outcomes.evidence for College-level Learning Outcomes.

• Next year: use by Program Coordinators to Next year: use by Program Coordinators to collect evidence of Program Learning collect evidence of Program Learning Outcomes.Outcomes.

• Next year: use by students for personal Next year: use by students for personal portfolios with Academic Advisor & Student portfolios with Academic Advisor & Student Affairs.Affairs.

  

Knowledge of Knowledge of Fundamental Fundamental

Ecological Ecological PrinciplesPrinciples

Quantitative Quantitative FoundationFoundation

Basic Basic Environmental Environmental

ParametersParameters

ESC 110 Introduction to ESESC 110 Introduction to ES

ESC 209 Geology & ESC 209 Geology & HydrologyHydrology

ESC 211 Fundamentals of ESC 211 Fundamentals of EcologyEcology

ESC 215 Quantitative ESC 215 Quantitative MethodsMethods

GEOG 220 Physical GEOG 220 Physical GeographyGeography

GEOG 221 Geographic GEOG 221 Geographic Information SystemsInformation Systems

Environmental Science Program

What are our ANGEL 7.2 What are our ANGEL 7.2 plans?plans?

• Import College-level and Program-level Import College-level and Program-level Learning Outcomes.Learning Outcomes.

• Train faculty to use these for their Train faculty to use these for their assessment activities. assessment activities.

• Develop scheduled export of results to Develop scheduled export of results to Program Coordinators, Associate Deans, Program Coordinators, Associate Deans, Dean, and Director of Assessment.Dean, and Director of Assessment.– Some reports built into system, others built Some reports built into system, others built

by queryby query

What potential problems What potential problems exist?exist?

• Differences in vocabulary between our internal Differences in vocabulary between our internal culture and ANGEL 7.2 (e.g., “Standards”).culture and ANGEL 7.2 (e.g., “Standards”).

• Affects faculty perception of difficulty of Affects faculty perception of difficulty of adoption.adoption.

• Not all faculty use LMS.Not all faculty use LMS.

• Not all assessments are conducive to on-line Not all assessments are conducive to on-line environment, but grades & feedback can be environment, but grades & feedback can be entered.entered.

SummarySummary

• ANGEL 7.1 & 7.2 and ePortfolio are ANGEL 7.1 & 7.2 and ePortfolio are already supporting assessment of already supporting assessment of college learning outcomes.college learning outcomes.

• Implemented solutions are institution-Implemented solutions are institution-specific.specific.

• Process is key for our faculty and staff.Process is key for our faculty and staff.• Software has not yet caught up with our Software has not yet caught up with our

needs, but we are getting there slowly.needs, but we are getting there slowly.

Thank you!Thank you!

Questions?Questions?

Suggestions?Suggestions?

Comments?Comments?

E-mail AddressesE-mail Addresses

• cstinson@ferrum.educstinson@ferrum.edu• dheck@ferrum.edudheck@ferrum.edu

NotesNotes• Blackboard: Blackboard: http://www.fool.com/news/commentary/2006/commentary06081803.htmhttp://www.fool.com/news/commentary/2006/commentary06081803.htm• eCollege: eCollege: http://www.fool.com/news/mft/2006/mft06081123.htmhttp://www.fool.com/news/mft/2006/mft06081123.htm• 2004 review of LMS: 2004 review of LMS: http://www.obhe.ac.uk/cgi-bin/keyresource.pl?resid=24http://www.obhe.ac.uk/cgi-bin/keyresource.pl?resid=24• UK Market share: UK Market share: http://mfeldstein.com/index.php/weblog/comments/359/http://mfeldstein.com/index.php/weblog/comments/359/

– combined market share of Blackboard and WebCT declined from 56.8% in 2001 to combined market share of Blackboard and WebCT declined from 56.8% in 2001 to 43.7% in 2005.43.7% in 2005.

– share for Open Source went from 0% to 11%, with 8% of the gains happening in the last share for Open Source went from 0% to 11%, with 8% of the gains happening in the last 2 years.2 years.

– number of homegrown VLE’s jumped from 7% in 2001 to 30% in 2005.number of homegrown VLE’s jumped from 7% in 2001 to 30% in 2005.• Desire2Learn: Desire2Learn: http://www.desire2learn.com/http://www.desire2learn.com/

– Canada's 18th fastest growing technology company (2006 Deloitte Technology Fast 50 Canada's 18th fastest growing technology company (2006 Deloitte Technology Fast 50 program). program).

• Sakai: Sakai: http://www.sakaiproject.org/http://www.sakaiproject.org/• ANGEL’s openness: ANGEL’s openness:

http://findarticles.com/p/articles/mi_qa4057/is_200504/ai_n13642051/pg_1http://findarticles.com/p/articles/mi_qa4057/is_200504/ai_n13642051/pg_1• WebCT/Blackboard merger: WebCT/Blackboard merger:

http://www.infotoday.com/newsbreaks/nb051017-2.shtmlhttp://www.infotoday.com/newsbreaks/nb051017-2.shtml• WebCT/Blackboard antitrust concerns: http://www.antitrustlawblog.com/article-WebCT/Blackboard antitrust concerns: http://www.antitrustlawblog.com/article-

on-line-learning-market-may-be-monopolized.htmlon-line-learning-market-may-be-monopolized.html• Moodle: As of July 15, 2006 there were some 13,526 Moodle sites involving Moodle: As of July 15, 2006 there were some 13,526 Moodle sites involving

nearly 45,000 courses, five million users, over seven million enrollments, and nearly 45,000 courses, five million users, over seven million enrollments, and over 150 countries. over 150 countries. http://www.isu.edu/itrc/resources/moodle-info.shtmlhttp://www.isu.edu/itrc/resources/moodle-info.shtml