Technology Tools for Technology Tools for Facilitating Learning Facilitating Learning

Outcomes AssessmentOutcomes Assessment

Jerry Rudmann, Irvine Valley CollegePat Arlington, Coastline Community

CollegeOctober 2007

Strengthening Student SuccessStrengthening Student Success

Agenda and GoalAgenda and Goal

Technology UsesTechnology Uses Technology ToolsTechnology Tools

Expected Outcome: Be able to select Expected Outcome: Be able to select and use technology-based approaches and use technology-based approaches to assess student learning outcomesto assess student learning outcomes

Assessment ChallengesAssessment Challenges

Assessing Students in Large ClassesAssessing Students in Large Classes Assessing Performance at a DistanceAssessing Performance at a Distance Minimizing Subjectivity in AssessmentMinimizing Subjectivity in Assessment Creating Authentic AssessmentsCreating Authentic Assessments Engaging Students in Self-EvaluationEngaging Students in Self-Evaluation Accommodating Diverse Learning Accommodating Diverse Learning

StylesStyles Assessing Conceptual ThinkingAssessing Conceptual Thinking

Technology ToolsTechnology Tools

CCC Confer (Web Conferencing)CCC Confer (Web Conferencing) Online Rubric BuildersOnline Rubric Builders eLumen (SLO Assessment/Tracking)eLumen (SLO Assessment/Tracking) Calibrated Peer Review (CPR)Calibrated Peer Review (CPR) Classroom Responders (“Clickers”)Classroom Responders (“Clickers”) Scannable and Online TestsScannable and Online Tests Cmap Tools (Concept Maps)Cmap Tools (Concept Maps) ePortfolioePortfolio

Robust SLOsRobust SLOs

Developed through faculty dialogDeveloped through faculty dialog Behavioral/measurableBehavioral/measurable Real-worldReal-world Higher-levelHigher-level ConditionsConditions Performance CriteriaPerformance Criteria Global, over-archingGlobal, over-arching Scored with rubricScored with rubric

CCC ConferCCC Confer

Small-group work in project-based Small-group work in project-based learninglearning

Assessment of DL studentsAssessment of DL students Foreign language classesForeign language classes Speech/oral communicationSpeech/oral communication Information literacyInformation literacy

Score using discussion or Score using discussion or presentation rubricpresentation rubric

http://www.cccconfer.org

CCC Confer Screen ShotCCC Confer Screen Shot

Online Presentation RubricOnline Presentation Rubric

Rubric for Online Presentation

Characteristic/Primary Trait

Poor1

Fair2

Satisfactory3

Excellent4

Score

Organization Presentation had no logical sequence, making content difficult to understand

Presentation jumped around; audience had difficulty following the presentation

Presentation followed logical sequence that audience could follow

Presentation was well organized with logical, interesting sequence that audience could easily follow

Subject Knowledge

Student does not demonstrate grasp of information and cannot answer questions about subject

Student appears uncomfortable with information and is able to answer only rudimentary questions

Student appears at ease with expected answers to all questions but fails to elaborate

Student demonstrates full knowledge (more than required) by answering all class questions with explanations and elaboration

Graphics Student uses superfluous or inappropriate graphics or no graphics

Student occasionally uses graphics that rarely support text and presentation

Student’s graphics relate to text and presentation

Student’s graphics explain and reinforce screen text and presentation

Technology Student had significant problems with the presentation technology

Student had some problems using the technology

Student used the technology well

Student used the technology with ease and to excellent effect

TOTAL 0

Excerpted and adapted from Evaluating Student Presentations rubric by Education Development Center, Inc.: http://www.internet4classrooms.com/doc/PBL_germs.doc

Discussion ForumsDiscussion Forums

ToolsTools Integrated discussion forum in CMS Integrated discussion forum in CMS

or standalone (e.g., WebBoard by or standalone (e.g., WebBoard by O’Reilly)O’Reilly)

UsesUses Practice/enrichmentPractice/enrichment Small-group project-based learningSmall-group project-based learning To assess understanding – score To assess understanding – score

with rubricwith rubric

Online Discussion RubricOnline Discussion Rubric

http://www.uas.alaska.edu/sitka/IDC/resources/onlineDiscussionRubric.pdf

Online Rubric BuildersOnline Rubric Builders

Rubrics to guide and measure learningRubrics to guide and measure learning ToolsTools

RubistarRubistar Landmark Rubric MachineLandmark Rubric Machine Coastline Rubric BuilderCoastline Rubric Builder

RubistarRubistar Art History RubricRubistar Art History Rubric

Rubric Builder Screen Shot

Coastline Rubric BuilderCoastline Rubric BuilderCoastline Rubric Builder

Digital Assessment SuiteDigital Assessment Suite

PlagiarismPlagiarism Grading (Grading (GradeMark)) Peer Review GradeBookGradeBook

http://www.turnitin.com

Peer Review Screen ShotPeer Review Screen Shot

Return to Turnitin ~ Next Slide

How Do You Report Results?

eLumen to Assess SLOseLumen to Assess SLOs

Reduce Time Spent Creating ReportsReduce Time Spent Creating Reports Assess Course, Program, and/or Assess Course, Program, and/or

Degree-Level OutcomesDegree-Level Outcomes Share Assessment Rubrics Across Share Assessment Rubrics Across

Classes and ProgramsClasses and Programs View Individual or Aggregated ResultsView Individual or Aggregated Results Use Online or OfflineUse Online or Offline

http://www.elumen.info

Use Online or Offline

Criterion-Based AssessmentCriterion-Based Assessment

Rubrics are attached to each SLORubrics are attached to each SLO

Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

Writes prose clearlyWrites prose clearly

Rubrics Describe CriteriaRubrics Describe Criteria

Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

Student activity Student learning outcome(s)

Rubric

““LinkLink”” a specific student activity a specific student activity (e.g. project, paper, test) to the (e.g. project, paper, test) to the SLO(s) for which the activity can SLO(s) for which the activity can serve as evidenceserve as evidence

Link Activity to RubricLink Activity to Rubric

Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

Read and write a response paper for the novel A Lesson Before Dying

Writes prose clearly

Critically analyzes a text

Considers ethical aspectsof a situation or text

Rubric

Rubric

Rubric

Here, one assignment stands as Here, one assignment stands as evidence for 3 different SLOsevidence for 3 different SLOs

Link to Multiple SLOsLink to Multiple SLOs

Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

Library of Degree-Level SLOsLibrary of Degree-Level SLOs

Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

And Rubrics Link to SLOsAnd Rubrics Link to SLOs

Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

Science and Gen Ed SLOs/RubricsScience and Gen Ed SLOs/Rubrics

from the Biology Departmentfrom the Science committee

from the facultycommittee oncritical thinking

from the facultycommittee oncommunicationskills

Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

Scorecard for All StudentsScorecard for All Students

Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

Class Scores by StudentClass Scores by Student

Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

Aggregated Data for CourseAggregated Data for Course

Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

Course Aggregates by ProgramCourse Aggregates by Program

Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

Calibrated Peer ReviewCalibrated Peer Review

Web-based program that enables Web-based program that enables frequent writing assignments with frequent writing assignments with minimal impact on instructor timeminimal impact on instructor time

Uses peer reviewUses peer review Promotes deeper learningPromotes deeper learning

http://cpr.molsci.ucla.edu/

Calibrated Peer Review in Psychology 101Calibrated Peer Review in Psychology 101

Critical Thinking Critical Thinking in Introductory Psych Coursein Introductory Psych Course

SLO on Pseudoscience skepticism: Students will correctly identify non-scientific explanations of human behavior and explain why those explanations are not based upon science and do not provide reliable or valid explanations of behavior or predictions of future behavior.

The Pseudoscience Belief TestThe Pseudoscience Belief Test

Please rate how much you believe the following statements. Use the 7-point scale provided.

1 – Do not believe in this at all.

2 – I doubt very much that this is real.

3 – I doubt that this is real.

4 – I am unsure if this is real or not.

5 – I believe that this may be real.

6 – I believe that this is real.

7 – I strongly believe this is real.

__ 1. A person’s personality can be easily predicted by their handwriting.

__ 2. A person can use their mind to see the future or read other people’s thoughts.

__ 3. A person’s astrological sign can predict a person’s personality and their future.

__ 4. An ape-like mammal, sometimes called Bigfoot, roams the forests of America.

__ 5. The body can be healed by placing magnets on to the skin near injured areas.

__ 6. Healing can be promoted by placing a wax candle in your ear and lighting it.

__ 7. A dinosaur, sometimes called the Lock Ness Monster, lives in a Scottish lake.

__ 8. Sending chain letters can bring you good luck; ignoring them can bring you bad luck.

__ 9. The government is hiding evidence of alien visitation at places such as Area 51.

__ 10. Voodoo curses are real and have been known to kill people.

__ 11. A broken mirror can bring you bad luck for many years.

__ 12. Houses can be haunted by the spirits of people who have died in tragic ways.

__ 13. Water can be accurately detected by people using “Y” shaped tree branches.

__ 14. Animals, such as cats and dogs, are sensitive to the presence of ghosts.

Adapted from…Walker, Hoekstra, & Vogl, (2002). Adapted from…Walker, Hoekstra, & Vogl, (2002). Science education is no guarantee of skepticismScience education is no guarantee of skepticism, , SkepticSkeptic, vol 9, no 3. , vol 9, no 3.

Critical Thinking Experiment Critical Thinking Experiment Using a SLO as the Dependent VariableUsing a SLO as the Dependent Variable

Pseudoscience Belief Pre-test

Randomly Assigned 90

Students

Calibrated Peer Review Lesson on Graphology

Pseudoscience Belief Post-test

Calibrated Peer Review Lesson

on Different Topic

CPR ProcedureCPR Procedure Students read assignmentStudents read assignment Students read resource materialsStudents read resource materials Students wrote a short essay in Students wrote a short essay in

response to the materials: Why or why I response to the materials: Why or why I believe graphology is a reliable, valid believe graphology is a reliable, valid way to measure and predict way to measure and predict personality.personality.

Students are “calibrated” – prepared to Students are “calibrated” – prepared to score essays written by their peers.score essays written by their peers.

Students receive a detailed grade Students receive a detailed grade report for the assignment.report for the assignment.

Graphology Belief ScoresGraphology Belief ScoresStatistical SummaryStatistical Summary

TreatmentTreatment

GroupGroup

Pre-test Average

Post-test

Average

Paired t-tests

GraphologyGraphology 4.41 2.33 t(26) = 6.40

p < .01

Conditioning Conditioning 4.12 3.69 t(25) = 1.31

p = ns

t(51) = 0.67

p = ns

t(46.7) = 2.93

p < .01

00.5

11.5

22.5

33.5

44.5

5

Pre-test Post-test

Graphology

Conditioning

Mean Pre and Post-Test ScoresMean Pre and Post-Test Scores

on Graphology Belief Questionon Graphology Belief Question

Example EssayExample Essay The Detection of a Pseudoscience: GraphologyThe Detection of a Pseudoscience: Graphology

Elaine Quigley’s posting on the website www.businessballs.com is littered with “red flags” that expose graphology as the pseudoscience/pseudopsychology that it is. While an attempt to promote graphology, Quigley’s posting fails to measure up to several of Cotton and Scalise’s guidelines for “baloney detection.” This paper will examine four areas in which graphology fails to live up to its claim of being “science.”In an attempt to display graphology’s validity, Quigley cites the notion that it is “a very old and respected science.” The fact that it has existed for approximately 3,000 years is used to justify Quigley’s notion that graphology is a science. However, one educated in the definition of science knows that the age of a theory is not a factor used to determine its validity. In fact, there are many beliefs that have been around for thousands of years that cannot be tested and therefore cannot be deemed as scientifically reliable. Graphology is just one of many ideas that cannot be justified despite their age. Quigley also fails to tell how the “science” of graphology has been tested and proven. Instead, she simply states that graphology is a “reliable indicator of personality and behavior” and expects her readers to accept this statement as fact. She also mentions that “the science is still being researched and expanded.” This is the extent to which she approaches the issues related to the research of graphology. Without explaining the testing that was done to prove the methods reliability, how is one to know that graphology is indeed reliable? Indeed, the answer is simple. It is impossible to be sure of the reliability of a measure of personality if the measure itself cannot be tested. In addition to not presenting methods for testing the claims of graphology, Quigley also fails to present evidence in support of its validity. Instead, she simply states that “it is not easy to explain how and why graphology works, nevertheless it continues to be used, respected and appreciated by many.” Could it be that the only “evidence” for the reliability of graphology is the satisfaction that its users experience? Unfortunately, being “used” and “accepted” characteristics required of a science. Finally, the vast majority of information provided by Quigley is anecdotal and leads up to a sales pitch for her services. She provides vague stories about how graphology has been used to produce more successful hiring processes and personal relationships. The information is presented more as an advertisement than a scientific work. Quigley goes into more detail on her experience as a graphologist than she does on the aspects of graphology that would qualify it as a science. In conclusion, it is quite clear that based on the evidence presented in this paper, graphology qualifies as a pseudoscience rather than a science. The claims of graphologist Elaine Quigley fail to show that graphology is indeed a science. Instead, she relies on the age of graphology and anecdotal evidence in support of graphology while ignoring issues related to methods for testing graphology’s claims and the results that have resulted in tests of its validity. Looking critically at “discoveries” is no doubt a useful tool that extends beyond the subject of graphology. The methods for recognizing pseudosciences compiled by Cotton and Scalise are certainly tools that would empower all people and prevent them from being fooled by pseudoscientific claims.

Questions and Answers for CPR Peer ReviewersQuestions and Answers for CPR Peer Reviewers

1. Did the essay begin with a topic sentence?1. Did the essay begin with a topic sentence?

2. Was the essay free of spelling and grammatical errors?2. Was the essay free of spelling and grammatical errors?

3. Did the essay present at least four (4) different reasons for 3. Did the essay present at least four (4) different reasons for supporting or denying the validity of graphology (or supporting or denying the validity of graphology (or handwriting analysis) as a method of assessing personality handwriting analysis) as a method of assessing personality and/or predicting behavior?and/or predicting behavior?

4. Did the essay have balance? Although this may seem 4. Did the essay have balance? Although this may seem subjective, do you feel that it provided a balance among each subjective, do you feel that it provided a balance among each of the points made? For example, was each point was of the points made? For example, was each point was explained in the same amount of detail.explained in the same amount of detail.

5. Did the author's arguments seem convincing to you?5. Did the author's arguments seem convincing to you?

6. Did the author conclude with any reflection about whether this 6. Did the author conclude with any reflection about whether this assignment was or was not helpful to his or her learning? In assignment was or was not helpful to his or her learning? In other words, did the author indicate that this assignment might other words, did the author indicate that this assignment might help him or her judge the validity explanations of behavior help him or her judge the validity explanations of behavior encountered in the popular media (newspaper, radio, TV, encountered in the popular media (newspaper, radio, TV, magazines, etc.)?magazines, etc.)?

7. How would you rate this text? (Scale of 1 – 10)7. How would you rate this text? (Scale of 1 – 10)

Student’s Screen: Detailed ResultsStudent’s Screen: Detailed Results

Instructor Screen: Student ProgressInstructor Screen: Student Progress

Instructor Screen: One Student’s ResultsInstructor Screen: One Student’s Results

Instructor’s Screen: Student ResultsInstructor’s Screen: Student Results

SLO Data

Classroom RespondersClassroom Responders

Engage studentsEngage students Monitor student understandingMonitor student understanding Quickly and easily collect and store Quickly and easily collect and store

assessment dataassessment data Use publisher item banks or create Use publisher item banks or create

your ownyour own

Renaissance Classroom Response System

PBS Demo

Renaissance LearningRenaissance Learningfor clicker training resourcesfor clicker training resources

http://www.renlearn.com

Scanning TechnologyScanning Technology

Embedding Questions in Multiple Embedding Questions in Multiple Sections and ClassesSections and Classes

Basing Multiple-Choice Questions on Basing Multiple-Choice Questions on Case Studies and Complex Scenarios Case Studies and Complex Scenarios

http://www.scantron.com and http://www.renlearn.com

Surveys and TestsSurveys and Tests

Online or ScannableOnline or Scannable SurveysSurveys

Pre and post surveys of student self Pre and post surveys of student self evaluation of progressevaluation of progress

Faculty and business community or Faculty and business community or advisory groups related to expected advisory groups related to expected learning outcomeslearning outcomes

Student satisfaction (indirect outcomes)Student satisfaction (indirect outcomes)

Quizzes/TestsQuizzes/Tests Practice and gradedPractice and graded

Cross

Match

Concept MapsConcept Maps

http://cmap.ihmc.us/ http://cmap.ihmc.us/

Why Concept Maps?Why Concept Maps?

Provides graphical means for Provides graphical means for organizing and representing knowledgeorganizing and representing knowledge

Addresses meaningful learning rather Addresses meaningful learning rather than simply rote learningthan simply rote learning

Allows students to construct their own Allows students to construct their own understanding of the contentunderstanding of the content

Provides opportunities for collaborationProvides opportunities for collaboration Provides alternative to essays or Provides alternative to essays or

multiple-choice testsmultiple-choice tests

Student Concept MapStudent Concept Map

High school student’s concept map about birds from http://cmap.ihmc.us/Publications/ResearchPapers/TheoryCmaps/TheoryUnderlyingConceptMaps.htm

ePortfoliosePortfolios

AdvantagesAdvantages Document artifacts of learningDocument artifacts of learning Support diverse learning stylesSupport diverse learning styles Authentic assessmentAuthentic assessment Course, program, or degree-level Course, program, or degree-level

trackingtracking Job skill documentationJob skill documentation

Proprietary or Open SourceProprietary or Open Source ePortfolio and Open Source PortfolioePortfolio and Open Source Portfolio

ePortfolio.org AssessmentePortfolio.org Assessment

Lock Assignments after submissionRandom selection of assignments by learning objectiveAnonymity of the student who produced the assignment and the instructor Access to the work and the scoring rubricsReports to aggregate scores; generate frequencies/means Ability to download raw data which can be analyzed in another format

http://www.eportfolio.org

Open Source PortfolioOpen Source Portfolio

Aligned with Sakai

Admins or Faculty can structure and review work

Learning matrix documents levels of work

http://www.osportfolio.org

ResourcesResources Calibrated Peer Review: http://cpr.molsci.ucla.edu/ CCC Confer: http://www.cccconfer.org Class Climate: http://www.scantron.com Cmap Tools: http://cmap.ihmc.us eListen: http://www.elisten.com eLumen: http://www.elumen.info ePortfolios:

ePortfolio.org: http://eportfolio.org Open Source Portfolio: http://www.osportfolio.org/ For others, see EduTools ePortfolio product comparison:

http://eportfolio.edutools.info/item_list.jsp?pj=16 Hot Potatoes: http://hotpot.uvic.ca/ Online Rubric Builders

Rubistar: http://rubistar.4teachers.org Landmark Rubric Machine: http://landmark-project.com/rubric_builder/index.php Coastline Rubric Builder: http://rubrics.coastline.edu

Remark Survey Software: http://www.principiaproducts.com/web/index.html Renaissance Classroom Responders: http://www.renlearn.com/renresponder/ Turnitin.com: http://www.turnitin.com

Contact InfoContact Info

Dr. Jerry Rudmann, Professor of PsychologyIrvine Valley Collegejrudmann@ivc.edu

Pat Arlington, Instructor/Coordinator Instructional ResearchCoastline Community Collegeparlington@coastline.edu