Analyzing and Sharing Assessment Results - … · Classical test theory ... Special Fields d) Dates...

Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

2011 European Users Conference Brussels

Analyzing and Sharing Assessment Results

Kate Soper

2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.

Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Reporting and analytics: Present and future - Greg Pope, Questionmark Thursday 1:30-3:00 PM

In Depth coverage of Analytics

Slide 2



Analytics are new reports, in addition to EM reporting

Analytics are part of the new QM Framework

8 new reports in analytics: 4 Course evaluation

1 General survey

1 Assessement results over time

2 Item Bank

Question status

Question type

Across repository or by topic

Questionmark Analytics vs. Enterprise Manager

Slide 3



Summary table of statistics All topics summary graph Topic breakdown graph

Compare the summary survey scores for all topics in each course - averages are ranked and color coded from lowest to highest score

https://www.questionmark.com/perception/help/v5/best_practice_guides/course_evaluations/default.htm

Course Summary Report

Slide 4







Summary survey scores for all topics for each instructor

Instructor Summary

Slide 5



survey scores for all topics for each participant that provided feedback for an instructor

Class Summary

Slide 6



Question level course evaluation information for a single instructor for a single course

Also includes comments

Class Detail Report

Slide 7



Graphs showing each question in a survey, and how many times each response was selected

Survey Frequency Analysis

Slide 8



Number of results over time

Mean score over time as well as 95% confidence intervals

Assessment Results Over Time

Slide 9



Number of questions in item bank that are assigned to each status

Question Status Report

Slide 10



Number of questions broken down by type

Question Type Report

Slide 11



Classical test theory

Item Analysis Report

Slide 12



Interactive Report

Colour coded to indicate questions that may need further review

Item Analysis Report Cont..

Slide 13


Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Slide 14



12 Standard report types

Types of Reports

Slide 15

= New Reporting Architecture



Report on … Report Type … Run on …

Assessment Statistics Assessment Overview Report Single Assessment Multiple assessments

Test Analysis Report Single Assessment

Results for a single assessment Score List Report Single Assessment

Export for Excel Single Assessment

Export assessment and question results

Export for Excel Single Assessment

Export to ASCII Single Assessment Multiple Assessments

Participants Transcript Report Single Participant

Coaching Report Multiple Participants, Single Assessment,

Grade Book Report Multiple Participants, Multiple Assessments

Comparison of two groups Gap Report Single Assessment, Two Groups

Items/Questions Question Statistics Single Assessment or Topic

Item Analysis Report Single Assessment

Survey Survey Report Single Assessment

Summary of Report Types and Uses

Slide 16



Regardless of whether it’s a “new” or “old” report, the basic process is the same

1. Select report template

2. Select assessment(s)

3. Set data set filters a) Groups

b) Participants

c) Special Fields

d) Dates

Reporting Process

Slide 17



Assessment Overview Report

Test Analysis Report

Common Statistics Histogram of test scores

Mean

Min Achieved

Max Achieved

Standard Deviation

Report on … Assessment Statistics

Slide 18

Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

2011 European Users Conference Brussels

Differences between Assessment Overview and Test Analysis reports

Assessment Overview

Date/time first taken

No. times started

No. times finished

No. time not finished

Least time taken

Most time taken

Mean time taken

Histogram of topic scores

Can be run on multiple assessments

Test Analysis

No. of items Max. possible score Median Mode Variance Standard error of mean Standard error of

measurement Skew Kurtosis Reliability (Cronbach’s Alpha) Can be run on a single

assessment

Slide 19



Assessment Overview Report - Output

Slide 20



Test Analysis Report - Output

Slide 21



Test Analysis Report - Output

Slide 22



Score List Report

Export for Excel

Report on … Scores for a single assessment

Slide 23



Score List Report - Output

Slide 24



Result index Assessment ID Assessment Name Assessment Author Assessment Last Modified Participant Participant Group Participant Details Hostname IP Address Monitor Feedback Given at End

Status Date/Time Started Date/Time Finished Total Score Maximum Score Percentage Score Time Taken Assessment Outcome

Label Special 1 – 10 Schedule Name Topic Percentage Score Topic Result

Score List Report – Template Options

Slide 25



Option to include question information Question ID

Question Description

Answer (given)

Score (achieved)

Outcome (triggered)

Option to include link to coaching report

Option to display average score for data set

Score List Report – Template Options

Slide 26



Export for Excel - Output

Slide 27



Main data options are the same as Score List Report (slide 11) Except no options to compute average score

Question Options Same as for Score List

Except can include question wording

Export for Excel – Template Options

Slide 28



Export for Excel

Export to ASCII

Export Reports

Slide 29



Can be run on multiple assessments

Can include Question details

Topic Analysis

Used often when there is a need to analyze results outside of Perception I.E. psychometric analysis

Export to ASCII

Slide 30



Export to ASCII - Output

Slide 31

Results w/o Question Details

Results with Question Details



Transcript Report

Coaching Report

Grade Book Report Participants across a set of assessments

Report on … Participants

Slide 32



Assessment “history” for an individual participant

Transcript Report - Output

Slide 33



Result Index Assessment ID Assessment Name Assessment Author Assessment last

modified Assessment description Participant group Participant details Whether Monitored Monitor Information

Hostname IP Address Status Date/Time Started Date/Time Finished Total Score Maximum Score Percentage Score Time Taken Assessment Outcome

Label Special 1 - 10

Transcript Report – Template Options

Slide 34

Option to include link to Coaching Report



New capabilities in v4.4 Report is broken down into 3 reportlets

Question

Topic

Comparison Batch Output to

PDF

Word

Excel

HTML-friendly E-mail Save and re-use report definitions

Coaching Report

Slide 35



Question Reportlet - Output

Slide 36



Topic Reportlet - Output

Slide 37

Participant Benchmark



Comparison Reportlet - Output

Slide 38

Participant

Topic level

Overall



Vary according to reportlet

Show options in Enterprise Reporter

Coaching Report – Template Options

Slide 39



Performance across a group of assessments

Supports the practice of assigning grades based on performance on several assessments E.g. Unit tests + Mid-term exam + Final exam = Final

Grade

The “final grade” (or score) can be weighted

Requires the creation of an assessment list

Grade Book Report

Slide 40



Grade Book Report

Slide 41



Score in points Max score Percentage score Assessment outcome

label Participant details Participant group Weighted total score Weighted maximum score Weighted percentage score Mean

Report on … All participants All participants in a

specific group All who have taken the

first assessment in list

Exclude incomplete/unfinished assessments

For Multiple attempts Display: latest, highest or

earliest Show number of attempts

Grade Book Report – Template Options

Slide 42



Gap report Raw and Relative differences between

Group A and Group

Administration 1 and Administration 2

Analyzes

Assessment score

Topic scores

Questions

Report on … Comparison of two groups (administrations)

Slide 43



Gap Report - Output

Slide 44



No. of participants Mean score Standard deviation Minimum score Maximum score Raw difference Relative difference Difference as a graph Individual divergence

from group t value Significance of t value

Assessment name Assessment author Assessment ID Analysis of score bands Histogram of assessment

scores Histogram of topic scores Analyze questions Analyze topics Display description of

filter Show question wording

Gap Report – Template Options

Slide 45



Question Statistics Report


Report on… Assessment Items

Slide 46



Can run by assessment or by topic When running by topic, Correlation is unavailable

Question Statistics Report - Output

Slide 47



Question description Question wording Topic Question type Times presented Times answered When last presented Maximum score Mean score Standard deviation of

score

Difficulty Correlation Significance of

correlation Outcome analysis Comments Question status Outcome name Times answered Percentage of times

answered Mean for outcome

Question Statistics Report – Template Options

Slide 48



Can only be run on assessments

Needs a minimum of 25 results to run

Template is not editable


Slide 49



Header and Histograms

Item Analysis Report - Output

Slide 50



Item Analyses

Item Analysis Report - Output

Slide 51



Survey Report

New capabilities as of v4.2

In core product as of v4.3

SP3 for v4.4 offers new date filter option

Report on … Surveys

Slide 52



Reportlets for analyzes different types of questions

Different graphical representations

Cross-tabs and trend analyses

Output to Word

PDF

Excel

Printer-friendly e-mail

Save and re-use report definitions

Survey Report

Slide 53



Survey Report - Output

Slide 54




Slide 55




Slide 56




Slide 57




Slide 58

Trainer A

Trainer B

Trainer C

Trainer D

Trainer A Trainer B Trainer C Trainer D




Slide 59



Access to Report Types

Access to Report Manager Access to Report Templates

Access to Results All results regardless of group?

Only specific groups

Reporting Permissions

Slide 60



Reporting Permissions

Slide 61



Enterprise Reporter Guide http://www.questionmark.com/perception/help/v4/manuals/er/welcome.

htm

Web Deployment Guide http://www.questionmark.com/perception/help/v4/manuals/wd/intro/ind

ex.html

See Managing Web Deliver | Administrators | Creating an Administrator | Setting Reporting Permissions and

Setting Security | Security on Report Templates

Enterprise Manager – Onboard Help

Related Resources

Slide 62

http://www.questionmark.com/perception/help/v4/manuals/er/welcome.htm

http://www.questionmark.com/perception/help/v4/manuals/er/welcome.htm

http://www.questionmark.com/perception/help/v4/manuals/wd/intro/index.html

http://www.questionmark.com/perception/help/v4/manuals/wd/intro/index.html



Questions?

Slide 63

Thanks for your participation and questions. Kate Soper [email protected]

Analyzing and Sharing Assessment Results - … · Classical test theory ... Special Fields d) Dates...

Documents

Transcript of Analyzing and Sharing Assessment Results - … · Classical test theory ... Special Fields d) Dates...