It Takes a Village: Collborating for success with high-volume assessments
-
Upload
questionmark -
Category
Technology
-
view
3.978 -
download
0
Transcript of It Takes a Village: Collborating for success with high-volume assessments
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
2013 Users Conference Baltimore | March 3rd – 6th
It Takes a Village Collaborating for Success with High-Volume
Assessments
2013 Users Conference Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Rio Salado College is part of the Maricopa Community College District
Rio Salado offers over 600 unique courses online Our model – one course, many sections Development/Production of courses is centralized Perception users since 2003
A Little About Us
Slide 2
2013 Users Conference Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Penny Freuler, Director of Instructional Technology and Course Production
Liz Burnett, Coordinator of Distance Learning Services
Heather Tyler, Testing Coordinator
Lance Cummins, Networking Manager
Introductions
Slide 3
2013 Users Conference Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
How we use Perception with our LMS How we use Perception Data outside of the application How we work as a team to support assessment
delivery What we have learned to do/what we have learned not
to do
What We Will Cover
Slide 4
2013 Users Conference Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Building Assessments
Delivering Assessments
Scoring Results
Supporting Services
Course Production
Slide 5
2013 Users Conference Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Topics and questions Outcomes Generator
Assessment controls Password protection Time limits Randomization Open access for self-assessment
Outcomes Feedback on question, topic, or assessment level Branch to another assessment
Building Assessments
Slide 6
2013 Users Conference Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
LMS Integration
Offline versions available for Testing Services
Schedule control Sequencing
Device support Perception 5.2 inclusion of device settings file
Delivering Assessments
Slide 7
2013 Users Conference Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Entry points from LMS to Scoring Tool
Scoring Tool updates have not kept pace with the rest
of the Perception application
Building, delivering, and scoring work together
Post Perception 5.4 Deployment Integrating scoring functions in LMS
Scoring Results
Slide 8
2013 Users Conference Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Work with helpdesks, testing services, students, faculty to troubleshoot issues Greatly improved in Perception 5.2, 5.4 Coordinate resolutions with Programming, Networking,
Technical Support, Instructional Design Evaluate coaching and transcript reports
Work with Questionmark staff on system issues
beyond our control and best practice advice
Supporting Services
Slide 9
2013 Users Conference Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Exam Administration
Proctoring Services
Academic Integrity
Data for Content Owners - Reports
Testing Services
Slide 10
2013 Users Conference Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
In-Person Exams Midterm and Final exams delivered through QuestionMark Password protected Identification verification Monitoring during exam
Close working relationship with Course Production and Networking allows issues to be identified sooner
Testing Services- Exam Administration
Slide 11
2013 Users Conference Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Same level of testing security required for all out of area students. Proctoring staff dedicated to vetting and communicating with
proctors to assure same testing standards are met Creates the challenge for Networking and Course Production
that there is potentially a student taking a high stakes exam 24 hours a day 7 days a week.
Issues encountered Transcript report research Branching profile workaround
Testing Services – Proctoring Services
Slide 12
2013 Users Conference Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Working with Faculty Supply supporting evidence for a reported academic integrity issue
• Turn-It-In Reports • Peer-to-Peer Reports • Adjunct Suspected (similar course work submitted) • AI Team suspected (data pulled from reports)
Working with Financial Aid Supply supporting evidence for a suspected FA issue Working with the Student Academic Progress process AI team works to alert FA to potential issues before they happen
Testing Services – Academic Integrity
Slide 13
2013 Users Conference Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Testing Services – Academic Integrity
Slide 14
What actions are taken by the Judicial Affairs Department?
What does Academic Integrity
look to identify?
Who/What can instigate an inquiry?
Financial Aid SAP Process
Sufficient Evidence to
confirm less than honest behavior
New Students: Send Internal Review
Letter to gather further documentation
Current Students: Actions necessary to
include suspension and expulsion
Insufficient Evidence to
confirm wrong doings
Record Research This database will
house all confirmed, suspected and cleared
student research
Staff Referrals Advisement Counseling Admission and Records
Financial Aid
Academic Integrity Reports
2013 Users Conference Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Testing Services – Academic Integrity
Slide 15
•Questionable Documents submitted for SAP. FA staff marks student record with ‘Referral’ Service Indicator and emails [email protected] with student information, description and supporting documentation. •Examples of questionable documents •Falsified obituaries •Altered id cards
Financial Aid
•Reviews and researches all referrals. •Sufficient Evidence •Compile all documentation necessary to support recommendation for further action.
•Insufficient Evidence •Remove Referral service indicator •Alert FA that student is not considered a risk •Forward all research to Judicial Affairs for storage.
AI Team
•Review all records and recommendations •Confirmed Sufficient Evidence •Pre-Student Enrollment •Send Internal Review Letter •Current Student •Actions necessary
•Insufficient evidence •Return to AI for further research or clarification or file with other insufficient evidence files
Judicial Affairs
2013 Users Conference Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Item Analysis Allows Content Owners to review Assessments based on • Question Difficulty – P-Value Proportion Correct • How did students perform on the questions • What can this tell us about the question or the content
Testing Services – Reports
Slide 16
2013 Users Conference Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Testing Services – Reports
Slide 17
2013 Users Conference Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Testing Services – Reports coming in 5.4
Slide 18
2013 Users Conference Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Team approach Programming Networking Production Support Services
Work closely with Questionmark staff
Version Testing
Slide 19
2013 Users Conference Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Some lessons learned Previously patched items Greatly improved with Perception 5.4
Items created in previous versions Items created that employ irregular practices Test, retest, test again, and go back and test If you think users won’t use it that way, they will.
Version Testing
Slide 20
2013 Users Conference Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Server Builds Install and configure Perception Servers Test and Production environments Maintain multiple versions of Perception throughout testing and
development cycle Integrate into Rio Salado’s custom LMS Coordinate testing and development cycle with other
departments such as Course Production
Network Administration
Slide 21
2013 Users Conference Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
New versions of Perception require testing and development by multiple teams We delivered over 2.3 million assessments in 2012 Networking test builds– Build and configure new servers,
copy database (~40-50GB). Course Production – Test features, look for bugs, ensure that
previously patched items are retained in the new version Programming – Ensure code works with potential QM
changes, modify as needed.
Version Testing
Slide 22
2013 Users Conference Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Course Production Takes initial reports of issues and troubleshoots to determine
if there is an issue with the Perception servers, the application itself, or some other issue that needs to involve Networking
Cuts down greatly on the amount of issues that come to Networking. Since we don’t use the application ourselves, it takes much longer for us to determine where the issue lies.
Now, if it comes to us we know there’s been a great deal of groundwork done and we know more about the sort of problem people are having. It also weeds out things like user error and problems with the course content.
Collaboration
Slide 23
2013 Users Conference Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Testing Services With students taking exams all day in our testing centers, and
proctored exams possible at any time, server downtime must be kept to a minimum
Perception servers are load balanced, so we can reboot them one at a time without affecting students.
SQL– On the rare occasion that there is an issue with our main SQL cluster, Perception and other applications may experience a few moments of downtime while SQL is moved over to another cluster node. This will cause problems for students currently taking exams.
We notify Testing as soon as possible before or during outages so that they can be aware of issues and take appropriate action in our testing centers to minimize lost work.
Collaboration
Slide 24
2013 Users Conference Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark – If/when issues are discovered, we work with QM support to see if it’s a configuration issue or a bug. If it’s a bug then they work on getting it fixed and either hotfixed or rolled into their next update, depending on severity.
For previous versions we’ve had weekly or semi-weekly meetings with multiple teams at QM (e.g. Customer Care, Tech Support, and QM developers) to go through our list. Some issues were bugs, and some were unique ways we were using Perception that didn’t work as expected, but weren’t necessarily bugs.
Testing/Support
Slide 25
2013 Users Conference Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Several years ago Questionmark began offering Enterprise support. It’s a paid service, but one I highly recommend
This has saved us a great deal of time Previously we’d have to bring multiple techs up to
speed on issues we were having, had to send the same files/logs, etc. repeatedly
Now we have one point of contact, and he deals with us often enough to have a grasp on our environment without having to start at square one each time we contact him.
Enterprise Support
Slide 26
2013 Users Conference Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
For 5.4, QM has instituted many changes to their development process, and we’ve been very pleased with our testing
In fact, for the first time we’ve not needed anything hotfixed, and are preparing to deploy 5.4 to production at the end of the month. We would have gone sooner, but we had some code changes on our end that held it up, not directly QM related.
Perception 5.4
Slide 27
2013 Users Conference Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Best Practices – read them Review and understand server settings Use system as designed
Questions?
Remember
Slide 28