Perception - Questionmark · PDF fileRoot Topic (Course) ... Practical Application in Resident...
Transcript of Perception - Questionmark · PDF fileRoot Topic (Course) ... Practical Application in Resident...
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
2012 Users Conference New Orleans March 20 - 23
Questionmark ™ Perception™
How e-Testing is Improving Assessment for the U.S. Coast Guard
Jim Parry, CPT, Test Development Manager
Art Stark, Instructor
USCG Training Center
Yorktown, VA
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Background
Moving from the past to the 21st century
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
" The Coast Guard is using a slow and archaic paper testing process to measure the knowledge of its members. The Coast Guard has a duty to provide the best operational programs to ensure missions are completed in an exemplary manner. To allow the Coast Guard to continue its long tradition of exceptional service and superior foresight, we must move the current paper based testing system to an electronic medium."
Commanding Officer US Coast Guard Institute, 2005
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Implementation Timeline
1999: USCG Institute (CGI) began transferring processes to electronic medium
2008: USCG awards contract to Questionmark ™
2012: CG wide implementation
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Wide use expected
The Enterprise Solution
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Over 300,000 test takers and over 800,000 test items available! http://www.cutteragent.com/atlas
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Participants
US Coast Guard Personnel 41,000 Active Duty 7,000 Reserve 9,000 Civilian 33,000 Auxiliary NAF and Contracted personnel
National Maritime Center (NMC) Over 240,000 active merchant mariners
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
High Stakes Authors
Enlisted Advancement Testing (40) 19 enlisted specialties 6 testable pay grades in each (E-4 – E-9) Two test types Criterion-referenced Rating Advancement Test (RAT)
Norm-referenced Servicewide Examination (SWE)
Certification Exams (10)
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
High Stakes Testing
Rating Advancement Test (RAT) 2 or 3 for each specialty (E-4 – E-6) Minimum 50 questions each
Servicewide Exam (SWE) 2 parallel exams each exam cycle 150 questions each Active duty & Reserve Forces
Law Enforcement Certification Aviation Certifications
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Low – Medium Stakes Authors
Resident Training (150 - 200) 15 locations Basic and advanced training courses Over 1300 courses One author may handle several courses
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Approval Process
Validity and Reliability
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Workflow
Test item requires minimum of 2 approvals before being published Author Writer/Editor Instructional Systems Specialist
Currently two CG workflows SWE/RAT Resident
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
SWE/RAT Workflow
Author generates/edits question
W/E edits, rejects or forwards
ISS edits, rejects or approves
No changes after final approval High stakes exams
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Resident Workflow
Author generates/edits question
W/E edits, rejects or forwards
ISS edits, rejects or approves
Changes can be made Low/medium stakes
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Standardization
Adds to defensibility of USCG testing
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Standardized Test Item Topic Structure
REPOSITORY (OSC Server) Root Topic (Course) Sub Topic (Unit)
Sub-Sub Topic (TPO) Sub-Sub-Sub Topic (EO)
Test items are derived from the enabling/lesson objectives in the case of EOCT/RAT and resident courses and EPQ/RPQ for SWEs
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Why Standard Structure?
Eliminates ownership confusion Better test item security
Root topic name includes course ID number Matches other CG databases
Allows drill down for coaching, statistical review & defensibility issues Many statistical and grading reports available
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Example Topic Structure
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Making it Real
Improving the Quality of the Testing Experience
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
The Enlisted Rating Advancement Training System
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Purpose
To support the performance of assigned enlisted personnel in preparing for advancement or change in rating
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Functions Define performance requirements Provide training and performance support Assess performance
Doctrine
Training
Assessment
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Realistic Rating Advancement Testing
Able to embed content into test Allows more
realistic approach to solving problem
If reference is available in field it will be available in test item
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Secure™ will prevent “surfing”, printing, etc.
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Selecting reference from list allows participant to view.
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Practical Application in Resident Environment
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Entering test items
As easy as pie!
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
How USCG Does It
Question Wizard Guides author through process
Question Editor Advanced users
Questionmark Live Contributers Allows work from home Cannot use for “For Official Use Only (FOUO)”
ASCII import Harvested from existing item banks
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Defensibility and security
Getting it right
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Use of Standards
ISO 10667-1 & 2 Assessment service delivery – Procedures and methods to
assess people in work and organizational settings AERA, APA, NCME Standards for educational and psychological testing
Interservice Enlisted Testing and Advancement Group (ISETAG) US Navy Advancement Center US Air Force Airman Advancement Division US Coast Guard Forces Command
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Defensibility
Test Plans
Insert reference as feedback
Angoff score
Analytics to support cut score
Ability to track test and item statistics
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Analytics
Test & Item Analysis
Coaching Reports
Gap Reports
Grade Book
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Security
Password protected “strong” password required
Security, non-disclosure, etc. statement is first item on all high stakes exams
Authors, Admins, etc. Required to execute non-disclosure agreement
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Potential Security Issues
Network printers Potential to send outside of secure area
Requires close monitoring when delivering assessments Test takers hidden behind screen
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Standard Operating Procedures
CG established SOP/TTP for e-testing Defines various roles (profiles) Establishes workflow rules Establishes topic structure Provides job aids for certain functions
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Community Sharing
Established collaboration site in the CG Portal Open access behind the USCG firewall Information and job aids available
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Certification & Accreditation of Software
Not just plug and play…
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Software C&A
USCG Telecommunications and Information
Systems Command (TISCOM) Checks for system image compatibility Reviews programming code for potential security holes Conducts penetration testing (Hacking)
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Stakeholders
Who’s who…
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
The Key Players
CG Force Readiness Command (FC-Tadl) Operations and Systems Center (OSC) E-testing managers Coast Guard Institute (CGI) Personnel Services Center (PSC-adv) All Coast Guard Training Centers (TRACENs) Educational Services Officers (ESOs)
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
to the future and beyond
We are on the move!
A wheel is _____. A. flat B. round C. square D. cool
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Shipboard and Remote Delivery
Sporadic or no connectivity Questionmark™ to Go Printing and scanning
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
What’s in Store?
Initial alpha/beta testing went relatively well Identified shortcomings Resolved most issues
Observational assessment Adaptive branching Growing pains are inevitable
2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questions?