June 19, 2012
description
Transcript of June 19, 2012
June 19, 2012
Evolving Your PBT ProgramThe Citrix Story
Kpayah Tamba, Citrix
© 2012 Citrix | Confidential – Do Not Distribute
About Citrix
© 2012 Citrix | Confidential – Do Not Distribute
Multiple-choice Items
Scenario-based Items
Simulations
Combination of Live Labs, Simulations, Board Defense, and Scenario-based Items
At Citrix, we continue to raise the bar
Real World Skills
Live Labs
© 2012 Citrix | Confidential – Do Not Distribute
Citrix Certification Program
CCIA
CCEE
CCAA
CCA
Top 3 cert tiers consist of exams with performance based items
© 2012 Citrix | Confidential – Do Not Distribute
Design Approaches at each Tier
Board Defense, Live Lab, and Simulation
• CCIA
Simulation and Scenario-based
• CCEE
Simulation and Scenario-based
• CCAA
Mostly Scenario-based, with one exam containing simulation
• CCA
© 2012 Citrix | Confidential – Do Not Distribute
Citrix Evolution
1997
2003
2006
2012
Multiple-choice based exams
First PBT based exam
75% of cert tracks now
includes PBT exams
First simulation
based exam
How we evolved our program
© 2012 Citrix | Confidential – Do Not Distribute
• Standard multiple-choice: single response and multiple response items
Phase I: MCQ Approach
© 2012 Citrix | Confidential – Do Not Distribute
Phase I: MCQ Cont’d
• Prosᵒ Easier to developᵒ Cheaper to developᵒ Easier to gradeᵒ Easier to deliver
• Consᵒ Highly vulnerable to item theftᵒ Not the best item type for assessing
performance based tasks
© 2012 Citrix | Confidential – Do Not Distribute
Phase II: Live Lab Approach
• Testing using the real products in a virtual environment
© 2012 Citrix | Confidential – Do Not Distribute
Phase II: Live Lab Cont’d
• Prosᵒ Test takers embraced this new form of
testingᵒ Citrix was able to measure hands-on
skillsᵒ The value (face validity) of our
certification program increased
• Consᵒ This new form of testing was not
scalableᵒ Candidates had to travel long distances
to take the examᵒ The design and delivery mechanisms
limited the number of exams that could be delivered at any given point
ᵒ The manual scoring resulted in long wait times for results
© 2012 Citrix | Confidential – Do Not Distribute
Phase III: Simulation Approach
• Testing using simulated environments that look and function almost like the real products
© 2012 Citrix | Confidential – Do Not Distribute
Phase III: Simulation Items Cont’d
• Prosᵒ Simulations are scalableᵒ Simulations allow for the measurement
of hands-on skillsᵒ Scoring is automated
• Consᵒ Simulations are costly to buildᵒ Simulations take a considerable
amount of time to design and developᵒ Our simulations are closed-ended;
therefore they imposed limitations on our test takers
© 2012 Citrix | Confidential – Do Not Distribute
Phase IV: Board Defense and Live Lab Items Approaches
• In 2012, Citrix rolled out a six-week intensive training program geared towards certifying our top level audience
• At the end of six weeks of training (eight hours a day, five days a week), candidates are required to complete a rigorous 14-hour hands-on practicum in one sitting, where they build a virtualization solution consisting of Citrix products
• Candidates must also defend the solution they build before a panel of SME judges
© 2012 Citrix | Confidential – Do Not Distribute
Phase IV: Board Defense and Live Lab Items Cont’d
• Prosᵒ The use of both live lab and simulation
items allow for Citrix to continue to measure hands-on skills at a high level, while using two different design approaches based on our need to scale
ᵒ The use of the board defense approach allows us to determine whether test takers know the “why” behind the “how”
• Consᵒ The cost to develop simulation items is
still a problemᵒ Simulations take a considerable
amount of time to design and developᵒ Live labs are difficult to scaleᵒ Live labs are difficult to scoreᵒ A uniform board defense experience is
also difficult to maintain
So what’s next for us at Citrix?
© 2012 Citrix | Confidential – Do Not Distribute
Next Steps
• Increase our PBT footprint
• Fully automate the scoring of our live lab items
• Complete the development of a grading tool for our board assessment
• Localize our live lab items
• Localize our board assessment exam
Conclusion
© 2012 Citrix | Confidential – Do Not Distribute
Conclusion
• As you embark on your next steps to expand your PBT program, ask yourself the following questions:ᵒ How will this design approach improve my program?ᵒ How do I start?
• Do I have the budget?• Do I have the skills, and if not, can I hire or contract the right talents?
ᵒ What is my end goal?ᵒ How do I get there?
© 2012 Citrix | Confidential – Do Not Distribute
Q & A
Work better. Live better.
© 2012 Citrix | Confidential – Do Not Distribute
Group Activity• The design approach document provides:
ᵒ Examples of design approachesᵒ Characteristics of those design approachesᵒ When to use themᵒ When not to use them
• Exercise Objective:ᵒ This exercise will help us identify solutions to potential issues that one could encounter
when using a design approachᵒ We will break up into four groupsᵒ Each group will focus on an approach
• Group A – Live Environment• Group B – Board Assessments/Oral Exams• Group C – Simulating Software Using Software• Group D – Simulating Hardware Using Software
© 2012 Citrix | Confidential – Do Not Distribute
Group Activity – Cont’d
• In each group, for the assigned design approach, identify solutions to the following issues:ᵒ Skills required to develop item typeᵒ Delivery challengesᵒ Psychometric challengesᵒ Other challenges
© 2012 Citrix | Confidential – Do Not Distribute
Wrap Up
• We will regroup
• A representative from each group will share with us how his or her group plans to address the challenges associated with their assigned design approach