Planning, Design, Executionww.w.cedma-europe.org/newsletter articles/Webinars/Performance-Based...

CEdMA SIG Meeting Sep 2013

Performance-based Testing Planning, Design, Execution

Kieran Milne, Juniper Networks Certification Program


• Company and Certification Team

• Juniper Networks Certification Program

• PBT Conception

• PBT Considerations

• Exam Design & Development

• Exam Logistics

• Exam Proctoring

• Exam Operations

• Benefits of PBT

• Lessons Learned

Agenda

2


• Juniper Networks

– Computer networking; routing, switching, security

– Competitors include Cisco, Alcatel-Lucent, Extreme,

HP, etc.

• JNCP team

– 6 employees

• Liz Burns, Director

• Kieran Milne, Program Manager / Tech Lead

• 2 Tech Leads / Exam Proctors

• 2 Operations folks

Company and Certification team

3




• PBT Conception

• PBT Considerations


• Exam Logistics

• Exam Proctoring

• Exam Operations

• Benefits of PBT

• Lessons Learned

Agenda

4


• Candidate reach: global

– Serves customers, partners, employees

• Content scope: validate technical expertise/depth

– Not for sales/pre-sales

• Certification/exam scope:

– 19 written exams

– 4 lab exams

Juniper Networks Certification Program

(JNCP)

5


JNCP Framework

6

Asso

cia

te

Pro

fessio

nal

Sp

ec

ialist

Ex

pe

rt

JNCIA -

Junos

JNCIS-SP JNCIS-

SEC

JNCIS-

ENT

JNCIP-SP JNCIP-

ENT

JNCIP-

SEC

JNCIE-SP JNCIE-

ENT

JNCIE-

SEC

JNCSP-

SP

JNCSP-

ENT

JNCSP-

SEC

JNCIP-SP JNCIP-ENT JNCIP-SEC P P P

Network Engineer Support Specialist

JNCIA-

E

JNCIS-

E

JNCIP-

E

JNCIS-

FWV

JNCIS-

WLAN

JNCIS-

SA

JNCIS-

AC

JNCIS-

QF

JNCIA-

IDP

P Out-of-track Prerequisite

=

Product/Technology Track

JNCIS-ENT P

Information current as of September 2013

Service Provider

Routing and Switching

Enterprise Routing

and Switching

Junos Security

Service Provider Routing and

Switching

Enterprise Routing

and Switching

Junos Security

E Series Firewall/ VPN

Wireless LAN

Junos Pulse

Secure Access

Junos Pulse

Access Control

QFabric IDP Series




• PBT Conception and Considerations


• Exam Logistics

• Exam Proctoring

• Exam Operations

• Benefits of PBT

• Lessons Learned

Agenda

7


• Factors in deciding to undertake a lab exam effort

– Desire to validate skills at highest level

• Attempt to replicate the actual skills being validated

– Desire to provide a high-level reference point for

expertise to the marketplace

– Desire to meet customer demand

• PBT candidates are most committed to the program

– ‘Me-too’ factor

PBT Conception

8


• Planning elements to consider

– Cost

• Input: resourcing, capex, opex

• Output: ROI? Strategic value?

– Scale

• Simulations in written exam?

• Full hands-on ‘lab’ exam?

• Live panel / adjudication?

– Staffing/resourcing

• Is there staff available to spend time on this?

• Does staff require specific skills?

PBT Considerations

9


• Planning elements to consider

– Time commitment

• Exam development time can be substantial

• Staffing requirement can be substantial

– Supporting/study resources

• Courseware?

• Self-study guides?

• Practice tests?

– Logistical constraints

• Equipment availability?

• Lab access limitations?

PBT Considerations

10






• Exam Logistics

• Exam Proctoring

• Exam Operations

• Benefits of PBT

• Lessons Learned

Agenda

11


• Pre-design/pre-development

– Establish some level of scope

• Scope of job role

• Scope of topics/content

• Scope of products

• Etc…

– Tends to be some degree of ‘business reality’ for

vendor-specific exams

• Cannot be all things to all people

• Cannot truly cover all aspects of a job role

– Role is often too diverse for one vendor

Exam Design & Development

12


• Exam development workshop (1 week)

– Mini-JTA / task expansion / blueprint

– Topology creation

– Task creation / development

– Technical review

– Fixes / edits / changes

• Subject matter experts (SMEs)

– Cross-section of experts (internals/externals) who:

• Do (or have done) the job role

• Have significant exposure to the environment of the job role


13


• Post-workshop

– Tech Lead takes on remaining steps

• Assemble tasks/sections into cohesive exam

• Translate network topology and tasks onto actual equipment

• Run alpha/beta testing

• Update exams based on alpha/beta feedback

• Create supporting documentation

– Answer guides / grading docs / grading

• Create supporting elements for lab equipment

– Starting-configs, start-up/reload/reset scripts, grading scripts


14

Post-workshop effort is generally much greater than workshop itself


• Post-workshop

– Sample task list


15

Planning note:

Timelines may be significantly

extended when tasks are

integrated among other work






• Exam Logistics

• Exam Proctoring

• Exam Operations

• Benefits of PBT

• Lessons Learned

Agenda

16


• Lab equipment

– Juniper uses real equipment

• One rack per candidate

– All hosted in Sunnyvale, CA

• No need to ‘travel’ the equipment

– Considerations:

• Equipment cost

• Space, power requirements

• Installation

• Location of equipment <> candidates

– Considering virtualized equipment

for future

Exam Logistics

17


• Testing locations

– Global coverage

– Combination of Juniper and training partner venues

– Can go onsite to a customer if required

• Lab access

– Secure/VPN access to Sunnyvale, CA

• Good network connection required

– Use ‘trusted’ PCs/laptops at testing location

• Never use candidate laptops

Exam Logistics

18

Internet

Cert lab

VPN

Access portal Lab racks


• Exam forms

– Provided to candidates

• Multiple sections with tasks, supporting info/exhibits

• Network topology diagram

– Currently paper-based

• Created as PDFs; provided to proctors, to be printed

• Hand out at exam start; collect at exam end

– Electronic forms are interesting

• Increase security; easier to manage

• Increase capex/opex

Exam Logistics

19


• Supporting documentation

– Provided to exam proctors

• Detailed Description doc

– ‘Walk-through’ guide of exam tasks and answers

• Grading doc

– Instructions to check correctness of exam tasks

– Strict criteria for correctness

• Infrastructure details doc

– Diagram of the underlying physical lab environment

– Useful for troubleshooting network issues

Exam Logistics

20


• Operations planning

– Develop global exam delivery schedule that coordinates

• Proctors (most not dedicated proctors)

• Lab equipment and PCs/laptops

• Facilities

– Provide an exam registration and payment system/process

• Communications/marketing

– Announcements to certified audience

– Releases to industry press

– Promotions to likely lab exam candidates

– Develop sense of ‘prestige’ around these certifications

Exam Logistics

21






• Exam Logistics

• Exam Proctoring

• Exam Operations

• Benefits of PBT

• Lessons Learned

Agenda

22


• Role of a proctor

– Ensure that every candidate receives a consistent and

fair testing experience

– Provide security for the lab exam event

– Onsite ‘representative’ for JNCP

– Protect the integrity of the JNCP and all certification

credentials

– Provide technical support/expertise

• Juniper proctors all JNCIE certified

Exam Proctoring

23


• Proctor tasks

– Prepare the testing environment

– Manage and control the distribution of proprietary

documents

– Support candidates during the exam • Verify the operation of lab equipment

• Answer candidates' questions

– Backup candidates’ work after the exam

– Grade exams

Exam Proctoring

24


• Building a proctor team

– Internal team

• Cert team staff

– External team

• Other internal employees

• Training partner instructors

• Channel partner engineers

– Primary consideration: the TRUST factor

Exam Proctoring

25


• Training a proctor team

– Pre-requisites

• Juniper requires that proctors have passed the exam

• Excellent way to expose proctors to the ‘front’/candidate side of

the exam

– Training materials and guides

• Proctor Training doc

• Policies and Guidelines doc

• Proctor Guide doc

• Lab Access doc

– Practice grading

• Excellent way to expose proctors to the ‘back’ side of the exam

Exam Proctoring

26

Full proctors are

3xJNCIEs.. a

small/select group






• Exam Logistics

• Exam Proctoring

• Exam Operations

• Benefits of PBT

• Lessons Learned

Agenda

27


• Candidate scheduling and payment

– Online exam registration page

• Candidates can choose exam, select location and date, pay

• JNCP Ops team can add/edit/open/close events, assign proctors

Exam Operations

28


• Lab equipment usage

– Online lab equipment scheduling tool

• Scheduled candidates > rack reservations

– Pre-scheduled exam sessions automatically start up

morning of exam

– Can borrow racks

from training as

needed

Exam Operations

29


• Grading/results

– Grading performed by proctors

– Grades sent to JNCP Ops team

– Ops team sends out results by email

• Allows Ops team to track candidate end to end

• Grading scripts/tools

– Script replaces manual grading

• Connects to lab equipment, checks candidate’s work, calculates

grade, provides output for review (text file, Web page)

– Significant reduction in grading time

• But must be measured against ROI

Exam Operations

30


• Granting credentials/rewards

– Results are manually entered into JNCP candidate

management tool

– Plaque sent out via fulfillment vendor

– Candidate is granted a ‘certification number’

• Ex. JNCIE-SP #380

• Important factor to candidates – ‘lower is better’

– Certificate available online

Exam Operations

31






• Exam Logistics

• Exam Proctoring

• Exam Operations

• Benefits of PBT

• Lessons Learned

Agenda

32


• For candidates

– Increased skillset, marketability

– ‘Marquee’ line item for resume

• For employers

– Provides a point of ‘known reference’ for high-end hiring

– Validates a high level of skills/knowledge/expertise

• For company/vendor

– Extends brand recognition

– Expands pool of expertise in marketplace

– Expands mindshare effort upward, towards decision-

makers

Benefits of PBT

33






• Exam Logistics

• Exam Proctoring

• Exam Operations

• Benefits of PBT

• Lessons Learned

Agenda

34


• Do not implement PBT unless committed to the

required investment in money and resources

• Consider carefully why implementing PBT

– Is there a compelling business return on investment?

• ‘Cascade effect’ of exam refreshes/updates

– Exam forms are just one element

– May also require a full workshop

– May also impact lab equipment, supporting docs,

grading scripts, proctor training, etc.

Lessons Learned

35


• Amount of support documentation required

– Don’t underestimate the time required to develop

and/or update necessary docs to successfully roll out

a PBT

• Trade-offs with equipment location

– Centralized, single location

• Easier to manage, support

• Latency can cause testing issues when testing far from

equipment

– Distributed, multiple locations

• Improved performance as candidates are closer to equipment

• Harder to manage, support

Lessons Learned

36


• Security of documents

– PBT requires a significant investment

– Exam documentation is the tangible output of that

investment

– Consider how much security is ‘enough’ for your

environment

Lessons Learned

37

Questions Kieran Milne

[email protected]

Planning, Design, Executionww.w.cedma-europe.org/newsletter articles/Webinars/Performance-Based...

Documents

Transcript of Planning, Design, Executionww.w.cedma-europe.org/newsletter articles/Webinars/Performance-Based...