Managing Multiple Credentialing Programs
Transcript of Managing Multiple Credentialing Programs
Managing Multiple Credentialing Programs
December 2018
Robert C. Shaw, Jr., PhD National Board for Respiratory Care
Lori M. Tinkler, MBA
National Board for Respiratory Care
2
Published by Institute for Credentialing Excellence 2025 M Street, NW, Suite 800 Washington, DC 20036 Copyright © 2018 Institute for Credentialing Excellence Printed in the United States of America
3
Table of Contents
Abstract ......................................................................................................................................................... 4
Introduction.................................................................................................................................................... 5
Adding a new program .................................................................................................................................. 6
Describing the interrelationships between programs .................................................................................... 6
Managing resources ...................................................................................................................................... 8
Financial .................................................................................................................................................... 8
Staff ........................................................................................................................................................... 8
Vendor ....................................................................................................................................................... 9
Examinations and their administrations .................................................................................................. 10
Examination committees and banks ....................................................................................................... 11
Project management ............................................................................................................................... 11
Panels ..................................................................................................................................................... 12
Volunteer turnover ................................................................................................................................... 12
Representing credential holders on the board ............................................................................................ 13
Standardizing those things that can be unified across programs ............................................................... 14
Choosing examination formats and methods to ensure equity across test forms ...................................... 14
Summary of key points ................................................................................................................................ 15
4
Abstract
This white paper describes one organization’s experience managing multiple credentialing programs. The
authors have in mind an audience of those who lead organizations responsible for more than one
credentialing program or who may add a program. Recognizing that every organization’s circumstances
are unique, the authors hope that readers will find value in the ideas described and in linkages of these
ideas to the National Commission for Certifying Agencies (NCCA) Standards.
The NBRC awards eight credentials based on results from six examinations. Each examination is the
responsibility of a committee that follows standard policies and procedures of the board of trustees. The
board consists of 31 trustees, each of whom is appointed to one or two examination committees, except
for the public advisor. Not only do trustees make policies, but they also serve as content experts on
panels during examination cycles. These panels are supplemented with other consultants who become
another part of the system to be managed.
The paper will address the following topics:
adding a new program,
describing the interrelationships between programs,
managing resources,
representing credential holders on the board,
standardizing those things that can be unified across programs, and
choosing examination formats and methods to ensure equity across test forms.
White papers are products of the ICE Publications Committee and offered as a complimentary resource
for ICE members.
5
Introduction
The National Board for Respiratory Care (NBRC) operates programs culminating in eight credentials
based on results from six examinations. Examinations have varying eligibility pathways in recognition of
the diversity of preparation of qualified applicants. Earning NBRC credentials demonstrates that allied
health professionals who perform respiratory care in the United States are committed to public safety.
Within the NBRC’s program portfolio, some were started by the NBRC, while others were started by other
organizations and then transferred to the NBRC. Milestones occurred as follows:
1960s: The first two credentials are awarded to respiratory therapists in general practice.
o 1961: The Registered Respiratory Therapist (RRT) program was the first program.
o 1969: The Certified Respiratory Therapist (CRT) program was started by the professional
association of respiratory therapists, which is now called the American Association for
Respiratory Care (AARC).
The CRT scope of practice was first a subset of RRT practice; today, the CRT
documents a lower proficiency level compared to the RRT within the same scope
of practice.
The CRT program was transferred to the NBRC’s control in 1975; the program
was first linked to state licensure in 1976.
Eventually, every state except Alaska linked licensure to the CRT; a handful of
states have linked licensure to the RRT.
1980s to present: The remaining credentials are awarded in specialty areas of respiratory care
that subsequently emerged.
o 1982: The Certified Pulmonary Function Technologist (CPFT) and the Registered
Pulmonary Function Technologist (RPFT) programs were transferred from the National
Society of Cardiopulmonary Technology.
o 1991: The Neonatal/Pediatric Specialty (CRT-NPS, RRT-NPS) program was started by
the NBRC at the AARC’s request.
o 2008: The Sleep Disorders Specialty (CRT-SDS, RRT-SDS) program was started by the
NBRC at the AARC’s request.
o 2012: The Adult Critical Care Specialty (RRT-ACCS) program was started by the NBRC
at the AARC’s request.
Each credentialing program is NCCA accredited; the first accreditation occurred in 1977 when the NCCA
itself began. After its incorporation in 1960, the NBRC could only start a new program when asked to do
so by the AARC or one of its three physician-sponsoring organizations: the American Thoracic Society
(ATS), the American Society of Anesthesiologists (ASA), and the American College of Chest Physicians
(CHEST). The physician-sponsoring organizations and the AARC lend support by nominating trustees of
the NBRC. There are 31 trustees, including one public advisor, 15 physicians, and 15 therapists, some of
whom have documented competence in at least one specialty area.
The NBRC financially supports itself through fees collected from candidates and credential holders.
6
Adding a new program The NBRC began running multiple programs through transfers in two cases and additions in other cases.
The organization follows a process that is written into its policy when a formal request for a new program
is made. The process involves the following studies, any of which could stop the pursuit of a new
program:
viability,
personnel, and
job analysis
To begin a viability study, the NBRC assembles a focus group of stakeholders. The focus group’s goal is
to evaluate whether consensus exists about the need for a new credentialing program and the likely
impact the program could have. A set of questions is proposed; brainstorming of responses follows until
all ideas have been documented. Multi-voting follows so responses that are most strongly endorsed can
be observed. The proposed questions vary with the proposed program. Our most recent experiences
have involved proposals for emerging specialty areas of practice. Responses to approximately a dozen
questions were evaluated, some of which were as follows:
“What do specialists do that generalists do not do (or do not do as well)?”
“To what extent are specialists uniquely prepared to anticipate patient’s needs?”
“How would the presence of credentialed specialists affect staffing?
If the proposal survives the viability study, then a personnel study follows. The goal is to evaluate whether
enough individuals will pursue a new credential over a multi-year period. A two-pronged sampling plan
can help: one focused on individuals and another focused on employers. Some questions we have asked
individuals during this phase include “Do you work in the specialty?” and “Would you seek a new
credential?” Some questions we have asked employers include “Would a specialty credential add value?”
and “Do you expect that your institution would support a new credential?”
If sufficient endorsement is observed in personnel study results, then the NBRC performs a job-analysis
study. We know that most do not see a job-analysis study as a test of new program feasibility. However,
should a group of content experts be unable to brainstorm a body of job-related content, or should job
incumbents’ responses to the body of content fail to endorse a sufficient proportion of the content, then
program development would stop.
Every program request that has made it to the third step has proceeded to full development in our
experience, but the NBRC did have one proposed program stop after the first step. In that case, the
program (RRT-ACCS) proposal returned about 20 years after the specialty had more fully emerged. A
second attempt at the steps outlined above produced a new credentialing program.
Describing the interrelationships between programs
Some NBRC credentials are based on passing one examination, while other credentials are based on
passing multiple examinations. These systems have contributed to the operation of multiple credentialing
programs. To understand the difference between these programs, compare the eligibility criteria plus the
required examinations for each credential shown in Table 1.
7
Table 1. Credential eligibility criteria and required examinations
Credential Eligibility Examination(s)
Take one examination
CRT Graduation from an accredited education program Therapist Multiple-Choice
CPFT1 Experience or CRT or RRT credential
Pulmonary Function Technology
RPFT1 Experience or CRT or RRT credential
Pulmonary Function Technology
Take two examinations
RRT Graduation from an accredited education program Therapist Multiple-Choice Clinical Simulation
CRT-SDS Six months beyond achievement of the CRT Therapist Multiple-Choice Sleep Disorders Specialty
Take three examinations
RRT-ACCS A year beyond achievement of the RRT Therapist Multiple-Choice Clinical Simulation Adult Critical Care Specialty
RRT-NPS RRT Therapist Multiple-Choice Clinical Simulation Neonatal/Pediatric Specialty
RRT-SDS Three months beyond achievement of the RRT Therapist Multiple-Choice Clinical Simulation Sleep Disorders Specialty
Details about the Therapist Multiple-Choice Examination and the Pulmonary Function Technology
Examination are worth discussing to explain how managing multiple programs has recently eased. When
originally conceived, the CRT credential covered a subset of content covered by the RRT credential.
Some candidates were only educated on the CRT content for a couple of decades, so they took one
examination. Other graduates were educated on the broader RRT body of content. Graduates of these
programs who chose to pursue the RRT credential took the same multiple-choice examination taken by
CRT candidates plus a second multiple-choice examination before taking the Clinical Simulation
Examination. Nearly two decades ago, the Commission on Accreditation for Respiratory Care (CoARC)
stopped accrediting education programs that only prepared candidates for the CRT credential. A few
testing cycles later, content expectations for CRTs and RRTs had merged, as revealed by subsequent
job-analysis studies. The second multiple-choice examination was removed from the RRT credentialing
system and replaced by a high cut score on the Therapist Multiple-Choice Examination. Candidates who
pursue the CRT credential and become licensed in most states still take only one examination.
Candidates who pursue the RRT credential now take two examinations instead of three examinations.
Likewise, the CPFT credential was linked to a subset of content covered by the RPFT credential. Over a
series of job-analysis results, a merger of content was observed. This led to the removal of what had
1 CPFT and RPFT credentials are attained by personnel who perform the tests of pulmonary function (e.g.,
airflow measurements, volumes, diffusing capacity) that are necessary to diagnose lung diseases. Within the same scope of practice, the CPFT documents a lower proficiency level compared to the RPFT. While each healthcare facility sets its own hiring criteria, a person with the RPFT credential is often in charge of a whole lab, while a person with the CPFT credential works with patients while testing is done.
8
been the RPFT examination. Candidates now take one examination; their scores are compared to a low
cut score and a high cut score. Achieving the low cut recognizes a new CPFT, while achieving the high
cut recognizes a new RPFT. The RPFT and RRT credentials document higher levels of proficiency today
rather than mastery of a broader scope of content.
Managing resources Financial Although the NBRC enjoys the support of four sponsoring organizations, financial support is neither given
nor needed. Fees collected from candidates and credential holders provide the necessary support. NCCA
Standard 4: Financial Resources states, “The certification organization must have sufficient financial
resources to conduct ongoing, effective and sustainable certification and recertification
activities.” There is a subtle shift in the language in this standard. Many standards hold each program
accountable, while Standard 4 holds the organization accountable.
The NBRC’s situation illustrates why this shift in syntax makes sense. Starting with the addition of the first
specialty program described in the Introduction, each program has fallen short of entirely paying for itself.
Speaking to Standard 4, the difference between the resources expended to sustain each specialty
program and the resources brought in by the program come mostly from fees paid by those who have or
will achieve the CRT and RRT credentials. Because any in this population could attain specialty
credentials in the future, this scenario is judged equitable.
The comparatively high number of people who seek initial CRT and RRT credentialing also leads to the
release of more test forms each year in support of these credentials than are released for any specialty
program. Our intent is to prevent the overexposure of examination content when we step up the annual
release of test forms. A specialty program bringing in less revenue than is expended to sustain it still
releases a new test form each year because board members know that doing so is vital to ongoing
content relevance, test score accuracy, and public protection.
Having added two programs in the last decade, we will speak to the budgetary impact of those additions.
Adding these programs has increased total expenses for the organization, as each examination
committee now requires more consultants than in the past; plus, there are more committees. However,
our board is large, containing 30 content experts. Reassignment of some of them to newly added
programs has avoided a complete one-for-one addition of volunteers whose travel expenses must be
covered. Hence, while having a large board increases the expense of hosting a board meeting, it has
given us capacity that we have tapped to add programs.
Staff Two elements of NCCA Standard 5 state that each program must identify personnel who conduct
certification activities and that the program must oversee those personnel. The board hires a chief
executive officer (CEO), who, in turn, hires staff to fill various roles. The board reviews the CEO’s
performance each year. Anticipating that it may help other organizations to learn about vital personnel,
the following is a breakdown of the people that the NBRC employs to operate its six examinations and
eight credentialing programs:
9
Chief
o Executive Officer (CEO)
o Financial Officer (CFO)
Executive Support
o Executive Assistant
o Administrative Assistant/Meeting
Planner
o Accounting/HR Coordinator
o Marketing/Communications Coordinator
Information Systems
o Principal Software Engineer
o Systems Administrator
Examinations
o Vice President (in-house
psychometrician)
o Research Associate
o Examinations Coordinator (2)
o Assessment Coordinator
Credentialing Operations
o Vice President
o Certification Services Coordinator
o Certification Services Specialist
o Customer Care Specialist (4)
The vice president for credentialing operations touches candidates and credential holders across all
programs. The vice president for examinations touches each form of each examination along with the job-
analysis, cut-score, and criterion-validation studies done for the programs.
No staff person is fully dedicated to just one examination or just one credentialing program in our
organization, which helps ensure that the needs of each examination and program are met. The closest
we come to dedicated staff occurs when each of the two examinations coordinators is assigned as the
primary person responsible for assembling forms of an examination. Each coordinator is responsible for
three examinations. The two examinations coordinators regularly cover for each other while using the
same checklists and procedures, so such crossover occurs without feeling extraordinary.
Crossing over to work outside one’s normal scope extends to other personnel in the examinations division
when it comes to covering face-to-face meetings and soliciting new items. Our board meets twice a year,
so examination committees meet three to four days leading up to the board meeting. However, some
examination committees only need to meet once a year, which helps staff manage coverage. There are
still some days when three examination committees meet simultaneously, at which point our research
associate becomes the person who facilitates item approval like our examinations coordinators do.
Cross-covering occurs in the credentialing operations division as well. While a pool of customer care
specialists takes phone calls from candidates and credential holders, other operations personnel can
cover calls. Likewise, the fulfillment tasks done by the certification services specialist can be done by
others from the operations area.
Finding smart, adaptable people who can learn to anticipate needs is key to making the system work
since it relies heavily on cross-coverage.
Vendor Absent from the list of NBRC staff are vendor personnel who administer examinations at nearly 300 sites
and support the network that allows those administrations to occur. Most sites are in the United States.
The vendor provides scoring services whose results are posted to item banks that the vendor hosts.
NBRC staff are authenticated to access those banks so they can assemble new test forms and maintain
item content. If the NBRC had no in-house psychometrician or in-house examinations coordinators to
assemble test forms, then a vendor could provide those services. Development and maintenance of a
database of credential holders and candidates plus related software are supported by a vendor on an as-
10
needed basis. Standard 10: Confidentiality holds NBRC staff and staff employed by the vendor(s)
responsible for maintaining confidentiality about examinations, candidates, and credential holders.
Examinations and their administrations Credentials that candidates earn are based on the results of six examinations. All but one of these
examinations are assembled from multiple-choice items, each of which contains four options. The
exception is the Clinical Simulation Examination, which is assembled from patient management
problems.2 If a proposal for a new program included an examination format other than one that contained
multiple-choice items or patient management problems, then it would be further scrutinized regarding its
likely budgetary impact.
Each of the six examinations is administered on demand, regardless of these two formats. We could
envision a testing-window scenario working, but we opted for on demand to maximize access for
candidates. Linkages to licensure for some examinations especially encouraged this choice. Pretesting of
items and problems is deployed for all examinations, with the primary intent to enable the delivery of
results at the end of each administration session. The benefit of pretesting is an elevation in the quality of
items that contribute to candidates’ scores.
Avoiding administration windows and instead deploying items throughout the year means that we do not
concern ourselves with calendar effects on items. Item sequences are automatically scrambled among
candidates: sometimes an item is placed early on a test form, and sometimes it is placed it toward the
middle or the end. Hence, we need not concern ourselves with preserving the home position of an item
on a new test form to be confident that it will perform the same. While on-demand test administrations
throughout the year produce their own challenges, administering all examinations under this model
simplifies the work of managing multiple programs.
Challenges created by on-demand testing with no waiting period are the potential for item overexposure
plus an increased likelihood that a repeat candidate may pass because of more experience taking the
tests rather than because he or she remediated and became a stronger practitioner. Starting in 2020, the
NBRC will follow a new policy about waiting between repeat attempts. Candidates may still make the first
couple of attempts without waiting until the probability of a false-negative outcome shrinks to zero.
Thereafter, candidates may still test on demand, but they will wait 120 days between attempts when
seeking CRT or RRT credentials or 180 days when seeking a specialty credential.
2 These problems present scenarios describing patients during a clinic visit, in the emergency department, just after hospital admission, or days into a hospital stay. Problems contain between four and six sections. Each section contains at least four options, although some may contain more than a dozen options. Each option is given a score of -3, -2, -1, 0, 1, 2, or 3. Some sections prompt candidates to gather more information, while other sections present opportunities to make clinical decisions. The problem cut score is the sum of positive option scores the examination committee has labeled as required or negative option scores labeled as forgiven, either of which is selected by minimally competent candidates. The examination cut score is the sum of the twenty problem cut scores.
11
This policy change illustrates another key point behind managing multiple programs. We have learned to
give ourselves time to implement changes in an orderly manner and to communicate about those
changes with stakeholders.
Examination committees and banks The content of each examination is the responsibility of an examination committee. The names of these
committees and the number of members on each committee are shown in Table 2. Note that some
individuals serve on two committees.
Table 2. Examination committee composition
Examination committee name Credentialing outcome
Number of therapists/ specialists
Number of physicians
Therapist Multiple-Choice CRT, RRT eligibility 6 4
Clinical Simulation RRT 6 6
Pulmonary Function Technology CPFT, RPFT 5 4
Adult Critical Care Specialty RRT-ACCS 5 5
Neonatal/Pediatric Specialty RRT-NPS 5 4
Sleep Disorders Specialty CRT-SDS, RRT-SDS 4 4
Examination committee members can access each new test form as it is assembled as a part of the
approval process, but they cannot access an entire item bank. Such access is restricted to staff in the
examinations division. Among the NBRC’s credentialing programs are several interrelationships
concerning eligibility and proficiency levels. From the perspective of a credential holder, this means taking
another examination to achieve a specialty credential or to document a higher proficiency level. The
board follows a policy that prohibits retesting candidates over a point of content at the same complexity
level. Such a policy has led to a practice in which no items are simultaneously shared across
examinations. We have had items cross banks when testing cycles begin and end. However, once
moved, the item has a new home, so there is no need to ensure subsequent revisions in one bank are
made in another bank.
Project management Purposefully disruptive projects linked to the start of a testing cycle, like a job-analysis study and cut-
score study, are planned for these six examinations more than a decade into the future. Doing so typically
allows us to tackle just one such project in a given year, but some years, there are two projects.
Experience tells us that this cadence spreads the burden for volunteers and staff, so these projects can
succeed without stressing the system. Another way that we create capacity for these projects is to
implement fewer test forms toward the end of testing cycles while letting each one run longer.
12
Panels When we start a new testing cycle, examination committee members may participate in the job-analysis
and cut-score activities. However, panels are expanded to include others who represent stakeholders.
When we perform a cut-score study, we specifically add others who recently achieved the credential to
ensure that people like future candidates contributed to the outcome.
NCCA Standard 13: Panel Competition provides guidance about managing examination committees and
other groups that are formed. Standard 13 states, “The certification program must use panels of
qualified subject-matter experts to provide insight and guidance and to participate in job analysis,
standard setting, and other examination-development activities.” Our observations from renewals of
NCCA accreditations for NBRC programs at the end of 2017 encourages us to share what became a
crucial point. It is no longer sufficient to list the name, academic degree, credential, and geographic
location of each panel member. The NCCA is likely to expect to see information supporting the conclusion
that each member was qualified. Hence, we also reported information about years of experience, name of
employer, and position or title.
While preparing for two cut-score studies linked to specialty programs in 2018, we decided to collect still
more information about panel member experience. Anticipating that a sole response about experience
was tied to overall experience in the mind of a panel member, participants in 2018 cut-score studies also
told us the number of years they have worked in the specialty and the years since their competence in the
specialty was documented. For example, someone could have 12 years of overall experience, 5 years
working in a specialty, and documented competence in the specialty 2 years ago. Collecting such
information permits a more nuanced way to describe qualifications.
Volunteer turnover Before leaving the discussion about panels, we will point out a statement in the Commentary section of
Standard 13: “A system of terms of service that includes a rotation schedule for panel membership
is a useful means of ensuring broad input into the examination program.” Just as it is useful to plan
years in advance about when to sequence the start of each new testing cycle among our multiple
programs, it helps to plan the replacement of a small number of examination committee members
regularly. There is a competing goal to maintain continuity among groups charged to do examination-
related work. We strive to typically replace no more than a couple of members on an examination
committee of eight to ten people.
We can share a strategy the NBRC has deployed to fill openings with qualified volunteers. We have
recently surveyed populations of credential holders so those who are interested in serving as an item
writer, examination committee member, or credential maintenance assessment panel member could
nominate themselves. After collecting demographic information and creating a scoring rubric to
emphasize characteristics we valued, a ranked list became a vital resource when filling openings.
We think that having multiple examination committees likely helps us retain the capable volunteers we
rely on. Part of the payoff for them is the opportunity to interact directly with high-performing, committed
professionals like themselves. Still, those other people are likely to do at least some things differently, so
navigating those differences while working together on items and problems is uniquely challenging and
rewarding for our volunteers. We also organize group lunches and dinners while ensuring their travel
expenses are covered, all of which helps us retain volunteers. We do not want to give the impression that
13
no examination committee member resigns before his or her term ends, because a few have. However,
most complete their full terms.
Representing credential holders on the board Within the respiratory care profession, there are three organizations, each responsible for a different
mission: the AARC is the professional membership organization and advocates for professionals, the
CoARC accredits education programs, and the NBRC credentials individuals. Standard 2: Governance
and Autonomy states, “The certification program must be structured and governed in ways that are
appropriate and effective for the profession, occupation, role, or specialty area; that ensure
stakeholder representation; and that ensure autonomy in decision-making over all essential
certification activities.” Maintaining autonomy over decisions about credentialing programs while
avoiding undue influence as required by essential element A is straightforward when there are three
independent organizations.
Essential element B of the Standard 2 requires an accredited organization to identify its status as a legal
entity. The NBRC is a nonprofit, as defined by section 501(c)(6) of the Internal Revenue Code. Other
essential elements mandate that the board must represent credential holders. Table 3 summarizes the
policy about populating the board to ensure that people who hold each of the credentials awarded by the
board are represented. Here again, having a board of 31 people gives us the capacity to ensure that each
credential is continually represented as trustees rotate on and off.
Table 3. Examination committee composition
Clinician type Sponsoring group At least one Number Voting Elected
Physician American College of Chest Physicians (CHEST)
Pulmonologist or critical care specialist
5 Yes Yes
American Society of Anesthesiologists (ASA)
Anesthesiologist 5 Yes Yes
American Thoracic Society (ATS)
Pulmonologist or critical care specialist
5 Yes Yes
Therapist
Specialty Therapist
American Association for Respiratory Care (AARC)
NBRC Credential:
CRT or RRT
CPFT or RPFT
CRT-SDS or RRT-SDS
RRT-NPS
RRT-ACCS
15 Yes Yes
Public Advisor 1 Yes Yes
Total 31
14
Standardizing those things that can be unified across programs The NBRC has about 50 volunteers touching the content of six examinations each year. Some NBRC credentials require taking multiple examinations, so consistency in the way that we present content across examinations is vital. Standardizing content format across examinations is a priority because allowing variation where it does not belong could be unfair to candidates. One way that the NBRC encourages consistency in the presentation of content is by managing a standing committee composed of the chairperson and vice chairperson of each examination committee. This committee is chaired by the immediate past president of the board. In addition to optimizing test form production procedures with staff, the committee of examination committee chairmen is responsible for an internal document called the Standardized Testing Guide. Attributes of the Guide include the following:
it provides examples of examination content and how each should be presented in items and
problems;
it summarizes evidence-based protocols that have been widely accepted as guides to clinical
practice;
it describes standardized presentations of clinical information about patients, drug names, and
data tables;
it becomes a reference for staff as items and problems enter the system and as examination
committees edit examination content; and
it contains about 30 pages.
The Guide is updated twice a year during the two weeks when meetings of examination committees,
standing committees, and the board occur. In recent memory, the Guide has been revised in small ways
during each meeting. Staff then use the new Guide over the next six months.
Choosing examination formats and methods to ensure equity across test forms A program may choose one of two methods when facing the challenge of deploying multiple test forms
within a testing cycle during which the same specifications for test form assembly are used. One can
transform scores from different forms so they are reported on a single scale, or one can manage test form
assembly so the scale of raw scores remains constant. The NBRC has chosen the latter method since the
year 2000, which was when computerized administrations began. Choosing a constant set of raw scores
is only available when the scored portion of each new test form is entirely assembled from items whose
keys are already validated and whose statistical properties are confidently known. A pretesting system is
required to make this happen. Either method can accomplish the goal described by Standard 21:
Examination Score Equating, which requires certification programs to “demonstrate that different forms
of an examination do not advantage or disadvantage candidates because of differences in the
content framework and/or difficulty of particular forms.”
Any who consider the choice between these two equating methods should understand that when
deploying the assembly management method, raw score characteristics (e.g., mean, standard deviation,
and standard error of measurement) may not persist into the next testing cycle. In fact, the start of each
15
cycle is an opportunity to strengthen the accuracy of test scores. Choosing to do so leads to the selection
of items that are more difficult and more strongly discriminate. If one can assemble the first test form(s) of
the next cycle from items that are generally more difficult and discriminate more strongly than occurred in
the previous cycle, then one should also expect the following changes to the scale of raw scores:
a lower mean because candidates will have more difficulty earning points,
a higher standard deviation describing scores further from what will be a lower mean, and
a lower cut score than was used in the previous cycle.
The third point is complicated to predict, because a cut-score study will be done that influences where the
cut score is located. Assuming cut-score judges properly detect that items in the next form are more
difficult than the past, the cut score should wind up below previous score cutoffs. If the previous cut score
is used instead, then the pass rate will decrease. Being fully informed of these contingencies should
encourage optimal choices about whether to push for more accurate scores at the beginning of a new
testing cycle.
Bringing these points back around to an environment in which there are multiple examinations starting
new testing cycles on a staggered schedule almost every year, we will argue that each program deserves
to be independently evaluated as it enters a new cycle. Some of our programs have capacity to increase
raw score accuracy, while other programs already produce raw scores that are practically as accurate as
they can be. The larger point is that those who operate multiple programs may choose what happens to
the raw score scale when each program transitions from one testing cycle to the next. When working with
a vendor, it is reasonable to describe the preferred outcome regarding raw scale stability or ask for a
recommendation.
Summary of key points The purpose behind this white paper was to describe how the NBRC has managed multiple credentialing
programs while illustrating where NCCA Standards intersect the discussion.
The NBRC recognizes the achievement of generalists and specialists with eight credentials based on the
results of six examinations. The first of these credentials was awarded in 1961. Programs were added
through transfers from other organizations or because a specialty credentialing program was added by
the NBRC at the AARC’s request.
Respiratory care has a formalized separation of organizations that advocate for professionals, accredit
education programs, and credential individuals. Such separation makes it easier, in our opinion, to keep
the NBRC board focused on its mission and head off undue influence from those who are not purely
motivated.
The primary reason the NBRC manages multiple examinations is that multiple credentials rely on results
from two or three examinations. Some examinations document the transition from generalist to specialist.
Other examinations assess competencies in two ways: breadth of knowledge through a multiple-choice
examination and clinical simulation through patient management problems.
Running multiple credentialing programs requires effective resource management while understanding
that those resources include finances, staff, volunteers, and a vendor. The NBRC definitively makes the
16
case that it is free from unwanted outside influence by supporting itself financially without relying on
outside money.
Volunteer management is affected by the expectation that board trustees will serve on at least one
examination committee (which will be supplemented by consultants). The NBRC employs an in-house
psychometrician plus staff who assemble test forms. In-house staff process applications and operate a
call center for inquiries from candidates and credential holders. Any of these functions could be provided
by a vendor, such as the one the NBRC uses to administer and score examinations, but we have decided
that in-house staff should directly control examination assembly plus candidate and credential-holder
communication.
We have found it valuable to plan transitions from testing cycle to testing cycle across programs many
years in advance, so, in aggregate, the organization typically only manages one big project a year, with a
maximum of two projects. Doing so avoids overtaxing the time of volunteers. We have recently found it
useful to solicit and rank potential volunteers to identify the best available people and to widen the scope
of our searches for these vital volunteers.
Board trustees have valued working directly on examinations linked to the awarded credentials. We have
been able to maintain continuous representation of each credential on the board up to this point in part
because there are 31 trustees.
Acknowledging that each credentialing program and each examination is unique in many respects, the
NBRC strives to standardize those things that can be standard. Having a committee that represents each
examination is key to ongoing standardization efforts. It has helped to create and continually revise a
document that is used internally to encourage standardized presentations of examination items and
problems.
The NBRC has chosen to create equivalent test forms within testing cycles instead of transforming scores
from different forms to a second scale. Choosing this alternative has meant that a separate choice must
follow about whether to instigate change in the scale of raw scores or to maintain stability as each new
test cycle starts.
Readers who have follow-up questions about the management of administrative aspects of the NBRC’s
multiple programs should contact Lori Tinkler ([email protected]). Follow-up questions about
measurement should be directed to Robert Shaw ([email protected]).