Evaluator Training Program Introduction · Similarly, Evaluators with a hot tapping background...

1 | ©2017 Veriforce, LLC. All rights reserved.

Rev 10 – 9/19/2017

Evaluator Training Program

Introduction

This training will provide Evaluators with the knowledge to perform an evaluation in accordance with their

company’s OQ program.

The steps and processes outlined in this training program are minimum requirements that are common to

pipeline operators who have adopted this program. For specific requirements, please review your company’s OQ

plan, procedures, and practices.

The Evaluator Training Program is a five-part series. The series will cover roles and responsibilities of an Evaluator,

covered task criteria and evaluation methods, span of control, records of evaluations, and evaluator protocol on

how to conduct an evaluation. You should review all modules in the series before attempting the final exam. We

recommend that you download and print a copy of the Evaluator Training Program manual which accompanies

this course. The manual can be accessed in VeriSource and in the resources tab of this course.

Module 1

Upon completion of this module, you will be able to:

1. Describe the definition of qualified.

2. Explain the key terms that make up the definition of qualified.

3. Describe the purpose and the contents of a covered task list.

4. Explain the role and requirements of an Evaluator.

The Department of Transportation’s Operator Qualification Rule requires pipeline operators of DOT-regulated

pipelines to develop a qualification program. The intent is to ensure a qualified workforce in order to reduce the

probability and the impact of accidents/incidents caused by human error.

The Department of Transportation defines the word “Qualified” as an individual who has been successfully

evaluated, can perform assigned covered tasks, and can recognize and react to abnormal operating conditions, or

AOCs, associated with the covered task.

Through evaluations, Evaluators play a major part in determining whether an individual is qualified or not.

Therefore, it’s important for Evaluators to have a good understanding of what it actually means to be qualified.

Before going any further, let’s take a moment to discuss some of the key terms that make up the definition of

qualified.

An individual is a person who, on behalf of the Operator, performs one or more covered tasks on a pipeline facility

operated by the Operator. This includes Operator employees, contractors, sub-contractors, and other entities.

Pipeline operators must ensure that all individuals who perform covered tasks are qualified to do so, or are

working under span of control. Span of control will be discussed in more detail later in the course.


Rev 10 – 9/19/2017

An evaluation is a process established and documented by the Operator to determine an individual’s ability to

perform a covered task by any of the approved evaluation methods. To evaluate someone means to carefully

assess or judge a person’s ability against an established set of standards or criteria.

Pipeline operators must have evaluation criteria for each covered task. This criteria is then used by the Evaluator to

determine an individual’s qualified status.

A covered task is an activity, identified by the Operator, that:

1. Is performed on a pipeline facility

2. Is an operations or maintenance task

3. Is performed as a requirement of 49 CFR 192 or 195

4. Affects the operation or integrity of the pipeline

Note: If the Operator determines that an activity poses significant risk to the integrity of pipeline facilities or if

an activity is deemed a “covered task” through some other means or reference, the Operator may choose to

make such activity a covered task.

An AOC is “a condition identified by the Pipeline Operator that may indicate a malfunction of a component or

deviation from normal operations that may:

• Indicate a condition exceeding design limits or

• Result in a hazard(s) to persons, property, or the environment.”

A qualified individual must be able to recognize and react to AOCs that are associated with a covered task. The

term “recognize” refers to the ability to identify the AOC based on sight, hearing, smell, and touch. The term

“react” refers to the ability to properly respond to the AOC.

Covered Task Lists – Covered Task Identification

As required by the Operator Qualification Rule, each pipeline operator is responsible for identifying and

documenting their covered tasks. Your company has identified and documented their covered tasks on their

covered task list. The covered task list can be a helpful tool for the Evaluator in preparing for an evaluation as it

contains relevant information about each covered task.

This is an example of a covered task list. Although your company’s covered task list may be different, most

pipeline operators will share the following categories of information.

A typical covered task list will include the task ID, the task description, regulatory reference(s) for the task (if

applicable), the evaluation method for the task, whether the task can be performed by a non-qualified individual

working under the direction and observation of a qualified individual, the span of control limit, and the

requalification interval.

Please keep in mind that your company may include additional categories of information and/or requirements to

their covered task list. Therefore, your covered task list may look different than the sample covered task list

presented in this course.


Rev 10 – 9/19/2017

Your company covered task list can be found inside VeriSource under the OQ tab. VeriSource is the OQ

compliance management software that is provided by Veriforce.

Authorized Evaluators – The Role of the Evaluator

So far we’ve discussed some of the fundamental aspects of operator qualification. But perhaps the most important

aspect of the entire process is the role of the Evaluator.

An Evaluator is a subject matter expert who is responsible for evaluating whether a Candidate is qualified to

perform a covered task based on your company-specific criteria. As subject matter experts, Evaluators must be

technically competent in the tasks they will be evaluating others on. For instance, Evaluators with a technical

competence in welding would evaluate Candidates on welding covered tasks. Similarly, Evaluators with a hot

tapping background would evaluate Candidates on hot tapping covered tasks, and so on.

Another requirement for Evaluators is that they must be authorized by the pipeline operator in order to conduct

evaluations. In addition, Evaluators must also be re-authorized according to the operator’s plan. Some plans

require an annual re-authorization, while others are on a three year re-authorization schedule. Because pipeline

operators have different requirements for authorizing and re-authorizing Evaluators, it’s important to check with

your OQ Administrator for your specific requirements.

An important concept for Evaluators to understand is the difference between Authorization and Qualification.

Evaluator authorization does not automatically grant qualifications. If the Evaluator decides to pursue qualification

on a given covered task, he/she must be evaluated by another Authorized Evaluator. Some companies require a

triangle method to mitigate evaluation issues, while others require supervisors to perform the evaluation. Please

check with your OQ Administrator for specifics. In addition, some operators may require the Evaluator to be

qualified in the tasks he/she will be evaluating on in order to be authorized as an Evaluator.

So why Evaluators? There are certain reasons why pipeline operators choose to use technically competent

Evaluators in their qualification process. For one, technically competent Evaluators understand the challenges and

obstacles that may occur while the task is being performed. They are able to assess a Candidate’s knowledge,

skills, and abilities, and stop the Candidate prior to performing an incorrect act. Also, if a Candidate is found to be

unsuccessful during an evaluation, a technically competent Evaluator can explain why, whereas a non-technically

competent Evaluator cannot.

Congratulations! You have successfully completed Module 1. To review:

1. The Department of Transportation defines qualified as an individual who has been successfully evaluated,

can perform assigned covered tasks, and can recognize and react to AOCs.

2. A covered task list is a document that is used by pipeline operators to identify and document their

covered tasks. It contains relevant information about each covered task and can be a helpful tool for

Evaluators in preparing for an evaluation.

3. An Evaluator is a subject matter expert who is responsible for evaluating whether a Candidate is qualified

to perform a covered task based on operator-specific criteria.

4. An Evaluator must be technically competent and authorized by the pipeline operator in order to conduct

evaluations.


Rev 10 – 9/19/2017

Module 2


1. Identify the most common types of evaluation criteria.

2. Identify the appropriate evaluation method for each type of criteria.

Criteria & Evaluation Methods – Common Types

Criteria may include knowledge points, performance skills, abnormal operating conditions, examinations,

certifications, and any other type determined by your company.

Although the evaluation methods that are used for each will ultimately depend on your company, here are the

most common pairings of evaluation methods and types of criteria.

Knowledge Points: Evaluation Method – Oral Examination

Knowledge points test a Candidate’s knowledge of a covered task. Here is an example of a knowledge point.

Knowledge points have the letter K listed next to them. They consist of a question statement as well as the

appropriate responses. Knowledge points are evaluated through the evaluation method of oral examination. In an

oral examination, the Evaluator questions the Candidate using the knowledge points from the evaluation criteria.

The Evaluator then makes a determination by comparing the Candidate’s response to the response statements

from the evaluation criteria. The Candidate must answer the question correctly, identifying each point that is

measurable.

Note: It’s important to note that the appropriate responses may not be listed on the evaluation criteria. If so,

the Evaluator should defer to the applicable operating procedures in order to assess the Candidate’s response.

When conducting an oral examination, Evaluators should not expect each Candidate to answer the question like

the candidate before him/her or verbatim from the criteria.

In addition, Evaluators may need to “rephrase” or “reword” a question. A best practice is for Evaluators to put

question statements in layman’s terms when Candidates do not understand the question.

It is acceptable to ask additional questions for clarification. However, the questions must be related to the task

and within the standard of the criteria. Do not make the criteria harder or easier than your company’s standard.

You can ask closed or open questions, depending on the situation.

Keep in mind that each Candidate is different. Some will answer each question quickly. Others have a harder time

formulating their thoughts and will need time to think about how to answer.


Rev 10 – 9/19/2017

A final consideration when conducting an oral examination is to make sure the Candidate can hear you and that

you can hear the Candidate. This is especially important if you decide to conduct the examination outside or in a

noisy environment.

Performance Skills: Evaluation Method – Observation

Evaluation criteria can also be in the form of performance skills. Performance skills test a Candidate’s ability to

perform a covered task. Performance skills have the letter S listed next to them.

They consist of an instruction as well as the appropriate actions or steps. For example, you may see the term

“demonstrate” in the instruction. This should remind the Evaluator that he/she is dealing with a performance skill

and, therefore, the Candidate will need to “perform an action(s).”

Performance skills are evaluated through the evaluation method of observation. During an observation, the

Evaluator watches (or observes) the Candidate perform the skills portion of the task criteria. The Evaluator then

makes a determination by comparing the Candidate’s performance to the actions or steps listed on the evaluation

criteria.

Note: It’s important to note that the appropriate actions or steps may not be listed on the evaluation criteria

for certain performance skills. If so, the Evaluator should defer to the applicable operating procedures in order

to assess the Candidate’s performance.

Observations are perhaps the best way to find out if a Candidate can actually perform a covered task. But just like

in an oral examination, there are certain considerations that an Evaluator should be aware of. When conducting an

observation, Evaluators are actually performing an informed or educated observation. This means that the

Evaluator already knows how the task is supposed to be accomplished. Sometimes there may be more than one

way a task can be performed. However, some tasks require the Candidate to follow a strict step-by-step

procedure. In those instances, there cannot be any deviation from the task steps.

It’s ok to ask the Candidate questions about their performance (either during or after). Sometimes you can obtain

more information if you can get the Candidate to think out loud. A best practice is for the Evaluator to use neutral

prompts. For example:

• What do you see?

• What are you thinking?

• What are you doing?

• What are you looking for?

• What should that be?

Observations may be conducted while someone is actually performing the skill on a regulated pipeline facility (if

the settings are practical). However, as an Evaluator, you should make sure the following considerations are taken

into account.

• Span of control requirements must be addressed, meaning a qualified person may need to be present in

order to direct and observe the Candidate if the task allows it.

• If the Candidate is not successful, the work will have to be verified/reaccomplished by a qualified person.


Rev 10 – 9/19/2017

• Also, coordination with operations management and Gas Control should be made prior to conducting any

activity.

If the observation cannot be performed on a regulated pipeline facility or if it’s not feasible to do so, then a

simulation can be used. A simulation is the demonstration of a performance skill for the purposes of an

evaluation through the use of a mock setup, rather than a regulated pipeline facility. Simulations should

sufficiently reflect the actual work setting in order to reflect work performance. During a simulation, the candidate

must perform some type of hands-on activity that imitates the actual performance skill to be measured. The

simulation cannot be limited to a candidate talking through the task, using hand gestures, imagining there are

tools and/or equipment present, or describing the activities of other individuals performing the task.

Let’s consider a couple of examples of simulations.

For example, Common Covered Task 008OP Measurement of Wall Thickness with Ultrasonic Device – to conduct a

simulation of this task you could use a scrap piece of pipe, take an actual measurement, and ensure that all steps

are performed as they would be in the actual task performance.

Another example to consider is Common Covered Task 426OP Inspect Pipe Coating with Holiday Detector – to

conduct a simulation of this task you would use a scrap piece of pipe with a working holiday detector in order to

reflect and imitate actual task performance.

A final example is Common Covered Task 412OP Install CP Leads on Pipeline Using Exothermic Welding – during

this simulation you would use a scrap piece of pipe and ensure all materials and equipment are used as they

would be in the actual task performance, performing each step until completion.

Remember, the keys to a proper simulation are to ensure that the simulation reflects the actual work setting and

task performance and to require the Candidate to perform some type of hands-on activity that imitates the actual

skill to be measured.

Should you have further questions about proper simulations, you can access the Veriforce Simulation Guidance

document which can be found in VeriSource under the Resources Tab. In addition, you can also contact the

Veriforce Customer Service group and ask to speak to a Compliance Auditor for further guidance.

AOCs: Evaluation Method – Oral Examination

Evaluation criteria also includes abnormal operating conditions. This criterion tests a Candidate’s ability to

recognize and react to abnormal operating conditions. The criteria consists of a description of the abnormal

operating condition and the proper responses or actions to take in the event the AOC is encountered.

Due to the nature and danger of abnormal operating conditions, an observation or even a simulation is not

practical or recommended to evaluate AOCs. For this reason, AOCs are evaluated through oral examination. When

evaluating a Candidate on AOCs, the Evaluator has the flexibility to address the AOCs in any appropriate manner,

as long as the Candidate has the opportunity to explain how to recognize and react to each AOC.


Rev 10 – 9/19/2017

One technique is for the Evaluator to describe a hypothetical situation using the examples for the AOC as listed in

the evaluation criteria. The Evaluator then questions the Candidate on the recognition of and the proper

responses or reactions to the AOC. Finally, the Evaluator then makes a determination by comparing the

Candidate’s responses to the evaluation criteria.

Let’s consider an example of this technique in action using the following criteria.

AOC: Unintentional release, vapors, or hazardous atmosphere.

Recognition: Examples include, but are not limited to, blowing gas, bubbles, or dead vegetation.

Response/Reaction: Eliminate potential ignition sources; move to a safe location; notify emergency

personnel; limit access to the location; and follow appropriate procedures.

Given this criteria, the Evaluator may use the following hypothetical situation:

Evaluator – “Let’s say that you were operating this valve, and noticed blowing gas coming from the valve. What

could this indicate?”

Candidate – “An unintentional release or a leak.”

Here the Candidate has recognized the type of AOC based on the hypothetical situation provided by the

Evaluator. But the Candidate must also be able to properly react to an AOC. The Evaluator will need to follow up

with an additional question that tests the Candidate on the react/response portion.

Evaluator – “If it were an unintentional release, what would you do?”

Candidate – “I would eliminate potential ignition sources; move to a safe location; notify emergency personnel, as

appropriate; limit access to the location; and follow the appropriate procedures for notification, documentation,

and remedial action.”

Here the Candidate has described the proper reactions he/she would take to address the AOC. In this example,

the Candidate has now recognized the AOC and has described how he/she would react to it.

Another technique involves the Evaluator asking the Candidate how a particular AOC can be recognized and

asking how they would respond. Using the same sample criteria from the previous example, this is how this would

unfold.

Evaluator – “How could you tell if an unintentional release or leak was happening?”

Candidate – “If there’s evidence of blowing gas, puddles, or dead vegetation.”

Evaluator – “How would you respond?”


Rev 10 – 9/19/2017

Candidate – “I would eliminate potential ignition sources; move to a safe location; notify emergency personnel, as

appropriate; limit access to the location; and follow the appropriate procedures for notification, documentation,

and remedial action.”

Regardless of what technique is used, Evaluators must ensure the Candidate possesses the knowledge to

recognize and react to the AOCs listed on the evaluation criteria.

Examinations and Certifications: Evaluation Method – Exams and Review

Criteria can also be in the form of examinations. Examinations test a Candidate’s knowledge of a covered task. The

Candidate is either given a paper test or is required to take the exam online, either in VeriSource, your company’s

intranet, or in an another location as specified by your company. Exams are typically completed in conjunction

with computer-based training, where the Candidate must take an e-Learning module and complete the required

exam, although this isn’t always the case. Check with your OQ Administrator for specifics.

Criteria can also be in the form of certifications. A Candidate may be required to provide current certification from

industry organizations, manufacturers, or vendors. Such requirements will be listed in the evaluation criteria of the

covered task (if applicable). An Evaluator should review the Certificate to ensure it is current and that it is the type

required by the covered task.


1. The most common types of evaluation criteria are knowledge points, performance skills, AOCs,

examinations, and certifications.

2. An oral examination is used for knowledge points

3. An observation (which can include simulation) is used for performance skills.

4. Simulations should reflect actual work setting; the Candidate must perform a hands-on activity; and the

simulation cannot be limited to talking, imagining, or describing activities.

5. AOCs are to be evaluated through oral examination.

6. Examinations are designed to test the candidate’s knowledge of the covered task.

Module 3


1. Define span of control.

2. Describe how to comply with span of control requirements during an evaluation.

Span of Control: Requirements

The OQ Rule allows nonqualified individuals to perform certain covered tasks on regulated pipeline facilities if

directed and observed by a qualified individual. Span of control is the maximum number of nonqualified

individuals that a Qualified Individual can direct and observe for the conditions under which the task is being

performed.


Rev 10 – 9/19/2017

While directing nonqualified individuals, the qualified individual must be able to take control of the task; be in

communication with the nonqualified individual, have close supervision over the nonqualified individuals at all

times, and respond appropriately to abnormal operating conditions should they arise.

If the qualified person walks away or for some reason is no longer in control of the task, the nonqualified

individual(s) must stop performing the task.

The span of control limit of a covered task, which is determined by your company, outlines the maximum number

of nonqualified individuals that a qualified person can direct and observe for the conditions under which the task

is being performed. Let’s look at a couple of examples of span of control limits.

Span of Control – 1:3

This ratio indicates that one qualified individual may direct and observe no more than three nonqualified

individuals during the performance of the covered task.


A ratio of 1:1 indicates that one qualified individual may direct and observe no more than one nonqualified

individual during the performance of the covered task.


A ratio of 1:0 indicates that only qualified personnel may perform the task and that performance of the task by a

nonqualified person is not allowed.

Span of control limits can be found on the covered task list and on the Record of Evaluation. Records of

Evaluation (ROEs) will be discussed in the next module.

Span of Control and Evaluations: Evaluator Options

Span of control (SOC) limits must be enforced anytime an individual performs a covered task on a regulated

pipeline facility whether it be during an evaluation, training, or actual task performance.

Therefore, in cases where actual task performance on a regulated pipeline facility is used as a means to evaluate

the Candidate, the Evaluator must ensure that appropriate span of control limits are followed. There are certain

scenarios that an Evaluator must be able to recognize and address in order to adhere to span of control

requirements. Let’s take a look at each scenario to get a good understanding.

Scenario #1 – The Evaluator is qualified and the Candidate is not qualified.

In this scenario, the Evaluator is qualified on a covered task while conducting an evaluation on a regulated

pipeline facility, so the nonqualified Candidate can perform the task under the Evaluator’s SOC.

Scenario #2 – The Evaluator is not qualified, and the Candidate is not qualified, but a qualified individual is

available to observe.


Rev 10 – 9/19/2017

In this scenario, the Evaluator could bring in a qualified person to oversee the nonqualified individual’s task

performance on a regulated pipeline facility. However, if the qualified person has to direct or stop the

nonqualified individual’s performance, the evaluation would be considered unsuccessful.

Scenario #3 – The Evaluator is not qualified and the Candidate is qualified.

This scenario can occur if the Candidate is qualified (such as when he’s up for requalification), the evaluation could

take place on the regulated pipeline facility, regardless of the Evaluator’s qualification status or span of control

limit.

Scenario #4 – The Evaluator is not qualified and the Candidate is not qualified.

If neither the Evaluator nor Candidate is qualified and there is no other qualified person around to observe, the

evaluation must be simulated.

Scenario #5 – The Evaluator is not qualified, the Candidate is not qualified, and the span of control for the task is

1:0.

For the last scenario, the evaluation must also be simulated if the span of control for the task is 1:0 and both the

Evaluator and Candidate are nonqualified. Even if there were qualified individuals available and willing to direct

and observe, the evaluation must be simulated since the span of control limit of 1:0 only allows qualified

personnel to perform the task.


1. The span of control limit of a covered task, which is determined by your company (the operator), outlines

the maximum number of nonqualified individuals that a qualified person can direct and observe for the

conditions under which the task is being performed. An oral examination is used for knowledge points.

2. Span of control (SOC) limits must be enforced anytime an individual performs a covered task on a

regulated pipeline facility, whether it be during an evaluation, training, or actual task performance.

Therefore, in cases where actual task performance on a regulated pipeline facility is used as a means to

evaluate the Candidate, the Evaluator must ensure that appropriate span of control limits are followed in

order to be in compliance with DOT regulations. Simulations should reflect actual work setting; the

Candidate must perform a hands-on activity; and the simulation cannot be limited to talking, imagining,

or describing activities.

Module 4


1. Describe the use of the record of evaluation (ROE) form.

2. Identify each section of an ROE.

3. Describe how to submit a qualification within VeriSource.

4. Describe how to submit documentation to Veriforce.

Record of Evaluation: Qualification Compliance Document


Rev 10 – 9/19/2017

Evaluators must use a record of evaluation (ROE) each time an evaluation is conducted. The ROE contains the

evaluation criteria and serves as the qualification compliance document.

ROEs are the first option under the Internal Evaluators Tab in VeriSource.

Once an Evaluator is authorized, he/she will have access to the ROEs they have been authorized on.

Prior to each evaluation, the Evaluator will need to print the ROE and take it with him/her to the evaluation.

Record of Evaluation: ROE Form

This section will introduce a typical ROE in sections.

The very top of an ROE begins with the record of evaluation instructions. This section outlines reminders on how

to conduct an evaluation and how to fill out the ROE. Make sure to fill out the ROE completely, but do not post

comments or write on the first page of the ROE other than where it is required. Never use white-out. If you make a

mistake, cross out the mistake and then initial the change.

The training requirements section requires the Evaluator to ask the Candidate each question that is listed.

• Is the Candidate currently disqualified due to his/her role in an incident/accident or for cause?

• Was the Candidate unsuccessful when last evaluated for this covered task?

• Has a High Impact change taken place related to this covered task since the Candidate was last

evaluated/qualified?

If a change that impacts evaluation criteria, task descriptions, or program requirements occurs, Veriforce and the

applicable pipeline operator(s) will categorize the change(s) as either low, medium, or high impact.

• A Low Impact change may include grammatical, formatting, or other modifications that result in no

material effect on the administration and implementation of the OQ Program.

• A Medium Impact change may include revisions to administrative procedures, evaluation methods,

operating procedures, or other items that affect the implementation and administration of the OQ

program.

• High Impact changes are those that require the requalification of individuals previously qualified on the

applicable task(s). A Candidate whose qualification is revoked as a result of a “High Impact” change is

required to complete training before being re-evaluated.

Note: Veriforce will communicate medium and high impact changes through any or all of the following: by

posting it in VeriSource or by broadcast email, fax, or US Postal Service (at pipeline Operator’s discretion).

So back to the ROE, if the answer is YES to any of the questions, the Candidate must have received training prior

to the evaluation. The Evaluator will need documentation of the training before the evaluation can begin.

(Documentation of training requirements will be discussed later on in the training.) If all of the answers are NO,

there is no need to submit training documentation.


Rev 10 – 9/19/2017

Note: Your company may have additional training requirement questions listed under this section. Check with

your OQ Administrator for specific requirements.

The employer information section requires the Candidate’s company name. The Candidate information section

requires the Candidate’s first, middle, and last name, and either their date of birth or their Veriforce ID. If they

don’t know their Veriforce ID number, you can fill it in when you submit the qualification data. (Qualification data

is submitted upon the conclusion of the evaluation.) It is extremely important for the Candidate’s information to

be documented exactly as it appears on the Candidate’s government-issued photo I.D. Furthermore, the name

and ID or date of birth must be legible. This information is often prepopulated by the system. However, if

Evaluators need to print this information, it must be legible.

The next section identifies the covered task number and description. This is also prepopulated by the system. The

supporting documentation area will indicate if a certificate or other documentation is required in addition to the

evaluation (e.g., Clock Spring certificate). Under the Evaluator information section, the Evaluator’s name will be

listed. Do not use an ROE with another Evaluator’s name on it. The only way to obtain an ROE with someone else’s

name on it is to log into VeriSource using their user name and password. It is not permissible to cross out the

other Evaluator’s name and insert your own. Evaluators must use their own VeriSource user name and password to

obtain the ROEs they are authorized on.

The results section requires the Evaluator to document the outcome of the evaluation. Check either Successful or

Unsuccessful.

Under Candidate and Evaluator acknowledgement, the Candidate and Evaluator must sign and date the form. The

signatures must be present and dates must be legible.

• Only the Candidate may sign for the Candidate. Only the Evaluator may sign for the Evaluator.

• The dates indicate the date the evaluation criteria was met and the evaluation was completed. This is not

necessarily the submittal date. The Candidate and Evaluator dates must be the same.

The disclaimer and no warranty section explains that the ROE alone does not constitute a qualification – only

documentation that the evaluation occurred. Qualifications are only issued after the ROE and data are submitted

to Veriforce and verified through the quality assurance process. It also explains that the qualification is not valid

for non-Veriforce operators and represents a breach of contract if used for that purpose.

The very bottom of page 1 displays the revision and date of the ROE. The most current revisions are kept in

VeriSource. Be sure to use the most current revision while performing an evaluation. If you submit an outdated

ROE, it will be rejected.

The top of page 2 contains the evaluation method, regulatory reference (if applicable), subsequent qualification

interval, and span of control.

The next area outlines the criteria used during the evaluation and the abnormal operating conditions associated

with the covered task.


Rev 10 – 9/19/2017

Submitting Qualification Data: VeriSource

Once the evaluation has concluded, the Evaluator must submit the evaluation data in VeriSource. Depending on

your company, you may or may not have to fax or email the first page of the ROE to Veriforce. Some operators

require the ROE to be submitted to their OQ admin for review and submittal. If you are required to submit the

ROE, make sure it is filled out completely and that it is legible.

Note: The steps outlined in this section may be slightly different than your company’s specific process. For

detailed instructions on submitting qualification data, please contact your OQ Administrator.

To submit an ROE:

• In VeriSource, under the Internal Evaluators tab, click Online ROE.

• Next, click on I want to Submit an Evaluation.

• Next, select the Candidate’s name from the drop down list.

• Select the covered task you evaluated the Candidate on. Multiple tasks can be selected by holding the

down arrow and CTRL key.

• Review your contact information.

• Select the date the evaluation was conducted.

• Adjust whether the Candidate was successful or unsuccessful as well as the answers to the training

requirement questions.

• After reviewing the information and making the appropriate selections, submit the evaluation result.

Submitting Documentation: QSF, ROE, and Supporting Documents

This section only applies if you are required by your company to submit the ROE and supporting documentation

to Veriforce. If you are, before faxing or emailing page one of the ROE, you will need to print out the Qualification

Submittal Form (QSF).

When submitting the paperwork, submit the documents in this order:

1. Qualification Submittal Form

2. Page 1 of the ROE

3. Any other supporting documentation (e.g., training record)

Send these documents to 1.866.447.9104 if faxing, or to [email protected] if scanning and emailing.

Before submitting the ROE to Veriforce, double check that all of the required information is filled out on the ROE,

make sure it is legible, and verify the evaluation dates are the same.

If anything is left blank or illegible, the ROE will be rejected. If an ROE submission is rejected, the correction

should be sent to [email protected] with “roefix” in the subject line of the email.


1. The ROE contains the evaluation criteria and serves as the qualification compliance document. Evaluators

must use a record of evaluation (ROE) each time an evaluation is conducted.

mailto:[email protected]


Rev 10 – 9/19/2017

2. Before submitting the ROE to Veriforce, double check that all of the required information is filled out on

the ROE, make sure it is legible, and verify the evaluation dates are the same. If anything is left blank or

illegible, the ROE will be rejected.

3. The evaluation documentation should be submitted in this order: 1-Qualification Submittal Form; 2-Page

1 of the ROE; 3-Any other supporting documentation (e.g., training record).

Module 5


1. Identify Evaluator responsibilities prior to an evaluation.

2. Describe how to brief a Candidate.

3. Describe the rules for conducting an evaluation.

4. Describe additional requirements for conducting an evaluation.

5. Describe how to debrief a Candidate.

6. Explain the training requirement after an unsuccessful evaluation.

Preparing for an Evaluation: Pre-Evaluation Activities

Evaluators should make sure that everything needed for the evaluation is available and present. You should

prepare for an evaluation by reviewing the materials that will be used during the session. These include the:

• Evaluation criteria

• Equipment and facilities

• Personnel

• Special needs

Let’s discuss each of these in more detail.

The first item on the list is the evaluation criteria. Each Candidate should review the evaluation criteria prior to the

evaluation. This allows the Candidate time to practice or ask questions. The Evaluator should confirm that the

Candidate has reviewed the evaluation criteria and/or provide it if necessary prior to the evaluation. Under no

circumstances should the Candidate be allowed to use the criteria during the evaluation.

The evaluation criteria can be accessed in two different locations within VeriSource. The first location is under the

OQ Tab. The other location is in the Internal Evaluators Tab under Online ROE. When an ROE is downloaded, the

evaluation criteria will automatically download with it.

As an alternative to the evaluation criteria, a Candidate can review a training guide. Training guides provide an in-

depth explanation of each item of the evaluation criteria.

Note: Please check with your OQ Administrator to determine if your company has training guides available for

their covered tasks.

In addition to the criteria, the Evaluator should also ensure that the necessary equipment and facilities needed to

conduct the evaluation are available. It is your responsibility to make sure the Candidate has access to everything


Rev 10 – 9/19/2017

needed to perform the task. A best practice is for the Evaluator to make every effort to provide the tools and

equipment that are familiar to the Candidate during the evaluation. Regarding the setting, it is highly

recommended to find a location for the evaluation away from distractions. Candidates will feel more comfortable

and perform to their fullest potential if they feel at ease with their equipment and surroundings.

Personnel is another consideration that an Evaluator must take into account when preparing for an evaluation. As

discussed in a previous module, if the task is to be performed on a regulated pipeline facility for the purposes of

an evaluation, the Evaluator must make sure to comply with the applicable span of control requirements. The

Evaluator may be required to have a qualified person present during the evaluation, depending on the task and

the qualification status of the Evaluator and the Candidate.

If the Candidate requires special needs to perform the task, you should seek guidance from your OQ admin

and/or refer to your company’s policies and procedures.

Briefing the Candidate: Starting the Session

Briefing the Candidate is extremely important for conducting a proper evaluation. It sets the stage for the

evaluation, allows the Evaluator to collect required information, and outlines Candidate and Evaluator

expectations. Briefing a Candidate consists of the following steps:

1. Introduce yourself

2. Break the ice

3. Explain the purpose

4. Provide an overview

5. Confirm identity

6. Determine training requirements

7. Verify supporting documentation

8. Explain the rules

As the Evaluator, you should greet the Candidate with a handshake when the evaluation begins. A handshake,

even to someone you know well, sets the stage for the beginning of the evaluation. The Evaluator should also

introduce him or herself.

Since most Candidates will be nervous, a best practice is for the Evaluator to take a few seconds to calm the

Candidate down by talking about something that interests the Candidate; for example, hunting, fishing, sports,

etc.

To help with Candidate’s anxiety, it’s also a best practice for the Evaluator to explain the purpose of the evaluation.

Often Candidates don’t know why they’re being evaluated and why they are being tested. Explaining the purpose

to the Candidate can help him/her obtain the big picture and can help with any anxiety or nervousness he/she

may have.

Another best practice is to explain how you, the Evaluator, are going to conduct the evaluation; for example, if it’s

a launching and receiving task, you might tell the Candidate that you are going to ask him/her a series of


Rev 10 – 9/19/2017

questions then go out in the field to watch him/her perform a launching and receiving operation, followed by

questions on abnormal operating conditions.

Once all of the preliminary and background information has been covered, Evaluators should proceed and ask the

Candidate for his/her government-issued photo I.D. This will be used to document the Candidate’s name and date

of birth on the ROE. As mentioned earlier, the Candidate’s name and date of birth should be documented as it

appears on their government-issued photo I.D. The Candidate’s company name should also be documented.

Note: Your operator may have different requirements or methods for confirming the Candidate’s identify (e.g.

the use of a company ID or prepopulated ROE or visual identification of an individual you know). Check with

your OQ Administrator for specifics.

The next step is to determine if training is required prior to the evaluation. To do this, you will need to ask the

Candidate the Training Requirement questions from the ROE. If any of the questions are answered YES, training

must have occurred prior to the evaluation and the Candidate must show proof by providing you with training

documentation.

Note: For some companies, Candidates must complete required training before the evaluation. Otherwise, the

Evaluator will not be able to download the ROE. In this type of system configuration, Candidates will not be

able to provide training documentation since the system tracks this automatically. Check with your OQ Admin

or company procedures to verify training acceptance and additional requirements.

You will also need to verify if the covered task requires any supporting documentation, such as a manufacturer’s

certificate, visual acuity test, etc. To verify, look at the Supporting Documentation field on the ROE. If supporting

documentation is required, the Candidate should provide the necessary documents.

The last step in briefing the Candidate is to explain the rules of the evaluation. Inform the Candidate that:

• There is no time limit – Candidates are allowed as much time as they need to perform the task. However,

it is the Evaluator’s call if they are taking too long.

• You cannot assist with any aspect of the covered task – Evaluators must maintain the integrity of the

evaluation process and therefore should not assist or help the Candidate.

• You will stop the evaluation for any safety violation (such as danger to personnel or equipment) and deem

the evaluation unsuccessful – Candidates must observe all safety rules and must be able to manipulate

equipment adequately.

• You will review his/her performance at the conclusion of the evaluation session. Evaluators should review

the Candidate’s performance at the end of the evaluation so that the Candidate is aware of the outcome.

To finish briefing the Candidate, ask if he/she has any questions before beginning the evaluation.

Conducting the Evaluation: Evaluator Protocol

There are five important rules that an Evaluator must follow during an evaluation.

1. Evaluate the Candidate individually.

2. Ensure the evaluation is done in a one-on-one setting.


Rev 10 – 9/19/2017

3. Follow the evaluation method as required by the covered task.

4. Adhere to the evaluation criteria of the covered task.

5. Do not train the Candidate during the evaluation.

It is extremely important for you, as the Evaluator, to follow these rules in order to maintain the integrity of the

evaluation process. Let’s discuss each of these rules in more detail.

The first rule requires each Candidate to be evaluated individually. Candidates cannot be evaluated two at a time,

in a group, or over the phone.

The second rule is that each evaluation must be conducted in a one-on-one setting. One-on-one means a

conversation or interaction between two people where no one else is present. One-on-one settings may include

areas such as in an office, vehicle, shop, field, or on the right-of-way.

Sometimes additional individuals may have to be present, such as when a qualified person needs to be present to

comply with span of control limits, when the operator requires internal auditing supervisors to be present, or

when state and federal auditors wish to see an evaluation process.

Regardless of the reason, if others have to be present, they need to meet the following requirements:

• They cannot be future Candidates – future Candidates will mimic what they just heard or saw.

• They cannot make your Candidate feel uncomfortable – no one likes to be tested and having other people

around may make the Candidate uneasy.

• They cannot distract you or the Candidate.

The third rule is that the Evaluator must follow the evaluation method required by the covered task. The required

evaluation method will be listed on the evaluation criteria of the covered task and on your company’s covered

task list. An important thing to note is that if a covered task requires more than one method, both methods must

be followed. For example, for common covered task 007OP operate valves, the evaluation method states

observation and oral exam. In this example, the Evaluator would have to orally examine the Candidate on the

knowledge points and AOCs AND observe the Candidate perform the performance skills of the covered task.

The fourth rule involves the evaluation criteria. This rule requires the Evaluator to adhere to the evaluation criteria

of the covered task. The criteria may not be altered or changed. Your company has established the criteria as task

standards. If for some reason you find that the criteria does not meet your company-specific requirements, get

with your company program administrator so a review can be conducted. Under no circumstances may the

Evaluator deviate from the criteria. Evaluators cannot add to the criteria, are not allowed to make the criteria

harder than what it is, and cannot require the Candidate to know or perform more than what is required by your

company. On the other hand, Evaluators cannot remove from the criteria. Candidates are required to answer each

knowledge point, perform each performance skill, and recognize and react to each AOC correctly.

The last rule prohibits Evaluators from training a Candidate during an evaluation. Training includes any type of

assistance with the task. Evaluators must not suggest a task performance technique or hand out or point out the

location of tools or equipment.


Rev 10 – 9/19/2017

There will be occasions when a Candidate asks a question and the Evaluator must “relay” the question back to the

Candidate because answering would be considered training. Some examples of how to “relay” these types of

questions back to the Candidate may include:

• Candidate: “Do you think I should clean this pipe better before fusing it?”

• Evaluator: “How would we know if the pipe was cleaned sufficiently?”

• Candidate: “What does this symbol mean?”

• Evaluator: “What do you think it means?”

• Candidate: “Should I check the pressure now?

• Evaluator: “How do we know when the pressure should be checked?”

Be careful not to ask leading questions. A leading question gives the Candidate a partial answer, which he/she is

expected to complete, or gives a complete answer and expects the Candidate to agree or disagree with it. Leading

questions are those that nudge, push, or shove the other person towards a particular answer and away from other

answers. Leading questions are another form of training. For example:

• Evaluator: “Are you going to refer to the procedures before starting the equipment?”

• Evaluator: “Are you going to lockout the machine before opening it?”

• Evaluator: “You are going to don your hardhat and glasses before performing that task – right?”

Additional Requirements: Evaluator Protocol

Following the rules mentioned in the previous section is essential in order to maintain the integrity of the

evaluation process. However, an Evaluator can improve the quality of his evaluations and to some extent can put

the Candidate at ease by adhering to the following items.

Perhaps the number one distraction is the use of cell phones. You should turn off your cell phone and give your

full attention to what the Candidate is saying or doing. Try not to be distracted by your surroundings or your own

thoughts. Candidates will sense that their performance is not important if they believe you are distracted. The

worst thing an Evaluator can do is to undermine the importance of the evaluation.

Pay close attention to the Candidate’s responses or performance. Make sure you don’t try to anticipate what they

are going to say or do, and don’t let your mind wander. Always avoid being in a position where you’re not sure

whether to deem the Candidate successful or unsuccessful because you weren’t paying attention. This could end

up being unfair to the Candidate, or worse, you could end up qualifying someone that really shouldn’t be.

Another item to adhere to is to remain as neutral as possible. If the Candidate picks up in your body language,

facial expressions, or non-verbal sounds (such as sighing) that he/she answered a question or performed a step

incorrectly, it’s possible that they may be unable to continue.

Evaluators should be aware of their surroundings at all times. You should immediately suspend the operation of

equipment during unanticipated or abnormal events, during accident conditions, or whenever you feel it is

necessary to ensure safe and reliable operation.


Rev 10 – 9/19/2017

Debriefing the Candidate: Concluding the Session

To be successful, the Candidate must have met 100% of the criteria during the evaluation. Once the evaluation is

over, you must review the Candidate's performance. Let him or her know if they were successful or not.

If the Candidate was successful:

• Review the Candidate’s performance with the Candidate.

• Inform the Candidate of the outcome.

• Congratulate the Candidate.

• Complete the ROE – mark the Candidate successful – both the Evaluator and the Candidate must also sign

and date the ROE.

• Inform the Candidate of the span of control limits of the covered task.

• Conclude the session.

• Submit the evaluation data in VeriSource.

• Fax or email the evaluation documentation to Veriforce, as applicable.

If the Candidate was unsuccessful:

• Review the Candidate’s performance with the Candidate

• Inform the Candidate of the outcome

• Explain that in order to be re-evaluated, he/she must fulfill the training requirement.

• Complete the ROE – mark the Candidate unsuccessful – both the Evaluator and the Candidate must also

sign and date the ROE

• Inform the Candidate of the span of control limits of the covered task. If the task’s span of control allows

it, let the Candidate know that he/she is only allowed to perform the covered task if directed and

observed by a qualified person.

• Conclude the session.

• Submit the evaluation data in VeriSource – It’s important to submit unsuccessful evaluations and

documentation. Just because the result of an evaluation is unsuccessful doesn’t mean that Evaluators

don’t have to submit the data and necessary paperwork. Submitting unsuccessful evaluations can help

validate the effectiveness of the Operator’s qualification program.

• Fax or email the evaluation documentation to Veriforce, as applicable.

Notice that the steps for debriefing a successful and unsuccessful Candidate are similar with the exception of the

training requirement. We’ll discuss the training requirement in more detail in the next section.

Training Requirement: Failing Candidates

After an unsuccessful evaluation, Candidates are required to complete training related to the task before they can

be evaluated again.

Note: Some operators may have time restrictions before a Candidate can be re-evaluated or possibly a limit to

unsuccessful evaluations. Check with your OQ Admin or company procedures for specific requirements

regarding the re-evaluation of unsuccessful Candidates.


Rev 10 – 9/19/2017

As far as the training requirement, the training may be in the classroom, online, or on-the-job (OJT). The training

must be specific to the task the Candidate failed and the training must be documented.

In order for training documentation to be acceptable, it must clearly identify the:

• Name or title of the training program

• Source of training

• Candidate’s name

• Date of completion

Examples of training documentation can consist of sign-in sheets, certificate of completion, attendance sheet, or a

valid record of training (ROT).

An ROT can be used to document on-the-job training. ROTs are available in VeriSource for each covered task. If

the operator allows it, the Evaluator is authorized to conduct training. The Evaluator can train a Candidate before

or after, but NOT during, an evaluation. If you do conduct training as an Evaluator, do not count a good training

or practice session as an evaluation. An evaluation must be a planned, separate event.

During an evaluation, if the Candidate stumbles through the task and begins to fail (non-safety related), the

Evaluator can choose to stop the evaluation or allow the Candidate to finish. Either option is acceptable.

Allowing the Candidate to finish will provide a complete picture of what the Candidate knows and doesn’t know.

This is helpful information with regard to the training that is required after an unsuccessful evaluation. However,

stopping the evaluation may be the best option if it becomes obvious that the Candidate needs more training or

experience. Stopping an evaluation should be handled delicately. Try not to embarrass or demean the Candidate.

Note: Check with your OQ Administrator for more information on Evaluators and their role in the training of

personnel.

As a final point, failing someone is generally difficult to do. It seems to be in our nature to help others when they

are in trouble, or make excuses for their poor performance, especially if it’s a fellow employee that you have

worked with for years or even a family member. However, if someone is failing, do not assist or pass them thinking

they are just having a bad day or that they will improve their skills with time. Doing this violates regulatory

requirements and jeopardizes the integrity of the pipeline and the safety of that individual, everyone around, and

the general public.


1. The Evaluator must ensure that everything needed for the evaluation is available.

2. 1-Introduce yourself; 2-Break the ice; 3-Explain the purpose; 4-Provide an overview, 5-Confirm the identity

of the Candidate, 6-Determine training requirements, 7-Verify supporting documentation, and 8-Explain

the rules.

3. 1-Evaluate the Candidate individually, 2-Perform the evaluation in a one-on-one setting, 3-Follow the

evaluation method, 4-Adhere to the evaluation criteria, and 5-Do not train during the evaluation.

4. Avoid distractions, pay attention to what the Candidate is saying or doing, remain neutral, and be safe.


Rev 10 – 9/19/2017

5. 1-Review performance, 2-Inform the outcome, 3-Congratulate Candidate if successful OR inform of

training requirement if unsuccessful, 4-Complete the ROE, 5-Inform Candidate of SOC, 6-Submit

evaluation data, 7-Submit evaluation documentation, and 8-Conclude evaluation.

6. Complete training related to the covered task.

This concludes the Operator Evaluator Training Program. Please review all modules in the series before attempting

the final exam.

If you have any questions concerning this module, please don’t hesitate to contact Veriforce at 800-426-1604 or

email us at [email protected].

Evaluator Training Program Introduction · Similarly, Evaluators with a hot tapping background...

Documents

Transcript of Evaluator Training Program Introduction · Similarly, Evaluators with a hot tapping background...