Because organizational environments are constantly

demanding that employees learn continuously, it is crucial that

training programs be effective in reaching every individual.

According to Kirkpatrick (1976), there are four levels of

evaluation training: reaction, learning, the performance in the

job and results. Trainee reaction to training is a very relevant

variable, usually associated with high levels of impact of

training at work. Despite such importance, valid and reliable

trainee reaction measures are still rare (Abbad, Gama &Borges-

Andrade, 2000).

According to Borges-Andrade (2002), studying reaction

usually involves the use of a questionnaire with a Likert-like

scale and at least one open question, which demands content

analysis. Usually, qualitative and quantitative analyses present

high degrees of correspondence.

This study presents two experiences in examining trainee

reaction to distance training programs. The first one consists of

the validation process of a questionnaire that measures trainee

reaction to the instructional procedures used in two distance

education courses. The second one describes the use of

qualitative methods in the formative evaluation of a web-based

training program.

Evaluating reaction to distance training programs: combiningEvaluating reaction to distance training programs: combiningquantitative and qualitative methodsquantitative and qualitative methods

ABSTRACTABSTRACT

METHODMETHOD

This study is an attempt to trigger a discussion about the

combined use of both quantitative and qualitative methods to

evaluate trainee reaction to training programs. Recent

qualitative research conducted in Brazil resulted in a valid and

consistent questionnaire containing 12 items that evaluated

training reactions to traditional procedures and procedures

based on new information technology. Research on trainee

reaction has also been done with qualitative methods, such as

the content analysis of learners’ answers to open questions

related to strategies, content, layout and other aspects of web-

based courses. Once both methods provide researchers with

precious, useful information, the combined use of them is

suggested.

As Borges-Andrade (2002) points out, the process of creating an instrument for collecting

quantitative data may also include qualitative tools (interviews and open-questioned surveys).

Therefore, it is suggested that qualitative and quantitative methods are used together to evaluate

reaction.

Study 2, in particular, exemplify the qualitative approach in the formative evaluation of a web-

based training program. Its results present agreement with previous findings from technical

evaluation (See the poster Formative evaluation: using diverse tools in order to refine and improve

instructional procedures in distance training). Such practice has shown great value, once the

qualitative methods have added specific information that clarifies data collected with quantitative

tools.

Combining both qualitative and quantitative methods is suggested for further research with web-

based training, once it allows a more holistic treatment of the multiple variables involved in the

continuous learning process that should take place in a sustainable organization of work.

DISCUSSION AND RECOMMENDATIONSDISCUSSION AND RECOMMENDATIONS

Abbad, G., Gama, A. L. G. & Borges-Andrade, J. E. (2000). Treinamento: Análise do relacionamento da

avaliação nos níveis de reação, aprendizagem e impacto no trabalho. Revista de Administração

Contemporânea, 4, 25-45.

Borges-Andrade, J.E. (2002). Desenvolvimento de Medidas em Avaliação de Treinamento. Estudos de

Psicologia, 7 (Número Especial), 31-43.

Kirkpatrick, D. L. (1976). Evaluation of Training. Em R. L. Craig (Org),Training and Development

Handbook (pp. 18.1-18.27). New York,NY: Mc Graw-Hill.

REFERENCESREFERENCES

RESULTS RESULTS

University of Brasília - Brazil

Institute of Psychology

Impacto: Research on Training and Organizations of

Work

Authors:

Lidia Parachin

André Wogel

Gardênia Abbad

Maria Emília Araújo

Talita Custódio

Karen da Matta

INTRODUCTIONINTRODUCTION

PRONEXPRONEX Fubra

F1 F2Sequence of content presentation.

0.82 0.61

Content compatibility with professional needs.

0.75 0.43

Language used in the didactic material.

0.72 0.54

Quantity of available didactic material.

0.70 0.41

Quality of available didactic material.

0.66 0.51

Period of time planned to the activities.

0.57 0.37

Adequacy of the exercises to the content.

0.55 0.54

LoadsItems h2

Debate at forums. -0.88 0.71Quality of the chats. -0.86 0.66News and reminders spread during the course.

-0.75 0.64

Guidelines for research on the internet.

-0.70 0.59

Content of the links available in the CBT.

-0.61 0.56

N 906 906Eigenvalue (Valor próprio)

5.040 1.527

% Explained Variance 42.003 12.725 54.73No. Of items 7 5 12

Alpha de Cronbach (α) 0,9727 0,8758

Table 2: Factorial Loads for the items in Study 1

Study 1:Study 1:Descriptives: Courses ‘MA’ and ‘FC’ were both well evaluated, as it is shown in the Table 1.Factorial analysis and internal consistency analysis presented two factors: Traditional procedures (alpha de Cronbach=0,97), and Procedures based on new information technology (alpha de Cronbrach=0,88), as it is shown in the Table 2.

Study 2:Study 2:First, researchers proceeded to a technical evaluation of the didactic material of the course, making use of the The Analysis of the Didactic Material Checklist.Then, participants of the course answered to a questionnaire containing five open questions about:

1. application of what participants learned;2. what aspects of the participant’s work improved because of the training program;3. participants’ opinion about the exercises;4. what aspects should be improved in the training program;5. what participants would like to learn in the course and haven’t learnt.

  Study 1 Study 2

What was the context?

2 distance education courses: (1) Program ‘MA’ — 60h at distance, 40h at presence; 710 participants. (2) Program ‘FC’ — 60h at distance; 223 participants.

A web-based training program with the main objective of capacitating consultants from a big Brazilian financial institution.

What was the object?

Training reaction to procedures: (a) Traditional procedures — the ones common to both distance and presential modalities (e.g.: quality of instructional objectives, sequence of content); (b) Procedures based on new information technology (e.g.: use of chat, tutoring).

Training reaction to different aspects of the course, such as: (a) Instructional Strategies; (b) Content; (c) Layout; (d) Type of language used; (e) Tutoring.How was

it done?

Quantitative method: Questionnaire 12 items Likert scale (0=awful; 10=excellent)

Qualitative methods: Qualitative Analysis of the didactic material; The content analysis of learners’ answers to open questions.

Technical Evaluation

  - Weakness of exercises - Layout

problems -

Navigability problems - Lack of

practical tools - Lack of interactive

alternatives -

Weakness of the content

Participants' Answers

1

2

3

4

5

Positive: “Today I feel more prepared to sign a contract with a client, acting accord his style, to negociate” / “The content helped me in the selection of the consultants in my area.”

Negative: “I haven’t applied it yet, at any kind of consult”. / “The contents at the web training are theorics, distant of the practice and the day by day of a internal consult.”

Positive: “I learned the differences of many types of a consult.” “We feel more secure to propose actions”.

Negative: "Nothing. I already knew what was there."

Positive: “The exercises were very good, because they helped me to memorize the contents” / “They are relevant.”

Negative: “There were few exercises” / “In general, they were not good”. / “I exercised my memory, not the application of the theory.” / “Totally incompatible with the contents in the unit.”

Navigability Interaction Grammar Errors Applicability Sequence and Distribution of content

“The use of tools for projects.”/“The real role of a consultant at the organization.”/“Practical guidelines about leading with clients.”

Traditional procedures

Procedures related to NT

Both courses

8.67< M < 9.30 8.67< M < 9.30

Program ‘MA’

8.64< M < 9.22 6.78< M < 8.3

Program ‘FC’

9.17< M < 9.69 8.00< M < 9.90

Table 1: Descriptives in Study 1

Table 3: Results in Study 2