The Effect of Computer-Mediated Collaborative Learning on...

The Effect of Computer-MediatedCollaborative Learning onSolving III-Defined Problems

EI Daniel UribeJames D. KleinHoward Sullivan

The positive effects of collaborative learning inaface-to-face environment are well known.However, little empirical research exists todetermine f such effects transfer to acomputer-mediated environment. The purposeof this study was to investigate the effect ofcomputer-mediated collaboration on solvingill-defined problems. Participantsfirst workedthrough a Web-based instructional programthat taught them afour-step problem-solvingprocess. Then they worked incomputer-mediated dyads or alone to apply thesteps to solve a realistic problem scenario.Results indicated that participants whoworked in computer-mediated collaborativedyads performed significantly better than didparticipants who worked alone. The resultsalso indicated that dyads spent significantlymore time than participants in the individualtreatment. Both treatment groups had positiveattitudes toward working collaboratively,Internet-based instruction, and transfer ofproblem-solving skills. Implications for theimplementation of computer-mediatedcollaboration in distance learning arediscussed.

E1T&D, Vol. 51. No. 1. 2003, pp. 5-19 ISSN 1042-1629

L Problem solving is regarded as one of themost important cognitive activities in everydaylife and a primary goal of the education process(Jonassen, 2000; Phye, 2001). A possible ap-proach for students to gain this critical skill isthrough problem-based learning. Proponents ofproblem-based learning describe it as a power-ful instructional approach that is engaging andthat leads to sustained and transferable learningof problem-solving skills (Mergendoller, Bel-lisimo, & Maxwell, 2000; Stepien & Gallagher,1993).

Problem-based learning uses authentic, com-plex problems as the impetus for learning andfosters the acquisition of both disciplinaryknowledge and problem-solving skills (Edens,2000; Flynn & Klein, 2001; Levin, 1995). Becauseof its potential to enhance knowledge acquisi-tion, problem-based learning has become apopular method to deliver classroom instructionin education, and has been widely implementedin a variety of other academic environments(Edens; Flynn & Klein; Kinzie, Hrabe, & Larsen,1998, Shulman, 1992).

Current research suggests that a collabora-tive learning environment can positively affectperformance on problem-solving tasks. Col-laborative learning is defined as "an activity thatis undertaken by equal partners who work joint-ly on the same problem rather than on differentcomponents of the problem" (Brandon & Hol-lingshead, 1999). A meta-analysis of the use ofcollaborative learning in higher education cour-ses indicated that collaborative learning promoteshigher achievement, higher level reasoning, more

5

ETR&D, Vol, 51, No. 1

frequent generation of ideas and solutions, andgreater transfer of learning than individual orcompetitive learning strategies Uohnson,Johnson, & Smith, 1991). In another meta-analysis of 122 studies that examined smallgroup and individual learning with technology,small groups were found to be a more effectivelearning structure than individual learning(Lou, Abrami, & d'Apollonia, 2001).

The research conducted to examine the effectof collaboration on problem solving supportsthe hypothesis that a collaborative learning en-virornent is well suited for problem-solvingtasks. In several case studies conducted toanalyze the impact of a collaborative environ-ment on problem solving, collaboration wasfound to improve performance on complex orhigher order thinking activities (Chang & Smith,1991; Johnson & Chung, 1999; Mergendoller etal., 2000). In these studies, learners appeared tobenefit from the ability to discuss the problem,brainstorm potential solutions, and arrive at afinal solution. However, these studies have beenconducted in face-to-face environments; there islittle empirical evidence to indicate if the posi-tive effects of collaborative learning duringproblem-solving tasks will transfer to a com-puter-mediated collaborative environment.With enrollments in courses delivered over theInternet in the United States already in the sixfigures (Simonson, Smaldino, Albright, &Zvacek, 2000), it is important to investigate if thepositive effects of a collaborative learning in aface-to-face environment transfer to a computer-mediated collaborative learning structure.

The characteristics of the Internet and a com-puter-mediated environment appear to makethem ideal for problem-based learning. Accord-ing to Laffey, Tupper, Musser, and Wedman(1998), computer-mediated learning on the In-ternet is suitable for project-based learning be-cause it provides ample resources, allowingstudents to do their own planning and presentnew forms of knowledge, which expands themechanisms for collaboration and communica-tion. Others also argue that computer-mediatedcollaboration and the Web are excellent tech-nologies for case studies and integrating higherorder learning (Jonassen, Prevish, Christy, &Stavrulaki, 1999).

Empirical research on the effects ofsynchronous computer-mediated learning onproblem-solving skills is sparse. According toMurphy and Collins (1997), research onsynchronous computer-mediated communica-tion has been limited to investigations of therecreational use of online chat systems, but theuse of these systems for instructional purposes,and specifically for problem-solving tasks, hasbeen explored only through case studies. Thesecase studies support the hypothesis that thebenefits of collaborative environment in a face-to-face environment transfer to a computer-mediated environment.

Current research on computer-mediated col-laborative learning indicates that it is effectivewhen students are faced with higher order cog-nitive tasks such as problem solving (Johnston,1996). In a short pilot study on the use of com-puter-mediated collaborative groups inpostcompulsory teacher education in the UnitedKingdom, results indicated that students usingcomputer-mediated communications workedbetter with higher order cogritive tasks thanstudents in the control group (Hall, 1997). Inanother case study where collaborative learnigfacilitated through computer-communicationwas used, nurse practitioners appeared toderive more benefit from the experience of theirpeers by working and sharing information viacomputer-mediated communications (Naidu &Oliver, 1999).

Current research also indicates that thequality of interaction between learners in a com-puter-mediated environment may actually bebetter than interaction in a face-to-face environ-ment. Findings in a case study suggested thatcomputer-mediated groups seemed to put morethought into the comments they made, thusproviding higher quality responses (Camin,Glicken, Hall, Quarantillo, & Merenstein, 2001).In another study findings indicated that the in-teraction patterns of computer-mediated groupsresembled thoughtful discussions whereas face-to-face interactions resembled recitations(Hillman, 1999). And in yet another study wherecomputer-mediated communications were com-pared to face-to-face interactions, findings sug-gested that in the computer-mediatedenvironment there was a tendency to share ideas

6

COMPUTER-MEDIATED COMMUNICATIONS IN PROBLEM SOLVING

without the restraints of typical social conven-tions, resulting in deeper and more thoughtfuldiscussions (Kruger & Cohen, 1996).

Another variable that has been shown to in-fluence achievement in a collaborative setting isability grouping. Ability grouping is the assign-ment of participants to small groups based ongeneral academic ability. Although studies ad-dressing ability grouping when solving ill-defined problems in computer-mediatedenvironment are rare, several studies have ad-dressed ability grouping in face-to-face col-laborative or cooperative environments. Some ofthese studies suggest that heterogeneous group-ing should be used in collaborative environ-ments since this allows for the higher abilitystudent to help and encourage the lower abilitystudent (ohnson, Johnson, & Holubec, 1990;Slavin, 1993).

In a meta-analysis of 27 studies dealing withability grouping conducted by Slavin (1993),there was little to no achievement differencereported between students groupedheterogeneously versus homogeneously byability. However, the lower ability students indi-cated more favorable attitudes toward learningwhen grouped with students of higher ability(as cited in Sherman & Klein, 1995). But otherstudies suggest that homogeneous groupingmay be the best alternative when workingcooperatively. In a study by Sherman and Klein(1995), findings showed that the self-confidenceof high-ability students was negatively affectedby being paired with a low-ability student. Thisis in line with other research, which has shown anegative impact on high-ability students whenpaired in heterogeneous dyads. (Hooper, 1992;Hooper & Hannafin, 1991).

The current study investigated the effects oftwo levels of learning structure (individualWeb-based learning versus computer-mediatedcollaborative Web-based learning) and abilitygrouping (high vs. low) on learner performancein solving ill-defined problems. Data on perfor-mance, attitudes, and time on task were col-lected for all participants. Data were alsocollected on the quality of interactions betweenparticipants in the computer-mediated col-laborative learning group. The research ques-tions for this study were:

1. Does learning structure (individual Web-based learning versus computer-mediatedcollaborative Web-based learning) affectlearner performance in solving ill-definedproblems in a Web-based program?

2. Does ability grouping (high vs. low) affectlearner performance in solving ill-definedproblems?

3. Does collaboration reduce variability in testscores among learners?

4. Do learning structure and ability groupingaffect time on task?

5. Does learning structure affect learner at-titudes toward collaborative learning, Web-based instructional programs, and transfer ofproblem-solving skills to other tasks?

Based on the review of the literature, fouroutcomes were predicted for this study. First,that participants working in computer-mediatedcollaborative groups would perform significant-ly better in resolving ill-defined problems thanparticipants who worked individually (Johnsonet al., 1991; Lou et al., 2001); second, that higherability participants would outperform lowerability participants (Sherman & Klein, 1995);third, that since participants working in dyadswould spend time in discussion, it would takethem longer to complete the program than par-ticipants working alone (Johnson et al., 1991;Lou et at., 2001); and finally, that participants inboth treatment groups would have a positive at-titude toward collaborative learning, Web-basedinstruction, and transfer of problem-solvingskills (Johnson et al., 1991; Lou et at., 2001; Sher-man & Klein, 1995).

METHOD

Participants

The participants in this experiment were 59Reserve Officer Training Corps (ROTC) studentsfrom a large southwestern university in theUnited States. All participants were enrolled inan aerospace studies course required of allROTC students. All participants were volun-teers and were selected based on whether or notthey had been previously trained on the prob-

I

7

ETR&D, Vol. 51, No. 1

lem-solving process that was used in this study.Only cadets that had not been previouslytrained were selected. There were 47 male and12 female participants.

Materials

The materials for this study were developed andincorporated into a Web-based interfacethrough the BlackboardTM course managementsystem. Blackboard is a course management sys-tem that provides faculty members the ability todeliver courses over the World Wide Web. Thestudy consisted of four items embedded withinthe Blackboard interface: (a) an instructionalprogram, (b) a knowledge quiz, (c) a problemscenario, and (d) the assessment questions forthe problem scenario.

Instructional program. A Web-based instruction-al program on problem solving was developedfor this study using AuthorwareTM. The instruc-tional program focused on a four-step problem-solving process derived from the Air Force"Six-Step Problem-Solving Process" commonlytaught to college juniors enrolled in the AirForce ROTC program. The six-step process wasstreamlined to four steps for ease of delivery in aWeb-based environment. The two major dif-ferences between the six-step and the four-stepprocesses were that in the four-step version, (a)two steps were combined into one (developingand testing possible solutions), and (b) a finalstep to implement and monitor the solution wasdeleted. The modified approach was intended toprovide students with a tool to solve complexproblems in a Web-based environment.

The instructional program is Web-based andinteractive. An animated cartoon (the "agent")leads the students through the following foursteps: (a) defining the problem, (b) gatheringdata, (c) developing and testing possible solu-tions, and (d) selecting the best possible solution.The agent explanations were all text-based; noaudio was included in this version of the instruc-tional program. The agent explains each stepand then uses a problem scenario to show theleamers how to apply the step. For example, aspart of the instruction for Step 1 (defining the

problem), the agent explains that the learnermust identify the individuals involved, the goal,and the obstacle preventing achievement of thegoal. The agent then uses a problem scenario tohighlight each of the elements of the problem en-vironment and how these lead to a problemstatement. The learner then practices describingthe problem environment and writing a prob-lem statement with a different problem scenario.Overall, the student is exposed to two differentproblem scenarios throughout the program. Thefirst scenario is solved by the agent as he ex-plains each step of the process. The second prob-lem scenario is solved by the student as he or shepractices each step. After completing the instruc-tional program, the participants are ad-ministered a knowledge quiz on the four-stepproblem-solving process.

Knowledge quiz. A 10-item multiple-choice quizwas developed to assess how well studentslearned the four-step process. The questions ad-dressed things such as the sequence of the steps,subtasks within each step, and critical outcomesof each step. The following is an example of thetype of multiple-choice question found in theknowledge quiz. The students were able tochoose only one answer:

Which of the following statements would you classifyas criteria?

(a) Survey results indicate the personnel like the newdorms.

(b) ASAP means As Soon As Possible.

(c) The chief of logistics thinks the job should onlytake a day.

(d) Your boss says you have three days to find a solu-tion.

The quiz scores indicated that the par-ticipants learned the problem-solving processwell (M = 8.98, SD = 1.06). The quiz results werepositive across both treatment groups andability levels. The average score for higherability participants working in dyads was 9.37,SD = .62, while the average score for lowerability participants working in dyads was 9.1,SD = 1.1. Higher ability participants working in-dividually scored an average of 9.1, SD = .99,while lower ability participants working in-dividually scored an average of 8.4, SD = 1.3.After completing the quiz the participants were

8


instructed to apply the process they had justlearned to a realistic problem scenario.

Assessment problem scenario. The assessmentscenario was developed with a military themeand portrayed a realistic personnel issue that theofficer candidates participating in the studymight face in the future. The following is the en-tire problem scenario text.

1000 Hours: somewhere in the jungles of a small Carib-bean island . . .

You are Capt. Klein, assigned to Joint Task Force-Bravo, participating in a multinational peacekeepingoperation in the island of Tamos (fictitious). Troopsfrom Peru, Honduras, Guatemala, Ecuador and theUnited States were deployed to restore and keep thepeace in this small Caribbean nation. You are in com-mand of an Air Force logistics squadron with 10 mem-bers composed of 2 members from all participatingnations. As you focus on accomplishing the mission,you receive a request from a member of theEcuadorian Air Force for a transfer out of the unit. Youknow your boss Col. Sullivan will not allow any per-sonnel transfers. You know you'll need to brief theproblem and your recommended solution at the nextstaff meeting....

As part of the problem scenario, the par-ticipants had access to additional information inthe form of simulated interviews with the in-dividuals involved in the problem scenario andWeb sites that provided information the studentcould use to solve the problem. Once the par-ticipants felt they were ready, they navigated tothe assessment area of Blackboard where theyanswered four essay questions that paralleledthe steps of the four-step problem-solvingprocess. All of the participants, including thoseworking in dyads, submitted individualanswers for each question. The Blackboard sys-tem was set up to allow the participants tochange an answer after it had been submitted.The four essay questions the students had toanswer as part of the problem scenario were:

1. Define the overall problem environment and writea problem statement.

2. List and categorize the data that are relevant forsolving this problem.

3. List as many solutions as possible that meet yourcriteria.

4. List additional criteria and select the best possiblesolution for this problem scenario.

Scoring rubric. The lead researcher developedthe scoring rubric used to evaluate theparticipants' responses to the essay questions.Given the nature of an ill-defined problemscenario where multiple solutions are pos-sible, there was not a single right or wronganswer. Instead, points were awarded for howwell the students applied each step of theprocess. The four essay questions paralleledthe four steps of the problem-solving process.Under Step 1, Defining the Problem, the par-ticipants were evaluated on the clarity of theproblem statement and how well they iden-tified the elements of the problem environ-ment, which include the individuals involved,the goal, and the obstacle. Under Step 2,Gathering Data, the participants wereevaluated on the amount and quality of datagathered and on the data classification. UnderStep 3, Developing and Testing Possible Solu-tions, the participants were assessed on thenumber and quality of solutions. Finally,under Step 4, Selecting the Best PossibleAnswer, the participants were assessed on theadditional criteria used and on the recom-mended solution.

The scoring rubric was designed to assessparticipant performance on each step by break-ing down the step into subcategories and using arating scale to assign a score based on three dif-ferent levels of performance. The participantsreceived 0, 1, or 2 points for each subcategory,depending on the quality of the answer. For ex-ample, for identifying the goal under Step 1, theparticipant received no points if no goal wasidentified, 1 point if a goal was identified butwas undear or inconsistent with the problemscenario, and 2 points if the goal was clearlystated (i.e., "The goal is unity within thesquadron"). As another example, when select-ing the best possible solution under Step 4, theparticipants received no points if no solutionswere provided, 1 point if a solution wasprovided but it was not a logical choice given thedata gathered in the previous steps, and 2 pointsif the solution was a logical choice given the dataand criteria gathered in previous steps.

The overall points assigned to each step werebased on the amount of information that was ex-pected from each participant. Step 1 had four

9


subcategories and was worth 8 points becausethe instructional program placed a heavy em-phasis on identifying the problem, while Steps2-4 were worth 4 points each and were dividedinto two subcategories. The participant respon-ses were blind graded by one of two evaluatorsto prevent bias in the grading procedure. Inter-rater reliability was determined by havingtrained evaluators score four participant respon-ses to the assessment scenario. The evaluators'scores were then processed using StatisticalPackage for the Social Sciences (SPSS) to com-pute an interrater reliability of .86.

Attitude Survey. A 10-item survey was devel-oped to measure participant attitudes towardworking alone or with a partner, as well as theirattitude toward the instructional program andtoward transfer to other tasks of the problem-solving skills leamed. Respondents used a 5-point Likert scale (1 = strongly disagree, 5 =strongly agree) to rate their attitude toward work-ing in a Web environment and working alone orin dyads, and their perception of transfer of theirproblem-solving skills to other real-life-typeproblems. Additionally, the students were askedto identify their preference for working on com-plex problems by circling one of two choices:alone or with a partner. Finally, 2 open-endedquestions asked the participants what they likedbest and least about the program. The alphareliability coefficient of the survey was .71.

Procedures

The two treatment groups for this study were (a)an individual Web-based learning group and (b)a computer-mediated collaborative Web-basedlearning group. The individual Web-basedlearning group initially consisted of an equalnumber of higher ability and lower ability par-ticipants, while the computer-mediated col-laborative Web-based learning group consistedof dyads composed of one higher ability and onelower ability student. Students in the individualWeb-based learning group worked alonethrough the entire program. Students in thecomputer-mediated collaborative Web-basedlearning group worked individually throughthe instructional part of the program and then

used synchronous computer-mediated com-munication to work collaboratively on the prob-lem scenario.

The participants were blocked into higherability and lower ability groups based on theiracademic composite. The academic compositewas calculated using the student's grade pointaverage (GPA) and current average in the ROTCclass. In computing the academic composite, theGPA was weighted heavier than current dassaverage because GPA was considered a morereliable indicator of the student's generalacademic ability. The median split for GPA was3.4 out of 4.0. Students were then randomly as-signed in equal numbers to the individual treat-ment or computer-mediated collaborativetreatment. Within the computer-mediated col-laborative group, one higher ability student andone lower ability student were randomly as-signed to form dyads. For participants workingin dyads, the higher ability participants had aGPA of 3.7, SD = .26, while lower ability par-ticipants had a GPA of 2.9, SD = .63. For par-ticipants who worked individually, higherability students had a GPA of 3.7, SD = .20, whilelower ability participants had a GPA of 2.9, SD =.44.

Four weeks prior to the study, the par-ticipants were instructed to ensure they had auser name and password that allowed them ac-cess to the university Blackboard system. If stu-dents did not have access to the system, theywere provided with instructions on how to get atemporary user name and password in order toparticipate in the study. One week before thestudy, participants were given a short briefingon how the study was organized, and its loca-tion and time. The participants were alsonotified that, as an incentive to perform well, thetop performers in each treatment group (i.e., topindividual student and the two students in thetop dyad) would receive a gift certificate to apopular dining establishment. Any technical is-sues related to access to the Blackboard systemwere also cleared up during this briefing.

The day of the study, participants weredirected to prearranged locations in the com-puter laboratory that ensured each member of adyad was physically separated from his or herpartner. This was done to prevent verbal or

10


bodily communications between the membersof the dyad, thus simulating a distance educa-tion environment. The participants were in-structed to log in to the Blackboard system andto navigate to the study Web page. Onceeveryone was properly logged in to Blackboard,the researcher instructed them to follow the in-structions on the screen and that they had 1.5 hrto complete the program. The participants werealso provided with a hard copy of the instruc-tions.

All students worked through the instruction-al program individually. Once the participantscompleted the instructional program, they wereinstructed to take the knowledge quiz. Once thequiz was completed, participants in the in-dividual group proceeded to the assessmentproblem scenario while the computer-mediateddyads were instructed by the program to estab-lish communication with their partners and tocollaborate on the assessment problem scenario.The communication between members of thedyads took place using the virtual classroomfeature of the Blackboard system. This feature al-lowed the students to chat with their partners byentering a virtual classroom that had been set upby the researchers for each dyad. Each dyad wasassigned a different virtual classroom to preventcross flow of information between dyads. Therewere no interactions between the participantsand the researchers or instructors except toremedy any technical difficulties.

The interactions between the members ofeach dyad were captured using an automaticfunction of the virtual classroom within Black-board that recorded the text communications be-tween members of the dyads. The logs for eachdyad were then transferred to a database pro-gram and separated by individual entry in orderto be analyzed.

Design and Data Analysis

This study was a posttest only 2 (Individual vs.Computer-Mediated Collaborative Learning) x2 (Higher Ability vs. Lower Ability) factorialdesign. The primary dependent variable wasstudent performance resolving an ill-definedproblem scenario. Time on task and learner at-

titudes were also analyzed. En route data in-cluded a computer record of the interactions be-tween the members of the dyads. Theknowledge quiz results were too skewed to beinterpretable, therefore only descriptive statis-tics were used to analyze those data. Analysis ofvariance (ANOVA) was conducted onparticipants' performance on the assessmentproblem scenario and on time on task. Multi-variate analysis of variance (MANOVA) wasconducted on the data from the attitude survey.One-sample chi-square tests were conducted onthe open-ended question: "When solvingproblems I prefer to work: by myself or with apartner." One chi-square test was conducted todetermine the overall student preference, andthen a chi-square was performed for treatmentgroup and for ability level.

The interactions between members of thedyads were analyzed using both qualitative andquantitative analysis techniques. The entries inthe communications transcripts captured viaBlackboard were first grouped into categoriesbased on the type of interaction. The entrieswere classified as questions, answers, discus-sions and off-task interactions as determined bythe lead researcher, and descriptive statisticswere then computed for each category. Previousresearchers have used these categories in studiesexamining small group interactions (Doran &Klein, 1999; Klein & Pridemore, 1994). An entrywas classified as a question if one member of adyad requested information or asked forclarification from the other team member (e.g.,"Have you looked into the information sites onthe Web yet?"). An entry was classified as ananswer if the communication was in directresponse to a question (e.g., "No I haven't. I readthe information of some of the members of thesquadron"). An entry was considered a discus-sion if the communication directly related to theproblem the dyad was trying to solve (e.g., "Ithink we should find out if sgt robles [sic] knowsthose comments were offensive . . . solve theproblem at the lowest level possible"). Finally,an entry was categorized as an off-task discus-sion if the communication was not related to theproblem-solving task (e.g., "dang . . . my key-board froze ... but I'm back now!").

11


Difference Scores Within Dyads

Performance by Learning Structure andAbility Level

The first two research questions investigated theeffect of learning structure and ability level onperformance in solving ill-defined problems.Table 1 shows the mean scores and standarddeviations for performance on the assessmentscenario. The table reveals that participants whoworked in a dyad had an overall average of11.94 (60%), while participants who worked in-dividually had an average of 9.89 (50%). Thedata also show that higher ability participantshad an overall average of 11.76 (59%) and lowerability participants had an overall average of10.27 (51%). A 2 x 2 ANOVA conducted on thesedata revealed that participants working indyads had a significantly higher performancescore than those working alone, F(1,55) = 6.58, p= .01, 112 = .11. Although there was a difference

between ability groups, this difference only ap-proached significance (p = .06) and there was nosignificant interaction.

Table 1 II Means scores and standarddeviations for performance onthe problem scenario.

Learning StructureAbility Level CMC Dyads Individuals Overall

High AbilityMean 12.37 11.00 11.76(SD) (2.94) (2.89) 2.95n 16 13 29

Low AbilityMean 11.50 8.86 10.27(SD) (3.12) (3.01) (3.3)n 16 14 30

Over allMean

(SD)n

11.94(3.01)

32

9.89

(3.09)

27

11.00

(3.19)

59

Note: The maximum number of points on the problemscenario was 20.

The third research question directly dealt withthe impact of collaboration on performance solv-ing ill-defined problems. A posthoc procedurewas conducted to better understand the effect ofcollaboration on performance. It was theorizedthat the pairs of participants who worked indyads (n = 16) would have performance scoresthat were more similar to each other than pairswho worked individually but were formed intodyads of one higher ability and one lower abilityparticipant posthoc (n = 13) to make this com-parison. The rationale was that the collaborativeeffort in dyads would influence the performanceof the two participants to make it more similar toone another, whereas no such influence could beeffected in pairs of participants who worked in-dividually but were constructed posthoc. A dif-ference score for each dyad was obtained bysubtracting the lower score from the higherscore. The data revealed that the average dif-ference between the two participants whoworked in each dyad was 1.5 points, SD = 1.6,while the difference between participants whoworked individually but were assigned to a pos-thoc dyad was 4.6 points, SD = 3.1. An ANOVAperformed on these data showed that the meandifference between the students who worked ina dyad was significantly lower than the dif-ference between those who worked individuallybut were formed into posthoc dyads, F(1,27) =

11.97, p = .002, 112 = .31.

Time on Task

The next research question pertained to the ef-fect of treatment on time on task. Participants inboth treatments were allowed a maximum of 90min to complete the program. The data revealedthat higher ability participants who worked in-dividually spent an average of 77.46 min, SD =13.61, while higher ability participants whoworked in dyads spent an average of 88.38 min,SD = 3.44. Additionally, lower ability par-ticipants who worked individually spent anaverage of 75.29 min, SD = 15.43, while lowerability participants who worked with a partnerspent an average of 87.88 min, SD = 4.33. A 2 x 2

RESULTS

12


ANOVA indicated that participants whoworked in dyads spent significantly more timeon task than participants who worked alone,F(1,55) = 19.24, p < .001, 12 = .26. The differencebetween ability groups was not significant andthere was no significant interaction.

Participant Attitudes

The last research question investigated the effectof the treatment on the attitudes of the par-ticipants. A 2 x 2 MANOVA conducted on stu-dent attitudes indicated that learning structurehad a significant effect on student attitudes,F(10,42) = 2.44, p = .022, 9

2= .37. Follow-up

ANOVA revealed two significant items. Thefirst item dealt with participant attitude towardtime available to complete the program. Par-ticipants who worked alone felt they had moretime to complete the program (M = 4.2, SD =1.22) than those who worked in dyads (M = 3.4,SD = 1.0), F(1,53) = 6.5, p = .014, i9 = .11. Thesecond item showing significant difference con-cerned ease of use of the Blackboard system.Participants who worked alone thought Black-board was easier to use (M = 4.4, SD = .65) thanparticipants who worked in computer-mediateddyads (M = 3.3, SD = 1.18), F(,53) = 17.42, p <.0 2 = .25. The attitude survey furtherrevealed that participants had a generally posi-tive attitude toward working on an Internet-based program (M = 3.82, SD = .86) and towardtransfer of problem-solving skills to their profes-sional lives (M = 3.84, SD = .76).

The attitude survey also included one optionquestion that asked, "When solving problems Iprefer to work: BY MYSELF or WITH A PARTNER." Aone-sample chi-square test revealed that of the54 participants who responded to this question35 indicated a preference for working with apartner while 19 indicated a preference forworking alone, X2 (1, N = 54) = 4.74, p = .03, effectsize = +0.09. The results by treatment grouprevealed that 17 of the 24 participants whoworked individually indicated a preference forworking with a partner, X2 (1, N = 24) = 4.17, p =.04, effect size = +0.17, while 18 out of 30 par-ticipants who worked in dyads selected "work-ing with a partner" as a preference, X2 (1, N = 30)

= 1.2, p = .27, effect size = +0.04. When analyzedby ability level, 15 of the 27 higher ability par-ticipants indicated a preference to work with apartner, X2 (1, N = 27) = .33, p = .564, effect size =+0.01. Additional tests showed that 20 out of 27lower ability learners indicated a preference towork with a partner, X2 (1, N = 27) = 6.26, p =

.012, effect size = +0.23.

The attitude survey also included two open-ended questions that asked the participantswhat they liked best and least about the pro-gram. The top four responses for what par-ticipants liked best included (a) the realism ofthe problem scenario (n = 13), (b) using the Inter-net to solve problems (n = 9), (c) the instructionalprogram (n = 8), and (d) learning a new ap-proach to solve problems (n = 8). The top fourresponses for the least liked aspects of the pro-gram were (a) the difficulties communicatingthrough Blackboard (n = 17), (b) working alone(n = 7), (c) having to work with a partner (n = 7),and (d) lack of time to complete the scenario (n =6).

Interaction Data

The interactions were first reviewed to deter-mine if these data revealed any strategies forproblem solving or any evidence that collabora-tion was beneficial to solving the problems.These findings are reported in the discussionsection and are used to support other findings.There was a total of 1,494 communications be-tween the members of the dyads-20% werequestions, 18% were answers, 51% were discus-sions and 11% were off-task entries.

DISCUSSION

The purpose of this study was to investigate theeffects of computer-mediated collaborativelearning and ability on learner performancesolving ill-defined problems. Results indicatedthat participants who worked in computer-mediated collaborative dyads performed sig-nificantly better than did participants whoworked alone. The results also indicated thatcomputer-mediated dyads spent significantly

13


more time than participants in the individualtreatment, and both treatment groups had posi-tive attitudes toward working collaboratively,toward the instructional program, and towardtransfer of problem-solving skills.

Performance by Learning Structure andAbility Level

The initial research questions dealt with the ef-fect of learning structure and ability level on per-formance in solving ill-defined problems. Thefinding that computer-mediated dyads per-formed significantly better than participantswho worked alone supports previous findingsthat showed computer-mediated collaborativelearning had a positive effect on achievement(Alavi, 1994; Hall, 1997; Johnston, 1996; Naidu &Oliver, 1999). This result also indicates that thebenefits of face-to-face collaboration on prob-lem-solving tasks transfer to a computer-mediated environment. Participants whoworked with a partner appeared to havebenefited from the ability to discuss the problemand possible solutions, which is in line withfindings from studies on face-to-face collabora-tive environments for problem solving (Chang& Smith, 1991; Flynn & Klein, 2001; Johnson &Chung, 1999; Johnson et al., 1991; Mergendolleret al., 2000).

The superior performance of computer-mediated dyads relative to participants whoworked alone also supports the hypothesis thata computer-mediated collaborative environ-ment is well suited for problem-solving ac-tivities and higher order learning Jonassen etal., 1999). It is possible that participants whoworked with a partner were able to access moreinformation related to the problem and couldgenerate better problem solutions than theircounterparts who worked alone.

A somewhat unexpected result of this studywas the nonsignificant difference finding byability level. There are two possible reasons forthese results. First, lower ability participants inthe computer-mediated dyads seemed to havebenefited by being paired with a higher abilityparticipant. This finding is in line with other re-search, which has shown that lower ability

leamers benefit from being paired with higherability learners (Johnson et al., 1990; Slavin,1993). An inspection of the means shows thatthere was a greater difference in the individualtreatment between lower ability and higherability participants than in the computer-mediated collaborative treatment; however, thedifference was not enough to yield a significanttreatment by ability level interaction. A secondpossible reason for the nonsignificant finding byability is that the GPA method used to assign anability score to each participant may not havebeen accurate enough. Although GPA can be agood indicator of general academic ability, itmay not be a good predictor of specific problem-solving skills.

Difference Scores Within Dyads

Evidence that collaboration in dyads for solvingcomplex tasks positively affected leamer perfor-mance came from the results of the posthoc pro-cedure conducted with the performance data.These data showed that the difference betweenthe scores of the two participants in a dyad wassignificantly smaller than in posthoc dyads com-posed of participants who worked alone. Theseresults are not surprising if we assume that col-laboration is indeed taking place. In a truly col-laborative environment it would be expectedthat the members of a dyad would workthrough the information and arrive at a solutiontogether and would therefore have a greaternumber of similar responses than posthoc dyadswho worked alone. By discussing the informa-tion available, the learners who worked with apartner were exposed to another perspective onthe problem, which may have helped them tobetter shape the mental model of the problemenvironment. This outcome supports previousfindings about the quality of interaction in acomputer-mediated environment wherelearners working in computer-mediated en-vironments exhibited more thoughtful discus-sions (Camin et al., 2001; Hillman, 1999; Kruger& Cohen, 1996).

A look at the interactions that took place be-tween members of the dyads seems to supportthis argument. The majority of the dyads ap-

14


peared to follow a similar pattern to arrive at asolution. The first pattern that emerged was anattempt by the dyads to develop a strategy tosolve the problem through a question-and-answer exchange. The following is an interac-tion example from a dyad that used a "divideand conquer" approach to solve the problem:

Participant X: Hello.

Participant Y: Hello!

Participant X: Have you looked into the informationsites on the web yet?

Participant Y: No I haven't. I read the information ofsome of the members of the squadron.

Participant Y: I began to read a little bit about the con-flict between Peru and Ecuador.

Participant X: Can we take notes?

Participant Y: Yes.

Participant X: Let's split up the web sites. . .

Participant Y: Ok.

Participant Y: Which three do you want to read?

Participant X: I'll read the first three articles.

Participant Y: Ok. I'll meet you back in here in a fewmin.

Participant X: ok.

Once the dyads developed a strategy to solvethe problem, the second general pattern ap-peared to be an attempt to follow each step ofthe problem-solving process by exchanging in-formation through interactions focused on arriv-ing at a problem solution. The following is anexample of this type of exchange:

Participant A: I think the problem can be found withMSgt Robles.

Participant A: This is because he doesn't appreciateTSgt Paredes at all.

Participant B: That would be a good assumption.

Participant B: well the research shows that there is con-flict with Ecuador and Peru.

Participant A: So what do you feel the overall problemenvironment is and how would we write a statementabout it.

Participant B: I think the problem is that robles andparedes can't get along.

Participant A: I would agree with this however, is thisjust a result of a bigger problem between theircountries.

Participant B: Yes.

Participant A: Our goal would be to have Robles and

Paredes get along.

Participant B: I agree.

The analysis of the interaction data may alsohelp to explain why the collaborative dyads per-formed better and more similarly to each otherthan individuals. A breakdown of the 1,494 in-teractions showed that 90% of the communica-tions between participants were focused onasking or answering questions or discussing in-formation related to the problem. It is reasonableto assume that if the participants were focusedon solving the task, then the effects of collabora-tion would be positive.

Time on task

The results of the time-on-task analysis may alsohelp to explain why computer-mediated col-laborative dyads performed better than par-ticipants who worked alone. The dyads spentsignificantly more time on the program thanparticipants who worked individually (88.48min to 76.33 min). This result supports the find-ing by a meta-analysis of 122 studies dealingwith small group and individual learning withtechnology where students working individual-ly accomplished tasks faster than those workingin groups (Lou et al., 2001). It is possible that thedyads generated more possible solutions be-cause they were able to discuss the problem.These interactions may well have been beneficialin developing a better understanding of theproblem and formulating more completeanswers. The following example highlights howthe members of one dyad generated and dis-cussed possible solutions to the problem:

Participant C: What do you think they should do?

Participant D: I think they should be mediated.

Participant C: What if they don't want a mediator?

Participant D: . ..well, what other solutions are there?

Participant C: ... ignore the problem or let them fightit out.

Participant D: How about disciplinary action?

Participant C: That sounds good.

The following is an example of an exchangethat shows how the dyads spent time helpingeach other get a better understanding of theproblem environment:

15


Participant P: ... well, what did you come up with forthe definition of the problem?

Participant Q: No idea. . ., but I would be wonderingwhy he wants to transfer in the first place.

Participant P: well . . I thought of . I am leading aunit from which one man wants to transfer. My bosswon't allow it," so my problem statement would be:"How can we keep the unit together?" . . . soundgood?

Participant Q: do you think that the problem is withthe man wanting to transfer, or the man not being al-lowed to transfer?

Participant P: well, I think the problem is with the manwanting to transfer. The unit needs to get the job done.Does the problem statement sound ok?

Participant Q: wait, I want to read real quick why hewants out ... you're right, sounds like disunity in theunit.

However, the additional time spent on thetask led to a perception by the participants whoworked in dyads that there was not enough timeto complete the program. Leamers who workedalone did not have to spend time communicat-ing with a partner and could concentrate onsolving the task, while participants who workedin dyads experienced the time delays en-countered when having to work with anotherindividual in a computer-mediated environ-ment.

ParticipantAttitudes

In general, participants indicated a preferencefor working collaboratively. This supports pre-vious research that has shown a preference bylearners to work in a collaborative environment(Johnson et al., 1991; Lou et al., 2001). However,the attitude data suggest that some challengesexist in a computer-mediated collaborative en-vironment. The responses to the open-endedquestions suggested that some of the extra timethe dyads spent on the program may have beena result of the problems associated with com-municating through a computer-mediatedmedium. Many participants who worked with apartner cited the difficulties of communicatingthrough the computer as the thing they likedleast about the program. The following are someexamples of the thoughts expressed by the par-ticipants when asked, "What did you like leastabout the program?"

Trying to communicate with my partner.

Keeping contact with my partner was difficult throughBlackboard ...

I didn't like working with a partner over the Internet.It made for slow communication.

I didn't really like using Blackboard to talk to mypartner. It is more helpful being able to talk with mypartner face to face.

It was a little hard to talk to my partner with justtyping.

These findings are supported by the sig-nificant differences found on the attitude itemsrelated to time available to complete the pro-gram and to perceptions on the ease of use of theBlackboard system. Participants who workedalone felt they had enough time to complete thetask and that Blackboard was easy to use. How-ever, participants who worked in dyads felt theydid not have sufficient time and also felt it wasdifficult to communicate through Blackboard.

The survey results also show that the par-ticipants enjoyed the Internet program and feltthey learned a lot from it. This finding supportsother research where student attitudes towardWeb-based instruction have been found to bepositive (Adelskold, 1999; Mclsaac & Ralston,1996; Savenye, 2001). The characteristics of theInternet, such as instant access to various sour-ces of information, appeared to make it attrac-tive for the learners. Some of the comments fromthe students on the open-ended questions seemto support this explanation, as indicated by theirresponses when asked what they liked bestabout the program:

Using the Internet to solve problems and do research.

I did enjoy doing the program over the Internet.

I liked using the Intemet. I liked having everythingthat I needed right at my fingertips.

It was nice having the articles on the Internet and nothaving to thumb through a bunch of books.

The attitude survey also attempted to gaugethe learner perceptions on transfer to theirprofessional lives of the problem skills learned.The results indicated that most participantsagreed that what they had learned would trans-fer to their professional lives. It is possible thatthe participants felt they would use this processin their professional lives because they thoughtthe problem scenario was realistic and could besomething they might face in their future as AirForce officers. The majority of participants indi-

16

17COMPUTER-MEDIATED COMMUNICATIONS IN PROBLEM SOLVING

cated that the realism of the problem scenariowas what they liked best about the entire pro-gram.

Implications

This study has implications for the design anddelivery of Internet-based courses. The study in-dicates that computer-mediated collaborativelearning is a more effective strategy when teach-ing problem-solving skills than is individuallearning. The findings of this study are ap-plicable to academic subjects where a high levelof real-time interactivity and higher order cogni-tive skills may be required. However, as thestudy suggests, text-based communicationshave some drawbacks that should be consideredprior to implementing this type of learningstructure. Instructional designers and instruc-tors should keep in mind the increased timenecessary in a computer-mediated collaborativeenvironment.

The results of this study also inform thetheory of transactional distance. The theorypredicts that if the structure of a course is keptconstant and dialogue is increased, the decreasein the transactional distance should have a posi-tive effect on performance (Moore & Kearsley,1996). The structure remained constant for bothtreatment groups in this study, but the amountof dialogue between learners varied dependingon treatment group. The promotion of dialoguebetween learners in the computer-mediated col-laborative dyads seemed to improve under-standing of the task and information available,leading to a difference in performance in favorof students working in the computer-mediatedenvironment.

clues such as eye contact, body motion, and soforth, are nonexistent, larger groups will havedifficulty communicating. With additional teammembers, the law of diminishing retums beginsto take place and the collaborative environmentbegins to lose its effectiveness. Future researchshould also focus on the effectiveness of com-puter-mediated collaboration on how well stu-dents learn a process to solve complex problems.Do students learn a process better, as they ap-pear to do in this study, if they are able to dis-cuss with a peer online?

Another area that could benefit from addi-tional exploration is the effect of computer-mediated collaboration on different types oftasks. It would be useful for instructional desig-ners and practitioners to know if there are cer-tain types of tasks that are better suited forcomputer-mediated collaboration. For example,the ill-defined problems identified by Jonassen(2000), such as decision-making, strategic per-formance, case analysis, and design problemscould be examined. Investigating these types ofproblems in a computer-mediated environmentshould help instructional designers identify thetypes of problems that are particularly wellsuited for a computer-mediated environment. Afactor that may affect the organization of col-laborative groups in a distance-learning en-vironment is social relationships. In this studythe dyads were randomly assigned, however,the effect of social relationships on attitudesabout heterogeneous grouping could be ex-amined to determine if partner self-selection hasan impact on leamer attitudes. As the number ofdistance education courses delivered over theInternet continues to grow, research on the dif-ferent aspects of computer-mediated collabora-tion should help us identify effectiveinstructional practices for promoting learningand problem solving. El

Further Research

The results of this study suggest several specificareas that should be addressed by additional re-

search. The size of the computer-mediated col-

laborative group should be explored to

determine an optimal size. It is possible that

given the characteristics of the synchronous

communication environment where nonverbal

Maj. Daniel Uribe [[email protected] isDirector of Operations of the Department of ForeignLanguages at the United States Air Force Academy,Colorado. During article preparation, he was a PhDstudent at Arizona State University (ASU), Tempe.

James D. Klein and Howard Sullivan areProfessors of Educational Technology at ASU.

The views expressed in this article are those of theauthors and do not reflect the official policy or

18

position of the United States Air Force, Departmentof Defense, or the United States government.

REFERENCES

Adelskold, G. (1999c7). Problem-based distance learn-ing of energy issues via computer network. DistanceEducation 20(1): 129-143.

Alavi, M. (1994). Computer-mediated collaborativelearning: An empirical evaluation. MIS Quarterly,June, 159-174.

Brandon,D & Hollingshead, A.B. (1999). Collaborativelearning and computer-supported groups. Com-munication-Education, 48(2),109-126.

Camin, C., Glicken, A., Hall, M., Quarantillo, B., &Merenstein, G. (2001). Evaluation of electronic dis-cussion groups as a teaching/learning strategy in anevidence-based medicine course: A pilot study.Education for Health: Change in Learning and Practice,14(1), p.21.

Chang, K.Y.R & Smith, F. (1991). Cooperative learningand CALL/IVD in beginning Spanish: An experi-ment. Modern Language Journal, 75(2), 205-211.

Doran, M.S., & Klein, J.D. (1999). Implementing in-dividual and small group learning structures with acomputer simulation. Educational Technology Re-search and Development, 47(1), 97-110.

Edens, KM. (2000). Promoting communication, in-quiry, and reflection in an early practicum ex-perience via an on-line discussion group. Action inTeacher Education, 22(2A), 14-23.

Flynn, A.E., & Klein, J.D. (2001). The influence of dis-cussion groups in a case-based learning environ-ment. Educational Technology Research andDevelopment, 49(3), 71-86.

Hall, D. (1997). Computer mediated communication inpost-compulsory teacher education. Open Learning,12(3),54-56.

Hillman, D. C. A. 1999). A new method for analyzingpatterns of interaction. The American Journal of Dis-tance Education, 13(2), 37-47.

Hooper, S. (1992). Effects of peer interaction duringcomputer-based mathematics instruction. Journal ofEducational Research, 85(3),180-189.

Hooper, S., & Hannafin, M.J. (1991). The effects ofgroup composition on achievement, interaction, andlearning efficiency during computer-based coopera-tive instruction. Educational Technology, Research andDevelopment, 39(3), 27-40.

Johnson, D.W., Johnson, R.T., & Holubec, E.J. (1990).Circles of learning: Cooperation in the classroom. Edina,MN: Interaction Book Company.

Johnson, D.W., Johnson, R.T., & Smith, C.A. (1991).Cooperative learning: Increasing college faculty instruc-tional productivity. Washington, D.C.: School ofEducation and Human Development, The GeorgeWashington University.

Johnson, S.D., & Chung, S.P. (1999). The effect of think-ing aloud pair problem solving (TAPPS) on the

troubleshooting ability of aviation technician stu-dents. Journal of Industrial Teacher Education, 37(1), 7-25.

Johnston, V.L. (1996), Toward a global classroom usingcomputer-mediated communications at UAA.University of Alaska Anchorage, Vocational TeacherEducation Research Report, (Eric document #ED356759)

Jonassen, D.H. (2000). Toward a design theory of prob-lem solving. Educational Technology, Research andDevelopment, 48(4), 63-85.

Jonassen, D., Prevish, T., Christy, D., & Stavrulaki, E.(1999). Learning to solve problems on the Web: Ag-gregate planning in a business management course.Distance Education, 20(1), 49-63.

Kinzie, M.B., Hrabe, M.B., & Larsen, V.A. (1998). Aninstructional design case event: Exploring issues inprofessional practice. Educational Technology Re-search and Development, 46(1), 53-71.

Klein, J.D., & Pridemore, D.R. (1994). Effects of orient-ing activities and practice on achievement, continu-ing motivation, and student behaviors in acooperative learning environment. Educational Tech-nology Research and Development. 42(4), 41-54.

Kruger, L.J., & Cohen, S. (1996). Using the Internet toextend training in team problem solving. BehaviorResearch Methods, Instruments and Computers, 28(2),248-253.

Laffey, J., Tupper, T., Musser, D., & Wedman, J. (1998).A computer-mediated support system for project-based learning. Educational Technology Research andDevelopment, 46(1): 73-86.

Levin, B.B. (1995). Using the case method in teachereducation: The role of discussion and experience inteachers thinking about cases. Teaching and TeacherEducation, 11(1), 63-79.

Lou, Y., Abrami, P.C., & d'Apollonia, S. (2001). Smallgroup and individual learning with technology: Ameta-analysis. Review of Educational Research, 71(3),449-521.

Mclsaac, M.S., & Ralston, K.D. (1996). Teaching at adistance using computer conferencing. Tech Trends,41(6),48-53.

Mergendoller, J.R., Bellisimo, Y., & Maxwell, N.L.(2000). Comparing problem-based learning andtraditional instruction in high school economics.Journal of Educational Research, 93(6), 374-383.

Moore, M.G., & Kearsley, G. (1996). Distance education:A systems view. New York: Wadsworth PublishingCompany.

Murphy, K.L., & Collins, M.P. (1997). Development ofcommunication conventions in instructionalelectronic chats. Journal of Distance Education, 12(1),177-200.

Naidu, S., & Oliver, M. (1999). Critical incident-basedcomputer supported collaborative learning. Instruc-tional Science, 27(5),329-354.

Phye, G.D. (2001). Problem-solving instruction andproblem-solving transfer: The correspondence issue.Journal of Educational Psychology, 93(3),571-578.

ETR&D, Vol. 51. No. I

19COMPUTER-MEDIATED COMMUNICATIONS IN PROBLEM SOLVING

Savenye, W.C. (2001). Evaluating Web-based learningsystems and software. Paper presented at the Con-ference on Instructional Design: Addressing theChallenges of Learnig Through Technology andCurriculum, Albert-Ludwigs Universitat, Freiburgim Breisgau, Germany, May 2001.

Sherman, G.P., & Klein, J.D. (1995). The effects of cuedinteraction and ability grouping during cooperativecomputer-based science instruction. EducationalTechnology Research and Development, 43(4), 5-24.

Shulman, J. (Ed.). (1992), Case method in teacher educa-

tion. New York: Teachers College Press.Simonson, M., Smaldino, S., Albright, M., & Zvacek, S.

(2000). Teaching and learning at a distance: Foundationsof distance education. Upper Saddle River, N.J.: Mer-rill an Imprint of Prentice Hall.

Slavin, R. (1993). Ability grouping in the middlegrades: Achievement effects and alternatives.Elementary School Journal, 93(5), 535-552.

Stepien, W., & Gallagher, S. (1993). Problem-basedlearnig: As authentic as it gets. Educational Leader-ship, 50(7), 25-28.

COPYRIGHT INFORMATION

TITLE: The Effect of Computer-Mediated Collaborative Learningon Solving Ill-Defined Problems

SOURCE: Educ Technol Res Dev 51 no1 2003WN: 0300103447001

The magazine publisher is the copyright holder of this article and itis reproduced with permission. Further reproduction of this article inviolation of the copyright is prohibited.

Copyright 1982-2003 The H.W. Wilson Company. All rights reserved.

The Effect of Computer-Mediated Collaborative Learning on...

Documents

Transcript of The Effect of Computer-Mediated Collaborative Learning on...