Running Head: ARTICLE CRITIQUE #2 1

Article Critique #2

Kendall Brown

Seattle Pacific University

EDU 6980 Applying Action Research in School Settings

ARTICLE CRITIQUE #2 2

Abstract

This study looked into the effectiveness of Problem-Based Learning (PBL) and

traditional instruction on high school seniors’ learning of macroeconomics. Within the

four high schools that were a part of this study, five veteran teachers taught eleven classes

of high school seniors, using PBL and traditional instruction, in order to determine the

impact of these two different instructional approaches on student learning. Different

student aptitudes of verbal ability, interest in economics, preference for group work, and

problem-solving efficacy were taken into account in order to determine the impact of

these two instructional approaches in teaching macroeconomics. Specifically, researchers

wanted to determine if students with preference for group work and high problem-solving

efficacy would learn more in PBL environments than students with lower group work

preference and problem-solving self-efficacy. The results of this study indicated that PBL

was more effective in high school seniors’ learning of macroeconomics, specifically for

students with average to low verbal ability, an interest in economics, and for students

with high and low problem-solving efficacy.

Keywords: Problem-Based Learning (PBL), traditional instruction, aptitudes, Simple Random Sample, Quasi-Experimental

ARTICLE CRITIQUE #2 3

Article Critique #2

Introduction:

In the study, the author defines Problem-Based Learning (PBL) as “a powerful and

engaging learning strategy that leads to sustained and transferrable learning,” providing

opportunities for collaborative, active, and self-directed activities that increase the likelihood that

students will retain and apply their learning to other situations (Mergendoller, 2006, p. 49).

When using the traditional instruction approach, students are involved in a much different

learning environment focused primarily on listening to lectures with little to no group work,

problem-solving, and critical thinking. With these two instructional approaches, researchers

wanted to find which one was more impactful on high school seniors’ learning of

macroeconomics, given students’ varying aptitude levels of verbal ability, preference for group

work, interest in economics, and problem-solving self-efficacy. The seniors who differed in their

aptitude levels were pre-assigned to the eleven classes studied. However, the five veteran

teachers did not know the type of learners they would be teaching in PBL and traditional lecture

environments, therefore allowing for manipulation of the groups and findings eluding to the

effectiveness of PBL vs. traditional instruction on student learning.

Purpose & Goals:

As mentioned earlier, the purpose of this study was to specifically discover if students

with preference for group work and problem-solving efficacy would learn more in PBL learning

environments than students who did not prefer to work in groups and had low self-efficacy in

problem-solving situations. By studying eleven classes of high school seniors, ranging in their

aptitude levels and receiving either PBL or traditional instruction, the goal of researchers was to

discover if PBL was in fact a more powerful teaching strategy on student learning.

ARTICLE CRITIQUE #2 4

Hypotheses:

Using pre-existing studies on PBL’s impact on student learning (discussed further in the

next section), researchers in this study had the following three hypotheses: 1) There is no

difference in achievement, as measured via pretest-posttest changes in macroeconomics

knowledge, between students in PBL and traditional instructional environments 2) There is no

difference in achievement, as measured via pretest-posttest changes in macroeconomics

knowledge, between students with different levels of verbal ability in PBL and traditional classes

3) There is no difference in achievement, as measured via pretest-posttest changes in

macroeconomics knowledge, among students with different levels of interest in learning

economics, preference for group work, or problem-solving efficacy (Mergendoller, 2006, p. 53).

Theoretical Constructs:

In coming up with these three hypotheses, researchers in this study referred to pre-

existing findings on the impact of PBL on student learning. The majority of these findings came

from empirical studies dealing with college students in the medical field, in which theorists

claimed PBL to be powerful and engaging for these types of learners. However, given the fact

that the majority of PBL research had been done with older, medical students versus high school

students, there was little evidence going into this study that supported PBL’s impact on this

study’s age group of students. In other words, these studies involving medical students and PBL

are difficult to generalize to the high school population. Since studies of PBL and college-aged

medical students were the only driving force behind this study, researchers should have looked

more into possible research done with PBL and elementary, middle, or high school-aged

students. This would have strengthened the theoretical orientations for their study.

ARTICLE CRITIQUE #2 5

Research Design:

The type of research design used in this study was Quasi-Experimental, in which “the

groups being compared are not assumed to be equivalent at the beginning of the study” (Ravid,

2011, p. 13). The fact that seniors with pre-existing differences in aptitude levels were assigned

to their classes ahead of time, there were no random participants in this study. However, there

would be manipulation to groups based on which classes were taught with PBL vs. traditional

instruction, therefore allowing for confounding variables. In this study, although the 246 high

school seniors were not randomly placed in the eleven classes, the five teachers had no

knowledge of the types of learners they would be teaching. Teachers simply chose their class

periods, as well as which class would be taught using the PBL or traditional instruction

approach. Therefore, the design of this study does match its purpose in determining if students

with varying aptitude levels (specifically preference for group work and problem-solving

efficacy), would learn more macroeconomics in a PBL environment than those with low

preference for group work and problem-solving efficacy.

Sampling Method:

Given this research design with teachers not knowing their students ahead of time, this

study’s sampling method can be seen as Simple Random Sample, with “every member of the

population [having] an equal and independent chance of being selected for inclusion in the

sample” (Ravid, 2011, p. 26). Given this sampling method with teachers randomly picking a

period and not the students, this study can be seen as probability sampling. Given that teachers

did not know the pre-existing aptitude differences of their students, only which periods would be

taught using either PBL or traditional instruction, this sampling method can be seen as

ARTICLE CRITIQUE #2 6

appropriate for this study, with its efforts in determining the effectives of PBL and traditional

instruction on student learning.

Sample Bias:

Even though this sampling method is appropriate for this study, there are areas of

possible bias and threats to internal validity in regards to the students involved. One such threat

involves not knowing the exact numbers of seniors from each of the four high schools in the

study. All we were told is that there was a total of 246 participants in eleven classes, all coming

from schools located in a large metropolitan area in northern California. Two schools were from

a suburban area, and the other two from an urban area. We were also not informed of how many

males or females were involved, their diversity, ethnic background or socioeconomic status, all

of which APA format requires. Finally, even though this sample size was larger than 30, findings

from this study cannot be generalized to the population given the lack of information provided

on the students. The authors themselves also made no claim of this study being generalizable.

Given the lack of demographic information provided in this study, I feel these findings cannot be

generalized to the elementary or high school populations in the area where I teach. Our school

community is rich in diversity, has varying levels of special learning needs, and middle to upper

socioeconomic status. Until my colleagues and I can understand more about the seniors that

participated in this study, we cannot generalize findings of PBL on student learning to our

student populations.

Variables:

The following are the independent and dependent variables found in this study, supported

by the researchers’ three hypotheses mentioned earlier. Based on the first hypothesis presented

by researchers, teaching different groups of students using either PBL or traditional instruction in

ARTICLE CRITIQUE #2 7

this study (independent variables) allowed researchers to discover any manipulation or change in

students’ knowledge of macroeconomics (dependent variable). Referring to researchers’ second

hypothesis, the difference in students’ verbal ability levels in the PBL and traditional instruction

classes (independent variable) allowed researchers to also see how this student aptitude may

have impacted students’ macroeconomics scores (dependent variable). Finally, researchers’ third

hypothesis focused on determining the effect of additional aptitudes of interest in economics,

preference for group work, and problem-solving efficacy, (independent variable), on students’

macroeconomics scores (dependent variable).

Reliability & Validity:

The measurements taken in this study included: students’ verbal ability levels, interest in

economics, preference for group work, problem-solving efficacy, and macroeconomics

knowledge. Even though the more measurements used in a study the more reliable and valid the

study becomes, the researchers themselves created all of the instruments used in this study in

order to realize student aptitude levels and their learning of macroeconomics. The fact that the

instruments used in the study were created by the researchers themselves creates room for error,

or unreliability, as well as the likelihood that the instruments were not measuring what they were

supposed to be measuring, or invalidity. The more a researcher stays out of the instruments used

in a study, the better. According to Ravid, “self-made instruments…tend to have lower reliability

levels than tests prepared by commercial companies or by professional test writers” (Ravid,

2011, p. 199).

When looking at each individual measurement and instrument used in this study in an

effort to determine the reliability and validity, it is important to keep in mind how each

instrument used in this study was created by the researchers. In measuring students’ verbal

ARTICLE CRITIQUE #2 8

ability levels, researchers used a Quick-Word Test, in which students were asked to circle a

synonym for a target word in capital letters, followed by four lower-case words. This instrument

resulted in correlations greater than .80, which shows high reliability. In measuring students’

interest in economics, a survey was used. Since researchers claimed that there wasn’t a survey

that already existed in understanding high school students’ interest in economics, they created

one themselves. In an effort to measure students’ interest in economics, researchers created the

stem “How interested are you in reading newspaper and/or magazine articles about…” which

was followed by four topics such as “economic issues faced by the poor” (Mergendoller, 2006, p.

55-56). After students responded using a scale ranging from 1 (very interested) to 5 (not

interested), the mean was taken, and the Cronbach’s alpha was .80, high reliability. Since the

purpose of the study was to measure the impact of Problem-Based Learning (PBL) versus

traditional lecture on students’ learning of macroeconomics, the questions on this survey didn’t

fully measure what the study was intended to measure. Furthermore, how students involved in

PBL (group-oriented learning) might answer questions regarding economic issues facing the

poor might differ from how students involved in traditional lecture would respond. Moving onto

how preference for group work was measured, researchers created another survey. This survey

used the stem “When I work with my classmates in small groups, I usually find that…” and was

followed by items such as “It does not help me learn” (Mergendoller, 2011, p. 56). In my

opinion, these questions used on the survey measured what it was intended to measure, seeing

that the purpose of the study was to determine the effect of PBL (collaborative learning) and

traditional lecture on students’ learning of macroeconomics. The Cronbach’s alpha for this

survey resulted in .79, high reliability. In measuring students’ problem-solving efficacy, a survey

was used with the stem “I have difficulty solving problems when…” followed by items such as

ARTICLE CRITIQUE #2 9

“I have to argue my own point of view” (Mergendoller, 2011, p. 56). After responding on a scale

ranging from 1 (strongly agree) to 5 (strongly disagree), Cronbach’s alpha was .82, high

reliability. I feel this survey measured what it was intended to measure, yet it would have been

interesting to add a stem of “I am successful solving problems when…” versus solely asking

students when they experience difficulty. Finally, in measuring students’ macroeconomics

learning, the instrument used was a unit-specific content test using 16 four-part, multiple choice

items taken from the Test of Economic Literacy and a high school economics textbook. These

questions covered the main concepts and topics students would be studying in the two week unit.

It was interesting to find that the reliability of this measurement was not taken; there is no

Cronbach’s alpha data to report. Furthermore, the resources used to create the test items are quite

dated, (mostly from the 1950’s), which adds onto the unreliability of this instrument. Another

thought I had in regard to this instrument has to do with the questions asked. The questions dealt

with the concepts and ideas that would be covered in the unit. I feel it would’ve been interesting

to see the results of the interest in economics survey if the concepts and ideas taught in the two-

week unit were asked here. This might have helped the validity of the study in understanding

students’ aptitude levels with their learning of macroeconomics.

As mentioned earlier, all of the instruments used in this study were created by the

researchers themselves. This again calls into question the reliability and validity of each measure

taken. With this being said, the measurements taken that resulted in Cronbach’s alpha data

showing high reliability needs to be called into question. Construct validity works to measure a

trait or characteristic that can’t be measured directly, such as the study’s measurements of

interest in economics, problem-solving efficacy, and preference for group work. According to

Ravid, “Evidence for construct validity is not gathered just once for one sample; rather, it is

ARTICLE CRITIQUE #2 10

collected with the use of many samples and multiple sources of data” (Ravid, 2011, p. 206).

Instruments to measure students’ interest in economics, problem-solving efficacy, and preference

for group work measurements were created by researchers because there was a lack of surveys

available, or a lack of other sources of data and supporting evidence. These measurements were

also taken once, solely for this study, which adds further to the lack of reliability and validity in

this study. In the article’s “Future Research” section, the researchers themselves even claimed

that they should not be basing their results off of one test in a two week unit because it does not

accurately measure what they are trying to measure. Based on these findings, it cannot be firmly

concluded that this study exhibits strong reliability and validity.

Analysis:

Data analysis requires a complete set of descriptive statistics, accurate measurements and

instruments used in a study. Since it has already been established that the instruments and

measurements used in this study are not reliable or valid, the data collected in this study can also

be called into question. According to Ravid, “If the instruments used to collect data lack in

reliability or validity, any conclusions or generalizations based on the results obtained using

these instruments are going to be questionable” (Ravid, 2011, p. 42). In the article, the authors

mention how the data showed a normal distribution with no outliers. However, the article does

not show any histogram or graph for that matter. Some descriptive statistics were provided in

three tables, yet necessary data was missing. Two tables show the mean results, standard

deviation and t scores of student aptitude levels, and another shows changes in pretest-posttest,

standard deviation, t values, and effect size in correlation with the student aptitudes. There is no

mention of skewness or kurtosis, which is needed in order to support the researchers’ claim that

the data from this study created a normal distribution without any outliers. If a normal

ARTICLE CRITIQUE #2 11

distribution did occur, we also need to know if any manipulation was done to the data by the

researchers (log or sine of values, removal of outliers, etc.) There simply is not enough data

provided from this study to draw these conclusions, support the authors’ statement about how

their assumptions were met, and how PBL is the more effective approach in teaching

macroeconomics to students.

Conclusion:

The following are the authors’ main conclusions of their study, regarding their three

hypotheses. In their first hypothesis, researchers felt that PBL would not be more effective than

the traditional approach in helping student learn basic macroeconomics concepts; this first

hypothesis ended up being rejected by researchers. Based on their results, PBL did prove to be

more effective than traditional lecture in students’ learning of macroeconomics. PBL gains were

higher for 4 out of the 5 teachers involved. As for the second hypothesis, researchers felt there

would be no difference between pre and post-tests of macroeconomics knowledge between

students with different levels of verbal ability in PBL and traditional classes; they were correct.

Finally, their third hypothesis was that there would be no difference between pre and post-tests

of macroeconomics knowledge between students with different levels of interest, preference in

group work, and problem-solving efficacy; they were wrong and instead found that there was a

difference in macroeconomics pre and post-tests for students with varying aptitude levels.

Keeping in mind the researchers’ main conclusions regarding their three hypotheses, the

following are some further conclusions drawn from the researchers in the “Discussion” section

of the article. Specifically speaking to the first hypothesis, which concluded that PBL was more

effective than traditional lecture in students’ learning of macroeconomics, the authors stated how

“Across all teachers, PBL classes gained .66 more than the traditional classes. This is equivalent

ARTICLE CRITIQUE #2 12

to a raw score of 4% - or the difference between a B and a B+” (Mergendoller, 2006, p. 62).

With this conclusion, more information is needed in understanding how they converted the

scores to a raw score; this would support the researchers’ main conclusions regarding the

positive impact of PBL on students’ learning of macroeconomics. Further down in this same

discussion section of the article, the authors go on to discuss interesting findings dealing with

Teacher A and C. In short, both teachers were more effective using different approaches;

Teacher A was more effective in teaching economics using the PBL approach, while Teacher C

was more effective when using the traditional approach. When looking at the chart on page 60,

the t-scores for both Teachers A and C in terms of verbal ability scores were significant. With

these conclusions, I agree with the authors when they recommend looking further into what

classroom conditions and social arrangements are necessary to maximize results when using

either PBL or the traditional approach (2006, p. 62). Finally, looking at the table on page 61,

other interesting conclusions resulted in regards to the impact of PBL on students’ learning of

macroeconomics. In this chart, the only significance was for the 86 students who had a high

interest in learning economics, resulting in a t score of 2.21*. Although this data supports

researchers’ conclusions regarding their third hypothesis, I again agree with the authors in how

further research is needed to support this study’s findings. In the next section of this conclusion, I

discuss further limitations I found in the study, as well as other recommendations for future

studies dealing with the effectiveness of PBL vs. traditional learning.

Although the article referenced a total of 346 seniors in the eleven classes, it mentioned

how only 246 completed both the pre and post macroeconomics knowledge instrument and the

verbal ability measure. Of these 246 seniors, “some of these students did not complete one or

more of the aptitude assessments” (Mergendoller, 2006, p. 54). The author goes on to then say

ARTICLE CRITIQUE #2 13

how they substituted the population mean for the missing score. This vague and incomplete

collection of data creates bias and threats to reliability and validity in the results of this study.

Looking specifically at the manipulation used on certain groups of students, I found areas of bias

in how PBL was set-up in the learning environments. Authors explained how those students

involved in PBL went through a one, two week unit. Towards the end of the article in the “Future

Research” section, authors state how “If PBL is to help students develop deep, applicable

knowledge and analysis skills, it is likely that students will need to solve multiple problems over

the course of a semester or school year” (Mergendoller, 2006, p. 64). To determine the true

impact of PBL on student learning, I agree with the authors that this study should have been

longer than two weeks.

The following are some limitations I found in regards to the design of the study,

specifically the teachers involved, as well as what additional information would have been

helpful in further understanding the study. As mentioned earlier, five veteran teachers from four

high schools taught the PBL and traditional instruction classes, but what, according to the

researchers, defines “veteran”? I recommend that in future studies, researchers define

terminology such as “veteran teacher.” Knowing the definition of not only “PBL” but

“traditional lecture” would also help in understanding this study further. Going off of the five

“veteran” teachers involved, a limitation in this study also involved not knowing exactly who

these teachers were and how they were chosen. Knowing how teachers were chosen for this type

of study would have been helpful. Furthermore, having five teachers in this study was too many,

and for future studies, I recommend using fewer than 5. In discussing preparation for this study,

the author goes on to state how “all teachers had attended at least one week-long training

workshop…to prepare them to use the PBL unit in their classes” (Mergendoller, 2006, p. 58). In

ARTICLE CRITIQUE #2 14

using PBL, teachers were also provided with curriculum guides, units, tips and strategies. It

would have been helpful to know if additional material and training was also provided when

using the traditional approach, as well as which teachers received this PBL training, seeing how

two of the five teachers involved in the study went on to be instructors. In addition to the

research design, the article mentioned how of the five teachers, 4 out of the 5 had one PBL and

one traditional class to teach, while “Teacher A” had two PBL classes and one traditional. It

would have been helpful in understanding how or why “Teacher A” was selected to teach two

PBL classes, as well as why out of the 246 total students, 139 students participated in PBL

classes and only 107 students participated in traditional instruction classes. In regards to

the students involved in this study, the following are certain limitations that I found, as well as

recommendations for the future. I agree with the authors in their discussion of future research

when they say how it would be beneficial to know what exactly students were doing in the PBL

environment. Authors refer to this study as a “black box study,” in which there was no record of

student interactions, which highlights this need for knowing if there was any kind of

accountability in these two learning environments. When students were working in PBL

situations, students had to use their own judgments as they worked together to solve a problem

with more than one solution. A lack of accountability might have resulted with certain students

taking on more of a leadership role while others took a back seat in solving these problems as a

group. The fact that some students prefer to work alone, versus with others in a group, might

have also been a threat to the internal validity of the study involving PBL, and I recommend

looking into this question for future studies. Finally, towards the end of the PBL unit, added

pressure resulted from students needing to present their findings, which also effects the overall

ARTICLE CRITIQUE #2 15

validity of the results. Based on these limitations, I recommend looking further into whether or

not PBL is truly more effective in students’ learning of macroeconomics.

Not only were students not held accountable, but teachers weren’t either; this is a key

limitation of this study. In order to support the findings that PBL environments are more

effective for students learning economics, we also need to know exactly what the teachers were

doing in both the PBL and traditional lecture classrooms. By doing so, we can understand key

differences that may exist between what teachers are doing in the PBL environment versus the

traditional, and how these difference attribute to gains in students’ knowledge of

macroeconomics. For future studies, this additional information would strengthen the overall

findings involving the effectiveness of PBL and traditional instruction on student learning.

ARTICLE CRITIQUE #2 16

References:

Mergendoller, J. R., Maxwell, N.L., & Bellisimo, Y. (2006). The Effectiveness of Problem-Based Instruction: A Comparative Study of Instructional Methods and Student Characteristics. Interdisciplinary Journal of Problem-Based Learning, 1(2). Available at: http://dx.doi.org/10.7771/1541-5015.1026

Ravid, Ruth. Practical Statistics for Educators. 4th ed. Lanham: Rowman & Littlefield, 2011. Print.

ARTICLE CRITIQUE #2 17