Integrating Alternative Assessment in a Project-Based Learning Course

for Pre-service Science and Technology Teachers .

MOTI FRANK, Technion – Israel Institute of Technology

ABIGAIL BARZILAI, Technion – Israel Institute of Technology

Paper presented at the Learning Communities and Assessment Cultures Conference organised by the EARLI Special Interest Group on Assessment and Evaluation,

University of Northumbria, 28-30 August 2002

ABSTRACT This paper deals with a required methods course, based on the

national curriculum of science and technology for junior high schools. The course

participants are pre-service teachers who study towards a B.Sc. degree in Education

in Science and Technology parallel to their studies in one of the faculties of Sciences

or Engineering. Working in small teams the students carry out a project in a Project-

Based Learning (PBL) environment. The final outcomes of the project are group and

individual written reports, a portfolio, a multimedia presentation and a physical

model. The research question was: what implementation issues and processes do

higher education students encounter in a Project-Based Learning environment which

involves an alternative assessment approach? Qualitative and quantitative tools for

collecting data included ‘the participant as observer’ observations at the classroom,

semi-structured interviews with students, questionnaire, and analysis of students’

reports and products. The findings relate to the benefits of PBL from the students’

point of view, the challenges students experienced and perceived while conducting

their projects, the benefits students may gain from formative assessment, and the

difficulties located by the course teachers throughout the assessment meetings.

Introduction

A required methods course, based on the national curriculum of science and

technology in junior high schools, has been developed in the Department of Education

in Technology and Science. The course participants are pre-service teachers who

study towards a B.Sc. degree in Education in Science and Technology parallel to their

studies in one of the faculties of Sciences or Engineering. The learning environment

of the course is based on a constructivist approach, and designed to enable the

students to construct their knowledge through active learning, and through interaction

with staff and colleagues. The teachers act as facilitators and mentors. The assessment

in the course is also derived from the constructivist approach and is based on

formative assessment of students’ performance in microteaching, active learning in

the National Museum of Science and Technology, group assignments, and

interdisciplinary team projects.

This paper focuses on an approach that is implemented to assess the students

while carrying out the project in a Project-Based Learning (PBL) environment.

Students are required to design a system based on scientific, technological, social and

environmental principles. To emphasize technological, and not merely scientific

literacy, the unique quality of the project is that the starting point is the technological

requirements and needs rather than a research question. The students first define what

would be required of the technological system, including the tasks to be performed,

and determination of the system outcomes. They investigate alternatives for

implementation, collect and analyze data through a process of investigation and

collaboration, and conduct a feasibility study, after which they designed the system,

using a top-down approach.

During the course students learn how to use project-based learning in their

classrooms. The course covers the theoretical background of PBL and some field

research examples which describe the use of PBL as well as the strategy for

conducting a student-project: where to begin, how to coach students through a project,

and how to assess student products.

The final outcomes of the project are group and individual written reports, a

portfolio, a multimedia presentation in classroom in front of the course colleagues and

staff and a physical artifact, which can assist the secondary school teacher while

demonstrating a scientific and/or technological principle underlying the system.

We believe that alternative assessment should take into consideration intangible

parts of the project and not emphasize the final products or presentations only. The

real learning is often in the doing or in the process leading up to the product. Students

presenting a product that fails to work may still have learned the central principles,

especially if they can describe why their product does not work and what is needed to

correct it (Buck, 1999). Therefore, in our course, the project grade is based on the

assessment of tangible (about one third out of the project grade) and intangible

outcomes .

Following are examples of projects performed by students: a car driven by solar

energy, a water desalination system, a remote cardiologic testing system, an

automated watering system, a hot air balloon system, and an automated purification

system for aquarium water.

Project-Based Learning

What is the project-based learning approach? According to Krajcik, Czerniak and

Berger (1999), this approach engages learners in exploring important and meaningful

questions through a process of investigation and collaboration. Students ask questions,

make predictions, design investigations, collect and analyze data, use technology,

make products, and share ideas. According to Buck Institute of Education (1999),

PBL is an innovative model for teaching and learning. It focuses on the central

concepts and principles of a discipline, involves students in problem-solving

investigations and other meaningful tasks, allows students to work autonomously to

construct their own knowledge, and culminates in realistic products .

Thomas (2000) also tries to define this approach, and emphasizes that in the PBL

environment students are, in fact, investigating solutions to a problem. They build

their own knowledge by active learning, interacting with the environment as

suggested by the constructivist approach, working independently or collaborating in

teams, while the teacher directs and guides, and they make a real product.

We shall now discuss the advantages of the PBL approach for teaching and/or

learning, firstly from the students’ point of view, and then as concerns the instructor.

Green (1998) notes that learning by means of a project is likely to increase motivation

and give the students a sense of satisfaction. It is also helps to develop long-term

learning skills. Krajcik, Czerniak and Berger (1999) suggest four benefits for the

student. Firstly, learners develop deep, integrated understanding of content and

process. Secondly, students learn to work together to solve problems. Collaboration

involves sharing ideas to find answers to questions. In order to succeed in the real

world, students need to know how to work with people from different backgrounds.

Thirdly, this approach promotes responsibility and independent learning. As a final

benefit, this approach actively engages students in various types of tasks, thereby

meeting the learning needs of many different students.

Orevi and Danon (1999) also list the advantages of PBL from the students’ point

of view - it develops collecting and presenting data skills, develops thinking skills,

suits personal learning styles, enhances motivation, and develops independent

learners.

So far we have discussed the advantages for the student. Krajcik, Czerniak and

Berger (1999) also suggest three possible advantages for the teacher. Firstly, the

teacher may find the work enjoyable, interesting and motivating, since teaching will

vary every year as he/she will be exploring new projects with each new group of

students. Secondly, in project-based teaching, the teacher continually receives new

ideas, thus becoming a ‘lifelong learner’. Thirdly, classroom management is

simplified because when students are interested and involved, they are likely to cause

fewer disciplinary problems.

What has research discovered about the usefulness of the PBL approach? In a

research described by Shepherd (1998), it was found that students who applied the

PBL approach statistically received significantly higher grades on the Critical

Thinking Test than did those in the comparative group who had studied in the

traditional manner. The PBL students also demonstrated greater self-confidence and

improved learning abilities.

Interestingly, Rosenfeld and Rosenfeld (1999) found that students with low

academic records who studied in the conventional framework did better in courses

based on PBL, whereas those with higher grades in regular studies had lower

achievement when PBL methods were applied. Based on their findings, the

researchers suggest that teaching styles and learning environments be adapted to the

student’s learning mode. Low academic grades do not necessarily demonstrate lack of

ability, but rather the unsuitability of the pedagogic system. They advise that students

be exposed to PBL in order to give those who fail a chance of doing better, and to

encourage those with high academic achievements in subjects taught traditionally to

develop additional expertise.

The teaching role in PBL is changed. The teacher is no longer the expert lecturer,

facts provider, and director of instruction but rather a resource provider, learning

environment shaper, how-to-learn teacher, advisor, tutor and colleague (Buck, 1999).

Teaching by means of PBL presents several challenges. These include: teachers’

content knowledge, students’ lack of experience in this new approach and their

preference for traditional-structured approach; their preference for learning

environment which require less effort on their part; and problems arising from time

stress. Students struggling with ambiguity, complexity, and unpredictability and are

liable to sense frustration in an environment of uncertainty, where they have no notion

of how to begin or in which manner to proceed.

The system is rather time-consuming and requires that the teacher make a lot of

efforts over a long period of time. Certain teachers may feel that they don’t have

sufficient time to meet the curriculum guidelines. Class management, in which the

students have the freedom to talk together, is often more difficult.

Teachers often feel a need to direct lessons to insure that students get the “right”

information. Teachers frequently give students too much independence without

structuring the situation, or providing feedback. Sometimes, teachers have difficulty

incorporating technology into the classroom. Other problems are discussed by

Krajcik, Czerniak and Berger (1999, pp. 322-328) and by Buck Institute of Education

(1999, Potential Problems section).

To sum up, we will present the main advantages of the PBL method (PBL that

based on identifying needs and designing system to meet that needs rather than PBL

that based on answering research question). Students are engaged in active learning

and gain interdisciplinary knowledge while working in a real-world context. Students

develop deep and integrated understanding of content and process and focus in depth

on central ideas and salient issues. Since students deal with relevant issues, their

motivation is increased. PBL promotes responsibility and independent learning, offers

multiple ways for students to participate and to demonstrate their knowledge, and can

be matched to the various learning styles of students.

Through PBL the students are familiarized with the synthesis processes (not just

the analysis processes), experiment in teamwork, become familiar with the design

procedures and the underlying principles of systems engineering. The students learn

to apply the important engineering principle - how to design to an optimum (not

necessarily to the maximum).

The learning environment promotes the development of inquiry skills, problem

solving skills, and information skills. Students develop a capacity for systems

thinking, acquire lab experience, and gain a higher level of cognitive skills (such as

asking questions) and affective outcomes (such as curiosity and skepticism). Finally,

the awareness of students to Scientific-Technological-Environmental-Social aspects is

increased, academic achievement may be improved and students learn to simulate the

professional work of the scholar, researcher, engineer, reporter, planner, manager, and

other practitioners.

Alternative Assessment

The traditional assessment in most courses of the higher education is mainly based

on paper and pencil tests. The grade in a given subject is mainly based on a final test.

Usually these tests measure how well the student can repeat the declarative

knowledge transferred from the teacher to the passive student. In our course the

traditional assessment methods were found less appropriate for measuring the kind of

understanding the students acquired in PBL.

According to Krajcik, Czerniak and Berger (1999, p. 207), traditional questions

would fail to assess the multimode of ideas students learned. They would not measure

students’ ability to work as a team. They would not show how students could apply

their knowledge and skills to everyday life. They would not show that they could

design and plan investigations .

Therefore we decided to apply an alternative assessment in our course.

Assessment of students’ performance in the project was based on meetings and

discussions between the course staff and each team during the semester, observations

of the students’ work in classroom, group report and portfolio, personal reflective

reports, and an exhibition at the end of the course - posters, multimedia presentations,

and the artifacts. A rubric was also developed and validated for this assessment.

The following table presents the characteristics of alternative assessment (Buck,

1999; Krajcik, Czerniak and Berger, 1999) and the assessment strategies implemented

by us in the course:

TABLE 1. The characteristics of alternative assessment, and the strategy of the

assessment implemented in the course.

No. Alternative assessment characteristics Assessment strategies implemented in the course

1. The assessment is formative. It relates to the learning processes not just to the final result as a

continuous process embedded in instruction.

Throughout the course students were given feedbacks regarding their progress as an integral part of the course. The criteria for assessment

were given to the students in advance.2 It encourages teachers to become more

reflective practitioners. It helps them justify the grades, plan instruction, and identify the

difficulties and misconceptions.

The course design was dynamic. The instruction has been altered a number of times according to the progress in practice. The assessment criteria were changed following interaction with the

students.3 It is based on a number of diverse information

sources, not just on tests.The assessment was based on various forms, not just on a final test (see details in the Method

section).4 Assessing of knowledge, understanding, skills,

attitudes, values, motivation, higher-level cognitive skills and affective outcomes in a real

life context .

Assessing processes and products enabled to assess diverse cognitive and affective abilities.

5 It indicates that the student is responsible for the learning process. Engaging students in the assessment process helps them to be self-

reflective and self-regulated learners .

To help students monitor their progress they were required to report their progress and plans, keep journals and submit a reflective report at the end

of the course.

6The focus is on student improvement rather than on comparison with others. It may reduce the

level of stress and anxieties.

The assessment was based on the progress and outcomes of each team rather than on a final

standardized test .7 Teachers come to know their students better. Regular meetings with course staff were held in

The teacher can help each student according to individual needs.

order to identify difficulties and offer individual assistance.

8 The assessment is qualitative and verbal and not just quantitative.

An analytical rubric was used. Students were given verbal comments .

In light of the above characteristics, Krajcik, Czerniak and Berger (1999) claim

that alternative assessment is more valid and reliable, more closely matches today’s

educational goals, better accommodates cultural diversity, is consistent with cognitive

learning theory, and measures deep understanding.

Method

Research Design

This paper deals with a required methods course, based on the national curriculum

of science and technology for junior high schools. The research question was: what

implementation issues and processes do higher education students encounter in a PBL

environment which involves an alternative assessment approach? The purpose of the

current study was to learn about ideas (cognitive aspects), emotions (affective

aspects), difficulties, and behavior (behavioral aspect) of the students in the above

course.

The study is based, mainly, on qualitative data analysis. The merit of a qualitative

study lies in its internal validity, not in its generalization. Therefore, we do not

recommend making generalizations on the basis of the case study described here.

However, the findings and implementation tips presented in this paper may serve as a

basis for a follow up empirical/quantitative study based on a random sample and

inferential data analysis .

The strategies adopted in the study served to reinforce reliability, inner validity,

and external validity. This was carried out by cross-referencing sources and

triangulation. Denzin (1988) outlines four levels of triangulation. The first is data

triangulation that includes time, space and persons. The second level is investigator

triangulation - using several observers. The third is methodological triangulation -

using more than one research method, and the fourth level is theory triangulation -

creating a broad theoretical framework .

The data analysis strategy used was ‘content analysis’: definition of analysis units,

setting categories (outstanding repeated elements), and examination of frequencies.

The study was conducted by two researchers. Qualitative and quantitative tools for

collecting data included ‘the participant as observer’ observations (Adler & Adler,

1994) at the classroom, semi-structured interviews (Spradley, 1979; Mishler, 1986)

with students, questionnaire, and analysis of students’ reports and products. Data was

collected at different times and stages during the course.

The Subjects

The course participants were pre-service teachers who studied towards a B.Sc.

degree in the department of Education in Science and Technology parallel to their

studies in one of the faculties of Sciences or Engineering. The duration of the course

is a fourteen-week semester. Every weekly class included a one-hour lecture, two

hours microteaching and three hours team project.

Multiple Forms of Assessment – Tools for Assessing Processes and Products

The principle of triangulation was applied also to the assessment process. Multiple

forms of assessment were used - multiple formats, multiple units, and multiple

assessors (Gonella, 2001). Multiple formats involve group reports, portfolios,

observations, media product, and a physical model. Multiple units refer to self-

assessment, group (peer) assessment, and whole class assessment. Multiple assessors

were involved – two lecturers, the TA, and the students .

The project grade was 55% out of the final grade of the course: 10% - physical

model, 5% - Power Point presentation, 10% - meetings with the course staff, 20% -

group report, and 10% - personal (reflective) report. Several rubrics were developed

for assessing the above outcomes: analytical rubric for assessing the group report, and

holistic rubrics (Birenbaum, 1997; CPS, 2000) for assessing the personal report, the

PPt presentation, the physical model, and the meetings’ documentation. Using the

rubrics enabled the instructors and the students to monitor the students’ progress and

guided them through the project .

Analytical rubric for assessing the group report

As indicated the students were required to submit a group report describing and

explaining their work. They were directed to divide the report to seven chapters.

Chapter 1 refers to the needs and the system requirements, chapter 2 presents

literature review, chapter 3 explains the design process, chapter 4 explains the block

diagram of the optimal system and how the system works, chapter 5 describes the

physical model, chapter 6 includes the references list, and chapter 7 contains the

transcriptions of the meetings. An analytical rubric was developed to assess the group

report. The development was mainly based on rubrics for PBL that were found in the

literature. Some of the dimensions and criteria were developed specifically for this

study, since suitable items were not found in the literature. In the first phase of the

rubric designing the dimensions were determined according to the project stages. In

the second phase the criteria and the benchmark (three performance levels) were set

for each dimension (see appendix A).

The rubric was content validated by three experts in science and technology

education. First, analysis of the dimensions list was done. Then, the matching of each

criterion to its dimension was examined, and finally the level of clarity and the

detailing of each criteria and benchmark were reviewed .

Two lecturers assessed the group report – one is an expert in Biology teaching and

the other is an expert in Technology teaching. The Spearman Correlation between the

scores of the two experts (inter-judgment reliability) was highly positive – 0.854 .**

The questionnaire

The questionnaire was designed to study the students' attitudes towards PBL

environment and towards the alternative assessment strategies implemented in the

course. The items of the questionnaire were based on characteristics of PBL (Krajcik,

Czerniak and Berger, 1999; Buck, 1999).

The first person singular was used intentionally, to obtain each respondent’s

perception of his/her own subjective experiences in the classroom, rather than

perceptions of the experiences of the others (Taylor & Fraser, 1991). Placing students

in the center of the instruction process and the willingness to understand the student’s

personal point of view is a basic principle in the constructivist education .

The questionnaire scale of measurement was that of Likert (five extents of

agreement from strongly agree to strongly disagree) and it contains 27 items. The

questionnaire was content validated by three experts. The categories’ list and the

matching of each item to the appropriate category were examined. The students

completed the questionnaire at the end of the course. Appendix B presents the format

of the questionnaire.

Personal report

Each student was required to submit a personal report. The students were

instructed to refer to the following points: a general description of the processes

occurring in the group, what happened to the student while working with the team; the

construction and formation stages of the team; what was the unique contribution of

the student to the team; what did the others in the team contribute to the student.

Furthermore, the students were requested to answer two questions: in light of your

experience in the course, what are the advantages and disadvantages of learning

through projects? In light of your experience – as a junior high school teacher, will

you incorporate such kind of teaching and assessment?

Content analysis of the personal reports was performed by the two researchers.

Data found in the reports can support, broaden, explain and increase the validity of the

findings in the other research tools.

Major Findings and Discussion

Analysis of the raw data collected from group reports, personal reports,

questionnaire, interviews, observations, and students’ scores (given by the teachers

based on the rubrics) revealed several frequently repeated items. These were used as

categories according to which the collected data were classified. The items came

under four categories – those related to the project-based learning environment; and

those related to the alternative assessment techniques used in the course. The first

category relates to the benefits of PBL from the students’ point of view. The second

considers the challenges students experienced and perceived while conducting their

projects. The third category refers to the benefits students may gain from the

formative assessment. The fourth relates to the type of difficulties located by the

course teachers throughout the assessment meetings.

The Benefits of PBL

While working on the projects, the students were exposed to some advantages of

PBL. For instance, many students noted that in the course of the project they acquired

interdisciplinary knowledge, which they believe is one of PBL’s advantages:

We were required to cope with issues from the various disciplines –

Biology, Chemistry and Technology. Each of us studies a specific

subject, and the need to perform a joint project forced us to study

subjects from other disciplines. I understood that in this method the

student can acquire knowledge from other domains.

Other students marked the mutual inculcation between the members of the team as

a means of acquiring interdisciplinary knowledge:

Each of us contributed his share. We were exposed to various work

methods of our own … we were exposed to a variety of ideas … we

learnt from one another … I learnt from my colleagues unique

subjects .

Or:

There existed among us in the team a readiness to share information

and ideas … each of the team member arrived from a different domain,

and through our mutual work, a variety of ideas and scientific aspects

were raised … I majored in Chemistry, whereas the other members of

the team majored in Agricultural and Civil Engineering…. I helped

them understand concepts in chemistry, which were totally new to

them.

Another student emphasized the importance of work within a team, as a means of

coping with a wide variety of issues, and a large amount of information:

In teamwork, one is able to encompass a larger amount of contents, as

compared with learning by oneself. Each team member learnt a

portion of the sum required, and subsequently taught the others. The

integration among us was very good.

The students’ perception of the interdisciplinary knowledge acquisition as an

advantage of PBL was also manifested in their answers to the questionnaire. Based on

their experience in the course, 95% of the students maintained that PBL allowed them

to acquire knowledge and enhance their understanding in interdisciplinary subjects.

Interdisciplinary knowledge acquisition develops the capacity for systems thinking

and increases the awareness of social and environmental aspects .

Indeed, according to Krajcik, Czerniak and Berger (1999) and Buck (1999),

students in PBL are engaged in active learning and gain interdisciplinary knowledge

while working in a real-world context. Students develop deep and integrated

understanding of content and process and focus in depth on central ideas and salient

issues. The learning environment fosters the acquisition of inquiry skills, problem

solving skills, and information skills .

Another advantage identified by a large number of students, is that in the PBL

approach, the responsibility for the learning lies on the student. And thus, for

example, one of the students mentioned the social pressure within the team as a factor

that stimulated her to strive harder:

In the course of the teamwork, I realized the magnitude of my

responsibility. I undertook a task and my team friends expected me to

perform it in the best way that I could. I realized that I could not let

them down. If I should “screw up” the quality of the teamwork could

be adversely affected, and we would all stand to lose.

Another student expressed a similar idea, only in slightly different words, while

mentioning the relevance as a learning motivation rising factor:

In the work allocation among us I was in charge of a certain issue. I

promptly understood that I had to master that issue so that I could

later teach it to the other members in the team. The subject I had to

learn was selected by me, since it whetted my interest. It was

something I had encountered in a different course at the Technion. It

was really “fun” to dwell on it.

And indeed, in answering the questionnaire, about 85% of the students (16 out of

19) responded to the questions that related to the issue of raising their learning

motivation and responsibility in PBL with the answer: “to a large/very large extent”.

About 90% of the students (17 out of 19) agreed “to a large/very large extent” that

PBL allowed them to be engaged in everyday relevant issues.

By viewing the literature, it seems that many researchers believe that in PBL the

student’s responsibility for learning is higher compared with the traditional learning

methods, and that in certain conditions, the students’ motivation for learning is

increased. Thus, for example, according to Buck Institute of Education (1999), PBL

allows students to work autonomously to construct their own knowledge. Green

(1998) notes that learning in PBL is likely to increase motivation and give the

students a sense of satisfaction. It can also help to develop long-term learning skills.

Krajcik, Czerniak and Berger (1999) suggest that PBL promotes responsibility and

develops independent learners. Orevi and Danon (1999) indicate that since students

deal with relevant issues their motivation is increased .

The third advantage identified by the students refers to acquiring communication

skills and methods of decision making within a team. A number of students

emphasized an improvement in their discourse skills. For example:

In the course of our discussions we learnt to listen to one another, to

conduct a conversation and offer and receive criticism in a

constructive manner.

Other students emphasized the challenge in making team decisions:

… Usually, following a long discussion, we needed to make a decision

in order to pursue a matter. All our decisions were made by consensus,

and there was no need for voting over them.

Other students emphasized the social aspect and the improvement of the

interaction with the teachers of the course;

Throughout the work, the social ties among us improved. Furthermore,

the interaction and ties with the teachers of the course were better than

in other courses at the Technion.

From the students’ answers to the questionnaire, it seemed clear that the majority,

90%, believed that in the environment of PBL one can improve the communication

skills and the ways of obtaining decisions within a team.

Many researches have shown that interpersonal skills can be learnt and acquired

gradually and developmentally (see for instance: Hertz-Lazarowitz, 1990). One

example of the communication skills expected of the interlocutor is active listening. It

is not sufficient to tell a student that he should listen. In order for a student to

comprehend what is the behavior expected of him, it is advisable to break up the skills

into observable and clear behaviors. It is also advisable to discuss it with the students

and involve them in identifying the skills, i.e., the listening skill may be broken into

behavior skills (the body is turned in the direction of the speaker, looking into the

eyes of the speaker, showing interest, responding with head gestures) and into verbal

response skills (could you please repeat, have I heard correctly that …, did you mean

that …, I understand that you claim …). One of the training lessons in the course was

devoted to exercises of the above kind.

Another lesson in the course was devoted to methods of decision-making in a

team. Various techniques were viewed, such as making decisions by majority

(voting), making decisions by consensus, making decisions by brain storming, making

a decision by the chairman after consultation, the Nominal Group Technique method

and the Delphi method (Dyer, 1987; Parker, 1990; Jewell & Reitz, 1984). By

analyzing the students’ reports, it emerges that with all the teams, the decisions were

made by consensus. A possible explanation to this is that time pressure during the

course did not enable the students to apply more time-consuming methods. Perhaps

there were students who preferred not to insist on imposing their opinion, provided

they would meet their scheduled dates. This point will have to be taken into

consideration, while planning the next course.

Another advantage identified by some of the students relates to the enhancement

of one’s self-esteem in PBL environment. One of the students attests about himself:

At first I had many apprehensions, but the more we progressed in our

work and were able to meet more and more tasks, the more did my self

confidence increase, and I began to believe in our ability to complete

the project and meet all the course requirements.

Whereas another student attests about a team-mate:

In the beginning M was the weak link in the team. We demanded that

she be a more active participant, and the more we made progress, the

more active and creative she became. Suddenly she started to raise

many new ideas … her self-esteem rose…

Researchers of the PBL method relate to the issue of the self-esteem in an indirect

manner. For example, Krajcik, Czerniak and Berger (1999) indicate that PBL offers

multiple ways for students to participate and to demonstrate their knowledge, that can

be matched to the various learning styles of students.

Self esteem is a well-organized system of qualities and values a person attributes

to him/herself. It is affected, among others, by the discrepancy between the

expectations that the person consolidates about him/herself and his/her achievements.

The self-esteem evolves within reciprocal relations between man and his

surroundings. Many studies tested the correlation between the self esteem of the

student and variables such as academic achievements, the need for achievement, the

level of aspiration, the level of anxiety, curiosity, locus of control and teachers’

assessment. In some of these studies, a positive correlation was found between the

self esteem and success and receiving positive assessment (Battle, 1991). Therefore, it

is likely that PBL environment, which is based on formative assessment and

continuous support, as the case was with us, an increase of certain students’ self

esteem was found.

And finally, the fifth advantage identified by the students is that in the

environment of PBL learning is active and experiential. One student articulated it very

eloquently:

While deliberating on a certain issue and searching for information

sources on that issue, I was able to come to a conclusion of my own. It

was a great experience, and I am sure that I will never forget this

material.

Another student emphasized the intensive activity of searching and sorting

relevant interdisciplinary information:

After extensive reviewing of dozens of Internet sites, we put a lot of

effort into sorting the data and select the sites that deal with scientific,

technological and social aspects related to a car that operates by

means of solar energy.

And indeed, about 80% of our students (15 out of 19) very strongly believe that

learning in PBL environment is active and experiential learning (as opposed to other

courses in the faculties of Sciences and Engineering at the Technion, where they

solely have a passive role).

Active learning is a principal characteristic of PBL environment that is based on

the constructivism approach to teaching. Constructivism is a theory concerning

learning and knowledge, which suggests that the human being is an active learner who

constructs his/her knowledge on experience and on his/her efforts to give meaning to

that experience. In the course presented here, students were, in fact, required to

construct their knowledge by means of active experience and learning in the form of

‘trial and error’.

As constructivist researchers, we hope that through their experiencing, our

students learnt to understand the advantages and challenges of applying the PBL

approach to teaching. We hope that now, that they have experienced it by themselves,

they will be able to integrate this approach into their teaching in the future.

Challenges of Students in PBL Environment

According to Hertz-Lazarowitz (1990) the interpersonal skills required of a team

member can be divided into three groups. The first group is the communication skill –

required behaviors from the message transmitter and required behaviors from the

message receiver. The second group relates to the vote-of-confidence skills. The third

group refers to the negotiation skills and coping with conflicts within the team. These

skills can be improved through training and practice.

The list of the interpersonal skills required of a team member was discussed with

the students in one of the course lessons, and was even practiced to some extent. From

analysis of the personal reports of the students, we have learnt that some of them were

indeed exposed to situations of conflicts in the teamwork. Following are a number of

examples:

While working with the team, some social problems that affected the

work process arose.One weak link in the team can severely harm the whole teamwork. We

encountered such a weak link.

Not everybody knows how to work or is interested in working in a

team. There was one member within us who preferred to work on his

own.In the course of our work, struggles of forces were formed, yet the

more we progressed the better we learnt how to cope with them.I learnt to adapt my work habits to those chosen by the team.

We hope that experiencing conflict situations during the work with the team, in

conjunction with referring to resources where exercises of communication skills

improvement can be found, will allow our students to better prepare their future pupils

for working in a team.

The second challenge located by the students relates to the large amount of time

which the teacher invests while implementing the PBL method in his/her teaching, as

compared to the time and effort required for teaching the same contents over and over

again in a traditional teaching method. So, for example, though we met with the

students a great number of times (in relation to what we do in our regular courses),

there were students that claimed that it was not enough. These students concluded

that:

The PBL approach requires that the teacher continuously follow up the

activities of each one of the group members and their degree of

progress.

Another student expressed the same idea only she phrased it differently:

I don’t think I will be able to integrate such a teaching method in the

first years of my work as a teacher. It seems that one needs a great

deal of time and effort, and I will probably be busy with many more

basic issues.

And one student, concluding that in order to implement the PBL method the

teacher should have a broad interdisciplinary knowledge:

This method requires of the teacher a great deal of effort and a very

wide interdisciplinary knowledge.

Another challenge faced by the students is the need to cope with new contents in a

learning environment which is neither structured nor organized in advance:

In the beginning I was frustrated, as I had no idea where to begin… In the beginning we all wasted a lot of time, because we really didn’t

know what to do, and at the end we were encountered by the most

severe time stress.I prefer the traditional-organized method to this one.

According to Krajcik, Czerniak & Berger (1999), teaching by means of PBL

presents several challenges. One of these challenges is: “students’ lack of experience

in this new approach and their preference for traditional-structured approach.”

The students’ attitude towards alternative assessment

The formative assessment strategy that we applied in the course served, among

others, as a means of locating difficulties of students and choice of intervention means

in order to assist the students who face difficulties. Each group met with one of the

course teachers for one formal meeting once every three weeks (sometimes there were

additional informal meetings). The meetings had a constant format: at the beginning

the students reported about what they had performed since the previous meeting.

Subsequently, the hardships encountered by the students were discussed, as well as

the ways of coping with those hardships. Finally, the students presented a work plan

for the next three weeks. The interaction between the teacher and the team also

included a feedback and assessment regarding the degree of the progress made. The

students were required to arrive at the meetings with a written report, specifying what

they had performed, the difficulties they had encountered, and a work plan. Each team

kept the reports and the summary of the meetings with the teachers in a group

portfolio. We wanted to know to what extent these meetings helped the students. For

this purpose a number of students were interviewed, and a questionnaire was

submitted to all the students at the end of the course.

The summary of the students’ answers to the questionnaire demonstrates that

about 80% feel that giving the criteria for assessing the students’ achievements at the

beginning of the course, helped in the implementation of the project. The majority of

the students maintain that continuous assessment throughout the course, not just at its

end, advances the learning for the following reasons: it assisted the student to evaluate

the degree of progress (95%), cope with difficulties (80%), locate the points which

require correction or improvement (90%), test additional aspects related to the project

(80%) and cope with conflicts among the members of the team (80%). The students

indicated that the feedback they received in the meetings with the course teachers,

assisted their learning in any of the following project stages: defining the problem /

the technological need (95%), formulating the system objectives (85%), locating the

information sources (80%), presenting and testing alternative solutions (80%),

designing and constructing the system (80%) and presenting the work (95%).

Similar findings also emerged from analysis of the data collected during the

interviews. Part of the students related in the interviews to additional issues. For

instance, there were some who indicated the contribution of the tri-weekly meetings,

and the feedback / assessment they received one from the other to raise the level of

the performance:

In addition to the feedback we received from the teacher, we were also

requested to give feedback and evaluation to one another. I believe

that the criticism was fair and constructive, and helped to raise the

level of our performance.

Other students emphasized that the meetings allowed the course teachers to locate

the students who had difficulties, and assist them:

The meetings allowed the teacher to become more familiar with us and

identify the strong and weak points of each of us. Anyone who faced

difficulties received personal tutoring.

A number of students pointed at the potential difficulties in alternative assessment.

For example, two teams stated the fact that the course teachers had difficulties in

assessing the personal contribution of each student:

There was a problem for the TA to monitor each student individually,

since the emphasis in the meeting was on the group as a whole.

Another student indicated that:

The course teachers had a hard time assessing the contribution of each

member of the group.

Another difficulty identified by the students relates to the vast amount of time and

effort the teacher would have to invest in order to implement the techniques of the

formative assessment. All the students indicated that continuous follow-up of the

teacher was required to ascertain the progress of the whole team and each student. For

example:

Such work requires a continuous follow-up of the teacher to check the

activity of the team members and their progress …

A continuous follow-up of the teacher is required to check the

development of the project at each stage.

I suppose that in the junior high school I will have to invest a larger

number of meetings with the students, than the number of meetings you

held with us.

Difficulties that were located by the staff during the assessment meetings

As said, the formative assessment strategy, implemented by us, allowed us to find

out the difficulties the students encountered in the course of performing the project.

Following are a number of quotations from the students’ reports and transcripts of the

meetings:

I had difficulties in processing and concluding the material that was

gathered.

In constructing the theoretical background, we had difficulties in

selecting the relevant material for the project out of the sources we

gathered.

We had doubts as to the level of detailing would have to be in writing

the theoretical background.

In order to perform the project, basic engineering now-how is

required, including familiarization with design and planning processes

as well as scientific knowledge for the purpose of defining and

understanding the scientific principles underlying this system.

It was difficult for us to define in advance the system requirements, and

we retrospectively defined them only after choosing the system and the

analysis of its components.

Understanding the manner by which the heart functions did not suffice

in order to define the design requirements for an artificial heart.

We did not understand the meaning of detailed design.

We had difficulties in choosing the system: we were looking for a

system in which the needs it meets would be easy to define.

We have encountered a number of technical problems, such as:

generating the required level of voltage for the system operation.

From analyzing the above quotations and the assessments, based on the rubric, it

seems that the difficulties can be sorted into six groups: problems in choosing a topic

for the project (different areas of expertise and desires), problems in implementing the

project (not finding alternatives for a solution, etc.), social problems (conflicts,

incompetent team members, etc.), problems related to submitting the group report (it

was not clear which level of profoundness is required, incompatibility between the

team members with regards to what each of them prepared, searching and locating

suitable information resources, etc.), problems related to the artifact (technical

difficulties, etc.), and problems related to preparation of the presentation (what to

present in the limited time allocated, etc.).

Conclusions

Summary of Findings

The study question was: what implementation issues and processes do higher

education students encounter in a PBL environment which involves an alternative

assessment approach?

Benefits of PBL. While performing the projects the students were exposed to five

benefits of PBL: gaining interdisciplinary knowledge, acquiring that knowledge

through active and experiential learning, taking responsibility for the learning,

acquiring communication skills and methods of decision making within a team, and

enhancing of one’s self-esteem.

Challenges of students in PBL. At the same time the students experienced three

challenges: coping with conflict situations in the teamwork, investing a lot of time and

efforts (both on the teacher’s side and the student’s side), and coping with new

contents in a learning environment which is neither structured nor organized in

advance (experiencing frustration in an environment of uncertainty).

The students’ attitude towards the formative assessment. The majority of the

students maintained that continuous assessment throughout the course advanced the

learning for six reasons: it helped to understand the course goals and requirements, it

assisted to evaluate the degree of progress, it helped to cope with difficulties and to

locate the points which require correction or improvement, it emphasized the need to

test additional aspects related to the project, it assisted to cope with conflicts among

the members of the team, and it allowed the course teachers to locate the students who

had difficulties. The students pointed at two difficulties: sometimes the course staff

had difficulties in assessing the personal contribution of each student, and the teachers

had to invest vast amount of time and effort in order to implement the alternative

assessment techniques.

Difficulties that were located by the staff during the assessment meetings. The

students difficulties can be sorted into six groups: problems in choosing a topic for

the project, problems in implementing the project, social problems, problems relating

to submitting the group report, problems referring to the artifact, and problems

relating to preparation of the presentation.

Implications for using PBL in course for training pre-service teachers

Integrating PBL in teaching requires training the teaching staff both in the content

knowledge as well as the pedagogical knowledge. In addition, integration of PBL in

teaching science and technology requires a teaching staff that is familiar with the

design process as well as with a scientific knowledge. Usually, novice teachers lack

the training necessary to teach PBL effectively mainly because they have not been

trained in this area.

While being involved in active experiencing of the project implementation in the

PBL environment, our students were exposed to the advantages and challenges of the

method. We are hopeful that this experience will assist them to implement the method

in the future, when they themselves become teachers of Science and Technology in

the junior high school. By looking at the study findings, it seems that indeed the

students were exposed to the benefits and challenges inherent in Problem-Based

Learning and in the accompanying assessment approach. The students acquired both

content knowledge as well as pedagogical knowledge. In the course of their active

experiencing, the students acquired interdisciplinary scientific-engineering

knowledge, and familiarized themselves with the design process.

Data analysis revealed that students found the alternative assessment more

suitable for a course based on PBL than traditional methods, and they indicated that

they would use this type of method in their teaching. According to the students,

implementation of formative assessment throughout the course would help to reduce

anxiety, identify and solve problems, monitor progress and guide them through the

project.

The students perceived the alternative assessment as a positive experience. They

felt that they had been given an opportunity to share, reflect on, and discuss their

ideas, and that this type of assessment should also be applied in other courses. The

results of this study can provide educators from different disciplines who are

interested in PBL with a set of assessment instrument as well as with some

implementation tips as to how to apply PBL in the classroom .

More than ever, universities are now expected to help students develop skills and

competencies in real-life "authentic situation” and to enable them to demonstrate

these abilities by performance on alternative assignments rather than standardized

tests. Our study findings show that PBL environment enables to realize this goal.

Since our research was a case study we recommend that the findings presented in this

paper will serve as a basis for follow up studies .

REFERENCES

ADLER, P. A. & ADLER, P. (1994) Observational Techniques, in: DENZIN N. K. & LINCOLN Y. S. (Eds) Handbook of Qualitative Research, (Thousand Oaks, CA,

Sage) .

BATTLE, J. (1991) Self-Esteem Research: A Summary of Relevant Findings, (Edmonton Canada, James Battle & Associates).

BIRENBAUM, M. (1997) Alternatives in Assessment, (Tel Aviv University, Ramot, in Hebrew).

BUCK Institute for Education (1999) PBL Overview. Available at: http://www.bie.org/pbl/overview/index.html

CPS – Chicago Public Schools website (2000) Analytical Vs. Holistic Rubrics. Available at: http://intranet.cps.k12.il.us/Assessment

DENZIN N. K. & LINCOLN Y. S. (2000) Handbook of Qualitative Research, (Thousand Oaks, CA, Sage) .

DENZIN, N. K. (1988) Triangulation, in: KEEVES, J. P. (Ed.) Educational Research, Methodology and Measurement: An International Handbook, (Oxford, Pergamon

Press).

DYER, W. G. (1987) Team Building, (New York, McGraw-Hill) .

GONELLA, F. (2001) Assessing Student Work with Project-Based Learning. Available at: http://pblmm.k12.ca.us/PBLGuide/AssessPBL.html

TAYLOR, P. C. & FRASER, B. J. (1991) CLES: An Instrument for Assessing Constructivist Learning Environments, paper presented at the Annual Meeting of

the National Association for Research in Science Teaching (Fontane, Wisconsin).

GOODRICH, H. (1997) Understanding Rubrics, Educational Leadership, 54 (4), pp. 14-18.

GREEN, A. M. (1998) Project-Based-Learning: Moving Students Toward Meaningful Learning, ERIC Database, ED422466.

HERTZ-LAZAROWITZ, R. (1990) An Integrative Model of the Classroom: The Enhancement of Cooperation in Learning, paper presented at the Annual Meeting of the American Educational Research Association (Boston, MA, April 17-20).

JEWELL, L. N. & REITZ, H. J. (1984) Group Effectiveness in Organizations, (New York, Warner Books).

KRAJCIK, J., CZERNIAK, C. & BERGER, C. (1999) Teaching Children Science: A Project-Based Approach (New York, McGraw-Hill College).

MISHLER, E. (1986) Research Interviewing: Context and Narrative, (Boston, Harvard University Press).

MORSE, J. M. (1994) Designing Funded Qualitative Research, in: DENZIN, N. K. & LINCOLN, Y. S. (Eds) Handbook of Qualitative Research, (Thousand Oaks, CA,

Sage) .

OREVI, N. & DANON R. (1999) Learning Ecology with Educational Technologies, paper presented at the International Workshop on Science Teachers Education toward the New Millennium (Haifa, Technion - Department of Education in

Technology and Science).

PARKER, M. G. (1990) Team Players and Team Work, (New York, Prentice Hall).

ROSENFELD, M. & ROSENFELD, S. (1999) Understanding the surprise in PBL: an exploration into the learning styles of teachers and their students, paper presented at the 8th Conference of EARLI - European Association for Research in Learning

and Instruction (Goteburg, Sweden, EARLI)

SHEPHERD, H. G. (1998) The Probe Method: A Project-Based-Learning Model’s effect on Critical Thinking Skills, Dissertation Abstracts International, Section A, 59 (3A), p. 779.

SPRADLEY, J. P. (1979) The Ethnographic Interview, (New York, Holt, Rinehart & Winston).

THOMAS, J. W. (2001) A Review of research on Project-Based-Learning. Available at: http://www.autodesk.com/foundation/pbl/research

Appendix A: Criteria of the Project-Based Technology Rubric (PBTR)

Dimension Criteria

Defining the problem / formulating the need for the system design.

The problem was clearly and unambiguously stated.

Defining the system requirements. The system objectives are clearly defined, are compatible with the problem definition and are attainable.

Theoretical scientific-technological background needed for the project.

The team has conducted sufficient literature review to support the project. All information is clear, appropriate and correct. Covers topic completely and in depth. Information includes scientific, technological and social aspects relevant to the project.

Project planning. The project planning includes:Presenting a number of alternatives (solutions) that meet the problem.

Presenting a number of advantages and disadvantages of each alternative.

Choosing an optimal solution based on technological, economic and environmental considerations, as well as schedules and anticipated difficulties.

Stages of the system design . The system design is presented in the Top Down method in two stages:

Preliminary design, which includes drafting of a block diagram accompanied by annotations and details of the relations between the sub-systems.

Detailed design, which includes flowchart and/or detailed explanation of each “block” in the sub-system of the hyper-diagram.

System description: Specifying the system components and its mode of operation .

The system mode of operation is explained in a clear manner and integrates drawings and illustrations. The system components are explained in details, commencing with a general explanation of the exhibit, and subsequently accompanied by a detailed explanation.

Description of the lab model: details of the model goal and its mode of operation.

The goal of the model is clearly defined. The information attached to the model is sufficient to build the model and understand its operation principle.

The scientific/technological principle intended to be demonstrated is coherently explained and illustrated through the model.

The model’s mode of operation is explicitly explained and integrates drawings and illustrations.

References. Complete list of all sources used.A variety of significant and relevant sources (internet sites, books), presented according to the rules of the bibliographic writing.

Documentation of the project stages

(Portfolio.)The documentation includes organized listing of all the meetings’ protocols. Each protocol presents:

A report regarding the activity performed by the team according to the design and the goals set forth in the previous meeting. Planning the tasks to be performed towards the

following meeting.Problems related to the project performance.

Appendix B: Sample Questions from the “Students’ Attitudes towards PBL” Questionnaire

Scale Name #of Item

Description Sample Item

Assessment Method

11 The extent to which the assessment method helped the student

The feedback I received during the project helped me:

To monitor my progress in the project.

To locate the points which require correction or improvement.

The feedback I received assisted me in the following stages of the project:

To formulate the system requirements.

To examine and choose alternative solutions.

Teaching Method

16 The extent of benefits gained by applying PBL as realized during the instruction process.

In light of my experience in the course, PBL:

Allowed me to cope with interdisciplinary knowledge.

Encourage active learning .