R e p o r t - T h e M o o d l e t e s tICT-based Approaches to Evaluation – mandatory task 2

Nina Tvenge, Susann M. Harlem, Irene Beyer Log og Rolf-Erik ØsteråsDate: 07.12.2009

Revised by Nina Tvenge 18.12.09

Table of ContentsIntroduction.......................................................................................................................3

Design Choices...................................................................................................................3We used............................................................................................................................................3Feedback..........................................................................................................................................6Students response.............................................................................................................................7

Process................................................................................................................................8Step 1: Understanding the task, pointing out main deliverables......................................................8Step 2: Production of test material...................................................................................................8Step 3: Running the test...................................................................................................................9Step 4: Learnings and report............................................................................................................9

Observation and the results...........................................................................................10The first questions..........................................................................................................................10Question 6:.....................................................................................................................................11Question 9:.....................................................................................................................................12

Conclusion.......................................................................................................................13

References........................................................................................................................14

2

IntroductionThis report is according to the mandatory task; “Design a digital test related to “nethics”, which you try out at the ICT2-students at HiO. Write a short report describing the process, the design choises, and the results”.

During the course “ICT-based approaches to Evaluation” we have spent some time studying opportunities for testing using different digital solutions. There is a diversity of learning resources on the Internet; many do also have embedded assessment tools. We tested, identified and evaluated tools like “Kasparia”, “Sirkel” and “Second Life”, representing open web-solutions. Of proprietary systems, like learning management systems (LMS), we analyzed the test-tool in Fronter.

Since the previous mandatory task was related to other assessment tools, we chose to look at a totally different one in this round, point being to get to know features and assessment qualities of an unknown tool. We decided to design the test in an LMS new to most members of the group; Moodle.

During the design process, we discussed whether we wanted to test simple, factual knowledge or also attitudes and ability of ethical reflection. And what kind of questions fit the purpose of the test? In the part of the test concerning “nethics”, we ideally wanted to test the whole specter. The test is relating to Blooms taxonomy, and the questions are formed as multiple choice (MC), “matching” and essay. Feedback after completion is automatically displayed. Even though aiming for it, we found it difficult to reach the top levels of Blooms taxonomy.  Time span to do the test was planned to be 5-10 minutes. We decided to write the questions in Norwegian.

Design ChoicesMoodle is an open source LMS. Free of charge except, of course, administration costs. The LMS is available for all, and students and teachers create their own users. Rooms can be open or closed, the latter by using security keys, this unlike the more familiar (to both student groups) tool; Fronter, which is a very closed off system.

Moodle was a new acquaintance for the students. According to Russell and Goldberg (2003), we discussed possible pitfalls related to introducing a “new” learning management system. The unfamiliarity of a new environment; moodle, could have an impact on the ICT2-students test

3

results.

We decided to design the questions mainly as multiple choice with automatic feedback. This question type consists of two parts: the stem, and two or more alternative responses, one is the correct answer, while the others are distracters. The difficulty of a question increases as the distracters are more and more close to the correct answer in both shape and content. (Baker, 2001) And another important aspect when creating questions is, according to M. Hoepfl (1994):

“Multiple choice tests should be able to measure the cognitive processes knowledge, comprehension, application and analysis (level 1-4 in Blooms taxonomy). For measuring level 5 and 6, open-response tests may be better suited”

To try to get around this problem we incorporated a question with two stems, number 9. By doing this; having two “stems” and many alternatives, the student is challenged to analyze and evaluate, and such, the question might measure on level 5-6. We also hoped to disregard the possibility to guess the right answers to a much greater extent. There are multiple combinations, and thus harder to guess. In order to answer correctly on this question you have to be able to separate the two concepts; have a true understanding of them.

One of Hoepfl's guidelines for making a test is “avoid the use of negatives in the stem”. If you use a negative, you must underline it for emphasis, and only when knowing the exception is as important as knowing the rule. In question nr 6 we used a negative stem, we'll come back to why.

The matrix on next page is made to illustrate on which level we aspired to reach with our questions in the test, according to Blooms taxonomy:

4

Bloom’s taxonomy of educational objectives describes six cognitive processes, arranged from lower to higher-order thinking skills1: Knowledge – the ability to recall facts, principles, terms, categories, rules and formulas. Comprehension – the ability to describe, illustrate, summarize, or interpret. Application – the ability to use previously learned material in a new setting. Analysis – the ability to describe separate elements and the relationships between them. Synthesis - the ability to judge the value of ideas or plans from previously learned material. Evaluation – the ability to judge the value of ideas, methods or procedures.

Criticism is made towards Blooms taxonomy, its’ origin dating back to 1956. One thing

being just the fact that it was created in a totally different era, and learning has been under

constant research and reformed since then. Also:

Anderson (2000) argues that nearly all complex learning activities require the use of several different cognitive skills.2

We are aware of this, but considering the time aspect here, we choose not to investigate

this further, but notices it as another possible weakness in our test.

We used  Multiple choice: Already commented Matching: “A list of sub-questions is provided, along with a list of answers. The

respondent must “match” the correct answers with each question”

1 Hoepfl, 19942 http://www97.intel.com/en/ProjectDesign/ThinkingSkills/ThinkingFrameworks/Bloom_Taxonomy.htm

5

Essay: For comments at the end

These were chosen mainly on the basis of theory we have read, rather than investigating the possibilities of the test tool in moodle3

FeedbackGiving value of each question and each correct answer is a huge research area. We've looked at “Item Response Theory” as Baker (2001) explains it.  Being far from experts, we've tried to value and grade the answers according to level of difficulty, and also looked at on which level we hoped the question would measure the students’ knowledge in the two subjects (“nettvett” and nethics). 

Because we’ve given this test only once, we cannot say anything about its validity. Does it measure what it’s supposed to measure? We said that we wanted to make at test for learning; formative assessment, not a test to measure what the students already know, summative assessment. Or, that also, but by giving questions where the answer could not be found directly in their curriculum, we hoped that the students had to think and re-use their knowledge in new contexts. Black and Dylan (1998) discuss the gap that often occurs between formative assessment and tests actually given:

“The grading function is over-emphasized and the learning function underemphasized” 4

Whether we have “hit the target” here or not is, as previously mentioned, hard to say because of the limited data obtained through only doing this once.

A weaknesses in our test is also that we really didn’t know what had been the emphasis in the ICT2-students’ course. Thus are our questions colored from what we, the master students, found important, rather than the view of teacher and the students in the test group.

3 Question types available in moodle, can be found here: http://docs.moodle.org/en/Question_types#Standard_question_types4 ) P.10, Black and Dylan, 1998

6

The illustration shows how we've valued the questions in this test:

(Click on picture to see larger illustration)

Students responseIn the last commentary field the students say they find this test relevant to their studies, it made them aware of matters they now realized were of importance in their everyday work and activities5

Again, it is difficult to evaluate the validity of an element, this time first and foremost because we don't know the students’ background, skills and knowledge. Vivien Sieber (2009) says:

“Weak students tend to overestimate their own skills, but strong students, which have a high degree of ICT-skills see that they still have a lot to learn, because they understand there are many things still to learn and then underestimate their own skill level.”

We find this to be an interesting area, and it could be meaningful to look into this field on a later occasion. How relevant is this test really according to the curriculum of the ICT2-course, according to the ICT-students experience or according to their needs?

5 Appendix 1

7

ProcessAt our first meeting on MSN and Skype, we agreed on how to collaborate and cooperate, and which test tool to be used. 

We decided:

Use of Skype for voice/chat MSN for chat and Etherpad for brainstorming and documentation. During the

work progress, we used different Etherpads. This to keep track of the process and to facilitate collaboration. We think this method mediated even more learning for the group.

Fronter for storage and Moodle for making the actual test for the ICT2 students. It is not possible to upload and share pictures into Etherpad, so Fronter was used for sharing documents of common interest

We delivered our finished product in the master wiki Elluminate to view tables, documents etc. in discussions.

Step 1: Understanding the task, pointing out main deliverablesWe worked as one team doing all activities together. All documentation from brain storming, references to documentation and suggestions on how/what to deliver was documented in our Etherpad 16. The documentation we produced where unstructured and gave the team limited overview of the task. However we felt we had touched all areas and had a good starting point, making sure we did not forget any important issues throughout the rest of our work. 

Step 2: Production of test materialWe worked in two teams; one focusing on producing the questions for the test, the other producing the room and the actual test in Moodle. A new Etherpad 7 was created, with copies of selected sections from Etherpad 1. 

We added three questions, giving our co-students the ability to give feedback on our test.  

Question 10:o ”Hvor lang tid tok det å gjennomføre testen?”

Question11: o ”Var spørsmålene relevante i forhold til IKT2 studentene?”

Question12: o ”Noen andre kommentarer til spørsmålene?”

We ran this test through the other group and vice versa giving/receiving feedback and adjusting the test accordingly.8

6 http://ibey.etherpad.com/3, password: nethics7 http://ibey.etherpad.com/4, password: nethics8 Appendix 2

8

For the ICT2-students we had only one extra open question: 

Q 10: “Var denne testen relevant for deg som IKT2 student? Hvorfor, hvorfor ikke?”

Step 3: Running the testIrene ran the test on ICT2-testgroup, and observed how it went and documented the verbal comments from the students. The results were discussed by our group, on Skype, and we picked out areas for learning, such as: Misunderstood questions, misleading questions, rules on not using negative questions9. We also discussed different ways of how to present our learning outcome and the process of choosing questions. All documentation were put down in Etherpad 310

Step 4: Findings and reportAt this stage we had discussions on what observations should be commented with what theory / documentation, and the highlights were documented in Etherpad 411. After having gained overview of what was left on the task given, we divided the rest of the work between us: 

Introduction / conclusion, process and Blooms-matrix, moodle design and choices and results

Relevant literature We have all given feedback and comments to all parts of the document as quality

measurement at the end. As quality assurance we have tried to look at how we had utilized the most important theories throughout the work.

9 We implemented one question in our test which contained a negative statement. Reason for this was to check how this question was interpreted by the students in particular10 http://susann.etherpad.com/8 password: rapport11 http://ibey.etherpad.com/rapport, password: rapport

9

Observation and the resultsWe had a representative present to observe and assist if any technical problems occurred during the test implementation. Only eight ICT2-students were present that day.  On advance we had emailed the students asking them to create users in Moodle, since we wanted to save some time and begin directly on the test.12 This had not been done, so we had to spend about five minutes on this before they could start the test. During the test, we got a few questions concerning terms used in the test.

Immediately after they had finished, we asked for comments from the students. One comment was that one student said he had been “tricked” to think that the number of points was equal to the number of right answers. This might be the case for several (all) if our analyses of the results are correct. 

The first questions

The first question measures knowledge on “nettvett” and here the students have submitted 16 answers, which can be interpreted as they all gave two answers each. The stem has six alternatives, only one is a distracter, the rest are correct answers. But the question states that total points given, when all are correct, are two, which indicates that the students misunderstood the compliance between points and answers. 

The next question, concerning nethics, also measures knowledge and therefore the two first questions reaches level 2 in Blooms taxonomy, which is listed above. “Alltid presentere seg korrekt, med fullt navn”, were one of the distracters and one of the students selected this. We mean that when we phrased the stem with the word “correct” it might be diffuse and the students had to think more. This distracter was also an alternative in question 3, and again one of eight students chose this. We also meant that the choice “Du må alltid bruke nicknames når du skal skrive på blogg, for å verne om egen identitet”, was a distracter because one can easily be led to think that that this is a probable alternative.

12 Appendix 5

10

It is reassuring to see that there was no one who chose what we thought were the wrong alternatives in the didactic question (number 5). This might show that the students were able to tie their pedagogical knowledge to the subject. The students are challenged in several areas, and will possibly reach level 3 at Blooms taxonomy.

Question 6:

“Hvilke av disse alternativene er ikke fornuftig regel for deltakelse i nyhetsgruppe?” Fifty percent answered: “Oppgi så få personlige opplysninger som mulig, både om deg selv og andre”. Only two out of eight answered this question correctly: “Dersom du kjenner de andre på nyhetsgruppen er det trygt å delta”. This can easily be interpreted as that they did not understand or misread the question, because we used a negative word, which is not recommended. One of the students commented that he or she didn't understand the question. Either we have worded it poorly, or they couldn't read the question when we used “ikke”. Looking closer, we see that we have misled them. The question clearly states that one should choose several alternatives, “hvilke”, and we had only one correct answer.  And as this source (Frary, 1995) puts it:

“Don't put negative options following a negative stem. Empirically (or statistically) such items may appear to perform adequately, but this is probably only because brighter students who naturally tend to get higher scores are also better able to cope with the logical complexity of a double negative.”

Only minutes before they took the test, they had discussed how to detect the reliability in a web page, so we are not surprised that no one selected the distracter “Hvis designet er profesjonelt utformet, kan jeg stole 100% på innholdet” in question 7. This was probably one of the most discussed subjects prior to the test, but still the compliance between points and the number of correct answers were a distraction, seemingly, since not all of them chose all the right alternatives.

The next question, number 8, was a didactical question and we wanted them to reflect on how to cope with identity theft. We had a specific idea behind the design of the answers. The distractions we included as alternatives should be plausible, but we wanted them to

11

“reveal” that not only talking to the persons involved, but all of the students had bigger effect. However, most respondents chose the option that only dealt with those involved. 

Question 9:

“Hvilke av disse elementene kan ligge åpent på web, og hvilke må administreres via et LMS.” This was the most challenging question because the students had to use their knowledge on several areas to pass. Due to this we believe that they have reached level 4 and possibly 5 at Blooms taxonomy. We didn't change the alternative on “blog”, the correct answer should have been “both”, partly because we disagreed with the test group response (co-students) on this question. We thought that a blog can only be published openly on the web, but our opinion may have been a little too firm, because the majority of the ICT2students, 75%, answered “both”. We find it fair to question our knowledge on this. The results from this question show that almost all of them had the alternatives with documents13 right. We discussed the “test” alternative in question number 9, and our decision regarding using this term and not specifying, e.g. “tests and quizzes”, was that we didn't want to make it too easy or obvious. Over 60 percent said that tests could only be administrated in a closed LMS, and this might indicate that they relate “tests” to school tests and did not think of what else this could mean.

13 Scanned documents from text books and non-fiction

12

We wanted to end the test with a comment box, to encourage students to give us feedback about the test. Not only did this help us see what they thought about the test and questions, but they also had the opportunity to explain themselves if they wanted to. A comment box is recommended in the article “Test and exercise learning” (Horton, 2000) because they then have the chance to comment on unfair or unclear questions. Overall, the feedback from the students was positive and they all thought the test was relevant to them and as a conclusion to the topic. 14

“Item difficulty” indicates the proportion of students who respond correctly to a test item.”15

This is helpful to analyze the results of a test to see if the question is too difficult or too easy. If the percentage of students selecting the correct alternative is below 30 percent, it should rarely or not be used. The question concerning “Hvilke av disse alternativene er ikke fornuftig regel for deltakelse i nyhetsgruppe”, (number 6), shows that only 25 percent selected the correct alternative and it might therefore be too difficult and/or worded poorly. Otherwise, it looks like that the other results meet that percentage.

“Few tests are perfect the first time out”.16

ConclusionAn early choice in making the “nethics”-test, was to use a “new” learning platform, called Moodle. The technical solutions in Moodle worked great, and it has a flexible and solid embedded assessment tool. For easy access to the test, we made an information paper which we posted to the ICT2-students via e-mail the day before the test. 8 of 19 students were present at the test. To get more reliable results we would wish that more students

14 Appendix 115 Hoepfl, 1994:2616 Horton, 2000 :322

13

were present. Some students did also have a misconception related to the number of questions and the maximum score. Maybe we should have described the rules more thoroughly? Next time we’ll put even more emphasis on being clear and concise, knowing how difficult this is, and we’ve experienced the necessity of implementing this in our own practice.

Some misspellings in the questions, (“hvilke/hvilket”) in question 6, lead to misunderstandings. Even though using negatives in the stem is not recommended, as told above, this was a conscious choice to check whether that statement was valid or not. We have also learned that our opinion of what is correct might not reflect what others think. To improve the tests even more, we would have provided diversified points for the various plausible responses, not only points for selecting our view. 

As we mentioned in the introduction, level 5 and 6 in Blooms taxonomy are hard to reach in a multiple choice test with automatic feedback. The results from the test show some of this. Constructing questions which are interesting and promotes learning for the students, is difficult!

This kind of test is probably best suited in secondary school and higher, first of all

because of the complicated form and content. Also, we aspired to measure analytic

and reflective qualities in the students, and primary school learning is more factual

oriented. That being said, the concept of multiple choice-tests might very well

function in lower grades. It would be interesting to try to get away from the

behavioristic inscription17 of multiple choice-tests, by for instance letting the pupils

create questions themselves, then discuss them afterwards and reflect on the various

suggestions together. This can be done on all levels, from primary school to higher

education, if adapted to the pupils/ students level.

17 Habib and Wittek (2007)

14

References Baker, Frank, 2001: (read 12.03.09) The Basics of Item Response Theory. ERIC

Clearinghouse on Assessment and Evaluation, University of Maryland, College Park, MD. http://echo.edres.org:8080/irt/baker/

Black, Paul and W. Dylan, 1998. Assessment in Education: Principles, Policy & Practice. Vol. 5 Issue 1

Hoepfl, Marie C, 1994. Developing and Evaluating Multiple Choice Tests. The Technology Teacher.

Horton, William: (read 12.03.09) Test and exercise learning. http://www.designingwbt.com/DesigningWBT/dwbttest.htm

Johansen/Storm, 2007. Hvordan digitale, selvrettende oppgaver/tester kan være et nyttig pedagogisk hjelpemiddel? Et pedagogisk utviklingsprosjekt. Høgskolen i Gjøviks rapportserie, bind nr:2007 nr 3

Moodle.org: (11.26.09) Standard question types http://docs.moodle.org/en/Question_types#Standard_question_types

Russell, M., A. Goldberg, et al., 2003. Computer-based testing and validity: a look back into the future. Assessment in Education: Principles, Policy & Practice, Routledge. 10: 279-293.

Sieber, Vivien, 2009. Diagnostic online assessment of basic IT skills in 1st-year undergraduates in the Medical Sciences Division, University of Oxford Volume 40, Issue 2. p 215-226

Wikipedia.org: (read 12.05.09) Item Response Theory. http://en.wikipedia.org/wiki/Item_response_theory

Habib, Laurence and Wittek, Line(2007).The Portfolio as Artifact and Actor',Mind, Culture, and Activity,14:4,266 282

Appendix’:1. Comments from the students - the ICT2/IKT2

http://www.ansatt.hig.no/ninat/master_ikt_laering/Test_moodle/appendix_1.pdf

2. Comments from our testgroup - the other master students http://www.ansatt.hig.no/ninat/master_ikt_laering/Test_moodle/appendix_2.pdf

3. Screenshot of the test: http://www.ansatt.hig.no/ninat/master_ikt_laering/Test_moodle/The_test.jpg

4. Screenshot of the room in moodle: http://www.ansatt.hig.no/ninat/master_ikt_laering/Test_moodle/The_test.jpg

5. How to register in moodle: http://www.ansatt.hig.no/ninat/master_ikt_laering/Test_moodle/Informasjon%20om%20test%20IKT2.pdf

15