Lecture 9: Evaluation of Information Retrieval Systems · School of Information and Communication...

College of Education

School of Information and Communication Studies

Department of Information Studies

2018/2019 ACADEMIC YEAR

Lecture 9:

Evaluation of Information Retrieval Systems

LECTURER: DR. DE-GRAFT JOHNSON AMENUVEVE DEI [email protected]

0243775571

mailto:[email protected]

Learning objectives

Dr. De-Graft Johnson Dei, Dept of Information Studies

Slide 2

EVALUATION

• A systematic, rigorous, and meticulous application of scientific methods to assess the design, implementation, improvement, or outcomes of a program.

• It is a resource-intensive process, frequently requiring resources, such as, evaluator expertise, labour, time, and a sizeable budget.

Dr. De-Graft Johnson Dei, Dept of Information Studies Slide 3

EVALUATION

• A systematic determination of a subject's merit, worth and significance, using criteria governed by a set of standards.

• The critical assessment, in as objective manner as possible, the degree to which a service or its component parts fulfils stated goals'.

• The focus of this definition is on attaining objective knowledge, and scientifically or quantitatively measuring predetermined and external concepts.


EVALUATION

• It can assist an organization, program, project or any other intervention or initiative to assess any aim, realisable concept/proposal, or any alternative, to help in decision-making; or to ascertain the degree of achievement or value in regard to the aim and objectives and results of any such action that has been completed.

• The primary purpose of evaluation, in addition to gaining insight into prior or existing initiatives, is to enable reflection and assist in the identification of future change.


Purposes of evaluating

• Keen (A)

• the need for measures with which to make merit comparison within a single test situation. In other words, evaluation studies are conducted to compare the merits (or demerits) of two or more systems

• the need for measures with which to make comparisons between results obtained in different test situations, and

• the need for assessing the merit of real-life system.


Slide 6

Swanson (B) • assess a set of goals, a program plan, or a design prior to

implementation.

• determine whether and how well goals or performance expectations are being fulfilled.

• determine specific reasons for successes and failures.

• uncover principles underlying a successful program.

• explore techniques for increasing program effectiveness.

• establish a foundation of further research on the reasons for the relative success of alternative techniques, and

• improve the means employed for attaining objectives or to redefine subgoals or goals in view of research findings.


EVALUATION CRITERIA

• An evaluation study can be conducted from two different points of view.

• Management-oriented – Evaluation conducted from managerial point of view,

• User-oriented – Evaluation conducted from users' point of view.

• Many information scientist advocate that an evaluation

of information retrieval system should always be user-oriented, i.e. evaluators should pay more attention to those factors that can provide improved service to the users.


Slide 8

• Other factors: – Performance

• The system’s ability to retrieve relevant and only relevant items. – Cost

• how economically does a system perform – cost incurred per search – o users' efforts involved – o in learning how the system works o in actual use – o in getting the documents through back-up document delivery

system – o in retrieving information from the retrieved documents, and o

users' time – o from submission of query to the retrieval of references – o from submission of query to the retrieval of documents and the

actual information.


Slide 9

EVALUATION CRITERIA

• Here are some criteria that should keep in mind when you when you evaluate your information sources:

– currency,

– relevance,

– accuracy,

– authority and

– purpose.


Currency

• Currency refers to the timeliness of information.

• In some fields information becomes outdated quicker than in others, for example medicine and engineering can change quickly due to new research and information technology so you should always use the newest information available.

• Having said that nearly all fields have their classics that stand the tests if time where publication year isn’t an issue.

• Here are a few questions to consider when determining the currency of your source: – When was the information published or posted online?

– Has it been revised or updated?

– Are the sources used in the publication up-to-date?

– Are the links are functional?


Slide 11

Relevance

• Relevance refers to the importance of the information for your needs.

• The suitability level of a system to the information needs of users.

– General

– Subject/Specialised areas etc

• Here are a few questions to consider when determining the relevance of your source:

– Does the information relate to your topic or answer your question?

– Is the information at an appropriate level (not too elementary or too advanced)?

– Have you looked at a variety of sources before deciding on which to use?


Authority

• Authority refers to the source of the information.

• The publisher and the publication usually give some indication about the quality and reliability of the information, for both print and electronic materials.

• You should always be critical about self-published material as it doesn’t go through the same quality control as materials published through publishing houses.

• You should also pay attention to who the author is, when you are evaluating authority.


• You can determine if they are qualified to write about the topic by checking their level of education (someone who has just graduated is rarely an expert in their field and that’s why you shouldn’t use thesis as a source for your essays) or professional experience.

• You can also check if the author has published other books or articles in the same field.

• Some databases will also tell you how many times the article or book has been cited by others.

• A high number of citations usually tells that the article/author is well-known and respected in that particular field.


• Websites can also be credible and informative sources, for example organizations like the WHO.

• Remember to be critical about the information you find online.

• Here are a few questions to consider when determining the authority of your source: – Who is the author/publisher/source?

– What are the author’s credentials or organizational affiliations?

– Is the author qualified to write on the topic? Are they a professional or published academic or a lay person expressing their own views on the subjects?

– Does the URL reveal something about the website?


Accuracy

• Accuracy refers to the reliability, the truthfulness and the correctness of the content.

• You can use your prior knowledge to determine the accuracy of information or you can compare the information to other sources to see if it holds.

• Here are a few questions to consider when determining the accuracy of your source: – Where does the information come from? – Is the information supported by evidence? – Has the information been reviewed or refereed before

publishing? – Can you verify any of the information from another source? – Are there spelling, grammar or typographical errors?


Purpose

• The reason the information exists. • Information is produced for different purposes, it can teach,

entertain, sell, inform, and influence opinion. • It’s advisable to consider the agenda behind the information

that you are evaluation, as it gives you a better understanding about the quality of the information and whether or not its biased.

• Here are a few questions to consider when determining the purpose of your source: – What is the purpose of the information? – Is the information fact, opinion or propaganda? – Do the authors/publishers make their intentions or purpose clear? – Does the point of view appear objective and impartial, or is the

information sponsored/produced by someone with an agenda?


Cleverdon (1996) criteria for the evaluation of an information retrieval system.

1. Recall – the ability of the system to present all the relevant items

2. Precision – the ability of the system to present only those item that are relevant

3. Time lag – the average interval between the time the search request is made and the

time an answer is provided

4. Effort – intellectual as well as physical, required from the user in obtaining

answers to the search requests

5. Form of presentation of the search output, which affects the user’s ability to make use of the retrieved items, – Nature of output – bibliographic reference, abstract, or full text

6. Coverage of the collection, – the extent to which the system includes relevant matter


Vickery (D) identifies six criteria, grouped into two sets :

• Set 1

• Coverage =

– the proportion of the total potentially useful literature that has been analysed

• Recall –

– the proportion of such references that are retrieved in a search, and

• Response time –

– the average time needed to obtain a response from the system.

• These three criteria are related to the availability of information, while the following three are related to the selectivity of output.


• Set 2 • Precision –

– the ability of the system to screen out irrelevant references

• Usability – – the value of the references

retrieved, in terms of such factors as their reliability, comprehensibility, currency, etc., and

• Presentation – – the form in which search

results are presented to the user.

• Management-centered evaluation

• User-Centred Evaluation

– User base evaluation is the most common evaluation system advocated by many information scientists. A criterion for evaluation of information retrieval system includes:

• Recall

• Precision

• Fallout

• Generality


• Precision is defined as the proportion of documents retrieved that is relevant, while

• Recall is defined as the proportion of the total relevant documents that is retrieved.


Slide 21

Number of relevant item retrieved

Recall= ———————————————— x 100

Total number of relevant items in the collection


Precision= ——————————————————— x 100

Total number of items retrieved



Recall= ———————————————— x 100

Total number of relevant items in the collection

• Example, if there are 100 documents in a collection that are relevant to a given query and 60 of these items are retrieved in a given search, then the recall is state to be 60% in other words the system has been able to retrieve 60% of the relevant items.


Number of relevant item retrieved Recall= ———————————————— x 100 Total number of relevant items in the collection

• Recall measures the extent to which the retrieval of

wanted items occurs. • Whenever a user puts his/her query, it is the

responsibility if the system to retrieve all those items that is relevant to the given query.

• When the collection is large it is not possible to retrieve all the relevant items.

• Thus, a system is able to retrieve a proportion of the total relevant document in response to a given query.

Slide 24

Precision


Precision= ———————————————— x 100


• By precision we mean how precisely a particular system function.

• Precision is defined as the proportion of documents retrieved that is relevant.

• In precision the non-relevant items is discarded by the user.

Slide 25


Precision= ———————————————— x 100


• Example, if in a given search the system retrieves 80 items, out of which 60 are relevant and 20 are non-relevant, the precision is 75%.


Slide 26

Example

• Following example show the relationship between recall and precision of a given search.

• In a given situation a system retrieved a+b number of documents, out of which a documents are relevant, and b documents are non-relevant.

• For example, c+d document are left in the collection after the search has been conducted.

• The number will be quite large, because it represents the whole collection minus the retrieved documents.

• Out of the c+d number, c document are relevant to the query but could not be retrieved, and document are not relevant and thus have been correctly rejected.

• For a large collection the value of d will be quite large in comparison to c because it represents the entire nonrelevant document minus those that have been retrieved wrongly (b).

Slide 27

• R= [a/ (a+c)] x 100

• P= [a/ (a+b)] x 100


Relevant Not-Relevant Total

Relevant A (hints) B (noise) A + B

Not-Relevant C (misses) D (rejected) C + D

Total A + C B + D A + B + C + D

• Fallout

– Fallout ratio is the proportion of non-relevant items that has been retrieved in a given search.


Slide 29

• Generality

–Generality ratio is the relevant items that have been retrieved in a given search.

– Salton state that larger the collection, the larger will be the number of nonrelevant item in given query. Hence, an increase in the level of recall will cause a decrease in precision.


Slide 30

Symbol Evaluation measure

Formula Explanation

R Recall a/(a+c) Proportion of relevant items retrieved

P Precision a/(a+b) Proportion of retrieved item that are relevant

F Fallout b/(b+d)

Proportion of non-relevant items retrieved

G Generality (a+c)/(a+b+c+d) Proportion of relevant items per query


Slide 31

Lecture 9: Evaluation of Information Retrieval Systems · School of Information and Communication...

Documents

Transcript of Lecture 9: Evaluation of Information Retrieval Systems · School of Information and Communication...