Resolving Power of Search Keysin MedEval, a Swedish Medical Test

Collection with User Groups: Doctors and Patients

PhD thesis by Karin Friberg Heppin, Göteborgs Universitet

Opponent:Prof Dr Stefan Schulz Freiburg University (Germany)

‘Resolving Power’ of Search Keysin MedEval, a Swedish Medical Test

Collection with User Groups: Doctors and Patients

PhD thesis by Karin Friberg Heppin, Göteborgs Universitet

Opponent:Prof Dr Stefan Schulz Freiburg University (Germany)

Background

• More than fifty years of research in Information Retrieval (IR) • Importance of IR as a key technology for dealing with large

amounts of information in the era of the Internet • Most IR research is done in English content and standard texts

(mostly newswire)• Specific issues in

• Other languages: Swedish • Sublanguages / language registers: Medicine

Example

• Single-word compounds are common in Swedish texts (10%)• Compare two search expressions:

(1) “narkotikapolitik”(2) “fotboll”

• What happens if used in a trivial IR setting?• Does a search for (1) retrieve enough relevant documents?

• If single-word compounds are decomposed and the component used as search keys?• Does a search for “fot” AND “boll” still yield relevant documents?• Or does it only add noise?

Focus of thesis

• Resource production• a medical test document collection in Swedish

• IR research questions • What are the characteristics of good search keys in general ?• Can professional language characteristics be used for

optimizing target-group specific searches?• Are compounds good search keys or is it better to use their

constituents as search keys?

Organization of thesis

IR background: exhaustive review of the state of the art:models, evaluation, linguistics, medical IR

Test environment: tools, resources, creation of MedEval test collection

Pilot studies: investigation of the behavior of terms and groups of terms; analysis of the patients and doctor documents

Literature / Appendix

259pp.

Main hypothesis

The resolving power of search keys is dependent on their frequency in the document collectionThis should guide the decision whether to use decomposition of single-word compounds

How is the hypothesis being validated?

• Creation of a medical test collection• Running IR experiments

• pilot study• manual inspection and error analysis

Creation of a Test Collection

• Subset of MedLex with medical texts from different sourcestotalling 42000 documents

• Two indexes• original tokens (e.g. “saltkoncentration”)• tokens and constituents of compounds (e.g. “saltkoncentration”,

“salt”, “koncentration”• Document processing: tokenization, lemmatization, non-

lexical decomposition

Tools used

• Indri search engine: • inference network approach• produces ranked output• complex query syntax (several proximity parameters, individual

weighting, Boolean AND)• TREC eval: evaluation toolkit• Query performance analyzer

Example Query Performance Analyzer

Topic collection and relevance assessment

• 62 topics were acquired by medical students• relevance assessment done by pooling (suboptimal but only

feasible strategy with given resources):• interactive searching and judging

(four runs * pool depth 100) • four grades of relevance judgements • judgements of target readers: patients vs. physicians vs. both• adjusted relevance scores

Six different scenarios

Creating baseline queries

• Division of the terms of a query into facets (conceptual aspects) in order to assess the impact of query components

• Using words of the topics + Swedish MeSH synonyms• Example for facet:

TREATMENT = #syn(behandla behandling strategi behandlingsstrategi behandlingsmetod behandlingsalternativ tillvägagångssätt genomföra)

• Parameter for assessment: Normalized discounted cumulative gain (nDCG)

Analysis of the contribution of

• words • word fragments• facets

to the query performance (resolving power) measured in nDCG (normalized discounted cumulated gain)

Test of the suitability of single terms

nDCG

Measuring resolving power by removing facets

retrieves noisenDCG

Recall vs. noise

Quality of search keys (dependent on topic)

ineffective keys

effective keys

Conclusions• Ineffective search keys are more likely to be found among

terms with very high and very low frequency (statistical significant), but the effect is not very strong effect and there are important exceptions

• Low frequency compounds can benefit from decomposition• Only split compounds if the constituents have greater

resolving power than the compound• If the compound has a head – modifier structure only use the

head, as the modifier is supposed to have a low resolving power

• No clear message in which sense professional language characteristics can be used for optimizing target-group specific searches

Questions to the candidate

Question 1

It is interesting that early IR work you cited referred not to the parameter pair precision/recall, but specificity/sensitivity.Whereas sensitivity = recall = rel docs found / rel docs

specificity = nrel docs found / nrel docs

precision = rel docs found / found docs

Is there a reason why the mainstream IR research abandoned specificity? Is precision really an unproblematic parameter?Do you know recent work that reintroduced specificity in IR?

Question 2

The F-measure allows to assign different weights to precision and recall. This is important when different user scenarios are to be studied. There are scenarios in which recall is more important and the user accept noise because no relevant document must be missed. Would it be possible to express these user scenarios using nDCG?

Question 3

You used normalized discounted cumulative gain (nDCG) for measuring the resolving power of search expressionsWhy did you chose this parameter (and not the widely used F-measure)?

Question 4

You describe a Swedish stemmer that produces a 15 percent increase in precision. Stemmers are supposed to increase recall. How can they increase precision?

Question 5

I am surprised that a lexicon-free compound splitter, guided by indicative consonant sequences, yielded relatively good results.Do these results also hold for neoclassical compounds, which are typical for medicine (e.g. “cerebrovaskulär“)? Could the performance have been improved if you had combined it at least with a basic lexicon of medical terms?

Question 6

Could you explain a bit more the difference between a synonym set and a facet? If you include hyponyms what does this mean with compounds? Does this mean that a facet for cancer would look like this:CANCER = (cancer bröstcancer lungcancer pankreascancer kolorektalcancer tjocktarmscancer magsäckscancer …)

Question 7

It seems that the conclusions re the two language registers studies are less obvious compared to the analysis of the effectiveness of decomposition of compounds. What had you originally expected from the distinction between registers and what could still be done to achieve the expected result?

Question 8

The conclusions of your “pilot” studies are based on a careful dissection of the elements of topic descriptions in order to pick out characteristic features the terms to assess their usefulness. Nevertheless what you found out is still hypothetic.How could an empirical study be devised that provides stronger evidence of the validity of your conclusions?

Question 9

On page 34 you address reliability: why does reliability matter and how can you measure it? Has reliability be assessed in the construction of your test collection?

Question 10

Your analysis of multiword units seems a bit off-topic. Could you explain the rationale for this investigation and why this matters for the discussion of language registers?

Question 11

Your conclusion that single-word compounds should be treated in a differentiated way (according to semantic and statistic criteria) seems not so original because this is already obvious when observing the behaviour of Web search engines.

Which related work do you know that addressed this issues, which are their conclusions and what is the specific contribution of your own research?

Question 12

The test collection you have built is certainly a valuable resource for further research. Which kind of research can you imagine or would you like to be seen using your test collection?

Question 13