8/11/2019 1-s2.0-S104281430580010X-main

1/12

Knowledge Acquisition(1990) 2, 167-178

Th e e ff ic acy o f kn ow led ge e l i c it a t ion t e chn iques acompar i son ac ros s doma ins and l eve l s o f expe r t i s e

A . M . BURTON N . R . SttADBOLT G . RU GG AND A . P. HEDGECOCKDepartment of Psychology, University o f Nottingham, Nottingham N G 7 2RD, UK

Received 17Novem ber 1989)

De spite an increased interest in knowledge elicitation, there is still very little formalevidence evaluating the relative efficiency of the techniques available. In this pape rwe compare fo ur KE techniques: structured interview, p rotoc ol analysis, card sortand laddered grid. Studies are re por ted across two classification domains, using eightexperts in each. Despite its common usage, protocol analysis is shown to be the leastefficient technique. Th e implications o f this finding are reviewed. Finally, a study isreported in which non-experts are subjected to know ledge elicitation . Subjectsentirely ignorant of a domain are able to construct plausible knowledge bases fromcommon sense alone. The ramifications of these findings for knowledge engineers isdiscussed.

n t r o d u c t i o n

Over t he p a s t tw o o r t h r ee yea r s , i n t e r e s t i n knowledge acqu i s i t i on ha s i nc r ea sedrad ic a ll y. I t is gene ra l ly a c c ep t ed i n t he expe r t sy s t ems com m un i ty t ha t know ledge

acqu i si ti on is a c o n s i d e r a b l e p ro b l em, and t ha t r e s ea r ch i s nee ded t o deve lopeff ic ien t t echn iques fo r acqui r ing codi f iab le kno wle dge . In resp ons e to th is , the re i sno w a l a rge l i te r a tu r e de sc r i b ing a ve ry l a rge nu m ber o f t e chn iques i n de ta i l ( e . g .Ga ine s & B oo se , 198 8; Bo os e & Ga ines , 1988 ) . Fu r the rm ore , cons ide r ab l e e f fo r tha s b ee n i nves t ed in a u toma t i ng so me o f t he se te chn iques (Bo ose , 1989 ), and i nt a il o ri ng t he m t o sp ec i fi c dom a i n s (M arcus , M cD erm o t t & W ang , 1985; K l inke r,B e n t o l i l a , G e n e t e t , G r i m e s & M c D e r m o t t , 1 9 8 7 ) .

D e s p i t e t h e l arg e- sc a le i n v e s tm e n t i n d e v e l o p m e n t o f t e ch n i q u e s f o r k n o w l e d g eacquis i t ion , there has been surpr i s ing ly l i t t l e e ffor t inves ted in eva lua t ing thee ff e c ti ven es s o f t h e t e chn i ques . I t i s o f t en a s se r t ed t ha t d i f f e ren t t e chn iques shou ldsu it d if f e re n t d oma in s , d i f f e ren t t ypes o f know ledge en g inee r and ex pe r t , an dd i f fe r e n t t ypes o f e xpe r t sy s t em a r ch i t e c tu r e . H ow eve r, t h is a s s e r t ion i s ve ry s e ldom ,b a c k e d w i t h e m p ir ic a l e v i d e n c e . I f th e t e c h n i q u e s u n d e r d e v e l o p m e n t a r e t o b eu s e fu l t o t h e e x p e r t s y s t e m s c o m m u n i t y, t h e n d e v e l o p e r s m u s t a d d r e s s t h e m s e l v e sto the ques t ion : und e r w ha t c ir c ums t ances w il l pa r t i cu l a r t e chn iques be mos t u se fu lfo r knowledg e acq u i s i t i o n?

In th i s pap e r w e p r e sen t a co m pa r i son o f f ou r kno wled ge e l i c it a t ion t e chn iquesa c ro s s t w o d o m a i n s . T h e w o r k p r e s e n t e d h e r e f o r m s a d e v e l o p m e n t o f p r e v i o u s l ypu b l i s he d w or k i n th i s a r ea , an d so we w i ll b r ie f l y ou t l i ne ou r p r ev ious r e su l t s ,be fo r e de s c r i b i n g t h e p r e s e n t s t u d y i n de t a i l.

I n p r e v i o u s e x p e r i m e n t a l w o r k , w e h a v e c o m p a r e d f o u r k n o w l e d g e e l i c i t a t i o n

t echn iques : s t r uc t u r e d i n t e rv i ew, p ro toco l ana ly s i s , l adde red g r i d and ca rd so r t s .T h e f ir st tw o o f t h e s e t e c h n iq u e s a r e c o m m o n l y u s e d i n e x p e r t s y s t e m d e v e l o p m e n t ,

167

1042-8143/90/020167 12503 00/0 1990 Academic Press Limited

8/11/2019 1-s2.0-S104281430580010X-main

2/12

168 A M BURTON E T A L

and we wi l l l abe l themtr dition l i n wha t f o l l ows . The l a t t e r two t e chn iques a r ei n t en d e d t o r e v e a l a n e x p e r t ' s " c o n c e p t u a l m a p " o f a d o m a i n , b u t d o s o in w a y swh ich a r e l i k e l y t o be un f ami l i a r t o t he expe r t . Fo r t h i s r e a son we l abe l l adde r ing

and ca rd so r tcontrived t e c h n iq u e s . F o r t h e p u r p o s e o f p re v i o u s e x p e r im e n t s , a n d o ft h e n e w w o r k p r e s e n t e d h e r e , w e h a v e d e v e l o p e d s ta n d a r d i z e d v e r s i o n s o f e a c h o ft he se t e chn i ques (Shad b o l t & Bu r ton , 1990 ) . The s t and a rd i zed ve r s i ons have bee nused i n cons t ruc t i o n o f com m erc i a l sy s t ems , a s we l l a s i n l abo ra to ry ex pe r imen t s . I nsummary, t h e s t r uc tu r ed i n t e rv i e w compr i s e s a p l anned s e s s ion i n wh ich t hei n t e rv iew er i s r e s t r ic t e d t o a s m a ll num be r o f p romp t s . Th e p ro toco l ana ly s i s cons i st so f th e e xp e r t so lv i ng a p rob l em in f ron t o f t he e l i c it o r, and be ing a sked t o " t h inka loud" du r ing t h i s p r oc e s s . The l adde red g r i d i s a t e chn ique i n wh ich t he e l i c i t o rco ns t ru c ts a we l l -de f i ned s tr uc tu r ed r ep re sen t a t i on o f t he do ma in i n co l l abo ra t i onwi th t he exp e r t . Th i s c an be d r aw n a s a g raph i ca l r ep r e sen t a t i on o f t he dom a in . T heca rd so r t i nv o l ve s t he expe r t r e p ea t ed ly so r t i ng i n to mean ing fu l p i l e s a deck o fc a rd s , o n e a c h o f w h i c h is m a r k e d t h e n a m e o f a d o m a i n e l e m e n t . S h a d b o l t a n dBu r ton (19 90 ) d e s c r i be ea c h o f t he se t e chn iques i n f u ll , w i th exam p le s f r om thes tandard ized vers ions .

Fo r t he pu rpo se s o f f o rm a l compa r i son , i t i s nece s sa ry t o compa re t he setechniques acrossm n y ex pe r t s . T h e re a r e c l e a r l y i d io sync ra t i c f e a tu r e s a s soc i a t edwi th any i nd iv i dua l KE se s s ion w i th any i nd iv idua l expe r t . I n o rde r t o p rov idegener liz ble r e su lt s , i t is nece s s a r y to pe r fo rm mu l t i - expe r t s t ud i e s wh ich t r an scendthe se i d io sync ra s i e s . F o r t h i s r e a son , two l a rge - s ca l e expe r imen t a l compa r i sons o ft e c hn iques we r e p e r fo r m e d , e a ch on 32 expe r t s , w i th in t he do ma ins o f igneous rockide nt if ic a t ion (B ur to n , Sh ad b o l t , H edg eco ck & Rug g , 1987) and g l ac ia l f e a tu r e

ide nt if ic a t ion ( B ur ton , Shadb o t , Rug g & He dge coc k , 1988) . I n o rde r t o r ende rthe se l a rg e - s ca l e s t ud i e s f e a s ib l e , t he doma ins we re s eve re ly r e s t r i c t ed . Fo r e achs t u d y, a n e x p e r i e n c e d p r o f e s s o r c o n s tr u c t e d a s u b - s e t o f t h e d o m a i n w h i c h w o u l d b ewe l l -known b y a su b j e c t po o l o f s t uden t s . H en ce t he su b j ec t s in the se exp e r imen t swe re expe r t s o n ly in d o m a in s o f v e ry l im i t ed s cope .

Th e r e su lt s o f t h e se s t u d i e s p r o v ide d a num ber o f s ta t is t ic a l ly r e l i ab l e r e su lt s . Thep ro to co l ana ly s is was t he l e a s t e ff ec t ive o f a ll t e chn iques s t ud i ed . T he con t r i vedt echn iques t ook l e s s t ime t o a dm in i s t e r and code t han t he i n t e rv i ew, bu t p rov ideda b o u t t h e s a m e a m o u n t o f in f o rm a t i o n. H o w e v e r, p r o t o c o l a n a ly s is t o o k l o n g e r, a n dp rov ided a s m a l l e r c o ve r ag e o f t he doma in t han any o the r t e chn ique t e s t ed . Th eimpo r t ance o f th i s r e su l t is h ig h l igh t ed b y t he ve ry h igh f r eque ncy w i th w h ichp ro toco l an a l y s i s i s u se d , a nd t he ve ry l ow f r equency w i th wh ich con t r i vedt e chn i ques a r e u sed by knowle d g e eng inee r s (Cu l l en & Brym an , 1988) .

The sub j ec t s u sed i n t h e se ex pe r imen t s we re i ndeed expe r t i n t he sma l l doma insunde r s t ud y. The r e w e r e v i r tua l ly no e r ro r s m ade i n any o f t he K E se s s ions u sed i nthe expe r i me n t s . H o w ev e r, i t is po s s ib l e tha t t hey have d i f f e ren t cha r ac t er i s ti c s t o apo pu l a t i on o f ex pe r t s t yp i ca ll y u s ed i n know ledge acqu i s i ti on . Ou r sub j ec t s we reyoung , a r t i c u l a te , an d po s s ib l y q u i te u sed t o be ing p l aced i n "o dd " s i t ua ti ons i n t hecou r se o f th e i r e d uc a t i o n . I t is n ece s sa ry t hen , t o e s t ab l ish w he the r t he se r e su lt sapp ly t o a popu l a t i o n o f g enu i ne e xpe r t s . I n t h i s pap e r w e de sc r i be s im i l a r s tud i e s t ot h o s e d e sc r i b ed a b o v e , b u t t h is t im e p e r f o r m e d o n " a d u l t " , r e al e x p e r ts . O f c o u rs e ,

i t is no t pos s i b l e t o t a ke t he t ime o f 32 rea l -wor ld expe r t s i n a hom oge neo us do ma in .In each do ma i n t hen , we h ave s t ud i ed j u s t e i gh t expe r t s . Th i s does no t a l l ow us t o

8/11/2019 1-s2.0-S104281430580010X-main

3/12

COMPARING ELICITATIONTECHNIQUES 169

make s t a t i s t i c a l c ompa r i sons o f t he k ind pos s ib l e i n p r ev ious s t ud i e s . Howeve r, i t i sr a t he r m o r e r e l i ab le t ha n t h e s i n g l e - expe rt c a se s t udy com m on in t he l i t e r a tu r e ( e . g .Smi th & Ba ke r, 19 83 ; Bu t l e r & Cor t e r, 1986 ; Schwe icke r t , Bu r ton , Tay lo r, Co r l e t t ,

Shadb o l t & H e dg e co c k , 1 98 7 ). I n sho r t , ou r a im in th i s s t udy is t o d i s cove r w he th e rt he r e su l t s f r o m p re v iou s l ab o ra t o ry -ba sed s t ud i e s a r e l i ke ly t o app ly when dea l i ngwi th fu ll y f l ed g e d e xp e r t s in t he c on t ex t o f r e a l wor ld know ledge eng inee r i ng . I neffec t we a re t ry ing to es tab l i sh th e eco log ica l va l id i ty o f our p rev ious s tud ies .

Exper imen t

DESCRIPTION O F DOMAINS

The tw o dom a i ns o f i n t e r e s t i n t hi s s t udy we re bo th a r chaeo log i ca l : t h e i den t i fi c a ti ono f f li nt a rt e f a c ts f r o m S t on e A g e t oo l p rodu c t i on , and t he i den t if i c a ti on o f po t t e ry

she rds f r om t h e m e d i ae va l pe r i o d o f Eng li sh h i s t o ry ( rough ly 1066 t o 1485 ). I ncommon wi th p r ev io us s t u d i e s , t he se a r e bo th doma ins r i ch i n c l a s s i f i c a t i onk n o w l e d g e .

Flint domah~ Analys i s o f f l in t a r te fac t s i s a wel l -es tab l i shed f ie ld in a rchaeology.Fo r t he pu r po s e s o f t h i s s t udy, we cons t ruc t ed a s e t o f s i x t een a r t e f ac t s w i th t hea s s is t ance o f s e v e ra l ex p e r t s . T h ese w e re chosen t o be a r ep r e sen t a t i ve s am p le o fflin ts . Eac h o f the se a r t e f a c t s i s e i t he r a t oo l , o r a by -p rod uc t o f t oo l p roduc t i on ,wh i ch h a s a gene r i c c la s s if ic a ti on i n the ap p rop r i a t e l i te r a tu r e . The se i t ems cove r t hep a l ae o li t h ic , m eso l it h i e a nd n eo l i th i c pe r i ods , t he t h r ee p e r i ods i n to wh ich t he S to neAg e i s u sua l ly d iv ided . A l l we re Br i t ish , and non - r eg iona l , t o avo id b i a se s be twe enexpe r t s fami l ia r wi th par t i cu la r reg iona l spec ia l i ti es . A ny e xp er t in th i s f i e ld w ou ldb e ex p ec t ed t o b e f ami l ia r w i t h a ll t h e se a r t e f ac t s .

Pottery domain ana ly s is o f po t t e ry she rd s o f t en t ake s p l ace a s pa r t o f ana r chaeo log i ca l ex cava t i on . E xpe r t s u se i n fo rma t ion f rom the se ana ly se s t o he lp t hemes t ab l i sh i n fo r m a t ion abou t t h e s i t e unde r s t udy ( e . g . t r ad ing p r ac t i c e s ) .D iagnos t i c d im en s i ons i nc l ude su ch f ac to r s a s t he co lou r o f t he f ab r i c , t he g l azeu sed , t h e t h i ck ne s s o f t he p o t wa l l , and so on . A s e t o f 16 she rd s was a s sem b led w i ththe he l p o f e xp e r t s in po t t e r y i de n t if i ca t ion . T hese we re cho sen a s a r ep r e sen t a t i ves amp le o f me d i aev a l po t t ype s . So , f o r exam p le , h a l f we re g l azed , t he r e was amix tu r e o f c oa r s ew a re s an d f i n e ware s , and t he r e was a sp r ead o f exam p le s w i thd i ffe ren t sur face decora t ions and mater ia l inc lus ions .

SUBJECTS

Eigh t f l i n t ex pe r t s an d e ig h t po t t e ry expe r t s ag r eed t o t ake pa r t i n t he expe r imen t .The 16 expe r t s w e re a l l p ro f e s s io n a l a r chaeo log i s t s engaged i n a cademic , museum o rs imi l a r wo rk , a n d sp ec ia l iz i ng in one o f t he dom a ins u nde r s t udy.

METHOD

T h e f o u r K E t e c h n i q u e s u n d e r s t u d y w e r e : s t r u c t u r e d i n t e r v i e w ; p r o t o c o l a n a l y s i s ;l a d d e r e d g r i d ; a n d c a r d s o r t . A s a l r e a d y i n d i c a t e d w e l a b e l t h e fi rs t t w o o f t h e s et e chn iques t r a d i t i on a l , and t h e l a st two con t r i ved . Ea ch expe r t t oo k pa r t i n two

knowledge e l i c i t a t i on s e s s i ons , o n e u s ing a t r ad i t i ona l , and one u s ing a con t r i vedt e c hn i q u e . F o r e a c h d o m a i n t h e n , t w o e x p e r t s w e r e s u b j e c t t o e a c h o f t h e f o u r

8/11/2019 1-s2.0-S104281430580010X-main

4/12

17 A M. BURT ON E T A L

possible combinations of techniques. This design provides four elicitation sessions ofeach type in each of the two domains. Experts were randomly allocated to theseconditions, and the order of testing was counter-balanced.

In the Structured Interview condition, experts were subject to the standardizedinterview referred to above. The protocol analysis consisted of experts being given arandon subset of the domain items (chosen independently for each session), andasked to think aloud while deciding on its classification. Similarly, in theLaddered Grid condition, experts were asked a series of questions while the elicitorsystematically explored the domain space. The Sorting Condition was different forthe two groups of experts. The flint experts were asked repea tedly to sort cards eachbearing the name of one of the sixteen domain elements. However, the potteryexperts were asked to sort not c a r d s but s h e r d s the artefacts themselves. This allowsus to compare the two variants of the sorting task within the experiment.

The choice of dependent variables for this type of study is always problematic.One needs to establish measures of effort taken in each session, and measures of

gain . As with previous studies (Burton e t a l . 1987; 1988), effort was measu red bythe time taken for each KE session and the time taken to transcribe and formulateeach session into pseudo-English production rules. This form of intermediaterepresentation (Young, 1987) has an IF, AND, TH EN structure, but is not of adirectly implementable form. We describe elsewhere the method by which raw datais coded into this intermediate representation (Shadbolt & Burton, 1990). Themeasure of gain was the number of clauses present in the transcribed rules. A clauseis defined as one conditional statement in a pseudo-English production rule. This isclearly a very broad measure of the amount of information elicited in each session.

An attempt to refine this measure is described below in Experiment 2.

RESULTS AND DISCUSSION

The effects on the main dependent variables are shown in Table 1. We will considerthe results from the flints domain first. It is clear that in this domain, the contrivedtechniques take considerably less time to administer than the traditional techniques.This disparity is also present in the time taken to transcribe t he KE sessions into theintermediate representation. However, for this latter variable, protocol analysis isseen to take much longer than its comparable technique, the interview. Considering

TABLEM e a n s c o r e s f o r s u b j e c ts in e a c h d o m a i n n = 4 p e r g r o u p

Domain Flints PotteryTechnique Interview Prot An L Grid Sort Interview Prot An L Grid Sort

Time to KE(min) 59 57 36 25 47 50 38 54

Transcriptiontime (min) 159 294 109 74 193 126 139 91

Total time 217 351 145 98 240 176 177 145No. clauses 270 269 188 123 317 184 278 216Clauses min -~ 1.2 0.8 1-3 1-3 1-3 1.0 1.6 1.5

8/11/2019 1-s2.0-S104281430580010X-main

5/12

COMPARING ELICITAT1ONTECHNIQUES 171

the ga in m e a s u r e , w e c a n s e e t h a t t h e t r ad i ti o n a l t e c h n i q u e s y i e ld m u c h m o r ein fo rma t io n t h an t he con t r i ved t e c hn iques . I n t e rms o fe ffo r t fo r ga ini t c an be s eentha t p ro toco l ana lys i s i s the l eas t e ff ic ien t t echnique , whi le the remain ing techniques

p e r f o r m c o m p a r a b l y.~ The r e su l t s f o r t he po t t e r y dom a in show a ra the r d i f f e r en t pa t t e rn . The t ime t akenfo r a know le d ge e l i c i ta t ion s e s s ion i s rough ly equ a l f o r i n t e rv i ews , p ro toco l ana ly se sand so r t s , w i t h l a dde red g r i d s t ak ing a sho r t e r pe r i od . The d i f f e r ence i n t he so r t i ngre su l t s b e tween t he t w o d oma i ns i s a lmos t c e r t a i n ly due t o t he d i f f e r en t t a sk sp re sen t ed i n e ach d oma in ( e . g . c a rd so r t ve r su s she rd so r t ) . I n t h i s doma in , t hesma l l es t num be r o f c lau s e s i s de l i ve r ed by t he p ro toc o l ana ly si s . The e f fi c iencyscores , as wi th the f l in t domain , show tha t p ro toco l ana lys i s i s the l eas t e ff ic ien tt e chn ique , wh i l e t h e c o n t r i ve d t e chn ique s a r e t he m os t e f f ic i en t .

I n sum mary, p r o t oc o l a na ly si s p e r fo rms t he l e a s t e f fi c ien t ly i n bo th t he se do ma ins .Th i s t he s am e pa t t e r n o f r e su l t s a s r epo r t ed i n t he l a rge s ca l e expe r imen t a l s t ud i e sd e s c r ib e d a b o v e ( B u r t o net al . 1987; 1988).

I n add i t i on t o t h e se m eas u re s , i t i s a l so pos s ib l e t o exam ine t het ype o f k n o w l e d g ee l i c i t ed by t e chn iq ues . R esu l t s f r om bo th doma ins show a ve ry sma l l ove r l apbe tween t h e two s e s s ions w i th t h e expe r t . I n t he f l i n t doma in , t he r e was an ave rageo f on ly 10% o v e r l ap be t we en kno w ledge ga ined f rom a t r ad i t iona l and a con t r i vedt e c hn i q u e . I n t h e p o t t e r y d o m a i n , f iv e e x p e r t s p r o v i d e d n o o v e r l a p b e t w e e nse s s ions , a nd t h e r e m a in ing t h r e e expe r t s p rov ided on ly a ve ry sma l l amoun t . Th i ssugges ts tha t t h e t e c h n i q u e s p r ov i de d i f f e r en t t ypes o f know ledge . A l though fu r t he rs t ud i e s wo u l d be n ec e s sa ry t o con f i rm th i s r e su l t , i t s ugges t s t ha t knowledgeeng inee r s shou ld co ns ide r u s in g t he se con t ri ved t e chn iques a s a supp l em en t t o t he

t r ad it i onal t e chn i que s . I n t he s e s t ud i es , no t on ly a r e t he co n t r i ved t e chn iquescompara t i ve ly e f f i c i e n t , t h ey a l so add t o knowledge ga ined f rom t r ad i t i ona ltechniques .

A p rob l em w i t h t he p r e se n t s t udy is t he l a ck o f a r igo rous m easu re o f i n fo rma t ionga ined . I n c l o se d -wor l d d oma in s such a s t hose u sed fo r ou r p r ev ious expe r imen t a ls tu d ie s , o n e ca n ta k e a m e a s u r e o f c o v e r a g e o f a k n o w n g o l d - s t a n d a r d r u le b a s e .In the presen t case th i s i s no t poss ib le . I t i s qu i te poss ib le tha t the coding ofi n fo rma t ion by c l au se s i s m i s l ead ing a s a me asu re o f ga in . Fo r t hi s r e a son , afu r t he r s tud y wa s pe r fo rm e d i n one o f t he dom a ins . I n fo rm a t ion cod ed a s Eng l i shru l e s was pa s sed b ack t o su b j ec t s who had t aken pa r t i n Expe r imen t 1 , and r a t ed bythem on a nu m be r o f c r it e ri a .

Experiment

METHOD

Al l the ex pe r t s f r om o ne o f t h e doma ins u s ed i n Expe r im en t 1 ( fl in ts ) we recon t ac t ed and a sked i f t h e y w o u l d b e w i ll ing t o t ake pa r t i n a f o ll ow up . F ive o f t heo r ig ina l e i gh t su b j e c t s ag r e e d t o p roc eed . Each o f the se expe r t s was p r e sen t ed w i ththe cod i f ied ru l e s e t g ene r a t e d f r om each o f t he i r two kn ow ledge e l i c i ta t ion s e s s ions .Th ey we re a sk ed t o r e ad t h rough t he se ru l e s e ts and t o c a t ego r i ze e ach c l ause as

true trivial garbledo r fa lse . T h e t rue and f a l se ca t ego r i e s a r e s e l f - exp l ana to ry. Thedef in i t ion oftrivial given i s tha t the c lause i s no t re levan t to the d iagnos i s .Garb led

8/11/2019 1-s2.0-S104281430580010X-main

6/12

172 A.M. BORTOr~ er AL.

was defined as being partially true, i.e. the clause would become true if somemodification was made. Although these definitions are rather informal, the expertswere perfectly able to understand and use them consistently.

After this coding had been made, subjects were asked to make one finalcontribution. Having seen the rule set derived from their original KE sessions, theexperts understood the form of the intermediate representation device used in thisstudy. They were now asked to construct such a rule base from scratch. They wereinstructed to present rules which would cover the domain under study. These newrule sets were then compared with the original rule sets generated from KE sessionsand measures of overlap were computed.

RESULTS AND DISCUSSION

The results of the rule-set coding were averaged by the technique used to elicit theinitial rule-set, and the means of this analysis are presented in Table 2. The resultsshow that very few clauses were rated as false for any of the techniques. Theladdered grid and interview techniques provided similar distributions of codes forclauses, with around 60% of the originals being rate d as true. The protocol analysisand card sort techniques both produced fewer true clauses, though the distributionof codes for remaining rules was different in the two cases. For protocol analysis, themajority of clauses not rated true were classified as garbled (32%), whereas for cardsort, most clauses not rated true were classified as trivial (44%). The garbled natureof the rules from the protocol analysis is probably due to the fact that raw data fromthis technique is difficult to interpret. The transcripts are often rambling, ungram-

matical, and give the appearance o f being gar bled . It is possible that theinefficiency of protocol analysis is due not to features of the session itself, but to thedifficulty of formulating data from these sessions into a suitable represent ation. Inshort, there may be more in a KE session than a knowledge engineer can extract.This result has important implications for the treatment of raw data from KEsessions. Work on auto mate d conce pt edito rs (e.g. Anjew ierde n, 1987) iscurrently aimed towards making this part of the knowledge engineering processmore efficient.

Table 3 shows the overlap scores between the elicited rule base and the rule baseconstructed directly by experts, averaged by the techniques of the originalelicitation. Two overlap scores are necessary as the knowledge bases are of differingsizes. We therefore take the proportion of rules from the KE session (lst pass)which also appear in the direct transcription (2nd pass), and also the converse. The

TABLE 2Experts coding o f the rule set generated from their own first pass elicitation

session. Mean percentages (n between 2 and 4 per group)

Code(%) True Trivial Garbled False

Interview 63 22 8 6Protocol analysis 46 17 32 5Card sort 43 44 7 6Laddered grid 55 22 17 6

8/11/2019 1-s2.0-S104281430580010X-main

7/12

COMPARING ELICITATIONTECHNIOUES

TABLE 3Comparison o f experts first and secon d pass rule sets. M ean

percentages (n between 2 and 4 per gro up)

173

Ov erlap ( ) 1st in 2nd pass 2nd in 1st pass

Interview 54 33Protocol analysis 25 11Card sort 29 33Lad dered grid 45 33

t ab l e c l e a r l y s ho w s t ha t p ro t o co l ana ly s i s p rov ide s t he sma l l e s t ove r l ap w i thsubsequen t l y e l i c i t e d knowl ed ge , c a rd so r t t he nex t l owes t , wh i l e l adde red g r i d andin t e rv i ew pe r fo rm ro ugh l y equ i va l en t l y.

The re a r e t w o po s s i b l e r e a s o ns fo r t he r e l a t i ve ly sma l l ove r l ap be tweenin fo rma t ion g a ine d f ro m the p ro t oco l ana ly s i s and f rom the d i r ec t cons t ruc t i on o fru les . F i r s t , i t i s poss ib le tha t the pro toco l ana lys i s i s s imply a poor t echnique in th i sdo m a in , an d t h a t a r e l a ti ve ly s ma l l am oun t o f i n fo rma t ion i s p i cked up . Se cond , i t i spo s s i b l e t h a t t h a t p ro toc o l ana l y si s p i cks up i n fo rm a t ion wh ich i s no t o f t he t ypesu i tab le fo r e li c i t a tion b y d i rec t e l i c i t a tion . W e fav ou r the f i rs t o f these ex plana t ions ,a s i t s e ems t o be i n kee p ing w i t h t he r e su lt s ga ined i n Exp e r im en t 1 .

Bo th t he man ipu l a t i ons made i n Expe r imen t 2 sugges t r e su l t s cons i s t en t w i thp rev ious ex pe r ime n t s . O n ce a ga i n, w e have dem on s t r a t ed t he r e l a t ive i ne ff ic i ency o fp ro toco l a na ly s is a n d t h e r e l a ti v e e f fi c acy o f t he two con t r i ved t e chn iques und e rs tudy. W e wi ll re tu rn to a genera l d i scuss ion of th i s find ing a t the en d o f the p ape r.Howeve r, we w i l l f i r s t cons id e r ano the r f a c to r con t r i bu t i ng t o t he p rob l em o f KE-the leve l o f exper t i se .

xpe r t s and nov ice s

The s t ud i e s d e sc r i bed i n Expe r im en t s 1 and 2 we re pe r fo rmed on sub j ec t s spec i a l l ychosen t o be e x p e r t s in t h e i r d o m a ins . F ro m the se s t ud i e s we have been ab l e t od raw co nc lu s ion s ab ou t t he r e l a ti ve e f fi c acy o f K E t echn iques . I n t h is s ec t i on weco ns id e r t h e e f f ect o f e xp e r t i s e i t s e l f on t he se t e chn iqu es . W e wi ll de sc r i be r e su l tsf r o m a s t u d y i n w h i c h t h e s a m e e x p e r i m e n t s w e r e r e p e a t e d o n c o m p l e t e n o v i c e s i nthe s ame d o m a i n . T h e p u r po se o f t he s t udy was twofo ld , f i rs tl y t o d i s cove r t heb a s e li n e p e r f o rm a n c e o f e a c h o f t h e t e c h n i q u e s, a n d s e c o n d l y t o d i sc o v e r w h e t h e rt he t e chn iq ue s o f f e r any r e l iab l e w a y o f d i s ti ngu ish ing be tw een expe r t s and nov i ce s .B e fo r e de sc r i b in g t he s t u dy , w e w i ll f ir st d i s cuss t he ba ckg ro und beh ind t he se i s sue s .

I t is pos s i b l e t h a t w h a t w e ha ve d i s cove red i n Expe r im en t s 1 and 2 , and i np rev ious expe r ime n t s , i s a p a t t e rn o f t he dep end en t m easu re s i nhe ren t i n t het echn ique s t h em se lv e s . I t may b e , f o r exam p le , t ha t p ro to co l ana ly s is w il lalwaysde l i v e r i n fo rma t io n i n a l e ss ef f ic i en t way t han o the r t e chn iques , no m a t t e r w he th e ri t i s u sed fo r kn owled ge e l i c i t a t i on , f o r t a sk ana ly s i s o r wha t eve r. A l t e rna t i ve ly, i tco u ld b e t ha t w e have d i s co ve r ed a pa t t e rn o f e f fi c acy o f t e chn iqu es w h ich i s spec if ict o t h is c la s s o f d om a in , w i th t h is c l as s o f expe r t . I n o rd e r t o d i s t ingu i sh be tw een

the se two hypo t he s e s i t i s n ec e s sa ry t o e s t ab l i sh abaseline eff ic iency for theset echn ique s , t he equ i va l e n t o fchance performa ncei n o the r expe r imen t a l s t ud i e s .

8/11/2019 1-s2.0-S104281430580010X-main

8/12

74 A M BURTON E T A L

The e s t a b l is h m e n t o f b a s e l ine pe r fo rm ance w i ll a l so be u se fu l t o know ledgee l ic i to rs in new domains . I t i s poss ib le tha t the techniques wi l l g ivenothingw h e n t h esub j ec t kno w s no t h i n g a b ou t t h e dom a in u nde r s t udy. A l t e rna t i ve ly, i t is pos s ib l e

tha t an i n te l li g en t su b j e c t cou l d m ake p l aus ib l e gues se s abo u t t he s t r uc tu r e o f thed o m a i n f r o m c o m m o n s e n s e a l o n e. I n t h e s u r v e y p u b l i sh e d b y C u l l en a n d B r y m a n( 19 8 8 ), b y fa r th e m o s t c o m m o n l y r e p o r t e d p r o b l e m a s s o c ia t e d w it h k n o w l e d g ee l ic it a t ion wasquality of expertise( 4 7 % o f r e s p o n d e n t s r e p o r t e d t hi s p r o b l e m ) . I n as ense t h i s i s no t su rp r i s i n g , and t a l l i e s w i th ou r own expe r i ence . Managemen tinvo lved i n su pe r v i s i ng t he co ns t r uc t i on o f an expe r t sy s t em a r e o f t en unwi l l i ng t odo na t e t h e t ime o f t he m os t e ff ic i en t expe r t . I t is o f t en t he expe r i ence o f know ledgee n g i n e e r s t h a t t h e y a r e f a c e d w i t h a n e x p e r t w h o h a s b e e n p r o m o t e d a b o v e t h e j o bwh ich fo rms t he ba s is f o r t he s y s t em.

Fo r th is r e a s o n , i t i s i m p or t a n t t o d i s cove r wh e the r t he ba se l i ne pe r fo rma nce onthe se t e c hn iq ues i s ve ry l ow, o r whe the r ba se l i ne pe r fo rmance wou ld ac tua l l ya p p e a r t o c o n ta i n u s e fu l k n o w l e d g e t o s o m e o n e n e w t o t h e d o m a i n . F u r t h e r m o r e , i fi nd eed t h e r e i s a h igh b a s e l i ne a s s oc i a t ed w i th t he t e chn iques , i t wo u ld be u se fu l tok n o w w h e t h e r a n y o f t h e m c o u ld f u n c ti o n a s a n e x p e r t s p o t t e r ( o r p e r h a p snon -expe r t s po t t e r ) . I f t h e r e w e r e a s tanda rd i zab l e wa y o f d i s c rim ina ti ng on pu re lysy n t ac t ic g ro unds b e tw een g enu in e expe r t s , and t hose w ho cou ld mere ly cons t ruc t ap l aus ib l e s t o ry a round t h e do m a in , t h i s wou ld be o f bene f i t t o t he knowledgeeng inee r i ng c omm un i ty.

W i t h b o t h t h e s e c o n s i d e ra t i o n s in m i n d , w e r e p e a t e d t h e s t u d y d e s c r ib e d a b o v e int he f li nt dom a in w i t h c om p le t e n ov i ce s i n t h is doma in . A l tho ugh t h is may so undb i za r re , i t is i n fa c t no t a h op e l e s s t a sk . F o r exa m p le , o ne o f t he dom a in e l em en t s

he r e was piano-convex knife.Wh i l e nov i ce s wou ld a lmos t c e r t a i n ly no t know thet echn ica l m ea n i ng o f th i s t e r m , m os t wou ld be ab l e t o cons t ruc t a p l aus ib l edesc r ip t i on o f t he a r t e f a c t . S imi l ar ly, two o f t he e l em en t s w e rebur in andmicroburin.W h i l e m o s t s u b j e c ts d id n o t k n o w t h e s e t e rm s , t h e y w e r e o f t e n a b l e t ohypo the s i z e a r e l a t i ons h i p be tw een t hem. In t h i s c a se , t he pa r t i cu l a r d i s c r im ina -t ion i s d iagnos t ic o f exper t i se , as aburin i s someth ing qu i te d i ffe ren t to amicroburin a n d t h e y d o n o t b e a r t h e c o m m o n s e n s e re l a ti o n s hi p i m p l ie d b y t h e irnames . W e n ow d esc r i be t he ex p e r imen t a l de t a i ls o f th i s s t udy.

Experiment

M E T H O D

Eigh t sub j e c t s w e r e r e c r u i t ed w h o had no exp e r i ence o r i n t e r e s t in a r chaeo logy. Thesubjec t s w ere a l l Un ivers i ty resea rchers in d i ffe ren t d i sc ip l ines , and s o i t can bea s sumed t ha t t h e y we re g e n e ra l l y ma t ched , on fo r examp le gene ra l i n te l l igence , t othe s amp l e u sed i n Exp e r imen t 1 .

Sub j ec t s w e re to l d t h a t t he y w e r e t o t ake pa r t in a P sycho log i ca l Ex pe r im en t , i nwh ich the i r t a sk wa s t o t ry t o m ake s ense o f t he f li nt doma in . Th e va r i ous K Etechn iques w e re ex p l a in ed t o t h em , t houg h t he co n t ex t o f know ledge e l i c it a t ion fo rexpe r t sy s t e m s wa s no t e x p l a i n ed a t t he s t a r t o f t he expe r im en t . Sub j ec t s we re

p rope r ly de - b r i e f ed a f t e r comp le t i ng bo th s e s s ions .Th e de s ig n o f t h e e x pe r i m en t w as exac t ly a s i n Exp e r ime n t 1 . Tw o sub j ec t s we re

8/11/2019 1-s2.0-S104281430580010X-main

9/12

C O M PA R I N G E L I C T AT IO N T E C t l N I O U E S 75

allocated at random to each of the four possible combinations of KE techniques(one traditional and one contrived technique). This provides data for four KEsessions for each technique. The same eff ort metrics as above were used.

However, to measure gain, only clauses gained was used, as there is no sense inwhich the information gained from these subjects could be compared withknowledge from genuine experts.

RESULTS AND DISCUSSION

Despite reporting that they found the task difficult, no subject withdrew from theexperiment, and all appeared to make a genuine effort to carry out the instructions.Analysis of the transcripts of KE sessions shows that all subjects were able toconstruct plausible accounts of some parts of the domain at hand. We will discussthis further below.

Table 4 shows the mean dependent measures taken by KE technique. Comparisonwith Table 1 shows that these measures give a relatively f l a t pattern across alltechniques. Unlike experts, these subjects do not show a characteristic pattern ofefficiency across techniques. Although there are some small differences in the

effo rt measures, these are not very marked , by comparison to those evidenced inthe expert studies repor ted above. The gai n measures are fiat across alltechniques. In short, it appears from these results that any technique is as good asany other to the non-expert. This is evidence that the results gained in previousexperiments are a consequence of the use of these techniques specifically as KEtechniques. It does not seem that we have merely being measuring aspects of the

techniques themselves.The base-line performances are interesting here. Looking at the ga in measure,it can be seen that the techniques appea r to provide information. In fact, subjectsare able to offer a set of quite plausible rul es which an engineer new to thedomain would have difficulty in faulting. Interestingly, the same information isrepeated across all combinations of techniques. Even though the subjects knownothing of the domain, they conclude the same in both the KE sessions they attend.

This is rather a worrying result for knowledge engineers. For someone approach-ing the domain for the first time, it is necessary to discriminate between genui neknowledge, and knowledge which is plausible but unfounded. In detailed analysis ofthe transcripts from these sessions, we were unable to discover a predictive reliablediscrimination metric between sets of rules generated by experts or non-experts. A

TABLE 4Mea n s c o re s f o r non -ex pe r t sub jec ts i n t he f l i n t do ma in(n = 4 p e r g r o u p

Technique Interview Prot An L Grid Sort

Time to KE (min) 15 29 17 15Transcription time (min) 43 48 49 43Total time 58 77 66 58No. clauses 75 85 69 77Clauses min -~ 1.29 1.10 1.05 1.32

8/11/2019 1-s2.0-S104281430580010X-main

10/12

76 A M B U R TO NE T A L

5

3 4

1 \7 2 5 3 5 4 3 0 2 9 2 7 6

7 1 7 0 3 9 3 8 8 7

5 6 9 5

P i a n o c o n v e x k n i f e

F IG U RE 1 T h e c o n n e c t e d n e s s o f r u l e s f o r a s i n g l e d e c i s i o n e l i c it e d f r o m a n e x p e r t i n t h e f li n t d o m a i nN u m b e r s r e f er t o r u le n u m b e r s

number o f metrics suggested themselves, though these were mostly circular, i.e. onewould need to know about the domain in order to discover whether one was talkingto an expert. This is clearly not the case in most knowledge elicitation sessions. Twopossible discrimination functions are suggested by the data here. Although the dataare not strong enough to make reliable recommendations, we will pursue thesepossibilities further. First, the results from non-experts are fla t , in that allelicitation techniques provide roughly the same amount of information (and usuallythe s m e information). This is not the case with genuine experts. If one wereprepared to ask experts to take pa rt in several types of elicitation session this mightserve as a possible discriminator. The second possible discriminator relies on thec o n n e c t e d n e s s of the rule set. Figure 1 shows a representation of rules leading to asingle decision, gained from one expert in the flint domain. This can be read as animplication tree, where the numbers refer to rules. So, for example, rule 35 has aconsequent which is used as an antecedent in rule 34. This shows complex

interconnectedness of knowledge in the expert's transcript. This degree of com-plexity is n e v e r present in the non-experts' rule sets. Decision trees are all veryshallow (i.e. very few rules chain to reach a particula r goal).

While this result is interesting, it is not conclusive. The degree of interconnected-ness is, of course, confo unded with the amount o f information gleaned from the twoclasses of expert. We plan further experiments (in new domains) to disentanglethese two factors, and we hope to be able to use the interconnectedness and otherfeatures of the top olo gy of rule sets as possible discriminators in the future.

o n c l u s i o n s

In this paper we have described an experiment comparing the efficacy of fourknowledge elicitation techniques, across two classification domains. As with

8/11/2019 1-s2.0-S104281430580010X-main

11/12

COMP RING ELICIT TION TECHNI QUES 77

p rev ious ex pe r ime n t s , we h ave s h own tha t p ro toco l ana ly s i s pe r fo rms poo r ly i n t h i st y p e o f d o m a i n . F u r t h e r m o r e , w e h a v e s h o w n t h a t t w o c o n t ri v e d t e c h n iq u e spe r fo rm ve ry w e l l , p ro v i d ingcomplementaryknowledge t o t he s t anda rd i n t e rv i ew,

and do ing s o ve ry e f f i c i e n t l y. We have t hen de sc r i bed fu r t he r expe r imen t s onnov i ce s wh ic h su gg es ts an un de r ly ing p ro b l em in know ledge e l i c i ta t ion , t ha t o fi nadeq u a t e e xp e r t is e . I n t h is f i na l s ec t i on , we w i ll o f f e r som e conc lud ing r em arks ont he na tu r e o f kn owledg e e l ic t a t io n , and on t he cho i ce o f KE t echn ique .

G iven t h e e v ide nc e aga in s t p r o t oco l ana ly s i s i n c l a s s i f i c a t i on doma ins , we shou lda s k wh y it con t i nu es t o be s o po p u l a r. W e sugges t tha t t he answ er l ie s i n t he com fo r to f u se o f th i s t e ch n ique . I t is by n ow c l ea r tha t expe r t s t yp i ca l l y p r e f e r K E se s s ionswh ich s eem f a mi l i a r t o t he m i n some way ( e . g . Schwe icke r tet al. 1986) .F u r t h e r m o r e , t h e y a re p r o n e t o o b j e c t t o t h e u s e o f c o n t r iv e d t e c h n iq u e s o n s u chg r o u n d s as I d o n ' t t h in k t h a t w a y . W e h a v e h e r e p r o v i d e d f u rt h e r e v i d e n c esupp o r t i n g a la ck o f co r r e s pon de n ce be tw een an ex pe r t ' s v i ew o f t he s e s s ion , andi t s u t i l i ty fo r a knowledge engineer.

G iven t he m a n y s o c i a l co n s t r a i n ts o n a K E se s s ion (e . g . H a r t , 1986) , i t is no tsu rp r is i ng t ha t e l ic i to r s tr y t o ma x imize t he com fo r t o f t he s e s s ion . H ow eve r, wesugg est t ha t t hey d o so a t t he ex p e nse o f e ff ic i en t e l i ci t a ti on . C l ea r l y t he r e a r e c a se sw h e r e one i s so c ons t r a in ed by t he o p in ion o f t he exp e r t , t h a t i s wo u ld be foo l i sh t or isk lo s ing h is o r he r co - o pe r a t i on . H ow eve r, we sugges t t ha t i n mo s t c i rcums t ances ,i t i s w or th bear ing a l it t le d i scom for t in o rder to im pro ve the e ff ic iency ofel ic i ta t ion.

Inadequa t e expe r t i s e i s l i ke ly t o con t i nue t o be a p rob l em fo r t hose work ing i nappl ied se t t ings . G iven the d i fficu l ty in find ing a rob us t an d re l iab le syn tac t ic

d i s c rim ina t o r b e t w e e n t r an s c r ip t s f r om ex pe r t s and non -expe r t s , e l ic i t o rs shou ld b eon t he i r gua rd aga i n st t h is t y p e o f e r ro r i n t he ir f u tu r e work . A l though i t i s no ta l w ay s p os s i b l e , w e s ug ges t t ha t cons ide r ab l e t ime b e pu t i n to t he o r ig ina l s e l ec ti ono f an expe r t . E x t e r na l va l i d a t io n o f an expe r t ' s su i t ab il i ty w il l s ave cons ide r ab l e t imeand wa s t ed e f fo rt i n f u tu r e s e s s io n s . P r e r au (1987 ) ha s p rov ided an accou n t o f howthe s e l ec t i on p roce s s may p roce e d . F ina l l y, we conc lude t ha t more emp i r i c a l s t ud i e so f t he e f f ec t i ve nes s o f K E m e tho ds a r e need ed . I t is imp or t an t t ha t p rog re s s i ndeve lop ing ne w t e c h n iq u es i s m ade i n pa r a ll e l w i th app rop r i a t e eva lua t i on o f t he set echn iques . T h i s is eq u a l l y t r ue o f au tom a ted kno wled ge acqu i s i t i on and t r ad i t iona lknowledge a cqu i s i t i on . Wi th ou t p rope r eva lua t i on , r e s ea r ch i n knowledge e l i c i t a t i onbecomes a n e nd i n i t s e l f , r a t he r t han an a t t emp t t o so lve an app l i ed p rob l em f acedb y t h o s e c o n s t ru c t in g k n o w l e d g e b a s e d s y s te m s .

This work was completed under an award funded by the Alvey Comm it tee (Grant num berIKBS 134). W e are grateful to the m any experts wh o contributed to the experiments reportedhere.

eferences

ANJEWlERDENA . (1 987 ). Know ledge acquisition tools.AI Communications0, 29-39.Bo osE , J. H . (1989). A survey of knowledge acquisit ion techniques and tools.Knowledge

Acquisition I , 3 -37 .Boo sE, J . H . & GAINES, B. R. , Eds. (1988) .Knowledge Acquisition fo r Knowledge-Based

Systems Volume 2.London: Academic Press.

8/11/2019 1-s2.0-S104281430580010X-main

12/12

178 A.M . BURTONET AL.

BURTON, A . M . , SIIADBOLT, N . R . , HEDGECOCK, A. D. & RUGG, G. (1987). A forma levaluat ion of knowledge e l ic i taf ion techniques for exper t sys tems: Domain 1 . In D. S .MORALEE, Ed . ,Research and Development in Expert Systems IV.Cambr idge : Cam-br idge U nivers i ty Press .

BURTON, A. M . , SHADBOLT, N. R. , RUG G, G. & HEDGECOCK, A. D. (1988). Kn ow ledgeelici tat ion techniques in classif ication domains.ECA I-gg: Proceedings of the 8thEuropean Conference on Artifical Intelligence,pp . 85 -90 .

BtrrLER, K. E. & CORTER, J . E . (1986). Use of psycho me tr ic tools for knowledg eacquis i t ion: A case s tudy. In W. A. GALE, Ed. ,Artifica Intelligence an d Statistics.Cambr idge , MA: Add i son-Wes ley.

CULLEN, J . & BRYM AN, A. (1988). T he know ledge acquis i t ion bo t t leneck: T im e fo rreassessment?Expert Systems,5, 216-225.

GAL~ES, B. R. & BOOSE, J. H . (1988).Knowledge Acquisition for Knowledge-Based Systems,Volume 1.London : Academic P ress .

HART, A. (1986).Knowledge Acquisition for Expert Systems.L o n d o n : K o g a n P a g e .KLINKER, G. , BENTOLILA, J. , GENETET, S. , GRIMES, M . & McDERM OTT, J. (1987).

K N A C K :Repor t -dr iven knowledge acquis i t ion .International Journal of Man-Machine Studies, 26,

65-79 .MARCUS, S . , M cDEmMOTr, J . & WANG, T. (1985). Kn owled ge acquis i t ion fo r const ruct ive

systems.HCAI-85.PRERAU, D. S . (1987). Kno wledg e acquis tion in expe r t sys tem develo pm ent .AI Magazine,

8 43-51 .SCtlWEICKERT, R. , BUR TON , .A.M. , TAYLOR, N . K . ,CORLE I F, E. N . , StlADBOLT, N. R . &

HEDGECOCK, A. P. (1987). Com par ing know ledge e l ic i ta tion techniques : A case s tudy.Artifical Intelligence Review,1, 245-253.

SHADBOLT, N. R. & BURTON, M . (1990). Kn ow ledg e elici tat ion. I n J . R. WXLSON & E. N.CORLE'rr, Eds. , Evaluation of H um an W ork: Practical Ergonomics Methodology.Basingstoke: Taylor and Francis .

SMrrn , R . G . & B AKER, J . D . (1983). T he Dip me te r Adv i so ry Sys tem: A case s tudy incommerc ia l expe r t sys t em deve lopment .HCAI-83: Proceedings of the 8th InternationalJoint Conference on Artifical Intelligence.

YOUNG , R. M . (1987).The Role of Intermediate Representation in Knowledge Elicitation.Key no te Address to Ex pe r t Sys tems 87 , Br igh ton , UK .