KBS development life cyKBS development life cycle Validation Uncertaintcle Validation Uncertainty y

Dr. R. Weber INFO612Dr. R. Weber INFO612

KBS development life cyKBS development life cycle Validation Uncertaintcle Validation Uncertainty y

Cop

yrig

ht R

osin

a W

eber

ES Development Life CycleES Development Life Cycle

PlanningPlanning Knowledge DefinitionKnowledge Definition Knowledge DesignKnowledge Design Code and CheckoutCode and Checkout Knowledge VerificationKnowledge Verification System EvaluationSystem Evaluation

Cop

yrig

ht R

osin

a W

eber

PlanningPlanning

feasibility assessmentfeasibility assessment resource managementresource management task phasingtask phasing schedulesschedules preliminary functional layout preliminary functional layout

(what)(what) high-level requirements (how)high-level requirements (how)

Cop

yrig

ht R

osin

a W

eber

Knowledge DefinitionKnowledge Definition knowledge source identification and selection

source identification, importance, availability, source identification, importance, availability, selectionselection

knowledge acquisition, analysis, & extraction acquisition strategyacquisition strategy knowledge identification and classificationknowledge identification and classification functional layoutfunctional layout control flowcontrol flow user’s manualuser’s manual requirements specificationrequirements specification knowledge baselineknowledge baseline

Cop

yrig

ht R

osin

a W

eber

Knowledge DesignKnowledge Design knowledge representation

choose the most appropriate knowledge choose the most appropriate knowledge representation formalism and then choose the toolrepresentation formalism and then choose the tool

control structure internal fact structure preliminary user interface initial test plan design structure implementation strategy detailed user interface design specifications and report detailed test plan

Cop

yrig

ht R

osin

a W

eber

Code and CheckoutCode and Checkout

coding testing source listings user’s manual installation/operations guide system description document

Cop

yrig

ht R

osin

a W

eber

Knowledge VerificationKnowledge Verification

formal tests test analysis

incorrect incomplete inconsistent

recommendations

Cop

yrig

ht R

osin

a W

eber

System Evaluation-System Evaluation-ValidationValidation

demonstrate the system serves its purposes efficiently and effectivelycomparing to other systemscomparing to alternate methodscomparing to humans

maintenance

Cop

yrig

ht R

osin

a W

eber

PlanningPlanning Knowledge acquisitionKnowledge acquisition Knowledge engineering: design and Knowledge engineering: design and

implementationimplementation Situation assessmentSituation assessment RetrieveRetrieve ReviseRevise ReviewReview RetainRetain

System EvaluationSystem Evaluation MaintainMaintain

CBR Development Life CycleCBR Development Life Cycle

Cop

yrig

ht R

osin

a W

eberCBR design and CBR design and implementationimplementation Situation assessmentSituation assessment

RetrieveRetrieve ReviseRevise ReviewReview RetainRetain (Validation)(Validation) Maintenance designMaintenance design

Cop

yrig

ht R

osin

a W

eber

casebase

Design decisions in CBR systems (i)

Which are the cases?What is the task?

How will the case base be organized?

How will the cases be represented?Which will be the indexing vocabulary?

What is the task?How will the case base be organized?

Cop

yrig

ht R

osin

a W

eber

How will new cases be input?How to perform retrieval?

Identify featuresInitially match (similarity assessment)

SearchSelect

Retrievalinputproblem initial

solutions

Design decisions in CBR systems (ii)

Cop

yrig

ht R

osin

a W

eber

How to implement reuse?From Select or with a combination?

How to display the proposed solution?

solution

Reuse

proposed

initialsolutions

Design decisions in CBR systems (iii)

Cop

yrig

ht R

osin

a W

eber

Is the proposed solution good?How to determine and find what to adapt?

Where is adaptation knowledge?

solutionReviseproposedconfirmed

solution

case repair

case adaptation

Design decisions in CBR systems (iv)

Cop

yrig

ht R

osin

a W

eber

Is it the type of task that it is worth learning?Index new case before retain.

Retain.

Retain

confirmedsolution

casebase

Design decisions in CBR systems (ii)

Cop

yrig

ht R

osin

a W

eber

Validation refers to establishing the Validation refers to establishing the effectiveness of a system in light of its effectiveness of a system in light of its intended purposesintended purposes

Verification indicates how correct a given Verification indicates how correct a given system can solve its proposed tasks system can solve its proposed tasks (Watson)(Watson)

Retrieval accuracy is indicated by the Retrieval accuracy is indicated by the result given by the system when the result given by the system when the target case is part of the case collection.target case is part of the case collection.

validation & verification (i)validation & verification (i)

Cop

yrig

ht R

osin

a W

eber

validation & verification (ii)validation & verification (ii) Retrieval consistency: the same retrieval Retrieval consistency: the same retrieval

when executed the second time must when executed the second time must retrieve exactly the same cases (e.g., with retrieve exactly the same cases (e.g., with the same similarity if k-NN is used)the same similarity if k-NN is used)

Case Duplication: when two distinct cases Case Duplication: when two distinct cases receive the same value for similarity in receive the same value for similarity in relation to a given target case. relation to a given target case.

When the same value is attributed to When the same value is attributed to different cases the user or the system has to different cases the user or the system has to decide which one to use by evaluating the decide which one to use by evaluating the value for each attribute. The same measure value for each attribute. The same measure of similarity does not mean the cases of similarity does not mean the cases necessarily teach the same lessons.necessarily teach the same lessons.

Cop

yrig

ht R

osin

a W

eber

Case Coverage is checked for the evenly Case Coverage is checked for the evenly distribution of cases when they are manipulated distribution of cases when they are manipulated and not actual experiences that are collected as and not actual experiences that are collected as they happen.they happen.

Efficiency verification : comparison to alternative Efficiency verification : comparison to alternative methods, empirical evaluationsmethods, empirical evaluations

Retrieval time Retrieval time Retrieval sorting Retrieval sorting Case base consistency can be indicated by Case base consistency can be indicated by

retrievals resulting cases with gradual values of retrievals resulting cases with gradual values of similarity. A retrieval that no case has a high value similarity. A retrieval that no case has a high value of similarity or too many cases have the same of similarity or too many cases have the same value suggests inconsistency in the case base value suggests inconsistency in the case base

validation & verification (iii)validation & verification (iii)

Cop

yrig

ht R

osin

a W

eber

MaintenanceMaintenance if the reasoner learns, the maintenance if the reasoner learns, the maintenance

is more elaborateis more elaborate statistics of case usagestatistics of case usage perform validation tests continuouslyperform validation tests continuously special issue on case-based maintenancespecial issue on case-based maintenance Neural networks and other soft Neural networks and other soft

computing methods have been proposedcomputing methods have been proposed methods for distributed case basesmethods for distributed case bases D. B. Leake, B. Smyth, D. C. Wilson, Q. D. B. Leake, B. Smyth, D. C. Wilson, Q.

Yang, “Special issue on maintaining Yang, “Special issue on maintaining case-based reasoning systems,” case-based reasoning systems,” Computational Intelligence, 17(2), Computational Intelligence, 17(2), pp.193-195, 2001. pp.193-195, 2001.

Cop

yrig

ht R

osin

a W

eber

Determine the domain and scope Determine the domain and scope Consider reusing existing ontologies Consider reusing existing ontologies Enumerate important terms in the Enumerate important terms in the

ontology ontology Define the classes and the class Define the classes and the class

hierarchyhierarchy Define the attributes of classes (slots) Define the attributes of classes (slots) Define the facets of the slots Define the facets of the slots Create instances Create instances

from Noy & McGuinness

Ontologies Development Life Ontologies Development Life CycleCycle

Cop

yrig

ht R

osin

a W

eber

Ontologies Development Ontologies Development ProcessProcess

ontology development is an iterative processontology development is an iterative process

determinescope

considerreuse

enumerateterms

defineclasses

considerreuse

enumerateterms

defineclasses

defineproperties

createinstances

defineclasses

defineproperties

defineconstraints

createinstances

defineclasses

considerreuse

defineproperties

defineconstraints

createinstances

from Noy & McGuinness

UncertaintyUncertainty

Cop

yrig

ht R

osin

a W

eber

Uncertainty in InformationUncertainty in Information

Forms of ignorance:Forms of ignorance:incompleteness:incompleteness:

value of a variable is missing value of a variable is missing imprecision:imprecision:

value of a variable is given but not with the value of a variable is given but not with the precision required precision required

ambiguity:ambiguity: more than one possible value of a variablemore than one possible value of a variable

uncertainty:uncertainty: the value of a variable might be wrong the value of a variable might be wrong

Cop

yrig

ht R

osin

a W

eber

Uncertainty in ElementsUncertainty in ElementsFuzziness or vaguenessFuzziness or vagueness

refers to ill-defined bounds to describe an refers to ill-defined bounds to describe an element.element.

RandomnessRandomness refers to the certainty of whether a given refers to the certainty of whether a given

element belongs or not to a well-defined element belongs or not to a well-defined set.set.

ProbabilityProbability quantifies the chance that an event quantifies the chance that an event

might occur or the belief that a might occur or the belief that a proposition is true.proposition is true.

Cop

yrig

ht R

osin

a W

eber

Uncertain ProblemsUncertain Problems

uncertain knowledgeuncertain knowledge uncertain reasoninguncertain reasoning

When trying to solve a problem:When trying to solve a problem:

input --> reasoning --> outputinput --> reasoning --> output

What’s the impact on the output when What’s the impact on the output when uncertainty of the input is not treated?uncertainty of the input is not treated?

Cop

yrig

ht R

osin

a W

eber

Methods for the Treatment of Methods for the Treatment of UncertaintyUncertainty

Cop

yrig

ht R

osin

a W

eber

Fuzzy Set TheoryFuzzy Set Theory

Fuzzy SetsFuzzy Sets Fuzzy LogicFuzzy Logic Extension principleExtension principle

Cop

yrig

ht R

osin

a W

eber

Set of Americancities in the East

Philadelphia

LA

New York q=1q=1

q=0

q=1

q=?

q=0

Fuzzy SetsFuzzy Sets

Wilmington

Set of big American cities

New York

DE

Wilmington

Philadelphia

LA

Cop

yrig

ht R

osin

a W

eber

Fuzzy Sets Fuzzy Sets Degree of membership (Degree of membership (qq))

Extension PrincipleExtension Principle

L.Zadeh, A.Kandel, R. Yager, G.Klir, L.Zadeh, A.Kandel, R. Yager, G.Klir, H. Zimmermann, B. KoskoH. Zimmermann, B. Kosko

Cop

yrig

ht R

osin

a W

eber

AggregationsAggregations

sound image WM

50 50

100 0 50

40 40 40

FI:fuzzy integral

WM: weighted mean

FI

0

40

Cop

yrig

ht R

osin

a W

eber

Theory of ProbabilityTheory of Probability Measures degree of belief Measures degree of belief 80% degree of belief is a fairly strong expectation80% degree of belief is a fairly strong expectation What does x% chance means?What does x% chance means?

Frequentist approachFrequentist approach x% means that x is expected for each 100 times x% means that x is expected for each 100 times Probability summarizes the uncertainty that Probability summarizes the uncertainty that

comes from ignorancecomes from ignorance Must follow statistical principles, for example Must follow statistical principles, for example

additivity and complementarityadditivity and complementarity

Cop

yrig

ht R

osin

a W

eber

Certainty Factors- CFCertainty Factors- CF ignores complementarityignores complementarity BeliefBelief DisbeliefDisbelief Positive CF=> belief > disbeliefPositive CF=> belief > disbelief Negative CF=> disbelief> beliefNegative CF=> disbelief> belief Zero CF-> belief = disbelief and Zero CF-> belief = disbelief and

they both can be zerothey both can be zero

Cop

yrig

ht R

osin

a W

eber

Uncertainty inUncertainty in knowledge-based systems knowledge-based systems

Cop

yrig

ht R

osin

a W

eber

knowledgebase

(frames, uncertain rules)

knowledgebase

(frames, uncertain rules)

explanationexplanation

generalknowledgegeneral

knowledge

userInterface

userInterface

expertproblemexpert

problem

expertsolutionexpert

solution

uncertain inference

engine

uncertain inference

engine

working memory(uncertain

information)

working memory(uncertain

information)

knowledge acquisition

knowledge acquisition

uncertainty in ES (i)uncertainty in ES (i)

Cop

yrig

ht R

osin

a W

eber

uncertainty in ES (ii)uncertainty in ES (ii)Fuzzy rulesFuzzy rulesIf the students are tall then recruit them to If the students are tall then recruit them to

the basketball teamthe basketball teamFuzzy LogicFuzzy LogicFuzzy implications define truth values of Fuzzy implications define truth values of

propositions; generalized modus ponens, propositions; generalized modus ponens, fuzzy modus ponensfuzzy modus ponens

Rules with Certainty FactorsRules with Certainty FactorsIF liquidity is IF liquidity is very high (very high (0.60.6)) AND benefit AND benefit

is is moderatemoderate (0.8) THEN (0.8) THEN ask for ask for discountdiscount

Cop

yrig

ht R

osin

a W

eber

uncertainty in ES (iii)uncertainty in ES (iii)

Fuzzy informationFuzzy informationThe patient is middle-aged, The patient is middle-aged,

exercises regularly, and is slightly exercises regularly, and is slightly overweightoverweight

Fuzzy inference engineFuzzy inference engineGrades of membership in inference Grades of membership in inference

engines using fuzzy implicationsengines using fuzzy implications

Cop

yrig

ht R

osin

a W

eber

Rules can have Certainty FactorsRules can have Certainty Factors

IF IF liquidity is liquidity is very highvery high

AND AND benefit is benefit is moderatemoderate (0.8) (0.8)THEN THEN purchase cashpurchase cash

Cop

yrig

ht R

osin

a W

eber

solution

Retain

FuzzyRetrieval

Revise

Reuse

proposed

inputproblemwith uncertaininformation

confirmedsolution

initialsolutions

casebase

case adaptationwith fuzzyrules

uncertainty in CBR (i)

solution

CBR assumptions:problems recursimilar problems have similar solutions

situation assessment

Cop

yrig

ht R

osin

a W

eber

uncertainty in CBR (ii)uncertainty in CBR (ii)

prediction:prediction: the set of most similar and look for the the set of most similar and look for the

most likely to be the solutionmost likely to be the solution summary or combination of the solutions summary or combination of the solutions

with measures of central tendency or with measures of central tendency or analogousanalogous

Cop

yrig

ht R

osin

a W

eber

uncertainty in CBR (iii)uncertainty in CBR (iii)

• retrievaltarget case candidate case

weightattr 1 a1t a1c w1

attr 2 a2t a2c w2

attr 3 a3t a3c w3

• nearest neighbor algorithms implement synthetic evaluation of similarity

• benefit of fuzzy methods is the relaxation of the additivity axiom

Cop

yrig

ht R

osin

a W

eber

Uncertainty in OntologiesUncertainty in OntologiesIgnorance in classes, objects, facetsIgnorance in classes, objects, facets

incompletenessincompleteness imprecisionimprecision ambiguityambiguity uncertainty uncertainty

Elements and facts (parameters)Elements and facts (parameters) fuzziness or vaguenessfuzziness or vagueness randomnessrandomness probabilityprobabilityUncertain reasoning: methods, functions, Uncertain reasoning: methods, functions,

axiomsaxioms

probabilityprobabilityvs.vs.

possibilitypossibility

Cop

yrig

ht R

osin

a W

eber

Probability Distribution Frequentist approachObserving John eating breakfast 100 times

freq eggs65 235 10 0,3,4,5,…

However, it would still be very easy for him to eat 3 or 0 eggs, so even though these would have very low probability, they would have very high possibility.

Cop

yrig

ht R

osin

a W

eber

Possibility Distribution

How "easy" it is for John to eat eggs for breakfast

PD= 2/0.9 + 3/0.9 + 4/0.75+5/0.6 +6/0.3+ 7/0.05

Cop

yrig

ht R

osin

a W

eber

possibility probability distribution distribution

1 egg 0.8 0.352 eggs 0.9 0.653 eggs 0.9 0.04 eggs 0.75 0.05 eggs 0.6 0.06 eggs 0.3 0.07 eggs 0.05 0.0

Methods based on probability are based on classic binary logic and must meet axioms such as additivityMethods based on Fussy Set Theory have weaker axioms

Cop

yrig

ht R

osin

a W

eber

probability vs. possibilityprobability vs. possibility What’s the difference in a real application?What’s the difference in a real application? Probability is easier to acquire Probability is easier to acquire Ease of happening requires elicitationEase of happening requires elicitation Possibility seems more amenable to specific Possibility seems more amenable to specific

and limited domains where the ‘ease’ will be and limited domains where the ‘ease’ will be availableavailable

More general problems should be more More general problems should be more difficult to validate with possibility, thus difficult to validate with possibility, thus more suited to be dealt with probabilitymore suited to be dealt with probability

Some solutions for some Some solutions for some problemsproblems

Cop

yrig

ht R

osin

a W

eber

What are the problems and solutions?What are the problems and solutions?

IncompletenessIncompleteness: : value of a variable is value of a variable is missingmissingIn CBR, having all values is a requirement;In CBR, having all values is a requirement;

In ES, a rule cannot be triggered;In ES, a rule cannot be triggered;

In NN, an example will not be trained;In NN, an example will not be trained;Solutions:•substitute the missing value for a degree of belief or truth (probability);•create rules that presents a direction for incomplete values based on domain knowledge;•Replace incomplete variables with fuzzy sets;

Cop

yrig

ht R

osin

a W

eber

Imprecision: Imprecision: value of a variable is value of a variable is given but not with the precision given but not with the precision required (e.g., grades)required (e.g., grades)Why does it happen?Why does it happen? originally imprecise (age in yrs)originally imprecise (age in yrs) generated by imprecise methodgenerated by imprecise method need to work at an abstract levelneed to work at an abstract level summarizedsummarized

Solutions:•replace values for intervals;•define degrees of belief/degrees of truth;•reason at an abstract level;•measure the imprecision and convey its effects to the final result;

Cop

yrig

ht R

osin

a W

eber

AmbiguityAmbiguity: : more than one possible more than one possible value of a variable; possible values value of a variable; possible values are knownare knownSolutions: Who can disambiguate? Depends on the source of clarification.• In listening, text understanding, information extraction• degree of belief, truth;• ontologies• choose which is the most likely meaning (with uncertainty)• test the meaning in similar natural language sources (e.g., text, people);

Cop

yrig

ht R

osin

a W

eber

UncertaintyUncertainty: : the value of a variable the value of a variable might be wrong might be wrong

prediction;prediction; during reasoning, partial values;during reasoning, partial values; when you must defuzzify or eliminate the when you must defuzzify or eliminate the

degree of belief;degree of belief;

Solutions:• reduce the uncertainty, where does it come

from?• typicality of a set as an alternative to measures

of central tendency;• degree of belief;• fuzzy implications for possible values,

possibility distribution;