SYST 542 Decision Support Systems Engineeringseor.vse.gmu.edu/~klaskey/SYST542/DSS_Unit4.pdf ·...

Unit 4 - 1 -

Department of Systems Engineering and Operations Research

Copyright © 2006, Kathryn Blackmond LaskeySYST 542

SYST 542Decision Support Systems

EngineeringInstructor: Kathryn Blackmond Laskey

Fall Semester, 2006

Unit 4: DSS Elements:The Model Subsystem (2)

Other Model System Technologies

Unit 4 - 2 -



Outline

• Other Model System Technologies– Heuristic methods for solving optimization

problems– Artificial Intelligence and Expert Systems– Statistical Methods– Some Additional Methods

• Matching Methods to Problems

Unit 4 - 3 -



Models for Supporting Decisions• Models can support decisions in a number of

ways:– Assist with problem formulation– Find optimal or approximately optimal (according to model)

solution– Assist in composing solutions to subproblems– Portray decision-relevant information in a way that makes

decision implications clear– Draw conclusions from data (data --> information ->

knowledge)– Predict results of proposed solution(s)– Evaluate proposed solution(s)– Can you think of others?

• Different modeling technologies are useful fordifferent kinds of support

Unit 4 - 4 -



Example Heuristics• Greedy hill climber

– Begin with a candidate solution– Change in direction that most improves solution– Never go downhill

• Decomposition– Break problem into simpler subproblems– Solve subproblems separately– Recompose solutions

• Heuristic search– Search space can be constructed as tree– Depth first, breadth first, best first: policies for deciding how to expand the

tree

• Approximate and adjust– Use cheap / fast / available approximation method– Adjust solution– e.g., use linear programming on integer problem and move to nearest

integer solution

Unit 4 - 5 -



Natural Analogy Heuristics• Nature is an efficient optimizer

– Apply methods based on analogy to natural systems

• Simulated annealing– Modify current solution randomly and evaluate objective function– Accept new solution if better than old. Otherwise, accept with probability

depending on system "temperature"– Gradually decrease temperature (make it harder to accept worse solutions)

• Evolutionary algorithms– Maintain "population" of solutions– Solutions reproduce with # offspring depending on objective function

(survival of fittest)– Apply evolutionary operators to change solutions from generation to

generation (e.g., crossover, mutation)

Unit 4 - 6 -



Knowledge-Based Systems• Basic premise: intelligent problem solving

requires flexible application of knowledge• Key feature: separate representation of

knowledge from application to solve problems• Advantages:

– opportunistic application of knowledge– flexibility to try different solution strategies– facilitates modularity, maintainability, ability to explain

reasoning to users

Unit 4 - 7 -



Elements of a Generic Knowledge Based System

• Knowledge base– Generalizations about the domain– R1: “IF GPA of <x> is less than 3.0 THEN <x> is not

eligible for combined BS/MS program”• Long term memory

– Facts about the world– F1: “GPA of G12345678 is 2.75”

• Inference engine– Uses R1 and F1 and the allowable proof steps to derive

the conclusion “G12345678 is not eligible forcombined BS/MS program”

• Working memory– Holds on to intermediate results of computations– Receives inputs from the outside world– Things can be read from and written into LTM and KB

• Control strategy– Sets priorities– Decides what order to do which tasks

LTM KB

WM

Data KnowledgeStructures(e.g. rules)

InferencesInferenceEngine

Unit 4 - 8 -



Representing Domain Knowledge

• Knowledge is different from data or information– Knowledge structures reflect structure of domain– Reasoners access and manipulate knowledge structures– Representation serves as a surrogate for represented entities

• Problems that are very difficult in one representation canbecome straightforward in a better representation

– e.g., arithmetic in Roman vs Arabic numerals• A knowledge base contains knowledge structures that

represent:– Types of entities that can exist in the world– Attributes of entities of each type– Relationships entities can have to each other– Processes in which entities can participate– Individual entities: their types, attributes, relationships and behavior

• A formal representation of knowledge about a domain isalso called a domain ontology

– Ontologies represent semantics to facilitate reuse, sharing, andsemantic interoperability

• Reasoners manipulate stored knowledge and incominginformation to derive new knowledge

Unit 4 - 9 -



Example Knowledge Structures• Rules

– IF-THEN knowledge– Allows system to derive consequences of knowledge– Rule-based system

» Represents knowledge as rules and reasons by applying rules» One of the first and most popular technologies of AI

• Object Classes / Instances– Represent types of entities and type-specific attributes– Subtypes can inherit structure and properties from parent

types– Instances represent specific entities of a given type– (Frames/Slots <=> Objects/Attributes)

• Semantic networks– Organizes knowledge about relationships between entities– Nodes represent entities; arcs represent relationships; arcs

are labeled by relationship type

Unit 4 - 10 -



Organizing Knowledge• Knowledge about things

– Knowledge about things» Types of things» Features a type of thing can have» Possible values for features

– Frames (artificial intelligence), objects (computer science), and relations(information systems)

• Knowledge about allowable transformations– Declarative knowledge: State axioms for allowable transformations and use

theorem proving techniques to do the transformation– Procedural knowledge: Set up triggers to execute sequences of

transformations• Applying knowledge to achieve goal

– Forward chaining: apply rules in forward direction to achieve goal– Backward chaining: look for antecedents of goal and declare achieving them

as sub-goal

Unit 4 - 11 -



Three-Legged Stool• Knowledge representation - formal representation for domain

semantics:– Entity types and instances, attributes, relationships, processes

• Decision theory– Goals are formalized as utility functions– Uncertainty is represented as probability distributions– “Ideal” inference and problem solving are formalized as Bayesian reasoning

and decision making– Practical inference and problem solving are formalized as “bounded

rationality”

• Algorithmic computer science– Find efficient algorithms for solving optimization and inference problems– Find efficient algorithms for approximating solutions to optimization and

inference problems and characterizing the quality of solutions– Apply heuristic methods to decompose problem, solve sub-problems,

compose results

• There is increasing cross-fertilization among OR andtraditional artificial intelligence

Unit 4 - 12 -



Statistics• Statistics is about using data to draw

inferences about uncertain phenomena– Many methods from OR, decision analysis, and AI use

probability distributions to represent uncertain outcomes– Distributions are specified by some combination of theory,

expert knowledge, and observations– Statistics provides methods for drawing well-justified

inferences from data

• Statistics is also about creating summaries ofobservations that will be useful for decisionmakers– Graphical displays– Numerical summaries

Unit 4 - 13 -



Statistical Methods• Exploratory data analysis

– graphical tools for data display– data transformation methods– outlier detection methods– robust methods for exploratory identification of trends and

patterns– clustering

• Statistical model construction and evaluation– Parameter estimation– Hypothesis testing and model evaluation– Model selection and model averaging– Statistical pattern recognition

• Sampling design methods– experimental design– survey design– design of observational studies

Unit 4 - 14 -



Parametric Models• Parametric statistical model:

y ~ f(y | θ), or y ~ f(y | x, θ), where~ means “distributed as”y denotes an uncertain variable or vector of variablesx denotes observed variables (covariates)θ denotes an unobserved parameter

• Goals of modeling and data analysis:– Determine functional form of parametric model– Estimate value of parameter

» “Best” point estimate» Interval of plausible values» Predict likely future values» Draw substantive conclusions (whether a variable influences another;

whether data support a theoretically interesting value of parameter)

Unit 4 - 15 -



Types of Statistical Models(some examples)

• Regression– Estimate an equation relating a dependent variable to

one or more independent variables– Example: examine relationship between students’

college GPA and high school grades• Analysis of variance

– Evaluate whether average value of a response isdifferent for different groups of individuals

– Example: evaluate whether patients taking a drug dobetter than patients taking a placebo

• Time series models– Examine trends and/or cycles in data over time– Example: predict price of a stock

Unit 4 - 16 -



Linear Regression in Excel• Data on tar, nicotine, weight and CO in 25 brands of cigarettes

downloaded from http://www.mste.uiuc.edu/regression/cig.html

Tar vs Carbon Monoxide

0

5

10

15

20

25

30

0 5 10 15 20 25 30 35

Tar (mg)

CO (mg)

Linear regression linecan be computedusing slope andintercept functions inExcelExcel statistics add-inhas more sophisticatedstatistical capabilityCan add trendline tochart without creatingnew column of data

Unit 4 - 17 -



Data Analysis Process• If you are collecting data, consider analyses to be run in the design of

your data collection– Simulate “dummy data” and analyze to test your procedures– How much data will you need to make the case you need to make?

• First step: clean and explore– Look for errors and problems– Check for outliers– Plot data and look for patterns

• Next step: confirmatory analysis– Select statistical model– Estimate parameters– Evaluate confidence levels or other indicators of strength of conclusion– Evaluate sensitivity to outliers and violations of assumptions

• Final step: report results– Conveying complex data understandably is an important human factors issue– A picture is worth a thousand words– Include information about uncertainty in result– Interface between model and dialog subsystems

Unit 4 - 18 -



Connectionist Modelsor Neural Networks

• Connectionist philosophy– Complex behavior comes from interactions among simple

computational units– Natural analogy: simulate intelligent behavior using process

modeled after human brains

• A neural network consists of– a large set of computationally simple units or nodes– links or connections between nodes

• Learning occurs by adjusting strengths ofconnections

– supervised learning: regression– unsupervised learning: clustering

Unit 4 - 19 -



A Simple Feedforward Network

Input layer

Hidden layer

Output layer

We can think of a feedforward neural network as anonlinear regression model:

y ~ f(y | x, w) y denotes the output node(s)w denotes the weight vectorx denotes the input node(s)

Learning the weights corresponds to estimating theparameters of the regression model

A good discussion of the relationship between neural networks andstatistical methods can be found at:http://www.faqs.org/faqs/ai-faq/neural-nets/part1/section-15.html

Unit 4 - 20 -



Machine Learning• Machine learning is the discipline devoted to

development of methods that allow computersto “learn” (improve performance based onresults of past performance)

• Machine learning draws from artificialintelligence, traditional computer science, andstatistics– Extract regularities from samples of data– Construct knowledge structures (typically rules) that

characterize the regularities– Evaluate performance against samples not seen before

Unit 4 - 21 -



Data Mining• The IT revolution has created vast archives of data• Data mining is a collection of methods from statistics,

computer science, engineering, and artificial intelligence forsifting through large stores of data to identify interestingpatterns

• There is a great deal of overlap with machine learning– In machine learning the emphasis is on using data to improve performance

on a well-defined task according to some performance measure (induction)– In data mining the emphasis is on identifying interesting patterns in large

volumes of data (discovery)– Both machine learning and data mining make heavy use of statistical

methods• The term data mining is sometimes used pejoratively to mean

fishing for spurious patterns and concocting post-hocexplanations

Unit 4 - 22 -



Knowledge Discovery and Data Mining

• Source: From Data Mining to Knowledge Discovery in Databases (PDF),Usama Fayyad, Gregory Piatetsky-Shapiro, and Padhraic Smyth, AIMagazine 17(3): Fall 1996, 37-54

• Available from http://www.kddnuggets.com

Unit 4 - 23 -



Economic Methods• Microeconomic models

– Analyze economic systems in which firms / agents are modeled as utilitymaximizers

– Static: analyze equilibrium– Dynamic: analyze behavior over time

• Game theory– Multiple players each have possible actions and objective functions– An economy is a many-person game

• Macroeconomic models (econometrics)– Statistical estimation of relationships between economic variables

• Cost / benefit analysis– Benefits of proposed policy option are quantified in dollar terms and

evaluated against cost

Unit 4 - 24 -



Cost-Benefit Analysisvs Multiattribute Utility

• Conceptually these methods are concerned with the sameproblem

• Both translate costs and benefits to common units– CBA: common units are dollars– MAU: common units are utilities

• Concerns with CBA as typically applied:– tendency to ignore "nonmeasurable" attributes– measuring social costs by "willingness to pay"– equity concerns due to maximizing net dollar benefit

• Objective:– CBA usually maximizes Benefit/Cost ratio (good for allocating limited budget

among multiple projects)– MAU chooses option with highest expected utility (good for selecting single best

option)

Unit 4 - 25 -



Management Science Methods• Project planning and scheduling methods

– Milestone charts– Gantt charts– Critical Path Method (CPM) charts

• Project monitoring methods– Earned value analysis

Unit 4 - 26 -



Methods from Accounting and Finance

• Not really a separate category of modelingmethods– Draws from optimization, statistics, economics, and other

disciplines• Examples of common model types

– Return on investment analysis» Identify time period for investment to pay back

– Break even analysis» Identify revenues needed for product to cover costs

– Forecasting supply, demand, interest rates, etc– Cost estimation

» Example: COCOMO is a model for estimating cost of softwareprojects

» See http://www.softstarsystems.com/overview.htm for adescription

Unit 4 - 27 -



Problem Structuring Methods• Group techniques

– Brainstorming– Nominal group– Delphi

• Methods for organizing large sets of elements– Checklists– Interaction matrices– Trees

• Structured modeling methods– Cognitive maps– Causal loop diagrams– Semantic networks

Unit 4 - 28 -



Some Philosophy• The most valuable output of a good model is

insight• Blind acceptance of "the computer's answer"

is almost always a mistake– sometimes closer examination of unintuitive result reveals

unmodeled asepct of the problem

• Often much insight can be gained from a less-than-complete analysis

Unit 4 - 29 -



Sensitivity Analysis• Sensitivity analysis means varying the inputs

to a model to see how the results change• Sensitivity analysis is a very important

component of exploratory use of models– model is not regarded as “correct”– sensitivity analysis helps user explore implications of

alternate assumptions– human computer interface for sensitivity analysis is

difficult to design well• In many models we need to make assumptions

we cannot test– Sensitivity analysis examines dependence of results on

these assumptions

Unit 4 - 30 -



Model Evaluation• Verification: evaluate whether model is

implemented correctly and produces outputs itwas designed to produce

• Validation: evaluate whether model accuratelyreproduces the phenomenon being modeled

• Why are verification and validation bythemselves insufficient for model evaluation?

Unit 4 - 31 -



Some Approaches to Model Evaluation

• Compare model output to ground truth– Quality of result (according to what metric?)– How many cases?– Coverage of all relevant conditions?– Can we instrument all relevant variables?– Fidelity of test conditions to real problem

• Compare model output to expert judgment– Same questions as above

• Model “walk through”– Have we included the right variables?– Are the methods appropriately matched with problems?– Are the methods applied correctly?

Unit 4 - 32 -



Model Evaluation for DSS• In a decision support environment we are

concerned with the quality of the Model +Decision Maker combination

• We need to evaluate whether supporteddecision maker performs better than– Decision maker alone– Model alone

• Metrics include– Quality of result– Time to solution– Understanding of solution by stakeholders– Buy-in by organization– Other measure?

Unit 4 - 33 -



Method Selection• Problem Type 1: General decision support for

poorly structured problems– Brainstorming or cognitive mapping to define the problem– Rule-based expert system to suggest solution approaches

and assist in constructing model» no uncertainty, multiple objectives --> MAU» uncertainty, simple objective --> influence diagram» uncertainty, multiple objectives --> MAU combined with

influence diagram– Rule-based expert system to suggest default values for

parameters (reduces elicitation burden)– Influence diagram solver– MAU model– Explanation generation expert system for interpreting

results– Sensitivity analysis methods

Unit 4 - 34 -



Method Selection• Problem Type 2: Optimization problem with

complex constraints and objective function– Use simplified mathematical programming model which does not

account for all aspects of problem– Use heuristic search (deterministic or stochastic) to modify

model output– Use statistical parameter estimation and/or expert system and/or

user input for values of parameters (constraints, objectivefunction)

• Problem Type 3: Plan design or enhancement ofcomplex system

– Build causal model of existing system– Use statistical methods to estimate parameters of model– Use simulation to predict system evolution– Use designed experiments to test cause and effect assumptions– Use sensitivity analysis to investigate how different values of

input parameters affect result

Unit 4 - 35 -



Method Selection• Problem Type 4: Identify patterns in large

database of high-dimensional data– neural network (unsupervised learning)– cluster analysis– graphical exploratory data analysis

• Problem Type 5: Classify new observationsinto categories, where categories are definedaccording to a large sample of exemplars

– discriminant analysis– neural network– genetic algorithm– simulated annealing– Bayesian network learning

Unit 4 - 36 -



Method Selection• Problem Type 6: Classify observations into

categories, when there is no pre-defined databaseof exemplars

– expert system– Bayesian network– neural network (trained by human-generated exemplars)– policy capture (this is a regression method in which the

dependent variable is human judgment and the independentvariables are cues used by human to classify)

• Problem Type 7: Select option from largedatabase classified on many dimensions

– multiattribute utility– screening– policy capture

Unit 4 - 37 -



Method Selection

• Problem Type 8: Select the best option froman implicitly defined option space with givenmeasure of performance

– mathematical programming– heuristic search– genetic algorithm– simulated annealing

• Problem Type 9: Forecast future values ofsome variable or variables

– time series methods– macroeconomic methods– simulation methods

Unit 4 - 38 -



Model Management• DBMS allows user to store and retrieve data

independent of physical representation of data incomputer

• MBMS allows user to invoke and run modelsindependent of particular implementation of models

– Paradigm of models as data is gaining favor– Parameters and assumptions represented explicitly rather than

being buried in computer codes– Facilitates flexible application and reuse

• Unlike DBMS, no standardized MBMS have emerged– Many different kinds of models, each with specialized structures,

uses, and methods– Building and applying models requires specialized expertise

Unit 4 - 39 -



Activities Supported by MBMS• Access and retrieve models• Exercise models

– specify model parameters– link models together– control execution of models

• Manipulate models– change assumptions and/or parameter values

• Store models and model results• Maintain models• Create new models

Unit 4 - 40 -



Simulation Methods• Given:

– Executable model of how a system behaves– Inputs representing policy options, environmental variables,

initial state of system, assumptions about system andenvironment

• Output:– Behavior of system state over time

• We use a simulation to conduct computerexperiments on “virtual reality” to evaluate options

– Can be exploratory, consolidative, or combination• Examples:

– Simulate future conflict to help decide which weapons systemsto procure

– Simulate economy to predict impact of economic policy options• Issues:

– Accuracy of simulation– Sensitivity of results to input assumptions– Design of experiment to maximize information from simulation

Unit 4 - 41 -



In Summary...

Unit 4 - 42 -



References• .

SYST 542 Decision Support Systems Engineeringseor.vse.gmu.edu/~klaskey/SYST542/DSS_Unit4.pdf ·...

Documents

Transcript of SYST 542 Decision Support Systems Engineeringseor.vse.gmu.edu/~klaskey/SYST542/DSS_Unit4.pdf ·...