June 2, 2005Slide 1 Analyzing the Risks of Information Security Investments with Monte- Carlo...

June 2, 2005 Slide 1

Analyzing the Risks of Information Security Investments with Monte-Carlo Simulations

WEIS05Workshop on the Economics of Information Security

James R. Conrad, University of Idaho Department of Computer Science [email protected]


Contents

Introduction to the Problem

The Monte-Carlo Solution

Overview of Monte-Carlo Simulations

Example

Analysis and Critique

Conclusions


Introduction to the Problem

• An information security investment may need to compete for resources with other business opportunities

• But many information security models rely upon experts’ estimates

• And the experts’ estimates may include significant uncertainty

• How can the analyst communicate an opportunity when so much is uncertain?


Monte-Carlo Solution

• Apply the Monte-Carlo technique to simulate and express uncertainty in information security models

• This is not a new model --- this is an enhancement of existing models

• While less common in the Computer Science discipline, many financial decision makers are already familiar with the Monte-Carlo approach


Monte-Carlo Simulations

• Specify uncertainty in probability distributions

• Monte-Carlo engine samples distributions

• Engine executes the security model once for each of several thousand iterations

• Monte-Carlo engine captures and collects the result of each iteration

distributions

engine model

results


Monte-Carlo Simulations

• Engine simulates uncertainty in the model parameters

• Model continues to operate with discrete values

• Extra complexity largely confined to the Monte-Carlo engine

• Results can be charted as probability distributions

distributions

engine model

results


Monte-Carlo Example

• Based upon Longstaff et al’s example appearing in “Are we Forgetting the Risks of Information Technology?” of IEEE Computer, December 2000

• Simulates the benefit/cost ratio of a proposed infosec investment for a financial enterprise

• Modeling parameters are similar to Longstaff’s example with an added complication…

• …The experts don’t agree!


Original (pre-Monte-Carlo) Parameters & Model

Intrusion Rate Parametersr1 2 Simulated annual intrusion count w/o investmente 5.00E-01 Effectiveness of investmentr2 =r1*e Annual intrusion count with investment

Other Parametersp1 =r1/365 Daily probability of intrusion w/o investmentp2 =r2/365 Daily probability of intrusion with investmentX $20,000,000,000,000 Asset valuey1 $100,000,000 Cost of software assurance w/o investmenty2 $200,000,000 Cost of software assurance with investmentz1 1.00% Losses w/o investmentz2 0.50% Losses with investment

Model Calculationsd1 =p1*z1 Calc damage w/o investmentd2 =p2*z2 Calc damage with investmentD =y2-y1 Calc cost to provide software assurance with investmentd =d1-d2 Calc percentage of losses prevented by investmentb =d*X-D Calc net benefit of investmentbcr =b/D Calc benefit/cost ratio for investment (bcr=7.22)

intrusionrates

otherparameters

model

benefit/costratio, bcr


Uncertainty in the Revised Example

• Consider a case in which the experts don’t agree upon an single value estimate for the annual intrusion rate (fixed at r1=2 events/year in the original problem)

• The hypothetical disagreement stems from uncertainty in anticipated business practices

• Experts do agree there exists a 20% chance that business practices will change in a way that will raise the intrusion rate to 20 events/year and an 80% chance that those practices will remain unchanged


Uncertainty in the Revised Parameters• Model variability of optimistic intrusion rate as a Poisson process (for purposes of this example), ro=randpoisson(2)

• Model variability of pessimistic intrusion rate as a Poisson process, rp=randpoisson(20)

• Model uncertainty of anticipated business conditions by choosing the optimistic rate 80% of the time and the pessimistic rate 20% of the time using randdiscrete(0.80,0.20,ro,rp)

• Variability refers to a truly random process

• Uncertainty refers to the experts’ inability to anticipate future business conditions


Revised Params & Model

Intrusion Rate Parametersro =randpoisson(2) Optimistic annual intrusion count w/o investmentrp =randpoisson(20) Pessimistic annual intrusion count w/o investmentr1 =randdiscrete(0.8,0.2,ro,rp) 80% Chance of ro. 20% Chance of rp.e 5.00E-01 Effectiveness of investmentr2 =r1*e Annual intrusion count with investment

Other Parametersp1 =r1/365 Daily probability of intrusion w/o investmentp2 =r2/365 Daily probability of intrusion with investmentX $20,000,000,000,000 Asset valuey1 $100,000,000 Cost of software assurance w/o investmenty2 $200,000,000 Cost of software assurance with investmentz1 1.00% Losses w/o investmentz2 0.50% Losses with investment

Model Calculationsd1 =p1*z1 Calc damage w/o investmentd2 =p2*z2 Calc damage with investmentD =y2-y1 Calc cost to provide software assurance with investmentd =d1-d2 Calc percentage of losses prevented by investmentb =d*X-D Calc net benefit of investmentbcr =b/D Calc benefit/cost ratio for investment

intrusionrates

otherparameters

model

benefit/costratio, bcr


Simulation of Revised Example

• randpoisson() and randdiscrete() sample the probability distributions in each iteration of the simulation

• The Monte-Carlo engine recalculates the model for each iteration and captures the results (bcr)

• The Monte-Carlo engine charts the captured simulation results (next slide)


Simulation Results


Why not use a weighted average of r1 and r2?• Why doesn’t the revised model simply compute a weighted average of the two possible intrusion rates?

r1 = randpoisson(2)*0.8+randpoisson(20)*0.2

• The randdiscrete() simulation preserves the bimodal nature of the experts’ disagreement.

• Any attempt to “average away” that uncertainty conceals the truth: The experts don’t agree.


Analysis

• The results reflect the experts’ strong preference for the optimistic intrusion rate in which the benefit/cost ratio remains unchanged at 7.22. Risk-tolerant decision makers might manage to this value.

• The mean value lies at 22 between the two modes.

• The results also reflect a second mode at about 81 along with a 10% chance of the benefit/cost ratio exceeding 81. Risk-adverse decision makers might manage to this value to avoid a catastrophe “on their watch.”


Critique

• But are real experts willing to provide even more estimates?

• The author’s industry experience with Monte-Carlo models is that many experts are relieved to disclose the uncertainty they know to be in their estimates

• What real experts truly dislike is being held accountable to an expected value they know is merely representative of the possibilities


Additional Critique

• Given a tool to express uncertainty as probability distributions, which distributions closely model the empirical evidence?

• How to extend the Monte-Carlo approach to graphical models?


Conclusions

• Monte-Carlo techniques offer an approach to simulate uncertainty in expert estimates

• Enables the use of probability distributions for model parameters and forecast results

• The Monte-Carlo engine simulates random variables, allowing a security model to continue to manipulate discrete values with only minimal changes

• May be particularly useful for visualizing the potential of an extreme event, the unlikely possibility of a catastrophic outcome


Questions and Optional Slides


Why Poisson Distribution?

• The example problem uses a Poisson process to approximate intrusion attempts

• If and/or when the Poisson process usefully reflects empirical intrusion attempts is an open question

• Review: Models the number of events occurring during a specified time interval for a Poisson process

• Review: Continuous opportunity for independent events to occur

• Review: Long-term rate is constant

• Review: Used to model lightening strikes in a storm


Correlated Parameters

• “Every iteration of a… model must be a scenario that could physically occur.” -- Vose.

• The parameters must “make sense” to the security model!

• One correlated parameter can usually be expressed as a function (relation) of another.

• Consider r1 and r2 in the example. These are likely related which is why r2 is calculated as a function of r1.

• If the relationship (e) between r1 and r2 is also uncertain, this too can be simulated.


Variability and Uncertainty

• Yes, this example lumped (simulated) variability and uncertainty together for simplicity

• Vose (Risk Analysis, 2000) offers an excellent treatment of this subject for those who need to keep them separated


Partitioning

• Yes, partitioning is an alternative technique

• The Monte-Carlo technique might be viewed as an automated approach to partitioning

• …and the Monte-Carlo technique avoids the subjective choice of partition boundaries

• …and the Monte-Carlo technique has commercial tool support for systems-level models.


Commercial Tools

• Yes, commercial off-the-shelf tools are available

• They are most useful for systems-level security models.

• They are less useful for low-level combinatorics security models

• Search for “monte carlo simulation” and pay particular attention to the “Sponsored Links”


Performance

• The author’s industry experience includes Monte-Carlo simulations using “hundreds” of random distribution parameters

• Yes, they required several hours to run…

• In 1997!

• My computer is more than 10X faster today.

• Simulation multiplies model complexity by n, the number of iterations. A simulation of an O(m2) model becomes n*O(m2).

• Opportunities for parallel approaches when n cannot be ignored.

June 2, 2005Slide 1 Analyzing the Risks of Information Security Investments with Monte- Carlo...

Documents

Transcript of June 2, 2005Slide 1 Analyzing the Risks of Information Security Investments with Monte- Carlo...