Training on Cost Estimation & AnalysisKaren RicheyJennifer EchardMadhav Panwar

OutlineIntroduction to Cost EstimatingLife Cycle CostsData CollectionData AnalysisCost Estimating MethodologiesSoftware Cost ModelingCross-checks and ValidationRisk and Sensitivity AnalysisDocumentation RequirementsCost Estimating ChallengesCost Estimating Auditor Checklists

Introduction to Cost EstimatingThe National Estimating Society has defined Cost Estimating1as:The art of approximating the probable cost of something based on information available at the time.Cost estimating cannot: Be applied with cookbook precision, but must be tailored to a particular system,Substitute for sound judgment, management, or control,Produce results that are better than input data, orMake the final decisions.Despite these limitations, cost estimating is a powerful tool because it:Leads to a better understanding of the problem, Improves management insight into resource allocation problems, andProvides an objective baseline to measure progress.

1 The NES Dictionary, National Estimating Society, July 1982

Introduction to Cost EstimatingThe reliability of cost estimates varies over time. The closer you get to the actual completion of a project, the estimate becomes more accurate.Four types of cost estimates represent various levels of reliability .Conceptual Estimate: Rough order of magnitude or back of the envelope.Often inaccurate because there are too many unknowns.Preliminary Estimate: Used to develop initial budget, more precise.Detailed Estimate: Serves as a basis for daily project control.Definitive Estimate: Accuracy should be within 10% of final cost.Important to repeat estimating process (i.e., re-estimate) on a regular basis as more information becomes availableThis will keep estimate current as well as increase the accuracy

Introduction to Cost Estimating (contd)

All cost estimates are constructed by the following tasks: Identifying the purpose and scope of the new system.New software development, software reuse, COTS integration, etc.Choosing an estimate type.Conceptual, preliminary, detailed, or definitive type estimateIdentifying system performance and/or technical goals.Laying out a program schedule. Selecting a cost element structure (CES).Collecting, evaluating, and verifying data.Choosing, applying, cross-checking estimating methods to develop the cost estimate.Performing risk and sensitivity analysisTime-phasing the cost estimate by fiscal year for cash flow purposes.Example: 4 years to develop and 10 years operations and support beginning in FY2003Providing full documentation.

Life Cycle CostsMost cost estimates in the federal government represent total Life Cycle Costs (LCC).LCC estimates include all costs to develop, produce, operate, support, and dispose of a new system.Important to look beyond the immediate cost of developing and producing a system and consider all costs of a systems life cycle.What may appear to be an expensive alternative among competing systems may be the least expensive to operate and support Life Cycle Cost Estimates can be used to:Compare various alternatives before committing funds to a project,Support Estimate-to-Budget transition after time-phasing to account for when funds will be spent.

Life Cycle Costs (contd)A LCC estimate is summarized using a detailed cost element structure (CES) that Identifies the activities required to complete project development and the effort, loading, and duration of each task,Provides a framework against which the cost estimate is organized.Enhances cost data collection and estimate reporting.Facilitates comparing estimates when a standard CES is used.

Life Cycle CostsCost Element Structure (CES)A CES provides a standard vocabulary for the identification and classification of cost elements to be used in cost estimating. Helps to identify costs that may be initially overlooked. Should be tailored for each project. The CES should be reviewed to ensure that there is no double counting of costs that could be allocated to more than one element.For example, logistics support costs could be included in the investment or operations and support phase.The CES is hierarchical in nature to accommodate early development (when relatively little data is available) through deployment, when more detailed data is available.

Life Cycle CostsExample of CES Elements (DOD)1.0 Investment Phase2.0 System Operations & Support Phase1.1 Program Management2.1 System Management1.2 Concept Exploration2.2 Annual Operations (supplies/spares)1.3 System Development2.3 Hardware Maintenance1.3.1 System Design & Specification2.4 Software Maintenance1.3.2 Development Prototype and Test Site Investment2.5 Outsource Provider Support1.3.3 Software Development2.6 Data Maintenance1.3.4Training2.7 Site Operations (personnel, training,etc.)1.3.6 Facilities1.4 System Procurement1.4.1 Deployment Hardware1.4.2 Deployment Software1.4.3 Initial Documentation1.4.4Logistics Support Equipment1.4.5 Initial Spares1.4.6 Warranties1.5 Outsource Provider Investment1.6 System Initiation, Implementation & Fielding1.7 Upgrade (Pre-Planned Product Improvement (P3I))1.8 Disposal Costs

Data Collection Data RequiredAll CES elements will need data to support the estimateHistorical cost and non-cost data need to be collected to support various estimating techniquesTechnical non-cost data describes the physical, performance, and engineering characteristics of a systemExamples: weight, # of design drawings, SLOC, function points, # of integrated circuit boards, square footage, etc.Important to pick data that is a predictor of future costImportant to have technical and schedule data because they act as cost drivers

Data CollectionData Required(contd)Identify both direct and indirect costs Direct costs are called touch labor and include direct manufacturing, engineering, quality assurance, material, etc. costs which have a direct bearing on the production of a good. Also included are direct non-wage costs such as training, supplies, and travel.Indirect costs are considered overhead and include such things as general & administrative support, rent, utilities, insurance, network charges, and fringe benefits. These expenses are typically charged to a company as a whole.Example: sick/annual leave, retirement pay, health insurance, etc.

Data CollectionData Required (contd)Some direct costs may be burdened with indirect costs and some may notNeed to know to avoid double-counting or, worse yet, underestimatingImportant to ask when collecting data whether costs are burdened with indirect costs

Data CollectionData Required (contd)Data can be collected in a variety of waysContractor site visitsData requests for all relevant CES elementsDocumented cost estimates, if available for earlier versions of the current systemCan save valuable research time for statistical analysisPublished cost studiesData collection is a critical and time consuming step in the cost estimating process!

Data Collection Typical data sources Two types of data sources: Primary & SecondaryPrimary data is found at the original source (e.g., contractor reports, actual program data, etc.)Preferred source of dataSecondary data is derived from primary data of another similar system such as documented cost estimates, cost studies/research, proposal data, etc.Second choice to primary data due to data gapsMay be best alternative when time constraints or data availability limit primary data collection

Data CollectionTypical data sources Current Program Milestone schedule (for phasing program costs over time)Current Program System Description Important to have a technical understanding of how the system will workApproved program funding (Compare to proposal estimate)Contractor actual cost reports (from internal Management Information System)Current program estimate documentation (if available)Contractor proposals (Compare to program funding profile)Commercial Off-the-Shelf (COTS) catalogsManufacturer websites (e.g., Dell, Microsoft, etc.)

Data CollectionTypical Data Sources (contd)Forward Pricing Rate Agreements (FPRA)Similar program historical actual costs and estimate documentationEngineering drawings/specifications,Interviews with technical and program management personnelSurveysProfessional journals and publicationsIndustry guides and standardsTechnical Manuals

Data AnalysisData Validity/IntegrityImportant to ensure that the cost data collected is applicable to the estimate.Identifying limitations in the historical data is imperative for capturing uncertaintyWhen using historical cost data for a similar system, appropriate adjustments need to be made to account for differences in the new system being estimated.Data may need to be mapped from the contractors accounting system to the Cost Element Structure (CES)

Data Analysis Data Validity/Integrity (contd)Proposal data should be validated to ensure that contractor motivations to buy-in or low-bid their estimates are not occurring.Compare previous contractor proposal bids and actual costs for similar programs.Look for trends in underbidding.Participate in a fact-finding trip to discuss contractor proposal estimates and gather supporting data/evidence.

Data AnalysisNormalizationInvolves making adjustments to the data so that it can account for differences inInflation rates, Direct/indirect costs, Recurring and non-recurring costs, Production rate changes or breaks in production, Anomalies such as strikes, major test failures, or natural disasters causing data to fluctuate, andLearning curve (cost improvement) effects due to efficiencies gained from continually repeating a processAnalysis of the data may indicate the need for more suitable data to add credibility to the estimate.

Data AnalysisNormalization (contd)Accounting for Economic Changes (e.g., inflation)Lack of cost data uniformity due to upward movement in prices and services over time.Index numbers are used to deflate or inflate prices to facilitate comparison analysis.Wrong to compare costs from 1980s to today without accounting for inflation over the past 20 years.Cost estimators use inflation indices to convert costs to a constant year dollar basis to eliminate distortion that would otherwise be caused by price-level changes.Constant dollar estimates represent the cost of the resources required to meet each years workload using resource prices from one reference year (e.g., constant 2003 dollars).Constant dollars reflect the reference year prices for all time periods allowing analysts to determine the true cost of changes for an item

Data AnalysisNormalization (contd)For the United States, the Office of Management and Budget (OMB) is responsible for developing inflation guidance by appropriation for government estimates.Most common indices used by United States Cost Estimators are published by the U.S. Department of Labor, Bureau of Labor Statistics.Producers Price Index (http://www.bls.gov/ppi/) for goodsEmployment and Earnings Index (http://www.bls.gov/ces/home.htm#data) for servicesImportant to pick the appropriate market basket of goods index that most closely matches the costs to be estimated.

Data AnalysisNormalization (contd)For budgetary purposes, however, data expressed in constant dollars should be inflated to represent current year (or Then-year) dollars.Current year estimates calculate the cost of the resources using the estimated prices for the year in which the resources will be purchased. Current year estimates reflect inflationNecessary to express estimates in current year dollars when requesting funding to avoid budget shortfalls.

Data Analysis Normalization (contd)Example: Assume 5% escalation rate compounded annually

Base-year cost Then-year costCurrent year $273,100 * 1.0000 multiplier = $ 273,100Current year + 1 $1,911,700 * 1.0500 multiplier = $ 2,007,285Current year + 2 $4,096,500 * 1.1025 multiplier = $ 4,516,391Current year + 3 $8,193,000 * 1.1576 multiplier = $ 9,484,217Current year + 4 $6,144,750 * 1.2155 multiplier = $ 7,468,944Current year + 5 $1,092,400 * 1.2763 multiplier = $ 1,394,230

Total Estimated Labor cost (budget quality) $25,144,167

Cost Estimating MethodologiesOnce data has been collected and normalized to constant dollars, there are five methodologies available for estimating costs:Expert Opinion,Analogy,Parametric,Engineering, and Actual.

Estimating Methodology ConsiderationsChoice of methodology is dependent uponType of systemSoftware, hardware, etcPhase of programDevelopment, Production, SupportAvailable dataHistorical data points from earlier system versions or similar systemTechnical parameters of system

Methodology Expert OpinionOften called Delphi method, proposed by Dr. Barry Boehm in his book, Software Engineering Economics.Useful in assessing differences between past projects and new ones for which no historical precedent exists.

Methodology Expert Opinion (contd)Pros: Little or no historical data needed. Suitable for new or unique projects.

Cons:Very subjective.Experts may introduce bias.Larger number of experts will help to reduce biasQualification of experts may be questioned.

MethodologiesExpert Opinion - StepsGather a group of experts together,Describe overall program in enough detail so experts can provide an estimate,Each member of the expert group then does an independent of the resources needed,Estimates are gathered anonymously and compared,If there exists significant divergence among the estimates, the estimates will be returned to the expert group,The expert group then discusses the estimates and the divergence and works to resolve differences, andThe expert group once again submits anonymous, independent estimates which continues until a stable estimate results.

Methodology Expert Opinion - Example Three software engineers are renown in the ERP software development. You hold interviews which each explaining the requirements, sizing level, and development process for your new system. Each member of the group submits their opinion of the final cost and the estimate converges to $50M.

MethodologiesAnalogyEstimates costs by comparing proposed programs with similar, previously completed programs for which historical data is available.Actual costs of similar existing system are adjusted for complexity, technical, or physical differences to derive new cost estimatesAnalogies are used early in a program cycle when there is insufficient actual cost data to use as a detailed approachCompares similarities and differencesFocus is on main cost drivers.

MethodologiesAnalogy (contd)Often use single historical data point.May have several analogy estimates of sub elements to make up the total cost.Comparison process is key to success.May have to add or delete functionality from historical costs to match new programConsult technical experts for advice on complexity factors, technical, performance or physical differences.(not to be confused with expert opinion method)Impact of differences on cost is subjective.

MethodologiesAnalogy (contd)Good choice for:A new system that is derived from an existing subsystem. Make sure actual cost data is availableA system where technology/programmatic assumptions have advanced beyond any existing cost estimating relationships (CER).Secondary methodology/cross checkProvides link between technical assumptions and cost.

MethodologiesAnalogy (contd)Pros:InexpensiveEasily changedBased on actual experience (of the analogous system)Cons:Very SubjectiveLarge amount of uncertaintyTruly similar projects must exist and can be hard to findMust have detailed technical knowledge of program and analogous system

MethodologiesAnalogy - StepsDetermine estimate needs/ground rules,Define new system,Breakout new system into subcomponents for analogy estimating,Assess data availability of historical similar systems,Collect historical system component technical and cost data,Process/normalize data into constant year dollars (e.g., constant FY2003$), Develop factors based on prior system, Example: Program Management is 10% of total development costDevelop new system component costs.Obtain complexity (and other translation) factorsExample: Historical database is for 4 million records and new database will need to house 24 million records => complexity factor of 6 (4*6 = 24) times the historical database costApply factors to historical costs to obtain new system costs

MethodologiesAnalogy - ExampleYour company is developing a new IT payroll system serving 5,000 people and containing 100,000 lines of C++ code. Another company developed a similar 100,000 lines of code system for $20M for only 2,000 people. Your software engineers tell you that the new system is 25% more complicated than the other system.

Other system development cost was $20M.

Estimated cost for new system = 1.25 * $20M = $25MPlus 5,000/2,000, or 2.5 * $25M = $62.5M total cost

MethodologiesParametric Utilizes statistical techniques called Cost Estimating Relationships (CER).Relates a dependent variable (cost) to one or more independent variables Based on specific factors that have a high correlation to total costNumber of software lines of code (SLOC) or function points,Square feet for office floor space,Number of floors in a high rise building for cabling estimates,Database size, etc.Can be used prior to development.Typically employed at a higher CES level as details are not known.Most cases will require in-house development of CER.

MethodologiesParametric (contd)Pros:Can be excellent predictors when implemented correctlyOnce created, CERs are fast and simple to useEasily changedUseful early on in a programObjectiveCons:Often lack of data on software intensive systems for statistically significant CERDoes not provide access to subtle changesTop level; lower level may be not visibleNeed to be properly validated and relevant to system

Methodologies Parametric ExampleYou have a previously developed CER to estimate a new IT system based on SLOC.

Cost = SLOC * 25 $/SLOC

The CER is based on systems ranging from 1,000,000 to 3,000,000 SLOC.

You have estimated 2,600,000 SLOC for new system

Cost = 2,600,000 * $25 = $65M

MethodologiesParametric (contd)Cost estimators can develop their own CERs or they can use existing commercial cost models.Various Software cost estimating models will be discussed nextLearning curves specialized type of CER.CERs can be cost to cost or cost to non-cost.Cost to Cost: e.g., Manufacturing costs are 1.5 times Quality Assurance costsCost to Non-Cost: $/pound, or engineering hours/# of engineering drawings yields hours/drawing metric that can be applied to new programFactors and ratios are also examples of parametric estimating.

Methodologies Parametric CERsCERs are defined as a technique used to estimate a particular cost by using an established relationship with an independent variable.Can be as simple as a one variable ratio to a multi-variable equation.CERs use quantitative techniques to develop a mathematical relationship between an independent variable and a specific cost.

Methodologies Parametric CERs (contd)Reliable, normalized data is most important for CER development.Must determine range of data for which the CER is valid.Useful at any stage in a program.Typically CERs are the main cost estimating methodology in early stages of a program. In later stages of a program, CERs serve as a cross check to other methodsMust be logically sound as well as statistically sound. High correlation (r2 = 0.75 or higher) for goodness of fit testDifferent statistical techniques may be used to judge the quality of the CER.Least squares best fit (regression analysis, or the ability to predict one variable on the basis of the knowledge of another variable)Multiple regression (a change in the dependent variable can be explained by more than one independent variable)Curvilinear regression (relationship between dependent and independent variable is not liner, but based on a curve)Learning curve (describe how costs decrease as the quantity of an item increases)

Methodologies Parametric CERs (contd)Statistics may be used to evaluate how well the CER will produce a reliable estimate.Coefficient of determination, R2:Percent of the variation in the Y-data explained by the X-data, (ie. How close the points are to the line)Standard Error, SE:Average estimating error when using the equation as the estimating ruleCoefficient of Variation, CV:SE divided by mean of the Y-data, relative measure of estimating errort-statTests whether the individual X-variable(s) is/are validF-statTests whether the entire equation, as a whole, is validNo single statistic may validate or invalidate a CER, quality of the input data is just as important.Best to use a statistical software package like SAS or SPSS to quickly evaluate alternative CERs.

MethodologiesParametric CERs - StepsDefine the dependent variable (e.g., cost dollars, hours, etc.) and what the CER will estimate,Select independent variables to be tested for developing estimates of the dependent variable, Variables should be quantitatively measurable and availableIf there is a choice between developing a CER based on performance or physical characteristics, performance characteristics are generally the better choice because they are known early onCollect data concerning the relationship between the dependent and independent variables,Most difficult and time consuming step, but essential that all data is checked to ensure that all observations are relevant, comparable, and free of unusual costsExplore the relationship between the dependent and independent variables,Use statistical analysis to judge strength of relationship and validity of equationSelect the relationship that best predicts the dependent variable, andA high correlation often indicates that the independent variable will be a good predictive toolDocument your findings.Identify independent variables tested, data gathered along with sources, time period (normalization for inflation effects), and any adjustments made to the data

Methodologies Parametric Learning CurvesBasic premise: Repetitive tasks should result in productivity for subsequent, similar tasksThis improvement is usually quantified at a rate Y = AXbIn simplest terms, the cost of manufacturing or installing a unit should decrease as the number of units involved increases.As the number of units produced doubles, the cost per unit decreases by a fixed percentageThe concept of learning curves is not new, originated in the mid-1930s with T.P. Wright in the Journal of Aeronautical Sciences.

MethodologiesParametric Learning Curve - ExampleSay that the first 100 tasks of an installation took 10 hours per task and the next 100 averaged 8 hours per task. Thus, the learning curve would be calculated as follows:

Learning curve = 8 hours per task/10 hours per task = 0.8

Implies an 80% learning curve meaning an improvement of 20% occurred between the first 100 tasks and next 200 tasks

Methodologies Rate ImpactBasic premise: The number of units produced in a single lot effects the overall cost of producing that lot.Costco theory that buying in bulk makes the unit cost lessMathematical equation showing rate impact along with learning curveY = AXbQrBoth rate and learning curves can be impacted by the following:Operator turnover rate (new employees do not meet expected productivity standards immediately)Production reworksMaterial handling and downtime (learning curves assume material is ready when needed)Engineering reworkRework of vendor supplied parts

MethodologiesEngineeringAlso referred to as bottoms up or detailed method.Underlying assumption is that future costs for a system can be predicted with a great deal of accuracy from historical costs of that system.Involves examining separate work segments in detail and then synthesizing these detailed estimates along with any integration costs into a total program estimate.Estimate is built up from the lowest level of system costs.Uses detailed cost element structure (CES).Must include all components and functions.Can be used during development and production.

MethodologiesEngineering (contd)Most useful for systems in production.Design configuration has stabilizedTest results are availableDevelopment cost actuals are availableTypically broken into functional labor categories e.g. engineering, manufacturing, quality control, etc.

MethodologiesEngineeringPros:ObjectiveReduced uncertaintyCons:ExpensiveTime ConsumingNot useful early onMay leave out software integration efforts

MethodologiesEngineering StepsUnderstand program requirements,Prepare program baseline definition,Define ground rules and assumptions,Develop detailed cost element structure,Develop functional estimates, andUse other program historyCompile estimate dataDevelop rates and factorsIncorporate supplier/subcontractor pricesInclude integration costs to glue the separate components into an integrated system (may need to use a CER for this estimate)Summarize estimate.

MethodologiesEngineering ExampleNew IT system can be broken down into Labor and Material costs.

Labor: Estimated number of hours250,000 hrs * $200/hr = $50MMaterial: From vendor quotesServer = $10M COTS = $3M Desktops (50 @ $100k) = $5M

Cost = $50M+$10M+$3M+$5M = $68M

MethodologiesActualBases future costs on recent historical costs of same system.Used later in development or production.Preferred method. Costs are calibrated to actual development or production productivity for your organization

MethodologiesActualPros:Most accurateMost objective of the five methodologiesCons:Data not available early onTime consuming Labor intensive to collect all the data necessary

Methodologies Actual - ExampleThe development process is nearing completion. The material has all been procured at a cost of $20M.

The labor cost to date is $30M. According to earned value cost performance reports (CPRs) the estimate at complete for the remainder of the labor is another $20M.

Cost = $20M + $30M + $20M = $70M

Methodologies Summary of the Five Cost Estimate ResultsExpert - $50MAnalogy - $62MParametric $65M Engineering $68MActual - $70M

Software Cost ModelsSoftware costs as a percentage of total system costs continues to increase while associated hardware costs decrease.Accurately capturing software costs can be difficult and cost overruns often occur as a result of software requirements being difficult to estimate and track.Software estimating problems occur most often because of the:Inability to accurately size a software project,Inability to accurately specify a software development and support environment,Improper assessment of staffing levels and skills, andLack of well-defined requirements for the specific software activity being estimated

Software Cost Models (contd)The requirement to develop cost estimates for systems at a time when few details are known has led to the development of cost models.The need to generate estimates quickly and for many alternatives can make the analogy and engineering methods impractical due to lack of detailed requirements.In the early stages of software development, few details other than performance requirements and some physical constraints are available.Software lines of code (SLOC) are often used to estimate software size which is the most significant driver in determining software costs.Programming languages also have a significant effect on overall cost.

Software Cost Models (contd)Software cost models are based on statistically derived cost estimating relationships (CERs) and various estimating methodologies used to predict the cost of a system.Models approximate the real world through a combination ofStatistical analysis of historical data,Informal/Intuitive analysis of rules of thumb based on experience,Standard procedures such as learning curves, inflation computation, and labor rate/overhead applications. Models may contain the following features:Support costs calculations, time-phasing to spread development costs by fiscal year, & inflation calculations to convert base year to then-year dollars.

Software Cost Models (contd)

Regardless of how software is programmed, it must proceed through certain steps or phases and must be supported after development. The combination of software development and support is known as the software life cycle.The Cost Element Structure (CES) discussed earlier accounts for the software life cycle using the following sub-elements for CES 1.3 Software Development:Requirements Definition Analysis, Design, and Integration Quality Assurance Program Configuration Management Software Design and Development HW/SW Integration, Assembly, Test and Checkout System Operational Test and Evaluation System Independent Software Verification and Validation Site Acceptance Testing Independent Operational Test and Evaluation Program Management Installation and Checkout Corrective Maintenance

Software Cost Models (contd)Examples of Commercially Available ModelsCOCOMOCOSTXPERTSLIMSEERCostar, REVIC, etc.

Software Cost Models (contd)COCOMOConstructive Cost Model (COCOMO) is one of the earliest cost models widely used by the cost estimating community.COCOMO was originally published in Software Engineering Economics by Dr. Barry Boehm in 1981.COCOMO is a regression-based model that considers various historical programs software size and multipliers.

Software Cost Models (contd)COCOMOCOCOMOs most fundamental calculation is the use of the Effort Equation to estimate the number of Person-Months required to develop a project. Example: # of person months * loaded labor rate = Estimated CostMost of the other estimates (requirements, maintenance, etc) are derived from this quantity.

Software Cost Models (contd)COCOMOCOCOMO requires as input the projects estimated size in Source Lines of Code (SLOC). In addition to SLOC, COCOMO has scale factors which allow you to tailor conditions to your project.Scale factors are used to calibrate or adjust the model that is based on data which is not necessarily representative of the system being estimatedIntent of calibrating is to allow for general application of a model developed from a specific databaseBased on the assumption that any differences between the predicted cost and the historical cost not explained by the model are due to:Differences in the type of systems and acquisition environment represented in the two data sets

Software Cost Models (contd)COCOMOScale factors include: Development flexibility, architecture, risk, team cohesion, and process maturity among others.Additionally COCOMO uses Cost Drivers grouped in broad categories such as personnel, product, platform, project etc.Product Cost drivers include: required reliability and product complexity.Personnel cost drivers include: language, platform, and tool experience

Software Cost Models (contd)COCOMOCOCOMO defaults to a nominal value for all factors when model is first used.Cost estimators must calibrate the model to the program being estimated to their specific development environment and productivity level of staff.COCOMO uses analogy and parametric methods based pm a historical, proprietary database of data submitted by consortium members.Allows COCOMO to be compatible with current design methods and evolving higher order languages.

Software Cost Models (contd)COCOMOThe model makes its estimates of required effort (measured in Person-Months) based primarily on the projects estimate of the software size (as measured in thousands of SLOC, KSLOC)):

Effort = 2.94 * EAF * (KSLOC)**E

Where EAF is the Effort Adjustment Factor derived from the Cost Drivers and E is an exponent derived from the five Scale Drivers.

Software Cost Models (contd)COCOMOFor example assuming a project with all nominal cost and scale drivers, EAF would have a value of 1.00 and exponent, E, of 1.0997. Assuming that the project consists of 15,000 source lines of code, COCOMO estimates that 57.77 Person-Months of effort is required to complete it. Effort = 2.94 * (1.0) * (15)**1.0997 = 57.77 Person-Months.Person-months multiplied by a loaded labor rate (includes direct and overhead costs) would yield the cost estimate:57.77 person-months * $15K per person-month = $866,000

Software Cost Models (contd)COCOMO Model output example

Software Cost Models (contd)COCOMO

Example of how model defaults to nominal scale factors.

These should be adjusted for every estimate to approximate the new, unique development effort.

Software Cost Models (contd)COCOMO outputs by phase

Software Cost Models (contd) COCOMO outputs by phase

Software Cost Models (contd)COCOMO overall estimate by phase

Software Cost Models (contd)COCOMO SummaryCOCOMO provides the following information:Software development, maintenance cost and schedule estimates.Cost, schedule distribution by phase, activity, and increment.A free updated version of COCOMO is available at: http://sunset.usc.edu/research/COCOMOII/

Software Cost Models (contd)COCOMOCOSTXPERTSLIMSEERREVICCostar, Price S, etc.

Software Cost Models (contd)CostXpertCostXpert builds on COCOMO by having a database of actual projects that is constantly being updated for various types of systems (commercial, military, etc.).When you enter project specific information, CostXpert maps it to the closest projects in its database and gives you results tailored to your project.CostXpert uses both parametric and analogy estimating methods.CostXpert can accommodate a variety of software sizing elements such as SLOC, function points, object metrics, GUI metrics, and feature points.

Software Cost Models (contd)CostXpertCostXpert also allows for customization specifically to your work environment.In addition, CostXpert creates a detailed cost element structure (CES), conducts what if scenarios, risk analysis, and provides a wide variety of reports to include:Cost allocations by task list,Labor loading by phase and functional categories,Maintenance and life-cycle costs,Software defects estimates, andDocumentation deliverablesGAO uses CostXpert during various cost analysis assessments to check the reasonableness of both the software cost estimate and accompanying schedule for delivery.Evaluation copies can be downloaded at http://www.costxpert.com/products/downloads.html

Software Cost Models (contd)CostXpert example of input screen

Software Cost Models (contd)COCOMOCOSTXPERTSLIMSEERREVICCostar, Price S, etc.

Software Cost Models (contd)SLIMThe Software Life Cycle Model (SLIM) is marketed by Quantitative Software (QSM). Set of tools include SLIM-Estimator, SLIM-Data Manager, SLIM-Control, SLIM-Metrics and SLIM-Master Plan.Input metrics include SLOC, Function Points, Objects, Windows, Screens, or DiagramsHistorical productivity and complexity are also inputs to the modelQuick start wizard enable rapid generation of estimates and details which can then be tailored for the specific project.

Software Cost Models (contd)SLIMResults are presented at the 50% probability and can be adjusted by using sliders for various parameters.Estimates can readily be compared to historical data and model allows for analysis of what if scenarios. A Productivity Index (PI) is calculated for your project and updates as new data is available.

Software Cost Models (contd)SLIMAs described in the users manual, the PI is a macro measure of the total development environment. It incorporates many factors in software development, including: Management influence, development methods, tool, techniques, skill and experience.Values for PI range from .1 to 40.Low values generally are associated with poor environments, process, tools and complex systems.High values are associated with mature processes, and capable management and well-understood, straightforward projects.Outputs of the model include: project description, estimation analysis views, schedule, risk analysis, staffing and skill breakout, effort and cost, reliability estimates, and documentation sections.Demo copies of SLIM can be downloaded at http://www.qsm.com/ .

Software Cost Models (contd)COCOMOCOSTXPERTSLIMSEERREVICCostar, Price S, etc.

Software Cost Models (contd)SEERThis set of tools provides both software and hardware estimation capabilities.Software Project Control Software Estimation, Planning and Project ControlHardware Project Control Hardware Estimation, Planning, Project Control and Life-Cycle Cost Analysis, Including Operations and SupportDesign for Manufacturability Cost Designer for Parts, Process and Assembly

Software Cost Models (contd)SEERThe toolset includes:SEER-SEM for evaluating software development. SEER-H for estimating hardware and hardware Operations and Support costs.SEER-DFM for manufacturability analysis. SEER-IC for estimating foundry costs. SEER-SSM for software sizing. Outputs of the model includeLabor estimates by function, A quick estimate providing a snapshot of size, effort, and schedule, andStaffing by month, cost by activity, person-months by activity, delivered defects, and SEI maturity rating.Demo copies of SER-SEM and other tools can be downloaded at http://www.galorath.com/tools_overvw.shtm

Software Cost Models (contd)COCOMOCOSTXPERTSLIMSEERREVICCostar, Price S, etc.

Software Cost Models (contd)REVICThe Revised Enhanced Version of Intermediate COCOMO (REVIC) model was developed by Mr. Raymond L. Kile formerly of Hughes Aerospace.

The main difference between REVIC and COCOMO is the coefficients used in the effort equations. REVIC changed the coefficients based on a database of recently completed DOD projects.

REVIC's schedule estimates are often considered lengthy because it assumes that a project's documentation and reviews comply with the full requirements of DOD-STD-2167A/498.

Software Cost Models (contd)COCOMOCOSTXPERTSLIMSEERREVICCostar, Price S, etc.

Software Cost Models (contd)Costar, Price S, etc.

There are many other models available.Most are based on COCOMO.Almost all need to be calibrated with project specific data.Using multiple models and comparing results provides better confidence.Crosschecking estimates leads to better cost estimate validationThe International Council on Systems Engineering (INCOSE) has a list of software tools that may be of use.http://www.incose.org/tools/tooltax/costest_tools.html

Cross-checks and ValidationAfter an estimate has been created, the next step involves validating the estimate by cross-checking.Cross-checking means using a different approach to create the estimate.If both estimates are close, the target estimate has some validity.If both estimates are very different.This increases the level of uncertainty which must be reflected in a risk analysis.This may lead to another estimating method to increase cost estimate confidence.It is a good practice to cross-check major cost drivers.If time is available, cross-checking other cost elements can further validate the entire estimate.Factors tend to be the most common approach for top level checks.Software cost models play an important role during later stages of development phase. Can be used to cross-check the reasonableness of costs forecasted using either the engineering or actual cost methods.

Cross-checks and Validation (contd)Validation is the process of demonstrating that either a CER or a cost model accurately predicts past history.Validation also includes a demonstration that:The data relationships are logical,The data used are credible,Strong data relationships exist, and The CER or cost model is a good predictor of costs.This can be done using independent test data (not included in models calibration).The models cost output would then be compared to the test points known value to provide an accuracy benchmark.

Cross-checks and Validation (contd)

Validation also includes a demonstration that:Model users have sufficient experience and training,Calibration processes are thoroughly documented,Formal estimating policies and procedures are established, andWhen applicable, information system controls are maintained to ensure the integrity of the models being used.

Risk and Sensitivity AnalysisRisk Analysis DefinitionRisk analysis is a process that uses qualitative and quantitative techniques for analyzing, quantifying and reducing uncertainty associated with cost goals.By nature, all cost estimates have some uncertainty. Earlier in development that uncertainty is higher. As the project matures these uncertainties decrease due to greater design definition, actual experience, and less opportunity for change.Errors can also occur from historical data inconsistencies.Cost risk analysis aims to achieve the following objectives:Identify program level confidence for development schedule,Provide credibility to the target estimate, and Identify technical, schedule, and cost estimating risk drivers for use in risk management.

Risk and Sensitivity AnalysisRisk Analysis Definition (contd)Risk is defined as a situation in which the outcome is subject to an uncontrollable random event stemming from a known probability distribution.Roll of two dice is an example since the roll can result in one of 11 possible outcomes.Uncertainty is defined as a situation in which the outcome is subject to an uncontrollable random event stemming from an unknown probability distribution.Example would be will it rain two weeks from today?Cost estimating falls more into the range of uncertainty than risk, but most managers use the term risk analysis.

Risk and Sensitivity AnalysisSensitivity Analysis Definition (contd)Sensitivity analysis answers the question:What happens if the assumptions change?Changes to some assumptions can have profound impact, while huge changes to other assumptions have little effect on results.Sensitivity examines the effect of primary cost estimating outputs to changes on individual CES inputs.Sensitivity analysis highlights the factors that have the strongest impact on the overall cost estimate.Points to management which factors deserve the most attention. Narrows down the number of lower level cost elements that should be examined using risk analysis techniques.

Risk and Sensitivity AnalysisSources of Cost RisksSchedule and Technical risksUnexpected design changesProject team experienceNumber of business units impactedRequirements changesIntegration considerationsTechnical difficulties or maturity issuesRevised project or acquisition plansQuantity changesNew labor ratesHigher inflationCost estimating risksImprecision associated with the estimating techniques used, errors, or oversights

Risk and Sensitivity AnalysisRisk Analysis TechniquesRisk factor approach Adds cost to the overall point estimate (most likely estimate) to cover risks by way of a factor.This factor is usually a percentage derived from past data and experience.Often applied to the estimate as a whole versus lower level cost elements.Monte Carlo Simulation Automatically analyzes the effect of varying inputs on outputs of a cost model using spreadsheet risk analysis.Randomly generates values for uncertain variables over and over to simulate a model.

Risk and Sensitivity AnalysisRisk Analysis Techniques (contd)Monte Carlo Simulation StepsAnalyst obtains cost distribution for each element identified as a major cost driver either through experience or sensitivity analysis. Cost distribution is usually triangular in the form of optimistic, most likely, and pessimistic cost estimates.These cost ranges are usually obtained through expert judgment (engineer or technical specialist interviews) or using a Delphi techniqueAfter all cost elements have been identified by a distribution, the simulation is run many times (1,000 10,000 times).Simulation calculates multiple scenarios of the cost model by repeatedly sampling values from the probability distributions assigned to the various cost elements.While the simulation runs, the forecasts stabilize towards a smooth frequency distribution called a cumulative frequency distribution or CDFAfter thousands of trials, statistics of the results and the certainty of any outcome can be obtained from the CDF.

Risk and Sensitivity AnalysisRisk Analysis Techniques (contd)Monte Carlo Simulation ResultsReveal not only the result values for each forecast, but also the probability of any value occurring.This is helpful to management because it can show the level of certainty of achieving a cost objective.For example, a simulation can show there is a 10% chance of the project finishing for $50 million, a 50% chance of it costing $ 70 million, and a 90% chance of developing the project for $100 million or less.Decision makers can use these results to decide which projects will continue funding based on quantifiable risk parameters.

Risk and Sensitivity AnalysisTypes of Risk Simulation ToolsUS GAO uses Crystal Ball to perform Monte Carlo Simulations Crystal Ball is an simulation program that helps you analyze the risks and uncertainties associated with Microsoft Excel spreadsheet based models. http://www.decisioneering.com/crystal_ball/Other tools are available, such as @Risk (http://www.palisade.com/html/risk.html)RiskEase (http://www.riskease.com/)Pertmaster (http://www.pertmaster.com/)

DocumentationAfter cross-checking the estimate major cost drivers, the next step, and most important one of all, involves documenting the entire estimate process.Although this is a difficult and time-consuming procedure, the level of detail and attention will pay big dividends when it comes time to re-estimateMay also help data collection of actual analogous costs.Important to document the estimate as it is being developed (i.e., document as you go).Hard to remember rationale and judgments for adjusting data months later when it needs to be documented.Documenting as you prepare the estimate leads to a better quality estimate and requires minimum effort at the end.

Documentation (contd)Best to provide more vs. less informationAssume reader knows nothing about the system being estimated.Include step-by-step instructions for how the estimate was developed.Aim to provide enough information for the estimate to be recreated by a cost analyst independent of the team.Most users of the documentation will either be updating the estimate at a later date or will relay on data for estimating an analogous system.

Documentation (contd)Documentation should include at a minimum:Introduction Purpose of estimateEstimate team members, POC information, and area of responsibilityProgram background and system descriptionProgram schedule (necessary for phasing costs over time)Scope of estimate (including what was omitted and why)Ground rules and assumptions (all technical and program specific assumptions that bound the estimate e.g., constant 2003 dollars using xx inflation rates)Summary of estimate by year to show phasing by CESOverview of main body that describes how each CES element was estimated

Documentation (contd)Summary of estimate by year showing phasing by CES element (In constant 2002 millions of dollars)

Documentation (contd)Main Body documentation should include:Enough detail for each CES element that would allow for another analyst to replicate the cost estimate.Each CES element should provide a description of the methods (with rationale for choosing it), techniques, and data used to generate the estimate as well as any cross-checks done to add rigor to the estimate.Include information such as data sources, direct/indirect labor rates, labor hours, material/subcontracts, learning curve data and methodology, factors, cost estimating relationships with corresponding statistics and data ranges, cost models used, inflation indices, time phasing (e.g., development lasts 3 years and 10 years operations and support), estimator judgments, cross-checks, risk and confidence levels.Documentation flow should follow the same CES structure shown in the estimate summary by year.If the estimate is an update to a prior estimate, then provide an explanation for any differences.

Documentation (contd)Be sure to spell out all acronyms when first introduced.Cross-reference the reader to where data is used multiple times.Show data at the lowest levels possible.Include copies of vendor quotes, studies used, statistical analysis printouts, cost model input and output reports, etc.Include disclaimers to show the level of risk and uncertainty surrounding estimate.Try to quantify the uncertainty by using a simulation model that will express the summary estimate in terms of level of confidence: Estimate is presented at the 90% confidence level, orPoint estimate of $1 million is bounded by a range of $750,000 to $1,250,000

Cost Estimating ChallengesAccess to historical dataNeed to invest in database capture of historical costs and technical data for proper CER developmentCostly, time consuming, and usually not fundedDevelopment costs for IT systems can quickly become outdated by new programming languages Maintenance costs are even more difficult to capture because they are seen as on-going support or overhead and not as metrics System architecture change effects on cost estimates can be hard to determineValidity and uncertainty of dataGarbage in = Garbage outLimited time to develop estimatesCan result in rough-order magnitude costs being used as budget quality estimatesCause important steps like validation and Monte Carlo simulation to be omittedResourcesLack of trained people is a problem

Cost Estimating Auditor ChecklistsVarious auditor checklists can be found in the many source documents used to create this briefing. A few of these are shown below:The Software Engineering Institute (SEI) has created "A Manager's Checklist for Validating Software Cost and Schedule Estimates." This checklist is designed to help managers access the credibility of software cost and schedule estimates.It identifies seven issues to address and questions to ask when determining your willingness to accept and use a software estimate. Each question is associated with elements of evidence that, if present, support the credibility of the estimate. http://www.sei.cmu.edu/publications/documents/95.reports/95.sr.004.htmlJoint Industry/Government Parametric Estimating Handbook (Second Edition, Spring 1999)http://www.jsc.nasa.gov/bu2/PCEHHTML/pceh.htmChapter 6, Auditing and Evaluating a CER & Auditing Parametric EstimatesAppendix G, Parametric Estimating System Checklist

Cost Estimating Auditor Checklists (contd)Various auditor checklists can be found in the many source documents used to create this briefing. A few of these are shown below:Department of the Army Economic Analysis Manual (U.S. Army Cost and Economic Analysis Center, Feb 2001) http://www.asafm.army.mil/pubs/cdfs/manual/economic.pdfAppendix M, Economic Analysis ChecklistDepartment of the Army Cost Analysis Manual (U.S. Army Cost and Economic Analysis Center, May 2002)http://www.ceac.army.mil/pubs/default.aspLink to a zipped MS Word file for downloading or printingChecklist on page 42, Figure 3-4 Validation ConsiderationsTASC (The Analytic Sciences Corp.), The AFSC Cost Estimating Handbook, Reading, MA, prepared for the U.S. AirForce Systems Command, 1986.Chapter 8, Cost Models pages 8-6 through 8-12Appendix D, Staff Review Cost Estimate Checklist pages D-1 through D-8

Cost Estimating Auditor Checklists (contd)Various auditor checklists can be found in the many source documents used to create this briefing. A few of these are shown below:Navy Post Graduate School Economic Analysis Handbookhttp://www.nps.navy.mil/drmi/chapter6.htmChapter 6, Guide for ReviewersNational Research Council Canada (NRC-CNRC) Software Cost Estimation and Control (1994)http://seg.iit.nrc.ca/English/abstracts/NRC37116.htmlAppendix A, pages 59-67, Software Cost Estimation Questionnaire

Contact InformationFor further information please contact us:Karen Richey (202) 512 4784 or richeyk@gao.govMadhav Panwar (202) 512- 6228 or panwarm@gao.govJennifer Echard (202) 512 3875 or echardj@gao.gov

* Pg 9-9 AF handbook* Parametric Estimating Handbook pg 1-1SEI Definition of RiskBusiness Case Essentials, pg 21FEA Guidebook, pg 4-10Economic Analysis Handbook, pg 13Business Case Essentials, pg 21. Modified example.