ERP Sample Planner 2006

Instructions & Main MenuQuick Guide to the ERP Sample PlannerThe Sample Planner is a tool to help you evaluate different approaches for conductingthe random inspections in the Environmental Results Program (ERP). Please fully readthis sheet before using the Sample Planner for the first time.This Quick Guide Sheet Contains:an overview of the tools within the Sample Planner,instructions about assumptions and use, andsuggestions for other resources.Overview of Spreadsheet Tools in the Sample PlannerThe worksheets in the Sample Planner let you evaluate your approach for conductinginspections two different ways:you can calculate the sample size needed to meet specified data quality requirementsfor a sample, oryou can evaluate the quality of your estimates based on the number of inspectionsyou can conduct. Each is described in the following table. (Click on the spreadsheetname to go to that sheet.)Spreadsheet NameDescriptionCalculate Number of Inspections NeededFind Sample Size (1-Sample)Calculates the number of inspections needed toachieve a specified margin of error andconfidence level for a single sample.Find Sample Size (2-Sample)Calculates the number of inspections needed toachieve a specified margin of error andconfidence level for the difference between results betweentwo rounds of inspections. Use this when yourmost important evaluation will be of thedifferences between the two rounds of inspections.of inspections.Calculate Margin of Error for One or Two rounds of InspectionsFind Margin of Error (1-Sample)Calculates the margin of error associatedwith a specified number of inspections ina single sample.Find Margin of Error (2-Sample)Calculates the margin of error associated with the difference inresults observed between two rounds ofinspections of given sizes. Use this when yourmost important evaluation will be of thedifferences between two samples.InstructionsColor scheme. Each sheet has a user-friendly color scheme that allows you to readilynavigate the sheet:Blue cells provide text information for you.Yellow cells are where you enter data.Green cells provide results.Gray cells contain assumptions and calculations, for advanced users.Terminology and definitions. Statistics often uses specialized terminology. As muchas possible, this tool presents plain language along with specialized terms andcalculations. There is also a glossary on the second tab of this workbook.Instructional comments. Be sure to read the additional information about the inputs andresults, provided in comments in the cells. Cells with comments have a small red flagin the upper right-hand corner. Place the cursor over the cell to see the comment.Protected cells. All the cells in this workbook are protected, except for the yellow oneswhere you enter data. Consequently, you do not need to worry about accidentallychanging the formulas. If you do want to modify protected cells, please do so with caution.You can unprotect a cell or worksheet under the Tools menu. There is no password.Planning your sample to estimate proportions and means. Note that this toolassumes you want to estimate a proportion. It does not let you design a sample with thegoal of estimating the mean and standard error of a continuous variable (e.g., the averagequantity of hazardous waste generated by facilities). Typically, ERP planners do not havethe necessary information in advance to undertake such designs anyway. You will still beable to measure continuous variables in your ERP, and the ERP Results Analyzer allowsyou to calculate results for continuous variables. However, you probably will not have anadvance idea of the uncertainty that will be associated with these estimates.Ease of viewing. Use of this tool is optimized for an average desktop monitor andMicrosoft Excel setup. Some users, including those with notebook computers, may findthe tool easier to use if they change the "zoom" to less than 100% -- e.g., to show moreof the comment boxes without scrolling. Choosing the "full-screen" option will also enableseeing more of the tool. Both of these options are accessible under the "View" menu inthe Microsoft Excel toolbar.Other ResourcesFor more information on the use of statistics in ERP, please refer to the Generic Guide toStatistical Aspects of Developing an Environmental Results Program. Please note thatthe formulas used in this spreadsheet are different than (and improved from) those in theApril 25, 2003, version of the Generic Guide.The spreadsheet ResultsAnalyzer2006.xls will help you conduct statistical tests andcalculate confidence intervals based on the results of your inspections.

Find Sample Size (2-Sample)Find Margin of Error (2-Sample)Find Sample Size (1-Sample)Find Sample Size (1-Sample)Find Sample Size (2-Sample)Find Margin of Error (1-Sample)Find Margin of Error (1-Sample)Find Margin of Error (2-Sample)

DefinitionsDefinitions of Key TermsPopulation (N) - This is the total set of facilities in your universe. You may want todetermine the proportion of this group of facilities that is in compliance with a certainregulation or makes use of a certain type of treatment technology.Sample Size (n) - The sample size refers to the number of random inspections. Asample that consists of the entire population is called a census.Point Estimate of the Proportion (p) - This usually refers to the proportion offacilities in the sample that meet your criteria (e.g., are in compliance).Confidence Interval and Margin of Error (e) - The margin of error and aconfidence interval associated with the random sample are computed to reflect theuncertainty associated with your point estimate of the actual population proportion ormean.The confidence interval gives a range of values that is believed to contain the actualpopulation proportion or mean with the confidence level prescribed by the user (seebelow). There are several different acceptable ways of constructing confidence intervals.The standard Wald confidence interval is symmetric about the point estimate. The Scoreinterval (used in this tool for single-sample estimations) is not symmetrical except for apoint estimate of 50%, but it is often shorter (i.e., more precise). It is particularly usefulfor small sample sizes and for estimating single proportions, especially when they areoutside the 30-70 percent range. For more information on this topic, please see thejournal article by Agresti and Coull, "Approximate is Better than 'Exact' for IntervalEstimation of Binomial Proportions," in The American Statistician, 1998.The margin of error is half the width of the standard confidence interval. It likewiserequires the user to select a confidence level. The Wald confidence interval is formed byadding and subtracting the margin of error to the point estimate. For example, if themargin of error of an estimated proportion is 5 percentage points, the confidence intervalwill range from 5 percentage points less than the observed value to 5 percentage pointsmore than the observed value. In this example, if the observed proportion is 50 percent,the confidence interval would be 45 to 55 percent. Notice that the length of the Waldconfidence interval is twice the margin of error.The length of the Score confidence interval is also twice the margin of error when theobserved proportion is 50% (but not for any other observation). When calculating samplesizes required for a particular margin of error, the Sample Planner assumes a 50%observed proportion, in order to provide the maximum sample size necessary as well as toprovide a symmetrical Score confidence interval that is twice the margin of error.Users interested in viewing the Wald confidence interval can unprotect the spreadsheets(see Instructions) and "unhide" hidden rows in the "Results" section of the "Find Margin ofError (1-Sample)" worksheet in the Sample Planner. In the Results Analyzer's "Proportion(1-Sample)" worksheet, the Wald confidence interval is shown in the "For AdvancedUsers" section.Confidence Level - The confidence level is the percentage of all possible randomsamples of size n whose corresponding confidence intervals contain the actual (unknown)population proportion or mean. (See the definition of Confidence Interval above.) Forinstance, suppose the true proportion of all facilities in compliance with a particularregulation is 62 percent. (In practice, you would only know this if all facilities were in thesample.) A 95-percent confidence level means that approximately 95% of all randomsamples of n facilities from the population will produce a confidence interval that includes62% (in a confidence interval, the lower limit will be less than 62% and the upper limit willbe greater than 62%). Since you are selecting just one random sample, thecorresponding confidence interval may or may not contain the true proportion, but the factthat 95% of all possible samples would contain the true proportion is expressed bysaying you have 95% confidence that your particular interval is one of these accuratesamples. You do not know for sure that this is the case, however.To determine the necessary sample size, the user must specify the desired confidencelevel and the desired margin of error.Comparing Two Proportions - A margin of error or confidence interval can bedeveloped for a difference in proportions or means, as well. Because uncertainty isassociated with each of the observations, the difference between the observations alsohas uncertainty associated with it. This uncertainty can be described with a margin oferror or confidence interval. For instance, if you observe 40 percent compliance in Round1 of inspections and 60 percent compliance in Round 2 of inspections, you have observeda difference of 20 percentage points. If you calculated a margin of error of 10 percentagepoints for this example, you could say that you believe performance improved by 20percentage points, 10 percentage points, or somewhere between 10 and 30 percentagepoints.Note that if the confidence interval contains zero, you can not be confident that anyimprovement or change occurred. For instance, if you observed a difference of 20percentage points 25 percentage points, the confidence interval of the difference wouldbe from - 5 to 45 percentage points. This interval includes zero, so you could not beconfident that there was a change.Go Back to Instructions / Main Menu

Go Back to Instructions / Main Menu

Find Sample Size (1-Sample)Estimate the Sample Size Required for One Round of InspectionsGo Back to Instructions / Main MenuTo Definitions of Key TermsWhen to use this spreadsheet: When you want to find out how many inspections you should conduct in a round of random inspections, based on the margin of error and confidence level you are seeking.Enter your data in the yellow cellsThe results are shown in green cells. These cells are locked.Assumptions and calculations are shown in gray cells. These cells are locked, too.Information neededInput/ResultsStatistical terminologyEnter Your InformationWhat confidence level do you want to have?Confidence levelHow many facilities is your ERP focusing on? (Estimate if you do not know.)1,000Population or NWhat margin of error do you want, above and below the estimate of the proportion of facilities that have the characteristic you are measuring (e.g., are in compliance)?5.0%Margin of error or Half-width of confidence intervalResultsNumber of inspections required277Sample required (finite population corrected)For Advanced Users: Assumptions and CalculationsInspections required with an infinite population382Sample required for Score intervalAssumed proportion, p0.50Assumed proportion, pp(1-p)0.25p(1-p)Confidence level95%Confidence level, (1-)Factor to achieve level of confidence required1.960Value of a standard normal, Z(1-/2)Reference:Derived from: Agresti and Coull. 1998. "Approximate is Better than 'Exact' for Interval Estimation of BinomialProportions." The American Statistician, v. 52, no. 2, 119-126.

This is half the size you would like to have for your confidence interval. E.g., to have an estimate that is within 5 percentage points, enter "0.05."

Assume the estimate of the proportion of facilities in compliance is 50 percent, you chose a confidence level of 95 percent, and the size of the interval is 5 percentage points. In this case, you would have 95 percent confidence that the true proportion of facilities in compliance is 45 to 55 percent.

The smaller the margin of error, the larger the sample size needed.

Note that the Score confidence interval is asymmetrical at proportions other than 50%, so the concept of a margin of error does not readily fit the Score confidence interval in those situations. You are able to use the concept of margin of error in this Sample Planner tool, however, because the Sample Planner assumes a proportion of 50%--a conservative approach that will yield the maximum sample size necessary. For more information on the Score confidence interval (and the Wald confidence interval), see the "Definitions" sheet.This is the confidence level, an estimate of the certainty associated with the margin of error or confidence interval for your result. Conclusions based on sampling, rather than inspecting every facility, will always involve some amount of uncertainty.

You can choose either 90% or 95%, which are two standard levels of confidence. Most, if not all, ERPs to date have used a 95% confidence level.

E.g., if you estimate that 50 percent of facilities are in compliance, with a 95 percent confidence interval of 5 percent, you can state that you are 95 percent confident that the true proportion of facilities in compliance is between 45 and 55 percent.

The higher the level of confidence, the larger the number of inspections needed.This is the sample required to calculate a Score confidence interval, which is considered a more accurate estimate than the standard (Wald) interval, especially with smaller sample sizes and proportions close to 0 percent or 100 percent.

For more information on the Score confidence interval (and the Wald confidence interval), see the "Definitions" sheet.This is the population about which you wish to make inferences. E.g., if you want to know what proportion of 500 dry cleaners is in compliance with proposed standards, enter 500. If you're not sure how many there are, put the largest likely number to be sure you inspect enough facilities to achieve the confidence level and confidence interval you are seeking.

Typically, the characteristic you intend to measure applies to all facilities in the population. If the characteristic applies to a subset of facilities, you should enter the number to which it applies rather than the total number of facilities.Go Back to Instructions / Main MenuTo Definitions of Key Terms

Find Sample Size (2-Sample)Estimate Sample Sizes Required for a Specified Margin of Error for aDifference in Proportions (Two Samples)Go Back to Instructions / Main MenuTo Definitions of Key TermsWhen to use this spreadsheet: When you want to find out how many inspections you should conduct in each round of ERP inspections, based on the margin of error and confidence level you wish to achieve for a difference between proportions observed in two rounds of inspections. Use this when your most important evaluation will be of the differences between two samples.Enter your data in the yellow cellsThe results are shown in green cells. These cells are locked.Assumptions and calculations are shown in gray cells. These cells are locked, too.Information neededInput/ResultsStatistical terminologyEnter Your InformationWhat confidence level do you want to have?Confidence levelHow many facilities is your ERP focusing on in Round 1? (Estimate if you do not know.)1,000Population or N1If it differs from Round 1, how many facilities is your ERP focusing on in Round 2? (Estimate if you do not know.)1,000Population or N2What is your desired margin of error? (+/-)(You may not be able to detect differences smaller than this amount.)5.0%Margin of error or Half-width of confidence intervalResultsNumber of inspections required for Round 1.435Sample size, n1Number of inspections required for Round 2.435Sample size, n2For Advanced Users: Assumptions and CalculationsConfidence level95%(1-)Factor to achieve level of confidence required1.960Value of standard normal, Z(1-/2)Assumed proportion, p0.50Assumed proportion, pp(1-p)0.25p(1-p)Number of inspections, before finite population adjustment768mReference:Derived from: Kish, Leslie. 1965. Survey Sampling. John Wiley & Sons, Inc. New York, NY. p. 41.

This is the maximum margin of error that will be associated with differences in proportions between the two samples. The smaller the margin of error of the difference, the larger the sample size you will need for a given confidence level.

E.g., say your confidence level is 95% and margin of error is 5 percentage points, and you find 62% compliance in Round 1 and 72% compliance in Round 2. The observed difference between the two samples is 10 percentage points. Since the margin of error of the difference is 5 percentage points, the confidence interval of the difference ranges from 5 to 15 percentage points. Consequently, you would be able to say that you are 95% confident that compliance improved by between 5 and 15 percentage points. In other words, since you observed a change greater than the margin of error for the difference (in our example, 10 percentage points), you can be confident that a change occurred i.e., that there is a significant difference between the results from Round 1 and Round 2.

Suppose, on the other hand, that with a margin of error of 5 percentage points you find 62% compliance in Round 1 and 65% compliance in Round 2. The observed difference is 3 percentage points, and the confidence interval of the difference is 3 5 percentage points. In other words, the confidence interval ranges from or -2 to 8 percentage points. In this case, since the confidence interval of the difference includes zero (because the observed difference is less than the margin of error of the difference), it is possible that there was no change in performance in the overall population. Therefore, you cannot infer that a change in compliance occurred; you can not say that the results from Round 1 and Round 2 are significantly different.This is the confidence level, an estimate of the certainty associated with the margin of error or confidence interval for your result (in this case, your result is the difference between the findings from two samples). Conclusions based on sampling, rather than inspecting every facility, will always involve some amount of uncertainty.

You can choose either 90% or 95%, which are two standard levels of confidence. A 95% confidence level means that you can be 95% confident that the confidence interval includes the true difference between the two populations. Most, if not all, ERPs to date have used a 95% confidence level.

E.g., if you find that compliance improved 10 percentage points 5 percentage points between the first and second rounds of inspections (with a confidence level of 95%), you can state that you are 95% confident that the actual improvement in compliance is between 5 and 15 percentage points. The higher the level of confidence, the larger the number of inspections needed for a given confidence interval.This is the population about which you wish to make inferences in the first round of inspections. E.g., if you want to know what proportion of 500 dry cleaners is in compliance with proposed standards, enter 500. If you're not sure how many there are, put the largest likely number to be sure you inspect enough facilities to achieve the confidence level and confidence interval you are seeking.

Typically, you will want to sample from the same population for both rounds of inspections. E.g., if you are looking at drycleaners in compliance in the state, it is important to ensure you have an entire listing of drycleaners before both rounds of sampling.

Typically, the characteristic you intend to measure applies to all facilities in the population. If the characteristic applies to a subset of facilities, you should enter the number to which it applies rather than the total number of facilities.This is the population about which you wish to make inferences in the second round of inspections. For more information, see the comment above (regarding population, Round 1). This population size will typically be the same as for the first round of inspections. If the population size is expected to change substantially between rounds, then insert the expected population for Round 2 here. By default, this cell is set to equal the population for the first round of inspections.Go Back to Instructions / Main MenuTo Definitions of Key Terms

Find Margin of Error (1-Sample)Estimate the Margin of Error for One Round of InspectionsGo Back to Instructions / Main MenuTo Definitions of Key TermsWhen to use this spreadsheet: When you know the maximum number of inspections you can conduct in a round of random inspections, and you want to find out the margin of error you will have.Enter your data in the yellow cellsThe results are shown in green cells. These cells are locked.Assumptions and calculations are shown in gray cells. These cells are locked, too.Information neededInput/ResultsStatistical terminologyEnter Your InformationWhat confidence level do you want to have?Confidence levelHow many facilities is your ERP focusing on? (Estimate if you do not know.)1,000Population or NHow many inspections will you conduct in a single round?100Sample size or nResultsMargin of Error or Half-Width of Confidence Interval9.8%Wald intervalMargin of error (+/-)9.1%Half-width of the Score confidence intervalFor Advanced Users: Assumptions and CalculationsConfidence level95%(1-)Factor to achieve level of confidence required1.960Value of standard normal, Z(1-/2)Assumed proportion, p0.50Assumed proportion, pp(1-p)0.25p(1-p)Reference:Agresti and Coull. 1998. "Approximate is Better than 'Exact' for Interval Estimation of Binomial Proportions."The American Statistician, v. 52, no. 2, 119-126.

This is the total number of inspections you think you can conduct. I.e., it is the sample size.

If there is a maximum number of inspections you can carry out due to budget constraints or other reasons, enter that number here. You can vary this sample size and see how it affects the margin of error.This is half the size of your confidence interval. E.g., assume the estimate of the proportion of facilities in compliance is 50 percent, you chose a confidence level of 95%, and the size of the interval is 5 percentage points. In this case, you would have95% confidence that the true proportion of facilities in compliance is 45 to 55 percent.

The confidence interval calculated here is the Score confidence interval, which is considered a more accurate estimate than the standard (Wald) interval, especially with smaller sample sizes and proportions close to 0% or 100%.

Also note that the Score confidence interval is asymmetrical at proportions other than 50%, so the concept of a margin of error does not readily fit the Score confidence interval in those situations. You are able to use the concept of margin of error in this Sample Planner tool, however, because the Sample Planner assumes a proportion of 50% -- a conservative approach that will yield the maximum sample size necessary. For more information on the Score confidence interval (and the Wald confidence interval), see the "Definitions" sheet.This is the confidence level, an estimate of the certainty associated with the margin of error or confidence interval for your result. Conclusions based on sampling, rather than inspecting every facility, will always involve some amount of uncertainty.

You can choose either 90% or 95%, which are two standard levels of confidence. Most, if not all, ERPs to date have used a 95% confidence level.

E.g., if you estimate that 50 percent of facilities is in compliance, with a 95 percent confidence interval of 5 percent, you can state that you are 95 percent confident that the true proportion of facilities in compliance is between 45 and 55 percent.

The higher the level of confidence, the larger the number of inspections needed.This is the population about which you wish to make inferences. E.g., if you want to know what proportion of 500 dry cleaners is in compliance with proposed standards, enter 500. If you're not sure how many there are, put the largest likely number to be sure you inspect enough facilities to achieve the confidence level and confidence interval you are seeking.

Typically, the characteristic you intend to measure applies to all facilities in the population. If the characteristic applies to a subset of facilities, you should enter the number to which it applies rather than the total number of facilities.Go Back to Instructions / Main MenuTo Definitions of Key Terms

Find Margin of Error (2-Sample)Estimate the Margin of Error for Differences in Proportions (Two Samples)Go Back to Instructions / Main MenuTo Definitions of Key TermsWhen to use this spreadsheet: When you know the maximum number of inspections you can conduct in each round of ERP, and want to know the maximum margin of error to expect for a difference between proportions observed in two different rounds of inspections. Use this when your most important evaluation will be of the differences between two samples.Enter your data in the yellow cellsThe results are shown in green cells. These cells are locked.Assumptions and calculations are shown in gray cells. These cells are locked, too.Information neededInput/ResultsStatistical terminologyEnter Your InformationWhat confidence level do you want to have?Confidence levelHow many facilities is your ERP focusing on in Round 1? (Estimate if you do not know.)1,000Population or N1If it differs from Round 1, how many facilities is your ERP focusing on in Round 2? (Estimate if you do not know.)1,000Population or N2How many inspections will you conduct in Round 1?100Sample size or n1If it differs from Round 1, how many inspections will you conduct in Round 2?100Sample size or n2ResultsMaximum margin of error for a difference in proportions (+/-)13.2%Half-width of the confidence interval for a difference in proportionsFor Advanced Users: Assumptions and CalculationsConfidence level95%(1-)Factor to achieve level of confidence required1.960Value of standard normal, Z(1-/2)SE (p1-p2)6.7%Reference:Derived from: Kish, Leslie. 1965. Survey Sampling. John Wiley & Sons, Inc. New York, NY. p. 41.

This is the sample size for Round 1 of inspections -- i.e., the number of inspections you expect to conduct in the first round.

If there is a maximum number of inspections you can carry out in the first round due to budget constraints or other reasons, enter that number here. You can vary this sample size and see how it affects the margin of error.This is the maximum margin of error associated with the difference in proportions observed in the two samples. The margin of error is half the width of the confidence interval of the difference.

E.g., if your margin of error is 10 percentage points, then you can be confident (at a 90% or 95% confidence level) that the actual difference is within 10 percentage points of the observed value. If you were to observe a 15 percentage points difference between Round 1 and Round 2 results, you can be confident that the actual difference was between 5 and 25 percentage points. If you were to instead observe a 7 percentage points difference between Round 1 and Round 2, the confidence interval of the difference would be -3 to 17 percentage points. In this case, because the confidence interval of the difference includes zero, you can not be confident that a change occurred in the population as a whole i.e., the results from Round 1 and Round 2 are not significantly different. On the other hand, if you were to observe a change greater than the margin of error (in our example, 10 percentage points), you can be confident that a change occurred i.e., that there is a significant difference.This is the sample size for Round 2 of inspections -- i.e., the number of inspections you expect to conduct in the second round.

If there is a maximum number of inspections you can carry out in the second round due to budget constraints or other reasons, enter that number here. You can vary this sample size and see how it affects the margin of error.

Since many ERPs will have the same number of inspections in both rounds, this cell defaults to the Round 1 value.This is the population about which you wish to make inferences in the first round of inspections. E.g., if you want to know what proportion of 500 dry cleaners is in compliance with proposed standards, enter 500. If you're not sure how many there are, put the largest likely number to be sure you inspect enough facilities to achieve the confidence level and confidence interval you are seeking.

Typically, you will want to sample from the same population for both rounds of inspections. E.g., if you are looking at drycleaners in compliance in the state, it is important to ensure you have an entire listing of drycleaners before both rounds of sampling.

Typically, the characteristic you intend to measure applies to all facilities in the population. If the characteristic applies to a subset of facilities, you should enter the number to which it applies rather than the total number of facilities.This is the population about which you wish to make inferences in the second round of inspections. For more information, see the comment above (regarding population, Round 1). This population size will typically be the same as for the first round of inspections. If the population size is expected to change substantially between rounds, then insert the expected population for Round 2 here. By default, this cell is set to equal the population for the first round of inspections.This is the confidence level, an estimate of the certainty associated with the margin of error or confidence interval for your result (in this case, your result is the difference between the findings from two samples). Conclusions based on sampling, rather than inspecting every facility, will always involve some amount of uncertainty.

You can choose either 90% or 95%, which are two standard levels of confidence. A 95% confidence level means that you can be 95% confident that the confidence interval includes the true difference between the two populations. Most, if not all, ERPs to date have used a 95% confidence level.

E.g., if you find that compliance improved 10 percentage points 5 percentage points between the first and second rounds of inspections (with a confidence level of 95%), you can state that you are 95% confident that the actual improvement in compliance is between 5 and 15 percentage points. The higher the level of confidence, the larger the number of inspections needed for a given confidence interval.Go Back to Instructions / Main MenuTo Definitions of Key Terms

DropDownMenusMenu for sample size, 1 sampleFor Confidence Interval Information90%95%2