Social Security: A-02-01-11012

8/14/2019 Social Security: A-02-01-11012

http://slidepdf.com/reader/full/social-security-a-02-01-11012 1/36

SOCIAL SECURITY

MEMORANDUM

Date: April 15, 2002 Refer To:

To: The Commissioner

From: Inspector General

Subject: Performance Measure Review: Reliability of the Data Used to Measure Employer Satisfaction(A-02-01-11012)

Following consultations with congressional committees, the Office of the Inspector Generalagreed to review the Social Security Administration’s (SSA) performance indicators over acontinuous 3-year cycle. We recently completed our first 3-year cycle. In conducting thiswork, we used the services of an outside contractor, PricewaterhouseCoopers, LLP (PwC),to assist us in our efforts.

For this report, we used PwC to conduct the review of two of the Agency’s performanceindicators related to the satisfaction employers have with the services SSA provides tothem. The objective of the review was to assess the reliability of the data used tomeasure the employer satisfaction with SSA services.

Please comment within 60 days from the date of this memorandum on corrective actiontaken or planned on each recommendation. If you wish to discuss the final report,please call me or have your staff contact Steven L. Schaeffer, Assistant Inspector General for Audit, at (410) 965-9700.

James G. Huse, Jr.

Attachment



OFFICE OF

THE INSPECTOR GENERAL

SOCIAL SECURITY ADMINISTRATION

PERFORMANCE MEASURE REVIEW:

RELIABILITY OF THE DATA USED TO

MEASURE EMPLOYER SATISFACTION

April 2002 A-02-01-11012

EVALUATION REPORT

.



Evaluation of Selected PerformanceMeasures of the Social Security

Administration:

Reliability of the Data Used to Measure Employer

Satisfaction

Office of the Inspector GeneralSocial Security Administration



2

BACKGROUND

These indicators have been created to measure the overall satisfaction rate of employers who performed wage reporting to SSA. The goal during FY 2000 is to have93 percent of employers surveyed rating SSA’s overall service as “excellent,” “verygood,” or good,” with 13 percent rating SSA’s overall service as “excellent.” Below is an

overview of the Employer Satisfaction survey and data collection mechanism.

Survey Frequency

This study was first conducted in 1996 by the Office of Quality Assurance andPerformance Assessment (OQA) to assess the satisfaction of employers with SSA-provided services and information. The results were used to develop employer-relatedcustomer service standards. Specifically, this study aimed to seek opinions on thefollowing issues:

q The completeness and clarity of SSA-provided written information;

q The methods used to report wages; andq The usefulness of the SSA/Internal Revenue Service (IRS) Reporter newsletter.

In 2000, this study was repeated for a second time using a questionnaire that wassubstantially modified. The 2000 administration of this survey collected information onemployers’ use, awareness, and satisfaction with the SSA wage reporting services. Italso sought opinions of employers on SSA’s plans for improving wage-reporting servicesvia the Internet.

Sample Design

Both administrations of this survey have used the SSA Employer Identification Master

File (EIMF) (1993 for 1996 study and 1998 for 2000 study) for the sampling frame. Eachtime a stratified sample of employers was selected, with three strata defined based onthe number of employees for whom the wage reports were filed. All employers with noknown number of employees have been grouped together to form a fourth samplingstratum. The following table provides a summary of the sampling design for the 2000survey.

Table 1. Sampling Design and Universe Counts for the 2000 Administration

Strata Number of Employees Population Sample

Strata 1 1 – 10 4,200,000 1,000Strata 2 11 - 250 1,700,000 1,000Strata 3 > 250 100,000 1,000Strata 4 Unknown 500,000 500

Total 6,500,000 3,500



3

Questionnaire Design

The 1996 questionnaire was designed to include 23 questions, 10 of which were open-ended. That questionnaire consisted of the following five sections.

q Services received from the SSA

q Level of service desired

q Annual wage reporting

q The SSA/IRS Reporter

q Comments and remarks

In order to improve the 1996 questionnaire, 16 focus groups were conducted inCalifornia, Florida, Illinois, and Maryland in May and June of 1999 by the Office of Communications (OComm). Prior to these focus groups, which included six major segments of employers and their representatives, OComm conducted a number of in-

depth interviews with representatives of employer and business groups. We applaudthese questionnaire improvement initiatives, as they have resulted in a more concisesurvey instrument while reducing respondent burden.

The resulting 2000 questionnaire, which was shorter than the one used for the 1996survey, consisted of the following three sections:

q Use and awareness of SSA wage-reporting services

q Opinions on SSA’s plans to improve wage reporting services via the Internet

q Quality of wage-reporting services received from SSA

One of the main highlights of this survey was that a notable number of respondentsexpressed unfamiliarity with the SSA services, as their wage reports were prepared by a

payroll service or by someone outside of the company.

Administration/Data Collection

The first survey was administered beginning in May 1996, while the second survey wasconducted between November 1999 and March 2000. Each administration followed aprocedure similar to what is outlined below:

q All sampled individuals were mailed an advance letter announcing the survey. Thisletter provided a brief description of the study and informed respondents that thequestionnaire would be arriving shortly.

q Next, the survey packet was mailed. This packet consisted of a copy of thequestionnaire and an explanatory letter regarding the purpose of the study. Thisletter assured confidentiality and provided an 800 number for those respondentsrequiring assistance.



4

q Soon after the initial mail-out, reminder postcards were sent to all sampledindividuals.

q Lastly, a second cover letter and questionnaire were sent to non-responders.

The following table provides a summary of the resulting response rates by strata for the2000 survey. As seen, the overall response rate is only 40 percent, which issubstantially lower than the 52 percent rate achieved during the 1996 survey.

Table 2. Sampling Design and Response Rate for the 2000 Survey

Strata (Employees) Sample Excluded RespondersResponse

Rate

1 – 10 1,000 68 421 45%11 - 250 1,000 50 422 44%

> 250 1,000 30 374 39%Unknown 500 51 108 24%

Total 3,500 1991 1,325 40%

Several potential reasons have been offered for the lower response rates. First, mostemployers do not seem to view themselves as customers or stakeholders, and insteadconsider wage reporting as a burden. That is, there is not as much empathy on the partof employers to participate in this survey as SSA might have expected. Second, a largepercentage of employers use a third party to process their wage reporting, making itdifficult for them to respond to questions with which they are not familiar. Finally, despitethe fact that sampled employers were asked to forward the questionnaire to individualsor organizations they deemed most knowledgeable about the survey, only a very smallnumber of such third parties responded to the survey. To compensate for this under-representation, OQA requested four large payroll services to complete the survey.Although the information gathered from the four payroll services was not combined withthe results of the employer survey, it provided OQA with additional insight about SSA’s

wage reporting services.

Analysis and Report Generation

Prior to data analysis, survey data were key entered and weighted to compensate for theemployed stratified sampling design. Subsequently, various estimates were generatedusing the weighted data. An example of such estimates is provided in the followingtable.

1Per OQA Employer Satisfaction Draft Report dated 10/12/2000: “The 199 employers were excluded because the

survey correspondence was undeliverable, or the questionnaire was returned with an annotation that the employer was deceased, the company was out of business, etc.”



5

Table 3. Overall Rating of Service Provided by SSA (2000 Survey)

Category 2000 Target

Excellent 3.4% 13%

Very Good 15.6%Good 28.8%Fair 9.3%Poor 0.7%

Very Poor 0.4%No Opinion 41.8%

Total 100%

Excellent, Very Good, or Good 48% 93%

Based on the results of these surveys, SSA is considering lowering the set targets tomore attainable levels. It is argued that since some employers view wage reporting as aburden, they might be giving lower ratings to SSA, hence deflating the overall estimate

of percent satisfied. Other reasons offered for the low ratings include the fact that alarge number of employers are not aware of the many services provided by SSA toemployers.

When reporting the results of the survey SSA calculated the percent of respondentsproviding a rating of Good, Very Good, or Excellent by eliminating all respondents whohave provided a “No Opinion” response. This approach increases the reported results of 48 percent to 82 percent.

RESULTS OF EVALUATION

During the period of September 21, 2000 to February 15, 2001, we evaluated the current

processes and controls, which support the FY 2000 SSA performance measurementprocess. In addition, we determined the accuracy of the underlying performancemeasure data. Our evaluation of the information provided by SSA management as of February 15, 2001 allowed us to determine that the reported FY 2000 results of the twoperformance measures tested (as itemized below) were reasonably stated based on themethodology used by SSA.

Performance Measure Reported Result

3. Percent of employers rating SSA’s overall service as“excellent,” “very good,” or “good.”

82 percent

4. Percent of employers rating SSA’s overall service as“excellent.”

6 percent

However, we did note the following five opportunities for improvement in SSA’smethodology:



6

1. Survey methodology may have led to a large non-response rate;

2. Survey results cannot be extrapolated to the target population;

3. Survey results may be upwardly biased;

4. Survey design needs improvement; and

5. Errors were identified in FY 2000 Survey Draft Report.

These items were noted as a result of our testing of the SSA survey instrument, as wellas the focus group documentation, sampling parameters, and discussions with SSA andits contractor staff.

1. Survey methodology may have led to a high non-response rate.

Overall 40 percent of employers surveyed responded to SSA. Although it is notuncommon for mail surveys to experience response rates that are of this level or lower, itis generally accepted, for example by the Office of Management and Budget (OMB), thatresponse rates of at least 70 percent2 are needed to provide credible results. Thisresponse rate was partially due to the fact that a large number of employers had their wage reports prepared by third parties and therefore experienced difficulties inevaluating SSA effectively on this aspect. In addition, a significant number of employershave minimal or no direct contact with SSA or knowledge of the type of services SSAprovides. This has led to a high non-response rate of 60 percent of employers surveyed.For this research to serve as an employer satisfaction survey, it is critical for thesampling frame to include only “eligible” employers. Eligible employers would be thosewho have experience with the SSA-provided services.

The OMB Resource Manual for Customer Survey states, “The ability to make generalstatements about your agency’s customers based on the survey results is linked to howcomplete and accurate your frame is.”

The data collection methodology should be improved to ensure a better than 40 percentresponse rate. SSA should consider the use of a Computer Assisted TelephoneInterviewing (CATI) method. By using a CATI method it would be possible to screen outthose employers who have had no direct contact with SSA. This will effectively eliminatethe need for forwarding the survey to third parties, which in turn will eliminate thecompounding effects due to responses from inappropriate respondents, and, based onexperience with such surveys, should increase the response rate. The PaperworkReduction Act of 1995, Implementing Guide, Chapter 6, F2 states, “Agencies shall useelectronic collection techniques where such techniques reduce burden on the public,increase efficiency of government programs, reduce cost to the government and thepublic, and/or provide better service to the public.” While mail surveys are sometimes

less expensive to administer, by utilizing the CATI method SSA could increase thequality of the resulting data, and therefore increase the efficiency and effectiveness of the survey.

If a CATI approach is not feasible, a second suggestion is to include a simple“screening” section at the top of the questionnaire that asks whether respondents ever

2

OMB Paperwork Reduction Act of 1995 Appendix C # 1.



7

used or attempted to use SSA services. If the answer is “NO,” the text of thequestionnaire would instruct the response to put the form in the return envelope andreturn it, without further effort. This should tend to increase the response rate (by addingthose who are out of scope), to allow for a count of those using versus not using SSAservices, and to avoid the confusion of a handoff between the employer and the third-party service providers. SSA could then send a separate survey to a sample of third-

party service providers, which is more controlled.

No matter what mode of data collection is employed, it is important to conduct non-response analysis after the survey has been administered. This way, it would bepossible to design and implement a non-response adjustment step in the weightingprocess, which can reduce some of the potential bias due to differential non-response.

2. Survey results cannot be extrapolated to the target population.

In order to minimize non-response, the current survey has instructed employers toforward the questionnaire to the third party they use for wage reporting. Thissnowballing feature has made it impossible to extrapolate the results to the target

population. Also, this approach mixes together the awareness results of the employersand third party providers. Since these two groups are likely to have different levels of awareness, it is best to produce separate awareness estimates for them.

The OMB Resource Manual for Customer Survey states, “The ability to make generalstatements about your agency’s customers based on the survey results is linked to howcomplete and accurate your frame is.”

3. Survey results may be upwardly biased.

The above issues have impacted the analysis and reporting of the survey data as well.For instance, when calculating the percent of employers satisfied with the SSA services,

a large number of respondents who have expressed no opinion are excluded fromcalculations. Because of this exclusion, the estimate of the percent of employerssatisfied with the SSA service changes from 48 percent (of the entire set of respondents)to 82 percent (excluding those who said they had “no opinion”). It can be argued thatsuch an estimate is upwardly biased. Instead, the report should reflect the percents for each of the answer categories, including no opinion, as shown in Table 3. The “noopinion” response, as it is currently implemented, does not clearly distinguish betweenthose in the population who do not have an opinion because they did not use theservices from those who did use the service, but just do not have an opinion.

4. The survey design needs improvement.

Section 1 of the 2000 survey (page A2) includes a battery of awareness questions withsimple Yes/No type answer categories. This is not an efficient use of valuable space onthe questionnaire. Questions regarding awareness with simple answer categories (yesor no) have limited informational value. Instead of allocating a significant part of thesurvey to such questions, it would be advisable to change the structure of thesequestions so that both awareness and degree to which respondents find the givenservice useful can be captured. An example would be:



8

On a scale of 1 to 5, where 1 means very useless and 5 means very useful, please ratethe usefulness of each of the following services provided by the SSA. If you areunaware of the given service, please mark the “Don’t know” box.

Changing the wording of these questions and their corresponding answer categorieswould allow SSA to obtain the awareness measures, while determining how useful

respondents find such services at the same time. In addition, the 2000 questionnaire didnot use question numbers, making it difficult for the respondent to delineate individualquestions, as some questions wrap to the next line. Numbering the questions will makeit easier to identify where questions begin and end.

5. Errors were identified in FY 2000 Survey Draft Report.

During our evaluation of the 2000 draft report, we identified a few minor mistakes. Whilethese errors will not have a significant impact on the reported results, it is important thatsuch errors be corrected in the final report. Examples of the errors are as follows:

q Table 1 does not report percentages for all categories. We believe this is an

oversight and that the excluded percentages should in fact be 14 percent.

q In Table 2, for “Secure reporting of W-2 wages to SSA over the Internet,” the samefrequencies are reported for both strata III and IV. Based on the data file provided bythe SSA, the frequencies for stratum IV should be 32 percent, 40 percent, and28 percent (instead of 24 percent, 26 percent, and 40 percent, respectively).

CONCLUSIONS AND RECOMMENDATIONS

Our evaluation found that the primary goal of this survey has been fulfilled, namely toreport FY 2000 results of the two performance measures. However, our evaluationnoted several methodological, as well as administrative, issues with the 2000 survey.

We recommend that SSA consider the following corrective actions:

1. SSA should enhance the sampling frame to include a larger percentage of respondents and only those which are “eligible” employers (i.e., those employerswho use SSA’s services and have direct contact with SSA).

SSA should consider the use of a more effective data collection methodology, such as aCATI method. It is likely that SSA can obtain as many respondents as they did for themost recent survey (approximately 1,300) at a cost comparable to the mailed outprotocol most recently employed. If SSA continues the mail protocol, telephone follow-ups should be done for employers who, after the second follow-up, do not respond.

2. In addition, SSA should conduct a non-response analysis after the survey has beenadministered and develop non-response adjustment factors for the survey data.

3. If a CATI approach is not feasible, SSA should consider including a simple“screening” section at the top of the questionnaire that asks whether respondentsever used or attempted to use SSA services.



9

4. SSA should change the structure of the yes/no questions so that both awarenessand degree to which respondents find the given service useful can be captured.

5. In addition, SSA should number the questions to make it easier to identify wherequestions begin and end.

6. When calculating the percent of respondents providing a rating of Good, Very Good,or Excellent, the denominator should include all respondents, regardless of their ratings.

This will provide an unbiased estimate of percent satisfied.

7. In future surveys, SSA should ask respondents whether they use SSA services or not, and exclude them from the computation as non-customers.

APPROPRIATENESS OF THE PERFORMANCE MEASURES

As part of this engagement, we evaluated the appropriateness of each of the

performance measures with respect to the Government Performance and Results Act of 1993 (GPRA) compliance and SSA’s APP. To do so, we determined whether thespecific indicators and goals corresponded to the strategic goals identified in SSA’sAPP, determined whether each of these indicators accurately measure performance,and determined their compliance with GPRA requirements.

Performance Measures #3 and #4 align logically with the SSA Strategic Plan butstill need improvement

The relationships between PMs #3 and #4 and the applicable SSA Strategic Goal aredepicted in the following figure:



10

SSA Mission

To promote the economic security of the nation's people

through comp assionate and vigilant leadership in

shaping and managing America's Social Security programs.

Strategic "Goal #2"

To deliver customer-responsive, world-class service

Strategic Objective

By 2001, to have 9 out of 10 customersrate SSA’s service as “good,” “very good”

or “excellent,” with mo st rating it “excellent”

Performance Indicators & Goals

Percent of employers rating SSA’soverall service as “excellent, “very

good,” or “good”

Performance Indicators & Goals

Percent of employers rating SSA’soverall service as “excellent”

The SSA mission is supported by five strategic goals, including Goal #2, “To deliver customer-responsive world-class service.” Goal #2, in turn, is supported by severalstrategic objectives, including the relevant objective dealing with customer perceptions,“By 2002, to have 9 out of 10 customers rate SSA’s service as ‘good,’ ‘very good,’ or

‘excellent,’ with most rating it ‘excellent’.“ Performance Measures #3 and #4characterize employer satisfaction with SSA services. Both PM #3 and PM #4 logicallyalign with SSA’s strategic planning process.

Based on the taxonomy of performance measures included in Appendix F, both PM #3and PM #4 are measures of accomplishment because they report on a result (employer satisfaction) achieved with SSA resources. They are further categorized as outcomemeasures because they indicate the accomplishments or results (employer satisfaction)that occur because of the SSA services provided. As shown in Appendix F, outcomemeasures include public perceptions of outcomes, such as employer satisfaction.

Within the framework of GPRA, Performance Measures #3 and #4 fit the intent of an

outcome measure because they provide “…a description of the intended result, effect, or consequence that will occur from carrying out a program or activity.”3 The intent of thesetwo performance measurements is to gauge employer satisfaction (i.e., the effect) for theactivity of providing services. They can both be useful to management and externalstakeholders, as encouraged by OMB’s Circular A-11 Preparation and Submission of Budget Estimates. Given the duplication of the measures, SSA should eliminate one of

3

OMB Circular A-11 Preparation and Submission of Budget Estimates , Section 200.2.



11

the measures, unless it finds utility in keeping both of them. Nevertheless, thesemeasures could be improved, as discussed below.

In its current form, the metric may not provide reliable results. The survey methodologymight not be appropriate for the underlying objectives. The results cannot beextrapolated to the target population and may be upwardly bias. Furthermore, our

evaluation of the draft report indicated that there were errors in the reported data. Inaddition, the survey design was confusing for employers and the corresponding resultsare skewed. For example, almost 42 percent of respondents stated they had no opinionon the services provided by SSA.

Ideally, a performance metric should help the agency to take action and affect theperformance of the indicator being measured. However, due to the presence of numerous third parties, the measurement becomes clouded and is, therefore, lesscredible and less useful.

The metric readily facilitates comparisons between geographic regions and businesssegments. With some enhancements, the measurement system could also be used to

benchmark against other agencies with employer relationships, such as the IRS, theHealth Care Financing Administration, the Pension Benefit Guaranty Corporation, andthe Office of Personnel Management.

Recommendations

8. SSA should enhance its measurement system to ensure greater accuracy.

9. They should also take steps to make the metric actionable so that it will provide

greater utility for the agency.

10. In addition, SSA should consider using the measurement to benchmark against other agencies that interface with employers.

11. Finally, SSA should eliminate one of the two measures, unless SSA finds a utility inkeeping both measures.

Agency Comments

The Agency agreed with or has a plan of action to address 8 of the 11 recommendationscontained in this report. In fact, it has already taken steps to implement some of therecommendations. For example, it has eliminated yes/no questions when it modified itssurvey instrument. The removal of this type of question was recommended sinceanswers to it provide limited informational value.

SSA disagreed with the recommendation that all respondents should be included in thedenominator, regardless of their rating, when calculating the percent of respondentsproviding a rating of Good, Very Good or Excellent. The Agency stated that basingratings on substantive responses is an acceptable practice, as long as accompanyingdiscussion indicates that the percentages reflect the opinions of those who provided arating.



12

The Agency also disagreed with the recommendation to consider using the employer satisfaction measurement to benchmark against other agencies that interface withemployers. The Agency stated that modifications of the performance measure, as wellas changes made to the survey instrument, addressed the need for benchmarking at thistime. It also stated that its day-to-day interactions with States and the IRS provided anawareness of “best practices” used by those entities. SSA provides due consideration to

incorporating those practices into its own service delivery model.

Finally, we recommended that the Agency consider eliminating one of the two measures.SSA, however, intends to keep both measures at this time. SSA stated that the firstmeasure, Percent of employers rating SSA’s overall service as “excellent,” “very good” or “good” , identifies the universe of satisfied employers/customers. The secondmeasure, Percent of employers rating SSA’s overall service as “excellent” , serves tocapture how well the Agency is achieving best in business service. The full text of SSA’scomments can be found in Appendix C.

OIG Response

We appreciate SSA’s comments on this report. The implementation of therecommendations contained within this report will help to ensure better measurement of the level of service provided to employers who have used SSA’s services. We are gladto see that the Agency has already taken steps to implement some of our recommendations through modifications made to its survey instrument.

We continue to believe SSA should reexamine the methods used to calculate theemployer’s level of satisfaction with its services. With the current methods used, theestimate of the percent of employers satisfied with the SSA service changes from 48percent (of the entire set of respondents) to 82 percent (excluding those who did nothave an opinion of SSA’s services). SSA‘s reporting of the results of the employer survey should reflect the percents for each of the answer categories, including those that

do not capture an opinion of the service provided. SSA should also work to differentiatebetween the respondents who did receive service, but did not express an opinion, andthose who did not receive service from SSA so they could not express an opinion.

In regards to benchmarking, we note the usefulness of SSA’s interaction with other Stateagencies and the IRS and suggest that SSA continue to look for opportunities tobenchmark its operations against other agencies that interface with employers.

Finally, while we respect the Agency’s prerogative to create its performance measures,we believe that the use of both measures is repetitive. SSA can provide decisionmakerswith a clear understanding of the satisfaction of employers who have had contact withSSA through the use of one performance measure.



APPENDICES

APPENDIX A – Scope and Methodology

APPENDIX B – Acronyms

APPENDIX C – Agency Comments

APPENDIX D – Performance Measure Summary Sheets

APPENDIX E – Performance Measure Process Maps

APPENDIX F – Performance Measure Taxonomy



Appendix A

A-1

SCOPE AND METHODOLOGY

The Social Security Administration’s (SSA) Office of the Inspector General contractedPricewaterhouseCoopers LLP (PwC) to evaluate 11 SSA performance indicators

identified in its Fiscal Year (FY) 2001 Annual Performance Plan (APP). We performedour testing from September 21, 2000 through February 15, 2001. Since FY 2001performance results were not yet available as of the date of our evaluation, weperformed tests of the performance data and related internal controls surrounding themaintenance and reporting of the results for FY 2000. Specifically, we performed thefollowing:

1. Obtained an understanding and documented the FY 2000 Employer SurveyMethodology;

2. Tested the reasonableness of the survey data;

3. Determined whether performance measures were meaningful and in compliance withthe Government Performance and Results Act of 1993 (GPRA); and

4. Identified findings relative to the above procedures and provided recommendationsfor improvement.

Our engagement was limited to testing at SSA’s headquarters in Woodlawn, Maryland.The procedures that we performed were in accordance with the American Institute of Certified Public Accountants’ Statement on Standards for Consulting Services, and areconsistent with appropriate standards for performance audit engagements inGovernment Auditing Standards (Yellow Book, 1994 version). However, we were notengaged to and did not conduct an audit, the objective of which would be the expression

of an opinion on the reliability or accuracy of the reported results of the performancemeasures evaluated. Accordingly, we do not express such an opinion.

1. Obtained an understanding and documented the FY 2000 Employer SurveyMethodology.

We obtained an understanding of the underlying process and operating proceduressurrounding the generation of the performance measures through interviews andmeetings with the appropriate SSA personnel. Our work involved a comprehensiveevaluation of the survey methodology used for this performance measure. The specificsteps included evaluating the sampling and questionnaire designs, data collectionprocedures, and data analysis and reporting. We evaluated the following documents:

q 1996 Final Report, issued in April 1998

q 2000 Draft Report, issued in October 2000

q 2000 Focus Group reports, undated.



Appendix A

A-2

2. Tested the reasonableness of the survey data.

To ensure the reasonableness of the number reported in the FY 2000 GPRA section of the SSA Performance and Accountability Report, we evaluated the Employer Satisfaction Survey data provided by SSA’s Office of Quality Assurance and

Performance Assessment. We verified the accuracy of the data by calculating thefrequencies and counts using SAS software. This included calculating the percent of employers rating SSA wage reporting service as “excellent,” “very good,” or “good.”

3. Determined whether performance measures were meaningful and incompliance with GPRA

As part of this engagement, we evaluated the appropriateness of each of theperformance measures with respect to GPRA compliance and SSA’s APP. To do so, wedetermined whether the specific indicators and goals corresponded to the strategic goalsidentified in SSA’s APP, determined whether each of these indicators accuratelymeasure performance, and determined their compliance with GPRA requirements.



Appendix B

ACRONYMS

APP Annual Performance Plane

CATI Computer Assisted Telephone Interviewing

EIMF Employee Identification Master File

FY Fiscal Year

GPRA Government Performance and Results Act

IRS Internal Revenue Service

OAS Office of Automation Support

OComm Office of Communications

OMB Office of Management and Budget

OQA Office of Quality Assurance and Performance Assessment

PM Performance Measure

PwC PricewaterhouseCoopers LLP

SSA Social Security Administration



Appendix C

AGENCY COMMENTS



"'.,

~\, SECu:JC.;~$/j/ts~~

~I,: IIIIIII~~~J"..,/sT?J'

SOCI-AL SEC-URITY

Rct~rTo: SlJ-3December 18, 2001

James G. Ruse, Jr.

Inspector General

+arry Dye "" ~ -

Chief of Sta~

From'

Office of the Inspector General (OIG) Draft Report, "Evaluation of Selected Performance

Measures of the Social Security Administration: Reliability of the Data Used to MeasureEmployer Satisfaction" (A-O2-01-11012)-INFORMATION

Subjcct'

We appreciate OIG's efforts in conducting this review. Our comments on the report content and

recommendations are attached.

Please let us know if we can be of further assistance. Staff questions can 1?e eferred to

Dan Sweeney on extension 51957.

Attachment:

SSA Response



Appendix C

COMMENTS ON THE OFFICE OF THE INSPECTOR GENERAL (OIG) DRAFT REPORT,“PERFORMANCE MEASURE REVIEW: RELIABILITY OF THE DATA USED TO

MEASURE EMPLOYER SATISFACTION” (A-02-01-11012)

Recommendation 1

Enhance the sampling frame to include a larger percentage of respondents and only those which

are “eligible” employers, i.e., those employers who use the Social Security Administration’s(SSA) services and have direct contact with SSA.

Comment

We agree. We have discontinued the generalized sampling frame used in the survey OIG

reviewed. We have already implemented a survey in FY 2002 targeted to employers who have

received direct service from SSA; i.e., employers who called the Employer Reporting Services

Center (ERSC) for assistance. This sampling frame includes specific contact information thatwill greatly enhance our ability to achieve an acceptable response rate.

Recommendation 2

Conduct a non-response analysis after the survey has been administered and develop non-response adjustment factors for the survey data.

Comment

We will conduct the suggested analysis if the response rate for the survey falls below acceptable

levels. We anticipate that our revised methodology, which includes use of Computer AssistedTelephone Interviewing (CATI) on a more targeted sample, will result in a much higher responserate.

Recommendation 3

If a Computer Assisted Telephone Interview approach is not feasible, consider including a

simple “screening” section at the top of the questionnaire that asks whether respondents ever used or attempted to use SSA services.

Comment

We have adopted a CATI for the current survey, which also utilizes a completely redesigned

questionnaire developed to assess satisfaction with service provided by the ERSC.



Appendix C

C-1

Recommendation 4

Change the structure of the yes/no questions so that both awareness and degree to which

respondents find the given service useful can be captured.

Comment

Our current survey does not include yes/no questions.

Recommendation 5

Number the questions to make it easier to identify where questions begin and end.

Comment

We agree. The CATI software being used for the current survey eliminates this problem.

Recommendation 6

When calculating the percent of respondents providing a rating of Good, Very Good, or Excellent, the denominator should include all respondents, regardless of their ratings.

Comment

We disagree. Basing ratings on substantive responses is acceptable practice, as long as

accompanying discussion indicates that the percentages reflect the opinions of those who

provided a rating, as was done in the report prepared by the Office of Quality Assurance andPerformance Assessment. It should also be noted that the rating questions did not include a

choice of "no opinion" as indicated in OIG's report; the choice was "not applicable/service not

used," which served as a screening device that fit into the format of the questionnaire.

Recommendation 7

In future surveys, ask respondents whether they use SSA services or not, and exclude them fromthe computation as non-customers.

Comment

Our FY 2002 survey specifically targets employers who have received direct service from SSA

(i.e., employers who call the ERSC for assistance).



Appendix C

C-2

Recommendation 8

Enhance the measurement system to ensure greater accuracy.

Comment

We agree. The revised methodology for the employer survey should produce this result.

Recommendation 9

Take steps to make the metric actionable so that it will provide greater utility for the Agency.

Comment

We agree. We have modified the employer performance indicator to "Percent of employersrating SSA's overall service during interactions with SSA as excellent, very good or good." This

performance measure will be based solely on results of a survey of employers who have calledthe ERSC. Therefore, the performance measure will indicate satisfaction specifically withservice from the ERSC. The survey results will identify specific areas where improvements can

be made in the Center's service in order to increase satisfaction as captured by the performance

measure. In the future, we intend to include additional types of employer interactions in thesurvey. At that time, we will determine if and how the employer performance measures should

be modified.

Recommendation 10

Consider using the measurement to benchmark against other agencies that interface with

employers.

Comment

We believe the modification to the indicator described above and changes to the survey instrument

address the need for benchmarking at this time. In addition, our day-to-day interactions with the

States and the Internal Revenue Service provide us with an awareness of “best practices” used by

those entities. Due consideration is given to incorporating those practices into our own servicedelivery.



Appendix C

C-3

Recommendation 11

Eliminate one of the two measures, unless SSA finds a utility in keeping both measures.

Comment

At this time, we intend to keep both measures. We developed the two performance measures to

capture how well the Agency was achieving both parts of the customer satisfaction strategicobjective--"...9 out of 10 customers rate SSA's service as good, very good or excellent" and "with

most rating it excellent." It is consistent with the two measures assessing the satisfaction of our

core business customers. In both instances, the first measure identifies the universe of satisfied

employers/customers. The second measure serves to capture how well the Agency is achieving best in business service.



Appendix D

Performance Measure Summary Sheets



A p p e n d i x D

D - 1

P E R F O R M A N C E M E A S U R E S U M M A R Y

S H E E T S

N a m e o

f M e a s u r e

M

e a s u r e

T y p e

S t r a t e g

i c G o a

l / O b j e c

t i v e

3 ) P e r c e n t o f e m p l o y e r s

r a t i n g S S A ’ s o v e r a l l s e r v i c e

a s “ e x c e l l e n t , ” “ v e r y g o o

d , ” o r “ g o o d . ”

P

e r c e n t a g e

G o a

l : T

o d e l i v e r c u s t o m e r - r e s p o n s i v e w o r

l d - c l a s s s e r v i c e .

O b j e c t i

v e : B y 2 0 0 2 , t o h a v e 9 o u t o f 1 0 c u s t o m e r s r a t e

S S A ’ s s

e r v i c e a s “ g o o d ” , “ v e r y g o o d ” , o r “ e

x c e l l e n t , ” w i t h

m o s t r a t i n g i t “ e x c e l l e n t ” .

P u r p o s e

S u r v e y

F r e q u e n c y

T o a s s e s s t h e p e r c e n t o

f e m p l o y e r s r a t i n g S S A ’ s o v e r a l l s e r v i c e a s e x c e l l e n t , v e r y g o o d , o r g o o d , S S A ’ s O f f i c e o f

Q u a l i t y A s s u r a n c e a n d P e r f o r m a n c e A s s e s s m e n t ( O Q A ) s u r v e y e d a s a m p l e o f e m p l o y e r s o b t a i n e d f r o m S S A ’ s

E m p l o y e r I d e n t i f i c a t i o n M a s t e r F i l e ( E I M F ) f o r C Y 1 9 9 8 . T h e E I M F i s u p d a t e d b a s e d o n d a t a

r e c e i v e d f r o m t h e I R S

M a s t e r B u s i n e s s F i l e .

Y e a r l y

T a r g e

t G o a

l

H o w

C o m p u t e

d

D a t a

S o u r c e

9 3 %

N u m b e r o f e m p l o y e r s s u r v e y e d b y S S A ’ s O Q A w h

o r a t e o v e r a l l s e r v i c e a s

“ g o o d , ” “ v e r y g o

o d , ” o r “ e x c e l l e n t ” d i v i d e d b y t h e t o

t a l n u m b e r o f r e s p o n d e n t s

t o t h a t q u e s t i o n .

E m p l o y e r S a t i s f a c t i o n

S u r v e

y



A p p e n d i x D

D - 2

D e s

i g n a

t e d S t a f f M e m

b e r s

D i v i s i o n

J e a n V e n a b l e

M a r k R u l e y

L o i s S m i t h

N o r m a n G o l d s t e i n

M a r v i n B r o d y

A l l a n K a u f m a n

P e g B l a t t e r

S a i l R e i n e r

B a r t o n a H a r r i s o n

O S M

O S F E

O S F E

O Q A

O f f i c e o f A u t o m a

t i o n S u p p o r t ( O A S )

T e s

t i n g a n

d R e s u

l t s

W e p e r f o r m e d a n e v a l u a t i o n o f t h e s u r v e y m e t h o d o l o g y u s

e d f o r t h i s p e r f o r m a n c e m e a s u r e .

T h e s p e c i f i c s t e p s i n c l u d e d e v a l u a t i n g t h e

s a m p l i n g a n d q u e s t i o n n a i r e d e s i g n s , d a t a c o l l e c t i o n p r o c e d u r e s , a n d d a t a a n a l y s i s a n d r e p o r t i n g . I n t h i s p r o c e s s , w e e v a l u a t e d

t h e f o l l o w i n g

d o c u m e n t s a n d d a t a .

·

1 9 9 6 F i n a l R e p o r t , i s s u e d i n A p r i l 1 9 9 8

·

2 0 0 0 D r a f t R e p o r t , i s s u e d i n O c t o b e r 2 0 0 0

·

2 0 0 0 s u r v e y d a t a

·

2 0 0 0 F o c u s G r o u p r e p o r t s , u n d a t e d

R e f e r t o “ R e s u l t s o f E v a l u a t i o n ” f o r a d e s c r i p t i o n

o f t h e f i n d i n g s .



A p p e n d i x D

D - 3

N a m e o

f M e a s u r e

M

e a s u r e

T y p e

S t r a t e g

i c G o a

l / O b j e c

t i v e

4 ) P e r c e n t o f e m p l o y e r s r a t i n g S S A ’ s o v e r a l l s e r v i c e a s

“ e x c e l l e n t . ”

P

e r c e n t a g e

G o a

l : T

o d e l i v e r c u s t o m e r - r e s p o n s i v e w o r

l d - c l a s s s e r v i c e .

O b j e c t i

v e : B y 2 0 0 2 , t o h a v e 9 o u t o f 1 0 c u s t o m e r s r a t e

S S A ’ s s

e r v i c e a s “ g o o d ” , “ v e r y g o o d ” , o r “ e

x c e l l e n t , ” w i t h

m o s t r a t i n g i t “ e x c e l l e n t ” .

P u r p o s e

S u r v e

y F r e q u e n c y

T o a s s e s s t h e p e r c e n t o f

e m p l o y e r s r a t i n g S S A ’ s o v e r a l l s e r v i c e a s e x c e l l e n t . S S A ’ s O f f i c e o f

Q u a l i t y A s s u r a n c e a n d

P e r f o r m a n c e A s s e s s m e n

t ( O Q A ) s u r v e y e d a s a m p l e o f e m p

l o y e r s o b t a i n e d f r o m S S A ’ s E m p l o

y e r I d e n t i f i c a t i o n M a s t e r

F i l e ( E I M F ) f o r C Y 1 9 9 8 .

T h e E I M F i s u p d a t e d b a s e d o n d a

t a r e c e i v e d f r o m t h e I R S M a s t e r B u s i n e s s F i l e .

Y e a r l y

T a r g e

t G o a

l

H o w

C o m p u

t e d

D a

t a S o u r c e

1 3 %

N u m b e r o f e m p l o y e r s s u r v e y e d b y S S A ’ s O Q A w h o

r a t e o v e r a l l s e r v i c e a s “ e x c e l l e n t ” d i v i d e d b y t h e t o t a l

n u m b e r o f r e s p o n d e n t s t o t h a t q u e s t i o n .

E m p l o y e r S a t i s f a c t i o n S u r v e y

D e s

i g n a

t e d S t a f f M e m

b e r s

D i v i s i o n

J e a n V e n a b l e

M a r k R u l e y

L o i s S m i t h

N o r m a n G o l d s t e i n

M a r v i n B r o d y

A l l a n K a u f m a n

P e g B l a t t e r

S a i l R e i n e r

B a r t o n a H a r r i s o n

O S M

O S F E

O Q A

O A S



A p p e n d i x D

D - 4

T e s

t i n g a n

d R e s u

l t s

W e p e r f o r m e d a n e v a l u a t i o n o f t h e s u r v e y m e t h o d o l o g y u s e

d f o r t h i s p e r f o r m a n c e m e a s u r e . T

h e s p e c i f i c s t e p s i n c l u d e d e v a l u a t i n g t h e s a m p l i n g

a n d q u e s t i o n n a i r e d e s i g n

s , d a t a c o l l e c t i o n p r o c e d u r e s , a n d

d a t a a n a l y s i s a n d r e p o r t i n g . I n t h i s

p r o c e s s , w e e v a l u a t e d t h e f o l l o w i n g d o c u m e n t s

a n d d a t a .

·

1 9 9 6 F i n a l R e p o r t , i s s u e d i n A p r i l 1 9 9 8

·

2 0 0 0 D r a f t R e p o r t , i s s u e d i n O c t o b e r 2 0 0 0

·

2 0 0 0 s u r v e y d a t a

·

2 0 0 0 F o c u s G r o u p r e p o r t s , u n d a t e d

R e f e r t o “ R e s u l t s o f E v a l u a t i o n ” f o r a d e s c r i p t i o n o f t h e f i n d i n g s .



Appendix E

PERFORMANCE MEASURE PROCESS MAPS



A p p e n d i x E

E - 1

E x e c u t i v e O r d e r 1 2 8 6 2 r e q u i r e s S S A

t o s u r v e y i t s c u s t o m e r s f o r t h e k i n d

a n d q u a l i t y o f s e r v i c e s t h e y w a n t a n d

t h e i r l e v e

l o f s a t i s f a c t i o n w i t h e x i s t i n g

s e r v i c e s .

A d m i n i s t e r e d S

u r v e y

( B e t w e e n N o v

1 9 9 9

a n d M a r c h 2 0 0 0 )

C o l l e c t e d a n d P r e p a r e d

D a t a f o r A n a l y s i s

T o m a x i m i z e t h e r e s p

o n s e r a t e ,

O Q A f o l l o w e d t h e T o t

a l D e s i g n

M e t h o d d e v e l o p e d b y

a

c o n t r a c t o r .

S a m p l e d e m p l o y e r s w

e r e n o t i f i e d

i n a d v a n c e b y m a i l o f

t h e i r

s e l e c t i o n f o r t h e s u r v e

y

A q u e s t i o n n a i r e a n d e

x p l a n a t o r y

l e t t e r w e r e m a i l e d 1 0

d a y s a f t e r

t h e m a i l i n g o f t h e a d v a n c e l e t t e r .

A p p r o x i m a t e l y 3 w e e k

s l a t e r ,

r e m i n d e r p o s t c a r d s w

e r e m a i l e d

t o n o n - r e s p o n d e n t s

O Q

A d e v e l o p e d a

s u r

v e y q u e s t i o n n a i r e

i n c

o r d i n a t i o n w i t h

t h e

S e n i o r F i n a n c i a l

E x e c u t i v e a n d t h e

M M

P w o r k g r o u p

P e r f o r m a n c e M e a s u r e # 3 & 4

E m p l o y e r S a t i s f a c t i o

n S u r v e y

S u r v e y r e s p o n s e s

w e r e

k e y - e n t e r e d i n t o a

n

A c c e s s d a t a b a s e

o n t h e

L A N b y O Q A

O Q A r e c e i v e d t h e

d a t a b a s e a s a n A S C I I

f i l e , a n d p l a c e d i t o n t o a

m a i n f r a m e I n

a d d i t

i o n t o t h e E x e c u t i v e O r d e r , S S A ' s

O f f i c e o f t h e S e n i o r F i n a n c i a l E x e c u t i v e

( O S F E ) r e q u e s t e d t h a t O Q A o b t a i n s p e c i f i c

i n f o r m a

t i o n f r o m e m p l o y e r s i n i t s s u r v e y .

G o a l # 2 i s t o d e l i v e r c u s t o m e r

-

r e s p o n s i v e w o r l d - c l a s s s e r v i c e

.

S t r a t i f i e d r a n d o

m s a m p l e

o f 3 , 5 0 0 e m p l o y e r s f r o m

a p p r o x i m a t e l y 6 . 5 m i l l i o n

e m p l o y e r s w h o

r e p o r t e d

w a g e s d u r i n g c

a l e n d a r

y e a r 1 9 9 8

C o l l e c t e d r e s p o n s e s

t o q u e s t i o n s r e l a t e d

t o S S A ' s p l a n s t o

i m p r o v e e m p l o y e r s '

w a g e - r e p o r t i n g

s e r v i c e s .

A f t e r a n a d d i t i o n a l 3 w

e e k s , a

s e c o n d c o v e r l e t t e r a n

d

q u e s t i o n n a i r e w e r e m

a i l e d t o a l l

n o n r e s p o n d e n t s

O Q A e d i t e d d a t a ,

a p p l i e d

w e i g h t s a n d p r o d u

c e d

t a b l e s u s i n g t h e

m a i n f r a m e d a t a a n d S A S

D

e s i g n e d S u r v e y

D e v e l o p e d S a m p l e

s e l e c t i o n

a n d

m e t h o d o

l o g y

M e a s u r e d e m p l o y e r s '

u s e , a w a r e n e s s , a n d

s a t i s f a c t i o n w i t h

S S A ' s w a g e r e p o r t i n g

s e r v i c e s

O f t h e 3 , 5 0 0 s e

l e c t e d

e m p l o y e r s , 1 9 9

w e r e

e x c l u d e d d u e t o d e a t h , o u t

o f b u s i n e s s , o r

u n a b l e t o

l o c a t e , r e s u l t i n g

i n a

s a m p l e o f 3 , 3 0 1

A n a l y z e d D a t a a n d

R e p o r t R e s u l t s

O Q A p r o v i d e d t h e

s u m m a r i z e d r e s u l t s o f

t h e s u r v e y i n a r e p o r t t o

S S A m a n a g e m e n t

E s t a b l i s h e d

S u r v e y o b j e c t i v e s

1 , 3 2 5 o f t h e 3 , 3 0 1 e m

p l o y e r s

r e s p o n d e d t o t h e s u r v

e y

q u e s t i o n n a i r e , r e s u l t i n

g i n a 4 0 %

r e s p o n s e r a t e

O C

o m m c o n d u c t e d

1 6

F o c u s G r o u p s

T h e O f f i c e o f

C o

u m m u n i c a t i o n s

( O C o m m ) c o n d u c t e d

i n - d e p t h i n t e r v i e w s

D e t a i l s :



DISTRIBUTION SCHEDULE

No. of Copies

Commissioner of Social Security 1

Management Analysis and Audit Program Support Staff, OFAM 10

Inspector General 1

Assistant Inspector General for Investigations 1

Assistant Inspector General for Executive Operations 3

Assistant Inspector General for Audit 1

Deputy Assistant Inspector General for Audit 1

Director, Data Analysis and Technology Audit Division 1

Director, Financial Audit Division 1

Director, Western Audit Division 1

Director, Southern Audit Division 1

Director, Northern Audit Division 1

Director, General Management Audit Division 1

Issue Area Team Leaders 25

Income Maintenance Branch, Office of Management and Budget 1

Chairman, Committee on Ways and Means 1

Ranking Minority Member, Committee on Ways and Means 1

Chief of Staff, Committee on Ways and Means 1

Chairman, Subcommittee on Social Security 2

Ranking Minority Member, Subcommittee on Social Security 1

Majority Staff Director, Subcommittee on Social Security 2

Minority Staff Director, Subcommittee on Social Security 2

Chairman, Subcommittee on Human Resources 1

Ranking Minority Member, Subcommittee on Human Resources 1

Chairman, Committee on Budget, House of Representatives 1

Ranking Minority Member, Committee on Budget, House of Representatives 1

Chairman, Committee on Government Reform and Oversight 1

Ranking Minority Member, Committee on Government Reform and Oversight 1

Chairman, Committee on Governmental Affairs 1

Ranking Minority Member, Committee on Governmental Affairs 1



Page 2

Chairman, Committee on Appropriations, House of Representatives 1

Ranking Minority Member, Committee on Appropriations,House of Representatives 1

Chairman, Subcommittee on Labor, Health and Human Services, Educationand Related Agencies, Committee on Appropriations,

House of Representatives 1

Ranking Minority Member, Subcommittee on Labor, Health and HumanServices, Education and Related Agencies, Committee on Appropriations,House of Representatives 1

Chairman, Committee on Appropriations, U.S. Senate 1

Ranking Minority Member, Committee on Appropriations, U.S. Senate 1

Chairman, Subcommittee on Labor, Health and Human Services, Educationand Related Agencies, Committee on Appropriations, U.S. Senate 1

Ranking Minority Member, Subcommittee on Labor, Health and Human

Services, Education and Related Agencies, Committee on Appropriations,U.S. Senate 1

Chairman, Committee on Finance 1

Ranking Minority Member, Committee on Finance 1

Chairman, Subcommittee on Social Security and Family Policy 1

Ranking Minority Member, Subcommittee on Social Security and Family Policy 1

Chairman, Senate Special Committee on Aging 1

Ranking Minority Member, Senate Special Committee on Aging 1

Vice Chairman, Subcommittee on Government Management Informationand Technology 1

President, National Council of Social Security Management Associations,Incorporated 1

Treasurer, National Council of Social Security Management Associations,Incorporated 1

Social Security Advisory Board 1

AFGE General Committee 9

President, Federal Managers Association 1

Regional Public Affairs Officer 1

Total 97



Overview of the Office of the Inspector General

Office of Audit

The Office of Audit (OA) conducts comprehensive financial and performance audits of the

Social Security Administration's (SSA) programs and makes recommendations to ensure hatprogram objectives are achieved effectively and efficiently. Financial audits, required by the

Chief Financial Officers Act of 1990, assesswhether SSA' s financial statements airly present

the Agency's financial position, results of operations, and cash flow. Performance audits reviewthe economy, efficiency, and effectiveness of SSA s programs. OA also conducts short-term

management and program evaluations focused on issues of concern to SSA, Congress, and the

general public. Evaluations often focus on identifying and recommending ways to prevent and

minimize program fraud and inefficiency.

Office of Executive Operations

The Office of Executive Operations (OEO) supports the Office of the Inspector General (OIG) byproviding information resource management; systems security; and the coordination of budget,

procurement, telecommunications, facilities and equipment, and human resources. In addition,

this office is the focal point for the OIG's strategic planning function and the development and

implementation of performance measures equired by the Government Performance and Results

Act. OEO is also responsible for performing internal reviews to ensure hat OIG offices

nationwide hold themselves to the same rigorous standards hat we expect from the Agency, as

well as conducting employee investigations within OIG. Finally, OEO administers OIG's public

affairs, media, and interagency activities and also communicates OIG's planned and current

activities and their results to the Commissioner and Congress.

Office of InvestigationsThe Office of Investigations (01) conducts and coordinates investigative activity related to fraud.

waste, abuse, and mismanagement of SSA programs and operations. This includes wrongdoing

by applicants, beneficiaries, contractors, physicians, interpreters, representative payees, hird

parties, and by SSA employees n the performance of their duties. Or also conducts oint

investigations with other Federal, State, and local law enforcement agencies.

Counsel to the Inspector General

The Counsel to the Inspector General provides legal advice and counsel to the Inspector General

on various matters, including: l) statutes, egulations, legislation, and policy directives

governing the administration of SSA s programs; 2) investigative procedures and techniques; and3) legal implications and conclusions to be drawn from audit and investigative material produced

by the DIG. The Counsel's office also administers the civil monetary penalty program.

Social Security: A-02-01-11012

Documents

Transcript of Social Security: A-02-01-11012