Trying Out New Survey Techniques – Lessons Learned

Trying Out New Survey Techniques – Lessons Learned

2013 Northwest Hazardous Materials Management Conference

Liz Tennant, Strategic Advisor, Office of the Program Director

June 25, 2013

2013 Northwest Hazardous Materials Management Conference

Liz Tennant, Strategic Advisor, Office of the Program Director

June 25, 2013

The Local Hazardous Waste Management Program in King

County

The Local Hazardous Waste Management Program in King

County

A regional partnership to protect and enhance public health and environmental quality by reducing the threat posed by the production, use, storage and disposal of hazardous materials.

Program PartnersProgram Partners

• 37 Cities and 2 tribes in King County

• King County Water & Land Resources

• Public Health--Seattle & King County

• Seattle Public Utilities

• King County Solid Waste

• 37 Cities and 2 tribes in King County

• King County Water & Land Resources

• Public Health--Seattle & King County

• Seattle Public Utilities

• King County Solid Waste

King County, WashingtonKing County, Washington

1.92 million residents

799,000 households

62,000 businesses

2,134 square miles

14th most populous

county in the U.S.

1.92 million residents

799,000 households

62,000 businesses

2,134 square miles

14th most populous

county in the U.S.

2012 Survey Goals2012 Survey Goals

Updated snapshot of residents’ attitudes, awareness and behavior about household hazardous materials and waste.

Update on who is using our collection services, and how satisfied they are with the services.

Better understanding of why people are not using the facilities and how to better reach and motivate them to dispose of hazardous wastes properly.

Updated snapshot of residents’ attitudes, awareness and behavior about household hazardous materials and waste.

Update on who is using our collection services, and how satisfied they are with the services.

Better understanding of why people are not using the facilities and how to better reach and motivate them to dispose of hazardous wastes properly.

ConsiderationsConsiderations

Last surveys conducted 5 years ago.

Need for statistically valid samples.

Need to align with increasingly diverse demographics.

Particular concern with input from South King County.

Last surveys conducted 5 years ago.

Need for statistically valid samples.

Need to align with increasingly diverse demographics.

Particular concern with input from South King County.

ApproachApproach

Three complementary surveys:

1. General survey of residents throughout King County

2. Focus groups of special populations in South King County

3. Survey of collection customers

Three complementary surveys:

1. General survey of residents throughout King County

2. Focus groups of special populations in South King County

3. Survey of collection customers

Decision to Try New MethodsDecision to Try New Methods

General residential survey using address based sampling instead of random-dialed phone calls

Oversampled the non-Caucasian population in South King County

Supplemented with focus groups

General residential survey using address based sampling instead of random-dialed phone calls

Oversampled the non-Caucasian population in South King County

Supplemented with focus groups

What is Address Based Sampling?What is Address Based Sampling?

Obtain a random sample of addresses in the area you are surveying.

Mail a letter or postcard to the selected addresses.

Provide information about how to respond.

Analyze the responses you receive.

Potentially can adjust the sample.

Obtain a random sample of addresses in the area you are surveying.

Mail a letter or postcard to the selected addresses.

Provide information about how to respond.

Analyze the responses you receive.

Potentially can adjust the sample.

Final Survey Design:Final Survey Design:

Worked with consultant to design an approach that would meet survey goals. Survey much shorter than 2007.

Letter and paper survey mailed to 10,000 randomly selected households.

Multiple ways to respond: mail back paper survey (postage provided) On-line survey in English or Spanish Toll-free phone number in English or

Spanish

Worked with consultant to design an approach that would meet survey goals. Survey much shorter than 2007.

Letter and paper survey mailed to 10,000 randomly selected households.

Multiple ways to respond: mail back paper survey (postage provided) On-line survey in English or Spanish Toll-free phone number in English or

Spanish

Final Survey Design:Final Survey Design:

Oversample of 300 non-Caucasian households in South King County. Telephone calls to targeted phone numbers

(cell and land line)

Monitored responses to ensure demographic alignment.

Built in contingency for random dialed phone calls if needed to get a demographically aligned sample.

Oversample of 300 non-Caucasian households in South King County. Telephone calls to targeted phone numbers

(cell and land line)

Monitored responses to ensure demographic alignment.

Built in contingency for random dialed phone calls if needed to get a demographically aligned sample.

2012 General Survey Responses2012 General Survey Responses

Survey Responses General Survey

South County Over-sample

Total

# of Paper Surveys 837 837

# of On-Line Surveys 176 176

# of Call-ins 3 3

# of Targeted Phone calls

0 300 300

Total 1,016 300 1,316

High Level Findings (n=1,270)High Level Findings (n=1,270)

Generally people are using the collection sites in their region.

Most people say they don’t have hazardous waste.

Slightly under half say they know about sites; about one third say that they have used them.

Strong support for importance of government providing these services from both collection site users and non-users.

Generally people are using the collection sites in their region.

Most people say they don’t have hazardous waste.

Slightly under half say they know about sites; about one third say that they have used them.

Strong support for importance of government providing these services from both collection site users and non-users.

South County Focus GroupsSouth County Focus Groups Selected 4 non‐English speaking

communities based on demographic data: Spanish‐speaking Vietnamese Filipino

(Tagalog‐speaking) Korean Focus Group

Korean

Conducted 5 focus groups (two Hispanic) recruited by and facilitated by a multi‐cultural, multi‐lingual team.

Selected 4 non‐English speaking communities based on demographic data: Spanish‐speaking Vietnamese Filipino

(Tagalog‐speaking) Korean Focus Group

Korean

Conducted 5 focus groups (two Hispanic) recruited by and facilitated by a multi‐cultural, multi‐lingual team.

South County Focus GroupsSouth County Focus Groups Focus group leaders worked to recruit a

diverse mixture of people in each 15 person focus group.

Conducted 43 individual interviews in the African American community.

General survey questions were modified for focus groups and interviews.

Materials were translated into the relevant language by each focus group facilitator.

Focus group leaders worked to recruit a diverse mixture of people in each 15 person focus group.

Conducted 43 individual interviews in the African American community.

General survey questions were modified for focus groups and interviews.

Materials were translated into the relevant language by each focus group facilitator.

Focus Group FindingsFocus Group Findings Results can’t be generalized due to small

sample size. Communities differ in their approach and

level of concern. Common perception that small amounts of

hazardous waste improperly disposed don’t make a difference.

Widespread confusion about what is hazardous.

Confusion about solid waste disposal and hazardous waste disposal.

Results can’t be generalized due to small sample size.

Communities differ in their approach and level of concern.

Common perception that small amounts of hazardous waste improperly disposed don’t make a difference.

Widespread confusion about what is hazardous.

Confusion about solid waste disposal and hazardous waste disposal.

Focus Group FindingsFocus Group Findings

The Spanish community members did not know about the services that we provide, despite our two year Hispanic outreach campaign.

The elderly were unaware of the home collection services we provide.

Generally that services are inconvenient and should be expanded.

The results help illuminate some of the responses in the other surveys.

The Spanish community members did not know about the services that we provide, despite our two year Hispanic outreach campaign.

The elderly were unaware of the home collection services we provide.

Generally that services are inconvenient and should be expanded.

The results help illuminate some of the responses in the other surveys.

Lessons LearnedLessons Learned

Project was too complex with too many moving parts.

Project management schedule was too ambitious.

Both consultants sought revised budgets to cover core activities.

There were problems with communications, oversight and work quality with both consultants.

Schedule regular face to face meetings with consultant at key checkpoints.

Make sure the consultant understands what level of statistical analysis you are looking for and other expectations.

Make sure consultant includes adequate time and budget for writing the report.

Project was too complex with too many moving parts.

Project management schedule was too ambitious.

Both consultants sought revised budgets to cover core activities.

There were problems with communications, oversight and work quality with both consultants.

Schedule regular face to face meetings with consultant at key checkpoints.

Make sure the consultant understands what level of statistical analysis you are looking for and other expectations.

Make sure consultant includes adequate time and budget for writing the report.

Lessons Learned - Next TimeLessons Learned - Next Time Don’t schedule the General Survey and the

Focus Groups at the same time.

Develop a realistic project schedule.

Make sure that the consultant includes adequate time and budget to cover: Needed meetings Analyzing the results Writing the report

Schedule regular face to face meetings with the consultant at key checkpoints.

 

Don’t schedule the General Survey and the Focus Groups at the same time.

Develop a realistic project schedule.

Make sure that the consultant includes adequate time and budget to cover: Needed meetings Analyzing the results Writing the report

Schedule regular face to face meetings with the consultant at key checkpoints.

 

Lessons Learned - Next Time Lessons Learned - Next Time

Make sure that the consultant understands: The type and level of analysis that you

expect. They need to translate technical presentation

of statistics into commonly understood terms. They are expected to meet standard report

requirements. An outline of the report must be approved

prior to drafting it. The first draft should not have grammar or

spelling errors and should be close to final.

Make sure that the consultant understands: The type and level of analysis that you

expect. They need to translate technical presentation

of statistics into commonly understood terms. They are expected to meet standard report

requirements. An outline of the report must be approved

prior to drafting it. The first draft should not have grammar or

spelling errors and should be close to final.

Lessons Learned – Next timeLessons Learned – Next time

Develop focus group questions before or after the general survey questions, not at the same time.

Work closely with the consultant to ensure that concepts are accurately translated and are asked consistently.

Interview consultants to be sure that consultant team members have adequate English proficiency and an understanding of the content that we are trying to collect.

Develop focus group questions before or after the general survey questions, not at the same time.

Work closely with the consultant to ensure that concepts are accurately translated and are asked consistently.

Interview consultants to be sure that consultant team members have adequate English proficiency and an understanding of the content that we are trying to collect.

2012 Surveys and Key Findings:2012 Surveys and Key Findings:

http://lhwmp.wordpress.com/category/publications/

Questions?Questions?

For more information:

LizTennant@kingcounty.gov

206-284-7974