April Webinar: Sample Balancing in 2012

Sample Balancing in 2012

Carter Cathey, Vice President, Excellence Initiatives

How to set and manage your sample balancing options to ensure quality data and happy clients.

2 © 2012 Research Now


Webinar Agenda

Introduction to Balancing

Limits of Balancing

Types of Balancing

“Good Mix”

Representative

Internet Rep

Population Sampling

Balanced

Census Balancing

Mechanisms to Achieve Balancing

Outbound

Inbound

Completes

To nest of not to nest?

2010 Census and Implications for Balancing

Quick Poll: What type of balancing do you use?

1. “Good Mix”

2. Representative

3. Internet Representative

4. Population Sampling

5. Balanced Sampling

6. Census Balancing

7. I use something different.

8. All of the above, depending on the needs of the study.

9. I don’t know / don’t care. Balancing is the responsibility of the panel provider, right?



Market Research = Learning about Specific Populations

To Learn about Specific Populations: Respondents

must Represent Target Audience.

This is the heart of sample balancing.

Introduction to Balancing



Type and Mechanism of Balancing MUST be CUSTOM.

The wrong type/mechanism of balancing can lead to:

Bad Data

Feasibility Issues

Higher Project Costs

Balancing doesn’t always make sense with TARGETED sample.

Limits of Balancing



Definition: An acceptable mix of ages, genders, and other demos, without major skews.

Feasibility Impact: This type of balancing is generally the least panel restrictive.

Key Concerns: This is fine for most research.

Types of Balancing – “Good Mix”



Definition: Representative of some known universe.

Feasibility Impact: Depends on the universe requested.

Key Question: Representative of what?

Key Concerns:

Clients sometimes request “representative” sampling without being specific on the details of the known universe.

Or, they can request “representative” sampling for universes that are unknown or unreachable.

Types of Balancing – “Representative”



Definition: Representative of the online universe.

Feasibility Impact, Key Questions, and Key Concerns: Largely the same as “Representative Balancing.

Types of Balancing – Internet Representative

Definition: Population sampling is representative balancing to the details of a specific population.

Feasibility Impact, Key Questions, and Key Concerns: Largely the same as “Representative Balancing.

Types of Balancing – Population Sampling



Definition: Generally refers to equal amounts of demographic groups, like 50/50 gender, 33/33/33 on three age ranges.

Feasibility Impact: Depends on the elements balanced.

Key Question: What elements need to be balanced?

Key Concerns: This is fine for most research.

Types of Balancing – “Balanced”



Definition: Generally refers to the census-based US population, with varying levels of nested targets.

Feasibility Impact: Typically, this is the most difficult to deliver and most panel restrictive.

Key Question: What elements need to be balanced to the Census?

Key Concerns: This type of balancing can be difficult to deliver and greatly reduce the amount of completed interviews Research Now is able to deliver.

Types of Balancing – Census Balancing



Definition: Creating a balanced set of respondents to invite to the research engagement.

Feasibility Impact: Marginal impact.

Key Question: What are the balancing goals?

Key Benefits: Can be easier to apply weighting controls.

Key Concerns: Frequently does not take into account differential response rates.

Mechanisms of Balancing – Outbound



Definition: Creating a set of respondents to invite to the survey designed to deliver representative respondents to the first question of the engagement.

Feasibility Impact: Can be more significant.

Key Question: What are the balancing goals?

Key Benefits: Corrects for differential response rates.

Key Concerns:

Can make applying weighting controls more challenging.

Response rates are not always consistent.

Can be harder to deliver.

Mechanisms of Balancing – Inbound



Definition: Balancing on “Completes” is essentially using the quota management functionality of the survey instrument to manage the distribution and attributes of the completed interviews.

Feasibility Impact: Varies to a high degree based on the quotas.

Key Benefits:

More control.

Easier to use multiple sample providers.

Minimizes impact of sampling errors.

Key Concerns:

Not a good fit for all research.

Must avoid impossible quota cells.

More sophisticated survey programming platform.

Mechanisms of Balancing – Completes



Top-Line (aka un-nested) Quotas: These quotas are not related to each other.

N=1600

Male: n=800

Female: n=800

< 35 YO: n=800

> 35 YO: n=800

HHI >$35k: n=800

HHI <$35K: n=800

Nested Quotas: These quotas are related to each other.

To Nest or Not to Nest… This is the question.



Key Benefits:

Ensures optimal distribution of sample.

Ensures that you will have minimum sample sizes to make assertions about these sub-sets of the respondent pool.

Key Concerns:

Can very easily create nonsensical quota cells. Think 18 year-old millionaires and Un-acculturated Hispanics with Graduate Degrees.

Can extend field time and make sampling more difficult.

Can lower qualification rates and raise project costs.

To Nest or Not to Nest… This is the question.


Sample Balancing in 2012 – 2010 Census



2010 Census – Key Changes Quick Poll: What version of the census do you use?

1. 2000 Census

2. 2010 Census

3. A hybrid that I alone have master-engineered

4. I don’t use the census.



Always Balance: At a minimum, balance for age, gender, and income.

Always Share Balancing Objectives: Always share your specific balancing needs, expectations, and objectives with your partners and providers.

Consider Nesting: When appropriate, nesting age, gender, and region should be applied.

Don’t Always Nest Ethnicity: Ethnicity targets should only be nested when required by the needs of a specific project.

Quotas should be discussed, and never assumed. Using quotas (rather than sampling targets) to control for sample mix may have a more significant effect.

Use 2010 Census as Default. If you haven’t already, you should consider using the 2010 estimates.

Tracking Studies. There will likely be a noticable impact in the data if you switch a currently fielding tracking study from 2000 to 2010 Census balancing.

What does it all mean? Research Now Recommendation


Questions?

Carter Cathey

[email protected]

April Webinar: Sample Balancing in 2012

Business

Transcript of April Webinar: Sample Balancing in 2012