HomeTown Competitiveness:Conducting Phase I Assessment

Jennifer Bott, PhD, SPHRAssociate Professor of Management Miller College of Business Ball State University

Agenda

•Background on HTC Assessment•Goals of Assessment•Types of Assessments

▫Focus Groups and interviews▫Designing surveys▫Archival data analysis

•Basic analyses and decision making•Resources

Background on Assessment

•Phase I Assessment phase typically lasts up to four months▫Community conducts local assessments of

development history, context, issues and assets

▫Create customized strategic opportunities•Leads into implementation of the

strategic development plan in Phase II

Assessment Goals

•Determine perspectives of community stakeholders on chosen pillar within the community: ▫Entrepreneurship▫Charitable Assets▫Youth Engagement▫Leadership

Assessment Outcome

•End product of the assessment:▫Collection of data that direct community

team toward path of action▫Summary of current thinking in community

on current status of selected pillar•Launching assessment with community

should engage them more in the process, outcome and implementation of the strategic plan

Types of Assessments

•Traditional measures of assessment:▫Focus groups▫Interviews▫Surveys (telephonic, online, mail, intercept)

•Other opportunities:▫Re-analysis of previously-collected data

Types of Data

•Two primary types of data emerge from assessments:▫Qualitative

Verbal responses to open or closed-ended questions

▫Quantitative Numeric responses to (usually) closed-ended

questions

Choice of Assessment(s)• Assessment method should:–Provide most insight into questions at hand– Involve stakeholders–Adhere to following criteria:• Reliable• Valid• Cost-effective• Feasible

• Combinations of assessment methods may be most beneficial

Focus Groups

•Characteristics:▫7 – 10 members▫Share some relationship to topic of interest▫Represent divergent opinions on topic and

typically represent different stakeholders•Repeated several times with different

groups of people•Requires collection of qualitative data

Topics for Focus Groups

•Strategic planning•Needs assessment•Current state analysis•Program design•Perspectives on a singular issue

Uses of Focus Groups

•Focus groups are especially useful:▫Exploratory, preliminary study▫Gap in understanding or communication▫Uncover data related to complex behavior▫Discover interactive synergies between

participants▫Uncover additional information leading to a

large-scale study effort▫High value placed on open-ended data

collection

When NOT to Use Focus Groups•Focus groups are not recommended for:

▫Emotionally-charged subject▫Researcher/HTC team not in control of the

environment or message▫Team expects statistical summaries from

data▫Is not economical to achieve data collection

needed▫Confidentiality can’t be guaranteed

Outcomes of Focus Groups

•Categorized data from focus groups can be used:▫As stand-alone, qualitative information▫As starting point for development of a

quantitative measure

Designing Successful Focus Groups• Create “safe” environment–May require use of a neutral facilitator

• Create a plan for focus group that details:–Purpose –Who to include • Participants• Number of sessions

– Incentives for participation (if necessary)–Location

Focus Group Questions• Categories of questions:– Opening • Everyone answers quickly (10 - 20 seconds)

– Introductory • General topic of discussion, foster conversation

– Transition • Approach the focal questions

– Key • Drive the study – 2 to 5 questions that require

most attention– Ending • Bring closure and allow for reflection

Focus Group Questions (cont’d)

• In most cases, focus group questions are open-ended– “What are our greatest weaknesses in

youth engagement?”• Closed-ended questions are not

appropriate for sparking discussion– “Would you participate in a training

success?”• Avoid asking why in response to a

participant’s answer

Analyzing Focus Group Data

•Determining analysis strategy:▫Least time intensive = Memory based

Report is created immediately based on facilitator’s memory

▫Most time intensive = Transcript based Combines field notes with transcript to write

final report▫Most common = Note based

Relies on field notes to create focus group report

Focus Group Report

•The focus group report summarizes:▫Participants▫Setting▫Key questions▫Themes that emerged from key question

responses▫Limitations▫Recommendations

Conducting Interviews

•Interviews can also be used as assessment tools▫One-on-one or panel (2 or more) ▫Collect qualitative data▫Can be used as singular collection method

or in tandem with focus groups or surveys

Uses of Interviews

•Appropriate when:▫Group perspectives are not valuable▫Topic is too sensitive to discuss in group

setting▫Want to collect limited information (e.g.,

only want to speak to a few individuals)▫Focus groups are not feasible

Uses of Surveys

•Surveys are appropriate when:▫Know the parameters of the content to be

assessed▫Have identified a sample that will provide

meaningful data▫Need quantitative figures (rather than

qualitative descriptions)•Can be coupled easily with qualitative

data collection

Considerations with Survey Administration and Design•Decision points on survey design:

▫Sample Who will respond? How many?

▫Administration mode Telephone Intercept (in-person) Snail mail Internet

▫Necessary response rates

Survey Questionnaire Development•Six basic steps:

▫Decide what information to collect▫Decide what type of questionnaire should

be used▫Develop first draft▫Revise questionnaire▫Pilot test▫Edit questionnaire and specify methods of

collection

Question Writing Guidelines

•Characteristics of “good” items:▫Simple, direct, familiar language▫Clear and specific questions▫Do not use leading, loaded, or double-

barreled questions▫Use short statements (20 words or less)▫Edit for readability

Designing the Scale

•Most common response options:▫Agreement▫Frequency▫Evaluation

•Number of response categories ▫3 to 5

•Neutral point – good or bad idea?

Evaluate Wording

•Which is a better question?▫Age?▫What was your age on your last birthday?

•Is there a problem with this item?▫What do you like best about this

neighborhood? (We’re interested in anything, like houses, the people, the parks, or whatever.)

▫Do you favor or oppose gun control legislation?

Item Stem Creation

•Agreement items – Declarative▫“I am uncomfortable around strangers.”

•Frequency items – Events, behaviors▫“You exercise strenuously enough to raise your

heart rate.”•Evaluation items – Persons, places, things

▫“How well your favorite sports team played last week.”

Designing the Survey

•Some simple rules for the ordering and design of questions:▫Keep items together if they use the same

responses▫Don’t overfill each page with questions▫Avoid including too many different

response options•Pilot test with individuals similar to those

being sampled

Importance of Instructions•Give direction for response for those

unfamiliar with format•Create a common frame of reference

▫“The following questions concern peoples’ opinions and beliefs about jobs and careers. These questions refer to jobs in general and not the job you presently have or a particular job you once had.” (Spector, 1992)

Using Archival Data

•In some cases, data has already been collected regarding your pillar of interest▫Ask groups that are associated with your

pillar (e.g., Chamber of Commerce, local schools, philanthropic groups)

▫Can be either quantitative (surveys) or qualitative (interviews, news coverage, etc.)

Possible Analyses with Archival Data•Content analysis•Assessment of trends•Analysis of communications

Limitations

•Archival data has many advantages and disadvantages▫Advantages:

Data is already collected (cheap, easy to use) Data may already be analyzed

▫Disadvantages: Can’t be tailored to fit needs Inability to determine the reliability of the

methods used to collect data

Conclusion of the Assessment

•Use the assessment data to identify “key findings” about the pillar assessment▫A key finding is a full sentence used to

summarize learning. Including a statistics strengthens this finding. “73% of high school students surveyed stated

that they would return to our community if opportunities exist.”

Conclusion (cont’d)

•Key findings are used to drive the development of the problems to be solved at the end of Phase I.

Tangible Outcomes of Assessment•A written report should be provided to

individuals who participated in the survey▫Details overall findings (at the group level

only)▫May provide information about uses of the

assessment Depends on confidentiality of the process

▫Can hopefully meet a need of the sample

Moving to Decision Making

•Totality of evidence from both qualitative and quantitative assessment efforts should be examined▫Looking for evidence that supports teams’

beliefs or uncovers new, unexpected information

▫Evidence should create a logical next step in assessment or implementation

Resources to Aid in the Assessment Process•Center for Business and Economic

Research at Ball State University▫Partnered with Building Better

Communities•Professional consultants•Other HTC Communities

Resources (cont’d)

•Books:▫Survey Research Methods (4th ed.), Floyd

Fowler (Sage Publications, $39.95)▫Focus Groups: A Practical Guide for

Applied Research, Krueger and Casey (Sage Publications, $45.32)

Links to Nebraska Surveys

•http://www.zoomerang.com/Shared/SharedResultsSurveyResultsPage.aspx?ID=L22Z2S57GK54

•http://www.zoomerang.com/Shared/SharedResultsSurveyResultsPage.aspx?ID=L23KD72VDQVC

•http://www.zoomerang.com/Shared/SharedResultsSurveyResultsPage.aspx?ID=L23KDC4HWP6J