Washington Weighted WAC

Survey Final Report

Summary of Focus Groups, Survey Methodology and Results

December 2016-April 2017

1

TABLE OF CONTENTS

FOCUS GROUP DATA COLLECTION ................................................................................................................ 2

SURVEY SAMPLING METHODOLOGY .............................................................................................................. 5

SURVEY RESULTS AND ANALYSIS ............................................................................................................... 11

2

FOCUS GROUP DATA COLLECTION

In December 2016 and January 2017, Public Consulting Group, Inc. (PCG), in partnership with Washington

Department of Early Learning (DEL), Thrive Washington, and local regional coalitions, facilitated 14 focus groups

across the state of Washington. The goal was to collect input from key stakeholders in the field of early childhood

education on which regulations within the Washington Administrative Code (WAC) should be included in a survey

to be weighted in accordance with best practices for differential monitoring.

This report provides a high-level summary of the methodology and findings from each of these efforts.

Methodology

Thrive Washington, a locally-based nonprofit early childhood advocacy group, reached out to the 10 Early Learning

Regional Coalitions (ELRC). The ELRCs are geographically organized, structured networks of early learning

stakeholders. ELRC leadership was responsible for logistics surrounding the focus group, including date and time,

location, invitation list, and follow-up with participants. The ELRCs reached out to a representative sample of 20

early childhood stakeholders from their regions to participate in WAC Survey focus groups, taking into account

occupation (provider types or community stakeholders), languages spoken, and demography. To verify that focus

group participants were representative of the larger state, PCG tracked focus group participation by geography,

stakeholder type, and demographics; additional focus groups were added to meet statewide representation goals.

Focus Group Table 1. Focus Group Distribution

Focus Group Number

of Focus

Groups

ELRC Region 11

Pierce County 1

Northeast 1

Central 1

Northwest 1

Olympic-Kitsap Peninsulas 1

North Central 1

Southwest 1

King County 2

Southeast 1

West Central 1

Parent Group (in King County) 1

Head Start/ECEAP Providers (in King County) 1

WA Department of Early Learning Licensors 1

TOTAL Number of Focus Groups 14

Through the 14 focus groups, 198 stakeholders participated. Below is a breakdown of representation and the

percentage of the total stakeholder group.

Focus Group Table 2. Focus Group Participation by Stakeholder Type and Demographics

Focus Group Participant Representation Count Percentage of

Total

Stakeholder Type

Center Providers 71 36%*

Home Providers 69 35%*

3

Focus Group Participant Representation Count Percentage of

Total

Union Members 44 22%

Tribal Providers 10 5%

Other Stakeholders 58 29%*

Demographics

American Indian or Alaskan Native 10 5%

Asian 7 4%

Black or African American 17 9%

Hispanic 49 25%

White 87 44%

Other 1 >1%

Prefer not to answer 1 >1%

Did not participate in optional survey 24 12%

Total 198 100% *The total of Center Providers, Home Providers, and Other Stakeholders is 100% or 198 participants. Union Members are home providers; this percentage is

representation of the total participants. Tribal Providers could be either center or home providers; this percentage is representation of the total participants. Focus

Group Participant Representation information was gathered through voluntarily, anonymous surveys taken by participants.

Focus groups were primarily conducted in English; however, interpretation services were offered at selected focus

groups for the following languages: Spanish, Somali, and Vietnamese. Written instructions were also translated into

Spanish and Somali for participants’ use; materials were translated into Russian, but no focus group participants

requested Russian interpretation supports.

PCG staff members and DEL representatives facilitated the focus groups. For consistency, all focus groups followed

the same format. After a short introduction to the event, participants were briefed on the methodology behind

differential licensing and weighted regulations. Participants were informed the purpose of the evening was to

determine which WAC regulations should be included in a subsequent survey for weighting. Given the length of the

WAC, participants were randomly assigned to one of three sub-groups to review a portion of the WAC. Sub-groups

were responsible for reading each WAC regulation and determining the value of including the WAC on the survey,

based on the perceived level of risk of harm to children. Then a representative of the sub-group recorded the final

decision on a master copy of the regulations by marking a check next to the specific regulation to include, or by

marking an X to exclude. At the end of each focus group, the master copy from each sub-group was collected by

PCG for analysis.

The length of each focus group was approximately three hours, with sub-group review of regulations accounting for

approximately 1½ hours of the time. At the end of the session, the three sub-groups were allowed to share their

decisions and methodology with the larger group, and answer questions from the larger group. Methodology for

decision-making on whether to include the WAC regulation on the survey included: democratic rule, erring on

including the WAC if there was a prolonged discussion or split vote, and excluding regulations appearing as

definitions or the DEL-specific provisions1.

As a final exercise, focus group participants were asked to recommend stakeholders to take the Weighted WAC

Survey. A variety of the recommended stakeholder groups were included in the pool from which survey respondents

were to be randomly selected.

Results

1 DEL specific provisions are classified as the duties and tasks DEL is responsible to complete. Examples include required paperwork DEL

must complete during the licensing process or the timeline in which DEL must complete inspections.

4

PCG reviewed all sub-group master copies and recorded the sub-groups’ recommendations in an Excel-based tool.

Recommendations for survey inclusion were tallied for each WAC regulation.

PCG determined, in accordance with focus group recommendations, that the following WAC regulations would be

excluded from the survey:

Definitions

DEL provisions and roles1

Provider provisions2

These types of regulations are unrelated to risk throughout the monitoring process.

PCG included specific regulations in the survey based on the majority of recommendations. In the event of a tie,

where the tallies of “include” and “exclude” recommendations were equal, PCG included the WAC in the survey.

The table below outlines the inclusion and exclusion trends.

Focus Group Table 3. Themes and Trends of Focus Group Results

Include in WAC Survey: Exclude from WAC Survey:

Written documentation of policy Definitions of terms

Documentation of child’s health and wellness DEL provisions1 and role

Job descriptions and qualifications for those in

direct contact with children

Provider provision2

Trainings related to health and safety;

emergency drills and preparedness

Written documentation of events, such as

trainings or absences

Illness, contagion, immunizations Professional development related to business

practices

Cleaning and sanitation Dietary requirements, such as menus and health

level of foods

Eating and food preparation Space requirements

Equipment and materials in child environment Curriculum requirements and philosophy of

education

Staffing, ratios, and supervision Play and creativity requirements

Special needs accommodations

Discipline policy, threat of harm, and bullying

Parent communication

As a result of the focus group recommendations, the Weighted WAC Survey was narrowed to 450 questions. The

following table illustrates the percentage of WAC regulations that will be included in the survey.

Focus Group Table 4. Percentages Included in Survey, by WAC Section

Section of WAC Percentage to

Include in Survey

Intent and Authority 29%

Child Outcomes 100%

Family Engagement and Partnerships 80%

Professional Development, Training, and Requirements 83%

Environment 92%

Interactions and Curriculum 54%

Program Administration and Oversight 56%

2 Provider provisions are regulations containing allowances. An example include a provider’s allowance to have pets. This permission is not a

standard one could violate and therefore risk of harm to children, if violated cannot be assessed (weighted).

5

SURVEY SAMPLING METHODOLOGY

Public Consulting Group, Inc. (PCG), a management consulting firm dedicated to working with the public sector, is

contracted to independently design, implement and analyze the Washington Weighted WAC Survey (Washington

Administrative Code for Child Care Regulations) on behalf of Department of Early Learning (DEL). DEL, in

collaboration with an Advisory Committee on Standards Alignment, has been undergoing a standards alignment

process3 since the fall of 2015. As part of the alignment, PCG gathered stakeholder input to design a survey to

assign risk “weights” to each regulation in the new draft proposed aligned standards issued in November 2016. 4

Risk weights will be used in a risk-based assessment for health and safety licensing and monitoring processes.

DEL will finalize the new risk-based assessment processes following final ruling on the aligned rules, pilot, review

and training. The risk-based assessment process will be a component of DEL’s differential monitoring model ‒ one

that is steeped in national best practices for child care licensing and monitoring.5

To engage stakeholders in all steps of the alignment and weighted standards processes, and to design a more

focused survey of key regulations and determine stakeholders for participation, PCG worked with Thrive

Washington (Thrive) and the state’s Early Learning Regional Coalitions to hold 14 statewide focus groups. Focus

group results drove the survey sampling methodology and survey design6.

Survey Design

The weighted WAC methodology adopted by DEL and PCG is founded in Dr. Richard Fiene’s national best practices

model for child care weighted risk-assessment.7

Risk Assessment is an approach that focuses on identifying and monitoring those rules that place children at greater

risk of mortality or morbidity if violations or citations occur.

The methodology indicates that state child care regulatory agencies put all child care regulations into a survey

format for risk-weighting by key stakeholders in the early childhood education and child care health and safety

community. The survey should be disseminated to at least 100 individuals, more if the state has over 3,000 licensed

facilities in the type of services being surveyed. DEL and PCG determined that not only should more than 100

individuals be included in the survey sample, but that the sample should be statistically representative of child care

health and safety experts in the stakeholder community. In other words, the results should have a high level of

reliability that the resulting weights for each regulation are representative of all stakeholders’ assessments of risk.

The survey was made available for a three-week period, from February 20 to March 13.8 PCG prepared frequently

asked questions (FAQ) and a live-recorded webinar to offer information about the process, survey methodology

and logistics. Both the FAQ and webinar recording were made available to all participants and posted on DEL’s and

Thrive’s websites. In addition, a dedicated email address and toll-free hotline were made available to participants

throughout the survey to provide participants with technical assistance. Assistance with interpretation and IT needs

was provided by the Early Learning Regional Coalitions in partnership with Thrive.

3 Washington State Department of Early Learning. Standards Alignment. https://www.del.wa.gov/government/standards-alignment 4 Washington State Department of Early Learning. Proposed Aligned Standards. https://www.del.wa.gov/government/proposed-aligned-

standards 5 Administration for Children and Families, Office of Child Care. Contemporary Issues in Licensing, Monitoring Strategies for Determining Compliance: Differential Monitoring, Risk Assessment, and Key Indicators.

https://childcareta.acf.hhs.gov/sites/default/files/public/1408_differential_monitoring_final_1.pdf 6 Washington State Department of Early Learning. Weighted WAC Survey Focus Group Summary https://www.del.wa.gov/sites/default/files/public/Weighted_WAC_Survey_Focus_Group_Summary.pdf 7 Fiene, R., Kroh, K. Human Service Licensing Measurement, Regulatory Compliance, and Program Monitoring Systems https://www.researchgate.net/publication/302876821 8 Initially, it was made available to participants for a 2-week period, from February 20-March 3, 2017. A one-week extension was provided to

allow sampled participants more time to complete.

6

Survey Questions

The WA Weighted WAC Survey was designed to ask participants to designate the weight—or level of risk—of the

proposed licensing rules of harm to children on a scale of 1 (lowest risk) to 8 (highest risk) if the rule is NOT followed.

In other words:

If the proposed WAC will not be followed by a child care provider, how likely is it on a scale from 1 (lowest

risk) to 8 (highest risk) that a child will be directly and/or indirectly harmed?

“Weighted”: means an assigned level of importance or value to each proposed licensing rule. The more “weight” is

assigned, the riskier the violation of the rule is and the more likely that children will be directly and/or indirectly

harmed if the rule is not followed. A higher weight also means that the level of harm could be higher.

In December 2016 and January 2017, PCG, in partnership with DEL, Thrive, and the Early Learning Regional

Coalitions, facilitated 14 focus groups across the state. The focus group participants provided input on which

stakeholders should be included as survey participants and which regulations should be included as questions for

weighting on the survey. Regulations that were not included in the survey were those that focus group participants

felt were “little to no risk” of harm to children if violated. The regulations not included will be calculated at a risk level

of 1. The resulting survey narrowed the regulations to be included on the survey from approximately 800 to 450

questions (when each regulation and sub-section were split into a single question).9

Sampling and Stakeholder Groups

PCG also took feedback from the statewide focus groups to determine the stakeholders that should be invited to

participate in the survey. Focus groups determined two survey populations: “experts” in child care health and safety

regulations—people who are very familiar with the impact of regulations on children, providers and licensors—and

other stakeholders. Other stakeholders are those who have a vested interested in health and safety but are not

necessarily proficient in all child care regulations and the impact on children.

Group 1: “Early Learning Experts”

o Providers: center, home-based, ECEAP, subsidy, Early Achievers, Head Start, Homeless child care

subsidy

o DEL Child Care Licensors

Group 2: Other Stakeholders

o Parents

o Professionals: medical, colleges, health and safety professionals and practitioners

Focus group participants believed “experts,” being well-informed on the regulations, should make up the vast

majority of the overall survey responses (at about 85%), while other stakeholders should make up a smaller portion

(approximately 15%).

Focus group participants stated that it would be difficult to recruit other stakeholder to participate due to the length

of the survey. Furthermore, stakeholders said some survey respondents could have difficulty completing risk

assessment ratings for all health and safety topics included in the WAC given their experience would fall primarily

outside of early childhood education.

Given the desire to have the results weighted by 85% “expert” and 15% “other stakeholder” and the difficulty of

reaching other stakeholder participants, PCG developed a two-pronged survey collection approach: one for experts

and one for other stakeholders.

9 Weighted WAC Survey Focus Group Data Collection summary

https://www.del.wa.gov/sites/default/files/public/Weighted_WAC_Survey_Focus_Group_Summary.pdf

7

“Experts” Sample

The random sampling methodology sought to reach a random representative sample of “expert” stakeholders

across the state. A random sample methodology is sometimes referred to as “probability sampling,” which means

that nearly every person in the population (included in the stakeholder groups) has an equal chance of being

chosen to participate in the survey.

As described above, DEL and PCG determined that the sample should be statistically representative of child care

health and safety experts in the stakeholder community. PCG designed the sample to meet a 95% confidence level.

The confidence level is a measure of reliability that the result represents the full population. In other words, 95%

confidence level means that if the survey were repeated with the expert stakeholder population, the results would

match the results of the actual full population 95% of the time. This means the average risk rating result for each

regulation cannot be explained by sampling error alone.

The survey was also designed with an acceptable range for the confidence interval for this survey to be 5-7. The

confidence interval provides more assurance about the certainty of the results (in this case risk score) in the

collected data. It is a range of results. In this survey, the confidence interval reflects the range the risk score may

be from the mean sample for a regulation. For example, if the average risk score for a regulation was 5, we expect

results to be within 4.55 and 5.45; in this example, both ends of the range, when averaged to the whole number,

equate to a risk score of 5.

Based on prior experience with similar surveys and the active early childhood education stakeholder community in

Washington, PCG assumed a 25%-30% response rate of completed surveys. With a total population of 5,111

providers, 358 providers would need to be sampled in order to meet the 95% confidence level and 5 confidence

interval.

To maintain a high level of reliability, PCG was seeking completed responses between a 5 and 7 confidence interval.

This level of reliability ensures that the results represent the population. The following table outlines the total

population, target sample at a 5 confidence interval, total outreach and results. The total responses of completions

and partial completions met the target with a confidence interval of 5.31.

Survey Sampling Methodology Table 1. Total Response Population and Confidence Interval

Stakeholder Group (85% of final sample)

Total Population

Target % of Group

Target Sample (95% confidence level)

Total Number of Participants invited

Results Confidence Interval

Providers 5,111 85% 304 1,217 243 6.14

Licensors 102 15% 54 102 77 5.56

Total "Experts" 5,213 100% 358 1,319 320 5.31

Tracking “Experts” Sub-groups

Participants were assigned identification numbers so that PCG could track responses and conduct targeted

outreach, as well as to maintain participants’ anonymity. Those identification numbers were only used for PCG’s

purposes and will not be shared with DEL or other stakeholders.

In addition to ensuring representation of the sample of the overall expert population, PCG used identification

numbers to targeted follow-up efforts to ensure representation by specific sub-groups and characteristics of the

state’s provider population. The sample results for all characteristics were representative of the full population. The

following tables provide results of the final sample representation by: provider type (center, family), primary

language, county, union affiliation, subsidy participation, Head Start participation, serves homeless children, and

Early Achievers participation.

8

Survey Sampling Methodology Table 2. “Expert” Sample by Subgroup

Provider Subgroup Family Child Care

Center Child Care

Total Total Population %

Sample %

Final Result %

Total 99 144 243 100% 100% 100%

ELC: Central 20 7 27 7% 7% 11%

ELC: King 41 31 72 35% 34% 30%

ELC: North Central 11 3 14 7% 8% 6%

ELC: Northeast 10 14 24 6% 6% 10%

ELC: Northwest 25 15 40 14% 14% 16%

ELC: Olympic Kitsap 3 2 5 3% 3% 2%

ELC: Pierce 13 12 25 9% 11% 10%

ELC: Southeast 10 7 17 6% 6% 7%

ELC: Southwest 6 5 11 5% 5% 5%

ELC: West Central 5 3 8 6% 6% 3%

Language: English 71 73 144 50% 51% 59%

Language: Spanish 25 0 25 13% 13% 10%

Language: Somali/Oromo 7 0 7 5% 5% 3%

Language: Other 3 1 4 3% 4% 2%

Language: Unknown 38 25 63 28% 26% 26%

Head Start Participation 0 7 7 2% 2% 3%

Homeless Subsidy 1 2 3 3% 2% 1%

Accepts Subsidy 92 63 155 69% 69% 64%

Union Affiliation 84 0 84 42% 42% 35%

Capacity: Large (21+) 0 91 91 28% 27% 37%

Capacity: Medium (11-20) 86 6 92 40% 40% 38%

Capacity: Small (1-10) 58 2 60 32% 33% 25%

Other Stakeholder Sample

Because the “other stakeholder group” sample is made up of parents and professionals, there is not a reliable

estimate of all stakeholders statewide that could be included in the survey. In other words, the size of the entire

population is unknown. As described above, focus group participants believed that not all individuals in the

population would be interested or knowledgeable to provide full responses and many are difficult to reach. For all

of the stated reasons, a “convenience sample methodology” was used for the “other stakeholders” sample.

A convenience sample is a non-probability sampling technique that survey researchers use to gather responses

from stakeholders who are readily available, and/or a self-selection of individuals willing to participate. PCG worked

with Thrive and the Early Learning Regional Coalitions to identify parents willing to take the survey. PCG also

worked with DEL to identify professionals in the early learning and health and safety fields. In total, PCG conducted

outreach via phone and email to 211 “other stakeholders.”

Because 85% of the final results would be “experts,” 15%, or 63 individuals would need to make up the “other

stakeholder” sample. In order to most efficiently allocate those 63 remaining responses to each of the different

populations, PCG split evenly among parents and early learning professionals: medical, colleges, and health and

safety professionals.

Participants for the “other stakeholders” sample were also assigned identification numbers so that PCG could track

responses and conduct targeted outreach, as well as to maintain participants’ anonymity. Those identification

numbers were only used for PCG’s purposes and will not be shared with DEL or other stakeholders. The following

9

table provides an overview of the “other stakeholders” outreach and results. The results of 66 respondents

exceeded the 63 target.

Survey Sampling Methodology Table 3. “Other Stakeholders” by Subgroup

Stakeholder Group Est. Size Target % of Group

Target Sample

Total Outreach

Results

Parents 61,401 50% 32 105 29

Professionals 200 50% 32 105 37

"Other Stakeholders" 61,601 100% 63 211 66

Total Responses

Of the 391 total survey responses PCG received, 231 were fully completed while 160 were partially completed. Of

the partial completions, a majority answered at least 100 questions or more, with most of the regulations in the

environment section receiving more than 300 responses. In later portions of the surveys, participants tended to

answer less questions, likely due to either fatigue or they felt that the content was not their specialty (other

stakeholders, in particular). The range of questions saw a high of 383 to a low of 213.

Based on a 95% confidence level, we can calculate that the margin of error for a question with 383 responses was

5%, while a question with 213 responses has a margin of error of 7%. This means that we can be 95% confident

that the whole population of providers, licensing staff, and other stakeholders agree with the risk as assigned

between 5%-7%, depending on how many people responded to the question in particular. Margins of error at this

level are fairly common and determined by PCG as acceptable for the purpose of this survey

Outreach

PCG conducted a series of outreach for both the “expert” and “other stakeholder” groups.

Email Campaign: Series of English and Spanish email notifications and reminders sent to participants

where email was available

o One (1) initial outreach email

o Three (3) reminder emails sent to participants who had not begun the survey

o Three (3) reminder emails sent partially completed their survey with a unique link to access their

responses

o One (1) extension notification reminder email sent to all participants who had not completed the

survey

o One (1) final reminder email sent to all participants who had not completed the survey

Phone Campaign: Series of pre-recorded English and Spanish call notifications and reminders to

participants where a telephone number was available

o One (1) initial outreach call

o Four (4) reminder calls to participants who had not begun the survey or only partially completed

the survey

o One (1) extension notification call sent to all participants who had not completed the survey

Thrive/Coalition Reminders

o PCG provided Thrive with a list of survey participants by ELRC; Thrive then disseminated the

participant list to ELRC leadership. The ELRC reached out personally to participants to provide

reminders to complete the survey, along with making participants aware of the technical and

language supports Thrive offered.

10

Technical assistance was provided by PCG through email support, hotline, and via partnership with Thrive and the

ELRC. Thrive and ELRC were responsible for coordinating translation and interpretive services, along with providing

access to technology needed to complete the survey (internet, computer).

In King County, Thrive and the ELRC worked to set up interpretive and technical assistance to the Voices of

Tomorrow, a group of Somali childcare providers.

Upon completion of the survey, participants could access another link where they could register for an incentive for

completing the survey. The incentive was provided by donation from the Washington State Library, funded by the

Institute of Museum and Library Services; it was a children’s book titled Words are Not for Hurting.

11

SURVEY RESULTS AND ANALYSIS

Data Analysis

In total, the proposed aligned standards of the Child Care WAC include about 700 regulations (when all subsections of each regulation are counted as one).10 254 WAC regulations were excluded from the survey through the Focus Group process, of those, 98 regulations were determined to be definitions or DEL/Provider provisions and authority.11 The 98 regulations, being definitions and statements of authority cannot be violated and therefore enforcement of an assigned weight is impracticable; thus, these regulations were given a weight of 0. The remaining 156 regulations identified by Focus Groups retained their weight of 1.

Survey results for the remaining 446 regulations were aggregated an analyzed using both the mean and median of all participant responses. Mean and median are two ways of determining where the central, or average, value falls in a data set. The mean is the sum of all data points divided by the total number of data points while the median orders all data points in sequential numerical order to determine the middle data point12.

Table 1 outlines the distribution of the mean weights as calculated from the survey responses, categorized by WAC Regulation Section. Also included are the focus group results.

Survey Results & Analysis Table 1. Mean Distribution

Focus Group

Results

Survey Results

WAC Regulation Section Total

Regulations 0 1 1 2 3 4 5 6 7 8

Intent and Authority 38 26 6 6

Child Outcomes 4 3 1

Environment 314 14 20 13 76 135 56

Family Engagement and Partnerships

6 4 2

Interactions and Curriculum 117 11 48 9 29 20

Professional Development and Training

81 9 20 6 19 25 2

Program Administration and Oversight

140 38 58 14 23 6 1

TOTAL Count 700 98 156 0 0 22 127 212 84 0 0

It can be noted the majority of the mean scores cluster between a weight of 3 and 6; with no mean scores from the survey reflecting a weight of 1, 2, or 8. Only one mean score resulted in a weight of 7.

Table 2 outlines the distribution of median weights as calculated from the survey responses, categorized by WAC Regulation Section. Also included are the focus group results.

10 The stakeholders in the 14 statewide focus groups reviewed all regulations and their subsections separately to determine whether weights

should be assigned as a whole or individually as subsections. When WAC regulations are counted as individual subsections there are over 800 WAC. The focus groups grouped and consolidated regulations into 700; some of which are grouped some are subsec tions. 11 Example of a definition rated a 0 is 170-300-0401 (4): the definition of the annual licensing fee; an example of a Provider provision is 170-300-0225 (1): stating a provider may have pets. 12 In an odd number of data points, the middle data point is easy to calculate; for example, in a set of 11 data points, 6 would be the middle

data point. In an even number of data points, the average of the 2 middle data points is taken; for example, in a set of 10 data points, the

average of data point 5 and 6 would be taken

12

Survey Results & Analysis Table 2. Median Distribution

Focus Group

Results

Survey Results

WAC Regulation Section Total Regulations 0 1 1 2 3 4 5 6 7 8

Intent and Authority 38 26 6 2 4

Child Outcomes 4 1 3

Environment 314 14 20 4 28 76 92 59 21

Family Engagement and Partnerships 6 4 1 1

Interactions and Curriculum 117 11 48 1 13 15 19 10

Professional Development and Training 81 9 20 4 10 19 13 6

Program Administration and Oversight 140 38 58 3 19 11 7 4

TOTAL Count 700 98 156 0 0 9 48 132 131 91 35

The distribution of median weights has a wider spread, with weights ranging from 3 to 8. Again, no regulations had a median weight of 1 or 2 resulting from the survey.

Weight Assignment Rational

According to Dr. Richard Fiene’s risk assessment methodology for child care differential monitoring13, the mean or

the median are both acceptable methods of calculating the representative risk weight of a regulation.

Taking the mean of a data set is most representative of data that are evenly distributed. When data are highly skewed14 or where outliers are present, the mean becomes less representative of the central concentration of the data. The mean can also result in an average that is not demonstrated in the original data set. For example, in a bi-modal5 data set where half the participants chose a weight of 8 and half chose a weight of 5, the mean would suggest the weight of the regulation should be a 6, a weight not distinctly chosen by any participant. 15

The median is less influenced by skewness in a data set. Where data is not normally distributed, the median is a better representation of the central concentration of the data sample16. It also assures that the central value is a data point clearly chosen by participants, as opposed to the example above where the mean can reflect a data point not explicitly chosen by participants.

During the analysis of the WA Weighted WAC Survey data, it became clear nearly half (47%) of data collected was skewed in nature. This means for nearly half of the regulations evaluated on the survey, participants’ responses were heavily concentrated around a particular weight or were bi-modal17. Because of this skewness, the mean and median were not in agreement, as they would if the data were normally distributed.

13 Fiene, R., Kroh, K. Human Service Licensing Measurement, Regulatory Compliance, and Program Monitoring Systems

https://www.researchgate.net/publication/302876821 14 Skewness refers to the distribution of data is asymmetrical, rather than a normal distribution that looks like a bell curve. In skewed data sets, the mean and median are not equal; in a normal distribution, the mean and median are equal. 15 Note that with the WA Weighted WAC survey, no outliers were identified. Normally, outlier data points can have a disproportional influence

over the mean. While outliers represent a small portion of the total sample population, outliers can significantly shift the mean away from the

central concentration of the data sample. 16 https://statistics.laerd.com/statistical-guides/measures-central-tendency-mean-mode-median.php 17 In bi-modal distribution, the responses are heavily concentrated around several weights; which means the data was not normally

distributed.

13

Table 3 outlines the number of regulations in the survey where data were normally distributed and where data was skewed in distribution.

Survey Results & Analysis Table 3. Normal vs. Skewed Distribution

Results of Survey Count Percent

Normal distribution (the mean is representative; equals the median)

273 53%

Skewed distribution (the mean is not representative; mean and median different)

209 47%

Total 446 100%

Given nearly half of the data received from the survey was not normally distributed where a mean score would be unrepresentative, statistical best practices indicate that the median is a more representative method for determining the result, or the risk weight, for this data set6.

The best practice of using the median to identify the average also allows for a wider distribution of risk-weights, that more accurately reflects the survey responses. The wider spread of the distribution of median scores seen in Graph 2 when compared to the tight distribution of the mean scores in Graph 1 reflects how the mean scores were highly influenced by skew. The presence of median scores of 7 or 8 in Graph 2 reflects a more accurate representation of the range of data in the survey responses.

Survey Results & Analysis Graph 1. Mean Distribution Survey Results & Analysis Graph 2. Median Distribution

The median scores will be the proposed risk-weight assigned to the proposed WAC draft regulations presented during the public comment period.

020406080

100120140160180

0

50

100

150

200

250

14

Results

Table 1, shown above, details the distribution of the total 700 proposed WAC draft regulations by median risk-weights as calculated from the Focus Group and Survey process for each section. Below, Table 4 provides a summary of the 700 regulations by proposed risk-weight.

Survey Results & Analysis Table 4. Summary of Regulations by Risk-weight Risk-weight 0 1 2 3 4 5 6 7 8

Number of Regulations

98 156 0 9 48 132 131 91 35

Percent of Total (700)

14% 22% 0% 1% 7% 19% 19% 13% 5%

Note: Due to rounding, percentages may not add up to 100%

To begin, 98 regulations, or 14% of the total proposed WAC draft regulations, are designated a proposed weight of 0; these regulations will be exempt from enforcement as they are definitions or DEL and Provider Provisions18.

From the Focus Group process, 156 regulations are proposed as a risk-weight of 1. No regulations received a risk-weight of a 1 or 2 because of the survey. This means 22% of proposed WAC draft regulations are proposed to be rated a 1, and 0% are proposed as a 2.

Proposed risk-weights of 3-8 are results of survey responses. Regulations receiving a proposed weight of 3 come from 3 sections: Child Outcomes, Environment, and Professional Development and Training. These represent 9 proposed regulations, or 1% of the total proposed draft.

Overall, 7% of regulations are proposed to be weighted as a 4, with at least 1 regulation from every WAC section receiving this weight. The heaviest concentration comes from Environment, with 28 regulations.

All regulation sections, except Child Outcomes, contain a risk-weight of 5. The most predominant categories include: Environment with 76 regulations, and Professional Development and Training, and Program Administration and Oversight, each with 19 regulations. Regulations assigned a proposed weight of 5 make up 19% of the total regulations.

Environment, Interactions and Curriculum, Professional Development and Training, and Program Administration and Oversight comprise the categories where regulations were proposed weighted a 6 and 7, representing 19% and 13% of the total respectively.

Those regulations where the median risk-weight was an 8 account for only 5% of the total proposed regulations. These regulations come from Environment, Interactions and Curriculum, and Program Administration and Oversight.

Table 5 shows a detailed look at the percent distribution of risk-weights by WAC regulations section.

18 Example of a definition rated a 0 is 170-300-0401 (4): the definition of the annual licensing fee; an example of a Provider provision is 170-

300-0225 (1): stating a provider may have pets

15

Survey Results & Analysis Table 5. Percent of Risk Level by Section

WAC Regulation Section Total Regulations 0 1 2 3 4 5 6 7 8

Intent and Authority 38 68% 16% 5% 11%

Child Outcomes 4 25% 75%

Environment 314 4% 6% 1% 9% 24% 29% 19% 7%

Family Engagement and Partnerships 6 66% 17% 17%

Interactions and Curriculum 117 9% 41% 1% 11% 13% 16% 9%

Professional Development and Training 81 11% 25% 5% 12% 23% 16% 7%

Program Administration and Oversight 140 27% 41% 2% 14% 8% 5% 3%

Note: Due to rounding, percentages may not add up to 100%

Another method of reviewing the distribution of the proposed risk-weights is by risk level: lower (1-3), medium (4-5), and high (6-8). Table 6 categorizes the number of WAC regulations by risk level. While the percent of lower and medium risk level weights are fairly even, there is a concentration of WAC regulations receiving a higher risk weight. Survey Results & Analysis Table 6. Risk Level Representation

Risk Level Count Percent of Total (700)

Weighted 0 98 14%

Weight: Lower Risk (1-3)

165 23.5%

Weight: Medium Risk (4-5)

180 25.7%

Weight: Higher Risk (6-8)

257 36.7%

Note: Due to rounding, percentages may not add up to 100%

Next Steps

Rule Making (NRM) and public comment process to solicit feedback on both the proposed regulation and risk-weights. The NRM and public comments period will begin in May 2017 and end in October 2017. PCG will partner with DEL in facilitating the NRM and public comment process. Throughout this process, DEL will use continue to develop the new compliance oversight and licensing review process.

www.publicconsultinggroup.com