CSO SI METHODOLOGY REVIEW:CONCEPTUALIZATION AND

MEASUREMENT

S. Wojciech Sokolowski

Washington DC, June 10, 2015

Areas of Examination

• Appropriateness and clarity of concepts• Validity and reliability of measures• Scoring of measures • Weighting of measures • Periodicity of data collection• Comparability with other indices and data sources• Assorted technical issues

Overall CSO SI Assessment

Good job, but there is room for improvement:

• Clearer conceptualization of measured dimensions;• More objective scoring technique that uses Thurstone

scale employing factual True/False statements to calculate the score; and

• Use communication process based on Delphi method• Add a “map” of what is measured in target countries;

Finding 1: Comprehensive CSO Definition, but not followed in data collection

The current Index methodology uses an internationally accepted definition that encompasses a wide variety of CSOs, including:• health, educational, and social services providers• foundations, charities, and international NGOs• grassroots community organizations• business, professional and labor organizations,• cultural and religious organizations, • nonprofit cooperatives, and • other membership associations.

Civil Society Conceptualizations

NARROW SCOPE:

1. Defined by law (e.g. 501c3,

Registered Charities)

2. Defined by a specific function (e.g.

NGOs, CBOs,

foundations)

COMPREHENSIVE SCOPE:3. NPI/CSO sector (UNSD, CSO SI)

4. Social Economy (EU)

BROAD SCOPE:5. Third Sector

6. Sphere between households, market and state (CIVICUS)

Data collection does not reflect conceptualization

The existing practices of data collection and assembly have an unintended consequence of de facto limiting the scope to much more narrow subsets of CSOs that vary from country to country, which risks comparing apples to oranges.

Recommendation 1: Map civil society in target countries

Type of organization Any CSOs of this type? (Yes, No)

Approximate size of CSO in scope (number of people or organizations)

Information availability (Good Medium Low)

Associations  Yes  about 20,000 orgs  MediumFoundations  Yes  about 500 orgs  GoodCooperatives  Yes  about 800 orgs  MediumInformal/Community based organizations  Yes  about 1 million volunteers  Low

Educational institutions  No    Religious organizations  Yes  about 50,000 churches  LowHealth care organizations  No    Social assistance organizations  Yes  About 3,000 orgs  MediumYouth organizations  Yes  about 200,000 members  MediumHousing associations  No    Water/electricity supply orgs  Yes  about 200 orgs  GoodEnvironmental organizations  Yes    Labor unions  Yes  about 2 million members Medium Business/employer associations  Yes    Professional associations  Yes    Political advocacy/lobbying organizations  No    

Civil/human rights advocacy organizations  Yes    

Museums/historical sites/cultural orgs  Yes    

Micro-finance institutions  No    International cooperation/aid orgs  Yes    Paramilitary groups  No    Other (specify)      

LIH

Finding 2: Inadequate conceptualization of CSO sustainability

• CSO operating in different environments have different sustainability models, some emphasizing government grants and contracts, other market sales, and still other volunteer mobilization and private fundraising.

• CSO sustainability is not “one size fits all” but rather involves the ability of organizational leadership to develop effective strategies of dealing with environmental challenges.

• Applying the concept of sustainability to an entire organizational environment borders on the logical fallacy of composition.

CSO Sustainability Problem

WHAT IS SUSTAINABILITY?

Are Human Settlements Sustainable in Greenland?

CIVICUS Diamond: Constrained Space

STRUCTURE

VALUES

IMPACT

SPACE

CIVICUS Diamond: Low Impact

SPACE

STRUCTURE

IMPACT

VALUES

Recommendation 2: Rearrange existing CSO SI into 3-dimensional framework

Organizational Environment:

1. Legal2. Political3. Social

4.Institutional

Organizational Inputs:

1. Professional staff2. Volunteers,

members, supporters3. Financial resources

4. Technology and information

Composite CSO Index

scoreOrganizational Outputs and Outcomes:1. Service

2. Advocacy3. Interest

representation4. Social inclusion

and cohesion

L/MI/EH

Findings 3. Problematic Validity of 7 dimensions

Absent a clearly defined “target”- a causal model of CSO sustainability- validity assessment leads to a circular tautology. They measure whatever they capture and they capture whatever they measure.

Reliability and Validity of Measures

0.5 1 1.5 2 2.5 3 3.5 40

1

2

3

4

5

6

??

Recommendation 3: More rigorous validity test

Engage a team of experts to test the validity of the measures of the three dimensional conceptual framework.

L/MI/EL/M

Findings 5-7: Problematic scoring technique

• Scoring methodology is the most frustrating, challenging and controversial aspect of the Index production.

• Scores are often driving the process of data evaluation, rather than the other way around.

• The subjectivity of data collection and interpretation has been identified as the most serious challenge in the Index methodology.

Scoring Method

7-point Ranking Scale aka Likert scale

Recommendations 4-5: Revise scoring methodology

• Replace the existing ordinal Likert scale with an Equal-Appearing Interval Scale.

• Use Delphi method to improve communication process among panelists, experts, and EC members.

MI/ET

LIM

Example of Equal-Appearing Interval Scale

Hypothetical indicators of legal environment TRUE FALSE CONF*.

1. The right of association is guaranteed by law or constitution.   5  

2. The law limits allowable purposes of CSO beyond general legality, morality or public order. 3    

3. Unincorporated or unregistered CSOs are legally permitted to operate.   3  

4. The law gives government agencies the authority to dissolve any CSO for any reason. 2    

5. The law provides severe penalties for operating unregistered CSOs. 4    

6. CSO registration process is burdensome and costly. 1    

7. The law does not allow appeals from decisions of administrative bodies/government agencies. 2    

8. The law gives government agencies the authority to alter CSO missions or operations. 2    

9. Appealing government/administrative decisions is burdensome and costly. 1    

10. CSOs are legally allowed to obtain funding from any generally legal source.   1  

Example of Delphi MethodR

OU

ND

1Panelists and experts answer questionnaire

RO

UN

D 2Panelists and

experts provide and review feedback from each other, amend answers if needed

RO

UN

D 3EC reviews

answers and provides feedback; feedback internally reviewed and coordinated

RO

UN

D 4EC feedback

reviewed by IPs, panelists and experts; answers amended if needed

RO

UN

D 5Answers

finalized and submitted for scoring

Other Recommendations

• Keep weighting all dimensions equally• Keep annual data collection and

publication• UNSD and ILO recommend compiling

data on CSOs and volunteering; take advantage of them when available

• Keep the USAID “brand” – it is a mark of high quality

LIM

CSOs in National Accounts Statistics

Thailand

Czech Republic

Portugal

Mexico

Kyrgyzstan

Cameroon

Brazil

Norway

France

Australia

Japan

New Zealand

Belgium

United States

Mozambique

Israel

Canada

AVERAGE

0.0% 1.0% 2.0% 3.0% 4.0% 5.0% 6.0% 7.0% 8.0% 9.0%

1.6%

1.9%

2.0%

2.2%

2.3%

2.5%

3.4%

4.5%

4.7%

4.9%

5.2%

5.3%

5.8%

6.6%

6.7%

7.1%

8.1%

4.5%

Percent of GDP

Source: JHU Center for Civil Society Studies

THANK YOU