Making Sense of Making Sense of SensemakerSensemaker®®: :

Evaluating development Evaluating development initiatives through micro-initiatives through micro-

narrative capture and self-tagging narrative capture and self-tagging in Kenyain Kenya

Partos PlazaPartos Plaza

October 14, 2010October 14, 2010

Dr. Irene Guijt, Learning by DesignDr. Irene Guijt, Learning by Design

with John Hecklinger and Marc Maxson (GlobalGiving)with John Hecklinger and Marc Maxson (GlobalGiving)

Connecting Needs ChronologicallyConnecting Needs Chronologically

Learning by DesignLearning by Design• find a methodological option for ‘seeing’ emergencefind a methodological option for ‘seeing’ emergence

Cognitive EdgeCognitive Edge• test application in international developmenttest application in international development

Rockefeller FoundationRockefeller Foundation• fund methodological innovationfund methodological innovation

GlobalGiving GlobalGiving • generate non-intermediated target group feedback generate non-intermediated target group feedback

flowsflows

Options to Find and ExpressOptions to Find and Express

StatisticsStatistics• Decontextualised fragments (end results/inputs)Decontextualised fragments (end results/inputs)• Outliers ignoredOutliers ignored• Hard to get to explanatory levelHard to get to explanatory level• Work well within the known/knowableWork well within the known/knowable

StoriesStories• Few in-depth voices, sometimes purposively selected Few in-depth voices, sometimes purposively selected

(dilution)(dilution)• Biases of the intermediaryBiases of the intermediary• Representativeness assumed; validity questionedRepresentativeness assumed; validity questioned• Richness cannot be dealt with if many storiesRichness cannot be dealt with if many stories

But when you’re off the beaten But when you’re off the beaten path…path…

future doesn’t necessarily mirror the pastfuture doesn’t necessarily mirror the past ongoing sensing of multiple perspectives ongoing sensing of multiple perspectives

neededneeded real time feedback loopsreal time feedback loops

Stories, then Statistics in SenseMakerStories, then Statistics in SenseMaker®®

GlobalGiving’s M&EGlobalGiving’s M&E

self-generated reportsself-generated reports incidental evaluatorsincidental evaluators postcards from the fieldpostcards from the field

ButBut closer to userscloser to users continuouscontinuous not mediatednot mediated

Design FeaturesDesign Features

Numbers derive meaning from their contextNumbers derive meaning from their context Stories told by people are filters through which Stories told by people are filters through which

they make decisionsthey make decisions Mass capture of fragmented material, self Mass capture of fragmented material, self

signified at the point of origin gives quantitative signified at the point of origin gives quantitative datadata

Semi-constrained signifiersSemi-constrained signifiers Visualization most important human sense-Visualization most important human sense-

making capability (patterns, then stories)making capability (patterns, then stories)

Methodological LogicMethodological Logic

1.1. Prompting question or image Prompting question or image • triggers a lived experiencetriggers a lived experience

2.2. Storyteller Storyteller self-indexes self-indexes storystory• gives meaning, but can be done by othersgives meaning, but can be done by others

3.3. Software detect patterns Software detect patterns • visual patterns among storiesvisual patterns among stories

4.4. Sense-making in dialogue Sense-making in dialogue • people discuss patterns and story clusterspeople discuss patterns and story clusters

5.5. Act on patternsAct on patterns• Weak signals phase shiftsWeak signals phase shifts• Stimulate beneficial patterns; dampen undesirable Stimulate beneficial patterns; dampen undesirable

onesones

Designing Use of SenseMakerDesigning Use of SenseMaker®®

1.1. Develop signification frameworkDevelop signification framework• Content and ‘translation’Content and ‘translation’

2.2. Strategies for data collectionStrategies for data collection• Who, how, how often, how long, etc? Who, how, how often, how long, etc?

3.3. Data analysisData analysis• Where to nest learning? Where to nest learning? • How will analysis take place, who will be involved, How will analysis take place, who will be involved,

what is the rhythm of analysis (how often at which what is the rhythm of analysis (how often at which level?)level?)

4.4. Feedback loops with the system Feedback loops with the system • Who needs to look at what aspects of the initial Who needs to look at what aspects of the initial

analysis and how, in order to enable change?analysis and how, in order to enable change?

1a. 1a. Triggering Triggering

storiesstories

Example of a Story (GlobalGiving Example of a Story (GlobalGiving data)data)

This library is located in Kibera Otiende opposite the upgrading This library is located in Kibera Otiende opposite the upgrading project. It started some years back not more than three years. project. It started some years back not more than three years. Their library came because most of children in the community Their library came because most of children in the community lacked textbooks. They demanded that in school everybody is lacked textbooks. They demanded that in school everybody is entitled to to homework and in order the homework to be done entitled to to homework and in order the homework to be done and saw the way forward is to contribute together and share the and saw the way forward is to contribute together and share the problem equally and raise some money for the library problem equally and raise some money for the library construction then for the books. They struggled to raise the construction then for the books. They struggled to raise the money and at long last they raised and built a well equipped money and at long last they raised and built a well equipped nursery textbooks because the of the victims were primarians. nursery textbooks because the of the victims were primarians. Now the children are happy and also the load that parents had is Now the children are happy and also the load that parents had is reduced. The community is very proud because their children are reduced. The community is very proud because their children are passing and doing well during their national exams. That is a sign passing and doing well during their national exams. That is a sign of excel to parents and also children since they are making good of excel to parents and also children since they are making good use of the library and materials available. The library is there use of the library and materials available. The library is there upto date and still going strong.upto date and still going strong.

Insert short Rimisp storyInsert short Rimisp story

1b. Questions to 1b. Questions to self-indexself-index

Storyteller gives meaning to own storyStoryteller gives meaning to own story

a.a. TriadsTriads

b.b. PolaritiesPolarities

a.a. Multiple optionsMultiple options

b.b. ‘‘About you’About you’

The questionsThe questions

Probes for additional story context: 9 Probes for additional story context: 9 questions about each story; 5 questions questions about each story; 5 questions about storytellerabout storytellerThis community effort improved

…Social relations

Physical conditions Economic opportunities

2. Strategies for Data Collection2. Strategies for Data Collection

paper-based, then typed uppaper-based, then typed up 3000+ stories (2700 usable) in 10 weeks3000+ stories (2700 usable) in 10 weeks via 50 volunteers (snowball) for 1500$via 50 volunteers (snowball) for 1500$

Incentives matter, so does chasingIncentives matter, so does chasing

Target a focused time to trigger flood of Target a focused time to trigger flood of storiesstories

3. Data Analysis3. Data Analysis

Any combination of triad, polarity, MCQ, Any combination of triad, polarity, MCQ, demographicsdemographics

Explorer: glance, browse, compare, range, Explorer: glance, browse, compare, range, distribute, cluster, graph, landscapedistribute, cluster, graph, landscape

Stories (n = 2530) Stories (n = 2530)

#1 Food#1 Food #2 Shelter#2 Shelter #3 Safety#3 Safety #4 Water#4 Water #5 HIV/AIDS#5 HIV/AIDS #6 Education#6 Education

Experts/implementers (n=65)

Why self-signify?Why self-signify?

4. Feedback loops4. Feedback loops

Dissemination and institutional feedbackDissemination and institutional feedback PartnershipsPartnerships Community feedbackCommunity feedback Geo-taggingGeo-tagging Global GivingGlobal Giving

Next Steps for GGNext Steps for GG

1.1. Improving story collection: signification Improving story collection: signification framework & data uploading framework & data uploading

2.2. Experiment with different incentivesExperiment with different incentives

3.3. Expand geographic scope Expand geographic scope

4.4. Integrated technology - combined web-Integrated technology - combined web-SMS platformSMS platform

5.5. Cross-organization learningCross-organization learning

6.6. Use ‘needs’ stories for advocacyUse ‘needs’ stories for advocacy

7.7. Partner rewards programPartner rewards program

ApplicationsApplications

Organizational aggregation from diversityOrganizational aggregation from diversity Rolling baselines Rolling baselines Cross-organizational comparisonsCross-organizational comparisons Focused moments – MTRs, final Focused moments – MTRs, final

evaluationsevaluations

More informationMore information

http://www.cognitive-edge.com/blogs/news/2010/09/final_report_published_by_glob.php

http://www.youtube.com/watch?v=ZV-lcTuUawc&NR=1 http://www.sensemaker-suite.com/sensemaker-http://www.sensemaker-suite.com/sensemaker-

videos.htmlvideos.html