Monitoring & Evaluation of Social Marketing Programs

7/24/2019 Monitoring & Evaluation of Social Marketing Programs

1/38

Module 7Presented by Poppet Celdran

Monitoring & Evaluation ofSocial Marketing ProgramsReport


2/38

WHO. 27. !dvocacy Communication and Social Mobili"ation #!CSM$ for %uberculosis Control. ! andbook for

Monitoring & Evaluating!dvocacy Communications & Social Mobili"ationPrograms


3/38

'$ conducting s(ort)term and long)term monitoring and t 2$ recogni"ing problems via feedback from t(e +eld* and

,$ making mid)course corrections based on feedback.

Monitoring

-efore !dvocacy Communications & SoMobili"ation #!CSM$ activities begin/ cremonitoring mec(anisms to receive feedt(e interventions and identify any problearly.0


4/38

1t is impossible to anticipate every problem/ a monitoring system ill (di3culties 4uickly so t(at t(ey can be addressed. 5etermine t(e rolesin solving problems. -Process evaluation0 is t(e day)to)day monitoringactivities and operations. 5etermine t(e ob6ectives of a process evaluestablis(ing t(e monitoring mec(anisms.

%(e ob6ectives s(ould focus on

(et(er !CSM activities are on track/

(o close t(ey are to meeting t(e pro6ected timeline and budget/ an

(et(er sta8 members understand and perform t(eir roles correctly

'. Conducting s(ort)term and long)term monitoringtracking


5/38

9nce ob6ectives (ave been determined/ select monitoring and tracking mec(anism

suggested sources of information for tracking t(e various components of !CSM ac

inventory of materials*

distribution list*

activity reports*

public diaries*

television and radio logs*

media)clipping services*

sta8 surveys or focus groups* partner feedback*

timeline and budget assessments*

nes and information searc(es*

legislative tracking systems* and

attitude or (ouse(old surveys.


6/38

Responding to relevant information in real time allos a programme toimmediately/ rat(er t(an reali"ing in retrospect (at it s(ould (ave dofolloing strategies for e8ectively using feedback from t(e +eld can bimplemented

1nvolve key decision)makers/ stake(olders and % advocates in (elp

analyse and use feedback* :se process evaluation to uncover problems or opportunities for t(e

intervention during implementation*

Conduct preliminary evaluations to identify potential improvements and s(are successes before t(e completion of !CSM activities* and

:se summative researc( to make future !CSM programme decision

process rat(er t(an 6ust an end point.

2. Recogni"ing problems via feedback from t(e +el


7/38

Consider t(e feedback 4uestions belo (en deciding (et(er mid)coursare necessary.

ave goals and ob6ectives s(ifted as activities (ave been conducted; 1f soriginal goals and ob6ectives to meet t(e ne situation.

!re particular ob6ectives not being met by programme or !CSM interven


8/38

'$ types of evaluations* 2$ setting evaluation goals*

,$ determining suggested indicators*

=$ developing an evaluation plan*

>$ selecting monitoring and evaluation met(odologies*

?$ developing and pre)testing data)collection instruments* and

7$ riting an evaluation report.

Evaluation

!t some point all programs need to ask t(e 4uestion/ -o e8ectiv!CSM strategies;0 %(is is t(e time to re@ect on (at (as been ac(i(as orked and (at (as not/ and to make recommendations to imfuture e8orts. ! summative evaluation ill not only (elp to anser 4uestions/ but also to demonstrate to funders and partners t(e e8eactivities.


9/38

Outcome evaluation measures (o ellt(e !CSM intervention (as met its ob6ectives and (ats(ould be c(anged to improve future !CSM activities.

Aollo t(e steps belo to conduct an outcomeevaluation

determine (at information t(e evaluation must

provide* de+ne t(e data to collect*

decide on data collection met(ods*

develop and pre)test data collection instruments*

collect data*

process data*

analyse data to anser t(e evaluation 4uestions*

rite an evaluation report* and

disseminate t(e evaluation report.

1. Types of evaluations


10/38

Impact evaluation connects be(aviour c(ange to (easocial outcomes. !n impact evaluation ansers t(e 4uet(e people (o adopted ne actions or be(aviour eBpeimproved (ealt( and ell)being related to t(eir % statu

1mpact evaluations are not often used by planners of !Cactivities D mostly because of t(e (ig( costs usually invcarrying t(em out successfully. oever/ evaluators canfe programmatic ob6ectives to measure/ suc( as c(anrates of treatment completion or increases in case dete


11/38

%o set evaluation goals/ +rst determine (o ants to learn from t(e evaill be primarily t(e stake(olders involved in !CSM activities or t(ose interest in t(e programme.

9t(er interested parties D suc( as t(e broader community/ t(e various l(ealt( system/ people from t(e municipaldistrictregionalnational gove

donors D s(ould also be invited to suggest evaluation goals or participatays.

%(ere may also be people and organi"ations t(at are interested in learnevaluation alt(oug( t(ey may not (ave been directly involved in t(e proPotential future partners mig(t also be invited to participate/ to provideperspective and to furt(er t(eir understanding of t(e approac( used an

2. Setting evaluation goals


12/38

What are the ACSM o!ectives"


13/38

C(anges in t(e evaluation indicators/ or key outcomes to be measure(et(er ob6ectives (ave been ac(ieved. Select t(e indicators t(at ide!CSM initiatives (ave been t(e most successful or (ere additional needed. Fink indicators to t(e ob6ectives set during t(e planning proceindicators must be measurable.

1ndividual)level indicators measure knoledge*

attitudes*

beliefs*

be(aviour*

stage of illingness to c(ange* and

demograp(ics.

'. (etermining suggeste% in%icators


14/38

!t a minimum/ an evaluation plan s(ould anserseveral key 4uestions

a) What are the pro!ect*s o!ectives an%e$pecte% outcomes" What +uestions shoul%e as&e%"


15/38

c) Who #ill collect the information"

1t is more cost)e8ective to use eBisting (uman resources and platforms/ if pcollect information. 1deally/ sta8 s(ould not collect information t(at directlysuccess or failure of t(eir on e8orts.

%) What resources-materials #ill e nee%e%"

Fist (o needs support in collecting t(e information from di8erent sources asupport t(ey need.

e) When #ill this information e collecte%"

Create a timeline for data collection.


16/38

Aor t(e broadest vie of areas (ere t(e !CSM programme (most e8ective and t(ose (ere improvement may be needed4uantitative and 4ualitative researc( met(ods to collect eval

ualitative metho%s may include in)dept( intervies/ foc

or anecdotal feedback mec(anisms suc( as diaries and obsstudies #atc(ing people in a natural setting it(out t(eir aand observing t(eir be(aviour$.

uantitative metho%s may include sales data/ service stasurveys.

/. Selecting the monitoring an% evaluation metho


17/38

Most outcome evaluation met(ods involve collecting data on participants t(roug( observation/ 4uestionnaimet(ods. %o develop data collection instrumentsD or to select and adapt eBisting ones D ask t(e 4uestions

Which %ata"

%(e data collected s(ould be directly related to t(e evaluation 4uestions. Aor eBample/ if members of t(e need to kno more about a topic before be(aviour c(ange can take place/ ask knoledge)related 4uestio

0rom #hom"

Evaluators s(ould decide (o many members of eac( group are re4uired in t(e evaluation to measure c(

be ade4uate resources to collect information from t(e number of people re4uired. 5i8erent data)collectiomet(ods for di8erent groups may be necessary.

Ho#"

!ssess available resources before deciding (o to collect data. !re skilled intervieers accessible or can trained;


18/38

%o prepare an evaluation report/ (ave sta8 it( appropriate eBpertise aoutcome evaluation data and ork closely it( t(e evaluators to interpand develop recommendations.

!n evaluation report s(ould

present t(e lessons learnt in a clear format t(at can be easily digested(o may be planning future !CSM activities*

demonstrate accountability to employers/ partners and funding agenc

provide evidence of t(e e8ectiveness of t(e !CSM interventions and a

create a formal record to serve as an institutional memory of (at (as(ic( partners (ad strong skills or eBperience in speci+c areas/ (at pere encountered and (at successes ere ac(ieved.

3. Writing an evaluation report


19/38

Weiner4 5enay.

Monitoring & Evaluatingfor Social and e(avioral C(ange Commu


20/38

! monitoring and evaluation #M&E$ plan is a document t(at ouan implementation researc( pro6ect is monitored and evaluatelinks strategic information obtained from various data collectioto decisions about (o to improve t(e pro6ect on an ongoing b

%(e M&E plan serves several main purposes/ including stating (o ac(ievements of t(e programmepro6ect ill be

documenting consensus/ t(ereby encouraging transparency/accountability and responsibility*

guiding implementation of M&E* and

preserving institutional memory.

Monitoring an% evaluation plan


21/38

!n e8ective M&E plan s(ould conform to t(e folloing standards

Utility: 1t must be useful and serve t(e practical and strategic information neintended users for decision)making purposes/ t(ese may range from assessinperformance to allocating resources/ etc.

Feasibility: e realistic and practical. Given t(e scarcity of resources/ t(e M

make t(e best use of eBisting data collection systems. oever/ if ne data systems are involved/ resources #cost and tec(nical capacity$ must carefully

Ethically sound: !bide by et(ical principles it( regard to t(ose involved inby t(e M&E activities.

Accuracy: Provide tec(nically accurate and useful information for decision)mprogramme improvement.

Stan%ar%s for an M67 plan


22/38

%(ere are four key components t(at form t(e founupon (ic( t(e M&E plan s(ould be built. !nserifour corresponding 4uestions is critical to M&E pla


23/38

Stakeholder consultation and participation

Stake(older consultations and participation s(ould be regular occurrences t(rentire process of developing and implementing your M&E plan. %(ese consultdialogue/ a clear understanding of t(e pro6ect goals and ob6ectives/ and (o tassessed.

%(ey also ensure t(at various perspectives are understood and integrated/ anaut(entic needs are being met. Stake(older participation in t(e design of t(e facilitates t(e selection of appropriate and useful M&E indicators. Aurt(ermoremeasures to promote stake(older participation creates a sense of oners(ip responsibility among partners.

Stake(older involvement increases t(e probability t(at t(e information and reby t(e M&E plan ill be consistent it( t(eir eBpectations.

8ey steps in %eveloping an M67


24/38

Consensus s(ould be reac(ed on key 4uestions in t(e folloing areas

9What %o #e #ant to &no# at the en% of the pro!ectand

9What %o #e e$pect to change y the en% of the pro!e

!gain/ ansering t(e 4uestion of (at you eBpect your pro6ect to c(guide decisions about (at strategic information is needed for pro6ecmanagement decisions as ell as (at elements s(ould be monitoreevaluated in order to assess progress.

%(e rigor and scope of your M&E plan ill depend on (at you comm(at results or outcomes your pro6ect is accountable for.

Developing the M&E plan


25/38

5eveloping t(e M&E plan provides your team it( a clear picture of t(e follo

o pro6ect activities are linked to eBpected outputs/ outcomes and populatimpacts.

o di8erent types of information ill be collected and used by di8erent lev(ealt( system.


26/38

9nce your team (as developed t(e M&E plan/ de+ned t(e indicatoidenti+ed t(e data sources necessary/ t(e appropriate met(ods bycan be collected and analysed s(ould be determined.

Aor eBample/ your team s(ould determine (et(er you ill use eBicollection systems or if ne systems need to be developed.

Hour team must also determine (o information ill be recorded/ areported.

Aurt(ermore your team s(ould also carefully consider t(e resourcein terms of tec(nical competencies/ costs/ and time (en determinmet(ods and tools to be used.

Determining the M&E methodol


27/38

!fter developing t(e M&E plan/ t(e roles and responsibilitiedi8erent stake(olders s(ould be described clearly.

%(is step ill determine (o t(e M&E plan ill be speci+calimplemented and (at reporting system ill be adopted.

%(e implementation of t(e plan s(ould include t(e data col#i.e. (o is responsible for collection of speci+c data* ensurcontrol at eac( stage* (o often t(e data ill be collected* ft(e data #e.g. ra/ summary$* (at resources ill be re4uirstage* (o ill analyse t(e data$ and t(e dissemination pla

Assign responsibilities for

implementation


28/38

%arget s(ould be set in consultation it( all stake(olders so t(at everyone understanpro6ect (as committed to ac(ieve.

y setting targets/ you ill (ave a concrete measure by (ic( to 6udge (et(er t(e pprogressing as eBpected.

%(e process of target setting/ must focus on ansering t(e 4uestion

9What can realistically e achieve% given the resources an% the environmenpro!ect is operating":

%(e factors to consider include baseline levels* past trends* eBpert opinions* researc(as been ac(ieved else(ere* client eBpectations* and t(e capacity and logistics to a


29/38

ere are some practical considerations in planning information reporting and utili"ation planning

5esign t(e M&E dissemination plan around t(e information needs of t(e users. 1t is important to becontent and format of data reports ill vary according to t(eir intended use. Aor eBample/ is t(e M&monitor processes; %o conduct strategic planning;

%o comply it( re4uirements; elp identify problems; Iustify a funding re4uest; 9r to conduct an i

1dentify t(e fre4uency of data reporting needs. Aor eBample/ pro6ect managers may ant to revie

fre4uently to assess pro6ect progress and make planning decisions/ (ereas donors may need dataa year to ensure accountability.

%ailor t(e reporting formats to t(e intended audience. Since reporting may entail di8erent levels oftec(nical language/ t(e report format and media s(ould be tailored to speci+c audiences and di8erto solicit feedback.

1dentify appropriate outlets and media c(annels for communicating M&E data. %(is s(ould considerreporting/ suc( as regular pro6ect reports/ to management and progress reports to donors/ as ell areporting/ suc( as public forums/ nes releases/ brie+ngs/ and ebsites.

Dening reporting system! disseminatioutili"ation of results


30/38

1mplementation of t(e M&E plan occurs in t(ree stnamely

c(ecking and measuring progress*

analysing t(e situation* and reacting to ne events/ opportunities and issues

Implementing the M67 plan


31/38

1deally/ monitoring focuses on t(e pro6ects t(ree main c(aracteris4uality/ time and cost.

%(e pro6ect manager coordinates t(e pro6ect team and s(ould alaare of t(e status of t(e pro6ect.


32/38

%(e second stage of monitoring consists of analyssituation. %(e status of pro6ect development is comt(e original plan/ and causes and impact of potentdeviation are identi+ed.

!ctions are identi+ed to address t(ese causes andimpacts of any deviations.

Analysing the situation


33/38

1t is important to anticipate and react 4uickly to nesituations/ events/ opportunities and issues/ and tot(e possible actions to be taken.

1f appropriate/ various options are considered anddiscussed it( t(e pro6ect team and a decision is tregarding t(e most appropriate pat( to pursue.

$eacting to ne% events!opportunities and issues


34/38

%(e M&E plan s(ould be seen as dynamic and s(ould alays re@ect t(e reality of (aunderstood. Eac( time a deviation from t(e original M&E plan is identi+ed/ (et(er oany furt(er action/ t(e M&E plan s(ould be revised and c(anges documented accordi

ere are eBamples of 4uestions t(at can be considered to (elp your team assess (oplan is orking.

!re M&E activities progressing as planned;

!re t(e evaluation 4uestions being ansered su3ciently; ave ne evaluation 4uestions been raised and/ if so/ s(ould t(ey be incorporated

plan;

!re t(ere any met(odological or evaluation design issues t(at need to be addressed

!re t(ere any outside factors #political/ environment$ t(at are a8ecting t(e M&E pla

!re appropriate sta8 and funding still available to implement t(e M&E plan;

!re M&E +ndings being disseminated and used by stake(olders for decision)makingprogramme improvement;

Adustments to'updating the M&plan


35/38

;MI.

Monitoring & Evaluation ofCommunication and Social Mobili"ation Gu


36/38

Aolloing is a list of re4uired minimum monitoring and evaluation standards for communactivities.

5escribe t(e process of strategy/ materials/ and activity development and implementatmet(odology and (en t(e steps (appened. %(e folloing steps s(ould be described in (o/ (en/ (o many/ and target audience

!udience researc( including instruments/ met(odology/ and results

Strategy formulation

Print materials developed

roadcast materials developed

Focal communication c(annels

Pre)testing including instruments/ met(odology/ and results

Production

roadcast

5issemination9rientation

%rainingorientation

Supervision

Monitoring including instruments/ met(odology/ and results

Evaluation including instruments/ met(odology/ and results

Minimum Monitoring and Evaluation StandardCommunication !ctivities


37/38

%(e Rapid Monitoring C(ecklist belo gives eBamples of 4uestions t(at can be used to collect relevant monitorinof tools to collect t(e information/ and suggested approac(es for using t(e tools. !s mentioned earlier/ t(is c(ecreceive feedback on t(e interventions and to identify any problems early so t(at t(ey can be addressed 4uickly.

%(e Rapid Monitoring C(ecklist


38/38

WHO. 27. !dvocacy Communication and Social Mobili"ation #!CS%uberculosis Control. ! andbook for Country Programmes. C(apter

Weiner4 5enay. Monitoring and evaluation for social and be(aviourcommunication

PM1 Communication and Social Mobili"ation Guidelines#(ttp.pmi.govresourcespublicationscommunicationKsocialKmuidelines.pdf.$

References

Monitoring & Evaluation of Social Marketing Programs

Documents

Transcript of Monitoring & Evaluation of Social Marketing Programs