Engagement in PCORnet 1.0:Results of the Engagement Assessment Project

Sharon F. Terry, MA

Andrea Goodman, MSW, MPH

Meanwhile, at today’s Feline Engagement Committee meeting, decisions were made on how to increase feline recruitment.

Project Background

PCORnet has prioritized engagement as a key function in transforming healthcare research

However, there was little investment in capturing and scientifically assessing engagement activities in PCORnet

PCRF is definitive on this topic – engagement is a critical tenant

GA CC set out to capture, review and publish the resources, best practices, and lessons learned from PCORnet and its networks

3

Project Overview and Goals Capture and assess engagement activities across the Network to learn and synthesize:

§ Engagement methods, tools, and models from across the Network

§ Successes, challenges, and lessons learned from PCORnet engagement activities

§ Gaps and opportunities for systematic, effective, and measurable engagement

§ Indicators of successes and challenges § A recommended framework for evaluation to assess impact§ Overall knowledge and capacity to execute and evaluate

engagement

4

Methods• Surveyed all 35 PCORnet networks to identify engagement goals,

strategies, activities, challenges, and lessons learned

• Interviewed representatives from each network after the initial survey to

probe deeper around gaps, challenges, and specific engagement

practices

• Analyzed survey data using Qualtrics and Microsoft Excel to produce

descriptive and inferential statistics

• Coded, categorized, and organized qualitative data into emerging

themes using NVivo, a qualitative data analysis computer software

package

• Data included interviews, network reporting, Commons resources,

past presentations, and prior open-ended survey results

5

A Snapshot of Respondents100% response rate for surveys and interviews (13 CDRNs, 20 PPRNs, and 2 HPRNs)

64 individuals participated in the interviews

Diversity of participant types:

6

Who Networks Are EngagingNetworks reported involving a variety of stakeholders across some aspects of network operations and research activities

All networks are engaging patients/participants (100%) and researchers (100%), the majority are engaging clinicians (97%) and systems leaders (94%), and some have also said they involved payers (47%)

7

Engagement Strategies and Definitions

Most networks (34/35) reported developing their own strategy for

engagement

60% use a formal framework and 86% use some tools or

resources

100% of networks use a formal definition of engagement (many

report using the PCORI or PCORnet definition, or both)

8

Different Approaches

VS

PPRNs

CDRNs

Reported Success of Engagement

0 2 4 6 8 10

Patients

Clinicians

Organizationalor Systems Leaders

Researchers

Health Plans/ Payers

Reported Success of Engagement by Stakeholder Group

PPRNs CDRNs HPRNs

Factors for network self-rating:

� How far they have come in a somewhat new research paradigm� Comparison against what they thought other networks were achieving� Experiences and satisfaction of the stakeholders � Variety of stakeholders and variety of activities they were engaged in� How they “felt” is was going, many sharing pride in the work they did� Failures, shortcomings, and plans for engagement that were never realized

Reported Success of Engagement (Cont.)Most Successfully Engaged Stakeholder Group by Network Type

11

Least Successfully Engaged Stakeholder Group by Network Type

65% of PPRNs reported they were most successful at engaging patients/participants, compared to 38% of CDRNs

Overall, networks report being least successful at engaging health plans/payers (60%) and clinicians (31%)

Rating the Success of Engagement Methods

“The type of outreach needs to be aligned with the medium… there are certain things that you can do in person that you can’t do digitally or do well digitally that you couldn’t do in person… it’s not a perfect one-to-one.”

-PCORnet CDRN

Themes of Engagement Methods: Reasons for Success

13

In-Person(e.g. conference or meeting)

• Builds trust and relationships

• Promotes active participation

• Helps engage ancillary partners with project

• Reach people that may not have been reached via other methods

• Supports creative thinking and brainstorming

Digital Tools (e.g. portals or surveys)

•More efficient in terms of time and resources

•Get answers to specific questions or input on topics

•Get candid feedback about stakeholder experience and engagement approaches

•More easily reaches a large number of stakeholders

•Allows for more representative group to be involved

•Increases visibility

Social Media(e.g. Facebook or Twitter)

•A good place for community listening

•Useful for recruitment and network registration

•Easy for community to control

•Useful for dissemination

•Can engage large groups and target specific communities or groups of individuals

•People are already comfortable and engaged with platform .

Email

•Easy to maintain regular contact

•Customized emails work to let stakeholders know you are paying attention to them

•Can get statistics about open rates and clicks

•Works well with established members of network

•Easy to share seamlessly across network members

•Get input quickly--especially useful for busy people

Phone Calls

•Helpful to reach underserved populations with limited internet access

•Used to build initial connection

•Regular calls are a good way to share updates and get feedback.

•Stakeholder preference: "People want phone calls."

•Good alternative to in-person meetings for collaboration

Themes of Engagement Methods: Challenges & Drawbacks

14

In-Person(e.g. conference or meeting)

• Time and

resource

intensive

• Can limit

consistency of

stakeholders/

individuals

involved

• Hard for people

across the

country to

gather regularly

Digital Tools (e.g. portals or surveys)

• Can add

additional cost

• Limited to those

with internet access

• Competing for

attention-- may

not be a priority

to engage, surveys may

have limited

response rates

Social Media(e.g. Facebook or Twitter)

•A lot of competition

for attention

•Limited exposure of messaging without

paid "advertising"

•Time to develop a social media

campaign

•Can be confusing if

network does not

have strong brand in community

•Needs to be

monitored and policed

Email

•May only reach a small percentage of those sent messages

•A lot of competition for attention

•Cannot reach those who don't actively use email or have no internet access

•Feels less personal, limited opportunity to build trust and relationships

Phone Calls

• Time intensive

(compared to low

touch methods)

• Not as engaging as in-person

meetings

• Length and

frequency of

phone meetings can be tricky

Stakeholder Involvement Across Phases

15

Phases of Research

# of

Net

wor

ks

Rep

ortin

g In

volv

emen

t

“…there have been times where our researchers just don’t realize that they actually are going to need patient or clinician stakeholder involvement.”

- PCORnet CDRN

Levels of Involvement

Challenges of EngagementAll networks reported some challenges associated with the development or implementation of engagement, most frequently:

Time – More time is needed to engage effectively

Resources – Lack of staff and monetary resources that quality engagement requires

Diversity - Lack of diversity, especially within the patient/participant stakeholder group

Navigating Roles - Challenges related to stakeholders navigating their own role and that of other stakeholder groups

Collaboration - Frustrations related to a general lack of stakeholder involvement or lack of collaboration

17

Engagement Impact

No significant difference between ratings by PPRNs and CDRNsNetworks report that engagement efforts resulted in stakeholders influencing many areas of the network and research activities (see table) Stakeholder impact was harder for networks to clearly define § Examples of some impacts reported

include:• Intrapersonal changes (i.e., how

individuals approach the work) • Structural changes (i.e., how the

infrastructure was set up to support the work)

18

Networks report stakeholders influencing:

culture network decisions and utilization

dissemination of information recruitment

execution of research or projects

relationships and partnerships

data sharing funding and applications

Overall, networks rated the impact of engagement on network operations at 4.2 with a range of 2-5.

Measuring Engagement For many groups, engagement was a binary variable – success was determined by the presence of engagement

Networks rely heavily on PCORnet milestones as a benchmark of successful engagement

Examples of quantitative metrics reported include: the number of stakeholders involved, the number of engagement activities, and/or the number of resources or outreach efforts

Most networks reported measuring engagement monthly or quarterly

19

Ways Networks Measure Engagement

PCORnet Logic Model

20

Inpu

ts

- Resources (e.g. time, staff, funding)- Expectations;

stakeholder or network specific and those set

by PCORnet milestones

- Strategy developed to navigate goals and

needs of network and research

- Leadership (e.g. network directors,

national leaders and support)

Act

iviti

es- Advisory group and governance

committee meetings- Stakeholder Trainings

- Opinion Surveys-Listening sessions, community engagement studios, and focus

groups-Stakeholder interviews - Network and Registry

Recruitment- Email Updates

- Conference and Workgroup meetings

- Webinars- Social Media Outreach

- Inclusion of stakeholders in research cycle (e.g. using online tools to prioritize study topics and

research questions) - Inclusion of stakeholders in network infrastructure (e.g.

participant/patient Investigators and governance committees)

Out

puts

- # of stakeholders involved- # of meetings held

- # of patients trained- # of stakeholders reached via

email, social media, etc.- # of voices captured to inform

research- # of stakeholders in governance

- # of people recruited into networks, registries, or research- # of developed engagement resources-and infrastructure

# of stakeholder-informed research questions and proposals - # of studies with results shared

in accessible language - # of meaningful relationships

developed

Impa

cts Observed Impacts

- Increase in project funding- Added value to research teams- More patient-centered research methods and approaches- Preliminary shift in culture of researchAnticipated Short-Term Impacts*- Increase in patient recruitment and retention- Increase in research relevant to patients and stakeholdersAnticipated Long-Term Impacts*- Increase in research translated into care improvements- Improved health outcomes

Compiled Engagement Resources

Type of Resource Example # Identified

# Networks Contributing

Engagement Frameworks & Strategies Engagement Key Driver Diagram

10 11

Evaluation Models and Tools Member Survey 18 13

Methods for Engaging Stakeholders Recruitment Guide 24 16

Outreach and Comms Tools Acronym Guide 29 14

Stakeholder Roles & Governance Engagement Committee Charter

40 19

Training & Orientation Overview PowerPoint; Trainings

6 5

21

Network members mentioned 127 engagement resources

Full directory of resources available on Commons: PCORnet Engagement Assessment Project: Findings and Recommendations

Conclusions

Networks have developed and implemented a variety of methods for engaging stakeholders in their network-building and research activities

Few are formally measuring or evaluating effectiveness or success of these methods and activities in a way that looks at nuance, quality, or impact

Networks largely see engagement as a binary variable, measured by the presence or absence of it

While anecdotes can be vivid and compelling, for now, the most common quantitative measures that PCORnet networks use to report success are process measures (e.g. the number of stakeholders involved in a project, the number of meetings held with stakeholders, the number of stakeholders in governance, etc.)

22

ConclusionsNetworks often cite needing additional funding, staffing, and resources to adequately support engagement efforts

Networks articulate a desire to learn from each other and wish there were more opportunities to collaborate and share best practices, lessons learned, and resources

However, networks are also hesitant to formally share materials, approaches, or tools until they are perfected, published, vetted and branded by their community

While many networks discuss the importance of engaging a diverse and representative group of stakeholders, few offer attempted solutions or strategiesto overcome this challenge§ Networks have not consistently captured or reported on specific approaches § There are few data showing what is or is not successful in this area

23

Recommendations: Standards for Stakeholder Engagement

A diverse and project-appropriate group of stakeholders should be involved in all network and project governance; including holding positions with the highest decision-making authority

All projects should have continuous patient/participant leadershipthroughout all stages of research and involve a diverse and representative group of appropriate stakeholders throughout the research cycle

Patient/participant and stakeholder engagement should be appropriately planned and budgeted for at the beginning of a research project

24

Recommendations: Impact of Approaches and Methods

On the Network level, evidence-informed approaches should be indexed, catalogued, tested, and modified based on post-implementation evaluation

Use evidence-informed approaches to build engagement strategy and conduct regular assessments to ensure the needs of both the scientific and stakeholder communities are being met

Gather evidence to document the value engagement brings to specific aspects of the research cycle (e.g. recruitment, retention, and participant satisfaction)

Embed evaluation activities across network initiatives§ Document if and how activities have led to an increase in patient

recruitment and retention§ Document if and how activities have led to an increase in research

relevant to patients/participants and other stakeholders

25

Recommendations: Expectations for Collaboration and Growth

A shift in culture is required on the Network level, including both carrots and sticks to align around “people-centric” methods and measures

Networks should be expected to share resources, collaborate on solutions, and implement promising approaches

Spaces (both online and in-person) should be resourced for these collaboration efforts

Assess representativeness of Network and national-level stakeholder groups, the depth and breadth of reach, and the diversity of teams and partnerships

Provide opportunities for entire teams to participate in continued training in Patient Centered Outcomes Research principles, patient engagement, and team dynamics

26

Recommendations: Outreach, Information Sharing, and Transparency

Conduct expansive outreach to patient/participant community that includes multi-modal methods of communication and leverages innovative partnerships

Return study results to stakeholders in a timely and appropriate manner, using channels indicated as most valued by the stakeholder community.

Increase transparency and invest in approaches that increase participant awareness of the research ecosystem including research agendas, priorities, processes, and data use.

27

Going Forward: Raising the Bar for Engagement

Build consensus around engagement definitions, strategies, and minimum requirementsPromote a culture of collaboration with wider sharing of tools and resourcesStandardize expectations around the level of involvement of stakeholders throughout the study cycleImplement universal metrics and a standard approach to evaluating quality and impact of engagementLead the development and advancement of engagement science

“…[engagement] has to start from the ground up

rather than coming at it after

the fact [to] fix things that aren’t

working.”

- PCORnet CDRN

Questions

?!

30

31